There may have been no such shift of moderates to conservative.
While the total turnout rate (based on VEP) increased by about 2.6%, the turnout for Southern states increased by about 5% and for the swing states by about 7%.
So it may well have been that while more liberals came out, even more conservatives also came out, and moderates didn't. In CA, total turnout dropped by 9%, in WA by 14%, in NY by 2%, etc.
[Data from GMU's elections site, and I guess the numbers for 2004 are provisional, as they don't have data for "Total Votes", only for "Vote for Highest Office", presumably meaning the counted votes]