This morning we made public our complete report evaluating the 2004 exit polls. It is available on www.exit-poll.net Also available there are methods statements, questionnaires and completion rates.
The individual respondent level data for all 51 states and the nation are being sent today to the Roper Center at UConn and to ICPSR at UMichigan. It will be available as soon as they can set it up. We are told that will be about two weeks for Roper and a month or more for ICPSR.
The executive summary from the report is posted below.
Thanks for your patience since the election while we put this together. It took a number of us a fair bit of work to put this together.
Joe Lenski and Warren Mitofsky
Executive Summary
On November 2, 2004, the Election System created by Edison Media Research and Mitofsky International for the National Election Pool (NEP) produced election estimates and exit poll data for analysis in 120 races in all 50 states and the District of Columbia. In addition, between January and March 2004, Edison and Mitofsky conducted exit polls for 23 Democratic Primaries and Caucuses. For every election, the system delivered on its main goals: there were no incorrect NEP winner projections, and the exit poll data produced on election day were used on-air and in print by the six members of the NEP (AP, ABC, CBS, CNN, FOX and NBC) as well as several dozen media organizations who subscribed to that data. However, the estimates produced by the exit poll data on November 2nd were not as accurate as we have produced with previous exit polls.
Our investigation of the differences between the exit poll estimates and the actual vote count point to one primary reason: in a number of precincts a higher than average Within Precinct Error most likely due to Kerry voters participating in the exit polls at a higher rate than Bush voters. There have been partisan overstatements in previous elections, more often overstating the Democrat, but occasionally overstating the Republican. While the size of the average exit poll error has varied, it was higher in 2004 than in previous years for which we have data. This report measures the errors in the exit poll estimates and attempts to identify the factors that contributed to these errors.
The body of this report contains the details of our analysis of the performance of the exit polls and the election system. In addition to the information included in this report, exit poll data from this election is being archived at the Roper Center at the University of Connecticut and at the Institute for Social Research at the University of Michigan and is available there for review and further analysis. This is the procedure that we have followed for all previous exit polls, which are also available at the Roper Center and ISR. The description of the methodology of the exit polls has already been posted on our Web site <http://www.exit-poll.net/>t - along with all questionnaires used on election day and the completion rates nationally and by state.
Here is a brief summary of our findings:
1. Exit Poll Estimates
The exit poll estimates in the 2004 general election overstated John Kerry's share of the vote nationally and in many states. There were 26 states in which the estimates produced by the exit poll data overstated the vote for John Kerry by more than one standard error, and there were four states in which the exit poll estimates overstated the vote for George W. Bush by more than one standard error. The inaccuracies in the exit poll estimates were not due to the sample selection of the polling locations at which the exit polls were conducted. We have not discovered any systematic problem in how the exit poll data were collected and processed. Exit polls do not support the allegations of fraud due to rigging of voting equipment. Our analysis of the difference between the vote count and the exit poll at each polling location in our sample has found no systematic differences for precincts using touch screen and optical scan voting equipment. We say this because these differences are similar to the differences for punch card voting equipment, and less than the difference for mechanical voting equipment.
Our detailed analysis by polling location and by interviewer has identified several factors that may have contributed to the size of the Within Precinct Error that led to the inaccuracies in the exit poll estimates. Some of these factors are within our control while others are not.
It is difficult to pinpoint precisely the reasons that, in general, Kerry voters were more likely to participate in the exit polls than Bush voters. There were certainly motivational factors that are impossible to quantify, but which led to Kerry voters being less likely than Bush voters to refuse to take the survey. In addition there are interactions between respondents and interviewers that can contribute to differential non-response rates. We can identify some factors that appear to have contributed, even in a small way, to the discrepancy. These include:
* Distance restrictions imposed upon our interviewers by election officials at the state and local level
* Weather conditions which lowered completion rates at certain polling locations
* Multiple precincts voting at the same location as the precinct in our sample
* Polling locations with a large number of total voters where a smaller portion of voters was selected to be asked to fill out questionnaires
* Interviewer characteristics such as age, which were more often related to precinct error this year than in past elections
We plan further analysis on the following factors:
* Interviewer training and election day procedures
* Interviewing rate calculations
* Interviewer characteristics
* Precinct characteristics
* Questionnaire length and design
We also suggest the following changes for future exit polls:
* Working to improve cooperation with state and local election officials
* Improvements in interviewing training procedures
* Changes in our procedures for hiring, recruiting and monitoring interviewers
Even with these improvements, differences in response rates between Democratic and Republican voters may still occur in future elections. However, we believe that these steps will help to minimize the discrepancies.
It is also important to note that the exit poll estimates did not lead to a single incorrect NEP winner projection on election night. The Election Night System does not rely solely on exit polls in its computations and estimates. After voting is completed, reported vote totals are entered into the system. Edison/Mitofsky and the NEP members do not project the outcome of close races until a significant number of actual votes are counted.
As in past elections, the final exit poll data used for analysis in 2004 was adjusted to match the actual vote returns by geographic region within each state. Thus, the discrepancy due to differing response rates was minimized and did not significantly affect the analysis of the vote. The exit polls reliably describe the composition of the electorate and how certain demographic subgroups voted.
2. Survey Weighting
Early in the afternoon on November 2nd, preliminary weightings for the national exit poll overstated the proportion of women in the electorate. This problem was caused by a programming error involving the gender composition that was being used for the absentee/early voter portion of the national exit poll. This error was discovered after the first two sets of weighting; subsequent weightings were corrected. This adjustment was made before NEP members and subscribers used exit poll results on-air or in print.
After election day, we adjusted the exit poll analysis data in three states (Tennessee, Texas, and Washington) to more accurately reflect the proportion of absentee ballots that came from each geographic region in those states. We have implemented a change to the survey weighting program to take into account the geographic distribution of the absentee votes in the future.
3. Technical Performance
While the computer system performed well for most of the night, a database server problem led to NEP member and subscriber screens “freezing up” shortly after 10:35 PM ET election night. This problem caused disruptions in the system until shortly after midnight when we switched to a backup server for the rest of the night. There was a second occurrence of this problem at approximately 2:45 AM ET. Details of the data server problems and other technical issues are outlined in the technical performance report being distributed to the NEP Technical Committee. We have isolated the reasons behind the database server problem and list several recommended technical changes in this report to help avoid a repeat of this problem in future elections.
MITOFSKY INTERNATIONAL 1776 Broadway, Suite 1708 New York, NY 10019
212 980-3031 212 980-3107 Fax
www.mitofskyinternational.com mitofsky at mindspring.com