Letter from Britain 12 November 2008
To the American pollsters and the media which (mis)report them
Never have so many trees been sacrificed to produce acres of newsprint and hours of broadcast media time wasted on anything before as the sum total of sceptical reporting of opinion polls in the America presidential election. It seems that every pundit and commentator has taken a swipe at the polls during the recent election. Yet on the night, triumph!
As an ‘outside/insider’, permit me to offer my congratulations and admiration for the pollsters of America. The American pollsters’ final tallies have now been examined by me and my team at Ipsos MORI in the light of the outcome. In all, we have been able to source 19 different eve-of-poll data sets, reported on or before November 3rd..
Remarkably, the American polls have come up trumps as never before.
All 19 sets of share figures from the American eve-of-election polls fell within a margin of plus or minus three percent. In fact, 18 of the 19 polls were within plus or minus two percent, the best record ever. The table can be found at www.ipsos-mori.com.
After days of working to collect, confirm and standardise the ‘final’ polling figures to make sense of the disparate ways American polling organizations conduct their political polls and report them, once again we have arrived at what we consider the definitive list of final polls (but would be glad to have evidence of any others we’ve missed)
We’ve scoured all the wonderful web sites which served us so well during this election, including the pollsters’ own sites, the media’s and other clients’ sites, and the now famous compilation sites including www.realclearpolitics.com, www.538.com,www.270togo.com, etc.
The hyperlinks to the pollsters’ final reports are in the box below, in the order that after standardising them we rank them in terms of error on share for Obama and McCain, averaging the two. Where two or more share errors tied, we used error on lead as the tie break.
We were not able to use, as we do in Great Britain, a three or even four (for Scotland and Wales where they have separate ‘national’ parties in addition to Labour, Conservative and Liberal Democrats) party accuracy measure. We do not rank them to present a ‘league table’ so much as to evaluate the performance of the polls generally in as objective a manner as some 40 years of political polling experience provides.
Nor do the American pollsters report like for like in other ways. Several did not ask so could not report the share figures for other candidates, and in some cases where they asked the question they failed to report the results in their press releases. Further, several of the press releases/final poll reports did not report the technical details, sample size, fieldwork dates, etc. As a result, it took us nearly as many days to finalise our data analysis as it did for the American election results to be counted.
Hyperlinks RasmussenReports Ipsos/McClatchy Diageo/Hotline Pew Research Daily Kos/Research 2000 Fox News,Opinion Dynamics YouGov/Polimetrix NBC/Wall Street Journal American Research Group Inc Democracy Coprs/Greenberg QR Marist Harris Interactive IBD/TIPP CNN/Opinion Research ABC/Wash Post CBS News Reuters/CSPAN/Zogby Gallup GWU/Battleground
Nearly 40 years ago the legendary founder of Opinion Research Centre in London, later the protégée of Lou Harris as President of the Harris Poll, Humphrey Taylor, called together the major British pollsters to establish ground rules they would agree to use to improve the reporting of their political opinion polls in the British media, and to clarify the role of voting intention results as not predictions of election outcomes days, months or even years hence but as commentators at the racetrack.
At a stroke, the better journalists and more responsible media improved their reporting and interpretation of poll findings to the benefit of their readers and viewers/listeners and each other. Some journalism departments even began the teaching of “Understanding and Reporting British Public Opinion”, and seminars have been held before every British General Election, most in the House of Commons, for the so-called “press lobby” of journalists who frequently report and inevitably comment on poll findings as the election progresses. These have proved popular, and have been well attended.
American pollsters have no such common basis of reporting their findings, certainly leading to me, and probably to many of them, being asked throughout the campaign, “How can I believe the polls if two companies’ final polls had Obama at 50% and yet he got 53% on the day; it can’t just be sampling error, can it?” My answer, “No, it isn’t”. In fact, if the two polls had reallocated to take account of the fact that they both reported 5% of their samples were ‘don’t knows’, they both were spot on, showing Obama with 53%, the outcome on the day.
All British pollsters now follow this convention, year in and year out, so that no one is any longer confused by different polling companies reporting on a different basis, and their findings can be used to compare with the election result four years before, and demographic comparisons.
In addition, having comparable ways of reporting their figures enables them and others to calculate ‘swing’, the statistic universally used in reporting British elections and polls which allows constituencies’ (states in the USA) results to be directly compared to each other and to the national result as well as to the previous party performance at previous elections.
This will be the subject of my next Letter from Britain.
Sir Robert Worcester Founder, MORI