Going Global? April 02, 2008 12:26 PM A poll put out by the BBC World Service that's getting some attention today headlines a 4-point improvement in positive views of the United States in the 34 countries in which it was conducted. But before jumping on the result, consider the methodology.
The poll's disclosure notes that nearly half the samples – 16 of the 34 – were urban-only, with noncoverage of the full adult national population ranging from 26 percent to 95 percent, averaging 66 percent (leaving aside India, where the coverage is unclear). That's the share of the national population in each country that had no chance of being selected for participation in the survey – a high to extremely high level of noncoverage.
Does it matter? Unknown, without full national data to compare. But as I always say, imagine an urban-only poll in the United States: President Kerry would love the results. The point is that we wouldn't accept urban-only coverage in a U.S. survey – which raises the question of why we'd regard it as acceptable in other countries.
There are other points to consider. The results have been aggregated across countries without weights that would make them proportional to population size. That means the approximately 1,000 urban-only respondents in Costa Rica (which has a national adult population of about 3 million) carry the same weight in the total as the 1,000 urban-only respondents in China, with about a billion adults.
Figuring out how to weight this kind of poll proportionally would be an impressive challenge in itself, given the range of population coverage. But simply aggregating data from countries of such disparate size hardly seems a solution; samples as a rule strive for proportionality to population distribution.
The poll is headlined as reporting "global" views, and its 34 countries do cross the globe. But with 34 included, that leaves about 160 other countries (depending on how you define the term) left out. The thought process behind the choice of these particular 34 isn't entirely clear. The argument that the biggest-population countries are included seems confounded both by the reported noncoverage in some of those countries (95 percent in Indonesia, 85 percent in Brazil, 57 percent in China) and the fact that the results from smaller-population countries are given equal weight.
The poll report is based on a single question, asking people their positive or negative views of several countries (half the respondents in each country were read the names of seven countries; the other half, seven other countries). The poll's analysis suggests that views of the United States may be improving based on "hope that a new administration will move away from the foreign policies that have been so unpopular in the world."
It'd be good to know the data supporting that hypothesis, since others may be equally plausible – beyond the fact that a 4-point change, from 28 to 32 percent positive, is a pretty thin reed in this kind of project. (It's a single country, but in our latest national poll in Iraq, nearly as many said a new U.S. administration will make things worse there, 27 percent, as better, 33 percent, with the rest expecting no difference.)
Based on our view that not all data are created equally, our longstanding policy at ABC News is to vet polls our programs are considering for air - sampling methodology, overall results and analysis alike - to try to ensure that they meet our standards and are reported in context. Admittedly we miss a few from time to time, but for details on what we strive to accomplish, see our standards and methodology page.