[lbo-talk] Ruy Teixeira on why most of the polls are crap

Wojtek Sokolowski sokol at jhu.edu
Thu Sep 23 10:49:08 PDT 2004


Doug:
> But they're supposed to compensate for this in the weighting scheme,
> or at least try to.
>

Doug, I think you put too much confidence in this whole business. I am not saying that surveys can't be useful, but more often than not they are what sociologists call "sentiments of rationality" - a behavior that pays lip service to rationality and performs rationalistic rituals like expressing everything in numbers because rationality has a high cultural value, but these numbers are de facto based on witchcraft and spin.

The chief reason why an opinion poll based on a sample of N=1,000 can be witchcraft rather than solid science is that the sample need to be representative of the opinions in question, which means that the holders of these opinions must be selected with known - but not necessarily equal - probability. Only then weighting i.e. compensating for those unequal probabilities makes sense.

The problem is, however, that the distribution of opinions is not known - if it were, we would need a survey. In that light, the only way to weight a sample is to use known characteristics of the respondents, such as sex, age, residence, etc. As you can easily see, that approach rests on an implicit assumption that the relationship between these observable characteristics and opinions being sought is determined and constant. Without such an assumption the whole procedure is not worth the paper on which it is written.

To illustrate that with an example, suppose that you want to draw a pre-determined sample of black and white chips in complete darkness, so we cannot see the color of the chip we are drawing. To compensate for that, we can touch each chip and knowing that white chips have a different texture than the black ones, and based on that knowledge we can assemble the desired sample. But that procedure will work only insofar as what we know about the relationship between texture and color holds. If for some reason it changes, i.e. both white and black chips have similar texture - the sample drawing procedure in question will fail and we cannot compensate it by weighting - which necessarily involves texture.

There are plenty of reasons to believe that the previously established connections between socio-demographic characteristics and opinions do not hold anymore: 9/11, economics, more aggressive propaganda, growing class and cultural polarization, to name a few. To give you an example - my wife just talked to an African American friend of his, a female social worker living in Philadelphia, strong civil rights advocate etc. - one would say, a sure Democratic vote. To her big surprise, my wife learned that this woman wants to vote for Bush - citing reasons such as "with all what's going in Iraq, we cannot change horses in the middle of the stream." In short, all what social and political scientists used to tell us about connections between socio-demographics and voting preferences does not seem to hold, at least in this case.

Another example. I frequently monitor the Yahoo news site, which contains a forum (not moderated) to comment on the current news. In the past, this was the site of raging testosterone, liberal bashing, and jingoism. To be sure, that is still going on, but these opinions are clearly in the minority. The prevailing tone is very skeptical of Bush, if not overtly hostile. That is a sea change for this site - which may indicate a sea change in the opinions of the population of white mostly male office workers and techies who tend to spew their bile on similar outlets.

To summarize, I suggest extreme caution in interpreting the pre-election polls. This, I believe, is what Teixeira asks us to do as well.

Wojtek



More information about the lbo-talk mailing list