On Thu, 28 Oct 2004, Doug Henwood wrote:
> But out of 15 major polls, 11 called it for Bush, 2 were tied, and 2
> called it for Gore. Of course anything's possible, but that looks
> like a systematic, and not a random, error.
Again, if the difference between the sample percentages is less than the margin of error, the poll's not "calling it" for anybody. People often make that mistaken assumption, but that's not a flaw in the research or the statistical procedure here. Any outcomes within the margin of error are feasible; thus the results don't necessary show a systematic error. (There may be one, but the evidence is ambiguous at best.)
> And the margin of error is a 95% confidence interval, right? That
> means that a 2 point lead has something like a 66% probability of
> measuring an actual lead, right?
>
> Doug
This is a big statistical can o' worms. According to the old school frequentist view of probability, a 95% CI means that if you were to gather random samples of the same size many times and calculate 95% CIs for each one, in 95 samples out of 100 the true population percentage would be somewhere in the CI range.
Example: if the sample percentage for Bush is 46% with a plus or minus 3% CI, then we can be fairly confident that the true population parameter is somewhere between 43% and 49%. However, we have to keep in mind that we could have gotten one of the 5 in 100 CIs that is a lemon (it misses the population parameter completely: say, when Bush really at 42% or 51%). Put another way: if 20 polls are conducted at once, one of those polls will report a CI range that misses the true population percentage for a candidate completely.
Also, it makes no sense to make claims about the relative likelihood of any percentage estimate within the CI. There's no coherent argument for the claim that 46%, in the middle of the CI, is more likely to be the population parameter than 48% or 44%. The true population parameter may fall anywhere within the range specified by the CI.
Fans of Bayesian analysis look at this a different way; I won't even go there. From my discussion above, though, I think it's obvious that these polls are less precise than people assume, and that's a necessary effect of the statistical methods that are being used.
Miles