New Hampshire: Graveyard of Pollsters
By Richard Morin and Claudia Deane Washington Post Staff Writers
"Now I know why they call this the Granite State," said candidate Bob Dole eight years ago after losing a Republican primary in the state for the second time. "It's so hard to crack."
Dole's assessment is as true for pollsters as for the candidates in New Hampshire, where bad methodology, bad timing or simply bad luck have produced some of the most memorable miscues in the annals of polling. Consider these flubs:
In 2000, the headline on an AP day-before-the-primary story was "Nearing the N.H. finish line; Polls declare GOP dead heat. . . . " John McCain then went on to beat George W. Bush by 18 percentage points.
The New Hampshire-based American Research Group's tracking poll ended up buried deepest in the snow bank: They had Bush winning by two the day before the primary, merely 20 points off the mark. On the Democratic side, the losing pollster at least got the winner right: The Quinnipiac poll predicted Gore would win by 17 percentage points, but he actually won by four.
It was the second debacle for ARG in as many New Hampshire Republican primaries. The day before the 1996 contest, ARG's Dick Bennett told the Union Leader, "It looks like Dole's going to win," based on the Kansan's seven point advantage in their tracking poll. He didn't, losing to Pat Buchanan by a single percentage point.
Exit pollsters aren't immune. In 1992, Voter Research and Surveys' exit poll showed George H.W. Bush beating Buchanan by a relatively narrow 6 percentage points, only to have Bush finish 16 points ahead on election night.
In 1988, it was the Gallup poll that fell victim. Gallup's final preelection survey had Bob Dole up by 8 percentage points. He ended up losing to Bush senior by 9. We "went on to call the rest of the primaries that season without a hitch . . . haven't had many questions about how we succeeded in so many of those subsequent races, however," former Gallup head Andrew Kohut, now with the Pew Research Center, reminisced the other day. "Moral of the story: on my tombstone it will say 'Here lies Andy Kohut -- got NH wrong in '88.'" [The Post had nothing to boast about here either, with our final product suggesting Dole up by three.]
So is New Hampshire just jinxed, or what?
Not necessarily. The dirty little secret in New Hampshire and elsewhere is that too many of the widely reported pre-election polls cut corners or otherwise use methods that are less than gold standard.
Perhaps the best-known of the bunch, Zogby International, does all kinds of controversial things to produce its headlines-grabbing tracking poll (see next item). Surveys taken by students for Franklin Pierce College, which is reporting a Democratic preference poll today, uses samples based on lists of registered voters that have proven to be incomplete, outdated or both. Suffolk University, which is polling for a Boston television station, asks a curiously convoluted candidate preference question that ends: "toward whom would you vote or lean?" Many professionals consider student interviewers unreliable, especially when unsupervised. Franklin Pierce and Suffolk University also use student interviewers, as does the University of New Hampshire. Polling directors at the schools insist that the kids are alright: "Their quality is tremendous," said Richard Killion, who oversees Franklin Pierce polls, later adding: "It really improved when I started paying them."
Research 2000, which does polls for the Concord Monitor, doesn't randomly select respondents within the households they contact. Instead, they interview the person who answers the telephone if they qualify as a likely Democratic primary voter on the basis of answers to subsequent questions. The data is adjusted so that the proportion of men and women, Democrats and independents as well as other key groups match the proportion who voted in the 2000 New Hampshire primary, reports Del Ali, president of the firm. That could improve the accuracy of the results if Tuesday's electorate is a carbon copy of 2000, but could be a problem if it is not.
Likewise, student interviewers at Franklin Pierce College also don't randomly select people once they reach a household. They talk to whomever answers the telephone unless they're sample is skewing more male or female. In that case, if they need more women, they ask in subsequent calls to speak to women, and vice-versa if their sample appears to be light on men.
The problem with not randomly selecting within households is that the resulting sample is not truly random and more likely to be biased in ways that are not readily apparent.
Despite these questionable practices, the track record of polls in New Hampshire may be no worse than elsewhere -- which may offer slim comfort to pollsters. "I think you probably would find this kind of thing in other primary states, but New Hampshire is much more visible," said UNH pollster Andrew Smith. "But New Hampshire does have some quirks that, if a pollster doesn't pay attention, they're going to be wrong."
Among them: wildly varying turnout; same day registration; rapid immigration of new voters; and a fine and confusing tradition of allowing residents to register as "undeclared."
"When you look at primaries, you'll probably see a lot more mistakes than you'd see in a general election," Smith said. "You're always going to have a more difficult time figuring out who's likely to vote."
What an Unkind Crack
That well-traveled pollster and media demi-star John Zogby is once again criss-crossing the country conducting tracking polls in primary states from New Hampshire to Arizona, apparently much to the chagrin of the political mavens at ABC News.
"The svengalis at ABC News and some major papers don't like Zogby's tracking," reported The Note, the widely read ABC News online politics briefing . Then they quote an anonymous "ABC guru" who called Zogby's tracking polls "crack for the weak."
"I saw that," laughed Zogby, adding he was not surprised. ABC's polling department has reviewed his methodology and rated his polls "not airworthy," he said. "But that's all right. We're doing okay without ABC."
Okay, indeed. Last year The American Prospect in an otherwise critical article suggested that his polling firm's reputation ranked second only to Gallup's. This year he's working for a gaggle of media clients, including NBC and Reuters. Zogby will be polling in no fewer than seven primaries and he had the only daily tracking in Iowa, which garnered huge media coverage but ended up a bit off the mark.
ABC isn't the only news organization that holds Zogby polls in some disrepute. Many media pollsters -- including these writers -- question at least some of his methods. (Not that our doubts keep his work from being prominently featured in newspaper and broadcast news stories, at least in part on the theory that good or bad, his polls are news.)
"Zogby is not a reputable pollster," said Warren Mitofsky, who is co-directing the media exit polls this year for the major television networks and the Associated Press. "He is more a salesman and a self-promoter than a pollster. He has made lots of mistakes on election outcomes -- five in 2002. . . . I have heard of volatile campaigns, but he has volatile polls." (Zogby acknowledged on his Web site last February that "this past election cycle was not my finest hour.")
Iowa was not particularly kind to Zogby or the other public polls, all of which underestimated Kerry's big win and Gephardt's dismal showing. Keating Holland, who directs surveys for CNN, noted that Zogby "was off by 10 points on Kerry and overestimated Gephardt by 7, which is not a good track record. We'll see what he does from here on out." (Zogby said things were changing fast in Iowa up through election day and his final poll did have Kerry leading, Edwards surging and Gephardt finishing fourth. Zogby was the only one doing tracking polling in Iowa.)
"We clearly have different points of view about how polling should be done," said Kathleen Frankovic of CBS News who discussed methodology with Zogby after the 1996 election.
Why do so many pollsters go squinty-eyed when Zogby's name is mentioned? Here are some of their biggest complaints about the ubiquitous pollster from Utica:
Zogby draws his samples only from computer files of people with listed telephone numbers, thereby missing 30 percent of the population with unlisted telephone number.
The biggest advantage to what Zogby does is that most of those telephone numbers are good. The disadvantage is that you miss everyone with an unlisted number, and people with unlisted numbers tend to be different than those who are in the telephone book. Most other public pollsters rely on Random Digit Dialing, a technique that captures both listed and unlisted numbers. (Zogby says he plans to release a study later this year that shows no demographic or ideological differences between people with listed numbers and those who are unlisted.)
Zogby also calls people during the day as well as in the evening. About 30 percent of his interviews are collected before 5:30 p.m. Daytime interviews are great if you want to talk to lots of retirees and housewives. But his critics charge that they're not so good if you want to interview working men and women. Zogby counters that daytime interviewing actually produces a more representative sample because he can talk to people who work at night.
Zogby also adjusts his sample based on historic trends and his judgment of "what is happening on the ground" in a particular race, and it is this imposition of his own judgment that disturbs many pollsters.
He will, for example, reduce the proportion of 18- to 24-year-olds in his sample of self-described likely voters if he suspects on the basis of past voting history and the "lay of the land" that a sample contains too many younger people. He also, on occasion, adjusts the religious composition of his sample if he suspects he has over or under-represented one faith.
Most pollsters cringe at such extra-curricular adjustments. "I know I do some things different that others," he said. "I know the so-called 'Poll-ice' would deny it, but there's art as well as science involved in this."
Some of his techniques that were once widely criticized have now been more generally adopted. Zogby adjusts his sample so it matches the proportion of Republicans, Democrats and independents, based on past elections.
Critics say people can change their party identification, so past estimates of partisanship may be outdated. In fact, party identification nationally has changed little over the past two decades, prompting some organizations, including The Washington Post and ABC News, to weight to party identification on some political polls.
Zogby knows he's not popular with many of his professional peers. But he doesn't seem to care. "I'm a humble guy from Utica just plying my trade," he laughed. "I'm just not a member of the club."