>How reliable is this?
The AAPORites treat Zogby as a bit of a charlatan. He uses questionable sampling techniques, then weights the results in odd and opaque fashion. Here's one critique. Zogby and Mitofsky have a bit of a beef going; in a subsequent NYT article, Zogby called Mitofsky (the inventor of the exit poll) "a grumpy old man."
Doug
----
<http://www.washingtonpost.com/ac2/wp-dyn/A41186-2004Jan23?language=printer>
New Hampshire: Graveyard of Pollsters
By Richard Morin and Claudia Deane Washington Post Staff Writers Friday, January 23, 2004; 7:55 AM
[...]
What an Unkind Crack
That well-traveled pollster and media demi-star John Zogby is once again criss-crossing the country conducting tracking polls in primary states from New Hampshire to Arizona, apparently much to the chagrin of the political mavens at ABC News.
"The svengalis at ABC News and some major papers don't like Zogby's tracking," reported The Note, the widely read ABC News online politics briefing . Then they quote an anonymous "ABC guru" who called Zogby's tracking polls "crack for the weak."
"I saw that," laughed Zogby, adding he was not surprised. ABC's polling department has reviewed his methodology and rated his polls "not airworthy," he said. "But that's all right. We're doing okay without ABC."
Okay, indeed. Last year The American Prospect in an otherwise critical article suggested that his polling firm's reputation ranked second only to Gallup's. This year he's working for a gaggle of media clients, including NBC and Reuters. Zogby will be polling in no fewer than seven primaries and he had the only daily tracking in Iowa, which garnered huge media coverage but ended up a bit off the mark.
ABC isn't the only news organization that holds Zogby polls in some disrepute. Many media pollsters -- including these writers -- question at least some of his methods. (Not that our doubts keep his work from being prominently featured in newspaper and broadcast news stories, at least in part on the theory that good or bad, his polls are news.)
"Zogby is not a reputable pollster," said Warren Mitofsky, who is co-directing the media exit polls this year for the major television networks and the Associated Press. "He is more a salesman and a self-promoter than a pollster. He has made lots of mistakes on election outcomes -- five in 2002. . . . I have heard of volatile campaigns, but he has volatile polls." (Zogby acknowledged on his Web site last February that "this past election cycle was not my finest hour.")
Iowa was not particularly kind to Zogby or the other public polls, all of which underestimated Kerry's big win and Gephardt's dismal showing. Keating Holland, who directs surveys for CNN, noted that Zogby "was off by 10 points on Kerry and overestimated Gephardt by 7, which is not a good track record. We'll see what he does from here on out." (Zogby said things were changing fast in Iowa up through election day and his final poll did have Kerry leading, Edwards surging and Gephardt finishing fourth. Zogby was the only one doing tracking polling in Iowa.)
"We clearly have different points of view about how polling should be done," said Kathleen Frankovic of CBS News who discussed methodology with Zogby after the 1996 election.
Why do so many pollsters go squinty-eyed when Zogby's name is mentioned? Here are some of their biggest complaints about the ubiquitous pollster from Utica:
Zogby draws his samples only from computer files of people with listed telephone numbers, thereby missing 30 percent of the population with unlisted telephone number.
The biggest advantage to what Zogby does is that most of those telephone numbers are good. The disadvantage is that you miss everyone with an unlisted number, and people with unlisted numbers tend to be different than those who are in the telephone book. Most other public pollsters rely on Random Digit Dialing, a technique that captures both listed and unlisted numbers. (Zogby says he plans to release a study later this year that shows no demographic or ideological differences between people with listed numbers and those who are unlisted.)
Zogby also calls people during the day as well as in the evening. About 30 percent of his interviews are collected before 5:30 p.m. Daytime interviews are great if you want to talk to lots of retirees and housewives. But his critics charge that they're not so good if you want to interview working men and women. Zogby counters that daytime interviewing actually produces a more representative sample because he can talk to people who work at night.
Zogby also adjusts his sample based on historic trends and his judgment of "what is happening on the ground" in a particular race, and it is this imposition of his own judgment that disturbs many pollsters.
He will, for example, reduce the proportion of 18- to 24-year-olds in his sample of self-described likely voters if he suspects on the basis of past voting history and the "lay of the land" that a sample contains too many younger people. He also, on occasion, adjusts the religious composition of his sample if he suspects he has over or under-represented one faith.
Most pollsters cringe at such extra-curricular adjustments. "I know I do some things different that others," he said. "I know the so-called 'Poll-ice' would deny it, but there's art as well as science involved in this."
Some of his techniques that were once widely criticized have now been more generally adopted. Zogby adjusts his sample so it matches the proportion of Republicans, Democrats and independents, based on past elections.
Critics say people can change their party identification, so past estimates of partisanship may be outdated. In fact, party identification nationally has changed little over the past two decades, prompting some organizations, including The Washington Post and ABC News, to weight to party identification on some political polls.
Zogby knows he's not popular with many of his professional peers. But he doesn't seem to care. "I'm a humble guy from Utica just plying my trade," he laughed. "I'm just not a member of the club."