What Some Much-Noted Data Really Showed About Vouchers By MICHAEL WINERIP
IN August of 2000, in the midst of the Bush-Gore presidential race, a Harvard professor, Paul E. Peterson, released a study saying that school vouchers significantly improved test scores of black children. Professor Peterson had conducted the most ambitious randomized experiment on vouchers to date, and his results — showing that blacks using vouchers to attend private schools had scored six percentile points higher than a control group of blacks in public schools — became big news.
The Harvard professor appeared on CNN and "The NewsHour With Jim Lehrer." Conservative editorial writers and columnists, including William Safire of The Times, cited the Peterson study as proof that vouchers were the answer for poor blacks, that Al Gore (a voucher opponent) was out of touch with his black Democratic constituency and that George W. Bush had it right.
"The facts are clear and persuasive: school vouchers work," The Boston Herald editorialized on Aug. 30, 2000. "If candidates looked at facts, this one would be a no-brainer for Gore."
Then, three weeks later, Professor Peterson's partner in the study, Mathematica, a Princeton-based research firm, issued a sharp dissent. Mathematica's report emphasized that all the gains in Professor Peterson's experiment, conducted in New York City, had come in just one of the five grades studied, the sixth, and that the rest of the black pupils, as well as Latinos and whites of all grades who used vouchers, had shown no gains. Since there was no logical explanation for this, Mathematica noted the chance of a statistical fluke. "Because gains are so concentrated in this single group, one needs to be very cautious," it said.
Several newspapers wrote about Mathematica's report, but, coming three weeks after the first round of articles, these did not have the same impact.
And Professor Peterson, a big voucher supporter, continued, undaunted. His 2002 book, "The Education Gap," largely ignored Mathematica's concerns and ballyhooed voucher gains for blacks. "The switch to a private school had significantly positive impacts on the test scores of African-American students," he wrote.
While he still couldn't explain why only blacks had gained, he offered theories. Perhaps heavily black public schools were even worse than urban Latino or white schools. Or, since most vouchers in New York were used in Catholic schools, perhaps a religious "missionary commitment is required to create a positive educational environment" for blacks.
David Myers, the lead researcher for Mathematica, is hesitant to criticize Professor Peterson. ("I'm going to be purposely vague on that," he said in an interview.) But he did something much more decent and important. After many requests from skeptical academics, he agreed to make the entire database for the New York voucher study available to independent researchers.
A Princeton economist, Alan B. Krueger, took the offer, and after two years recently concluded that Professor Peterson had it all wrong — that not even the black students using vouchers had made any test gains. And Mr. Myers, Professor Peterson's former research partner, agrees, calling Professor Krueger's work "a fine interpretation of the results."
What makes this a cautionary tale for political leaders seeking to draft public policy from supposedly scientific research is the mundane nature of the apparent miscalculations. Professor Krueger concluded that the original study had failed to count 292 black students whose test scores should have been included. And once they are added — making the sample larger and statistically more reliable — vouchers appear to have made no difference for any group.
Some background. In 1997, 20,000 New York City students each applied for a $1,400 voucher to private school through a project financed by several foundations. A total of 1,300 were selected by lottery to get a voucher, and 1,300 others — the controls, who had wanted a voucher but were not selected — were tracked in public schools. When the first test results came back, the vouchers made no difference in test scores for the 2,600 students as a whole. So the original researchers tried breaking the group down by ethnicity and race, and that's when they noted the sixth-grade test gains for the black voucher group.
But there was a problem. The original researchers had never planned to break out students by race. As a result, their definition of race was not well thought out: it depended solely on the mother. In their data, a child with a black mother and a white father was counted as black; a child with a white mother and a black father was counted as white.
When the father's race is considered, 78 more blacks are added to the sample. Professor Krueger also found that 214 blacks had been unnecessarily eliminated from the results because of incomplete background data. These corrections by Professor Krueger expanded the total number of blacks in the sample by 292, to 811 from 519.
In recent weeks, Mr. Myers, of Mathematica, has reviewed Professor Krueger's critique and found it impressive. Mr. Myers has now concluded that Professor Krueger's adjustments mean that "the impact of a voucher offer is not statistically significant."
It is scary how many prominent thinkers in this nation of 290 million were ready to make new policy from a single study that appears to have gone from meaningful to meaningless based on whether 292 children's test scores are discounted or included. "It's not a study I'd want to use to make public policy," Mr. Myers said. "I see this and go `whoa.' "
Professor Krueger of Princeton (who also writes a monthly business column in The Times) said, "This appeared to be high-quality work, but it teaches you not to believe anything until the data are made available."
As for Professor Peterson of Harvard, the star of newspapers and TV news in 2000 remains curiously mum these days. In a brief interview, he declined to comment on Professor Krueger's or Mathematica's criticisms. He said he stood by his conclusion that vouchers lifted black scores, and would "eventually" respond in a "technical paper." But he said he would not discuss these matters with a reporter.
"It's not appropriate," he said, "to talk about complex methodologies in the news media."