New York Times - September 15, 2000
New Doubt Is Cast on Study That Backs Voucher Efforts By Kate Zernike
Two weeks ago, prominent researchers released a study showing significant gains by black students who had been given vouchers to help pay for private school. The finding lent support to backers of voucher programs at a time when they have become an election-year issue.
But now a company that gathered data for the research in New York, one of three cities studied, says the gains, as measured by scores on standardized math and reading tests, were overstated by the lead researcher, a Harvard professor known within the academic community for his exuberant support of vouchers.
In fact, the company says, in New York there was no significant test- score difference between students who attended private school on vouchers and those who stayed in public school.
Bothered by what it describes as the report's exaggerated claims, the company, Mathematica Policy Research of Princeton, N.J., has now taken the unusual step of issuing a statement that cautions against leaping to any policy conclusions. Mathematica calls the original finding "premature."
The researchers, led by Paul E. Peterson, director of the Program on Education Policy and Governance at Harvard University, acknowledge that the gains among black students who used vouchers were concentrated heavily in Washington, where the improvement was twice as great as in New York and one-third greater than in the third city studied, Dayton, Ohio.
But in an interview, Professor Peterson stood by his characterization of the overall result as significant.
"An average is an average," he said.
The study measured test scores among 1,400 poor students given vouchers worth $1,700 a year to attend private school. While there were no significant overall gains among students of other ethnic groups, black students in their second year of private-school attendance improved their test scores by 6.3 percentile points — a striking advance at a time when schools around the country are trying to close a persistent gap between scores of white and black students.
But the study quickly came under attack from several quarters, in part a reflection of how heated the debate over vouchers has become.
The work was initially criticized because it had been underwritten by several conservative pro-voucher foundations. But the more recent criticism has focused specifically on the way the conclusions were arrived at, with even some of the researchers saying the larger picture is much less positive than Professor Peterson maintained.
"If you ask the question, `When I offered students vouchers, did I make a difference in their test scores?' right now you come away saying, `No, there's no impact,' " said David Myers of Mathematica, who was a principal investigator for the study.
Professor Peterson said in the report that the gains by black students using vouchers were statistically significant in each city after two years. But in the study's New York portion, which involved the largest and most diverse group of students, the gains were limited to sixth graders; the black students in Grades 3, 4 and 5 made no gains.
"Because the gains are so concentrated in this single group, one needs to be very cautious in setting policy based on the overall modest impacts on test scores," Mr. Myers said. "We really need to learn why this group stands out so much. Until we understand it, we cannot place much policy weight on it."
Critics point to another issue as well: Professor Peterson initially said the study's results were particularly reliable because this was the first time students had been randomly selected to receive vouchers; family background, then, would presumably not influence the results.
But the critics note that a high percentage of students offered the vouchers — 47 percent in Washington, 46 percent in Dayton and 24 percent in New York — did not use them, and that those who ended up using them had higher family incomes and higher levels of parent education and were less likely to be on welfare. This information was not included in the report, although Professor Peterson acknowledged it in the interview.
The difference in the two groups, the critics say, suggests that the poorest students — those whom politicians maintain vouchers would most help — may not use them.
In follow-up interviews and surveys, Mr. Myers said, parents say they cannot afford private schools even with the help of vouchers, or find that private schools do not have enough space for their children.
Martin Conroy, a professor of education and economics at Stanford University, said: "I'm worried that there are all sorts of nonmeasurable characteristics of these kids that made it difficult for them to get into these private schools. Even if they got vouchers, they might not have been able to pay the other costs associated with private schools, and even if they could pay, they might not have been able to get into a private school."
Professor Conroy is among a group of academics who have called on Professor Peterson to release his data so they can analyze the results, which were not peer-reviewed. They question whether he adequately adjusted the data for differences in income and parent education level.
"The pressure to get something out at election time was a much more dominant theme than the idea of letting it go through review," said Henry M. Levin, a professor at Teachers College at Columbia University.
Professor Peterson dismissed the criticism, saying his work "was looked at by lots of people lots of times."