On Nov 10, 2009, at 3:02 PM, Jordan Hayes wrote:
> After reading all about this dustup, I think your original point of
> putting the CREDO results out there as "Charter schools don't work"
> is (very) wrong
Their own conclusion says: "And yet, this study reveals in unmistakable terms that, in the aggregate, charter students are not faring as well as their TPS [traditional public school] counterparts. Further, tremendous variation in academic quality among charters is the norm, not the exception. The problem of quality is the most pressing issue that charter schools and their supporters face."
This study was, by the way, peer reviewed. NBER working papers are not. Hoxby is an intense right-wing partisan of charter schools.
An interesting critique of Hoxby's methodology (there more, this is just an excerpt):
<http://morethoughtful.blogspot.com/2009/09/what-is-gold-standard.html>
Issue #3: Selection Bias on the School Level
The goal of this lottery-based study design is to avoid self-selection bias in the data. However, those who use it do not acknowledge the additional selection problems they create.
The most important problem is that not all charter schools are oversubscribed, so not all charter schools can be included in these studies. This wouldn't be a problem if we had good reason to believe that a random selection of charter schools were included, but that is obviously not the case. Clearly, the "better" charter schools are far, far, far more likely to be oversubscribed than the "worse" charter schools. This biases the sample rather severely towards better charter schools.
Unfortunately, the sample bias problem doesn't stop there.
A really strong traditional non-charter public school is not going to lose a lot of students to a simply above-average charter school. In order to be oversubscribed, a significant number of students and/or families have got to believe that the charter school option is superior to the non-charter public school option, which suggests a level of dissatisfaction with the local traditional public schools. This biases the sample towards inferior non-charter schools.
Issue #4: Generalizability
The hardest thing in educational research -- and perhaps research overall -- is to be able to generalize one's results to the broader population or wider world. And yet, that is usually the end goal of policy-oriented research.
These kinds of lottery-based studies only include the kinds of students and families that apply to charter schools in the first place. Even if the previous issues could be corrected, how can one know that other sorts of students and families would see the same benefits? The fact is that different populations might benefit less or more from going to a charter school. It is simply impossible to know from this kind of study. Of course, if you are only concerned about benefitting the kids of families who already opt for charter schools, then this is not a problem. But if you aim to help a broader population than that, you need a better methodology.
These generalizability concerns also apply to schools. Oversubscribed charter schools might well be better than average non-charter public schools, and I do not really question whether they are better than their local traditional alternatives. But on a policy level, we need to be concerned with charters more generally than that. If we raise or lift caps on charter schools, or approve new charter schools, we have to expect an average charter school to result, not an exceptional one. But these studies really tell us nothing about the majority of charter schools that are not oversubscribed. Nor do they tell us anything about the relative quality of non-charter public schools that lack charter school alternatives.