> this is precisely what I want to hear about: the statistical problems. it's where I'm the weakest in training and experience. everything else you've talked about again seems more or less straightforwardly the case. but it doesn't help me see the statistical flaws. i just got served with it as part of a long running argument about the possibility of measuring good teaching. and I don't, I'm sorry, have the statistical chops to just look at the article and know what's wrong with the analysis in the study. that is where, I think, I need some concrete assistance. but I won't keep whining about it. I just thought joanna or alan or someone might have something to hand.
Hi Jeffrey - I unfortunately don't have time to go through this particular paper - it's almost a hundred pages - but I know the genre and can make some general comments. I worked for a couple of years in a corner of the New South Wales education bureaucracy where a part of my job was interpreting this literature for people up the chain.
It is of course reactionary stuff and ultimately just a weapon against teacher unions. Although I agree there are fundamental problems with quantifying successful teaching, especially at the level of the individual teacher, I don't think we should be afraid of stats as such - they are really on our side. When I read this 'value-added' stuff it seems like a really transparent reframing of educational problems - but it is obviously very successful propaganda, partly because the reporters and even well-meaning policymakers don't have the background to read between the lines.
Without going into specific criticisms of the methodology - i.e. if you grant that it does what it says it does for the sake of argument, there are several objections:
(1) The biggest problem with the whole endeavour is that it strips away the biggest factor explaining differences in educational success - socioeconomic background. This is the whole point of 'value-added' analysis, it's what 'value-added' means in this context - to find the 'value added' by a teacher you have to strip out the 'value' contributed by other factors. In the language of this paper, you strip out the effect of the "vector of student characteristics" and the "vector of classroom-level characteristics". This is of course what education policymakers find so helpful about it, since they can conceive of controlling the hiring and firing of teachers but not the economic landscape. It also seems to sidestep the traditional teacher objection to rating teachers based on student test scores, by stripping out non-teacher influences.
(2) It invites (clueless) people to take an exaggerated view of the potential effect of policy which tried to act on these results. Results are typically sold on the basis of comparing what would happen to students if they had teachers two standard deviations better than they do. That's what this paper does in the abstract and executive summary - compares someone taught by a bottom 5% teacher with what they would have achieved with an average teacher. Then the relatively modest gains are inflated into a big number by posing it in terms of lifetime earnings. It is very highly unlikely that any feasible policy would mean upgrading 'teacher quality' by two standard deviations across the board (adopting the 'teacher quality' framework for the sake of argument). Actual 'gains' from this agenda would be much more modest, unless of course outmaneuvring teacher unions counts as a gain.
(3) It erroneously draws conclusions about macro-level gains from education from results on individual income differences. Most of the impact of academic success on an individual's income comes from sorting them into a better paying job, at the expense of someone else who would have got it. Educational improvements may raise real productivity to some extent, but how much has not been shown, and it is very misleading to imply that across-the-board gains would raise society's output to anything like the full extent of individual gains.
Got to run, but if I get a chance later I might drag up some criticism of the value-added methodology itself.
Mike