economists, value-added, and the value of criticismPosted: January 25, 2012
In case you missed it, a study came out in December that compared individual teachers’ students’ test scores to determine who the best teachers were, then compared the longitudinal earnings, health, education, and employment information for those groups of students to determine the economic impact of having a “better” teacher. It has everything an ed talking head could want: econometric analysis, teacher evaluation, standardized testing, longitudinal data, conclusions, policy suggestions, and the words “value added”. It also has a lot of people very excited, at least judging from the internet.
On her shared EdWeek blog, Diane Ravitch questioned the value of the study and predicted that it would be used as a rationale for more testing, more test prep, and less non-testing activity in schools. Matthew Di Carlo wrote at Shanker Blog about the strengths of the study and the perils of overreaching in interpreting it, while Nicholas Kristof of the New York Times was quite happy to make those overreaching intepretations (and freely admitted to doing so). Education Matters took a different perspective on the study: that economists need to stay out of education research entirely.
I don’t share that last opinion, but for some specific reasons. If we accept the findings of a growing body of research regarding the real, lifelong, tangible benefits and the excellent return for each dollar invested in early education, and particularly if we cite them as support for the practice of providing education as a public service, then we are accepting that econometric research has a place in education as evidence for the strategies, practices, and policies that are involved in schooling. Of course, good research admits its limitations and does not reject the usefulness of things it can’t directly address. Well-regarded studies on the returns to education normally will clearly note that there are likely to be additional, “hidden” benefits as well which quantitative research cannot address, making their estimates of the overall value of education less than the likely actual value.
“Value-added” measures (VAM) in particular are problematic in education (among other fields), and they are the main means of analysis used in the Chetty et al. paper. Criticizing the use of value-added analysis in studies of education is quite legitimate, if not popular; that kind of specific criticism is quite different from attempting to keep economists out of education entirely, as there are some persistent and serious theoretical, methodological, and practical questions about VAM as a tool for evaluation, accountability, and compensation/hiring purposes. Some of the most substantial arguments against VAM include its relative lack of utility for helping teachers to improve their teaching and better serve their students; its cost in the face of other, more effective and more efficient means of evaluating and improving schooling; and the tendency among some policymakers and economists (including those outside the U.S.) to nonetheless pursue it as a single, stand-alone measure of school and teacher effectiveness, even in the face of evidence for its limited use as a “complementary” measure.
But to reject the economics of education wholesale based on the Chetty et al. study or on the overt free-market and pro-privatization biases of some vocal and visible economists is to shoot ourselves in the collective foot, by cutting off an area of research that can actually help us to make decisions and to evaluate our prior choices. It represents a rejection of material that challenges our positions, which is a dereliction of intellectual duty and a failure of imagination that we really can’t afford in a world that requires multidisciplinary approaches to policymaking and problem-solving at all levels. (To be skeptical of the economics of education on the basis of the Chetty et al. study or on the biases of some vocal and visible economists is a different story; skepticism involves thoughtful consideration and looking at things directly, rather than avoiding them altogether.)
The Chetty et al. study can easily be discussed on its actual merits or the lack thereof, without rejecting it out of hand or because of how people we disagree with might use it. And, fortunately, this is already happening, and people are already reading about it.
Michael Winerip of the New York Times described one of the major limitations of the study: it was conducted way back in the 1990s, when high-stakes testing did not exist in its current form, and the lack of high-stakes makes it doubtful that the conclusions of the study can be directly applied to teachers, students, and test scores in the post-NCLB 2000s and 2010s. Bruce Baker at School Finance 101 gave one of the most complete analyses of the paper itself, which came out just before a study on teacher evaluation by the Gates Foundation. Baker has the technical expertise to dissect the study, but he also pays attention to the conclusions and prescriptions in the paper, the reactions to the paper from outside parties, and the comments of the study’s authors themselves.
What does this study and the response to it tell us? I’m not sure about “us”, but I have a couple ideas about what it tells me:
1) High-profile research can, as always, be flawed, problematic, and can be used for all sorts of purposes, including those the researchers would vigorously criticize as inappropriate. But it’s what we’ve got to work with, and it is useful.
2) People are actually reading the relevant research, writing about it, and discussing it; some of them have produced facile, unhelpful arguments, but others have really insightful critiques and (perhaps most importantly) thoughtful alternative interpretations of the results.
That’s all part of the nature of the media and of empirical research, so it appears that they’re both functioning properly in this case. With respect to Diane Ravitch, the “damage” has not been done with the publication of the Chetty et al. study, because the study’s effects are just beginning and the number of people paying skeptical attention to it is growing, and what’s been “done” is mostly a lot of talking by pro-merit pay and anti-tenure groups rather than actual hiring/firing decisions or policy changes.
3) Advocates for genuine teacher preparation, professional development, and evaluation, as well as those who are seeking fair and effective labor practices in education, have their work cut out for them.
But that’s not a bad thing. For every researcher or group who draws highly problematic conclusions from their studies, there’s one who does not. And for every one whose work is used to justify slash-and-burn policies, there are many others whose work strongly challenges received notions of educational quality and reform and proposes progressive, constructive policies that support teachers and students without punishing them. It’s our job as readers and thinking citizens to consider what’s out there, discuss it all, and act (or not act) upon it after giving it our best analyses.