• Researchers warn against overuse of test scores in teacher evaluation
  • Study: “Legislatures should not mandate a test-based approach to teacher evaluation that is unproved and likely to harm not only teachers, but also the children they instruct.”
  • Merit pay study:  “By and large, POINT had little effect on what these teachers did in the classroom.”

Christie’s education agenda:

Political grandstanding, or research-based reform?

Published on Tuesday, September 28, 2010

Gov. Christie is expected to announce his education reform agenda at a town hall meeting in Old Bridge this afternoon.  Details have not yet been released, but based on past public statements and the state’s failed Race to the Top application, it is expected to rely heavily on a system of teacher evaluation that researchers say is fatally flawed.

Among the things Gov. Christie is expected to pursue is a new teacher evaluation system that bases at least 50% of a teacher’s evaluation on test scores, perhaps under a so-called “value-added” system that measures student growth throughout the year.  Those evaluations would become the basis for decisions about whether to fire experienced teachers as well as for an expected merit pay plan the governor has previously said he wants to implement.

But research published by the Economic Policy Institute and authored by several of the nation’s top education researchers cautions that using students test scores for personnel and compensation decisions is unreliable and could lead to both firing effective teachers and harming students’ education.

Highlights from the EPI study include:

  • “Some states are now considering plans that would give as much as 50% of the weight in teacher evaluation and compensation decisions to scores on existing tests of basic skills in math and reading. Based on the evidence, we consider this unwise.” (page 2)

  • “Nonetheless, there is broad agreement among statisticians, psychometricians, and economists that student test scores alone are not sufficiently reliable and valid indicators of teacher effectiveness to be used in high-stakes personnel decisions, even when the most sophisticated statistical applications such as value-added modeling are employed.” (page 2)

  • “One study found that across five large urban districts, among teachers who were ranked in the top 20% of effectiveness in the first year, fewer than a third were in that top group the next year, and another third moved all the way down to the bottom 40%.” (page 2)

  • “The nonrandom assignment of students to classrooms and schools—and the wide variation in students’ experiences at home and at school—mean that teachers cannot be accurately judged against one another by their students’ test scores, even when efforts are made to control for student characteristics in statistical models.” (page 3)

  • “Research shows that an excessive focus on basic math and reading scores can lead to narrowing and over-simplifying the curriculum to only the subjects and formats that are tested, reducing the attention to science, history, the arts, civics, and foreign language, as well as to writing, research, and more complex problem-solving tasks.” (page 4)

  • “Tying teacher evaluation and sanctions to test score results can discourage teachers from wanting to work in schools with the neediest students, while the large, unpredictable variation in the results and their perceived unfairness can undermine teacher morale.” (page 4)

  • “Adopting an invalid teacher evaluation system and tying it to rewards and sanctions is likely to lead to inaccurate personnel decisions and to demoralize teachers, causing talented teachers to avoid high-needs students and schools, or to leave the profession entirely, and discouraging potentially effective teachers from entering it. Legislatures should not mandate a test-based approach to teacher evaluation that is unproven and likely to harm not only teachers, but also the children they instruct.” (page 4, emphasis added)

These researchers are not alone in their concern about the negative educational impact of using value-added modeling (VAM) for high stakes decisions about pay, evaluation and employment.  They also cite research from the Board on Testing and Assessment of the National Research Council of the National Academy of Sciences, which concluded:

  • “…VAM estimates of teacher effectiveness should not be used to make operational decisions because such estimates are far too unstable to be considered fair or reliable.” (Page 2)

Similarly, the Policy Information Center of Educational Testing Service, which develops many of the tests used for such evaluation, cautions:

  • “VAM results should not serve as the sole or principal basis for making consequential decisions about teachers. There are many pitfalls to making causal attributions of teacher effectiveness on the basis of the kinds of data available from typical school districts. We still lack sufficient understanding of how seriously the different technical problems threaten the validity of such interpretations.” (pages 2-3)

And Rand Corporation researchers have noted:

  • “The estimates from VAM modeling of achievement will often be too imprecise to support some of the desired inferences…” (page 3)
  • “The research base is currently insufficient to support the use of VAM for high-stakes decisions about individual teachers or schools.” (page 3)

The Economic Policy Institute report is not the only recent research to cast doubt on Gov. Christie’s proposed education agenda.  Last week, a major study on merit pay from the National Center on Performance Incentives concluded that a major merit pay experiment in Nashville, TN, the Project on Incentives in Teaching (POINT) had failed to achieve the intended educational results.  The authors stated:

  • “Thus, POINT was focused on the notion that a significant problem in American education is the absence of appropriate incentives, and that correcting the incentive structure would, in and of itself, constitute an effective intervention that improved student outcomes. By and large, results did not confirm this hypothesis.” (page xi, emphasis added)

  • “While the general trend in middle school mathematics performance was upward over the period of the project, students of teachers randomly assigned to the treatment group (eligible for bonuses) did not outperform students whose teachers were assigned to the control group (not eligible for bonuses).” (page xi)

  • “As already noted, we find no effect of incentives on test scores overall (pooling across all years and grades).” (page xii)

  •  But by and large, [participating teachers] did not endorse the notion that bonus recipients in POINT were better teachers or that failing to earn a bonus meant a teacher needed to improve. Most participants did not appear to buy in to the criteria used by POINT to determine who was teaching effectively. Perhaps it should not be surprising, then, that treatment teachers differed little from control teachers on a wide range of measures of effort and instructional practices. Where there were differences, they were not associated with higher achievement. By and large, POINT had little effect on what these teachers did in the classroom.” (page xiii, emphasis added)

As Gov. Christie sets out to pursue an education agenda that has significant implications for the future of New Jersey’s entire public education system, New Jersey residents and policymakers would be wise to ask whether it is an agenda based on sound educational practice, or simply another attack on New Jersey’s excellent public schools.

More information

Bookmark and Share