Last month I joined a team of UPD-ers and traveled around the state of Oklahoma training district-level trainers on Value-Added.  During one of the sessions, a participant raised his hand and asked our team how value added could be relied upon as a valid measure of teacher effectiveness when districts like Houston Independent School District[1] are currently involved in lawsuits surrounding the legitimacy of their value-added model, and the American Statistical Association (ASA) released a statement[2] that has been described as “slamming the high-stakes ‘value-added method’ (VAM) of evaluating teachers.”    Although we were familiar with both the Houston lawsuits and the ASA statement, this question created an opportunity to take a closer look at recent articles and information opposing (or seeming to oppose) value added.

First, a little background:  According to our partners at Mathematica Policy Research, “Value-added methods (sometimes described as student growth models) measure school and teacher effectiveness as the contribution of a school or teacher to students’ academic growth. The methods account for students’ prior achievement levels and other background characteristics.”  Value added does this via a statistical model that is based on educational data from the given state or district, and uses standardized test scores to evaluate teachers’ contribution to student achievement. Although value added and similar measures of student growth had been used in various places in the United States without much opposition, criticism peaked around 2010 when districts such as Chicago, New York City and Washington, DC began incorporating value-added into high-stakes teacher evaluation models.  Since then various individuals and organizations have published their views on the merits or pitfalls of value added including, most recently, the American Statistical Association (ASA).

The ASA statement has garnered considerable attention because as described by Sean McComb, 2014 National Teacher of the Year, “… I thought that they are experts in statistics far more than I am. So I thought there was some wisdom in their perspective on the matter.”[3] Of course as statistical experts they shed some light on what can and cannot reasonably be expected from the use of value-added measures, but here are a few ways that we can address parts of their statement that may be misunderstood:

• The ASA mentions that value added models “are complex statistical models, and high-level statistical expertise is needed to develop the models and interpret their results. Estimates from VAMs should always be accompanied by measures of precision and a discussion of the assumptions and possible limitations of the model.”  Although it is true that the models themselves are complex and require advanced statistical expertise to compute, we would argue that people without this level of expertise can be trained on the concepts behind how the models work and also how results should be interpreted.  In Oklahoma, part of the training we provide is designed to help teachers build a conceptual understanding of the statistics behind value added.  Although we do not look at the regression formula itself, we help to define components of the measure including how it is developed, its precision, etc. so that teachers are able to better understand how value added can provide additional data to help inform their instruction.

• In the report, the ASA cautions that since value added is based on standardized test scores, and other student outcomes are predicted only to the extent that they correlate with test scores, it does not adequately capture all aspects of a teachers effectiveness – “A teacher’s efforts to encourage students’ creativity or help colleagues improve their instruction, for example, are not explicitly recognized in VAMs.”  This statement is true and it is one that we are quick to highlight when we train on value added.  Value-added models are not designed to measure teacher effectiveness in isolation as they only tell part of the story.  When used as part of an evaluation system with multiple measures (such as classroom observations and student surveys), a more complete and stable picture becomes available.

• Finally the ASA clearly states that “VAM scores are calculated using a statistical model, and all estimates have standard errors. VAM scores should always be reported with associated measures of their precision, as well as discussion of possible sources of biases.”[4] Since we are always transparent about the fact that all value-added estimates have confidences intervals, this is almost always something that trips people up during training sessions.  Many will say, “If there is a margin of error, then how can this measure be trusted enough to include in an educator evaluation system?”   What is easy to forget is that all measures, statistical or not, come with some level of uncertainty.  This includes more traditional methods of teacher evaluation such as classroom observations.  Although efforts should be made to limit or decrease the margin of error where possible, there will never be a way to completely eliminate all error from something as wide and deep as teacher effectiveness. Despite this, this does not mean that value added should not be used to evaluate teachers but, as mentioned previously, it should be considered alongside other measures.

 

[1]Strauss, Valerie. April 30, 2014. “Houston teachers’ lawsuit against the Houston Independent School District” Washington Post. http://apps.washingtonpost.com/g/documents/local/houston-teachers-lawsuit-against-the-houston-independent-school-district/967/

[2]American Statistical Association. April 8, 2014. “ASA Statement on Using Value-Added Models for Educational Assessment.” http://www.amstat.org/policy/pdfs/ASA_VAM_Statement.pdf

[3] Valerie Strauss. April 30, 2014. “2014 National Teacher of the Year: Let’s stop scapegoating teachers” Washington Post. http://www.washingtonpost.com/blogs/answer-sheet/wp/2014/04/30/2014-national-teacher-of-the-year-lets-stop-scapegoating-teachers/?tid=up_next

[4] American Statistical Association. April 8, 2014. “ASA Statement on Using Value-Added Models for Educational Assessment.” http://www.amstat.org/policy/pdfs/ASA_VAM_Statement.pdf