Posted on by arotherham

Adding Value

Important, measured, and sensible report on value-added methodology from Brookings.  There has been way too much craziness on the blogs and interest group-driven misinformation about what value-added can or can’t do.  This issue is complicated (more so than the rhetoric allows), there is promise but also a lot of reason for caution in how this tool is employed.   The Brookings paper lays all that out clearly and via a group of respected and middle of the road analysts and lays out the contextual issues – perfection is not the standard.

My short take on value-added and evaluations more generally here via TIME.

23 Replies to “Adding Value”

  1. The tests used in a lot of places are a bad match for the value-added methodology, which is a lot more complicated than subtracting one year’s score from the next. Meanwhile, different value-added models can yield different conclusions about the same teacher. A small detail like that matters a lot if you’re going to use this data to start firing people.

    That’s true, but its a huge understatement. In my district alone, there have been three rounds of test-making since NCLB, each with different purposes. With the Repubs taking over, we’ll get new tests. None are compatible with each other for the purposes of VAMS. And just wait until they start firing freshman Biology teachers for their test scores based on data and targets from middle school Science tests. One test scientific thinking while the other tests info from the “worksheet science” that plagues middle school. The same applies to using middle school Math tests that test arithmetic being used to fire freshmen teachers based on Algebra test results.

    If peer review is too modest, then use peer review for interpreting VAM results.

    You simply can’t allow administrators who set policies, or at least take the blame for those policies, to evaluate whether it was those policies or the teacher who was to blame.

    Again, if theorists would look into it with open eyes, I bet we’d find that the issue that trumps everything in the inner city is the refusal of administration to enforce attendance and discipline policies. Rank-in-file will never accept the people who allow chaos and violence to reign to interpret test scores.

  3. JT: “Rank-in-file will never accept the people who allow chaos and violence to reign to interpret test scores”

    Well, you can say that for much pretty much every potential evaluation tool. Basically you are saying that teachers refuse to be evaluated by administrators because they don’t think administrators are good at their jobs.

    To that I say: welcome to the real world. Out here in the private sector, we get evaluated by our bosses even if we don’t think they are great bosses! It’s a stunning reality.

  4. People talk about “in the private sector we” … I don’t really know any job in the private sector that even remotely compares to the job of educators.

    Hm, maybe private schools?

    It’d be neat if we stopped pretending teachers were very similar to bankers and that kids are stock whose value we need to raise.

  6. John Thompson:

    “The tests used in a lot of places are a bad match for the value-added methodology”

    This is not a valid argument against whether or not VA should be used in evaluations. You’re assuming that VA can’t yet be used in many places, but this does not argue either for or against its actual usage in evaluations.

    “Meanwhile, different value-added models can yield different conclusions about the same teacher.”

    The solution here probably entails choosing one of the best possible models (smallest error bars) and sticking with it. This is such an annoyingly trite argument, too, since given the error, running the SAME value-added model can also yield different conclusions about the same teacher. This is not exactly a big surprise, nor is it a damning one for VA, since no expert is arguing for VA to be the only indicator of teacher effectiveness.

    “If peer review is too modest, then use peer review for interpreting VAM results.”

    Can you elaborate?

    “You simply can’t allow administrators who set policies, or at least take the blame for those policies, to evaluate whether it was those policies or the teacher who was to blame.”

    1) Yes, you can. It happens in every school with whatever evaluation model that is currently in effect. As the report shows, the alternative to VA is more imprecision. Every time a teacher gets canned, they could use your reasoning to suggest that their bosses shouldn’t be the ones in charge of determining whether they are truly bad teachers or whether their established evaluation policies are faulty.

    2) This happens everywhere else, too, as edconsumer noted.

    “Again, if theorists would look into it with open eyes, I bet we’d find that the issue that trumps everything in the inner city is the refusal of administration to enforce attendance and discipline policies. ”

    Do you twirl your mustache and cackle sinisterly when you write stuff like this in vain hopes of derailing the thread?

    *****

    TopherGL:

    “It’d be neat if we stopped pretending teachers were very similar to bankers and that kids are stock whose value we need to raise.”

    And it’d be fantastic if we stopped pretending that anyone should give a damn if teachers don’t like being evaluated on similar standards to others.

    That teachers have a unique job does not imply that they shouldn’t be thoroughly evaluated on their effectiveness. If anything, your argument should imply the opposite: since their unique role implies that teacher effectiveness impacts student achievement, we should hold teachers to *higher* standards so that we can help more kids succeed. After all, how many lives are invested in the effectiveness of investment bankers (no offense to any that may lurk here)?

  8. This is such an annoyingly trite argument, too, since given the error, running the SAME value-added model can also yield different conclusions about the same teacher. This is not exactly a big surprise, nor is it a damning one for VA, since no expert is arguing for VA to be the only indicator of teacher effectiveness.

    Yes–but people (see Houston ISD) are planning to fire teachers based solely on VA scores. And they get bonuses based solely on VA scores.

    So, its trite to you, but not to teachers in the system.

  9. What’s trite by now is you tossing out conspicuous lies and assuming I won’t double-check your facts:

    “Q: If a teacher has a negative or regressive value-added score in one subject but is doing well with students in other subjects, what will happen to the teacher?

    A: Contract decisions will not be based solely on a teacher’s value-added scores. In this case, a principal may decide to provide focused professional development and/or assign a mentor to help the teacher improve his or her knowledge or skills in the subject area in which the teacher is struggling. The principal may also decide to change teaching assignments so that teachers are teaching content areas in which they are facilitating the greatest academic growth with students.”

    “Q. Why does HISD want to include value-added as part of the teacher evaluation system?

    A: Nothing matters more to student success than teachers. Having great teachers in the classroom is the single most important thing we can do at HISD to meet the needs of our students and prepare them to graduate on time and be successful in college, careers, and life. Value-added measures how well schools and teachers are doing in
    accelerating the academic progress of their students. Value-added, when used with other measures, provides a more complete picture of performance.”

    “HISD already uses a statistical analysis of student test scores to decide which teachers get performance bonuses. That same complex formula now would be used to highlight ineffective teachers.”

    “HISD’s performance pay program for teachers is one of the largest in the country. The program measures teacher and school impact on student academic progress from year to year using both achievement data and value added analysis”

    (Google: “houston isd value added”)

    If you read the report, you’ll also find a few arguments for why VA, even given its imprecision, could/should be used despite false positives/negatives.

  10. “What’s trite by now is you tossing out conspicuous lies and assuming I won’t double-check your facts”

    Ouch!

  12. Nice try Chris, but that document precedes the HISD school board passing a rule that will allow the district to fire teachers based on their value-added score. Of course, they aren’t stupid enough to say it will be solely based on the score, but the comments from the superintendent and school board members make it clear that this is their intent.

    Why don;t you prove how supportive you are for VAM and take a job as a teacher where you can be fired based on a statistical error?

    If you are not willing to do so, then you are a fraud and hypocrite.

  14. Harry:

    Always.

    (https://www.eduwonk.com/2010/08/kipp-and-catholic-schools.html#comment-208856)

    *****

    The Anti-Chris:

    “that document precedes the HISD school board passing a rule that will allow the district to fire teachers based on their value-added score.”

    There were several documents and articles I found, actually, all suggesting that you are, in fact, lying.

    Here’s a novel idea: how about you give a link that supports what you’re saying above?

    “Of course, they aren’t stupid enough to say it will be solely based on the score, but the comments from the superintendent and school board members make it clear that this is their intent.”

    Sure, because of course we should never trust written documents signed off by the district but instead assume that whatever happens to be publicly said at the time by school officials is actually the ultimate truth.

    “Why don;t you prove how supportive you are for VAM and take a job as a teacher where you can be fired based on a statistical error?”

    Why don’t you first prove that you’re not an idiot by taking any other job and realizing you can be fired due to a laundry list of reasons more unfair than an assessment of objective data concerning your effectiveness?

    To be very clear, you will not win this debate if you keep setting the goal for evaluations at 100% accuracy. You will never achieve 100% accuracy for evaluations, either in education or in any other field, and you would never know even if you had achieved it. Luckily, the key players on this issue seem to clearly understand this, and so the recommendations tend to always include VA as part and parcel of a thorough evaluation system.

    Humorously, such a system (as any other) will also have statistical error. Can someone here try this reasoning on their boss the next time they get a bad evaluation? “I refuse to be evaluated due to the possibility of error!”

    Here’s a question that we can ask: do they need to make VA only part of evaluation? The report gives justification for the use of VA and compares its precision to various other fields, and really, IF the argument were that VA by itself should be utilized to parse good and bad teachers, the merits of such might be somewhat convincing, but good luck to anyone trying to get teachers on board with that. Luckily, there is other data that can be used to further supplement VA, reducing the probability of false positives and negatives.

  15. The Anti-Chris (again):

    “Not that you will understand the arguments here Chris (Ive sent similar links and you totally missed the point)”

    What is the point I am missing? That you’re trying to deflect arguments addressed to you and further derail the thread by linking to something unrelated? What a wholly unique approach to online edu debates.

    Linking to a WordPress blog to better support your arguments should be rightfully responded to with raucous laughter and derision, as it is the typical gambit of those losing a debate here and inconceivably still not realizing it.

    Try, if you could, to flesh out your arguments here, as I’d rather not be a dick and start attacking some random blogowner’s arguments when he isn’t even here to give his rebuttal (this sounds vaguely familiar to someone we know). You can start by responding to what I wrote above.

  18. As usual, Chris, you don’t respond to anything you cannot explain away. You just deride others. As if your use of TIME magazine as a reference was scholarly.

    You are the poster child for why these reforms will never be fully implemented nor effective.

  22. I told ya’ll before, the HISD board passed policy after this came out. Wow–can’t ya’ll read??????

  23. “I told ya’ll before, the HISD board passed policy after this came out. Wow–can’t ya’ll read??????”

    We’ve asked you several times now to substantiate your above claims. Why can’t you simply do so?

Leave a Reply

Your email address will not be published.