One last RTT thought for the day: Everyone (in no small part because of the tireless efforts of Rick Hess and Andy Smarick) is now chattering about whether Secretary Duncan is favoring collaboration and consensus too much in the choice of TN and DE as Race to the Top winners. The teachers’ unions in those states are more supportive of the RTT applications than in some other states — but we’ll see how long that lasts…The other big theory, that this is about vote-greasing on ESEA, is too ridiculous to waste time discussing.
But to the “consensus complaint” it doesn’t look like Duncan put his hand on the scale here. The reviewers made the call not the Secretary. And that may be the problem based on a look at the score sheets. So if there is a critique isn’t it that Duncan didn’t get involved enough rather than that he influenced the process too much? Under the law he had a great deal of discretion and it seems not to have come into play. If I were FL, LA, RI, or a few others tonight, that’s what I’d be pissed off about.*
Ockham’s Razor? A review process that isn’t bulletproof isn’t the most sexy explanation for the outcome, but it is the most likely.
*Meaning that between some questionable reviewer calls on some key issues and the prioritizing of LEA buy-in over reform by some reviewers, the Secretary’s call for boldness seems not to have permeated parts of the review process.
3 Replies to “Thumb Off The Scale?”
I think what you are asking about here is called inter-rater reliability. My perusal of the scoring tells me that the wide divergence in scoring has roots in unclear scoring rubrics or rater ideology or the weighting of different goals or of course, some combination of all those.
I have difficulty not choking over your use of “Ockham’s Razor”.
The entire process of education reform is over-elaborate and uses metrics and adjustment factors that are not proven to have any causation. In many cases these false measurements may be harmful.
Calling Scotus and Ockham from their rest to battle for education reform would likely result in its destruction. Better to stay with proven education philosophers like R. J. Rushdoony who never let rational thinking interfere with his vision of rebuilding civilization. /sarcasm
Reliability is an important component of a good test. After all, a test would not be very valuable if it was inconsistent and produced different results every time.