Had some time to read through more of the Race to the Top reviewer comments today and the scoring itself. I think there is a problem. No, not the various conspiracy theories or the word parsing partisanship but the more mundane problem I discussed below: You have some anomalies and outliers in the scoring as well as a seeming lack of consistency. The latter will lead to some focus on inter-rater reliability but reading a few apps (DC for instance) I’m not sure there may not be an intra-rater reliability problem as well. Of course, Secretary Duncan is in a no-win situation because you’d have a lot of people screaming if he’d overruled the reviewers. Still, to ensure that Round II isn’t a crap shoot some changes to the process seem necessary.
Will be interesting to see how much pushback there is, especially given what appears to be a winners premium for winning in the first round. You had a lot of people in the states and in various big consulting firms invest a lot in this so the reputational stakes are high — and there is a great deal of grumbling as well as a lot of analysis going on.
Ron Tomalis, a former state and federal official, has a theory on the role grant size played in the selection of just two states relative to natural breaks in the scoring. He’ll be publishing that soon. Update: It’s here.
One Reply to “RTT Scoring”
Not so fast, Ron. I didn’t think Tennessee was even supposed to get that much money in the first place.
http://online.wsj.com/article/SB10001424052748703837004575012860304884840.html says, that only the largest states get the biggest payouts, not little Tennessee.
Something stinks somewhere. Maybe many places.
I think this may end up being a fail for Duncan and Obama. It’s not that they could not score a win here but rather it’s that, once again, the whole theoretical foundation of the Great Race was flawed. They could have won by choosing say, Florida and Louisiana and there would be few groans. Had they done so, the scoring would have had to have been different and for that to have been different, the rubrics would have to have been different, and so on…
I get the scoring categories for the Great Race; I just don’t yet understand how to score such a complex mashup of different, quite subjective variables without problems. If statewide buy-in was so darned important, according to Duncan, then many states never even stood a chance and the prognosticators clearly failed to see the neon sign of buy-in. Something’s fishy.