Had some time to read through more of the Race to the Top reviewer comments today and the scoring itself. I think there is a problem. No, not the various conspiracy theories or the word parsing partisanship but the more mundane problem I discussed below: You have some anomalies and outliers in the scoring as well as a seeming lack of consistency. The latter will lead to some focus on inter-rater reliability but reading a few apps (DC for instance) I’m not sure there may not be an intra-rater reliability problem as well. Of course, Secretary Duncan is in a no-win situation because you’d have a lot of people screaming if he’d overruled the reviewers. Still, to ensure that Round II isn’t a crap shoot some changes to the process seem necessary.
Will be interesting to see how much pushback there is, especially given what appears to be a winners premium for winning in the first round. You had a lot of people in the states and in various big consulting firms invest a lot in this so the reputational stakes are high — and there is a great deal of grumbling as well as a lot of analysis going on.
Ron Tomalis, a former state and federal official, has a theory on the role grant size played in the selection of just two states relative to natural breaks in the scoring. He’ll be publishing that soon. Update: It’s here.