Sin Of Commission?

As the dust settles on the Race to the Top selections a general consensus has emerged that again (pdf) there were problems with the scoring.   Not the sensational political tampering claims that some people are trying to allege, there is no evidence of that, but rather problems with the process.   Those problems are at once more mundane and a lot more far-reaching.  The Race to the Top is over for now but the problems have broader implications for federal grant competitions, especially because the Administration would like to shift toward using more competitive grants for some programs.

The rationale for moving to more competitive grants is that they involve more planning, more substantial amounts of money to make real changes, and that the competition pays dividends as competitors (even those that ultimately lose) vie to get ready to win.  The Race to the Top certainly illustrated this as did I3 and several of the other competitions running this year, including smaller below-the-radar ones.  And as long as the priorities of the competitions are well-constructed there is no reason that competitive grants can’t also serve the equity goals.  You wouldn’t want to allocate all aid this way, but on some big programs, such as Title II teacher quality funds (pdf), it makes a lot of sense.

However moving in this direction means that the competitive grant process must be reliable.  For example, scores should align to program goals and reviewers should be able to separate rhetoric from actionable proposals.  And there absolutely must be consistency among reviewers so it’s not a game of bingo for applicants.  In the case of Race to the Top, while it wasn’t a disaster there were enough problems that some people favorably inclined toward using more competitive grants are now asking if the federal government, with all the political and substantive constraints upon it, can really run a reliable high-stakes competition. (The back and forth around the social innovation fund isn’t helping either).

Meanwhile, you’re already hearing a lot of concern that because pretty much everyone with deep expertise on assessments was conflicted because of work with various states and vendors that the review pool is there is sub-optimal.  That was certainly the case on Race to the Top.   Watching some of the video interviews is discomforting as were some of the questions reviewers had on relatively basic issues.  Surely there is a better way to mitigate conflicts of interest but also engage people with deep knowledge of the work.  For my part I helped several state teams prep for their RTT live interviews and the feedback after their actual presentations was that the prep teams were much harder on them around the guts of the applications and the connective tissue that really makes plans like this rise or fall.  I heard the same from other prep projects.  That’s not encouraging.  Especially with hundreds of millions of  public dollars on the line.  Likewise, the actual reviewer comments and scoring variances in Round 1 and Round 2 don’t always inspire confidence, to put it gently.

It would be easy to say the problem was people at the Department of Education and a different team would have avoided these problems.  I don’t think that’s the case.  Rather, the competitions are thoughtfully designed but constrained by a flawed process (and of course a tortuous ‘gotcha’ politics).

That’s why Secretary Duncan must move quickly to head these problems off at the pass.   The best way for him to do so would be to convene a commission or Secretary’s technical working group to study and report on what can be learned from the federal competitions so far and, more importantly, what can be learned from other high-stakes competitions in the public and non-governmental sectors.  There are other fields, for instance, where a small subset of experts have to at once make decisions and manage conflicts of interest.  What are best practices there?  What are best practices for ensuring reliability among and between reviewers?  What aspects of current federal grant making policy (which was really not designed for high-stakes competitions like these) should be changed?  Is more training needed, and if so what kind?  What else has to change if substantial amounts of federal aid are to be allocated this way?

Granted (ha ha) commissions are generally considered the place to go when you don’t want much to happen or want to punt an issue.  But Secretary Duncan has proven clever at leveraging issues in creative ways and just by putting his brand on it and sanctioning a candid review and study he could make such a process meaningful.  Substantive benefits aside, such a project would also help the Secretary make the case for moving in this direction.

Bottom line:  The fallout here extends beyond states like CO, IL, and LA and it behooves the Secretary to get in front of it.  Not in a way that invites pointless recriminations, but rather in a forward looking way that improves future initiatives.

Standing Disc: Bellwether personnel, including me, were involved in advising a number of states, winners and losers, about policy and strategy but had no direct interest in a specific outcome for the states mentioned here.

6 Replies to “Sin Of Commission?”

  1. sort of a vague disclosure, andy — even the gates folks eventually told us which states they helped — how about you?

  2. Well said. The bright line defining conflict of interest for programs with requiring broad national expertise and deep state roots is tricky. The professional community may be big enough to successfully review these applications, but I doubt that it will ever be deep enough to make everyone happy.

    As the Department continues to press forward with competitive applications, I think it’s worth reviewing the process. These discretionary dollars are here to stay for a while. I like that these funds can be targeted and I like that they complement existing compensatory funds. I want it to succeed – so it’s fair to check and improve the process before diving into the next round.

  3. Having done work for ED on grant peer review processes and how to improve them, launching a study or commission is only going to reveal the inherently subjective nature of aspects of the selection process and its inability to make fine distinctions. It is an ok decision making process, and maybe better than the alternatives, but it is far from perfect. I’d not recommend going down that rabbit hole to defend further competitive grant making.

    And, I think you too generous in letting the Admin off the hook in the design of its competitions. Nobody but the Admin came up with the 500 point schema for RTTT; i3 got what it was designed to get. Stepping aside from the speeches and rhetoric, the application processes could have been designed better from the very start.

  4. You write, “Pretty much everyone with deep expertise on assessments was conflicted because of work with various states and vendors that the review pool is there is sub-optimal.”

    This point completely misses the basic question: Did these experts ever apply? If they did, your point is well taken. However, my appreciation is they did not.

    You advised states on policy and strategy but had no direct interest in a specific outcome for the states. I’m sure if you applied to peer review USDOE would have used your expertise and found a way to mitigate any perceived conflict of interest by, for example, giving your states you did not advise.

    These experts can’t have it both ways. Complaining talent was sub-optimal, while not attempting to improve the talent by offering their expertise.

Leave a Reply

Your email address will not be published.