The interim federal evaluation on Reading First is now out (pdf). Critics, advocates, everyone, to the barricades!
A few thoughts. First, the study is hardly good news but also hardly that surprising. It’s a huge program, varying degrees of implementation fidelity, etc…Still, given the other politics around it the study is a political problem for the program’s supporters and does also beg a closer look at implementation. That newer grantees seemed to generate better results is interesting and one thing in particular worth looking at more closely. If there is a weakness in the study it’s that we can’t really account for the variance among sites in how they implemented the program and even how aggressively they wanted to do so. But, as with after-school programs, for instance, variance that results in indiscernible effects is nonetheless a problem in a program of this size and cost. It’ll be good to see the next evaluation later this year. And you also want to look at the CEP evaluations of Reading First.
Ironies: Those who say single studies shouldn’t be determinative (and they’re right) will nonetheless jump all over this. It’s almost as though there is a disconnect between the political and research process or something… Perhaps this will finally put to rest the idea that Russ Whitehurst cooks studies at IES? And, per the CEP evals, should what educators say only matter when it coincides with various political ideologies? And, as a macro issue, shouldn’t people hope this program works and not be gleeful when the evidence comes back mixed like this? It’s about teaching kids to read for God’s sake!
Do we have an example of a large scale experimental study run out of IES showing positive results for the intervention?
What really struck me was how utterly miniscule the “intervention” really was, in terms of what the teachers actually did. From page 44:
QUOTE: In first grade classrooms, the impact on phonics was statistically significant, while the impact on comprehension was not statistically significant. First grade classroom instruction in schools with Reading First included about 21.4 minutes on phonics and about 23.6 minutes on comprehension per daily reading block. This reflects an estimated daily impact of 3.9 additional minutes for phonics and 2.3 more minutes for comprehension.
• Second grade classroom instruction in schools with Reading First included about 29.2 minutes per daily reading block on comprehension, and about 14.0 minutes on phonics. This reflects an estimated daily impact of 5.3 extra minutes for comprehension and 3.9 extra minutes for phonics, both of which were statistically significant.
Classroom instruction in both first and second grade in schools with Reading First included less time per daily reading block on other dimensions of reading than on comprehension and phonics, as follows: vocabulary (7.8 and 11.6 minutes, respectively), fluency (4.5 and 4.3 minutes, respectively), and phonemic awareness (2.1 and 0.4 minutes, respectively). ENDQUOTE
So it was just a few minutes more phonics, and a few minutes less vocabulary/fluency. Why would anyone be surprised that a few minutes don’t make that much difference?