Last Friday the National Board for Professional Teaching Standards released the now infamous Sanders study. Ed Week’s Bess Keller reports here. It was a good move by the board because national reporters were increasingly focusing on the story (see also Josh Benton and Linda Seebach) and they needed to get in front of this.
As for the study itself (pdf) it’s pretty much what it was billed as. The topline finding is: the amount of variability among teachers with the same NBPTS Certification Status is considerably larger than the differences between teachers of different Status. Consequently, a student who is randomly assigned to a National Board Certified teacher is not much more likely to get an “effective” teacher (or an “ineffective” teacher) than a student assigned to a teacher who has never been in the NBPTS process (or one who failed certification, or one who may in the future become certified).
That’s not inconsistent with other research about teacher credentialing including the recent Hamiltonian opus. In other words, these screens often don’t serve as the quality assurance that proponents say they do. That’s an enormous issue for policymakers to get a handle on that goes far beyond the National Board.
That said, this isn’t outright damning for the National Board. For starters, it’s just one study. The National Board is undertaking a comprehensive evaluation and it’s wise to withhold judgment until that process provides more evidence. They didn’t help themselves with how they handled this and mortgaged a lot of the cred they’d built up by doing the evaluation project, but the jury is still out. But, they might want to encourage their partisans to temper their rhetoric a bit. The Goldhaber study was, as I said, significant as these things go but not a slam dunk and this one raises some legitimate questions.
So what’s the big picture difference between Sanders and Dan Goldhaber’s study (pdf)? Not a lot actually, roughly similar effect sizes but Sanders uses multiple models and finds a lot of noise in some so he consequently highlights the not insignificant issue of intra-route variability, meaning there more variability within routes (e.g. National Board certified or not) than between them. That’s significant because it (a) raises cost-benefit issues relative to the National Board as a process to ensure or improve quality (b) raises the related question of whether there are more cost-effective ways to identify top teachers and (c) raises questions about the reliability and value of the certification process. Sanders and Goldhaber use different methodological approaches so look for some back and forth about the relative value of each going forward. Sanders makes a pretty strong claim that his method is superior which as these things go is like slapping Dashing Dan across the face with a glove…
The National Board also includes a summary of the peer reviews on their site (pdf). Some of the issues are smokescreen stuff, for instance sample size. But the reviewers also question the impact of a change to North Carolina’s test during the study and whether there is a ceiling effect on student achievement since National Board teachers generally teach higher performing students (a problem in its own right)*. I don’t know enough about NC’s system to make heads or tails of those claims but I have a hunch that if they were genuinely serious threats to the validity of the findings more would be made of them** and there is no reason to think, at least before this recent episode, that Sanders had any agenda with the National Board.
The National Board has some strategic thinking to do going forward. Like it or not, student achievement is the coin of the realm here so the “trust us, we know best, now bugger off” days are a thing of the past. National Board President Joseph Aquerrebere signaled as much when he told a reporter, “I am disappointed with the results.” That’s the right posture and hopefully the board will take the study to heart and evaluate their programs. [Here is where they should put aside the anti-TFA animus and learn from what Teach For America does to refine its recruitment criteria. TFA is the industry standard right now for that process.] By itself, the Sanders study will only be as damning as the National Board makes it through their public and substantive response going forward. A more portable and real credential-based system for teachers makes a lot of sense and the National Board has a role to play there. But there is a lot of public money at stake here and a lot on the line for them going forward. Too many more episodes like this, either on form or content and they’re going to have some problems.
*Several readers want to know what I think this means for efforts to get National Board certified teachers into high-poverty schools? Well, as I’ve said, all else equal states are spending a lot of money on incentives for NBCT’s and so those incentives ought to be tied to efforts to improve equity in teacher quality and in most places they are not. States are still going to be spending a lot on those incentives for the foreseeable future so all else is still equal.
**Meaning in the year+ that the National Board had the report prior to its public release they would have had the data reanalyzed if this were indeed significant.