OK, it’s Labor Day week and you want to beat feet to the beach ASAP but you know this big charter report is coming out this week (Thurs. say those in the know) and that’s going to kill your plans for a fast exit. Fear not, here’s a quick primer to get you to the sand without undue delay:
So, I hear this big new government charter school study based on the NAEP is coming this week, pretty exciting huh?
Well, this is not new data, it’s been analyzed before in previous reports. What is new about this analysis are some new controls for demographics and governance.
Great, so then it will finally settle the question about whether charter schools are better than public schools. That’s big news.
Not exactly. Though it will be treated as such by advocates, it’s not causal data. In other words, the data tell us about some characteristics of students, by school type, at a point in time but not what caused that. Since it cannot tell us about the previous experiences of the students it’s impossible to say, from this data, about what kind of school is more effective. I’d say that this data is most useful for generating some research questions — especially about school governance type (a particular interest of mine) — that future analyses using longitudinal data that tracks students over time can answer.
In addition, the charter sector is getting so heterogeneous (for good and ill) that increasingly “charter” is a meaningless label for a school. For instance what does MATCH, the best open-admission high school in Boston, have in common with some out-of-control online school in Ohio? That problem is going to become even more pronounced if lots of low-performing charter schools get converted into “charter” schools. Update: A reader writes to note that the same variety exists among public schools. I think that’s true and have argued for a long time that we basically have two public school systems. But, I think that’s much better understood while charters are still assumed to be pretty similar to one another–if not a singular program–by casual observers. That said, I would argue that your average traditional public school in Ohio and Boston have more in common than MATCH and say the ECOT. Those two schools are totally different in their operations, how they’re authorized and held accountable, etc…
Wait a minute, isn’t all this prior achievement stuff just a big smokescreen? After all some of the same folks who support charter schools have used the NAEP to castigate the public schools haven’t they?
No, prior achievement matters a lot. Without knowing how students were doing before they came to a particular school it’s impossible to judge the effect the school is having. And this study, and the earlier public – private studies that have been getting so much attention, have the same issues.
That doesn’t mean it tells us nothing, it is an interesting point-in-time data point. Some analysts think that in the land of the blind the one eyed person is king so this data is still useful. Others argue that because this stuff inevitably gets politicized and distorted it’s worse than nothing because it just confuses the issue.
Ironically, in the past when private school proponents would argue that private schools had a greater effect on student learning than public schools, public school defenders were quick to point out that even controlling for various demographics did not mean there were not certain unobservable traits that exerted leverage on the findings. That was true and it also applies here, without controlling for prior achievement there is no way to know if charter school students might also be systematically biased in some way — for instance doing worse in school than other students. And, even controlling for prior achievement there can still be unobservable characteristics, for instance charter students might be more motivated than others or having more trouble fitting into their prior schools. Doesn’t mean that the research is meaningless, far from it, but it does mean that consumers of it need to understand that there is no unmitigated punchline here.*
That said, there is no doubt that some charter proponents have misleadingly attacked the public schools in the past. Some of this is just turnabout being fair play in a high stakes debate where advocates on all sides are going to use whatever data they can to make their case.
But isn’t the NAEP the “gold standard”?
As CGCS’ Mike Casserly once quipped, the NAEP may well be the one “unsullied” thing in public education. But as with all research, the NAEP is good for answering some questions and not for others. Its greatest utility is that it tests a representative sample of students on a regular basis to provide a barometer on educational achievement trends as measured against the NAEP’s standards. Though it’s not testing the same kids each time, the sample is the same so it’s basically an apples to apples comparison over time along that dimension.
So then what’s the issue here, why is everyone so excited about this study?
Because charter schools are a high stakes political and policy issue right now. In fact, to really understand what is going on here you’d be well advised to spend as much time delving into the political theory literature around power as the literature around public schools and charter schools. That’s the crux issue here: Charter schools displace existing power arrangements, people naturally get bent out of shape about that. To be sure there is an empirical component to this debate, but it’s more where the battle is being fought out than its true cause.
C’mon, are you saying there are no bad charter schools?
No, on the contrary, there are too many. The real story on the charter sector is the staggering dissonance, some of the nation’s best public schools and some inexcusable laggards. But this new data is really here nor there in that debate. One really has to look state by state, here and here are case studies of various states, some encouraging, some not. For my money the best doctrine here is Joe Williams’ “Anti-Crap” one whereby we acknowledge that all low-performing schools in the public sector are a problem and that it’s not unreasonable to expect a great deal more from schools.
*This graf is altered from the original, it didn’t paste into the blogger software correctly. Sorry for any confusion.