Charter Schools & Spaghetti

Here’s a story you’ve probably heard before, more than a few times. First, a researcher releases a study comparing the performance of charter and non-charter public schools (in this case, Stanford economist Caroline Hoxby and her colleagues’ report showing substantially higher gains for students who won charter slots in NYC’s lotteries vs. those lotteried out.)  Then, charter proponents and critics square off on the op-ed pages about the results (in this case, see Albany charter school ringleader Tom Carroll vs. NYU education historian Diane Ravitch.)

If all of this seems a little unsatisfying, here’s why.  Say you set out to improve your mother’s beloved spaghetti sauce recipe (treading on even more sacred ground than public education!)  You try ten different variations.  Despite your best efforts, three are worse than the original.  Five are no better, but two are markedly superior.  On average, the new batches are a little worse than your mom’s. But—would you say your experiment was a failure, or a success?

It really depends on what you do next.  It’s a failure if, the next ten times you make spaghetti, you cook the same 10 trial recipes.  But what if instead you avoid the eight bad and OK recipes, make more of the two good ones, and try more new recipes that build on the ones that pleased your palate?  Your average experiment in round 1 was a “failure,” but your average meal going forward is going to be pretty tasty.

You can debate (and we’re sure you will!) whether charter schools match the 3-5-2 distribution in the spaghetti story—we’re not claiming those are the right numbers.  But whatever the exact spread, the basic pattern applies—and so success or failure of chartering depends on what policymakers and sector leaders do next.  Will we vigorously scale-up and replicate schools that work?  Launch a new round of experiments that learn from the lessons so far?  Close or “Try, Try Again” with the ones that haven’t measured up?  Bon appétit.

—Guestbloggers Bryan and Emily Hassel

18 Replies to “Charter Schools & Spaghetti”

  1. Your error is assuming the two good recipes will turn out good again when you try to replicate them. What if all of the variation in your ten experimental recipes was caused by something other than what you controlled – proportions of ingredients, cook times and temperatures, etc.? For example, what if the variation was caused by highly variable quality of the tomatoes or other ingredients, or something about the tasters on a particular day (how hungry the were, or whether they just ate a candy bar or brushed their teeth)?

    It is necessary to run the replication experiment and see the results before we can determine if the average quality will be improved!

  2. You’re also in error because the three bad recipes are still existing and still causing kids to get sick. And why wouldn’t you take the conventional recipes that are superior to the new ones and find out why they work better?

  3. From my experience as a former teacher in a variety of schools, I think student success stems in large part from how motivated and well-behaved the students are (individually, and in the school at large). If a school has many students who disrupt the classes, it is hard for any student to learn. If a school has many students who disparage educational achievement, it will be difficult for the studious students to resist the negative peer pressure.

    Charter schools may have an advantage over public schools because, by definition, they accept only those students who care enough about their education to apply, and then they surround those students with other students of the same mindset.

    While I never taught in a charter school, I taught in other selective schools. I believe that if the most disruptive students were removed from public schools, and all public school students were required to come to class, meet minimal behavior standards, and complete their homework (and their parents were required to be involved as well), their scores would improve as well.

    For good or for ill, public schools are not allowed to expel students who do not attend class, do not do their homework, act disruptively in the classroom, or have parents who do not attend conferences or back to school night. Unless this is changed, I doubt public schools will ever be able to “successfully” educate the most challenging student demographics.

  4. The exact distribution does matter for what policy-makers do next because it could indicate how difficult it is to replicate success. And the study by Hoxby et al is encouraging — the mean distribution of outcomes shown on IV-21 is definitely to the positive side of zero, meaning the distribution of bad-mediocre-good charters is more like 2-2-6, suggesting that scaling up and replicating success is eminently do-able, given how many charters are performing well.

  5. Another question about replicability: Are the ingredients available to make 10 batches of the wonderful spaghetti? Can we find endless supplies of the teachers willing or able to work very long hours at KIPP schools, for example?

    Also, Attorney DC has a point. Even in New York City, successful charter schools were filled by motivated students who could enjoy the schools’ quality but also enjoy the peer effects of being in schools with other motivated students. (Of course, NYC charter schools that did not draw an overflow of applicants could not be included in Hoxby’s study, which might bias the results). I do worry, however, about the prospect of creaming in public schools. What happens to students whose apparent lack of motivation results from outside factors? We do need to reach those students, too.

    I don’t mean to criticize the best charters. But we should think carefully about the replication issue.

  6. I actually really like the metaphor. But here’s where it breaks down: while you have a good deal of control over how a certain recipe will turn out the next time, we have very little control over how a charter school will turn out the next time.

    I suppose we can follow certain authorization methods or only approve charters that do certain types of things, but the whole idea behind charters is that there is little to no oversight, freeing the operators to do as they please. In other words, I’m not sure we can replicate the two great charters with any assurance that they’ll turn out great again. Though it sure would be great if we could.

  7. @Claus: “Even in New York City, successful charter schools were filled by motivated students who could enjoy the schools’ quality but also enjoy the peer effects of being in schools with other motivated students.”

    That’s a really good point, and doesn’t seem to be addressed by Hoxby et al …

  8. Claus: Thanks for backing up my point (above). In your post, you asked: “What happens to students whose apparent lack of motivation results from outside factors? We do need to reach those students, too.” I agree that this is a difficult question. Regardless of the reasons for the lack of motivation or poor behavior on the part of certain students, their disruptive behavior can affect the education of all students. How should these students be handled? Alternative schools? Smaller classes? Military academies? Expulsion? It’s a tough question, with no easy answers.

  9. During my many years as a mother and a teacher, I became certain that the main difference between a neighborhood public school and a private school was the student population. Any public school teacher, especially those in low-performing city schools, will tell you that out of thirty students there might be three or four with severe learning and behavioral problems. Once these children are removed from the class, performance for all will improve dramatically. This is the “secret” of the parochial school. The elite private school takes it a step further by taking only the brightest children. Almost none of these schools welcome the severely learning disabled and the seriously disruptive child.

    Charter schools base their success on the “secret” but they don’t admit to it; and that’s the part many of us object to. Some are undoubtedly very successful but when advocates say they work with the same population as the public schools, we know they are being disingenuous: “Oh, we take children of similar backgrounds, except of course their parents had to apply and sign a contract.” What perplexes me is why so many intelligent people support this fiction. Are they lying or do they want to believe so badly that they are fooling themselves?

    There are some very successful magnet programs, charters and parochial schools in my city and every one of them is selective. The more successful they are, the more selective they are. The same applies to colleges and universities.

    I am not against charter schools, especially if they are strictly nonprofit. However, their leaders should be honest and admit that the severe discipline problems are not welcome. There’s nothing wrong with that. I’d like to see all such children placed on home study or accompanied by a parent or aide at all times. Every child with a severe learning problem should also have an aide to offer individual assistance.

    The best thing we can do to improve education is to be honest about the problems we face.

  10. Gosh isn’t it amazing that so many people say the same thing in the blogosphere. Perhaps people outside of neighborhood schools don’t realize how much effort goes into to keeping classroom teachers from articulating these sentiments. But guess what? As soon as the theorists and the administrators are out of earshot, almost all teachers continue to repeat the same thoughts.

  11. The study that you are referring to did not reach any conclusions about why charter schools succeeded. The reason that the scores are high is because these charter schools can and do expel students out of their schools if they do not perform at a certain level in terms of test scores. See “Charter schools pawn off
    flunking students, says public school principal”

    Read more:

    The reasons given for the results were anecdotal. The study did not reach any conclusions about why charter schools succeeded, but noted that many had extended school days and school
    years, mandatory Saturday classes, performance-based pay for teachers and a disciplinary policy that punishes small infractions and rewards courtesy.

    Also, the individual doing this study, as you noted, is an economist, not an educator. If you want to see a study regarding charter schools that was done by a team of educators also from Stanford, see “PACE issues scathing report of charter schools”. This study was paid for by the Wal-Mart Foundation who were proponents of charter schools.

    A small study that doesn’t ask the question of “why” in terms of the results is in my view not a valid study. It appears that test scores over a certain amount of time were gathered and used to provide the statistics necessary to support the existence of charter schools. That’s all that was done.

    Also, charter schools hire young and inexperienced teachers who don’t mind working the longer hours and receiving minimum pay and benefits. They also don’t mind the merit pay system where their income is based on how well their students perform on a test. See “David B. Cohen and Alex Kajitani: Test scores poor tool for teacher evaluation” .

    This one study does not validate anything about charter schools one
    way or the other.

  12. All charter schools are not bad. Most charter school where form to look out for the best intrest of the children. Then somewhere the school system loose focus of the best interest of the students. It become a money issue and put anybody in the school may it be teacher or students. I as I was reading it didnt give the pro’s and con’s of charter school.

  13. Hopefully I’m not going to state the obvious. Do Businesses create culture, people, and societies in the way schools do? Is it the school or the community that needs fixing or both? Research and data are more valuable than opinions. Can there ever be a panacea? Transformational leadership on the level of Ghandi, MLK, and other martyrs is a necessary ingredient, who among us has that to give? Money matters. There’s no real lack of it but we’re still busy maintaining nukes. De facto segregation and dilapidated schools are good indications of the residue of the past. I attended schools in a smallish town in Oregon on a shoestring budget but they were comparable to governor’s schools in terms of educantional programs and infrastructure. How is this possible? We must listen to Socrates and remember that as long as we fail to enlighten the misguided they will be a danger to all and it is our responsibility to be our brother’s keepers. If you doubt this then show me a prison with a population of highly educated people. Schools are not making products, they are molding societies; therefore, we must improve the quality of life at the same time that we improve schools. Please excuse my writing style for there are many thoughts compacted in this note which I am willing to defend with research and further explain/discuss on request.

  14. It was interesting to read people’s assumptions about charter schools. I have to admit I had them too until I started teaching at a charter school. Our Science Academy does 1) have higher standards than public schools, even though we follow the state standards; 2) level the classes so we actually have advanced students in different classes than grade level students; 3) have a very long waiting list of students whose parents want them to get in; 4) have smaller class sizes (but not by much – my Advanced class is 26 and my grade level is 27) than the public schools; 5) get funding but it’s much less than public schools; 6) have SPED students on IEPs and 504s; 7) have just as many ADDs as any public school classroom [I’ve got five in my grade level class right now]; 8) have more involved parents who do make their children do their homework and follow the discipline guidelines; 9) have just as many parents who dump their children off every day and don’t care about them at all; and finally, 10) have teachers who are highly qualified with Masters degrees who get paid less than or only just comparable to the low end public school beginning teacher. So, while we do have discipline problems where we’d like to expel students — we don’t dump them on the public schools any more than we’re dumped on. I always thought charters were where the problem students went when the public school was just short of… well, you know. We can, however, tell the problem child and their parent that student is not keeping up with their peers and highly recommend that they find somewhere else to go. In other words, we have no problem failing a student, and we do it unless they have a 504 or IEP. That part of the charter school myth is true. We just adopted a school uniform policy this year and, since we are a college preparation (not preparatory) school, we can demand more than a public school in terms of homework and research papers, etc., and get it. Our test scores are high and we work very hard to keep them that way. Arizona’s state test is based on our state standards, so the teach-to-the-test issue isn’t one — if we’re teaching the state standards, our students are getting what they need to pass the tests.
    The schools are the staff as much as they’re the community. While I would hate to be a teacher at the school in Detroit where the honors student was killed by railroad tie-wielding thugs last week, if I did teach there I would still have high expectations for my students. Chances are, the killers were not students, but out-on-the-streets males who didn’t know how to read and write but didn’t stay in school to learn how to either. I know I’m the one making assumptions now, but I bet I’m not far from the truth. Those kids let themselves down and so did their parents. Choices were made. It’s their responsibility — just like it’s my students’ responsibility to do their homework every night. Very few people take responsibility for their own actions and consequences — everybody just wants to blame somebody else. If you ask me, that’s the main thing that’s wrong with education these days. Not charter schools. (And please stop making assumptions that are not based in fact about us.)

  15. There are two unstated assumptions not in this discussion. First, that school officials (charter or otherwise) move beyond the newspaper and review the original report. The second relates to the first, that those new charter schools will learn from the mistakes of other school administrators. From my observations, school admin’s make the same mistakes as in other schools. There is no moving forward because these people are learning only by personal experience.

Leave a Reply

Your email address will not be published.