Lotteries, Monkey Business and Other Silliness

We can take a joke, so we had a good chuckle at Andy Smarick’s response over at Flypaper to our Try, Try Again post here on Eduwonk.  But the thing is, we think he may be serious, so we better respond!  In Try, Try Again, we argue that even if turnarounds (or new schools, for that matter) succeed only 10-30% of the time, if we rapidly retry when they don’t work, we can achieve 40-80% cumulative success rates over time.

The essence of Smarick’s response is that the chances of success for turnarounds are much much lower than 10-30%— more along the lines of lottery odds (one in millions) or the chance that a monkey hitting random keys would type the full text of Hamlet (one in some incredibly large number).  But this is silly.  In sectors other than education, where real turnaround attempts are much more common, the odds of success aren’t, as Smarick suggests, “infinitesimal.” Some change gurus say major change efforts in companies succeed 30% of the time.  But even if we could hit 10-20% in K–12, if we also retried rapidly, we could turn around half of all failing schools in several years, helping millions of kids.

Andy doesn’t say so in this post, but from prior debates we know that instead of turnarounds, Andy advocates new school creation.  We’re strong backers of that strategy as well, both to replace failing schools and to create wholly new options.  But what Andy doesn’t seem to want to acknowledge is that failure rates are very high for new start-ups as well.  According to a Harvard study, even proven entrepreneurs who launch new businesses only succeed 34% of the time.  Since the scale-up pace of proven new school operators isn’t nearly fast enough to meet the urgent need we have here, with millions of kids trapped in thousands of failing schools, we’ll need new entrants too, who in the Harvard study succeed 22% of the time.  So let’s stop monkeying around and place serious bets on both turnarounds and new school creation—both of which will require rapid retries a la “try, try again” to help a lot of kids fast.

—Guestbloggers Bryan and Emily Hassel

8 Replies to “Lotteries, Monkey Business and Other Silliness”

  1. What is really needed is for people to analyze what makes schools successful and then apply those lessons to existing schools. If I had to guess, the answers are probably discipline, selectivity, and money. Many attendance admissions high schools I know of in Chicago don’t have effective detention policies, but Noble Street does. That is the kind of change these schools need, rather than the massively more expensive intervention of closing, phase out, smallification, staff replacement, etc.

  2. Thanks, Hassels, for interesting discussion.

    My thoughts:

    1. What are odds of creating a good urban charter school from scratch? Depends on “good.” Perhaps 250 out of 2500 are No Excuses high flyers, right? So that’s 1 in 10.

    But then there’s another group of “solid” charters. School culture and test scores are definitely better than district. What % of from scratch charters fit this category?

    2. What do we want from a turnaround? Let’s say Locke High progress, just for sake of argument, stopped right here. Much more stable place for a kid. Not measurable gains in learning yet.
    Success or no?

    If yes, then chances of success are much higher. If goal is for turnaround to end up with a No Excuses caliber school with huge academic gains and not just decent school culture – which seems to be 1-in-10 with all the advantages of being from scratch – then we’d probably guess 1-in-100 perhaps?

    3. If a “turnaround” can be mostly a from scratch school which has a few bells and whistles to make it more politically palatable, then we have a terminology grey area.

    Ie, think NOLA model for a failing high school. Most people fired. Some teachers and a new principal teach Grades 10, 11, 12 in Year 1; then 11, 12 in Year 2; then 12 in Year 3, then gone.

    Meanwhile, new operators bring in a new 9th grade with entirely new staff, culture, rules, norms.

    Is that a turnaround or a new school? It’s mostly a new school which can slip under the turnaround umbrella.

  3. I’m willing to take Rapid Retry and the 33% success rate for granted and agree with our authors here that more turnarounds can be achieved more quickly — simply by retrying more often.

    But I’ve been thinking a lot about the work our company has done and tried to clarify for myself those instances when it has succeeded and when it has failed. The area of change I’m most committed to and experienced with is change in instruction. And my experience tells me that some very small changes have very large impacts in certain school situations.

    For example, we saw a nice spike in teacher morale and student productivity in one large district when we merely taught teachers some basic classroom management tools — mostly procedural management and classroom participation criteria with student self-assessment.

    At other times, I’ve received reports back from people who use some of our stuff and have heard that one particular teaching strategy has made a huge impact on test scores.

    In several other cases, moving teachers away from “formula” writing to authentic writing forms (along with student-selected topics) has also made a big difference.

    Since we don’t get to change building leadership in our work, my insights are generally about working with existing leadership. To that effect, if I can get a principal to actually tell his or her staff to use some of our teaching tools, we see good results. (Most of the time, principals would prefer to remain quiet and simply stay in their offices.)

    As I’ve consulted my way from school to school, I’ve noticed that changes in teaching practice, especially in literacy, assessment, and classroom management, can have a dramatic effect on student achievement and teacher culture.

    If one were to add to Rapid Retry and the 33% success idea the regular and consistent influx of new and high quality teaching practice, my hunch is that change would come even more quickly and sustain itself much longer. Or, perhaps just as important, the retry process would “fail faster” thereby making it easy to decide when to start up again.

  4. Failure rates are due to students not having the skills they need to be successful. As a musician, the learning that takes place is one note at a time and one new rhythm being introduced and repeated enough so the students make the connection between the language of music and their instrument. When students do not make these connections, they will not continue to play their instrument. This is the same with learning new concepts or skills.

    I taught in one school where my homeroom students were failing at the 40% rate to become sophomores. I encouraged these students to complete their assignments, their homework, and complete extra work in order to be successful. Some of these students chose to respond, while others chose not to. Is it how we bring the “horse to water” that matters the most?

  5. I think guys are getting yourself deeper in a hole. You seem to forget that it s not estimated rates of success that turnaround schools. It is human beings that turn around schools. Your letting the debate take you further away from reality and into a question along the lines of what would be the optimal number of angels dancing on a pin.

    I checked out your links earlier in the week but didn’t have time to respond. None volunteered the most important data if you want an honest conversation. We can’t have constructive discussions unless you guys volunteer information upfront about he amount of intended and unintended “creaming” that occurs. Creaming can come from an overt selection process, or just having the students apply, or by setting academic and/or other standards that create selectivity, and it can happen in an open or a covert manner.

    The Westwood HS in Memphis implies creaming when reporting that the biggest disciplinary challenges were “targeted” but then the reader is left hanging. Chicago shipped out large numbers of more difficult students using rationales as ridiculous as claiming that a change in scenery might do their disruptive kids some good. But its that refusal to be candid that is so maddening. How many times this year have you guys broken up fights and ended on the floor? How much blood have you cleaned up? How many of your kids have been arrested in the first six weeks? And please remember that when you get the right to create more optimal conditions, that means that some of your toughest problems are dumped on us.

    I’d define creaming as selectivity that occurs when schools do not have to follow that policies of neighborhood schools have to follow. Nobody should begrudge you some relief from the suicidal policies imposed on us – in a written and unwritten manner. All schools should be able to enforce their disciplinary and attendance policies, and if you are empowered to do so, good for you. Nobody should begrudge a certain amount of creaming, as long as you come clean.

    And all of the above is mirrored in the teacher side of the equation. I won’t begrudge you guys more freedom to selective your teachers to get things up and running. But there has to be a balance. Take away too many contractual rights and progess won’t be sustainable.
    And take away creaming and you’ll have to offer a lot more protections to keep a quality staff.

    Here’s my thought experiment. The AFT Toledo Plan regularly removes nearly 8% of teachers who shouldn’t be in the schools. Kane estimated a smaller percentage need to go, but lets’s take those ballpark figures.

    What would happen if 5 to 8% of teachers were removed, and a comparable number of students whose behavior shows that they are not emotionally capable of functioning in the chaos of neighborhood school were provided alternative services? The good news would be that many poor schools would have never become dysfnctional and they would not have needed to be turned around.

    The bad news is that we might not have anything to fight over.

  6. John,

    As always, you bring up many good points. What intrigues me about BOTH your argument and that of the authors is that you both mention relatively simple changes, with common sense statistical thinking, that would almost certainly make a huge difference in many schools.

    I’ve often felt, while teaching 30 middle or high schoolers that the chaos I was encountering was really a chain reaction initiated consistently by one or two kids who really had a hard time just being in school. And I think everyone could agree that 5% of the teaching profession might be more successful working in other jobs. As you point out, however, additional services would need to be available for kids who struggled with the school environment.

    Interestingly, if we combine your thoughts with the those of the authors’, we might save enough money, while enhancing student achievement, to provide additional services for our most struggling kids.

    As someone who spends a lot of time in front of many different kinds of teachers and kids in many different school situations, I am sympathetic to both sides of the coin that has been tossed here. I’ve also bought into the popular notion that education is a civil right. So in my heart of hearts, it is getting harder to tolerate that very small number of kids who impinge on other kids’ ability to learn. And because I have such an up close and personal view of teaching talent nationwide, I would like very much to see plans like the AFT Toledo plan you mention where some small percentage of low-achieving teachers are replaced. Many companies, for example, routinely replace their lowest-achieving workers — or figure out cost-effective ways to “manage them up” to some standard. Having logged almost 15 years training teachers, I have not experienced much success managing up the weakest candidates. Kids, however, are entirely different. With the right services, I have seen rough, tough kids make their way back into school in a relatively short period of time. Admittedly, few of these kids get the help they need. But providing for this could be a partial strategy for reform if more people wanted to advocate for it.

    The Rapid Retry model suggested by the authors would, it seems to me, have to include aspects of all of these ideas. If moving the toughest kids from “school” to an alternative service were part of a reform paradigm, we might not have to call it “creaming”. And if we routinely removed the lowest-performing teachers, we might improve a school’s teaching corps dramatically over the “retry” period.

    Knowing a little about you and the authors, I would imagine you’re closer to each other than either of you might think.

  7. Bryan and Emily – There are cases when a “turnaround” (or creating a new school) may make sense, but I think I’m with Andy on this one on the futility (and dangers) of speeding up turnaround efforts. I think there are a couple of issues here. 1) In addition to the lottery ticket example, think about baseball: sending a hitter up to the plate with a .200 batting average will yield more hits, but also more outs/failures. (Of course, if, as you’ve implied elsewhere, the .200 hitter gets better by getting more practice, that’s a different story and a different argument, but right now that’s not happening fast enough either!). 2) From a systemic perspective, under current circumstances – where the need for “turnarounds” far exceeds the experienced personnel, resources, and programs available – trying more often is only likely to strain the system further and perhaps do even more harm than good. 3) The real problem is that simply trying more often does nothing to actually build the capacity for large numbers of schools to be successful and sustain high performance over the long-haul. That comes from the difficult, basic, work of creating the political, social, economic and educational conditions that can support improvements on a wide scale…

  8. What percentage of schools have effective “behavior disorder” programs? Ours just got cut:( We now get “inclusion” because we’re short dollars. Go figure.

Leave a Reply

Your email address will not be published.