CMOs – Are We There Yet? Guest Post By Lake & Hill

Last Friday a major study looking at charter school management organizations was released.  Here’s my take on it from TIME. And here’s an Eduwonk post about it.  Now, below, for the first time the researchers spell out what they see as some key takeaways:

CMO’s: Are We There Yet?   By Robin Lake and Paul Hill, Center on Reinventing Public Education at the University of Washington

Charter management organizations were built to bring consistency and scale to the charter movement by replicating high-performing schools in large numbers. Have they delivered? A new study by the Center on Reinventing Public Education and Mathematica Policy Research shows that, unfortunately, the answer is not yet.  CMOs have created a number of high-performing charters, but many others don’t perform as well as nearby public schools. As is the case with independent charters, there is a great deal of variation in the effectiveness of CMOs.

The good news is very good. A number of CMOs have stellar results. They make it possible for students to make three years of learning gains after just two years of enrollment. CMOs that get strong results in reading and math also do well in other subjects, suggesting that these schools are not simply prepping students for the most tested subjects. And the most effective CMOs have positive impacts that are substantially larger than the negative impacts of the least effective CMOs. We found that high-performing CMOs tend to have intensive professional coaching of teachers and high expectations for student behavior, with clearly defined discipline policies. Teacher coaching is often packaged with a sophisticated use of data and more instructional time.  But the news is not all good. CMOs included in our study appear to differ dramatically in the value they add as measured by test scores. While about half of the CMOs we studied produced positive results, about a third showed negative results in math or reading. This is well short of the consistency that we should expect from efforts to replicate high quality charters.

(An important caveat – the overall quality of CMOs may be better or worse than we know right now.  Our study used the most rigorous methods available to eliminate selection bias and isolate the value CMOs add to student achievement. That level of rigor requires achievement measures for each student before entering the CMO school—measures which do not exist in pre-kindergarten, thereby eliminating elementary schools from our analysis. What’s more, we haven’t yet looked at college admission and success, which some CMOs emphasize over test scores.  A fuller picture of school effectiveness should include college attainment and success.  We’ll have more results on that front soon.)

Many have argued that CMOs deserve fast-track approval processes and require more limited government oversight. While streamlined and customized application processes have promise, they can’t substitute for rigorous oversight.  And charter school authorizers must realize that scale brings its own challenges.  In some cases, CMOS are in danger of being too big to fail, given the thousands of students they serve. Poor authorizing or investment strategies, then, carry all the more risk, putting a premium on discriminating up front due diligence and ongoing oversight.

So where do we go from here? Giving up on replication of excellent schools is not an option. U.S. public education needs thousands more schools that help students excel in the ways that the top-performing charters, CMOs, and conventional public schools do. But funders and charter authorizers need to recognize that all CMOs are not created equal. They need to carefully assess CMOs’ achievement records—carefully measured in “value-added” terms—their instructional and operational models, and their financial viability before they give the green light and dollars to start new schools.

For their part, CMO leaders must attend to uneven quality, perhaps starting by focusing on promising practices such as intensive coaching of teachers and comprehensive behavior policies. In addition, they need to figure out how they can grow without continued infusions of philanthropic dollars. That means experimenting with technology to reduce costs, engage students, and help teachers reduce their workloads. It means innovating with strategies to run central operations more efficiently, partner with other organizations that can provide services at lower costs, or negotiate with school districts to provide affordable buildings.  And while doing this, many CMOs need to find ways to serve more students with disabilities and English language limitations.

The bottom line is that many CMOs are very effective, but charter authorizers and funders should not assume that schools affiliated with CMOs are necessarily better than independent charters. More than a half-billion philanthropic dollars have supported CMO development and expansion in the last 10 years. Large grants by the U.S. Department of Education are supporting the replication of effective charter schools generally and the growth of CMOs specifically. Our study suggests that such investments must be directed with care to support quality and scale in the charter sector. Put plainly: The next generation of scaling strategies should build on CMO successes and acknowledge their limitations.

19 Replies to “CMOs – Are We There Yet? Guest Post By Lake & Hill”

  1. As with virtually all charter school studies, the results must be discounted to reflect the charter schools’ two inherent systemic advantages vis-a-vis the neighborhood schools — 1) skim-the-cream; and 2) credible-expulsion-threat.

    Skimming the pages of the study where the authors outline their efforts to control for “selection bias”, it appears that none of the charters studied were populated by students assigned from the neighborhood schools — that is, all of the charters studied were populated by students whose parents who were motivated enough to apply for the charters and functional enough to complete the application process.

    The study looked at race and income in the charter vs. the neighborhood schools, but this does little to address the motivated/functional-parent bias. Although the study looked at pre-charter-enrollment student achievement, the study reports that, for minority/poor students at least, the charter students had slightly higher pre-enrollment achievement levels than the neighborhood students — evidence of selection bias. (Of course, even if charter pre-enrollment achievement levels were comparable to neighborhood-school achievement levels, the charters’ motivated/functional-parent advantage would still exist; the presence in the neighborhood schools of large numbers of children of unmotivated/disfunctional parents might reasonably have prevented the charter students from reaching the achievement levels they were capable of in a school where all the students had motivated/functional parents.)

    And, as in most charters, the charters had fewer LD and ESL students than neighborhood schools.

    Apparently, the study made no attempt to control for the charters’ credible-threat-of-expulsion advantage — not a criticism of the study (hard to see how a study could control for this), but it’s an inherent charter advantage.

    Until/unless charters are populated by students assigned randomly from the neighborhood schools, comparisons of charters to neighborhood schools cannot be cited as evidence that charters are better than neighborhood schools. This study might tell us something about which charters are better than other charters, but it tells us nothing re the major issue of whether charters are good for US education.

  2. Labor lawyer – please read the study, not just skim, for a description of experimental and other methods designed to address exactly the questions you raise, including expulsion. We certainly never made any claim, nor was it the intent of the study, to assess whether charters writ large are “good for public education”. Robin Lake

  3. Robin — The study is 179 pp; the table of contents references several sections including the appendices) that might address these issues. Could you respond in a comment listing the pages that address these issues? Thanks.

  4. LaborLawyer, if you didn’t bother to read up on the study’s methodology, why did you write a detailed comment about the flaws of said methods?

  5. “all of the charters studied were populated by students whose parents who were motivated enough to apply for the charters and functional enough to complete the application process.”

    There’s no reason to think that this is an overall advantage. Perhaps many of the most “motivated” parents are the very ones whose children aren’t doing very well (hence the motivation to seek alternatives). That would be a disadvantage for charter schools.

  6. All of you who post on this blog sound like very intelligent people, so you must know that the only way to compare a charter school with a traditional public school is to open it up to all comers, even the ones who come in September. There can be no unusual application forms (other than the same forms used in the traditional schools) or promises from parents. In other words, the conditions for being accepted into the school must be EXACTLY the same as for the traditional school. There can’t be any selection bias or the appearance of one.

    And it is my OPINION (for Chris) that the most critical variable in the education of the child is an involved parent.

  7. Linda, this would be a great time for you to also read the methods section.

    And yes, I’m very aware of the status of your opinions, as I wrote before:

    “Yes, I know, thus the inherent lack of utility in your comments when you make far-reaching claims concerning teacher accountability and the plight of teachers everywhere.”

    Opinions are generally not worth much if they rely on logical fallacies and/or rarely have a basis in reality, but as you sound like a very intelligent person, you must know this already.

  8. Linda,

    Can you at once claim that parental involvement and SES are the most critical factor in the educational success of a child?

    Since we’re all in the mood to offer anecdotes, I once had a student that had a parent that wanted to move them to a charter school. Certainly a more involved than normal parent, but this student had only a D in my class. Parental involvement isn’t all its cracked up to be, and just as none of the other variables in the equation are a silver bullet, parental involvement isn’t either.

  9. Yes, it is my opinion that parental involvement is the most critical variable in the education of the child, but of course it is not the only important variable. Another critical variable would be the health of the child. That said, there were many, many sickly children with uninvolved parents who grew up to be splendidly educated and made important contributions to the world.

    I do agree with you that parental involvement is no silver bullet. I too had students who performed very poorly despite heavily involved parents.

  10. We certainly never made any claim, nor was it the intent of the study, to assess whether charters writ large are “good for public education”. Robin Lake

  11. Robin, Chris Smyr — Re the skim-the-cream and credible-threat-of-expulsion advantages that charter schools enjoy vis-a-vis neighborhood schools:

    I skimmed the study’s table of contents and then read the sections of the study that, per the table of contents, seemed to address the skim-the-cream advantage. Those sections described how the study tried to address the advantage by looking at 1) charter vs. neighborhood pre-charter-enrollment test scores; 2) charter vs. neighborhood low-income percentages; and 3) charter vs. neighborhood minority percentages. The table of contents seemed not to reference any text that would have addressed the credible-threat-of-expulsion advantage.

    None of these approaches described in the text are likely to have substantially controlled for the skim-the-cream advantage that an application-based charter school inherently enjoys over a neighborhood school. The charter-vs.-neighborhood-pre-charter-enrollment-test-score analysis might, to some extent, control for the motivated-parent factor — that is, if a student has motivated parents, the student is likely to score higher on standardized tests and therefore, if charter school students and neighborhood school students score roughly the same on the pre-charter-enrollment standardized tests, this arguably suggests that the charter students do not have more motivated parents than the neighborhood students. However, given that charter school parents are, almost by definition, more motivated/functional — on average — than parents of neighborhood school students (that is, they all made the effort to apply to the charter school and were sufficiently functional to complete the application process — a significant accomplishment for a low SES inner-city parent), the fact that pre-charter-enrollment standardized test scores were comparable for charter and neighborhood students suggests that these test scores do not measure parental motivation. Rather than suggesting equal parent motivation, the fact of comparable pre-charter standardized test scores suggests that the presence of children of unmotivated parents in the neighborhood schools overwhelmed the effect of motivated parents in the pre-charter-enrollment neighborhood school classes. Granted, there may be some ciruclar reasoning at play here.

    Bottom line — particularly when dealing with low-SES inner-city parents, any charter school that restricts enrollment to children of parents who submit applications and complete a multi-step application process will necessarily have a significant motivated/functional-parent advantage over neighborhood schools that do not have an application process. The only way that a charter school can convincingly refute the skim-the-cream argument is for the charter school to be populated entirely by students randomly assigned to the charter school from the neighborhood school population. To my knowledge, this has happened only once — in a Denver-area charter school and the charter experiment there quickly failed.

  12. How many charter schools are there where they have taken over an existing school and kept the student population?
    I know of only three:
    KIPP in Colorado, which ended with KIPP withdrawing failure.
    Dunbar HS in DC
    Coolidge HS in DC both run by Friends of Dunbar
    How have they succeeded?

  13. LaborLawyer,

    Definitely some circular reasoning going on there wrt parental motivation and measurable impacts on student achievement. Stuart Buck had a good point on the matter.

    As for more convincing controls, read further into the executive summary:

    “We tested the validity of our “propensity-score matching” method in a subset of CMO schools where it was possible to conduct a randomized experiment—the “gold standard” of evaluation methodology. The matching approach successfully replicated experimental results, thereby providing some confidence that it can produce valid impact estimates in the much larger number of CMO schools where it can be applied but where the experimental analysis is not possible. We report impacts in standard deviation units (also known as z-scores) to allow comparisons across grades and states that have different test scales.”

    and later:

    “Test scores measured prior to CMO entry are critical to our NXP approach. Prior research in various topical areas has demonstrated that NXP methods can replicate the findings of randomized experimental studies if the researchers have a good pretreatment measure of the outcome of interest (Glazerman et al. 2003; Cook et al. 2008). In addition, our study provided an opportunity to directly test the NXP methods against experimental results in the subset of CMO schools for which admissions lottery data are available and usable. The success of this test—described later in this chapter and in more detail in Appendix B—confirms that in the CMO context, nonexperimental methods can reproduce rigorous experimental findings if they can make use of pretreatment measures of relevant outcomes along with other student-level covariates that are widely available in administrative data.

    […]

    The experimental sample frame consists of students who applied to an oversubscribed CMO school that used a random lottery to admit students. The treatment group is composed of applicants offered admission to a participating CMO school at the time of the lottery.12 Applicants not offered admission at the time of the lottery form the control group. All students who provided consent, were in the correct application grade at the time of the lottery, were randomized in the lottery, were in the proper grade for the lottery, and had baseline test scores were included in the analysis”

    If you’re interested, much relevant discussion can also be found here:

    eduwonk.com/2011/04/still-going-2.html#comments

  14. Chris: I think that LaborLawyer’s point is twofold: (1) The charter schools must de facto “skim” the more motivated families, because only families that are interested and capable of completing the application procedures can send their kids to the schools; and (2) The students attending charter schools attend school ONLY with other similarly-motivated students, while the students remaining at the public schools (including those who “lost” the charter lottery) continue to attend school with many other students, including those who are much less motivated.

    In my experience (having taught in several schools, including low-income schools), many students in a typical inner city school misbehave in class to an extent that they interrupt the learning of their fellow classmates and disrupt even the best laid lesson plans of their teachers. They are usually allowed to continue to remain in class, even with a substantial history of disruptive and inappropriate classroom conduct. Based on the above, I would surmise that the charter lottery “losers” who remain in their low-performing public schools are significantly negatively affected by their classmates (most of whom didn’t apply to charter schools).

    Point being that studies such as the one referenced here don’t “prove” that charter schools don’t skim.

    If I were a parent living in the inner city, I’d prefer my kids to go to school only with other motivated, well-behaved kids as well. I don’t fault the parents for their choice to remove their children from public schools for the charter school option. However, charter schools continue to have the advantage of teaching only motivated students/families and also have the advantage that the kids attending the school believe that they can be expelled if they don’t live up to the charter school contract.

  15. Chris Smyr — I read the cited pages, but it’s so complicated I don’t clearly understand what the authors are saying. As best I can make out, the study supplements the usual lottery-loser-control approach by identifying non-lottery control students in the neighborhood schools whose achievement scores were comparable to the pre-CMO-enrollment achievement scores of the charter students.

    To the extent that pre-CMO-enrollment achievement scores adequately measure parental motivation, this approach has some validity in addressing the issue of whether charters enjoy a parental-motivation advantage. However, as I noted above, it is equally likely that, in the neighborhood schools, the presence of large numbers of students of unmotivated parents so disrupts pre-charter-enrollment instruction that the students of motivated parents and of unmotivated parents have comparable achievement scores. In other words, in inner-city schools, having motivated parents does not significantly impact a student’s achievement scores until/unless the student is placed in a school environment where most of the fellow students also have motivated parents.

    And, even if this approach allows one to control for parental motivation (in comparing post-enrollment student achievement for the charter school to the neighborhood school), it does not allow one to control for the effect on student achievement of placing students in a school where virtually all the students have motivated parents (or for the charter’s credible-threat-of-expulsion advantage). Therefore, comparing the post-charter-enrollment achievement scores of charter students to the achievement scores of “comparable” neighborhood students cannot show that charter schools are better/worse than neighborhood schools.

    The only way to eliminate the skim-the-cream advantage is for the charter to be populated entirely by students randomly assigned from the neighborhood schools. The fact that extremely few charters have ever sought to populate their schools with students randomly assigned from the neighborhood schools is, itself, strong evidence that those who actually run charter schools recognize and desire this skim-the-cream advantage.

  16. In my post above (Nov. 11, 2:15 pm) I made a mistake. If a researcher wanted a scientific study designed to compare a charter school with a traditional school in the same neighborhood, he’d have to find a charter that took over a traditonal school and brought in its own faculty and methods. In addition to that, there would have to be a way to ensure the validity of the testing. By this I mean the test would be unseen by school personnel until the day of administration and it would have to be handled by a neutral agency.

    I’d like to stress another point (again) because I think it is not understood by many people, even though it is extremely important in any discussion of charter schools. It is this:

    In even the poorest schools there are children who excel. By this I mean they are grade level or above and they are well-behaved and motivated to succeed. These children often come from homes that are just as poor as the other children in the class. So, for example, “Jess” and “Kim” might both come from homes with similar demographics, but Kim could actually be an educationally advantaged child in the sense that her parents are very involved in her education and take advantage of the many educational opportunities provided by the community. Based on my experience, I would guess that about one-fourth to one-third of every class in a very poor school would fit this description of “educationally advantaged.” If you started a school that attracted these children, you’d have a school that competes nicely with any school in the suburbs.

  17. LaborLawyer said:

    However, as I noted above, it is equally likely that, in the neighborhood schools, the presence of large numbers of students of unmotivated parents so disrupts pre-charter-enrollment instruction that the students of motivated parents and of unmotivated parents have comparable achievement scores.

    You’d have to substantiate the “equally” descriptor, but yes such peer effects are possible. The study is able to confirm that there is a difference between academic achievement attainment for similarly motivated students/families if they attend charters or not. The impact likely stems from in-school variables. This fact alone ought to shake many critics of ed reform from their stance on the prerequisite for fixing poverty before we address school policy, as many students from disadvantaged families can do much better academically in certain charter schools, as the paper focuses on in more detail.

    This paper does objectively address some of the hypotheses it is unable to test, such as the impact of peer effects:

    Third, because CMOs operate schools of choice, the families they attract are different in both measurable and unmeasurable ways, which may give rise to peer effects. The selection process of students is driven in part by who learns about and chooses to apply to CMO schools. It is possible that the parents or students who end up enrolling in some CMO schools are more motivated or have other assets. In addition, CMOs can encourage certain families to apply or enroll in their school; even those with random lotteries can target their recruitment efforts and ask students to sign agreements to attend regularly and do their homework. An individual student may benefit from being in the same school and classroom with other students with higher levels of motivation or parental support. If peer effects are contributing to CMO impacts, this does not mean that our impacts are improperly measured. Indeed, our experimental results suggest the impacts are accurate. But it could affect our understanding of the mechanisms behind the impacts: Peer effects may explain why CMO students do better than they would have had they been placed in a school or classroom where there are fewer students like themselves. If that turns out to be true, it would also have important implications for policy: Similar effects might not be achieved, for example, if CMO practices were directly applied to conventional public schools that are not schools of choice. While peer effects can be challenging to estimate, future research should explore their importance.

    Of course, a crucial point here is that this is a possibility and shouldn’t be stated as fact, as is often posited here. There are numerous reasons why motivated/”functional” students/families wouldn’t choose a charter, just as there are reasons for why unmotivated/”not functional” families would choose a charter if possible. It’s also uncertain whether the motivation/”functionality” of students/families differ dramatically between the two populations, enough at least to bolster achievement levels as dramatically as shown. Furthermore, if the argument is that peer effects impact achievement through the disruptions of instructional time, this should stress the importance of effective teaching practices, something the study alludes to addressing next time. This was a contentious issue in the other thread I linked above and was addressed at length by both sides, and so I’ll point to it once more for good measure:

    eduwonk.com/2011/04/still-going-2.html#comments

  18. “all of the charters studied were populated by students whose parents who were motivated enough to apply for the charters and functional enough to complete the application process.”

  19. KIPP in Colorado, which ended with KIPP withdrawing failure.
    Dunbar HS in DC
    Coolidge HS in DC both run by Friends of Dunbar
    How have they succeeded?
    Anyway forget everything and try to visit http://jetsodoor.info which is a Just for Fun site and relax, Hope everybody smile and laughing all round the year

Leave a Reply

Your email address will not be published.