Still Going…

Two stories from last week that will have legs going forward and are worth keeping an eye on:

Cheating in D.C.  schools.  This story seems likely to have some twists and turns and it seems like there is more to come.  Smoke, fire, all that.

The Gary Miron “study” on KIPP from Western Michigan University (pdf). As a rule, when a “study” is shared with the schools being studied after it’s already been circulated to reporters you should be leery. These issues are complicated so common – and understandable – mistakes, for instance conflating or mixing-up capital and operating expenses are easily addressed through a review process – before a study is released. More generally, as Brian Gill (Mathematica) and Robin Lake (who has herself raised questions about KIPP) have pointed out a lot of this report is “apples to watermelons” comparisons and KIPP’s own response is a good overview (pdf). That’s less understandable. But the issue to keep an eye on is whether some easily-fooled reporters should have been more skeptical and asked some obvious questions they apparently did not.  Stay tuned.

66 Responses to “Still Going…”

  1. john thompson Says:

    Explain again who miscategorized revenue. Explain again why an accounting mistake made by KIPP reflects on Miron et al? Miron used a sample of only 1/2 of the huge 99 school metric. Is it Miron’s fault that KIPP’s approach, though worthy, is fundamentally a niche approach and can’t produce bigger samples. If KIPP’s one experiement of tackling all of the challenges of neighborhood schools had not failed, that could have helped researchers.

    Given the very different population served by KIPP, with half as many kids on IEPs, and nearly half as many on ELLs, how can you say that states don’t fund KIPP fairly. Check out Bruce Baker’s similar study on New Jersey charters.

    the entire KIPP system results in creaming, so Mathematic cites miss the point. As Mathematica showed, KIPP loses its lowest performers. Do you expect to find neighborhood schools losing their toughest kids to KIPP.

    This is not a criticism of KIPP. But KIPP doesn’t serve the same kids. Could we learn from KIPP? Yes, if we invested in services for the traumatized kids who don’t make it at KIPP and other charters, neighborhood schools could learn alot from KIPP.

  2. Attorney DC Says:

    I agree with John Thompson: “This is not a criticism of KIPP. But KIPP doesn’t serve the same kids.” It’s a given that the only students who go to KIPP are those whose parents are committed enough to complete the application process. The kids who stay at KIPP are the ones who are able to meet the behavior, attendance, homework and other requirements over and above those required by public schools (for example, longer days and Saturday school).

    If I were a motivated, low-income parent in a poorly performing public school district, I would likely want to send my child to KIPP. I would feel good knowing my child was attending a school that could concentrate on her schooling without the need to deal with the most disruptive and difficult-to-educate students at the local public school.

    However, this does not mean that we should pretend that, as an application-only charter school, KIPP schools don’t ‘cream’ from the general student body.

  3. Harry Says:

    “If I were a motivated, low-income parent in a poorly performing public school district, I would likely want to send my child to KIPP.”

    This is false. Read the mathematica study, or the summary of the mathematica study, and you will see:

    “KIPP middle schools enroll students whose entering
    fourth grade achievement is lower than the district-wide average”
    link: http://kipp.org/files/dmfile/KIPP_statement_WMUreport_03_30_20112.pdf

    I see your larger point, but let’s not muddy the waters with falsehoods.

  4. Attorney DC Says:

    Harry: If what you say is true (that KIPP middle school kids enter with lower scores than the district average), I would counter that it is quite possible that KIPP’s students have similar scores to the students in their actual neighborhood schools (versus the entire district). In any event, my point is not that particularly high-scoring students tend to enroll in KIPP (although it wouldn’t surprise me if intelligent students in low-performing schools tend to post lower-than-expected scores because they attend school with low-performing and/or misbehaving classmates).

    My point is that students whose families are particularly motivated and plugged in to the education of their children are more likely to end up at KIPP than students from other families.

  5. Butler Bulldogs Says:

    Different issue, but the “obvious questions” link goes to Western Michigan’s website listing of funders for research. Looks like the AFT gave Miron $50,000 in October, about the time he was doing the KIPP analysis. Whether or not Miron has it right, shouldn’t that be something he needs to disclose so that the rest of us can evaluate potential bias?

  6. edharris Says:

    One take on the study that Andrew didn’t acknowledge:
    Study Finds High Dropout Rates for Black Males in KIPP Schools

  7. john thompson Says:

    Butler Bulldog,

    Being an AFT member is doesn’t bother me a bit.

    Just kidding!

    but, on the other hand, I saw that $50,000 instantly when I followed the hyperlink to the report, so I figure we can be proud to be so transperant. Also, there is an antidote – carefully reading methodology and the evidence, which by the way I haven’t done thoroughly yet.

  8. Chris Smyr Says:

    I invite John and Attorney DC to actually read further into KIPP’s response as well as the papers in question. It’s not hard to find other sticking points in Miron’s analysis:

    * On pg. 27 under the heading “Selective attrition of low-performing students”, the WMU study claims that the 2010 Mathematica study “noted that KIPP had attrition rates similar to those of the local districts and that the performance of students leaving was lower than average.”

    This is vaguely misleading, as Mathematica summarized their findings with this: “there is little evidence that the transfer pattern of low-performing KIPP students differs from the pattern at other public schools.” (pg. 16)

    * Immediately after that sentence, Miron claims the Mathematica report overlooked something important: “while both KIPP and the local district schools were losing lower-performing students due to attrition, the resulting vacancies were filled in the traditional public schools with low-performing students leaving KIPP or other schools; at the same time KIPP was not replacing its departing students”

    Notwithstanding the fact this does not change the average performance level of students who transfer out of either type of classroom, Miron seems to have not read very far into Mathematica’s design:

    “We address [selection bias through attrition] by ignoring early exits from KIPP schools: Any student who ever enrolled at KIPP remains permanently in the KIPP treatment group, regardless of whether the student remains in a KIPP school or transfers elsewhere. In other words, a student who enrolled at KIPP at fifth grade in the 2002-03 school year but left KIPP after completing sixth grade at the end of the 2003-04 school year is included in the treatment group all four years he or she appears in the data (from 2002-03 through 2005-06 inclusive). This approach is analogous to an “intent to treat” analysis conducted in an experimental context. By including all students observed attending a KIPP school, regardless of whether they leave or stay through eighth grade, we avoid the problem of overstating the effect of KIPP. Instead, this method is likely to produce a conservative estimate of KIPP’s full impact on the students who continue attending KIPP schools. From the perspective of parents, students, or policymakers, this method appropriately accounts for the fact that not all students entering KIPP schools will in fact finish at KIPP schools.” (pg. 22)

    This accounts for all students who transfer out of KIPP, something that local school districts would not necessarily do. While it is possible that students who leave a non-KIPP classroom will be replaced with other students (and on average with similar achievement levels), this replacement will not always occur. Even if it does, the testing data for a classroom may not always include data from students who recently transferred into the class. Thus, Mathematica’s method is more conservative than meets the eye when comparing numbers to non-KIPP schools.

    Furthermore, since KIPP enrolls students whose average achievement levels are lower than the district’s, there should be no immediately inherent positive peer effects in a KIPP classroom. The positive peer effects that are generated should be rightfully accepted as an outcome of the KIPP program, to make high-achievers out of students who were not before. To assert that these students were all “high-achievers in disguise”, or something equivalent, requires more evidence than has been offered.

    The issue of admitting students at different grade levels has also been responded to before:

    “However, it is not the case that KIPP cuts off enrollment after sixth grade. Many KIPP middle schools, including those at KIPP DC, now regularly enroll new students at all grade levels, fifth through eighth. KIPP’s high schools also take students at all levels, from ninth to twelfth grade.

    As more schools are reaching full enrollment and sustainability, this issue of “backfilling” classes is also subsiding.

    The SRI study data Mr. Kahlenberg cites cuts off in 2006-07, when the eighth grade class was at 55% of the starting size of the entering fifth grade class. But that data is now several years old, and those numbers have improved dramatically. As of 2010-11, the KIPP Bay Area eighth grade class is at a full 86% of its starting fifth-grade size. We are working hard to increase that percentage even farther, at KIPP Bay Area and in all other regions.”

    (http://www.kipp.org/news/washington-post-kipp-responds-to-criticism-on-attrition-rates )

    * Miron claims in the executive summary that “students with disabilities and students classified as English language learners are greatly underrepresented” (pg. iii). By his numbers, the percentages for the former are 5.9% vs 12.1%, while the percentages for the latter are 11.5% vs 19.2%. Slightly excessive rhetoric there.

    If we want to continue with this heavy-handed approach, African American students are also greatly underrepresented in local school districts (55% vs 32%), and thus the relative absence of these students results in more homogeneous non-KIPP classrooms. Since local school districts also serve on average fewer low-income students — and 5 times as many white students — than their KIPP counterparts, this all obviously suggests a conspiracy of selectivity perpetuated by local school districts (!!1!)

    ****

    Finally, a question for Attorney DC:

    If your main point is that “students whose families are particularly motivated and plugged in to the education of their children are more likely to end up at KIPP than students from other families”, what was the reason that these KIPP students all did not similarly excel in a non-KIPP classroom?

    From your reasoning in the last thread we discussed this (http://www.eduwonk.com/2011/03/must-reads-3.html#comments ), where you asserted that generally little-to-nothing could ever be attributed to the teacher’s ability to teach, it should seem that students with better family support at home ought to have been the ones that actually succeeded, or at least succeeded more on average than their peers. Yet that’s not what we see happening here. Why?

  9. Art Says:

    When more students apply to a charter school than the school has room for, the school must admit students by lottery (excepting preferences for siblings of already-admitted students, children of staff members, and children exercising transfer rights out of certain Title I schools). In the main, though, the lottery procedure creates an opportunity for “apples to apples” comparisons of the achievement of children from motivated families who were admitted to the charter with the achievement of children from equally motivated families who were not admitted. Angrist and colleagues conducted this type of analysis for the KIPP school in Lynn, MA, a school serving mostly Hispanic students and substantial numbers of LEP students and special ed students. They found statistically significant and educationally significant effects for attending the KIPP school with stronger results in math than reading. Because the analysis compared the achievement of like groups of students, “creaming” or differences in parental motivation can not account for the advantage of the KIPP school.

  10. edharris Says:

    Do any of the KIPP studies breakdown the academic achievement of the students they take through the lottery and the achievement of those who lose the lottery per instruction time at the respective schools?
    That is, to say, does the advantage of KIPP schools come from the instructional time in excess of the standard 6.5 hours of instruction in public schools?

  11. Attorney DC Says:

    Chris: You asked, “Finally, a question for Attorney DC:

    If your main point is that “students whose families are particularly motivated and plugged in to the education of their children are more likely to end up at KIPP than students from other families”, what was the reason that these KIPP students all did not similarly excel in a non-KIPP classroom?”

    My answer is that I believe that students at KIPP have the advantage of being surrounded by other students who will behave in class and do their homework. They have the advantage of being in a peer culture that helps them succeed, rather than in a culture that denigrates school and/or actively disrupts the learning of the other students.

    From my experience teaching in low-income, minority schools, there were always a few students who came to school eager to learn but who were thwarted by their classmates who acted out, disrupted the classroom, and diverted their fellow students’ (and teacher’s) time and energy away from the lesson. I would suspect, based on my experience, that KIPP has the advantage of NOT having these types of students in their classes.

    KIPP students benefit from attending a school with a stricter discipline and behavior policy than is allowed in public schools. For example, in public schools many students with severe behavior problems cannot be removed from the classroom or disciplined for their behavior if such behavior is related to a diagnosed ‘emotional disability’ (ED). Generally, I’ve found (for whatever reason) that principals are very reluctant to impose any real discipline or behavior standards on their students, such that in many of these schools, learning is continuously disrupted by kids who come late to class, mouth off to the teacher, don’t bring their school supplies, don’t do any homework, bang their chairs on the floor, refuse to get out their pencils and paper or participate in any classwork (just to name a few examples).

    This point ties into my previously expressed ideas that we should be focusing on other issues (especially student behavior and discipline) in our efforts to effect school reform, rather than spending increasing amounts of time trying to weed out the bottom 5 or 10% of our teachers.

  12. john thompson Says:

    Chris,

    I have now reread and reread the studies. D.C. attorney answers your questions. The KIPP SYSTEM, by definition, creams. Mathematica’s sample of 22 ESTABLISHED KIPPs showed they lost the lower performing students, as did the horrific neighborhood schools that nobody sees as a solution.

    But, if we looked honestly at Miron’s 60 school study, as well as the other evidence, we should ask with open eyes whether we are doing a disservice to poor students by refusing to provide high-quality alternative services to the most traumitized kids who can’t function in neighborhood schools, KIPP, or other charters. By denying reality, we deny services to our most vulnerable, most suffering kids, and undermine entire systems. Why not look at the evidence with open eyes to help kids rather than attack educators?

  13. Attorney DC Says:

    John: Thanks for backing up my point (above). I’d also like to note that if schools can effectively address student discipline issues, ALL teachers would benefit by being freed from exhausting classroom management and therefore spend more time planning and executing quality lessons. This is a more practical solution than continously trying to identify and target the “weakest” teachers (particularly if the “weakest” teachers are “weak” in large part from having devote their energy to addressing student misbehavior rather than lesson plans).

  14. sm Says:

    john thompson,

    I am a little confused by your arguments here. You write, “The KIPP SYSTEM, by definition, creams. Mathematica’s sample of 22 ESTABLISHED KIPPs showed they lost the lower performing students, as did the horrific neighborhood schools that nobody sees as a solution.”

    So I take it you are defining “creaming” by not only choosing who enters but also by removing those that they do not want. You say that Mathematica’s study shows they lost the lower performing students… yes, it did, but the exact quote is, “Cumulative rates of attrition vary widely in different KIPP schools, but we did not find systematically higher (or lower) levels of attrition among these KIPP middle schools as compared with other schools within their districts. In approximately one-third of cases, attrition was higher at KIPP than at other district schools by a statistically significant margin; in another third, attrition was lower at KIPP than at other district schools by a statistically significant margin” (ES, xiv).

    Any school is going to lose students, “as did the horrific neighborhood schools” (your quote). The studied KIPP schools displayed no systematic differences: some lost more, some lost less. What do you want to prove otherwise? That every single child who enters does not leave?

    And if you are defining “creaming” in the inverse, the entire section of the study you (supposedly) read on selection (p.17) deals with that, and it ends with: “Across the descriptive analyses presented in this chapter, there is little evidence that KIPP middle schools are systematically enrolling more advantaged or higher achieving students from their districts.”

    Note that I am not necessarily arguing with your points– I am arguing with what I think is your misuse of the MPR study.

  15. Art Says:

    EdH … I’m not aware of studies that have adjusted estimates of the KIPP effect for differences between KIPP schools and other schools in “time on task”, though the Angrist study reports that the Lynn KIPP school offers 1900 hours of instruction and other Lynn public schools offer 1250. That 50 percent additional time bought KIPP students 0.35 standard deviations of growth a year in math and .12 sd a year in reading, with larger gains for the lower achieving students. The study team says that “Measured against
    Lynn’s Hispanic-White score gaps of about 0:5 in math and 0:6 in ELA, both the math and ELA eff ects [for KIPP] are substantial.”

    John … The Mathematica study controlled for attrition in the KIPP schools by including all students who ever enrolled at KIPP in the KIPP treatment group even if they left the KIPP schools early. This means that even if it is true that low achieving students leave KIPP schools, Mathematica’s estimates of the effect of KIPP would not be biased upwards because of that. The Mathematica team says that if anything their procedure biases estimates of the KIPP effect downwards.

  16. john thompson Says:

    sm, and also Art,

    I was in a hurry but my key word is “system.” It is no criticism to say that the KIPP system, as a whole, produces creaming. Worse, creaming, by definition, occurs at KIPP and the other charters who sincerely would like to avoid creaming.

    The problem is not that KIPP creams in order to look good. The problem is that the challenge of educating critical masses of the worse victims of generational poverty is so extreme. It is that problem that creates the dynamic that teachers like DC Attorney, and I see.

    I used to swim, play b-ball, camp, garden, and teach many of the kids in the 700 student middle school that was replaced by the 120 something KIPP. It explanded slowly and rationally and it came to serve a few more kids over a decade, which is another luxury that neighborhood schools don’t have. You can’t say that those kids, were the same kids as at the school which was 35% special ed, and was written up Harpers Index for its riot. I take that back. Arne Duncan did.

    Mathmatica said that these ESTABLISHED schools did not do worse that the neighboring schools. But inner city neighborhood middle schools are as dysfunctional as any institution in America. Nobody seeks to replicate them. When the lowest performing KIPP kids don’t make it, they go back to the regular schools. When the lowest performing neighborhood middle school kids don’t make it, they go to the streets.

    Before the kids who are most failed by neighborhood schools leave, they help disrupt learning for everyone. But that’s not blaming kids for acting out their trauma. That’s the irrational system.

    Art, if we had a rational system, we’d have a data base of time on task in all types of schools. That’s the type of data-INFORMED decision-making that kids need. If we adults had to face the facts about the non-stop anarchy that kids have to endure, we might attack the real problem and not attack educators trying to survive in this nonsystem.

    Think of how Paul Tough and Nadine Burke explained the problem of the most extremely traumatized kids. And I mean extremely traumatized. Non-inner city teachers might have been taken aback to read that ten of those kids might end up in an inner city class of 30. Before NCLB and the proliferation of choice, I would have been taken aback too. Then, it was rare to have more than one or two kids with fullblown mental illness in a regular class. Now, in the toughest schools, those extreme concentration of kids who KIPP could not be expected to teach are common.

  17. john thompson Says:

    I should also add that, in my experience, when the lowest performing KIPP students go back to regular school, they end up in HONORS classes. That reflects well on KIPP, but it also shows how different the KIPP population is. How many of their 6% of students on IEPs are Seriously Emotionally Disturbed? Its been years since I had a class, regular or honors, that did not have more than 6% of its students diagnosed with extreme conduct disorders. I had more sophomores on IEPs every morning, in addition to my class load, than every sophomore served by every charter in the metro put together. And in my nontested subject, it was worse every afternoon. But that was high school. It was the middle school that really had it rough.

    I don’t see why we’re even arguing about that. The researchers who used math to conduct their study were not teachers but they could have read Balfanz. As he explained, when a school has tens of kids who are two or three years behind, we produce a lot of success stories. In schools where hundreds of kids are four and five years behind, everyone is overwhelmed. That’s just common sense.

  18. sm Says:

    john thompson,

    You write, “It is no criticism to say that the KIPP system, as a whole, produces creaming. Worse, creaming, by definition, occurs at KIPP and the other charters who sincerely would like to avoid creaming.” I disagree. I think it is a criticism, and I would like you to explain the definition of creaming and your proof for your statement, and not at every charter– I don’t care about this generalized argument you are making– but at KIPP.

    You write, “Mathmatica said that these ESTABLISHED schools did not do worse that the neighboring schools.” Actually, they did better; way to frame it in the negative. That’s the same rhetorically as UConn did not do worse than Butler last night.

    I don’t care about your argument with the “system” and all of that other stuff. I’m a teacher too and deal with all sorts of stuff. Above I said I am arguing with your misuse of the MPR study; I maintain that in your rebuttal to me, you merely misused it some more.

  19. Harry Says:

    John, I agree that its unrealistic to expect KIPP to serve 100% of a district or believe we can turn every district school in to a KIPP look-a-like. But I don’t understand why you continue to denigrate KIPP results. All of the critics you raise have been addressed. KIPP does amazing things with the kids they serve. The research is pretty settled on that.

    You raise some good points about the problems a school like KIPP doesn’t and wasn’t designed to address. I actually agree with them. But they say nothing about the efficacy of the KIPP model. They are irrelevant to this conversation.

  20. Harry Says:

    actually forget my comment. I’d much rather hear you respond to sm

  21. john thompson Says:

    Harry,

    I can address your point quickly. When I get back, I’ll respond further to sm and others.

    To my knowledge, the achievements of KIPP are amazing. I don’t dweel on them because I concentrate on what I know from firsthand experience, and serious reading.

    From what I know, KIPP deserves all of the praise it gets. I know firshand that theitr success in OKC is amazing, and I’d think that it would be assumed that the accounts of the problems cut both ways.

    I’d like a conversation about the way KIPP uses music and activity and how it is similar to the way I use it, and what we could learn from each other. Got a few hours for that?

    I’d like a conversation about what KIPP teaches us on how to best provide humane and high-quality schooling for the 5 to 10% of kids who aren’t emotionally capable of dealing with the chaos of urban schools, and how to deal with that chaos.

    I’d also like to efficinetly and LEGALLY get rid of the 5-10% of teachers who don’t belong in any school.

    But gotta go now.

  22. Attorney DC Says:

    It seems that Harry, sm, and other posters are discussing two distinct issues: (1) Is KIPP effective for the students it serves; and (2) Does KIPP cream and/or are KIPP’s students representative of those students in the neighboring public schools?

    I think the verdict’s still out on the first question due to the difficulty of controlling for all the variables; However, the available evidence appears to show that students who enroll in KIPP are likely to benefit from the experience.

    However, there is no evidence that KIPP students are the equivalent of the general population. The difference is that they are more motivated (and/or have parents who are more motivated) AND that they attend school with other motivated students AND that the school tends not to enroll students with the most difficult problems. This is no different, in my mind, than a gifted-and-talented program or AP classes: Taking the most motivated/talented students and letting them attend class with other similar students. Separating students in this fashion enables the teachers to teach at a higher level/faster pace and avoids much of the discipline and behavior problems present in other classes.

    Based on the above, my opinion is that the KIPP model cannot be considered a cure-all for general low-performing public schools UNLESS we are prepared to give the public schools the same tools KIPP has at its disposal: Requiring parents to go through an applicaiton process; Requiring parents and students to agree to follow strict behavior, homework, and attendance policies; Encouraging students who have difficulty with the rules and requirements to leave…

    However, in the public school setting, this begs the question — where would the students who failed to live up to these more stringent requirements GO? Seriously: This is a difficult question which has no easy, politically acceptable solution.

  23. tom Says:

    Attorney DC, you’ve got to be kidding me. Have you been reading the comments above? All of your critiques have been addressed. Furthermore, the level of proof you seem to be looking for is impossible to attain. I can only assume you have some bizarre vendetta against this organization.

    If you want to talk about larger questions fine. But stop trying to denigrate the results of this organization. You’re not contributing anything.

    How about this new rule for criticizing research: If can’t describe an alternative methodology or research design that would address your critique, your criticism is invalid.

  24. Chris Smyr Says:

    Attorney DC:

    Your response confirms you think it is because all of these kids had to deal with “disruptive and difficult-to-educate peers” before KIPP and it thus reduced their potential. This is what I wanted to see you write, because you here imply that

    * School factors are more important than out-of-school factors (rowdy kids can offset the effects of a supportive family)

    and

    * Were a teacher able to effectively deal with outbursts of the other kids, he/she could help generate achievement gains

    This is different from what you were arguing before in the other thread.

    Considering there are probably more than a few families that similarly support their kids (surprise surprise), it seems likely by your logic that good teachers should often be able to generate gains with their kids. This assumption is unnecessary, however: we can measure achievement growth of students and expect certain gains even when they have crappy home lives.

    You say that KIPP students “have the advantage of being in a peer culture that helps them succeed”. Guess who established that peer/classroom/school culture? The educators.

    Finally, your analogy of KIPP to AP classes is poor because students entering into KIPP do not enter with past evidence of high achievement. You’re pushing the assertion that these kids are all essentially high-achievers in disguise, yet you aren’t providing any evidence to support this assertion.

    *****

    John:

    There’s already good responses here to how you have misread the Mathematica study, so there’s not much else for me to say.

    *****

    In short: it will be useful to have further experimental studies address exactly how KIPP students do in relation to those students that lost the lottery (Art talked about one such study above). Mathematica is aiming to do just that.

    In the meantime, it’s important to make sure the nonexperimental data is adequately analyzed, and it is evident which of the two, Mathematica and WMU, actually did a better job at this. Also, no one here so far has really addressed KIPP’s response or even my own above.

  25. sm Says:

    And to buttress tom’s points, I will once again reference the actual MPR study which it doesn’t seem any of the people who are criticizing the organization are willing to do:

    Attorney DC, you write: “However, there is no evidence that KIPP students are the equivalent of the general population.”

    Well, MPR wrote: “To examine the characteristics of the students who enter KIPP middle schools, we compared the fourth grade characteristics of future KIPP students to those of non-KIPP students in the same districts and in the same elementary feeder schools (in other words, elementary schools attended by students who later enrolled in KIPP). We found no evidence that KIPP middle schools are systematically enrolling more advantaged students from their districts…The prior achievement of students entering KIPP schools varies, but KIPP schools most often enroll students whose average fourth-grade achievement is lower than the districtwide average.”

    So unless you are in a position that is of a stronger research pedigree than MPR, I will defer to them.

  26. john thompson Says:

    I see my post on this is now up at:

    http://scholasticadministrator.typepad.com/thisweekineducation/2011/04/thompson-why-deny-that-kipp-schools-have-an-extra-edge.html

    How’s this. Mathematica’s study showed that KIPP attribution wasn’t as bad as that of awful middle schools, but its internal study and Miron indicate that it may be worse? That doesn’t diminish KIPP’s successes. It calls into question whether it can be scaled up.

    mr, I used to be an academic and I don’t challenge the nuts and bolts of the Mathematica study. But their statistical evidence just approximates reality, and it can’t carry the freight of some of their conclusions. But that would get us into a deeper discussion of how you define “creaming.”

    Since I submitted my post, the Baltimore story revealed that five districts, as I recall, have diagnostic assessments as a part of the intake that result in some 6th graders being assigned to 5th grade. This may or may not be a good policy, but it is a part of the gestalt that in its totality is creaming.

    Perhaps we need another term. We all need safety nets and buffers. That’s why tenure is absolutely necessary for systems. That’s why I don’t complain if KIPP has a buffer/ability to cream enough to protect its system.

    Neither am I threatened by its approach to testing, as long as other people don’t cite KIPP as a rationale to impose teach to the test on me. Neither am I threatened by most of its 99 schools being resistant to unions. If by some miracle they had 999 schools, I admit that I might be threatened by KIPP. But if there are KIPP teachers out there that think they face the same challenges as neighborhood school teachers, that’s no skin off my nose.

    The problem is “reformers” who misuse KIPP, pretending that classroom-driven approaches can solve systemic challenges. The bigger problem is thinking that KIPP’s success should be used as a club against teachers in neighborhood schools. The biggest problem is “reformers” is using KIPP as an excuse to flush our most troubled 5 to 10% of students down the toilet. I don’t mind the term dance of the lemons being used for the 5 to 10% of teachers who shouldn’t be in schools. I don’t want anyone to use derogatory terms, however, for the 5 to 10% of kids who are being denied services. They didn’t ask to be raped, to be born with a predisposition toward mental illness, to have the data-DRIVEN war on drugs lock up huge numbers of nonviolent parents, to be born with alcohol and crack in their systems, to be left as an infant in a bayou or be taken on a kidnaping spree, to see a parent murdered or commit suicide, or witness their murder-suicide …

    Check out the “hallwalkers” who cut class for years in neighborhood schools, and listen to their stories. Ask why the same kids get in trouble almost every class, almost every day, from say 6th grade until they finally drop out in high school. My sense is that KIPP is able to “tithe” and do wonders for a reasonable percentage of those traumatized kids. I sure don’t blame KIPP for so many kids being dumped in my school, for instance, where we have $7,600 per student, even though a huge percentage of them should be in a school like SEED.

    A little accountability is fine, but what we really need is the opposite of accountability. We need buffers to shield people from the blame game. Take the dilemma of a central office administrator who has just heard that another teacher has been assaulted, needing medical treatment, and the CYA questions he or she must ask. How many other reported and unreported assaults on teachers have happened at that school? How many other teachers has he assaulted, and has the district just responded by shuttling him to another school. Was the teacher grossly incompetent? Was the principal? If the kid is on an IEP, was the special ed teacher incompetent? Is his BIP up-to-date? Is it possible we have an alternative slot available? How many kids has he assaulted? Do I dare address this problem? If I ask basic questions, will I open a can of worms that I don’t dare address?

    Real world, the answer is usually going to be that you sweep the problem under the rug, intimidate or guilt-trip the teacher into silence, and perpetuate the law of survival. That happens multiple times per year with adults. With assaults on students it happens weekly or daily.

    Lastly, I don’t mind a detailed give and take on the nuances of methodology. For instance, you are ignoring DC Attorney’s huge point on peer effects, and not even the best scholars can address it. So, think of the opportunity costs of this fight. As I said in my TWIE post, people of good will should have plenty to agree upon. Let’s concentrate on how we can stop flushing or most vulnerable kids down the toilet.

  27. sm Says:

    This is insufferable.

    I already asked you, above, “I would like you to explain the definition of creaming and your proof for your statement, and not at every charter– I don’t care about this generalized argument you are making– but at KIPP.”

    You wrote above, “But that would get us into a deeper discussion of how you define “creaming.””

    I already asked you to define it. You see, I am not making any grand statements, really. I have continually asked you to define this, and you aren’t. Here, I will even show you how to do it: I define “creaming” as purposefully accepting only the top student applicants and also forcing out those who cannot meet the expectations. How do you do it? Your turn!

    Next, it isn’t the Baltimore study that shows the grade assignment issues– it’s MPR’s own study. For the love of everything, just read the darn thing. (It may also be in this Baltimore thing you allude to, but MPR does as well).

    To jump around a bit, you also write: “It calls into question whether it can be scaled up.” So, going from 0 to 99 schools in around 15 years isn’t good enough for you? Fair enough. Because they aren’t the largest and most successful charter school operator in the country or anything.

    Also, please explain how their evidence “can’t carry the freight of some of their conclusions.” Exactly explain that– not just some general statement that goes something like, “Well, p values are arbitrary and statistics aren’t reality as some things change blah blah blah.” I want to know which conclusions they reach are not supported by their analysis.

    And I don’t really care about anything else you write. Once again, I am simply strongly objecting to your misuse of research even though you apparently are/ were an academic.

    Because you tend to avoid my specific questions, I will summarize them:
    1) Define creaming. Explain how KIPP specifically creams and the proof you have for that explanation.
    2) Explicitly explain which conclusions of the MPR study are not supported by their evidence. Please cite them exactly.

  28. john thompson Says:

    Chris, I mean, sm,

    What is so insufferable about trying to change the subject from “Well, p values are arbitrary and statistics aren’t reality as some things change blah blah blah” to the suffering of kids?

    Creaming is the intentional or unintentional reduction of the hard-to-educate populations so that the challenge of raising student performance becomes recognizably less difficult. A key is the point where the challenge becomes recognizably less difficult because that gives hope to the system and lessen the motivation to give in to the culture of compliance.

    The diagnostic test in Baltimore, as well as the threat of being returned to 5th grade is the archetypical example of unintended creaming. The difference in families’ motivation between those who would go along with that system would not show up in free and reduced lunch numbers or student achievment data.

    What is it you want me to explain about the studies, and the D.C. Attorney’s answers that you haven’t addressed? gestalt? peer pressure, Cambell’s Law? the difference between ESTABLISHED schools and KIPP schools that are still struggling?

    Not suprisingly, Mathematica did an excellent job of addressing the topics of their research, although they seem to have a narrow definition of creaming. As was explained, I have no problem with their research on how worthy KIPP is. The problem comes from issues regarding scaling it up. As I explained, if KIPP has a lower attrition rate than the neighboring middle schools – schools that often are the most dysfunctional institutions in America – that is persuasive in defending KIPP as it exists from its critics (of which I do not count myself). But if you are interested in scaling up, the better comparison is with the system’s attirtion rate.

    I think you misread or took my words “the system” of of context, but you accurately quoted Mathematica. “Across the descriptive analyses presented in this chapter, there is little evidence that KIPP middle schools are systematically enrolling more advantaged or higher achieving students from their districts.”

    As James Heckman documents, and most teachers know, its the socio-emotional factors such a stick-to-it-ness that are the key to academic success. Achievement scores are poor surrogates for measuring those traits. Similarly, the word advantaged connotes economic or social advantage, not temperment advantages. And if the 5.9% of KIPP students are broken down by serious emotional distrubance and conduct disorders, I missed that. But anyway, measuring how much of an advantage that gives to KIPP is a major project. If you’ve got research on that, I’d be happy to read it.

    The mere existance of lotteries, as has been documented, is a major problem for researchers trying to disprove KIPP has advantages. It is the most successful programs that have lotteries. Even when it gets to the point where all 99 KIPPs have lotteries, that remains a huge issue for scaling up. Also, as I recall, there is research about the extremely high percentage of unsuccessful applicants in one selective school that get accepted by other schools. That dynamic over-estimates the size of the market of KIPP schools and reinforces arguments that KIPP families have more stick-to-it-ness.

    There is a large literature on peer effects. The latest from Sass et al at Calder is especially important. Follow Steve Sawchuk’s links.

    Peer effects, as Caroline Granham explains, are even more important in a system where they keep the same kids throughout and don’t replace those that fall out. I don’t begrudge KIPP that advantage. The Special Forces, Marines, are other elite units take full advantage of that.

    Since you “don’t care about anything else I write,” I can close this. Previously, Chris was about the only commenter who I tried to actively ignore.

    And that gets me back to my point. I’m not opposed to KIPP. I’m opposed to using KIPP to beat up on neighborhood school educators. It sounds like you are less interested in discussing solutions than defending KIPP supporters who want to attack teachers in the toughest schools.

  29. sm Says:

    Thank you for defining “creaming” in your opinion. If we can’t agree on definitions of terms, we can’t argue over anything. You still maintain they cream; I do not.

    Although above you wrote, “But their [MPR's] statistical evidence just approximates reality, and it can’t carry the freight of some of their conclusions,” you now write, “Not suprisingly, Mathematica did an excellent job of addressing the topics of their research.” I asked you to cite which conclusions were not substantiated. You didn’t.

    That’s all I care about– not anything more. Don’t get all petty with me, “miswriting” my name on the top. That’s very mature. Also, don’t play this card about the “suffering of kids” with me.

    Lastly, Chris writes his things; I am not him. If he said something I didn’t agree with, I would say so. You tend to generalize and, in my opinion, misrepresent research. So I asked you about it. “Actively ignore” me all you want.

  30. Chris Smyr Says:

    Just a little to add as I await Attorney’s reply:

    John usually does all of the quirky things sm has referenced above (and more), and the longer his replies get, the more incoherent they become. There’s a lot that can be torn apart up there, but the response he receives by me or otherwise usually generates active ignorance. Or was that he actively ignores me? In any case, let me know if you want a longer reply, John.

  31. john thompson Says:

    sm,

    Thanks for giving me a chance to explain a contradiction in my comments. I meant to write that the author of the MPR arguments on TODAY’S debate can’t carry the freight.

    I have no qualification to criticize his Mathematic report, other than his weird understanding of the word “cream,” I didn’t challenge that study. I was disagreeing with his statements here:

    http://www.edweek.org/ew/articles/2011/03/31/27kipp_ep.h30.html?tkn=UVLFkSFQ4fzF9DdalcbqVZb89bvN9Bnmw4Zv&intc=bs&sms_ss=blogger&at_xt=4d949cf14cafdda5%2C0

    I personally don’t see that many differences between Miron and the Mathematica studies, except in their interpretation.

    And that’s my point. I have no complaints with KIPP. I have complaints with the way KIPP is used in this politicized environment.

  32. Art Says:

    John … The CALDER study by Tim Sass and colleagues found that even though overall teacher quality (as assessed by value-added statistical rituals) seems to be similar in high-poverty and low-poverty schools, there are more really bad teachers in high-poverty schools. Sawchuck also cites a recent study suggesting that imbalances in teacher quality are more pronounced at middle school. I am not aware of any study that tracked whether kids applying to KIPP schools experienced bad teachers in their traditional public school. But my point is that the peer effect hypothesis might offer less comfort than you think.

  33. Attorney DC Says:

    Art: I’d like to reply to your comment that there may be more ‘bad teachers’ in low-income schools. That seems to be a reasonable theory, based in part on the idea that teachers at low-income schools are often less experienced (coupled with more turnover of staff).

    However, I’d like to point out that perhaps some teachers at low-income schools APPEAR less effective than teachers at higher-income schools because teaching at low-income schools is much more difficult. Based on my experience working in schools, I think that many of the teachers who run efficient, productive classes at cheerful surburban schools would be much less efficient and productive if they were placed, instead, in a high-poverty school, with their students having all the behavior issues typical of youth from broken families and gang-controlled neighborhoods.

    Point: The teacher’s effectiveness is determined not simply by intrinsic characteristics of the teacher, but also by her surroundings.

  34. john thompson Says:

    Art,

    Reread Sass. It shows that the lower quality teachers are less productive in high-poverty schools, and then asks why. His likely explanations are exactly what teachers have beenn saying. It keeps getting back to peer effects, to conditions that burn out teachers and students, and the overall culture of schools.

    Isn’t it intersting the D.C. Attorney, a former teacher, already made the same point.

  35. Chris Smyr Says:

    Attorney DC:

    If you extrapolate from your basic point, you implicitly agree that teachers should be able to have some impact on their students’ achievement gains. Going off my last post addressed to you, it seems so long as a teacher can create a positive learning environment in their classroom there ought to be some measurable gains for his/her students. It will certainly be obvious with classroom observations, at least.

    Also, your hypothesis for why some teachers are labeled as ineffective is not testable or falsifiable; it’s just another accuracy argument. Pulling effective teachers from “cheerful surburban [sic] schools” and simply putting them into high-poverty schools is not an adequate test of this, as there are many variables confounded with the switch. It should further be stated that, in your words, to “appear less effective” means that they actually *were* less effective as measured by classroom observations and testing data that accounts for past achievement.

    Your rhetoric also continues to ignore that with the help of various sources of data, a measure of effectiveness with adequate precision can be made for teachers even in high-poverty schools. Even considering peer effects should not be off-putting since these effects will have also likely impacted past achievement.

    In short, making small gains with a large group of students who have been tough to teach in the past would rightfully be seen as a success. Establishing a positive classroom culture would also be seen as a success.

    *****

    John:

    If you want to continue implying that peer effects explain the gains made in KIPP, you need to provide evidence that KIPP students either:

    1) Are high-achievers prior to their attending KIPP, or

    2) Are better students prior to their attending KIPP.

    Please provide this evidence.

  36. Art Says:

    John … Not really, because despite “peer effects, burnout, and the overall culture” of high-poverty schools, highly effective teachers in high-poverty schools are as effective as highly effective teachers in low-poverty schools. You can see how that plays out in the research team’s suggestions that we should find ways to keep highly effective teachers in high-poverty schools and recruit effective and experienced teachers to them, as well as develop teachers’ skills over time.

    If the least skilled docs are working in the intensive care unit, training may well be called for but I doubt that the best solution is to admit healthier patients to the ICU.

  37. Terri Says:

    Art and Chris… I agree with your position on keeping highly effective teachers in high-poverty schools. My last teaching position was with an inner city, high-poverty, charter school. The gains that the students made were slow, but steady. Of course, the steady progress did not meet the standards for AYP in Ohio. During the fourth year of the school, the principal and teachers were confident and whole heatedly believed that our students and our school would meet the requirements for AYP. If we didn’t, our school would be phased out over the year. Our curriculum director, hired during the school year, reassured the school board that our students were not being taught a curriculum that would lead the students to success on the high stakes testing. The chart school made the decision and notified all teachers (plus the school cook, aides, etc.) on the last day of school (2010), that we were no longer employed (knows as a clean-sweep), because we were not effective teachers. The principal was fired during the last six weeks of that school year as they blamed him for our “short comings” to make AYP. When students take the test in the spring, we have to wait until the fall to find the results. The scores came in and all we could say was, “Wow!” We were happy for our students and angry with the administration. If we could have stayed, where would our students been now. Our schools ranking went from academic emergency to continuous improvement. We, the teachers, could see the baby steps turning into leaps. The students were gaining confidence as learners and building relationships with teachers who truly cared about their futures. The student population at my previous school was made up of kids on probation, young parents, gang members, homeless students, and the list goes on and on. Inner city schools in high-poverty areas struggle just as much as traditional high risk public schools, if not more. A school that is young and in its first years of creating a safe and effective learning environment for students who had many struggles with traditional public schools is expected to make the same progress as other schools which have been trying to reach AYP for several years. When the students came through the front doors of the school, they were there for many reasons, such as Judges and parents forcing them to attend the school and some students looking for a school that would help them reach their goals of earning a high school diploma and lead them to higher education. The teachers at this school over the previous four years had volunteered more than just sweat equity, they were willing to take substantial pay cuts to keep the doors opened, as did the principal who was fired. Teachers brought food and clothing to the students and helped them get social security numbers, drivers licenses, birth certificates and housing, even if it was the at the YMCA. The students were grateful and worked hard to improve their skills. When all of the teachers were let go, they were not given the chance to say goodbye and explain. We were making more than academic progress, we were improving the living conditions and chances for opportunities for our students. The majority of the teachers at this charter school were effective teachers. The proof was in the scores of the tenth grader’s OGT scores after being in the district for two or three years and the quality of the lives of the students. Just because a teacher is in a high poverty, urban school and the progress being made is slow, does not mean they are an ineffective teacher.

    Now, I teach in a high poverty rural school district. The district is so far in debt, that massive cuts (even after the community passed a levy) leading to lay-offs of teachers, aides, guidance and non-academic courses. When the I was hired at the beginning of the 2010-11 school year, I was only hired because I had less than five years teaching experience; I was cheap! The school was/is so focused on the money issues that the students of the schools academic needs were being compromised to save money, but yet the state demands for AYP continue and yet they continue to cut the budgets. The funding for professional development was limited and only senior teachers were getting to attend the classes, but yet the new and younger teachers are held to the same caliber. The students in the rural areas face just as many challenges as their peers in urban schools; the challenges are just labeled differently. Students come to school hungry, abused, starved for attention, unfocused, and with learning disabilities. They are each different, academically and physically, but yet we demand that each of them can slam-dunk a basket ball, on the same day in early spring. Several schools have effective teachers, but not every classroom is full of eager, engaged and alert learners. Not every classroom is going to have all students being successful on a set date as determined by the state or federal government.

  38. Attorney DC Says:

    These are my positions on the issues in this post, which I think are pretty much unassailable:

    My point is that schools like KIPP give an advantage to their students by enrolling only a group of students who are more motivated and geared toward academic success than the average public school students in their neighborhood.

    My other point is that attempting to evaluate and compare the ‘quality’ of teachers who teach in very different environments may lead to false judgments about the teacher’s effectiveness, because, in my opinion, it is difficult (if not impossible) to separate a teacher’s innate skills and abilities from the teacher’s current environment.

  39. Chris Smyr Says:

    Attorney DC:

    Well, I believe those two points, as well as the other stuff you have ignored thus far in this thread and the other thread, IS assailable. Your decision to ignore some of it doesn’t make the valid criticism go away.

    To reiterate your reasoning: students enter KIPP performing academically worse than the district average, even though they have “particularly motivated” and supportive families, all because other students “disrupted the classroom”, and yet this still doesn’t imply that gains should be possible for students with good teachers. Did I rephrase your logic correctly?

    You have this sort of two-faced approach to reform where you continually claim the moral high ground in how we must nurture and support “cultural attitudes toward education” (and proceed to dump empty platitudes about how to do so), yet you throw nearly all low-income families under the bus when you assume that they don’t care about their kids, and that when their kids misbehave it indicates the forgone conclusion that they weren’t meant to succeed in a classroom and that they should be exited and made to pick up trash (all to promote said cultural attitudes, of course).

    KIPP is an example of how some families are motivated/able/willing to enroll their low-achieving students in a charter school (suggesting out-of-school factors can’t always be to blame). While you may have been a teacher, you missed the forest for the trees if you still cannot imagine how many other families find their children’s education just as important but are unable/unwilling to put their kids in a different school. A lot of these families are in old-fashioned public schools, too. Were it up to you, these families will continue to draw straws hoping they don’t end up with any ineffective teachers.

    Your other opinion about how teachers who are labeled as ineffective may not really be ineffective (even though they actually *were* ineffective in helping their kids learn) is duly noted. It ignores a laundry list of reasons for why it is incorrect, however, and those reasons include facts about how these evaluations are performed and what accuracy entails.

    Put simply: “Disruptive kids” and “poverty” should not be the go-to excuses every time a teacher fails to make achievement growth targets. As KIPP’s success suggests, a focus on in-school factors like teacher effectiveness and peer/classroom/school culture should be considered as we move forward with reform, to do what we can to help other low-achievers rise to the top.

  40. john thompson Says:

    Sass et al did not establish that the least effective teachers who worked in high-poverty schools were less qualified than their counterparts in less challengng schools. They concluded that those teachers, working in those environment brought down student performance more than the counterparts in lower poverty schools.

    Also, combine their study with the IES study cited by Sawchuk, and you see that these problems are far worse in middle schools where anarchy is preordained for neighborhood schools.

    The better analogyis that it makes no sense to tell patients who need heroic measures in the ER to go home, take two aspirin, and walk it off. That’s what we’re doing to poor kids because we’re more interested in punishing teachers than helping kids. KIPP, for instance, succeeds with some of those traumatized kids, but it doesn’t have to keep them and wreck the system for everyone. Now those kids are dumped on neighborhood schools, but they need adequate treatment.

  41. Attorney DC Says:

    Chris: Don’t have time to address all your points, but I’ll address one of your more important one: You wrote, “KIPP is an example of how some families are motivated/able/willing to enroll their low-achieving students in a charter school (suggesting out-of-school factors can’t always be to blame).” I agree with this statement, which is similar to what I wrote in my posts.

    However, you also wrote: “…many other families find their children’s education just as important but are unable/unwilling to put their kids in a different school.” This is where I disagree with you, in part, for the following reasons: Many studies and surveys have demonstrated that the vast majority of parents and families IN THEORY would like their children to have a good education. However, this does not mean that they all provide equal supports for their children.

    Many parents do not help their kids with their homework, do not come in for parent-teacher conferences, do not have consistent punishments for misbehavior, do not set a strict curfew for school nights, do not limit their child’s TV, and do not give their child a quiet space to do homework at night. Studies have also shown that parents of different cultures/races have different academic expectations for their children: For instance, a survey reported by both Abigail and Stephen Thernstrom and John McWhorter showed that Asian parents were dissatisfied if their children brought home less than an A- while white parents were, on average, satisified with a B and African-American parents were OK with a C.

    My point is that, while almost all parents want academic success for their children, the definition of that “success” differs across families. The family’s ability to assist their children in obtaining scholastic success also differs dramatically. Therefore, when KIPP enrolls only those children whose parents are educated enough and motivated enough to search out charter school options (who presumably also communicate the importance of this option to their children), KIPP is enrolling a subset of families who are more able to provide the necessary support and attitudes to their children to achieve scholastice success.

  42. Attorney DC Says:

    One more point: Chris wrote, “A lot of these families are in old-fashioned public schools, too. Were it up to you, these families will continue to draw straws hoping they don’t end up with any ineffective teachers.”

    On the contrary, that’s not my attitude at all. FIRST, as I’ve said before, I don’t think the main problem in low-income schools is the teachers, I think it’s the school culture and the students/families who attend the schools. SECOND, I strongly believe that public schools should be reformed in the area of student discipline/personal accountability: Public should enforce a consistent discipline code (similar to KIPP); students who violate the code should not be allowed to simply stay in class each day, disrupting the attempts of the other students to learn.

    When I worked in a military-style boarding school, the school determined EVERY DAY which students had done their homework and which students had misbehaved in class, and consequences were doled out appropriately. Currently, in public schools, students simply do no homework, skip class, and act disruptively, day after day, with little or no consequences for their behavior. This is unacceptable. Schools should support their students/families, but should be able to hold students and families responsible for their own behavior.

  43. sm Says:

    And I return to this fray after a brief hiatus. You all keep going at it, don’t you?

    Anyway, I have a question here: Attorney DC, you write that “Therefore, when KIPP enrolls only those children whose parents are educated enough and motivated enough to search out charter school options (who presumably also communicate the importance of this option to their children), KIPP is enrolling a subset of families who are more able to provide the necessary support and attitudes to their children to achieve scholastice success.”

    So before I go on, even though a lot of research, which has been cited above, shows KIPP students are systematically usually worse off than other neighborhood students at enrollment, you argue that that is a moot point and not what we should be looking at, that is, we should be looking at the parental characteristics of the students who enter versus those who don’t? And that that would override everything else?

    If I am right so far, okay, sure, you can ARGUE that. However, you definitely cannot flat-out STATE that: we have no clue. You phrase this as if you know this for a fact: “Therefore, when KIPP enrolls only those children…” and “KIPP is enrolling a subset of families…” Where are you getting the support for these statements?

    You write this as if you know this– have you conducted this research comparing the parental structures, perhaps a randomized controlled trial looking at those who got in versus those who did not and the parental supports given to each child over a period of time? I would love to see it if you have. Or, have you worked in admissions for KIPP? Have you worked at KIPP and can first-hand prove this?

    You see, I completely disagree with you because of a brief first-hand experience when I helped enroll students at a KIPP, and I will be absolutely astonished if you tell me that we systematically knocked on doors with a strong predictive ability of the parents inside of them, and then the lottery was entirely scammed and rigged to only pick those parents who looked really good and helpful and supportive. To think, I didn’t even know I was somehow parentally-support-psychic and that lottery with bouncing balls was ingeniously rigged.

    However, I actually don’t think you either did this research or have first-hand experience in admissions of KIPP.

  44. Attorney DC Says:

    sm: It’s true that I haven’t done wide-range research on KIPP (or any other education topic). I’m not a researcher. I’m a former teacher and current attorney. However, I have taught, substituted, and volunteered in many different types of schools (including public and private schools), and have worked with a wide range of students, from low-income to wealthy, special education, general education, honors, English-Second-Language, and students of practically every race/cultural group.

    In my experience, I have found that the behavior and motivation of the individual student (which often is closely linked to the behaviors and attitudes of the student’s parents/guardians) is CRITICAL to the academic success of the student. To the extent schools can help conrol/influence student behavior (for example, by having strict attendance or behavior requirements, like KIPP) or that schools can select their student body based on demonstrated performance or behavior, the school can have an effect on educational achievement.

    Of course, students influence each other as well. For instance, peers set a school culture (e.g., is it ‘cool’ to get good grades), denigrate (or applaud) other students who work hard, disrupt (or not disrupt) the class with inappropriate behaviors, etc… I’m not arguing that KIPP’s longer school day or other ideas have no effect on the students (is that what you’re saying?) but I’m simply stating that, in every school I’ve attended or worked in, the behavior and attitudes of the students/parents are extremely important. Countless studies show that out-of-school factors are much more influential when it comes to academic peformance than in-school factors.

    While a great teacher is (of course) welcomed by all students, I’ve seen that hard-working, motivated kids with supportive families do pretty well in school no matter which teacher they happen to get. What exactly are you arguing – that student/family behavior and attitudes DON’T contribute heavily to academic success?

  45. sm Says:

    No. No, no, no.

    No, you aren’t just “…simply stating that, in every school I’ve attended or worked in, the behavior and attitudes of the students/parents are extremely important.” You are stating that AND that KIPP enrolls only a subset of families who have supportive families. Don’t back out and generalize now, like you did in your second paragraph when all of a sudden “KIPP” became “schools” (your full quote: “schools can select their student body based on demonstrated performance or behavior, the school can have an effect on educational achievement”).

    You wrote that KIPP only enrolls certain students here:

    “when KIPP enrolls only those children whose parents are educated enough and motivated enough to search out charter school options (who presumably also communicate the importance of this option to their children), KIPP is enrolling a subset of families who are more able to provide the necessary support and attitudes to their children to achieve scholastice success.”

    And here:

    “My point is that schools like KIPP give an advantage to their students by enrolling only a group of students who are more motivated and geared toward academic success than the average public school students in their neighborhood.”

    I pretty plainly stated what I was arguing; I will even cite it again:

    “You phrase this as if you know this for a fact: “Therefore, when KIPP enrolls only those children…” and “KIPP is enrolling a subset of families…” Where are you getting the support for these statements?”

    Here it is in one line: where is your support that KIPP only enrolls a subset of families or children?

    Also, I never argued “that student/family behavior and attitudes DON’T contribute heavily to academic success.” I have no idea why you got that– please cite where I stated that.

    To summarize, since it worked so well with the other commenter:

    1) What is your exact and explicit evidence that KIPP is enrolling only a subset of families?
    2) Cite where I argued that student/family behavior don’t contribute to academic success.

    If I am to argue anything– and don’t take issue with this before you answer 1 and 2– I argue that KIPP systematically enrolls students whose family structures are worse off. If we take the MPR study that PROVED (not just some commenter ranting) that 22 KIPPs systematically enroll students who are worse off than their neighborhood peers, it is a logical albeit tenuous extrapolation that those students’ families may be worse off than their neighborhood peers’ families, and that therefore KIPP is systematically enrolling students whose family structures are worse off and doing something well to produce greater academic gains than their peers.

    To break it down very simply, MPR showed KIPP students are worse off when they start in academics, FRL and are more likely to be a minority, and that if KIPP kids are worse off when they start than the kids that live down the block, their families may be worse than those kids’ families down the block. So KIPP is doing something well to affect parental structures in a positive way which in turn is helping academic achievement despite KIPP kids being worse off from the start, both in academics and MAYBE in their families. (Note: all the “maybes” and “mights” are placed at points where I make the tenuous extrapolation that is not founded in any research whatsoever and is merely my opinion.)

    So if you read this far, though, don’t forget Attorney DC: you still need to answer 1 and 2, and here they are again!

    1) What is your exact and explicit evidence that KIPP is enrolling only a subset of families?
    2) Cite where I argued that student/family behavior don’t contribute to academic success.

  46. Attorney DC Says:

    sm: You asked, “What is your exact and explicit evidence that KIPP is enrolling only a subset of families?” My answer: I don’t have ‘exact and explicit evidence’ (does anyone?) with regard to the motivation levels of the parents of KIPP students compared to other parents. What I have is the knowledge that ANY program/school that requires people to take affirmative steps to enroll will, per se, only attract a subset of the general population — specifically, that subset with the desire to participate. Unless ALL the families in the city applied to KIPP (and of course they didn’t), how can you argue that there is no difference between the motivation/academic interest of the families who applied to KIPP and those who did not apply?

    This is a separate question from whether the KIPP students were high-performers prior to entering KIPP. Some of my best students were those with lower than average knowledge and/or raw intelligence, but who had the dedication and hard work necessary to devote themselves to their studies. It’s quite possible that students entering KIPP only had mediocre grades prior to entering the charter school (quite possibly due to peer effects at their local school) but were pushed by their parents to attend KIPP and, once at KIPP (surrounded by other like-minded students and following strict KIPP rules) had more success.

  47. sm Says:

    Thank you for answering the question, although you did not explain where I ever said that student or family behaviors don’t contribute to academic success (here’s the answer: I didn’t. You suggested I did.).

    And you also answered my main issue: “my answer: I don’t have ‘exact and explicit evidence’ (does anyone?)…” No, no one does. You don’t. I don’t. No one does. There is no explicit evidence that KIPP enrolls a subset of families who are more proactive.

    And here is my follow up: you write that you the “knowledge that ANY program/school that requires people to take affirmative steps to enroll will, per se, only attract a subset of the general population.” And this goes all the way back to one of my two first questions: hey, I will once again cite it: “Or, have you worked in admissions for KIPP? Have you worked at KIPP and can first-hand prove this?”

    You neglected to address that last time. You addressed my assertion that you that you don’t have the research, but you never addressed that. In fact, you blindly trumpet out the assertion that charter schools will inevitably attract more involved parents because of the application process without any specific information about how KIPP enrolls students.

    And that is why I wrote that paragraph about me helping in admissions once. Do you have first-hand experience with how KIPP conducts their admission process? Because you seem to suggest this generalized idea of how all charters do it with zero actual knowledge of how KIPP does it.

    So it took me two posts from you to admit there is no research to prove your point, so now I go to the next one, which is the same exact question I asked all the way above:

    Have you worked in admissions for KIPP? Have you worked at KIPP and can first-hand prove this?

  48. John Says:

    Attorney DC,
    “Desire to participate” does not equate with academic performance or other character traits. It may well include things like the desire to have a child be in school from 7:30 until 5PM so that after school care isn’t necessary.

    I think the Boston Charter School study put this issue to bed by comparing lottery winners to lottery losers with a clear advantage going to the kids that ended up in the charter schools.

  49. Chris Smyr Says:

    Attorney DC:

    Sm addresses nearly the entirety of your latest replies to me (I’m eager to see your response to him/her — please give it).

    You continue to imply that there is a different brand of family that is recruited to KIPP, yet you are unable to provide evidence on this matter. You also agree that in-school factors are critical for getting these kids to achieve but won’t admit that teaching ability is one of these factors. Right.

    What I will respond to is some of your 2nd reply to me:

    You disagree when I say that “were it up to you, these families will continue to draw straws hoping they don’t end up with any ineffective teachers”, yet that is the state of affairs implicitly being defended by your logic. I don’t care if you think teachers are not a big problem here. Realize, again, that I’m not just disagreeing with you, but am also showing why your reasoning is wrong.

    In-school factors matter (you admit this), and teachers are one of the biggest of these in impacting achievement (you need to admit this). If you continue to make excuses for ineffective teaching — where excuses are unnecessary to be made — you are defending the system that results in families essentially picking straws to avoid ineffective teachers. We simply cannot act on teaching effectiveness in your vision of reform.

    You then go on to say that “the main problem in low-income schools [...] is the school culture and the students/families who attend the schools.”

    Well how about that. I don’t know if you intended this to read so bluntly, but I think this does clear up how two-faced your ideas of reform really are.

  50. Rob Clifford Says:

    A school wants to start a baseball team. Let’s assume they could build the team 1 of 2 ways:

    1) Advertise and recruit saying all are welcome to sign up and join the baseball team. They will accept everyone. No experience or baseball aptitude required.

    2) Randomly pick out 20 names from the school yearbook.

    Which option will most likely yield a better team? My money is on option 1. Even though they’re not trying to “cream” since everyone is accepted and no one is getting cut, it is likely that these kids at least like baseball (or their parents do) and have some interest in doing well since they voluntarily chose to sign up.

    Option 2 may also end up finding some good potential players too, but we can’t make any assumptions about their initial interest in playing, and the coach may end up with some players who lack buy-in because they have to be there and aren’t necessarily a match to the coach’s approach.

    Short of a scenario involving random assignment of students to a KIPP school vs. a neighborhood school, I don’t think either side is going to be swayed by the other’s comments (though the back and forth banter is amusing). Even if such a study took place, I’m sure the findings would be attacked by the side that wasn’t happy with the conclusion.

    One of KIPP’s core operating principles or “Pillars” demonstrates that being a school of choice DOES matter to them:

    “Choice & Commitment. Students, their parents, and the faculty of each KIPP school choose to participate in the program. No one is assigned or forced to attend a KIPP school. Everyone must make and uphold a commitment to the school and to each other to put in the time and effort required to achieve success.” (http://www.kipp.org/about-kipp/five-pillars)

    Since neighborhood schools are not “Schools of Choice” there is no starting point of buy-in or assumed willingness to make the kind of commitment described above. Students and parents have not made any promises to do all they can to create a successful academic school culture.

    Really, the majority of comments boils down to whether one thinks the act of having people sign up and pledge to do certain things constitutes creaming.

    I think that KIPP does a number of things well to help students earn the results they get (see that Pillar reference again). Being a school of choice is one of the things that contributes to their ability to achieve such results and should not be discounted.

  51. sm Says:

    Rob,

    I understand your example, and I do agree with you on the fact that a majority of the comments do boil down to what you say. I have one tack here, and I have been working it for awhile: there is no explicit evidence of this– mostly because the random assignment you mention would be impossible, probably– and the points that KIPP enrolls a subset of families is entirely based on one’s personal assumption and feelings.

    On your point about the baseball team, personally I think there are shades of grey in there that could convolute the example, and I think those are meaningful (e.g., do they “recruit” from every household in the neighborhood or not, and how is the recruiting phrased?). Most of what I have been getting at are these shades of grey– you will see that I have rarely (I did it once but qualified it a lot) actually made an argument, but instead I am desperately trying to point out that we don’t know these things for a fact no matter how much one writes them as a fact. That and people pretend to read research but do not (since when did that become trendy and cool?).

    Otherwise, while I think you do a minor little switch-a-roo– simply because KIPP states that everyone must make a choice does not suggest the inverse, that they only take those that make that choice– I do agree with you, but the weight to which we give this “school of choice” factor with respect to everything else in determining student outcomes is an issue: some think that it is the only factor, and that is what I take issue with.

  52. Attorney DC Says:

    Rob: Well said. Your baseball team analogy is on-point. I had thought of a similar analogy (to a little league team) myself: As long as students or players must take an affirmative step (registering, attending interest meetings, filling out an application) to join a particular school/team, such students/players will automatically be more likely to ‘buy in’ to the activity than would a randomly selected group from the community.

    In addition, the students/players who affirmatively joined the school/team would likely be more swayed by the threat (implicit or explicit) to be kicked off the team (or expelled from school) if they do not follow the rules, than would a student/player who had not affirmatively selected the school/team in the first place.

  53. Attorney DC Says:

    I’d like to add that the above concept of “buy-in” correlates to my experience in the real world of students being more scared to be expelled from a private or charter school (or selective program) than from a traditional public school. This tended to translate into appreciably better attendance and behavior patterns among the students.

    For example, when I taught SAT prep classes for the Princeton Review, there were almost no absent students, and students uniformly behaved well and paid attention in class. This included, for the most part, students attending the free classes we provided for low-income DC students who wanted to do well on the SAT (primarily to qualify for college athletic scholarships). All these students made the choice to attend these classes (and, for the regular classes, to pay for them). Consequently, they had a certain amount of ‘buy-in’ or they wouldn’t have taken the time to register and show up. Their behavior and attendance reflected that ‘buy-in’, which I would also expect to occur in a school like KIPP.

  54. sm Says:

    Hey Attorney DC, you didn’t answer my question yet. This is the fourth or fifth time you didn’t, and I keep asking it:

    “So it took me two posts from you to admit there is no research to prove your point, so now I go to the next one, which is the same exact question I asked all the way above:

    Have you worked in admissions for KIPP? Have you worked at KIPP and can first-hand prove this?”

  55. Attorney DC Says:

    SM: I haven’t worked in admissions at KIPP, but I’m not sure how relevant that is to this discussion. Someone working admissions at KIPP would presumably have knowledge about the KIPP parents, but wouldn’t have any research-based method of comparing these parents to the vastly larger group of non-KIPP parents, correct?

  56. Chris Smyr Says:

    Still waiting for a reply from Attorney DC, particularly about how his opinions are different from facts. “Don’t have time to address all your points” he said last week. What about now?

    http://www.eduwonk.com/2011/04/still-going-2.html/comment-page-2#comment-219306

    http://www.eduwonk.com/2011/04/still-going-2.html/comment-page-2#comment-219390

    *****

    Rob,

    Your analogy fails because:

    * It presupposes that there is a sizable group of families/students who do not “at least like [school]” or “have some interest in doing well [academically]“. You acknowledge that we can’t make any assumptions about these families’ initial interest levels, yet you do just that when you suggest there will be many families who “lack buy-in” in traditional classrooms. Given this, it’s not clear if the intrinsic differences in motivation to succeed are large or even apparent between the schools.

    * Whereas option 1 will likely yield a group of kids that will play better than those gained from recruitment under option 2, for KIPP this is not the case. Before starting at KIPP, these students are not performing any better academically than their peers.

    * The analogy implies that these kids will play baseball only if they are recruited. In reality, every student is participating in a classroom. Thus, if a family chooses to keep their kid in a traditional school instead of being recruited to KIPP, that does not have any implicit bearing on said student’s/family’s motivation to “play ball”, as there may be many other reasons for not wanting to attend KIPP.

    KIPP is a school of choice, but I think the critical point here is whether KIPP is effectively “creaming” (intentionally or unintentionally) or whether the possibility for such exists but is not yet shown to be fulfilled. I guess answering this question does depend on how one defines “creaming”, but I hope that what I’ve written above explains why your definition is too weak to be useful here: “Signing up” doesn’t directly translate into a better KIPP/baseball team. That these kids become great after signing up ought to also suggest how important in-school factors still are.

  57. sm Says:

    How do you not see its relevance? My point is precisely what you say: someone working at KIPP would have that knowledge about the parents, but you do not work at KIPP, so you do not have that knowledge.

    So you haven’t worked in admissions at KIPP– so you don’t have first-hand experience of how they recruit and admit– and there is no explicit research supporting your argument that KIPP enrolls a subset of families.

    So if you don’t have first-hand experience and there is no evidence to support your argument, your continually maintained point that KIPP enrolls a subset of families is just a guess with no basis.

  58. sm Says:

    Chris,

    Is there a specific report or research or whatever that lays out the relationship or relative weight of in-school vs. out-of-school factors and their weight on student outcome variables?

    My area is charters/ vouchers/ teacher preparation… not as much this in-school vs. out-of-school argument (peer effects… don’t know that much).

  59. Chris Smyr Says:

    sm,

    I don’t have specific papers that I can highlight but scattered along Google/ERIC one can find a large swath of papers suggesting the (apparently) exceedingly bold idea that teacher effectiveness is an important variable in the student outcome equation.

    Folks can argue ’til their blue about how which of these factors are more important, but that’s besides the point. We can do a lot of good by keeping a focus on in-school factors, particularly because we can readily change these regardless of which social revolutions are being plotted on comment boards throughout the blogosphere…

  60. john thompson Says:

    Here’s my latest.
    http://www.huffingtonpost.com/john-thompson/kipp-and-its-critics-are-_b_848911.html

  61. Art Says:

    John … You seem to labor under the misconception that students on IEPs are completely the responsibility of a particular school. They aren’t. They are the responsibility of the local education agency. So if kids with IEPs are underrepresented in KIPP schools, it’s a structural problem more than a KIPP problem.

  62. john thompson Says:

    That’s what I said. Its not KIPPs fault that 40% of my students are on IEPs and that 2/3rds to 3/4ths of my afternoon classes are on IEPs, ELLs, 504s for mental illness, or parole for extreme felonies. The problem is pretending that they teach the “same kids.” The problem is using KIPP to argue that 5/6ths of my 237 students who aren’t counted under NCLB don’t need services because KIPP shows it just takes no excuses.

  63. sm Says:

    And I return.

    Why is “gangbangers” not hyphenated early in the post but then hyphenated later on?

    Anyway, I am sure you don’t want my take or care since you vowed to “actively ignore me” and totally adorably “mistyped” my name, but here you go:

    1) You set up with this illogical beginning, like MPR claimed KIPP admitted students mid-year by virtue of their terminology (“late arrival”), and you use that to launch your piece. It’s a silly piece of writing.

    2) You make this absurd jump:

    “Its research tells us nothing about whether the KIPP system of 99 schools could be scaled up.

    To do that, we need studies that act like “race track monitors” to make sure that no interest groups get “an unfair advantage” in promoting their preferred agendas.”

    Well, MPR or its research or anyone mentioned anything about “scaling up.” Also, really? How in the world would “race track monitors” assess the possibility of large-scale replication? You write this merely to get in the Miron study. It doesn’t make sense.

    3) In the comments, you write “As I recall, the attrition data comes from 19, not 22, of their 99 schools, and two excluded schools are illustrati ve. They are studying the success stories. Give the other schools you and I have heard about a decade to implement their vision, they may reduce their attrition rate – or they may not.”

    Really? Come on. Just read the darn selection methodology and state the facts.

    4) Your last question is absolutely ridiculous: “Why not give our neighborhood schools the same chances to help poor kids that we give to KIPP?” That doesn’t even make sense. What are these “chances?” You write a whole piece on basically how the two entities, KIPP and a public school, are different, but then you end with this question that assumes the same things could be done.

  64. Attorney DC Says:

    John Thompson, well said: “Its not KIPPs fault that 40% of my students are on IEPs and that 2/3rds to 3/4ths of my afternoon classes are on IEPs, ELLs, 504s for mental illness, or parole for extreme felonies. The problem is pretending that they teach the “same kids.”

    I agree with you: There’s nothing wrong w/ KIPP (or private schools or gifted pull out programs or any other specialized programs). The problem arises when charter school advocates proclaim in the media that their schools teach ‘the same students’ as the traditional public schools. In fact, charter schools teach different students and are released from many of the contraints placed on public schools (for example, KIPP doesn’t have to accept transfer students mid-year). Charter school advocates should not pretend that they’re teaching the same students (with IEP’s and parole officers), under the same constraints, that the public schools have no choice but to deal with every day.

  65. Chris Smyr Says:

    Really can’t stand these “here’s a link to something else, but no I won’t respond to criticism” tactics. These debate strategies are not befitting a self-proclaimed academic, nor are other commenters giving a good-faith attempt at debate when they similarly do this.

    ***

    John writes: “I was excited to read that Mathematica had analyzed “late arrivals” who attended KIPP. That seemed to be an inspired methodology for determining whether KIPP “creams” by excluding the most difficult-to-educate kids. I thought that KIPP did not admit students after the year began, but if they had a big enough sample to compare the characteristics of KIPP students who arrive in October, for instance, with students who first enter neighboring schools at that time, we would have real evidence!”

    The goal posts are moved yet again. Apparently, it is not considered “real evidence” when Mathematica analyzes characteristics of KIPP-bound vs. non-KIPP-bound students along several different variables and finds little difference between the two.

    ***

    John writes: “Saying that KIPP’s attrition rate is a little better than some of the nation’s worst middle schools’ rate is not a ringing proclamation of success.”

    No, but what it does show is that critics who argue that KIPP is forcing out the low-achievers at high rates are incorrect. KIPP’s success becomes apparent rather when one looks at their achievement data.

    ***

    John writes: “Mathematica simply confirms that KIPP succeeds greatly with the kids where it succeeds, while indicating that its failure rate with their more difficult-to-educate students seems comparable to the failure rate of the toughest neighborhood schools. Its research tells us nothing about whether the KIPP system of 99 schools could be scaled up.”

    This is misleading. That KIPP does better for the majority of students who stick with the program than if they were in another local school suggests they *are* succeeding with some of the “difficult-to-educate students”, as their students on average achieve at lower levels when starting out at KIPP. In other words, the same percentage of attrition for either school still may leave KIPP with more prior low-achievers. It’s silly to suggest KIPP is failing because some of their students drop out, when many of these low-achievers stay and succeed at high levels.

    Also, there’s just a slight difference between 99 schools and ~100,000 schools. KIPP could increase in number 10x and not hit any of the roadblocks you are implying above. That there are students that drop out of KIPP does not imply that scaling up KIPP schools would stifle their noted average academic achievement. There are still plenty of kids/families that would likely benefit from such increases.

    ***

    John writes: “And to my knowledge, it has never been claimed that KIPP has been more successful with IEP students diagnosed with conduct disorders or serious emotional disturbances.”

    Sure, but this doesn’t imply that they are poorly serving these populations, either. Since the SPED populations of KIPP versus the control group are not all that different (9% and 12%, respectively, per MPR 2011), it seems foolish to imply that KIPP isn’t helping SPED students.

    ***

    John writes: “By pretending that KIPP serves our most vulnerable students, society is given an excuse for starving alternative services for our most traumatized kids.”

    KIPP does serve some of these “vulnerable students”. Considering KIPP is successful with a large number of them suggests we ought to consider their methods and/or encourage their scaling up to help more students further succeed.

    To be clear, none of what I’ve written above implies that we should ignore efforts to increase other “investments for our most damaged children”, and I think that is a given for most of the folks you disagree with. Until these vague investments are enacted (and concurrently afterward), we ought to continue pushing for school-related reform.

    ***

    John writes in the comments of his blog: “Deformers don’t play fair.”

    You respond with playground taunts to those you agree with but refuse to debate your critics? That’s exactly what we need more of in education.

  66. Phillipmarlowe Says:

    John,
    Is there any evidence that KIPP’s “success” is based on more than the longer school day?
    It is not right to compare KIPP to the neighborhood school if the additional instructional time is not figured in.

Leave a Reply


five + = 12