“What Works” in Education Is Not Merely A Question of Effect Sizes

Here’s a pet peeve: A champion of some particular education intervention will point to some research study showing Intervention X led to positive outcomes for participating students. Ok, great. Assuming it’s a good study, we now have evidence that Intervention X “worked” in a given Situation Y.

That does not mean Intervention X will easily replicate to new Situation Z.

Anytime you hear someone attach the phrase “high-quality” in front of some intervention, they’re talking about this problem.  These caveats pop up frequently in debates over particular reforms:

  • The small school reform effort produced long-term gains for students, but their backers largely abandoned them.
  • School integration efforts produced large gains for black students (with no harm to white students), but formal integration programs were and remain relatively small.
  • “High-performing” charter schools produce large gains for students, but there is wide variation in those results.
  • Teacher evaluation reforms produced gains in some cities, but the effects were smaller or non-existent when similar reforms were spread more broadly.

Some of these reflect implementation challenges. Others are more about politics (which is itself a particular type of implementation challenge). I am by no means the first person to make this point, but we can’t just say something “works” or “doesn’t work” without giving some consideration to where the policy worked, for whom it worked, what outcomes changed, and by how much it changed the status quo.

–Guest post by Chad Aldeman

Weekend Reading List

Fun graphs on the change in educational attainment over time by state.

Teach for America keeps getting better.

Hailly Korman and Kelly Robson on the fragmentation between schools and social service agencies.

Max Marchitello on how Arizona keeps cutting teacher retirement benefits.

Bellwether has two new short briefs on charter schools in California, on special education and facilities. In particular, the special education brief addressed one common myth: That California charter schools enroll fewer special education students and thus force traditional school districts to carry a disproportionate burden for educating those students. As the brief outlines, California uses a regional, census-based model that shares special education costs across entities and helps address this problem:

This structural arrangement for special education upends the commonly held belief that charter schools, through a kind of selective enrollment, choose not to enroll their share of students with disabilities, and thereby overburden traditional school district budgets. Instead, districts retain special education funding that is based on enrollment across both district and charter schools and determine how to best deploy those resources.

That funding arrangement system is still far from perfect, but it’s not as simple (or as bad) as the raw statistics might lead you to believe.

Today marks the 65th anniversary of Brown v. Board of Education. If you haven’t already, I recommend listening to Malcolm Gladwell’s podcast episode on the specific direction and argument the Court took in deciding the case, as well as the subsequent impact on black teachers.

–Guest post by Chad Aldeman

The Side Effects of Reading Robert Pondiscio. Plus, What’s Eating Teacher Salaries in Kentucky?

Warning: Reading Robert Pondiscio may cause side effects of unintentional chortles and knowing grimaces.

Neerav Kingsland on New Orleans. His piece is a nice reminder against comparing effect sizes in isolation. It’s much easier to see positive effects of an intervention when dealing with a small sample, but getting the same effect size for an entire city, let alone an entire state, is quite a bit more meaningful in the real world.

Cory Curl with an excellent analysis of school spending trends in Kentucky. Like in California, health care costs seem to be a primary culprit for stagnant teacher salaries.

Lake Wobegon, New Mexico?

–Guest post by Chad Aldeman

Questions and Answers About Teacher Pension Plans, Boston Charter Schools, and Guns in Schools

Can we fix teacher pension plans? In a new piece for Education Next, I show it is possible for pension plans to offer adequate benefits to all workers, but it would require states to re-examine their current offerings:

There are two basic ways states could accomplish this. One would be to significantly increase the generosity of current plans. But to reach our adequacy thresholds this way, states would have to increase benefits substantially, essentially doubling the cost of their current plans.

Alternatively, there are cost-neutral ways for defined benefit plans to provide adequate benefits to all workers, but states would have to radically restructure their current plans.

Check out the full piece for more detail.

Can Boston’s charter sector scale up effectively? Apparently yes. The authors note that beginning teachers in Boston charter schools don’t suffer through the same steep learning curve that teachers in Boston’s traditional schools do, and they find that “the charter sector reduces variation in teacher effectiveness within schools, which may be due to charters’ centralized management of teachers and standardized instructional practices.”

On Twitter, Cory Koedel asked what would happen to school district insurance premiums if they started allowing employees to carry guns into schools. Here’s a good overview from Milliman, and one telling anecdote from Kansas:

At least 24 states across the country have policies that allow security personnel to carry weapons in schools, and at least nine states have policies that allow other school employees to do the same. But how could these laws affect school districts’ insurance policies and coverage? It’s not quite so easy, as the state of Kansas found out when it passed a law in 2013 allowing school staff to carry guns, and an insurer that covers most districts in the state subsequently issued a letter denying coverage to schools that took on this risk. Five years later, no Kansas school employee has carried a gun into a K-12 school.

Basically, insurance companies may play a quiet but important role in this policy question. Insurance companies are having a similar effect on school district and university decisions about whether they can afford to field football teams.

–Guest post by Chad Aldeman