Eduwonk

The debate over standardized testing is an odd one. It’s highly political, often divorced from the facts. But that’s hardly unusual in education – and other issues.

What strikes me, in addition to the pervasive lack of appreciation for how these assessment systems work and why, is that testing and accountability proponents generally like tests much less than their critics think they do. It’s more of a Churchill and democracy kind of sentiment. A lot of people don’t realize that the information and quality assurance yield for pennies on the dollar is the envy of most industries. And almost everyone, frequently the media spoon fed by activists, underestimates the generally broad support for assessment among the general public and especially parents. People want to know how their kids are doing in some comparable way.

Bellwether recently released six primers on major elements of assessment practice. They are not intended to convince you that testing is great. Some of this is a values debate and some of this is, as yesterday’s post mentioned, a ‘for what purpose’ debate.

And there is plenty of room for nuance. I have some concerns about high stakes testing for kids, but also think employing scores aggregated at the classroom, school, district, and state level can provide important analytic leverage.

Others disagree. And that’s all fine and healthy insofar as that debate is grounded in how these things actually work rather than misinformation or caricatures. That’s the point of these new Bellwether briefs, which were funded by a consortia of assessment providers, not to convince but to inform. They look at how assessments are designed, special populations, comparability, reporting, and other key issues:

• What Statewide Assessments Are Designed to Measure
• Ensuring That Assessments Accurately Measure Academic Standards
• Developing High-Quality Assessments and Items
• Ensuring Comparability Across Administrations
• Making Assessments Accessible for Students With Disabilities and English Learners
• Reporting Assessment Scores

As with all work that carries our byline or brand, we retained absolute editorial control so nothing in these briefs should be attributed to any entity or anyone other than the authors.

Bellwether’s Michelle Croft, an actual psychometrician, and I sat down to talk about some of the issues:

In a new video series, @arotherham sat down with Bellwether's own former psychometrician Michelle Croft to ask some FAQs about state standardized assessments. 📽️

First up: what's the biggest myth about state standardized tests?

Learn more: https://t.co/jWKMVkuMcU pic.twitter.com/P6ue4A0BOJ

— Bellwether (@bellwetherorg) April 17, 2023

Previously on this issue: Here’s a 2021 look at testing. Here’s Shilpi Niyogi on testing. Here’s Anne Wicks. And here’s Jenn Schiess, Bonnie O’Keefe, and me for NASBE on the broader issues.

We haven’t had music in a bit, so here are the Chainsmokers.

You know how they say don’t answer a question with a question. It’s sometimes good advice. Along the lines of don’t use a word to define that word.

Here in education, however, the inverse may be true. When someone asks a question, the best answer is often another question, or three.

That’s not only because asking good questions is key to getting to good answers. It’s also because, in broad strokes, America’s education system is sprawling and diverse while most of the questions turn on specifics and particulars. The issues tend to be blunt force, but the answers require nuance and texture.

For example, when we talk about standardized testing you often hear the debate framed as for or against or good or bad or more or less. In fact, if someone asks about standardized testing the key questions are, what kind of test? What age level? For what purpose? High stakes for who, adults or kids?

Policy around transgender athletes. The debate is quick to yes or no. But the actual questions that matter are what age, what level of competition, what specific sports?

We talk a lot about whether big funding streams work? When the questions should be about what specific intervention(s)? For which students? Under what conditions? That question bedevils the ed tech industry and seems likely to be an issue with AI as well.

With today’s debates about censorship and talk of – and actual – book banning questions about what age group, what book, for what reason, and in what setting, for instance classroom library versus school library versus community library, are key to sorting out various claims and counterclaims. And it’s worth asking, is this a new issue (and some are the past few years) or a long running debate with a new culture war veneer or click-seeking take on it.

Of course, the evergreen – is college worth it? When the questions are for who, what schools, what program or course of study, at what cost? (Look for more on that from Bellwether later this week).

Other perennial favorites include, does school choice “work”? When the actual questions are what kind of choice schemes (publicly controlled choice, charter, private, tax credits, ESAs), with what rules or regulations, for what students? And work how? Grades? Graduation? College or post-secondary going? Parent satisfaction?

Or does money matter? That’s another one everyone can’t get enough of even though the devil is in the details.

Performance pay as a binary “yes” or “no” or does it work or not, obscures all the permutations of those polices. School segregation is often taken about in broad terms that ignore local context and variance.

You get the idea.

We’ve talked in the past about the definitional confusion problem, and that’s a part of this. Just simply asking what exactly are we talking about here and being specific matters.

And some of it is sloppiness around base rates. Is what we’re seeing really a change, or is it just being noticed (or reported on) now? And category errors, not asking if things being discussed are really the same?

But asking, and answering, good questions in education is also about some more sector specific follow-on questions like, again, what works under what circumstances and for what students? Or is this an outlier or edge case or something we see a lot around the sector – with more than 100,000 schools, across 50 states and 13,000 school districts you can find an example to prove almost anything.

We should not let the education debate turn into some exercise in nihilism or relativism where the answer to every question is just more questions. There is too much evasion now, and used a certain way questions can be a deflection. And there are actors who use this strategy in order to retain power – it’s all so unknowable, so many questions, just trust us.

Good questions can help punch through bad faith obfuscation (and preference falsification) and can separate sheep from goats on hard issues. Not everyone questioning standardized tests or accountability policy is an apologist for poor school performance or achievement gaps. People concerned about fairness in women’s sports are not all axiomatically transphobes or bigots. Not everyone skeptical of performance pay is a hack. And not everyone raising questions about appropriate content in schools is censorious or unsophisticated. But you won’t know if you don’t ask.

Especially in this hopped up time becoming better at the follow up questions might elevate the conversation some and be the kind of thing that lowers the temperature some. A question I ask a lot that might apply here is, if everyone starts asking more questions, then what’s the worst that can happen?

In The 74 today Drew Pache and I take a look at school security. The defeat of the shooter at Covenant School was professional and highly competent, which is not always the case. Still, two things are true at once. The violence there could have been far worse. It was also not the best outcome, six people still died.

From The 74:

The police body cam videos from the shooting at Nashville’s Covenant School left few unimpressed. The training and professionalism of the officers was obvious, and despite the tragic loss of life, it could have been far worse. Their actions were a stark juxtaposition with what transpired in Uvalde, Texas, or Parkland, Florida, where disorganized or delayed responses compounded tragedy. 

The outstanding performance of the Nashville Metro Police officers likely drastically reduced the number of casualties, but six killed instead of 15 or 20 doesn’t feel like a win. We must not lose sight of an underlying reality: By the time a gunfight breaks out at a school, many systems have already failed. Defeating a shooter at a school should be the very last line of defense. Politicians and educators should commend the heroism of the Nashville police without creating the impression that this response is the ideal outcome.

That’s because when it comes to schools, the most effective gunfight is the one that never happens.

You can read the entire commentary here in The 74. We talk about the measures – most importantly healthy school culture – that schools can take upstream to prevent shootings. When four in five shooters tell you what they’re going to do ahead of time we should take that seriously. Don’t take our word for it, the Secret Service says the same thing.

Drew Pache and I previously wrote about security in The 74 here, and why training for adults is more important than scaring kids. Drew, who is a retired Special Forces officer with substantial global experience, wrote here about why arming teachers is a lousy idea. And I’ve written about while these episodes are horrifying, they’re rare and there is far more common gun violence we should be more focused on.

Clearly, there are not enough middle-aged guys with podcasts.

So Jed Wallace and I decided to address this market gap. Jed’s popular “Charter Folk” newsletter and his interviews focus on the charter sector and issues in and around it. I’ve been a guest a few times. It’s a great medium, despite that.

Now, as I mentioned recently, Jed and I are going to do an informal and semi-regular podcast on education goings on more generally. It will be us, and some guests. In our first one, just out today, we discuss some charter stuff but also the absurdity in Los Angeles and the politics in Chicago – and why Karen Lewis still doesn’t get the credit she deserves for reversing the teachers’ unions fortunes – there and more generally. She walked so Brandon Johnson could run. We speculate wildly about what AI might mean for schools and we discuss some of the nuance around school safety. It’s the first one so feedback extra welcome.

You can listen here:

Watch here:

Transcript here if that’s your thing.

Two reactions to recent posts here on Eduwonk.

On philanthropy and metrics and investing in people, reader Mike G writes,

If I ran a big charity, I’d invert the whole thing.  Rather than metrics, I’d tell them I will ONLY consider additional funding based on the quality of them telling me everything going WRONG.  “True confessions” is what I’d reward.  With hypothesis that in telling me that, it may plausibly trigger an authentic what will we do.  

My weekly sends to all grantees would be like this: 

Dear Grantees,

This week we frigging LOVED xyz tutoring org’s report, of the absolute chaos in their NYC expansion.  We sent $10,000 of vodka just to help them cope.  Seriously, they really shared the nuance of what was going on, that they had this theoretical anti-racist framework but the kids laughed in their face about how “weak” the tutors were and just played on their phones.  So now at least CEO is trying to lead an authentic convo on what to try, that makes up happy.  

Also we got a total bullshit spin job from ABC ed tech grantee we won’t mention.  It wasn’t actually sent to us, it was sent to all their donors, but it was the classic annual report pack o’ lies.  Remember the test: we cold call your people on the front lines.  And we find ways to hack access to the kids themselves.  For ABC, we cold-called some teachers who said ABC was really struggling, and then they put some of their students on a Zoom who said ABC was lame.  Again, fine for ABC to be lame if they’re fighting to make it NOT lame.  But they’re in the charity collection business, the prestige business.  They’re dead to us.  

Sincerely,
Rich Mike

On the what we should expect from schools question and the deBoer essay, David G. writes,

As you know, there are aspects of Freddie deBoer’s thesis that resonate with me, but there are two major issues with his piece that I don’t think that you picked up on that are perhaps interesting. 

Optimism bias is good/essential

He makes it as if optimism bias is bad, but actually it’s essential for innovation.  Silicon Valley is optimism bias central  — every deal is going to make a gillion dollars, and they keep on pouring money into it whether it seems to be working or not, until it’s apparent it isn’t, and then they walk 10 feet and invest in the next crazy idea. If all education people were of the opinion that nothing works, everyone would go home.

Yes, deBoer is right to the extent that we need to realize eventually when things don’t work, and in fact, education does course correct, if slowly. But being pessimism-biased would destroy our ability to dream, innovate, make mistakes, ignore the last failure and start the next one without looking back. My rose-colored glasses are welded to my face, and without them, I’d be writing grouchy pieces on my substack instead of out there fighting the good fight.  Without optimism bias, Bellwether would close its doors (and that would not be good for anyone).

To the extent that he’s right, the system doesn’t spend much time on post-mortems – that was my take home from the time that I spent at Bellwether talking years ago about the failures of education reform. If you look at the amazing success in reducing the number of planes that fall out of the sky or patients that died from anesthesia, it was all due to the system carefully analyzing why people died and coming up with better methods. I’m not sure that education really hasn’t learned many lessons from the failures of ed reform (though I’m not really a member of the inner circle of ed policy people, and so don’t know if those conversations are happening in private).  More generally, I don’t think that education has come up with a framework for analyzing successes and failures – there’s no group dedicated to this – what we at SeeMore call the Theory of No Change, which is necessary to developing a coherent Theory of Change.   

And one last point – the issues that he states about orthodoxies (and they are, as you acknowledge, not all wrong) are largely true of sociology, development economics, all economics, psychology, political science, and not only “soft fields”, but also hard sciences like biology and physics. All of these areas are cultures with their own norms and beliefs – someone should show him Kuhn’s The Structure of Scientific Revolutions. To the extent that education fits here, I can’t see that it’s any worse than other fields, and it’s fundamentally built into human endeavors and progress. Change happens and generally to the better, but it’s on its own timeframe – faster probably in the sciences, but even there it can take a while (cf geocentric model of the universe, Newtonian physics, etc.

Much of education is actually pessimism-biased   

deBoer is complaining really about policy people.  But if you go to a lot of schools and colleges, they’re pretty skeptical about every new last great idea, and many are resistant to new ideas. Many teachers don’t actually believe that their students can learn.  I still don’t have a good sense of the distribution of optimism/pessimism among teachers in a turnaround high school, for example, but I definitely get a sense that some amount of pessimistic fatalism is there. And there’s at least some strong suggestive evidence that many teachers don’t have a growth mindset attitude about their students (and I get the sense that almost no one believes in growth mindset around personal characteristics, which I believe is very wrong). On the ground, and not in the policy clouds, I would guess that pessimism is more rampant and dangerous than unbridled optimism.

In fact, why is it that charter schools work?  Is it really that their pedagogy and curriculum is better?  That TFA teachers outperform veteran teachers? My suspicion is that their inherent optimism was a big part of what made the good ones work so well, and especially in comparison with public schools, that often have a pall hanging over them.

One of our major focuses is on self-efficacy – we think that it’s the major factor in personal success (and there’s a raft of both educational psychology and organizational development in the business world to back that up).  But we think that self-efficacy isn’t only a personal characteristic, but a characteristic of organizations and entire systems. Optimism-bias is not true self-efficacy, since it’s not always informed by actual success and a meta-cognition for what does and doesn’t work. But pessimism is pure poison to self-efficacy. Let optimism reign!

It turns out some people don’t realize the Boomtown Rats hit “I Don’t Like Mondays” was born of a school shooting. When the shooter, who killed 2 and wounded 9 with a .22 rifle, was asked by a reporter why she did it she responded, “I don’t like Mondays.”

The silicon chip inside her head
Gets switched to overload
And nobody’s gonna go to school today
She’s gonna make them stay at home
And daddy doesn’t understand it
He always said she was good as gold
And he can see no reasons
‘Cause there are no reasons
What reason do you need to be shown?

– I Don’t Like Mondays

I was pretty young at the time but people who were actually at the scene tell me the remark didn’t set off a big discussion about Mondays, whatever deeper meaning or cause that remark portended, or ascriptive claims about people who don’t like Mondays. She wasn’t in a good mental state and did a horrible thing.

After Nashville we’ve heard quite the opposite. The details are still sketchy, in part because of media fecklessness*, but it seems the shooter at Covenant School was transgender or was transitioning their gender identity. This has occasioned all manner of speculation and toxicity. People have asked if the LGBT “community” should denounce the shooter. Really.

I get it. Every time some jackass shoots up a school, church, club, or store there is a race to ascribe some motive or cast widespread blame (that often ends up being wildly discordant with the facts, fog of culture war), so some people see a goose – gander situation, political opportunism, or just a clickbait opportunity here. But the only thing mass shooters have in common is derangement and brokenness. It’s an unhinged act not a rational one, so the search for rational causes and logical motives is a fool’s errand. Most shooters don’t intend to survive the scene as may have been the case in Nashville, it’s often explicitly an act of suicide by law enforcement. None of this is normal or a normal reaction to information or circumstances.

Sure, the 2022 Buffalo shooter was motivated by racial hate. He also was a self-described “eco-fascist” and anti-corporatist. Should anyone criticizing corporations apologize for that part? Of course not, because he’s a whackjob, much of his “manifesto” was cut and paste and all over the place. You’ll search in vain for a coherent ideology or worldview. Whatever the Nashville shooter’s grievances were, at some level they were not rational because rational people don’t kill others at schools.

Remember the whole panic about Goth kids and Marilyn Manson after Columbine?

These episodes are horrifying, so a human instinct is to make sense of them by placing them in a broader context or cause. Perhaps because it’s even scarier to think that sometimes people just snap and do incredibly deranged things because they believe incredibly deranged things because they’re not thinking straight. Lots of people get exposed to hateful or crazy ideology and don’t go shoot up public spaces or drive into parades. The randomness is even more terrifying than the sense that perhaps there is a pattern we can discern and address.

Here’s the thing about school shootings: They’re random, but rare, and often preventable. For starters, by definition these are small n events. When you ask people how many people have been killed in mass shootings at schools since Columbine almost a quarter century ago, you usually get numbers in the high hundreds and the thousands (or even tens of thousands among people who really spend too much time online). The actual answer is 175. That’s quite obviously 175 too many and each one is uniquely awful in its impact on families, friends, and others. One is too much. But it’s a pretty rare episode given the scale of American education. Both those things are true.

More common is the interpersonal violence and neighborhood or domestic situations playing out at schools one small episode at a time because of the ubiquitousness of schools. More common still is all the out-of-school violence that takes a staggering toll on American youth.

Important and not discussed enough: https://t.co/AutpSONMxx

— Andrew Rotherham (@arotherham) March 24, 2023

In addition, about four in five school shooters tell you what they’re going to do ahead of time. Episodes like Parkland, Florida or Oxford, Michigan were preventable. That’s what should haunt us. (We don’t yet know the details, but the Nashville shooter’s parents were apparently concerned enough about their behavior to try to restrict their access to firearms.)

Professionals call this “leakage” and it’s often present upstream with mass shootings in general. School shooters signal their intentions or just tell you outright what they’re planning in a majority of cases. That’s what we should take seriously, not toxic theories about this group or that one. You can’t profile school shooters, they’re too varied, but you can act when someone shows you their hand. Attacks by outsiders – for instance in Uvalde or Nashville – are rarer still.

The loud rhetorical approach to all this is mostly politics not safety. And the focus on politics means the school safety industry is largely becoming a reaction industry when it should be a prevention industry. There is a lot of money in reaction-based security. Yet it doesn’t matter how many lockdown drills or how much ALICE you do, if your school culture sucks and kids don’t trust adults then you are at risk.

*The coverage has been something. People can’t even live up to their prior (like week before last prior) commitments and routinely misgender the Nashville shooter, who apparently wanted to be called he/him not she/her. Until a few minutes ago misgendering was violence. Except not in this case. Because…something.

That sort of panicked situational BS just fuels more mistrust and the idea that something is up. Echoes of Loudoun County. That this apparently trans person in Nashville did something horrible doesn’t tell you really anything about trans people any more than white, Black, or Asian shooters lend themselves to generalizations about their race or ethnicity. In the social media era, idiotic reactions on the left and also the right get magnified, but they’re outliers. It would be helpful if people more forthrightly stepped up to the plate on all that so we could focus on the real lessons here. What’s seen as good intentions or allyship makes an awful situation even worse and does little to prevent the next one.

Earlier this year I took a look at how different ways of grantmaking – in particular project based funding –  might be a contributing factor to the ongoing meltdowns at progressive organizations. And more recently a look at philanthropy and risk.

In both cases, some of the most interesting responses came from people inside philanthropy – blink three times if you’re being held against your will.

So I want to share a third and related observation. In an effort to improve diversity in the education sector we may be hindering effective operations of non-profits more than helping. Some people in the non-profit and philanthropic sector have raised concerns about this as well, it’s not my unique insight.

Also, at the outset let me be clear: Improving access to philanthropic capital (and private capital for that matter) is an essential goal and a goal that involves a focus on diversity. As in many areas of American life patterns, in particular race, ethnicity, and gender, should cause concern and action. To the sector’s credit, philanthropists saw this problem, too, and are trying to address it. (It’s also a problem along ethnic and class lines. Going to the “right” schools, for instance, still helps a lot.)  The end result is that lot of people are shut out. Changing that is for the good.

My concern, instead, is about the how of changing that, not the goal.

A brief history of grantmaking that is over-simplified but good enough for our purposes goes like this: For a long time grantmakers gave money to people, usually men, and usually white men, who they saw as change agents. There were generally agreed upon goals, open up schools, develop new university or secondary school curriculum, or pilot this or that. But the specifics of how to do the work were generally left up to the grantees. Sometimes this strategy led to positive change, sometimes it didn’t.

It was an invest in people strategy. Give people money to do things they seek to do to try to improve the world.

While this work certainly led to some progress in a variety of ways, it also had one big problem. Owing to the structure of American life and prevailing views it was mostly white guys getting the money. Sure, there were exceptions. But the fact that we can quickly name so many of those exceptions, historically and more recently, basically illustrates the problem.

This is not a pre-1964 or 19th-century problem. This was still a pretty common pattern when I started doing this work a couple of decades ago.

To address this, grantmakers decided they needed metrics and measures to help ensure more diversified philanthropy. This became something of a best practice. And they turned to management consultants to create those metrics, often consultants who said, ‘well, why use five metrics when 50 will do?’ (I’m obviously not anti-consultant, Bellwether consults. But my tastes run more to KISS or stuff like this).

Meanwhile, woe to the grantees who have to live under all of those metrics. It’s especially hard for new, scrappy, and under-resourced  grantees – who are more likely to come from what’s now fashionably called “marginalized” backgrounds. And even in well-established organizations it can distort priorities and create paper chases. And yes, it can lead to the kind of internal discord fueled by lots of small grants that leads to meltdowns.

So here’s the question: What if this solution is a misreading of the problem and consequently a badly designed remedy?

I’d suggest the problem might not have been the method of giving – invest in people and be tight on goals and results and loose on the rest. Rather, it was the more basic problem of who was and was not getting the money. In other words, we might ask if the method was sound but the field of view was too constrained. Invest in people, yes, but make sure you are investing in a far more diverse and representative group of people.

This would point to a strategy of opening the aperture, but perhaps not changing the foundational methods of giving.

The problem with the “old” way of grantmaking, it seems, was a problem being far too narrowcast. But the method itself, give money to people and expect them to do good things is not inherently flawed. In fact, it may be preferable to burying everyone in metrics.

The detailed metrics all feel safe. Everyone is doing the same things, being measured the same way. Using metrics and data to compare, that feels very progressive and equitable.

But there are costs.

First, there is a noise to signal problem. Grantees are going to try to meet these metrics or at least appear to meet them and sometimes it’s stuff that is measurable but a distraction from the core goal of a grant.  Take for example “press hits,” a popular metric for policy work. The media is a key part of the policy ecosystem but getting a press hit and getting something done are two different things. Some of the most impactful work is behind the scenes. And a focus on press hits creates an incentive for bad or distracting behavior. Or meetings, a metric for how many you take. With both media or meetings, quality matters more than quantity. Yes, requirements can focus on high quality earned media or decision-maker level meetings. But why? If you don’t trust grantees to have that political sophistication a better strategy is to provide support to develop it.

Second, there is a capacity inequity. Large grantees can bring on talent to manage these grant processes. Setting up all these metrics and tracking progress against them takes time. Smaller grantees – and smaller organizations are more likely to be led by people from exactly the kind of backgrounds funders now want to include – often don’t have that capacity. It’s really not a highest and best use of scarce resources for them anyway.

Third it leads to informal evasion and bad practice. I’ve been told by funders not to worry about metrics, “no one reads them.”I know others get that advice. I don’t have any reason to believe that different people are being held to different standards in any way that’s not pretty random and program officer driven. But why leave people wondering what metrics actually matter when you could just ask for data on the ones that really do matter. It’s sort of mandarin.

Leveling down. The idea that a focus on compliance or common outcomes leads to a leveling down of quality is hardly a new one. But it seems relevant and hard to miss here.

Now look, to be clear* – there’s always a ‘to be clear’ graf on complicated stuff and especially on complicated stuff being oversimplified – I’m not against measurement and evaluation. Or against diligence. We do a lot of that work here at Bellwether. As I’ve noted I don’t think blank check giving it all it’s made out to be. I just think the sector can find a happier medium.

The point is that metrics and measurement should be the minimal amount to help discern impact, not tied to every aspect of a logic model – especially on dynamic work like policy or advocacy. It should certainly not take on a life of its own within any project. So more MVP than pseudo-scientific or creating false precision.** Measurement should serve the work, not the other way around. I should also note, there are also a variety of methods of giving out there, some like what I’m arguing for. Not every grant is a blizzard of metrics. But too many are.

And also to be clear, I don’t want to conflate metrics about who gets dollars with the metrics attached to a grant. Collecting data on the who is important and not especially burdensome. And tracking the who is pretty essential to ensuring that efforts to diversify portfolios are working. I’d suggest a broader set of metrics than what is sometimes used, to also include economic class, veterans status, political orientation, and other measures of diversity.

The larger point is pretty obvious, was the problem with the old way a problem of method of giving or rather the size of the aperture in terms of who was involved? Broadening the aperture and also giving people a lot of room to run seems worth considering?

More diverse grantees and more flexibility for them. Not a great bumper sticker or tee shirt. But maybe worth trying more? A more diverse set of grantees all laboring under the same constraints doesn’t seem like true progress.

*Also, to be clear, at Bellwether we get some foundation funding (though not as much as would help us achieve all our goals and it’s a fraction of our overall annual budget). All sources of funding are disclosed on our website and in any project or publication where it’s relevant. All our funders are brilliant, virtuous, beautiful, visionary people. 

**The tyranny and craziness of small n sizes in all these things is a whole separate issue.