Most everyone in the political and policy world was fixated on all the “what does it mean” questions about Sunday’s NYT Mag story on Mark Warner. But there was also some chattering about the Outlook spread on No Child Left Behind in the Wash. Post. It was well done including reactions from DC-area principals, an NCLB primer by Jay Mathews, and a map of DC-area schools (pdf) not making “adequate yearly progress” or AYP.
But despite the primer, readers might have been left wondering about these adequate yearly progress targets. That’s understandable, it’s confusing, and they’re not the result of a single calculation. Instead, it’s a multi-step process with opportunities to increase or decrease the level of difficulty at each one. It goes something like this:
First, the state chooses a test to use. This can be a pre-existing test used elsewhere, a custom-designed one based on the state’s standards, or a combination of the two. Obviously, the degree of difficulty is a big issue here.
Second, the state decides what the cut score on the test will be for a student to be “proficient” as well as “basic”, “advanced”, and any other delineations of performance the state wants to have. In other words, how many questions does a student need to answer correctly? For No Child Left Behind the most important category is proficient because that is what the law’s “adequate yearly progress” ratings are based on. There are several methods for determining cut scores. What’s most important to remember about them is that they all rely on professional judgment. There is no revealed source of truth about what a fifth-grader or a high school student needs to know and be able to do. At the risk of oversimplifying too much, the three most common methods are based on using expert judgment from a panel of experts to come up with cut scores, comparing and contrasting how various groups of test takers do on the test, and scaling the questions from easy to hard and determining various delineations for performance along the scale. Again, plenty of chances to increase or reduce the level of difficulty in this process.
But, while newspapers commonly report the percentage of students passing a test, they rarely report on what the cut scores are and when and how they are set. The composition of the professionals involved also matters a lot. Is it just K-12 teachers, or outside experts for instance representatives of higher education, too? Lack of attention to this process is unfortunate because there is plenty of opportunity for mischief and a state with a difficult test and a high cut score, say 40 out of 50, is going to have different results than a state with an easier test or a low cut scores. But, cut scores of half to 2/3 of the questions correct in order to be “proficient” are not at all uncommon. All this is public information or can be obtained through a FOIA. And it’s all extremely relevant to all this.
Finally, the state uses the test and the cut scores for its own accountability system and now for No Child Left Behind. What NCLB basically requires is for state to set escalating targets for how many students at a school must be proficient on the state test in order for that school to make adequate yearly progress under the federal law. This is where there is a lot of confusion. Over time the No Child Left Behind doesn’t make the tests harder, in other words it doesn’t require states to change the cut scores, rather it raises the percentage of students at a school who must pass the test. It’s worth noting that this percentage of students who must pass for a school to make AYP is still often relatively low, you can see state plans and targets here. And, in addition to looking at how many students overall pass the test, No Child also holds schools accountable for how well minority and low-income students do as well. It’s this latter requirement that is causing a lot of the angst about NCLB as schools that were previously considered to be excellent schools don’t make AYP because poor youngsters or low-income youngsters are lagging far behind. Good AYP primer here (pdf).
So, while the Post package on Sunday was excellent and nice attention to the issue, a chart clearly showing readers cut scores on these tests, the percentage of students at a school who need to pass the test in order for the school to make adequate yearly progress, and perhaps some professional evaluation of the state standards (divergent groups like the Fordham Foundation and the AFT periodically produce these and groups like Achieve also study standards and tests) as well as any other relevant context might have helped give readers a fuller picture of what’s happening and why.
That’s because the mix among these variables matters a lot. Difficulty of the test, what the cut scores are, and what the state’s adequate yearly progress targets are all bear on the final percentage of schools making or not making AYP.