Wednesday, November 22, 2006
Sayonara
This is it for me--thanks much to Andy for giving me a few days to write, and to the many of you who over e-mail or on your own blogs took the time to argue or occassionally even agree with what I said here. This blog has a very wide and thoughtful audience -- credit both to Andy and to those out there in eduland. For those loonies who want more from me, I blog (though not daily) on New Vision's blog Foresight, and sometimes on Tpmcafe -- I'll see you in the comments. Happy Turkey Day!
From the department of you can't make this up: ASCD is doing a "poll" asking: "How has NCLB negatively affected your school or district?" Seriously.
Next week's question: Do you still beat your wife?
Update: ASCD responds that they ran a poll looking at the positive side earlier this year. Just didn't find any!
My biggest concern as we look towards the reauthorization of NCLB is that most of the existing policy talk is about fixing the problems with the law rather than moving the discussion forward on how to better educate kids. Changing the formula or the grids on AYP or expanding the use of growth models may make life more reasonable for teachers or principals (which may build greater support for the law outside of the “Washington consensus”) but it does not go to the fundamental problem of improving student learning. In a sense, to have a reauthorization that is about refining and fine-tuning an existing law is only to be expected after the tidal wave that was NCLB. But progress on achievement has been slow, and this is probably going to be the last reauthorization before the deadline comes due in 2014. If NCLB is going to work at all, this needs to be a moment for fresh thinking in terms of how best to build the schools that will achieve those goals.
What might that look like, wise guy? I have some ideas, but I don’t know yet, and I hope some of you will be inspired to help figure it out. (Shameless plug alert.) New Vision is looking to hire two people to do some work on an educational “Vision project” due out next September. Depending on what happens between now and then, and what we come up with, the project could either be geared to reauthorization or to the policy window that comes with the 2008 election. You’d brainstorm with us, and write a paper that would become part of the project. We have had some success in the past in influencing debate—one of our briefs was introduced as a bill by Senator Barack Obama—so this is a chance to get out of your cubicle and influence policy. Think tankers, independent researchers and policy experts, graduate students, post-docs, and professors urged to apply. About three hours a week (enough to yield one paper), January to September, lump sum payment commensurate with experience. E-mail resume and short note to me at jmehta@newvisioninstitute.org if you are interested.
-- Posted by Guestblogger Jal Mehta
It may be too soon to write the obituary, but it is hard not to notice that the action around higher education accountability is getting weaker and weaker as the months pass. In February, higher education commission chair Charles Miller said, "What is clearly lacking is a nationwide system for comparative performance purposes, using standard formats," raising fears of an NCLB-like system for higher education. The final report released in September warned against a one-size fits-all approach of standardized testing, urging colleges and universities to collect their own data on student learning, and create a national database that would allow students more comparative information across colleges and universities. Finding little support for the database idea in Congress, Spellings is now set to push the outputs agenda with accreditation agencies, perhaps the weakest potential tool of external reform.
What happened? I think the explanation is more institutional than political. Even before the Congress switched over to the Democrats, there was limited support for federal accountability measures as applied to higher education. As I've written in the past, the key factors which protect higher education are enduring: their generally strong reputation, their much greater degree of professionalization than K-12 education (longer training, body of knowledge that creates deference to expertise, higher status and pay), and the fact that they can plausibly argue that their success is due in part to the combination of decentralized professional autonomy and market competition. In my dissertation I discuss other reports in the past seeking to bring accountability to higher ed, most notably one released in 1984 modeled after A Nation at Risk, with similar fanfare at the time, and little effect after the fact.
The irony of all of this, of course, is that higher education is professionalized around research, but the professionalization of teaching is non-existent. Not only are all of the incentives set around research--from completing a Ph.D. to gaining tenure to attaining greater status, prestige and pay--but research also undergoes a professional process which ensures the improvement of practice. Writing is shared with colleagues, revised, and goes through peer review, which assures that the collective eyes of the profession work gradually towards better scholarship as defined by the disciplines (which have their own limitations, but we will leave that question for another day). Teaching at research universities has none of these things: it is largely not rewarded materially or for tenure, it does not clearly result in greater status either for individual teachers or universities as a whole, and it does not undergo any kind of collective process by which it is refined over time. As a result, some of the "best" universities get the worst marks from students in terms of engaging students in the classrooms: among the lowest scorers in "bringing material to life" according to a Princeton review survey quoted in the NY Review of books were UCLA, Texas, Michigan and Harvard.
I largely agree with Sara's take on the debate at present about teaching to the test. The problem now lies largely in the incentives for a race to the bottom in state tests, and in the poor multiple choice tests that states are using (to be fair, in part b/c of the expense of creating so many tests).
But what is lost in this debate is why we wanted to move to a standards-based system in the first place. If you go back and look at the movement to create standards-based reform in the early 1990s, you will see two critical assumptions that have gotten lost in the intervening years: 1) That the purpose of setting standards was to move away from the American view of tests (like the SATs) whose purpose was to sort students on fixed notions of ability, and to move instead to challenging criterion referenced tests, which were more sensitive to effort, and set an absolute bar that both students and teachers could work towards. 2) That the tests themselves would reflect really higher-order thinking skills. People like Marc Tucker of the National Center on Education and the Economy who had gone to look at the European models we were intending to emulate came back with stories that the best end of high school exams looked like college bluebook exams--e.g., essay questions that asked student to draw parallels between the French and American revolutions. These kind of exams, Tucker argued, were sensitive to effort: you knew you needed to know about the American and French revolutions to do well on them. But they were not sensitive to "test prep," in that there was no way to game it -- you had to know both revolutions, you had to be able to write, and you had to be able to think at a level that allowed you to see connections between the two. That's a standards-based system that I think a large majority of us could support.
How did we get from here to there? The idea of setting challenging standards, in terms of higher expectations, was widely embraced by all but the fiercest local control advocates; teachers supported this movement through the activites of their subject matter associations, most notably the NCTM in 1989. But policymakers became impatient with the pace of change, and the institutional conservatism of a school system that was not responding or even implementing state and particularly federal reforms. The result, somewhat understandably, was an emphasis on the accountability functions of testing as a mechanism to force improvement, which in turn required annual testing and high stakes, which has led to off the shelf tests, states lowering standards, and a fair amount of teachers teaching to the test in the narrowest sense. There is plenty of blame to go around here, but it is revealing to see the distance we've traveled from our initial hopes to our current reality.
Where is Margaret Spellings?
Tonight, she is on Celebrity Jeopardy. The press notice from the Department of Ed says that she is believed to be the first cabinet member to go on the show. No word on how she did, although the fact that they are sending out a press release suggests it didn't go too badly. Fortunately, she is not up against this guy, who would have to the favorite among an interesting list of celebs competing this year.
AFTie One-L raises the issue of capacity in response to the lack of gap closing under NCLB. I think this is clearly right in one sense, namely that our theory of change is long on motivational sticks and much weaker on creating the means to improve practice. But what exactly does it mean to increase capacity, and how might it be done?
I define capacity as the ability of an organization to accomplish an objective, in this case improving student learning. I can see four ways to boost capacity, which can be laid out in the following two by two typology:
| | External | Internal |
| Individual | Training/prof. dev. | Get better people |
| Collective | External assistance | Learning organizations |
Ed School types tend to favor one of the two strategies in the “external” column. In this view, teachers’ skills can be boosted either by revamped training at teacher preparation institutions, or through ongoing professional development that will provide them with the most up to date best practices. This external-individual strategy can be supplemented by an external-collective strategy, whereby outside experts visit failing schools and consult on how to improve practice. It should be noted that teachers themselves are somewhat skeptical of both of these strategies, often seeing one-off professional development workshops as of very little use, and outside “experts” as lacking the on the ground knowledge to deal with their particular problems.
A third view we might label “individual-internal.” It challenges the importance of teacher preparation institutions and professional development, and takes the view that the best way to improve practice is to find better teachers. On this view, teachers are at least as much born as made, and the objective is to recruit better people into teaching through higher or differentiated pay, reduced barriers to licensing, and more non-traditional routes into teaching. There are arguments and evidence for and against this view, but however these are resolved the “better people” strategy is unlikely to help us in the short to medium run, because, as our late Defense Secretary might have said, you go to war with the teaching force you have.
There is a fourth alternative, which has been rarely utilized in education but is gaining increasing attention in other fields and in the organizational literature. We might think of these as “internal-collective” strategies, or to use their catchier names, creating “learning organizations” or “problem-solving organizations.” The idea here is that problems are too frequent and context-specific for one-time general outside expertise to be of much use, and thus that an organization’s most important resource is its ability to identify and address its own problems. It also assumes that problems are better solved by organizations than by individuals, as different kinds of local expertise are brought to bear. The 1990s emphasis on “Total Quality Management” was one example of such a strategy, as a variety of businesses and non-profits sought to use internal data to identify problems and work to address them. Some schools are using frequent internal testing and assessment in a similar fashion today. I was recently at a conference where a national health care expert raved about the internal use of data as the key to the remarkable performance of the Mayo Clinic. But being a problem-solving organization also doesn’t have to take such a business or data-heavy tint – it could be as simple as teachers and a principal getting together to discuss why a group of 3rd graders aren’t learning math, and taking steps to address it.
These strategies are not mutually exclusive, and they could be mixed and matched: external experts could help facilitate problem-solving or offer best practice models from other schools; finding better people would certainly increase schools’ abilities to be learning organizations. Boosting capacity is the key to making progress in improving student learning; we should use all the strategies at our disposal.
Said I might pop in, so don't blame Mehta for this one...Reading my hometown paper this AM I noticed that WaPo's usually solid Ian Shapira turned in something of a lemon on test scores and student grades. It's a complicated issue and the story doesn't do it justice. Of course there are going to be students who don't test well, that's a pretty minor issue that garners headlines but is dealt with relatively easily in public policy through a meaningful appeals system that takes into account multiple measures. Hardly front page news.
Where Shapira falls down is by not engaging on the larger question about whether teacher grades are the best indicator of student learning. He's got anecdotes, but on this one there is also data. Grades are surely one important indicator, but the best or only one? The chart below shows data from the Prospects evaluation of Title I (pdf). What it shows is the wide-variation in achievement among students with the same grades between high and low-poverty schools in 1991. Basically, students getting an "A" in 7th-grade math and reading were only at the 35th and 36th percentile on standardized tests while "A" students in low-poverty schools were at the 87th and 81st. Some of those As aren't so meaningful.* The gaps do decline as achievement declines, but the trend is hard to miss.
Source: Prospects: Final Report on Student Incomes. ABT Associates, 1997.
All industries have some sort of external accountability. For some it's government, for others the relentless discipline of the marketplace, and for some a hybrid, which is basically where we're driving toward in public education. The question of grades versus test scores really boils down to that question, what sort of external benchmarks do we want in a public system like ours? Right now, standardized tests, which help provide information and leverage to attack the inequities you see in the chart above are the worst way to do that, except for all the others.
*By the way, this is not some radical revelation, it's something just about anyone who has taught at the collegiate level will tell you. Shapira probably could have landed some devastating blind quotes from VA and MD college profs about the differences in preparation in those states, irrespective of high school grades.
Rorschach Redux -- More True Than I Knew
As predicted, this morning's NYT story has only served to reinforce previously held positions: Joanne Jacobs sensibly calls for richer curriculum, after school tutoring, and diagnostic testing, D-Ed Reckoning somehow criticizes me and the NYT for assuming that racial gaps are not immutably given by nature, and, in a rare moment of detente between AFTie One-L and Guestuwonk (Andy, don't fire me!), Michele "pimps" Richard Elmore's arguments for greater capacity. In an interesting twist, I'm at the GSE in part because I wanted to work with Elmore, so I'm sympathetic to Michele's argument, but saying that capacity matters is not the same as knowing how to create it. More on this tomorrow.
In honor of the death of Milton Friedman, I wanted to offer an argument for choice, albeit one that he would not have made. The discussion around choice usually centers on the equity/efficiency trade-off, with conservatives lauding the magic of the invisible hand, and liberals worried about the potential for educational markets to increase inequality. I'm in the middle (for public choice and charters, not for vouchers), and I don't want to have that discussion today.
But there is another side to the choice debate that is under-appreciated, which is the way that choice can afford greater school-level autonomy by providing an accountability metric that is less centered on tests and more on parents. If teachers' main complaint is that they are over-regulated from above, then choice can provide an opportunity to establish an educational identity at the school level, as teachers are accountable to parents rather than the state as a whole. It also provides for greater educational pluralism, which should be attractive to students, parents and teachers alike. This is the genius of charters, and it is frustrating that it has not been more widely embraced by exactly the people--teachers, principals, and the unions that represent them--who could benefit from the increased autonomy and discretion it could potentially afford. The debate around choice should not only be one of left vs. right, equity vs. markets, but also about centralized control and bureaucracy vs. professional discretion and autonomy. Call this approach "choice + professionalism," and consider American higher education as the foremost example of how it might work. It just might be the antidote to the prevailing emphasis on top-down accountablity -- too bad that no one other than Ted Sizer will stand up for it.
What to make of this morning's Times lead story on the lack of gap closing under NCLB? The obvious first question is whether the claim itself is supported. The NAEP graphs presented suggest some progress since 92 in overall achievement, but little progress in gap closing. The story also effectively gathers together a number of recent reports, including some from those like Fordham who are pro-NCLB but are empirically honest in reporting a lack of progress in gap closing. Given that there has not been a RAND-style evaluation of NCLB, for now, I think this story and the reports it references will be accepted by partisans on both sides of the debate as evidence that there has been little overall progress in closing achievement gaps.
If so, then what to make of it? As the day goes on, I predict it will serve as a kind of Rorschach test for NCLB: critics will claim vindication, proponents will accuse critics of "making excuses" and argue for the need for redoubled efforts, and policy wonks of all kinds will use it as an opportunity to pimp their preferred education strategies (Ed Trust has already done so in the NYT story). If there are interesting reactions today I will post them here. (Eduland can also e-mail me at jmehta@newvisioninstitute.org.)
I don't claim to be immune from this dynamic, but with that caveat, here are two quick thoughts.
First, from a research perspective, it is not surprising news. The state programs that served as a basis for NCLB in North Carolina and Texas showed some evidence of overall progress in raising NAEP scores (although there were conflicting analyses on Texas, as you may remember), but they did not show progress in reducing racial achievement gaps on NAEP. See the Thernstroms summary of the research in No Excuses -- even those generally partial to the approach had to acknowledge that the evidence just wasn't there. From this perspective, these reports are progress in that they bring a dose of realism to a discussion that has seen too much in the way of grand utopian pronouncements -- i.e., all children proficient by 2014 -- and too little in terms of serious policy talk about the sheer difficulty of closing achievement gaps and the various strategies that would need to be employed to make even incremental progress.
Second, from a policy perspective, I think (and here is that Rorschach test) it reveals the weakness of external accountability as a lever to accomplish what is one of the most difficult of our social policy objectives -- breaking the link between family poverty and school outcomes. In a sense, setting standards and measuring progress through assessments are the low-hanging fruit -- what they leave undone is the much more difficult task of actually helping kids do tomorrow what they couldn't do today. These questions--of which accountability systems are one part but a relatively small part--is where the policy dialogue needs to go. Let's hope today's story sounds the bell and starts that conversation.
What a day to guestuwonk -- Dillon leads the NYT with a piece on the lack of progress in closing gaps under NCLB. You don't know me yet, but trust me, this is a must read, even before you catch up on the weekend TomKat nuptials (ugh!).
My thoughts shortly, but first a word of intro. My name is Jal Mehta. I am co-director of New Vision: an institute for policy and progress, a small but growing think tank that connects university-based academics with Washington policymakers around domestic social and economic policy. I also have recently completed a dissertation tracing the rise of the standards and accountability movement. I just graduated from a Ph.D. program in Sociology and Social Policy at Harvard, and now am teaching and post-docing there at the Graduate School of Education. While I don't always agree with Eduwonk, I have learned as much from this site as from any other single source, and hope to repay a bit of that over the next few days.