TAP That?

Mathematica just released a study that did not find learning gains or improvements in teacher retention as a result of the Teacher Advancement Program (TAP)* in Chicago.   The initiative was funded via the federal Teacher Incentive Fund. There is plenty of bloggy noise about this so just five quick observations mostly about context:

1) This is just one study (and a small one, eight schools each year, that also had some attrition as schools were closed) of TAP, so the rush to judgment is lacking in context.  Overall the evidence on TAP is mixed but encouraging and the big challenge has been controlling for selection effects (teachers choose to participate), which Mathematica** sought to do through their design here.   TAP is doing a second study to see what’s going on in Chicago, stay tuned for that.  Hopefully some lessons for Chicago but also more generally.

2) TAP is a lot more than merit pay, it’s about creating a career ladder for teachers that creates opportunities for new responsibilities and opportunities that do not mean a teacher has to leave classroom teaching.  That’s why it’s supported by organizations like the American Federation of Teachers.   So the rhetoric about how this is some sort of verdict on merit pay is uninformed.

3) This is just one Teacher Incentive Fund location so everyone should slow down with the rush to a verdict on that initiative.   And anyone who thinks the field knows how to do any sort of incentive pay has not been paying attention.   What’s known is that the current system of allocating about 4/5ths of the $600-plus billion we spend annually on K-12 schools is not linked with outcomes and creates some perverse incentives.  How to fix that is going to require a lot of innovation, experimentation, and failure.

4) As with Reading First and other initiatives, the apparent glee and celebration when things don’t work is one of the most depressing aspects of education policy today.   Something went wrong with this initiative in Chicago, to their credit TAP wants to figure out what it is rather than attack the study, make excuses, etc…and yet what you get from the bleachers is some cheering and crowing.  Really?

5) If this was not Chicago would it be getting as much attention?

*Bellwether works with TAP and its parent organization the National Institute for Effective Teaching.

**Mathematica continues to come up with creative evaluation strategies on issues (eg. teacher effectiveness) that previously were assumed to be impervious to this sort of evaluation.

11 thoughts on “TAP That?

  1. edlharris

    From the trenches:

    1. Pay for Performance

    When Pay for Performance was introduced to Fairfax County more than twenty years ago, I was in a pilot school. Teachers participated because it eventually meant more money. We embraced the hope that what we did in the classroom would be appreciated and rewarded.

    It was a fiasco—albeit one that lasted several years before the Fairfax School Board voted it out during the budget crisis of the early 90s. The reasons were simple, and we should have anticipated them: principals and vice principals don’t always make decisions based on classroom performance, and some don’t recognize effective teaching when they see it.

    There were principals who championed the quiet and obedient class; they were not impressed by group interaction and lively discussions. Others only rewarded teachers who had student-centered classrooms, and didn’t appreciate marvelous lecture-format ones.

    Principals often became defensive about their judgments and wouldn’t let teachers participate in the process. In that first pilot year, I was new and didn’t realize my school had decided not to rate any new teacher highly. I was given a “competent” rating rather than an “effective” one. Even though money was not yet tied to this decision, I appealed, won my case, and later underwent a much-improved process at another school and was awarded “merit pay.”

    The difference between the first process and the second was the quality of the principal. Ten years later I became the first “National Board Certified” high school English teacher in Fairfax County—a great honor, yet one not reliant on variable principals.

    So before you introduce Pay for Performance to schools, you have to figure out how to guarantee principals will reward teachers for the right reasons.

    Read more at the Washington Examiner: http://www.washingtonexaminer.com/opinion/columns/EricaJacobs/An_educators_letter_to_the_new_president_.html#ixzz0putERA00

    And a curent trench:

    I participated in the PG pay for performance program called FIRST last year. I am not doing it this year for several reasons:

    1. I teach newcomer ESOL, the lowest of the low English speakers. Most of my students are exempt from the MSA reading. Therefore, I did not get money in the category of “classroom test scores.” However, my students take the LAS which is the test for measuring how much English a student knows. That was not used for evaluation.

    2. Emphasis is made on teacher “growth.” Because I scored so highly in the first evaluation I had very little room to “grow” and so did not get the money in that category.

    3. We had to participate in mandatory professional development. I have 2 Master’s degrees and am certified in ESOL, SpEd and social studies. These professional developments were a waste of time. Most of them focused on the evaluation process itself as if we could not read the book we were given about it.

    4. Even though I have taught for many years in other counties (and countries), I was new to PG so as a “new teacher” I was not allowed to pick the 4 areas I wanted to be evaluated in (where I actually wanted to try to grow like communicating with parents).

    Overall, what I learned was how to game the system if this excuse for an evaluation ever is mandated. Sit at my desk and let the kids do a worksheet for the first evaluation then teach like I regularly do the second.

    Finally, my biggest beef with tying teacher evaluations to standardized assessments is that they measure a snapshot. If they measured how much individual students grew I might be more interested.

    With the current system, I can bring a student from the 3rd grade to the 6th grade reading level, but she still isn’t proficient if she is in 8th grade. All the teacher = test programs out there say that I have failed. What do you believe?

    Posted by: Bramblerose | June 2, 2010 8:05 PM

    http://voices.washingtonpost.com/answer-sheet/research/analysis-of-chicagos-teacher-p.html#comments

  2. JSP

    I remain amazed that data and evidence that support someone’s preexisting viewpoints are good and valid and impressive and that evidence and data that raise questions about those viewpoints are limited and incomplete and questionable. Shouldn’t we just question it all?

  3. Bob Calder

    The failure of pay incentives is now at the NTY best-seller level. That is to say, the reading level of many education reformers. Here’s hoping they notice it next to their horoscopes.

  4. JSP

    To quote a song that dates me–“A man sees what he wants to see and disregards the rest.” If all you are seeking in research is validation of your opinions, you generally find it whether it’s there or not. Support for pay incentives is the reform du jour–someone somewhere will say something that validates the pro-incentive stance. And vice-versa.

  5. Linda/RetiredTeacher

    People who advocate merit pay just don’t understand the mindset of teachers. When I applied for my first teaching job, I never even asked about the salary before signing the contract. During my tenure, I rarely knew what my salary was and only during the period that my sons were at college did I seek extra income. After college tuitions and mortgages were paid, I found myself with a lot of extra money, so what did I do? Yes, I spent it on “my kids,” just as many of my colleagues did. I started out spending 50 cents per child on books I presented them at Christmas but ended up paying for all kinds of extras for them (concerts, books, CDs, VCR, PTA memberships etc.) Now that I’m retired, I’m surprised at how much extra money I have. I still have an old, battered car, but the cruises around the Mediterranean are quite nice.

    We DO need workers who are motivated mainly by money; otherwise it would be impossible to get people to do those jobs (try being a corporate attorney and suing some big hotel for losing an expensive dress; that’s what my son had to do before he quit). But teachers are generally not in this group. The job is extremely rewarding in the intrinsic sense, and many teachers wake up each day with a feeling of excitement. The job satisfaction is usually very, very high (but not at the present time because the recession has targeted public servants). I LOVED being a teacher and money had very little to do with it.

    Of course, teaches would appreciate higher salaries like everyone else, but what they really want are good working conditions and professional autonomy. My guess is that when faced with the next big teacher shortage, teachers will become fully professional and won’t have to grovel for “merit pay.” Just the words alone are so demeaning.

  6. edlharris

    I asked my principal on the plans to base teacher evaluation on test scores and merit pay, and she remarked that she did not get into teaching for bonus pay and, as me, wonders who does.
    There will be those professional “reformers” who will try to turn it around to the question “Wouldn’t you want extra pay for getting test scores up?”

    Reminds me of theFrank Zappa album from 1967, We’re Only In It For The Money

  7. Carl Jepsen

    Linda is right. Because teachers do not care much about money and do the job for all those other reasons schools should stop raising salaries and put the funds into better support for students. We could still get great teachers like Linda and provide more services for students.

  8. edlharris

    More from Valerie Strauss:

    Study: N.Y. teacher performance pay program flops

    Just the other day we heard that a program in Chicago that attempted to link teacher pay with student standardized test scores wasn’t working, at least not in the first two years.

    A 2009 analysis of a major program in Texas that also linked teacher pay to student achievement gains on tests showed no evidence of success. The multi-year Texas Educator Excellence Grant involved teachers at about 1,000 campuses, with a total of more than 140,000 students in lower-income neighborhoods. It was discontinued because of “design problems.”

    Now, a paper prepared by two Columbia University researchers for a recent education conference at Harvard University said that the New York City Bonus Program, which attempts to raise student achievement by paying teachers for it, was — you guessed — also unsuccessful.

    http://voices.washingtonpost.com/answer-sheet/no-child-left-behind/ny-teacher-performance-pay-pro.html

    “And the walls, come tumblin’ down”
    -John Cougar Mellencamp

  9. Jesse

    I have mixed feelings on merit pay but am troubled by comments like Linda’s that “People who advocate merit pay just don’t understand the mindset of teachers.”

    Of course it is true that many teachers care more about things like job stability than pay. But that’s really just begging the question. Obviously, the system we have set up now is attractive to people who don’t place the highest emphasis on money. (Guaranteed middle-class salary with good job protections but few economic incentives to improve). And maybe it just so happens that the people who would make great teachers are all people who don’t care that much about money. But maybe it’s also true that plenty of people who would make great teachers would pursue the profession if the incentive structure were different (rather than becoming bankers, lawyers, etc.)

    I actually am skeptical that changing pay structure (at least by itself) would make a huge difference. But I think our analysis needs to take into account not just the teaching corps we have, but the teaching corps we want to have.

  10. Linda/RetiredTeacher

    Jesse, I do agree with you. If teachers were offered higher salaries, we might attract talented people who now go into other professions. I can think of several young people who wanted to be teachers but said something to the effect “There’s not enough money in it.”

    To me, “merit pay” and salaries are not the same. As I said in my earlier post, “teachers would appreciate higher salaries like everyone else” but to many of us, “merit pay” is demeaning.

Leave a Reply

Your email address will not be published. Required fields are marked *