Adding Value?

Value-add measures for teachers are complicated.  Two takes fresh out today. Shorter versions:

From the teachers’ union-funded EPI (pdf):  We don’t want to say don’t use value-add, but use it only a very wee little bit!    We’re more bullish on peer review, but ignore the evidence there please!

From U of W’s Dan Goldhaber: Use it responsibility and beware of the limitations.  Why on earth is the LAT doing what it’s doing?

Goldhaber’s take is sensible.  EPI is right that the fetishising of 51 percent of evaluation from value-add isn’t wise (and it’s also not practical as a comprehensive tool).  And they sensibly call for a federal push to innovate with various evaluation models.  But isn’t that what’s happening under Race to the Top and related initiatives?* And since we really don’t know what works here yet there is nothing wrong with states innovating with heavy value-add models (meaning weighted at 50 percent or more), too, is there?  Besides, it’s worth nothing that models that use value-add for much less than 50 get attacked, too.

In fact, I’d argue the underlying issue is less the specifications of any value-add model, or any evaluation system that uses value-add, and more the underlying issue of outcome-based evaluation.  Most of the debate today is camouflage for that.

*Take for instance the DC IMPACT model, which is a pretty good tool.

59 Responses to “Adding Value?”

  1. Marktropolis Says:

    So, EPI is “teachers union-funded” (clearly designed to have the reader assume there’s some conflict of interest there, and Dan Goldhaber is just, “Goldhaber”? Should we start calling Andy the “billionaire boys’s club-funded Rotherman?

    I know there was thread a year or so ago about conflicts and disclosures. And while at this point, I don’t expect Andy to restate his disclosures on every post, but at least be consistent in how you do it within a post. In this one, your biases are readily apparent. Clearly EPI can’t do any decent research or analysis, since they’re funded by those evil teachers unions. But Goldhaber, now that’s another story. He’s clean.

  2. Linda/RetiredTeacher Says:

    For those of you who want to know who is truly dedicated to the education of our nation’s children without a conflict of interest, here is a clue:

    Who spends his day teaching children without the expectation of riches, recognition, or even help from other adults?

    Yes, teachers and parents! A big thank you to all the people who actually educate our young citizens! Your reward will come daily from the children and young adults who express gratitude and love for you. And there are probably few joys in life that exceed the thrill of watching and facilitating the unfolding of a child’s gifts.

    Teachers, you are our nation’s heroes! Best of luck to you as you begin a new year!

  3. TFT Says:

    Thanks Linda.

    But please, we don’t need any more support of the “teacher as altruist” notion.

    We should be compensated like professionals and not expected to work out of love.

    Sure, it’s nice to get love from the kids. It’s also nice to be able to pay the mortgage and send our kids to college, like the policy makers do.

  4. john thompson Says:

    Kate Walsh took the same approach on NPR today. Reformers should be rejecting their own extremists from the Times to Rhee to Kevin Carey. IMPACT for instance would be a pretty good tool but not in the hands of Rhee’s administration. Not when she’s started off the year telling principals to “go hard or go home.” in other words, fire people, ask questions later.

    All evaluations are a political process, just as all testing regimes come from a political process. That doesn’t mean that all must be rejected. But it does mean that there must be checks and balances, and one step for an appropriate political climate is to reject extremists who are so convinced they are right that they reject checks and balances.

  5. Linda/RetiredTeacher Says:

    TFT:

    In my state (CA) teachers receive decent salaries. It’s nice to wish for more, but I just don’t think public servants will ever make that much. Traditionally these workers want to serve the public and are satisfied with modest salaries, decent pensions and job security. I think I mentioned that my son just gave up his $160,000 salary so he could serve in public office, which pays $30,000. Just like Mom! (Can’t you just picture the disdain on the faces of some of these “reformers?” )

    That said, I think there’s going to be a huge teacher shortage when all the baby boomers have retired. Until the 1970s women had few choices so many chose teaching or nursing. Because those “captive” workers are no longer there, I predict that urban districts will have a terrible time recruiting and retaining qualified teachers. Because of this, teachers might have the opportunity to make more money than they do now. They certainly deserve it. You are right about that.

  6. Chris Smyr Says:

    John Thompson:

    So IMPACT is a good evaluation tool except when Rhee uses it? Why?

  7. Marktropolis Says:

    TFT: I’ll take a teacher as altruist over teacher as capitalist any day.

    Yes, teachers should be paid a decent salary (avoiding for the moment what defines decent), but I’d rather have a person motivated by the love of teaching, than someone who’s just looking to make a buck. We already have jobs for folks like that – on Wall Street.

  8. Joe Hill Says:

    What do you say about a so-called “reform” movement that does a touchdown victory dance for any scrap of data that seems to support destabilizing public schools and undermining the autonomy, expertise, and professional status of teachers . . . but each time that scrap of data is exposed as faulty and insufficient, shoots the messenger?

    It’s in this dance that the motives of this “reform” movement are revealed. Their personal theology will simply not let them accept a sector of society that is not linked to the market of late stage capitalism. Their attacks come from an anti-union, anti-children, and, in the end, an anti-intellectual position.

    The future will not forgive us if we hand public education over to the greedy little hustlers who already control far too much of America. Fight back!

  9. john thompson Says:

    Chris,

    All evaluations are a political process, just as all testing regimes come from a political process. That doesn’t mean that all must be rejected. But it does mean that there must be checks and balances, and one step for an appropriate political climate is to reject extremists who are so convinced they are right that they reject checks and balances.

    did you read Bill turque today about the apparent political retaliation at Wilson in DC, and Rhee’s justification for transferring him. IMPACT like VAMs like most everything are tools. Its how you use them. When you start off the year telling principals to go hard or go home, how do you think its supposed to be interpreted? In other words, fire people first, ask questions later.

    I don’t know how old you are or how many jobs you’ve had, but real world, a wink is as good as a nod to a blind horse.

  10. Chris Smyr Says:

    John Thompson,

    You copied and pasted the same paragraph again, but I’m still wondering why– give an example of how Rhee et al. reject checks and balances that are implicit in the IMPACT model.

    You also linked to an article that — surprise surprise — doesn’t have much to do with what I asked you. The article specifies that Rhee actually didn’t give the justification for transferring him, and the reason for his transfer very likely could be due to exactly what the article explains in the first 2 paragraphs: standoffs between him and the school admin. I’ve worked enough jobs to know not to pick fights with my employers and stir up dissent among staff if I want to stay employed.

    Rhee didn’t use IMPACT to fire Wilson. So my question remains: how would Rhee use IMPACT, a good evaluation tool, for devious purposes? Hinting to principals that they should fire people (which is a bit of an overstatement) wouldn’t change the way IMPACT could be utilized. If the argument is that Rhee is forcing principals to be harsh in their evaluations of certain teachers,

    1) it is quite ironic to hear that evaluations, instead of test scores, are now the troublesome element in the DCPS accountability system,
    and

    2) teachers ought to be clamoring for some reprieve from these unfair evaluations and should contact their union to push for other measures of their effectiveness, and

    3) the evidence from their dismissals should be collected and given to Bill Turque so he can write another damning article about the evil afoot in DCPS.

  11. Billy Bob Says:

    Sheesh Chris. John;s explanation was pretty straight forward. Rhee is using the IMPACT model (it could be any model as far as she is concerned) to push hard on firing teachers. She is using the tool to advance a political agenda rather than waiting for evidence and then making a decision.

  12. Chris Smyr Says:

    Billy Bob:

    No, it wasn’t straight-forward at all. HOW is she pushing hard on firing teachers? He needs to EXPLAIN how she’s doing so, other than with these empty assertions. How is IMPACT being circumvented to fire teachers that she wants fired?

  13. Marktropolis Says:

    As Sherman Dorn points out (in his ever eloquent way, and linking to much of the history), part of how Rhee is “doing so” is by making VAM measurements account for 50% of a teacher’s evaluation. See http://shermandorn.com/wordpress/?p=2384 (sorry, not talented enough to embed the link). Rhee is using an untested measure as an excuse to fire teachers. Similar to what she did about ayear ago when she fired another swath of teachers – and in the press made accusations about gross violations of the law, much of which was successfully challenged and a number of those teachers got their jobs back. This IMPACT system has the appearance of giving Rhee an extra stick with which to “go hard” on her teaching staff.

    In other words, it’s not so much that IMPACT is being circumvented, it’s that IMPACT is having too much of an impact.

  14. Anne Says:

    “And since we really don’t know what works here yet there is nothing wrong with states innovating with heavy value-add models” — Maybe we do not know what works, but we know (if we believe the EPI report and the research it contained, which I personally am inclined to do) what does NOT work when it comes to reliably gauging teacher effectiveness. Value-added models. Why should we innovate using something that researchers in general do not support?

  15. Chris Smyr Says:

    I don’t see why this is so hard to understand.

    John Thompson claimed that, while IMPACT is a good evaluation tool, in the hands of Rhee it is not. IMPACT includes the 50% VAM component REGARDLESS of who is the chancellor of DCPS. The question still remains: how does Rhee utilize a good evaluation tool for devious purposes?

    This is like the third time this month that John has done this, dropping empty assertions in the comment threads here and refusing to substantiate them with evidence or some reasoning.

    Marketropolis:

    You linked to a blog post that didn’t really substantiate the claim that “VAM is an untested measure as an excuse to fire teachers”.

    Since VAM seems to be specifically intended to measure teacher/school effectiveness while attempting to control for student/family background, all studies with it likely test whether or not it is a reliable way to determine teacher/school effectiveness. VAM supplemented with evaluations seems to be one of the better options on a short list of possible teacher evaluation tools.

  16. Marktropolis Says:

    Chris, I’ll let John defend himself. As for me, maybe the specific posting I linked to didn’t “substantiate the claim: directly, but the paper linked (at least twice in the post, http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1461508) I think does. To quote from the abstract of that paper: “The most important practical conclusion is that point-based systems with fixed components and weights cannot capture the dynamic and political benefits of a reciprocal relationship between professional judgment and quantitative student outcome data.”

    Maybe I didn’t do a good job of phrasing things. And the EPI paper posits that “there is not strong evidence to indicate either that the departing teachers would actually be the weakest teachers, or that the departing teachers would be replaced by more effective ones.” Meaning, roughly, there’s no data to indicate that using VAM scores as *the* reason to fire a teacher is reasonable. And further, they make the case that yes, using VAM as one of many components of a system is reasonable, but giving it a weighting of 50% is not. Which goes back to John’s argument, that IMPACT in the hands of Rhee (who pushed to have it account for 50% of teachers’ evaluations) is a dangerous thing – she’s doing something that the research says she shouldn’t do.

    So maybe “untested” was a poor choice of words. Maybe what I should have said was that experts in VAM have stated repeatedly that VAM should not be used as the sole reason for firing anyone. And that’s what Rhee did in at least 20-some cases – as reported in the media and not denied by her or her administration, that many teachers (can’t recall the exact number, and it’s not really important to the argument) were in fact fired for their VAM scores.

    Chris, methinks you may want to spend a little more time reading the research on VAM – what works, what doesn’t work, and in what cases should it be used, or not used. Irrespective of John’s capacity to make an argument, doing a little homework might help.

  17. Marktropolis Says:

    A late addition, Liam Goldrick just posted a nice summary of what’s wrong with VAM (as well as what’s right). http://eduoptimists.blogspot.com/2010/08/adding-value-to-value-added-debate.html

    And a bevy of links if you want to get deeper into this. I will add that i heartily agree with Andy’s first line: VAM *is* a very complicated issue, technically, politically and logistically. And most of that complexity is being ignored in this discussion.

  18. Chris Smyr Says:

    Marktropolis:

    Your first link directs to a theoretical paper discussing the possible application of a Bayesian perspective in the evaluation process, rather than an experimental study on the uses or misuses of VAM (the latter would much better support your argument).

    Even the EPI paper relented that VAM could be useful if it were supplemented by other evidence of effectiveness. The 50% number given is rather flimsy, and the appropriate proportion of VAM to other modes of evaluation seems to be more based on the sentiment of the authors rather than on hard data. And while “experts in VAM have stated repeatedly that VAM should not be used as the sole reason for firing anyone”, it’s important to realize that IMPACT does not use VAM as the sole method of evaluating teachers and schools.

    Also, you’re not letting John defend himself when you say “IMPACT in the hands of Rhee is a dangerous thing”. Your argument is that IMPACT itself is a poor tool for evaluation. You and I can start from there. John’s argument, on the other hand, was that IMPACT would be useful if it weren’t for Rhee trying to use it. Very different viewpoints.

  19. TFT Says:

    Marktropolis

    So it’s either/or with you? Teachers are either altruists or capitalists?

    Linda,

    My teaching experience is in California as well. All I am saying is that the meme you put forth–that teachers get all they need from the love their kids show–is crap. We get a lot from the kids and appreciative parents, but we also get paid. Teaching is hard work, and like all hard work, we should be compensated like other hard workers–with a decent salary.

    Saying teachers do it out of love may be true, but it is our job too–we do it for money.

    Whenever we reduce teachers to altruistic lovers of children, however true that may be, we preclude any chance we have of being considered professionals.

    We do it because we are good at it, hopefully, and like it, and we get paid for it.

  20. Marktropolis Says:

    TFT, no it’s not either or for me. That said, Rhee (a recurring topic on this thread) is from the venture capitalist school of education (TFA, NLNS, etc.) who believe that there’s “nothing wrong with making a buck” in the field of education. No, there isn’t. But there is something wrong when that’s the motivation. Hey, I think teachers should get paid as much as dentists. you’ll notice I didn’t say doctors and lawyers – too many of them actually DID going into their fields because of the money. Of course, there are teachers out there getting paid more than a lot of lawyers and doctors, as many of those doctors and lawyers have chosen to do things like be public defenders, or run community clinics.

    In other words, my point was not to promote up a false dichotomy, rather point to the notion that I don’t want teachers who are there for the pay check (and there are already a lot of those out there – we don’t need any more). I want folks who are committed to a career of being good teachers. Career. Not a couple of years in the jungle, a la TFA.

    So no, it’s not either or, but perhaps you didn’t read the rest of that post.

  21. TFT Says:

    I think you would have a hard time finding teachers who do for a reason other than a buck–except the independently wealthy ones.

    You seem to be creating the false dichotomy.

    You want folks who are committed to a career of being good teachers, as do I. You have to pay for that.

    I think we agree on TFA and Wall Street.

    I am tired of hearing how my low salary is justified because I get other non-monetary compensation–the love of my students. One cannot pay for food with love.

  22. Linda/RetiredTeacher Says:

    TFT:

    I just reread my posts. I never said that teachers don’t expect to get paid; of course they do – it’s their job! My point, not well expressed apparently, is that teachers don’t expect “riches,” or very large salaries. People who want to make a lot of money just don’t go into teaching at any level. My husband was a university professor who doubled his salary when he went into industry. I believe it’s this way all over the world.

    In my opinion, the satisfaction that teachers get from the job itself is a huge reward but I do agree that teachers need to advocate for higher salaries and professional autonomy. What bothered me much more than the modest salary was the fact that administration tried to take away the decision-making abilities of teachers after NCLB. It was a good time to retire and I’m glad I was able to. For most of my career I made almost all decisions about the students in my class.

    Personally I think the profession is having so many problems now because it’s still predominantly a female occupation and women make easy targets. Many do not fight back. We need more people like you, TFT! Go for it!

  23. TFT Says:

    I appreciate your response, Linda.

    My friend calls me an “unreasonable man” in the George Bernard Shaw sense. I suppose he is right.

  24. Chris Smyr Says:

    Marktropolis:

    “Rhee (a recurring topic on this thread) is from the venture capitalist school of education (TFA, NLNS, etc.) who believe that there’s “nothing wrong with making a buck” in the field of education. ”

    Can you elaborate?

    “I want folks who are committed to a career of being good teachers. Career. Not a couple of years in the jungle, a la TFA.”

    This statement falsely implies that TFA teachers are less committed to a career in teaching than are other new teachers. Any new teacher is not committed to a lifetime of teaching, just as any young person entering a particular job field right now is not committed to sticking with it for their entire working lives. Anecdotal evidence, but most of my TFA colleagues were uncertain as to whether they would continue teaching after two years or not, and wanted to discover if their passions lie in staying in the classroom or not. Do you have evidence suggesting otherwise?

  25. TFT Says:

    Chris, isn’t the deal with TFA that one commit to 2 years if one is to avail themselves of the TFA program? Because if so, it would seem Mark’s statement is not only true, but codified.

    And are you sure that your definitive statement, “any young person entering a particular job field right now is not committed to sticking with it for their entire working lives” is true? Do you know all the young people entering particular job fields right now?

    I know quite a few young people who have gone into particular fields with a plan to do it forever.

  26. Chris Smyr Says:

    TFT:

    The two year commitment is a minimum, not a maximum, so no, you’ve not found something that bolsters his claim.

    And I don’t know what all young people are thinking as they embark on their careers, but I guarantee that no one is signing a contract that holds them to any one career until retirement, even if they assume they’ll stay forever. I also know that labor statistics suggest that we have a mobile workforce, and that college career centers tend to impress on students that some flexibility is good to have:

    http://www.bls.gov/news.release/pdf/nlsoy.pdf
    http://www.bls.gov/opub/ooq/2009/summer/art02.pdf
    https://career.berkeley.edu/article/070119a-sbd.stm

    Furthermore, that you “know quite a few young people who have gone into particular fields with a plan to do it forever” is not evidence that, more often than not, non-TFAers fit this description more readily than TFAers.

  27. TFT Says:

    Thanks for the confirmation. TFA is a two year commitment. And most leave the classroom after 2 years.

    That we have a mobile workforce is true, but that doesn’t make it good.

    The fact that TFAers leave after two years is only evidence that Mark was right.

    That I know some who plan to stay is evidence–that you support–that your claim of knowing their plans is wrong.

    Your other stuff is irrelevant.

    You say your passion is social justice. Why haven’t you ever mentioned that in a comment? How do you see social justice getting more traction? Or was TFA the extent of your knowledge and commitment to it?

  28. Chris Smyr Says:

    TFT:

    You are incredibly obtuse:

    1) That TFA is a two year commitment does not imply that all TFA teachers have no desire to stay past the 2nd year. This is also irrelevant to the claim that TFA teachers are “not as committed” as other non-TFA teachers to staying longer; in fact, if you wanted to go anywhere with that example, it would suggest that TFA teachers are more committed since they are the ones that are actually signing a contract. I’m not arguing that, however, but that’s really all you could correctly assert with that example.

    2) “That we have a mobile workforce is true, but that doesn’t make it good.”

    So then you were wrong to question my initial statement. I didn’t offer any commentary on the fact that young people entering the work force will likely change jobs several times before settling down. All I said was that it happens that way in general, and not specifically only with TFA.

    3) “The fact that TFAers leave after two years is only evidence that Mark was right.”

    Non-TFAers can also certainly leave after (and even before) two years of teaching….

    You know what? You’re really botching Mark’s arguments here. Why don’t we just let him reply?

  29. Billy Bob Says:

    No program has an attrition rate that even approaches the TFA attrition rate.

    Alternatively, there are numerous programs from elite universities that have far, far lower attrition rates. Loom at UTEACH–these teachers have math and science degrees from one of the top universities in the country and the five year attrition rate is less than 20%. Now THAT is COMMITMENT to children and teaching. TFA is a commitment to one’s own resume.

  30. Chris Smyr Says:

    Billy Bob:

    That’s because you refuse to take off the blinders and even look at the other studies that run contrary to your screed.

    (we did this dance before, and the conclusion was hardly a surprise: http://www.eduwonk.com/2010/07/teach-for-america-and-the-problem-of-study-laundering.html#comment-208479)

    Also: can you give a citation for those numbers on UTEACH?

  31. Chris Smyr Says:

    Billy Bob:

    And before you cite UTeach’s website, I’ll do you a favor and analyze the data myself, since I’d hate to make you look foolish again.

    This is not to knock on UTeach’s success, but there are some key points about the program that you fail to mention (data comes from the UTeach at Austin website and the UTeach Institute website):

    1) UTeach recruits undergraduates early on before they’ve graduated, exposing them to ed coursework and student teaching throughout their time in college before they become full-fledged teachers. 75.6% of students who enroll into UTeach continue with the program after their first introductory field-placement course. 63.6% are retained after the next round of student teaching, and nearly 90% actually move on after the program to a teaching position. “About 50 percent who enroll in Step 1 eventually complete the program and become certified.” None of this attrition is calculated into any official measures of teacher attrition of UTeach once participants begin teaching, which biases the retention data upward and makes for a lousy standard to grade TFA on. This is only the first strike against making your comparisons between UTeach and TFA.

    2) Nearly all of the current graduates are located within Texas, and the program does not specifically articulate any desire to put new teachers into high-poverty schools. TFA, on the other hand, *only* places into such schools (recall that their mission is to close the achievement gap). Since teacher attrition normally exists at higher level at schools that TFA places compared to schools that UTeach places in, it’s not really a surprise that the retention rates are different. Strike two.

    3) Can we even make any useful inferences about UTeach teacher retention after 2 years of teaching versus TFA? For TFA, 61% continue to teach into their 3rd year (Harvard study). For UTeach, the 2009 Spring data they have on file shows that their 2006 cohort (which would have been teaching their 3rd year that year) comprises 66 students, of which 51 are still teaching in a traditional setting. While that’s 77% retention, the extremely small sample size doesn’t make that a very reliable estimate. Furthermore, this number does not include the previous numbers of attrition that I discussed in #1, and also samples a set of schools that are likely very different from what TFA teachers experiences (#2).

    4) We can’t make reliable comparisons of long-term retention data to that of TFA, either, for the same reasons as above. The Harvard study discussed before gave a 26% retention of TFA teachers staying 6 or more years in teaching. The Spring 2009 data for UTeach shows that, of the 134 students that graduated in 2003 or earlier, 75 are still teaching (56%). Again, not only is the sample size incredibly small, but it also does not include the attrition of would-be teachers after their student teaching within the program.

    5) AFAIK, there is no student achievement data available to rate UTeach teachers and compare this to the results TFA teachers have shown. Furthermore, if there were a study out there that measured UTeach’s effectiveness, the validity of the results in any case would be questionable due to the small numbers of UTeach teachers.

    6) The premise inherent in your arguments, that commitment to children should be measured only by the time they spend as teachers, is a weak one to base arguments on. The main goal ought to be increasing student achievement, since that is what directly impacts their lives for the better. Considering that TFA teachers are proven effective at this– even if they don’t remain in teaching– I don’t see why you are throwing a fit about how UTeach is doing so much more for children than TFA.

    Your premise also fails to take into account the importance of having a growing base of TFA alumni that are committed to making positive changes for students outside of the classroom. If I recall correctly, you’re the one that suggested that the top priority (and you also unconvincingly argued that it should be the only priority) be to fight for fixing the gaps in preconditions for students. TFA alums are fighting for just that (http://www.teachforamerica.org/mission/documents/2009_ASIR_Final.pdf ).

    7) Finally when asked in a survey if UTeach student participants “plan on teaching in school in grades 4-12?”, only 30% answered they will definitely teach, 36% probably will teach, while 33% were still deciding. This is nothing I would ever try to fault, but since you want to bring up the level of “commitment” that non-TFA teachers have, you should similarly blast away at these folks for not being “committed” enough.

    Perhaps you should start a new teacher prep program that forces participants to become contractually obliged to teach for 10, 20 years. Would that be enough “commitment” for you?

  32. Billy Bob Says:

    I got my UTEACH numbers from Time–always used by scholars when in need of information. ha ha ha

    The TFA study had a low response rate and there was no evidence that they checked for any type of bias resulting form differences in respondents versus non-respondents. Given the pride of TFA folks, it would be reasonable to assume that those not in the field of education would be less likely to respond to the survey. But, the onus is not on me, it is on the creators of the study. Given that it was from Harvard, I would have expected an entire section on potential biased results. Ten again, the writer was a TFA grad and we all know TFA people like to avoid facts and issues that conflict with their pre-conceived notions.

    UTEACH has reports on-line that track their grads retention rates. They use both survey data and state administrative databases to identify whether people are still in the field. So they don;t have a response rate problem.

  33. Chris Smyr Says:

    Billy Bob:

    1) “I got my UTEACH numbers from Time–always used by scholars when in need of information. ha ha ha”

    That’s a ballsy move, trying to incorrectly phrase my arguments from a thread only a couple days old (and one you also cowered away from):

    http://www.eduwonk.com/2010/08/whole-lotta-news.html#comment-209171

    2) “Given the pride of TFA folks, it would be reasonable to assume that those not in the field of education would be less likely to respond to the survey.”

    No one could substantiate the premise then, and you won’t likely do it now:

    http://www.eduwonk.com/2010/07/teach-for-america-and-the-problem-of-study-laundering.html#comment-208368

    http://www.eduwonk.com/2010/07/teach-for-america-and-the-problem-of-study-laundering.html#comment-208448

    3) “UTEACH has reports on-line that track their grads retention rates.”

    Yup, I analyzed those, too, if you look above.

  34. Billy Bob Says:

    You stated you used Time as a reference–its there in print.

    You don’t get it. The TFA study did not address potential bias. Any good researcher would address the issue. You believe that people other than the researcher has to PROVE bias. We don’t have the data. The researcher does. The fact that potential bias might exist but was not addressed calls into question the conclusions of the author. You clearly don’t understand research and how research reports should be written. It is not up to me or anyone else to prove there is bias–it is the author’s responsibility to prove that there is no bias or caution about conclusions because of potential bias. I’m through taking you to school on this point. Go take a research course or two and get back to me.

  35. Chris Smyr Says:

    Billy Bob:

    “You stated you used Time as a reference–its there in print.”

    To point to the completely transparent conclusion that EVERYONE who has EVER heard of the Coleman Report already knew about! I even pointed you to the Coleman Report itself which wrote THE EXACT SAME THING as what Time had written. This is an amazingly stupid point that you keep wanting to reference, and I really don’t understand why.

    “The TFA study did not address potential bias.”

    The potential bias that you are implying exists is one that relies on the faulty premise that “TFA teachers aren’t proud enough to admit that they stopped teaching after their two-year TFA commitment was over”. Survey results for similar types of surveys given to TFA teachers offer a similar response rate as that found in the Harvard study. None of the TFA teachers I know, both those that stayed in teaching and those that left, felt sorry for themselves in either case. And those that did leave are probably somewhat more likely to respond due to the fewer restrictions on time in other non-teaching careers.

    What else would you have liked the authors do to suggest that such an unlikely bias is even more unlikely than it already is?

    “I’m through taking you to school on this point.”

    That’s hilarious. Are you ever going to respond to any of the past counterarguments addressed to you? Or is this your cowardly attempt to avoid any further probing of your peculiar perspectives?

  36. Billy Bob Says:

    Sheesh. I give up. You don’t understand potential bias in survey research and evidently never will. Its the AUTHOR’S RESPONSIBILITY. Why can’t you understand that?

  37. john thompson Says:

    Billy Bob,

    I’d trust IMPACT in your hands. Chris, not yours.

  38. Chris Smyr Says:

    Alright, so to summarize this thread:

    John Thompson makes a(nother) claim that he can’t/won’t justify with evidence/reasoning (how IMPACT would/could be used for devious purposes in the hands of Rhee)

    Marktropolis reminds us that VAM should not be the sole reason for firing anyone, and luckily it’s not

    TFT awkwardly attempts to defend Mark’s claim that TFA teachers are not committed (because they make a commitment to teach for at least 2 years)

    Billy Bob declares that UTeach is more “committed” than TFA because of higher retention rates, but doesn’t bother to read beyond face value into the numbers, nor does he trust a study that didn’t offer evidence to disprove his claim even though the claim is spurned by a growing biased bitterness of anything TFA rather than anything of objective reality.

    My postulated Law of Eduwonk Debates seemed to be quite effective at predicting all of this…. (http://www.eduwonk.com/2010/08/kipp-and-catholic-schools.html#comment-208856 )

  39. FTW! Says:

    Chris Smyr FTW!

  40. Billy Bob Says:

    Let me summarize this thread.

    Chris thinks everyone is wrong except for him.

    Chris cannot understand any arguments except his own (although I am not so sure he understands his own arguments), therefore blames everyone else.

    Chris does not understand research methodology, thus blindly believes numbers that support his world-view.

    Chris rambles on and on and on burying everyone and everything in a huge pile of BS spewing from the mouth of someone who clearly has not studied education and education research to form any valid conclusions.

  41. BeatTheDevil Says:

    This exchange reminds me of a time back in early 1986.
    Christopher Hitchens was on Crossfire and after receive abuse from Robert Novak, he called him a McCarthyite bum and a polecat.
    Then at the end of another Novak tirade, Hitchens responded “More musk from the polecat.”

  42. Chris Smyr Says:

    Billy Bob:

    The Harvard study also did not provide evidence showing there isn’t a grand conspiracy afoot, where all TFA teachers are actually contractually obliged to spread misinformation on any such surveys or questionnaires by only marking down choices that boost the performance, retention, and awesome factor of TFA.

    An equally valid claim goes unnoticed! Harvard is in league with the enemy! Quick! John Thompson, TFT or Bill Turque needs to write a scathing article about it at once!

  43. Billy Bob Says:

    Chris, go take a survey research course and then come back and talk. You know you are wrong, you backed yourself into a corner, and refuse to admit you are wrong on the Harvard study.

    When looking at REAL data–from state administrative data–almost none of the TFA teachers remain in teaching positions. Yet, programs like UTEACH and others from elite universities keep up to 80% of their teachers over five years. Why plow money into TFA when we could plow money into programs like UTEACH and see a lon-term difference?

    Because TFA people can’t accept actual evidence.

  44. hu? Says:

    Billy bob, did you actually read chris’ response to your UTEACH claims? He pretty thoroughly eviscerated your numbers. It’s telling that you have no response.

  45. hu? Says:

    in case you missed it here’s his response, reposted for your convenience:

    Billy Bob:

    And before you cite UTeach’s website, I’ll do you a favor and analyze the data myself, since I’d hate to make you look foolish again.

    This is not to knock on UTeach’s success, but there are some key points about the program that you fail to mention (data comes from the UTeach at Austin website and the UTeach Institute website):

    1) UTeach recruits undergraduates early on before they’ve graduated, exposing them to ed coursework and student teaching throughout their time in college before they become full-fledged teachers. 75.6% of students who enroll into UTeach continue with the program after their first introductory field-placement course. 63.6% are retained after the next round of student teaching, and nearly 90% actually move on after the program to a teaching position. “About 50 percent who enroll in Step 1 eventually complete the program and become certified.” None of this attrition is calculated into any official measures of teacher attrition of UTeach once participants begin teaching, which biases the retention data upward and makes for a lousy standard to grade TFA on. This is only the first strike against making your comparisons between UTeach and TFA.

    2) Nearly all of the current graduates are located within Texas, and the program does not specifically articulate any desire to put new teachers into high-poverty schools. TFA, on the other hand, *only* places into such schools (recall that their mission is to close the achievement gap). Since teacher attrition normally exists at higher level at schools that TFA places compared to schools that UTeach places in, it’s not really a surprise that the retention rates are different. Strike two.

    3) Can we even make any useful inferences about UTeach teacher retention after 2 years of teaching versus TFA? For TFA, 61% continue to teach into their 3rd year (Harvard study). For UTeach, the 2009 Spring data they have on file shows that their 2006 cohort (which would have been teaching their 3rd year that year) comprises 66 students, of which 51 are still teaching in a traditional setting. While that’s 77% retention, the extremely small sample size doesn’t make that a very reliable estimate. Furthermore, this number does not include the previous numbers of attrition that I discussed in #1, and also samples a set of schools that are likely very different from what TFA teachers experiences (#2).

    4) We can’t make reliable comparisons of long-term retention data to that of TFA, either, for the same reasons as above. The Harvard study discussed before gave a 26% retention of TFA teachers staying 6 or more years in teaching. The Spring 2009 data for UTeach shows that, of the 134 students that graduated in 2003 or earlier, 75 are still teaching (56%). Again, not only is the sample size incredibly small, but it also does not include the attrition of would-be teachers after their student teaching within the program.

    5) AFAIK, there is no student achievement data available to rate UTeach teachers and compare this to the results TFA teachers have shown. Furthermore, if there were a study out there that measured UTeach’s effectiveness, the validity of the results in any case would be questionable due to the small numbers of UTeach teachers.

    6) The premise inherent in your arguments, that commitment to children should be measured only by the time they spend as teachers, is a weak one to base arguments on. The main goal ought to be increasing student achievement, since that is what directly impacts their lives for the better. Considering that TFA teachers are proven effective at this– even if they don’t remain in teaching– I don’t see why you are throwing a fit about how UTeach is doing so much more for children than TFA.

    Your premise also fails to take into account the importance of having a growing base of TFA alumni that are committed to making positive changes for students outside of the classroom. If I recall correctly, you’re the one that suggested that the top priority (and you also unconvincingly argued that it should be the only priority) be to fight for fixing the gaps in preconditions for students. TFA alums are fighting for just that (http://www.teachforamerica.org/mission/documents/2009_ASIR_Final.pdf ).

    7) Finally when asked in a survey if UTeach student participants “plan on teaching in school in grades 4-12?”, only 30% answered they will definitely teach, 36% probably will teach, while 33% were still deciding. This is nothing I would ever try to fault, but since you want to bring up the level of “commitment” that non-TFA teachers have, you should similarly blast away at these folks for not being “committed” enough.

    Perhaps you should start a new teacher prep program that forces participants to become contractually obliged to teach for 10, 20 years. Would that be enough “commitment” for you?

  46. Chris Smyr Says:

    Billy Bob:

    Go and actually LOOK at the UTeach data. I even broke it down for your lazy self upthread. The only way you could possibly validly compare those data with TFA retention data would be if, after TFA teachers finished their two year commitment, those that wanted to continue teaching were funneled into a new program called Keep Teaching For America (KTFA), and retention data from THAT program were analyzed with respect to UTeach. That still wouldn’t address, however, the other 6 (out of 7) critiques that I wrote.

    And claims that have exceedingly diminishing chances of actually being true do NOT have to be a focus of every goddamn study! There are INFINITE alternative hypotheses for EVERY scientific question that has EVER been addressed in the research, be it education or in any other field. So why is it that every paper isn’t 500 pages long? Because useless claims that stem from a prejudiced outlook on the subjects (and boy do you deal in TFA prejudice!), but given any of the objective evidence available likely have no chance of being correct, do not even deserve the few sentences that it would take to further denigrate them.

  47. steve f. Says:

    Hi Chris,

    Just curious, in your opinion could one compare TFA retention data to any other programs?

    How should one objectively evaluate TFA retention, if not by comparing it to another program?

    Thanks.

  48. Chris Smyr Says:

    Steve F.:

    I imagine it depends on the program and how their data is generated. UTeach is a terrible comparison due to the reasons I outlined. Are there other programs that are more similar to TFA? Yes. But even so we still need to look past the face value of numbers for these programs and determine how their data is generated as well, which is something that our expert researcher Billy Bob doesn’t want to do.

    It’s also revealing that the discussion currently is now focused on teacher retention. What initially brought us down this path was someone’s inept comment about TFA teachers not having “commitment”. Useless and despicable comments such as the above should NOT be given the power they are currently given on this blog to dictate topics of discussion, and yet were I not to have responded, the issues originally discussed (John Thompson pulling arguments out of his ass about anything he doesn’t like) would have nonetheless been dropped.

  49. steve f. Says:

    Hi Chris,

    Thanks for your response.

    Could you name a few of these similar programs that you might compare TFA retention data to if you were comfortable with data set?

  50. Chris Smyr Says:

    Steve F.:

    If you’re interested in finding a comparable program to TFA, the Internets would be a great place to start looking. I’m not interested in doing more legwork when the data analysis I’ve already done has proven fruitless in making others open their eyes.

  51. steve f. Says:

    Hi Chris,

    Thanks for your response.

    I just figured that anything we were to find and bring back to you – you would take issue with making a comparison to TFA.

    So, I was trying to avoid that middle step.

    And maybe help everyone in the thread see that some real comparison could be made to TFA.

    You know, to stop all the TFA bashing around here.

    Thanks anyway, though.

  52. Chris Smyr Says:

    Steve F.:

    Interesting dilemma. Do I wait for someone to bring up another teacher prep program that has amazing numbers, or do I painstakingly search myself for such a program?

    In either case:

    1) Whatever data analysis I present will continue to be ignored.

    2) The initial topics brought up in this thread will continue to be ignored.

    3) I will continue to be vilified.

    4) There will continue to be resentment of the big TFA conspiracy.

    I also like that you’re putting the onus on *me* to stop the TFA bashing that goes on here, rather than on the lazy commenters that continue to frequent these discussions.

    And, no, thank you for the response.

  53. Billy Bob Says:

    Why would TFA people have to be put in a different program? Using state data, one can track TFA grads as long as they stay in teaching. The TFA designation does not automatically disappear. SO UTEACH has a much, much higher retention rate. Why can;t you compare the two? UTEACH provides initial mentoring and support, but so odes TFA. UTEACH tries to place their grads in the same schools so they can provide support to each other. TFA does the same thing. Yet, almost no TFA teachers make it past three years, while a huge chunk of UTEACH grads do. So which programs affects students more? UTEACH of course!

    Its really pretty simply.

    And your link did not work.

    You seem mighty angry that UTEACH has a much better track record on retention than TFA even tho both are elite programs.

    Fortunately, the business community is funding replication oof UTEACH across the country. Maybe they will put TFA out of a job. One can only hope–for the kids’ sakes.

  54. Chris Smyr Says:

    Billy Bob:

    You again ignored all 7 critiques. That is quite a feat.

  55. Billy Bob Says:

    1) Yes, many UTEACH students do not teach. Thank god they figured out they shouldn’t be teachers before they actually screwed up some classrooms full of kids. And I would venture to guess that not everyone who enters TFA eventually teaches either.

    My fav comment:”exposing them to ed coursework and student teaching.” Hmmm–you mean they PREPARE their teachers? The horror!! Who do they think they are actually PREPARING PEOPLE TO TEACH.

    And a fair percentage of UTEACH grads teach in high-poverty schools. And they stay in teaching despite that. Many teach in Austin, Houston, and Dallas. TFA teachers teach in Houston.

    By the way, an internal Houston ISD memo found that TFA teachers were more likely than other teachers to be ineffective under their VAM model.

    In a southeastern state, teachers from traditional prep programs that tiaught in the VERY SAME schools as TFA teachers were far, far more likely to stay for a 3rd, 4th, 5th, and 6th year. Same schools. Can’t compare TFA in Texas–TFA does not want its teachers identified in the state data system.

    UTEACH has multiple cohorts of teachers now and the retention rate hovers between 75% and 80%.

    The validity of the TFA teachers is in question as well, especially when you take into the negative effect on students of greater attrition.

    Face it–TFA and UTEACH can be compared and TFA is no better and may be worse.

    And at least UTEACH uses REAL data to calculate retention, not some survey with a 62% response rate. Error rate on UTEACH retention–0%. Error on TFA study–unknown.

  56. Chris Smyr Says:

    Billy Bob:

    It’s like you’re reading something other than my arguments when you respond to me. Is it a good read, whatever it is?

    “Yes, many UTEACH students do not teach. Thank god they figured out they shouldn’t be teachers before they actually screwed up some classrooms full of kids. And I would venture to guess that not everyone who enters TFA eventually teaches either.”

    You really don’t get it. You are trying to compare retention rates of two very different programs. It’s not even about a vague sentiment of “some don’t teach”; there is an accumulating percentage of attrition that continues throughout UTeach. And this is not just from people who decide “hmm, this coursework isn’t right for me.” The numbers I cited were loss of students *AFTER* they started student teaching. The official teacher retention percentages offered by UTeach do not include any of the many students that decide, *AFTER* teaching, that they don’t want to teach anymore. The program is designed that way, which is not to fault the program itself, but rather careless attempts at comparisons to different data sets.

    In contrast, TFA teachers commit to at least two years of teaching without this accumulating attrition before they began their careers. Whatever number is reported for attrition rates of TFA teachers encompasses the career decisions made of the ENTIRE cohort of TFA recruits; with UTeach, their attrition rate encompasses ONLY THOSE that have already taught and decided they wanted to stay in teaching. This latter number is an estimate of teacher attrition from a group of teachers that have already decided they want to remain in teaching.

    “And a fair percentage of UTEACH grads teach in high-poverty schools. And they stay in teaching despite that. Many teach in Austin, Houston, and Dallas. TFA teachers teach in Houston. ”

    Can you cite that “fair percentage” of grads teaching in high-poverty schools? Is it 100%? If it’s not, then my #2 critique still stands.

    “By the way, an internal Houston ISD memo found that TFA teachers were more likely than other teachers to be ineffective under their VAM model.”

    Yup, because an internal memo is a much more reliable metric of teacher effectiveness than all of the research with sound methodologies, right? (http://www.teachforamerica.org/about/research.htm#card )

    “In a southeastern state, teachers from traditional prep programs that tiaught in the VERY SAME schools as TFA teachers were far, far more likely to stay for a 3rd, 4th, 5th, and 6th year. Same schools. Can’t compare TFA in Texas–TFA does not want its teachers identified in the state data system.”

    It seems you are citing data from your unpublished dissertation again, am I right?

    “The validity of the TFA teachers is in question as well, especially when you take into the negative effect on students of greater attrition.”

    This also seems to cite evidence from your unfinished manuscript. Would you care to explain how the student achievement of a 1st grader is affected retroactively by his teacher leaving the profession 2 years later?

    “And at least UTEACH uses REAL data to calculate retention, not some survey with a 62% response rate.”

    Surveys that receive the same response rate as other TFA surveys don’t give real data? But data from a sample size of 66 teachers *is* real data? We shouldn’t trust studies that haven’t addressed the biases of an angry anonymous researcher on Eduwonk? But we should trust a study that said researcher hasn’t finished yet?

    Look. I’ll let you try again. Here’s the link to my critiques (http://www.eduwonk.com/2010/08/adding-value-2.html/comment-page-2#comment-209225 ). Read it. Respond to it. There’s plenty of other counterarguments of mine that you similarly let fall to the wayside since you didn’t want to address them. I can link to them as a reminder if you’d like.

  57. Chris Smyr Says:

    And while my first comment patiently awaits moderation, I’m wondering if you were trying to refer to Hammond’s 2005 study? And if so, link: (http://eduwonk.net/2009/08/if-the-race-to-the-top-were-the-olympics.html#comment-94394 )

  58. Billy Bob Says:

    OK Chrius–you convinced me. TFA is the BEST teacher preparation program ever invented. The teachers are extraordinary individuals and elicit great gains from the poor minority kids who are evr so thankful to be taught by privileged white teachers from Ivy-league schools. And the teachers from TFA are so committed that the majority stay much past their two-year commitment because they want to teach in these schools for extending periods of time. They don;t want to pad their resume, but really, really care about these kids and want to stay in the community for long periods of time. If we could only prepare a million TFA teachers, we could close the achievement gap, be number 1 in the world in every subject, solve our financial crisis, and everyone would live happily ever after.

    Keep dreamin buddy.

  59. Chris Smyr Says:

    Billy Bob:

    I didn’t intend to convince you of anything. My only intention is to make sure readers knew that you, as several others here, deal only in deception and prejudice, rather than good-faith arguments.

    On that note, I’m greatly looking forward to reading your coming paper.

Leave a Reply


seven − 1 =