Teach For America Evaluation, Reading Parters Pops, Charter School Debate, & Tom Kane Attacks!

Light posting this week, I’m traveling in the west. A few things I’m reading this week though:

MDRC evaluation of Reading Partners (pdf). Volunteer tutors exceed expectations. There might be implications here!

Mathematica evaluation of Teach For America (pdf). It’s being treated as though it’s groundbreaking but Mathematica’s Teach For America* evaluation is just the latest serious evaluation of Teach For America over the past decade to show that TFA teachers perform as well or better than other teachers (that includes, among other work, evaluations by states like TN and LA, think tanks like Urban Institute, research initiatives like CALDER, and other evaluations by, yes, Mathematica). If there is any news here it’s around scale and quality questions – an area where TFA has broken the traditional education mold.  Yes, it’s legit to argue that TFA’s theory of change/action is wrong for the education sector, everyone is entitled to their own opinion. But the ongoing “debate” about effectiveness in assessed subjects/grades – abetted by a statistically illiterate and conflict addicted media – is a waste of time and energy.

Scott Pearson/Skip McKoy and Neerav Kingsland debate the all-charter district versus the blended charter/district approach.

Tom Kane proposes defunding the regional education labs to support more fast turnaround R & D evaluation.

*Bellwether recently did a project for Teach For America – evaluation of different dimensions of their growth. You can read it here.

4 Replies to “Teach For America Evaluation, Reading Parters Pops, Charter School Debate, & Tom Kane Attacks!”

  1. In conclusion, read at face value, here is the message Mathematica appears to promulgate with the report:

    We do not need experienced (read: more expensive) teachers when non-experienced, less expensive teachers get the “same” —though not statistically significant— outcomes.
    We do not need a more diverse workforce of teachers, again, because TFA teachers, who are overwhelmingly white, get the same outcomes.

    http://tinyurl.com/mfll28t

  2. David, from the study (and the lit review on prior studies, not pasted here) is especially useful with regard to the context:

    TFA impacts on math and reading achievement
    On average, the TFA teachers in our sample were as effective as comparison teachers in both reading and math. In both subjects, differences in test scores between students assigned to TFA teachers and those assigned to comparison teachers were not statistically significant.
    We found that TFA teachers in lower elementary grades (prekindergarten through grade 2) had a positive, statistically significant effect on student reading achievement of 0.12 standard deviations, or about 1.3 additional months of learning for the average student in these grades nationwide. However, for both math and reading, we did not find statistically significant differences in either direction for other grade levels or for TFA teachers compared with either novice or traditionally certified teachers…

    Conclusions
    In this evaluation we documented TFA’s experiences as it undertook an ambitious five-year scale-up effort, and we provided rigorous estimates of the program’s effectiveness in the second year of the scale-up. We found that TFA elementary school teachers recruited in the first and second years of the i3 scale-up were as effective as other teachers in the same high-poverty schools in teaching both reading and math. We found that TFA teachers in lower elementary grades had a positive, statistically significant effect on student reading achievement, but we did not find statistically significant impacts for other subgroups of TFA teachers that we examined.
    Our main findings are consistent with earlier studies showing that TFA teachers were just as effective as other teachers in teaching reading; however, they differ from the findings of several prior studies showing that TFA teachers were more effective than their colleagues in teaching math. Although we cannot definitively determine why our impact estimates for math differ from previous studies, we found some evidence that corps members’ satisfaction with the program declined in the first two years of the scale-up relative to the two prior years and the quality of comparison teachers in the schools served by TFA might have changed for the better…

  3. This deceptive Prof. Helig found that the

    Finding 1: TFA Teachers did NOT statistically improve achievement relative to comparison teachers
    Finding 1: TFA Teachers did NOT statistically improve achievement relative to comparison teachers

    Let’s being with the first “finding.” The infamous “as effective” graph was first sent to me on Twitter by one of my former UT-Austin stats students the day the study was released— he noticed the spin right away.

    Mathematica knows better than to spin this graph, but they did anyway. In the introductory statistics course that I teach, we of course talk about the difference between the “eye ball test” and results that are statistically significant. The eye ball test is what the average person does when they look at a bar graph produced by a study and say “one bar is higher than the other” so, therefore, TFA is slight better. This is false. What statistical significance tells you is that you can be at least 95% confident that the difference is not solely due to chance in the sample. This is important because if a result is not statistically significant, you could resample and the results in bar graphs could be flipped to the opposite or even. In other words, if a graph is not statistically significant, then there is nothing to see here that we should be confident about. But Mathematica chose to spin this graph as “they perform about the same.” They know better— the result is not statistically significant.

    Let’s being with the first “finding.” The infamous “as effective” graph was first sent to me on Twitter by one of my former UT-Austin stats students the day the study was released— he noticed the spin right away.

    Mathematica knows better than to spin this graph, but they did anyway. In the introductory statistics course that I teach, we of course talk about the difference between the “eye ball test” and results that are statistically significant. The eye ball test is what the average person does when they look at a bar graph produced by a study and say “one bar is higher than the other” so, therefore, TFA is slight better. This is false. What statistical significance tells you is that you can be at least 95% confident that the difference is not solely due to chance in the sample. This is important because if a result is not statistically significant, you could resample and the results in bar graphs could be flipped to the opposite or even. In other words, if a graph is not statistically significant, then there is nothing to see here that we should be confident about. But Mathematica chose to spin this graph as “they perform about the same.” They know better— the result is not statistically significant.

Leave a Reply

Your email address will not be published.