If Bill wants to discuss the issues he brought up, great. In the other thread I asked him to elaborate; in this thread I’ve simply pointed out that his citing of his degrees seemed rather useless and that his ideas regarding accuracy are slightly unhinged. Since you’ve brought up different topics than he, perhaps we can start there.

If value-added scores are such accurate measures of teacher knowledge and skill why is it that these VA score correlations are .2 in ELA and less than .5 in math? Why so much variation? Wouldn’t one expect a “good teacher” to have comparable scores across a number of class section?

First, questioning the accuracy of VAM should take into the account the notable results from research in the field. The NBER working paper by Chetty et al, for example, makes a strong case that VAM is accurate enough to forecast VAM changes at the test score level as well to forecast long-term outcomes. The paper got some bad press from the usual suspects, but none of the critiques leveled thus far hold water. Yes, it’s obvious there is variation. This is not a novel concept nor one that removes the benefit from utilizing VAM. Despite the variation and the error, there are key results that support VAM’s reliability.

I should also question some of your own assumptions regarding teacher quality and variability. Suppose there were an absolute measurement that could track the effectiveness of a single teacher, based on any number of criteria at 100% accuracy throughout a number of years and classrooms. I wouldn’t be surprised to still see such a measurement vary for very mundane reasons given the way teachers can have a different fit in certain classes with certain students and certain subjects and certain content organized within certain curricula. Such a measurement — even without any error — could show variation, and I’d expect it to do so. But such a measurement would still be useful because of the trend. We’d be able to identify exactly which teachers are most often effective and ineffective at what they do.

There is likely more variation in VAM estimates of teacher quality than actually exists, but VAM scores over time, while noting trends and teacher ratings relative to others, would still be a useful and objective (and accurate!) data point to add to the measures of accountability we currently have. In short, I think that variability is likely more than an artifact of the research design, but that heeding this caution while utilizing it within a system of accountability would still be useful.

Yes, a teachers has a huge amount to do with the learning experience in the classroom, but so does the curriculum and how it structures and conveys knowledge. So, does the culture of the school and school and peer norms. Many researchers are implicitly conflating teachers with teaching. They are related but not the same.

Chetty et al. also showed that the measured accuracy of VAM was not conditionally tied to any particular curriculum, school culture or norms. Good teacher who left for another school brought their high VAM scores with them. It was not situational.

Teacher effectiveness is thought to be the largest in-school factor of significance, and it seems intuitive. Why is it that a teacher shouldn’t be responsible for the curriculum they use? And why assume that school norms must weigh down on every teacher? Research (like the CALDER study Andy referenced) suggests that teacher quality fluctuates greatly even within underperforming schools, evidence that good teachers can lead their students to large learning gains despite obstacles outside the classroom. If factors outside of a teacher’s control were the dominant in-school factor, we shouldn’t see this kind of phenomenon taking place.

I actually think Bill raises some points you ought not dismiss too quickly. I’ve actually commissioned a VAM study and worked with the results. Although VAM shows promise for certain uses, I think it’s always important to remind oneself of the taken-for-granted assumptions underlying statistical models. It is easy to get impressed by some of the elegant highly sophisticated techniques that researchers use in building their model and fail to fully appreciate some of the questionable assumptions and attributions that they make on a routine basis. At the very least these assumptions ought to be scrutinized carefully. I wonder what you make of the relatively low year to year and even classroom to classroom correlations of Value added scores? If value-added scores are such accurate measures of teacher knowledge and skill why is it that these VA score correlations are .2 in ELA and less than .5 in math? Why so much variation? Wouldn’t one expect a “good teacher” to have comparable scores across a number of class section? I don’t think it’s simply a matter of measuring error but rather something more fundamental. I think many researchers using VA models make some fundamental attribution errors in interpreting their results. They assume that teachers are 100% responsible for student learning, when it is more accurate to state that what they are estimating is the amount of learning gain that students achieve as a result of spending a year in a particular classroom. Yes, a teachers has a huge amount to do with the learning experience in the classroom, but so does the curriculum and how it structures and conveys knowledge. So, does the culture of the school and school and peer norms. Many researchers are implicitly conflating teachers with teaching. They are related but not the same.

@Scott, I get the draw that appeals to authority have, but why also the faulty assumption that the experts who argue in favor of using VAM are only dabbling in “anecdotes, personal opinions, and predetermined think tank reports”?

@Bill, degrees in math and physics do not make you an expert on VAM. Are you a researcher who has worked with VAM? Your last comments on this blog suggest you are a teacher. Your ideas about accuracy also suggest that maybe you’re not the go-to guy on this.

@Kent, we already know that in-school factors are not the largest factor influencing student outcomes (teacher quality is considered the largest of these). However, that fact doesn’t impose a maximum on the influence that in-school factors could seek to cause.

“Now back to the issue at hand. Income and demographics account for nearly 85% of the variation in student test scores. The other 15%, that variation which can be explained in a statistically significant way, is related to teacher quality. Thus, when edu-reform advocates talk about the power of good teaching they are talking about his 15%.”

Interesting. So only 15% of the variation is “related” to teaching, or more broadly put “instruction.” And of that 15%, how much is within the control of individual teachers? Because instruction includes such things as:

technology in the classroom
textbooks and curriculum
safety and discipline
attendance rates
support for special education/special needs
class size

And probably dozens of other instructional factors that I’m forgetting.

If 85% of variation can be explained by income and demographics. That means by definition that all other factors combined account for the remaining 15%. So again I ask, what part of that 15% is really in control of individual teachers?

Because if individual teachers account for the entire 15% then we are saying that technology, class size, school safety, curriculum, etc. have no measurable effect.

The VAM controls for INCOME AND DEMOGRAPHICS. There is no argument in the edu-refom movement that these two variable are excluded from the two VAM models: The one at SAP and the other at UOW Madison. Both were made by economist.

As a bit of history, economist have been laboring since the 70′s to bring a sense of quantitative analysis to a more or less fly the seat of the pants, descriptive, weak discipline. This relatively new field is called mathematical economics.

As one of my graduate professors in econometrics told me: “Econometrics rests entirely upon apriori assumptions.”

Now back to the issue at hand. Income and demographics account for nearly 85% of the variation in student test scores. The other 15%, that variation which can be explained in a statistically significant way, is related to teacher quality. Thus, when edu-reform advocates talk about the power of good teaching they are talking about his 15%.

The VAM creators open admit that their model is not ROBUST enough to incorporate income and demographic data without CAUSING their explanatory variables to generates coefficients that have very low t statistics.

Thus they have a VERY SERIOUS AUTO-CORRELATION PROBLEM when they model is built the way it should be.

So, right now, there is hidden in the teacher variables a tremendous amount of explained variation that is DUE TO VARIABLES THAT HAVE BEEN EXCLUDED FROM VAM MODELS.

This is a SERIOUS DEFECT and it is one that should ENTIRELY INVALIDATE present VAM MODELS.

But what is the edu-reform movement saying? Well they say that something is better than nothing. This is, on its face, risible.

As a tongue in cheek comment to end this piece, South Africa has now passed law that imprisons and fines meteorologists for issuing incorrect weather predictions. I kid you not.

Perhaps it is time to do the same thing for blogger, journalists, economists, and all edu-reformers. Unless they demonstrate a 90% accuracy rate, they should be fired, banned, and imprisoned for faulty research.

Their faulty predictions have cost the economy trillions of dollars in BAD DECISIONS made by free market participants. And as an aside, what do they have to fear. The good predictors should get merit pay.

Perhaps we could pass a law that states simply: Academics, and bloggers are no longer allowed to post academic qualification. They are, by law, required to post their accuracy rate. And if they falsify that rate, they will be imprisoned.

Isn’t it time for the edu-reform movement to spread throughout the free markets? Oh we could also outlaw that long treatise at the bottom of your mutual fund report that says that past performance is NO GUARANTEE OF FUTURE PERFORMANCE.

Cheers.

http://bit.ly/xuF7US

Research in Educational Researcher, one of our very top, methodologically stringent, peer-reviewed education research journals, notes the methodological concerns about VAM:

http://bit.ly/A55Ozd

For educational policy and practice, we should be choosing actual data, scholarly analysis, and peer review over anecdotes, personal opinions, and predetermined think tank reports EVERY TIME.