PISA! 5 Reasons It Matters And Doesn’t

In the education sector today sounds like a never-ending ad for Little Caesar’s….PISA PISA…

The new PISA results are out.  Usual suspects saying the usual things.   Few things to keep in mind.

– Bottom line: The sky is not falling, but the ceiling may be closing in on us some.  But we don’t need international comparisons to tell us we have serious educational problems here.

These results matter, but not as much or as little as various combatants would have you believe.  The best thing to look at is the patterns because as a rule any precision will be false precision given the limitations of the assessments.  And the pattern seems to be that we’re holding steady but other countries are getting their educational acts together.  Over time that’s a problem for the United States in a globalized economy. We’re making a big bet that our political, immigration, and economic system can overcome the drag of a subpar education system. That worked in the 20th Century but may not be a smart bet in this one.

– International comparisons are fraught with loose claims. In particular, correlation does not equal causation. So beware when someone points to a single element of country X and says that’s why they are doing well. Context matters, including a country’s political, policy, economic, and social context. That doesn’t mean there is not plenty to learn from other countries, there is (just as there is plenty to learn from other sectors and institutions) but we should respect the limits of comparisons. Likewise, consider how various ideas and systemic features would fit with our educational system and values- for instance we heavily value giving students second chances.

– When someone tells you that we’d be doing great on these tests except for all the poor kids who pull the average down, remind them that it’s not entirely correct but it’s also largely irrelevant: Those poor kids do live here, are being educated (or not) here. It’s important to disaggregate the data for analytical purposes but as a practical matter of making policy today their lives are not a thought experiment or debating point.

– If you want to obsess about this issue read Amanda Ripley’s book instead.

15 Replies to “PISA! 5 Reasons It Matters And Doesn’t”

  1. Correlation does not equal causation, except of course, where Andy is pushing the Duncan line that some modest NAEP progress for DC and Tennessee proves that ed reform policies are dandy. What a difference a few weeks makes.

  2. Rosa – please post where I wrote that or said it? I don’t mean what you heard, or think, or whatever, but where did I actually write/say that?

  3. Yeah, that pretty much makes the point. The post doesn’t mention Duncan or the administration and says explicitly that,

    “Causes? Probably multiple but the gains are noteworthy.”

    It’s less than a few hundred words in total so it shouldn’t be that hard to follow.

  4. Well, I am not an American and I don’t even leave there but, I must say that some other countries are having similar ‘problem’ . In my country for example, every year, most national/international exam results keep getting poorer and poorer.

    Well, I think the part of the problem is from the government which is not willing to provide basic facilities required in the educational sector.

  5. When I see these scores, I always wonder how typical my own son is. Now 42 years old, he played his way through grade school and high school and majored in “making stuff.” His test scores were always mediocre. But once he got to college, he settled down and got serious. He became an A student at the University of California, and then won a fellowship to Stanford where he earned a Ph.D. in electrical engineering. Today he is the “principal scientist” at his company. He has many friends in other countries, and can hold his own with anyone in the world. As Salmon Khan said today on CNN: “Test scores don’t tell the whole story.” The United States is still up there in creativity and innovation.

    That said, the PISA scores do give us some critically important information: Test scores in the United States are highly correlated with income. Our richest kids get the highest scores, the affluent kids are next, then middle income, then lower middle, then poor and so on down. Of course, we’ve known this for many years, but have yet to do anything about it.

    It’s time to end the shameful status quo of education by zip code. Other countries have done it and so can we.


  6. As usual, Steve Sailer’s racially reductionist take summarizes the results pretty well…


    “As I wrote in 2010, the main focus of the 2009 test was reading, and American students read relatively well:

    • Asian Americans outscored every Asian country, and lost out only to the city of Shanghai, China’s financial capital.

    • White Americans outperformed the national average in every one of the 37 historically white countries tested, except Finland.

    • Hispanic Americans beat all eight Latin American countries.

    • African Americans would likely have outscored any sub-Saharan country, if any had bothered to compete….

    “At the continental level, PISA (and the competing TIMSS test) show the same general broad patterns as seen in countless tests in the US: On average, Northeast Asians are out front, then Europeans, then Latin Americans, then blacks. So the PISA results have some prima facie validity. Of course, it’s discouraging to learn that PISA’s global results are largely the American bell curve writ large, so nobody notices.”

  7. As usual, pointing out simple and obvious truths in Sailerian style is grounds for excommunication from polite society. In particular, multiculturalism, immigration, diversity, and blacks are all bad for PISA scores…

    “What do Finland, Poland, and South Korea have in common?

    “First, all three achieved their independence only within the last 100 years, and they were each invaded at Stalin’s command in 1939-1950. They remain relatively patriotic in the old-fashioned nationalist modernizer mode that did so much to lift educational levels beginning in Prussia. They see education as a way for their people to get ahead in the world.

    “In contrast, India—even though it is always grouped with China in President Obama’s rhetoric about America’s main educational rivals—has dropped out of the PISA after a disastrous trial run in which two Indian states performed at sub-Saharan levels. India has been the darling of Davos Man for its seeming postmodern post-nationalism, but unfashionable old nationalism retains the best track record at uplifting the masses.

    “Second, none of the three had colonies and therefore don’t feel any imperial guilt.

    “Third, the three countries have very few students of first- or second-generation immigrant background. A recent PISA study showed Finland’s student body is only 3 percent non-Finnish (compared to 19 percent immigrant in the US), while Poland’s and South Korea’s immigrant percentages rounded down to zero.

    “Ripley visits a Finnish school that has many immigrants and breathlessly reports that everybody is above average. In reality, children of immigrants score well over a half standard deviation below average in Finland. And the three largest sources of immigrants in Finland remain neighboring Estonia, Russia, and Sweden.

    “In 2009, immigrants dragged down average PISA scores in almost all countries except Australia, which works hard to keep out illegal immigrants and uses a points system to help pick promising legal ones.

    “Fourth, even though Ripley is excited to see some Somalis in one Finnish school, none of these countries has many blacks. What giant pandas are to the cause of biodiversity preservation, blacks are to the reigning dogma of diversity: “charismatic megafauna.” Thus, not only do blacks tend to lower test scores and disrupt classrooms, they also tend to warp elite thinking and public policy.

    “For example, the Obama Administration has been persecuting local school districts that suspend higher percentages of black students. Now, you know and I know and even Barack Obama knows that the reason for this is because on average black students cause more trouble. But in America today you aren’t supposed to know that, so the federal government is pushing a policy on school districts: If they want to stay out of trouble with the feds, they should ignore black troublemakers.”

  8. Here’s a simple graph of the 2012 PISA results by country:


    Asian Americans did pretty well versus Asian countries, white Americans did a little better than the average for white countries, and Latino Americans did better than all Latin American countries. No black countries participated in PISA, but almost certainly none would have matched the performance of African Americans.

  9. Most Americans would rather be poorly educated and highly paid, than well educated and poorly paid.

    Michelle Rhee is of mediocre academic accomplishments and professional achievements, but is handsomely compensated.

    It is silly to treat students and parents like customers, and then in the same breath call the education product a RIGHT.

    R and his posse have gone on and on about the validity of comparisons using data, and now they urge moderation in parsing the PISA data. Why?

    Easy answer really. PISA does not support their 15 year narrative. Many of the solutions they insist will drive education improvement in this country are not being used in the rest of the world where better results are achieved.

    Their response? The USA is NOT like the rest of the world. So, does that mean the USA is holding on to a market/political model that is doomed to failure since none of the sure fire remedies that work for the rest of the world will work here at home?

    One interesting pattern. The wildly successful nations in PISA do not have sports fields near their schools.

  10. If manufacturing techniques can be duplicated around the world with identical results, then why can’t education techniques also be duplicated?

    R and his posse have defended the industrial/data model of education for 15 years. Now, when their remedies are proven to give our nation’s education system cement boots, they run for cover with the wonderful encomium: Not all solutions are scalable.

    So, the USA is NOT allowed to adopt proven low cost methods of improving schools, but must BE FORCED to use “made in the USA” school reform products.

    Can it be this bad? American business is DEMANDING TRADE PROTECTIONS for its lousy school reform products?

  11. Our backwards thinking.

    Low skill set, in surplus, education reformers make two to three times as much money with better working conditions that a critical need, in short supply, math or physics teachers.

    Pay and working condtions tell the truth. Top talent math and physics teachers are NOT needed.

    If they are, then relative pay scales must change and their social status must also change.

    And R and his education reform elite will never let that happen.

    Here is their logic. ” We love you so much that we must kick you around.”

  12. The only top performers in education will always be the reformers with their high minded abstract speeches and moral demands.

    The work of the math and science folks will always be held in low regard and treated with suspicion.

    The problems in this country run much deeper than a poor quality teaching force.

    Wage scales in education say it all. Math and science teacher DO NOT MATTER. Sports and easy electives do.

Leave a Reply

Your email address will not be published.