Are the Chinese cheating in PISA or are we cheating ourselves?


Today’s post, courtesy of educationtoday, is by Andreas Schleicher, in charge of the OECD PISA Programme and Deputy Director for Education and Skills as well as Special Advisor on Education Policy to the OECD’s Secretary-General.

Whenever an American or European wins an Olympic gold medal, we cheer them as heroes. When a Chinese does, the first reflex seems to be that they must have been doping; or if that’s taking it too far, that it must have been the result of inhumane training.

There seem to be parallels to this in education. Only hours after results from the latest PISA assessment showed Shanghai’s school system leading the field, Time magazine concluded the Chinese must have been cheating. They didn’t bother to read the PISA technical report, which shows there was no cheating, whatsoever, involved. Nor did they speak with the experts who had drawn the samples or with the international auditors who had carefully reviewed and validated the sample for Shanghai and those of other countries.

Others were quick to suggest that resident internal migrants might not be covered by Shanghai’s PISA sample, because years ago those migrants wouldn’t have had access to Shanghai’s schools. But, like many things in China, that has long changed and, as described by PISA, resident migrants were covered by the PISA samples in exactly the way they are covered in other countries and education systems. Still, it seems to be easier to cling to old stereotypes than keep up with changes on the ground (or to read the PISA report).

True, like other emerging economies, Shanghai is still building its education system and not every 15-year-old makes it yet to high school. As a result of this and other factors, the PISA 2012 sample covers only 79% of the 15-year-olds in Shanghai. But that is far from unique. Even the United States, the country with the longest track record of universal high-school education, covered less than 90% of its 15-year-olds in PISA – and it didn’t include Puerto Rico in its PISA sample, a territory that is unlikely to have pulled up US average performance.

International comparisons are never easy and they are never perfect. But anyone who takes a serious look at the facts and figures will concede that the samples used for PISA result in robust and internationally comparable data. They have been carefully designed and validated to be fit for purpose in collaboration with the World’s leading experts, and the tests are administered under strict and internationally comparable conditions. Anyone who really wants to find out can review the underlying data.

Short of arguments about methodology, some people turn to dismissing Shanghai’s strong performance by saying that Shanghai’s students are only good on the kind of tasks that are easy to teach and easy to test, and that those things are losing in relevance because they are also the kind of things that are easy to digitise, automate and outsource. But while the latter is true, the former is not. Consider this: Only 2% of American 15-year-olds and 3% of European ones reach the highest level of math performance in PISA, demonstrating that they can conceptualise, generalise and use math based on their investigations and apply their knowledge in novel contexts. In Shanghai it is over 30%. Educators in Shanghai have simply understood that the world economy will pay an ever-rising premium on excellence and no longer value people for what they know, but for what they can do with what they know.

PISA didn’t just test what 15-year-olds know in maths, it also asked them what they believe makes them succeed. In many countries, students were quick to blame everyone but themselves: More than three-quarters of the students in France, an average performer on the PISA test, said the course material was simply too hard, two-thirds said the teacher did not get students interested in the material, and half said their teacher did not explain the concepts well or they were just unlucky. The results are very different for Shanghai. Students there believe they will succeed if they try hard and they trust their teachers to help them succeed. That tells us a lot about school education. And guess which of these two countries keeps improving and which is not? The fact that students in some countries consistently believe that achievement is mainly a product of hard work, rather than inherited intelligence, suggests that education and its social context can make a difference in instilling the values that foster success in education.

And even those who claim that the relative standing of countries in PISA mainly reflects social and cultural factors must concede that educational improvement is possible: In mathematics, countries like Brazil, Turkey, Mexico or Tunisia rose from the bottom; Italy, Portugal and the Russian Federation have advanced to the average of the industrialised world or close to it; Germany and Poland rose from average to good, and Shanghai and Singapore have moved from good to great. Indeed, of the 65 participating countries, 45 saw improvement in at least one subject area. These countries didn’t change their culture, or the composition of their population, nor did they fire their teachers. They changed their educational policies and practices. Learning from these countries should be our focus. We will be cheating ourselves and the children in our schools if we miss that chance.

International comparisons are never easy and they aren’t perfect. But PISA shows what is possible in education, it takes away excuses from those who are complacent, and it helps countries see themselves in the mirror of the educational results and educational opportunities delivered by the world’s educational leaders. The world has become indifferent to tradition and past reputations, unforgiving of frailty and ignorant of custom or practice. Success will go to those individuals, institutions and countries which are swift to adapt, slow to complain and open to change. And the task for governments is to help citizens rise to this challenge. PISA can help to make that happen.

Useful links

PISA 2012

Guest author

6 comments to “Are the Chinese cheating in PISA or are we cheating ourselves?”

You can leave a reply or Trackback this post.
  1. Simon Goss - 11/12/2013 Reply

    It does need to be stated, however, that, unlike other countries, China’s results reflect a city, not the country, in this situation. A bit like Australia being able to nominate Sydney as a way of representing Australia’s performance.

    As an ademndum to this, why is PISA using a standardised test, which only encourages standardised learning, in a non-standardised world?

  2. Tim Werwath - 08/01/2014 Reply

    “..resident migrants were covered by the PISA samples in exactly the way they are covered in other countries and education systems.”

    I beg to differ. At the time of PISA 2009, there were almost 9 million rural-urban migrants living in Shanghai. Many are not considered “resident migrants”, but have temporary hukou instead.

    I conducted an analysis by looking at responses of Shanghai students in PISA 2009 to question 18 of the student questionnaire, which asked students explicitly if they were born in Shanghai. Over 99% of students reported that they were born in Shanghai. So where are the rural migrants?

    I know there are a lot of smart, dedicated people working on PISA, but at the end of the day, the “international auditors who had carefully reviewed and validated the sample for Shanghai” are going on faith that what the Shanghai PISA team has reported is truthful.

    I am conducting the same analysis now using the 2012 data and am finding very similar results. I am planning to present or publish on the topic later this year.

    • Yvonne LI - 03/03/2014 Reply

      When will you finish your analysis?
      Could you tell me your result when you finish?

      • xingfenzhen - 16/08/2016 Reply

        There is no analysis, because he is lying. However, from student questionnaire I noticed something interesting. namely ST27 series of question should probes learning strategies. Memorization, Elaboration and control.

        Sample survey

        Memorization: When I study, I try to memorize everything that is covered in the text
        Elaboration: When I study, I try to relate new information to prior knowledge acquired in other subjects
        Control: When I study and I don’t understand something, I look for additional information to clarify this

        Survey results
        Shanghai-China 0.64 0.72 0.72
        Japan 0.70 0.76 0.77
        Korea 0.73 0.76 0.82
        United States 0.76 0.81 0.82
        Sweden 0.70 0.78 0.74
        Finland 0.69 0.77 0.79
        So much for rote learning vs creative learning arguments….

        page 296

    • A Shanghai student - 18/04/2015 Reply

      I was a student from Shanghai.In 2009 ,I was 16-year-old student in HuangPu District.Even though I didn’t participate in the test in 2009,I seriously doubt that you have conducted 99% of the students are local.Shanghai is a migrant city which has 23 million citizens,about 14 million are local (their last generation are probably from Zhejiang or Jiangsu Province which next to Shanghai) and the rest 9 million poeple come from all over China.
      As far as I know,several of my friends attended the test in 2009.Most of them are Shanghainese but two of them weren’t.One comes from Changzhou the other comes from Wenzhou.And all of them said that it’s random to pick students to finish the test.They were very glad to attend the test because it’s so easy compared with routine test papers.Of course,this example can’t prove that your 99% conclusion is wrong but I think my personal experience is much more convincing than your ridiculous statistic which may occured in your dream.
      Let me tell you more,I have to admit that we Shanghai students are so poor in Maths compared with other provinces such as Shandong Hubei Henan,etc.Next time maybe 2015 or 2018 their performance will frighten your fragile heart.

      • xingfenzhen - 16/08/2016 Reply

        Well, from “leaked” Zhejiang results, it has 623 Math, 570 Reading and 582 Science; while Shanghai had 613 Math, 570 Reading and 580 Science. Aware that Zhejiang did not officially participate, so the schools are random sample as well (only 150 schools), however, from the report it appears the schools sampled tended to rank in mid-to-lower tier range and half of them are rural schools.

        Also if the much more difficult college entrance exam results from 2nd and 3rd tier cities are anything to go by, cities like Wuhan, Jinan, Fuzhou, Changsha etc will produce scary numbers compared to Shanghai. However, numbers of truly remote rural counties will likely drag the entire thing down. The same Zhejiang leak also included China’s 2009 averages from 22 provinces, with same caveat that schools are sampled. The national average are 486 reading, 550 Math and 524 Science. Respectable numbers, but much lower than Shanghai.

Leave a Reply