Keita Takayama
Rethinking the PISA Debate

Will Brehm  2:00
Keita Takayama. Welcome to Fresh Ed.

Keita Takayama  2:25
Oh, thanks very much for having me.

Will Brehm  2:27
In 2014, a group of academics and teachers wrote a letter to Andreas Schleicher, who’s the head of the Education and Skills division of the OECD and he was almost synonymous with the PISA test, criticizing PISA in fact. Who wrote this letter, and what did they say?

Keita Takayama  2:48
So the open letter was authored by two individuals. They are Heinz-Dieter Meyer, he’s a professor at the SUNY State University of New York, Albany and the other author is Katie Zahedi, who is a principal at Linden Avenue Middle School in upstate New York. And this letter was basically authored by these two lead authors and then endorsed by in total 84 signatories.

Will Brehm  3:17
And what the letter say?

Keita Takayama  3:21
So I think there are five key criticism of PISA that they lay out in the open letter. The first one is that PISA has promoted standardized testing, teaching to test and narrowing curriculum, and more multiple-choice testing and scripted curriculum in school systems. And the second criticism they make is that PISA has promoted short-term solution to climb up the ranking quickly. And the third criticism they make is that PISA has sort of promoted narrowing focus on employability, and then they argue that this reflects the OECD’s interest in economic development. And a fourth criticism they make is that the entire way in which the PISA has been conducted, it lacks the transparency in democratic participation. For example, why is it that teachers and parents are not invited to be part of the decision-making processes and so forth? And the last criticism, the fifth criticism they make is that they are concerned about the increasing role of multinational edu-business in the design of the PISA as well as the operation of the PISA. So these are the set of key criticisms that they develop in the open letter.

Will Brehm  4:38
And they concluded by saying that PISA should not be implemented any longer?

Keita Takayama  4:45
Any longer. So, the PISA needs to be immediately suspended until all these issues that they highlight are sorted out

Will Brehm  4:51
Across the globe. In every country where PISA is done?

Keita Takayama  4:54
That’s correct, yes.

Will Brehm  4:55
Okay. About a year later, in March of 2015, two other scholars responded to this open letter. What did this letter say, and who wrote this letter?

Keita Takayama  5:08
So the response was made by Pasi Sahlberg and Andy Hargreaves, two education scholars. Both are based in United States currently. So the response was entitled: “The tower of PISA is badly leaning. An argument for why it should be saved.” which was published in The Washington Post in March 24 in 2015. So in this response, basically they say that they agree with the criticism about PISA. But the PISA has also done some many good things for students in schools and societies around the world. And they argue that these good things should be acknowledged. So what are the good things? They basically highlight three good things that PISA has done to societies and education systems around the world. The first one is that PISA has helped countering what they call GERM, that is the global education reform movement. So what they mean by that is that there’s an education reform movement on a global scale happening that is guarded by this quasi market principle of school choice, competition, accountability, standardized testing. So all these what we call neoliberal education reform, which some would argue originate primarily from United States and the UK, now globally disseminated. But the PISA has actually helped us, or people around the world, to question this neoliberal education reform and then help us to develop alternative reform movements. So that’s one positive that they highlight. And then I guess as an example of the counter reform movement, they talk about what they call the “Finnish way to Finland”. So the PISA actually helps us to identify educational systems in the world that had not been recognized up to that point. And they talk about Finland and Canada. And in these countries, they argue that the kind of education reform that we see in the United States and UK are note there. Basically there’s not much emphasis on school choice, competition and standardization of curriculum and so forth, so something else is going on in Finland and in Canada. So the fact that we now know what’s happening in Finland and Canada is one of the positive outcomes of PISA.

And another thing that they talk about as a positive outcome of PISA is that PISA does have certain focus on equity issues. They argue that OECD has become a strong advocate of equity in education by reminding policymakers that the highest performing education system that is basically in Ontario education system in Canada, and Finland, as an example. So these education systems have combined quality with equity. So in some countries, PISA actually has helped them to bring equity back on a policy table, so these are the sort of positives that they highlight as a result of PISA. So to them – Sahlberg and Hargreaves – the problems are the following: First, many countries actually failed to treat PISA as just one of many indicators. So they are also concerned about the many countries try to climb up the PISA league tables as quickly as possible. And then they argue that PISA needs to be treated as just one of many indicators that the government can look at. And the government need to stop short term solutions and the global academic horse race. And Sahlberg and Hargreaves are also concerned about the increasing edu business involvement in the operation of design of PISA. And they also particularly concerned about Andreas Schleicher’s endorsement of American Common Core State Standards Initiative, which has been quite controversial in American context. And they think that it’s wrong for someone like Andreas Schleicher, the director of the OECD Education Division to be explicitly endorsing particularly controversial education policy in American context. So these are the sort of issues that Sahlberg and Hargreaves highlight.

Will Brehm  10:22
And they conclude that PISA has problems, but it shouldn’t be dismantled. It needs to be reformed.

Keita Takayama  10:30
That’s their conclusion, yes.

Will Brehm  10:32
And then a month later, a third letter appears in The Washington Post, basically saying that PISA should be dismantled. Who wrote this letter, and what did it say in detail?

Keita Takayama  10:46
So this refutation of the previous letter was written by Harris and Zhao. It was published in The Washington Post the 19th of April, 2015. Basically, they are trying to refute the rather a positive appraisal that Sahlberg and Hargreaves had made earlier by saying that, “Look, PISA completely disregard the historical, social, economic, cultural context and how they impact on student performance. So it’s actually not that useful.” And, I think it was Zhao in particular is concerned about the fact that East Asian countries, for example, achieve enormous success in PISA, and oftentimes their successes are explained in terms of particular policies and programs that they have implemented. But he would argue that, “Look, it’s the historical cultural factors that are driving their success. Namely, it’s the test craze and high academic pressure in those countries that are contributing to their success in a major way. So it is wrong and inaccurate to identify certain systemic features of their education system as if they are the factors that have led to their PISA success. So therefore because PISA doesn’t allow us to identify any specific programs and policy features that have contributed to a successful country’s outcomes, it is actually inaccurate; that it actually doesn’t allow us to factor in historical, cultural context of education and the success of some countries. So it is very dangerous to romantically talk about successful PISA countries, not just as East Asian countries, but also Finland and Ontario – those two countries that Hargreaves and Sahlberg talk about. So from Zhao’s point of view, for example, it’s inappropriate to talk about those PISA successful countries when in fact their success can be explained in terms of cultural, historical factors, as opposed to specific policies and programs.

Will Brehm  13:29
This open debate that’s been happening in newspapers since 2014 – you argue each of the letters, each of the authors, make certain unspoken assumptions in their argument. What sort of assumptions do you find in these letters?

Keita Takayama  14:01
So I have been closely following this debate for the last two to three years. But I have always been slightly uncomfortable with the way the debate has been developing. And I think, as you pointed out, there are certain unspoken assumptions embedded into the debate. And a first assumption I want to talk about is the fact that they seem to assume, rather and problematically, in a very uncomplicated way, the effect of OECD and PISA on national and subnational education policy making, as if there’s a direct correlation between what was tested in PISA and what gets implemented on national and subnational education policy. So for example, the open letter authored by Heinz-Dieter Meyer and Katie Zahedi, they seem to argue that the kind of negative effect of standardized testing that we know – that’s basically narrowing curriculum, teaching to the test and all those negative consequence of testing is now seen in educational systems in across the United States as a result of PISA. So one would have to question whether or not that’s actually happening United States and elsewhere. And even those who criticize the open letter, that is Sahlberg and Hargreaves, they seem to accept this direct impact of a Finnish way, or counter reform movement against GERM (the Global Education Reform Movement). That is, for example, they talk about how the kind of exemplary practices and policies identified in Ontario, Canada and Finland are now spreading around the world. So again, there is an assumption of uncomplicated processes of international or transnational knowledge transfer from one context to another, as if what happens in Finland, and how they are talked about, is now shaping education policies and programs elsewhere. So that’s to me one big assumption that seems to be underpinning the debate.

And the second assumption I want to talk about is the fact that that whole debate seems to be driven by particular concerns and policy issues that are specific to American context. I remind you that all the people that that we’ve reviewed up to this point, those people who participated in the debate, are US based. They might not necessarily be Americans, but they are affiliated with, and based in American research or educational institutions. And I think to a great extent, the kind of critique of PISA for example, as expressed in open letter, reflect the particular policy concerns and issues of the United States, not necessarily reflecting the actual consequences and effect of PISA in different parts of the world. So to me, these are the major underpinning assumptions of the debate that needs to be carefully critiqued.

Will Brehm  17:35
So what would be an example of where the PISA effect is mediated through domestic actors instead of simply being global best practices being transferred from one place to another without any consequence at all of the national actors?

Keita Takayama  17:55
So I’ll give you two examples here, drawing upon my own work out of Japanese context. For instance, when the results of PISA 2003 came out, which was December 2004, there was a major major shock and a crisis in Japanese education. What happened was that the ranking of Japan in reading literacy dropped considerably, from the eighth in 2000 to 14th in 2003, and the drop was statistically significant. But also there was a ranking drop of Japanese students in mathematical literacy as well from the first in 2000 to the sixth in 2003. Mathematics has three components within it, and in the first two components there’s no statistically significant drop. It was only in the last section of the mathematical literacy where a statistically significant drop was registered. Despite that, the drop from first to sixth seems like a significant drop, but from a statistical point of view, which was not so statistically significant drop. Anyway, so there were ranking drops in a mathematical literacy and reading literacy, for sure. And then what happened was the minister at the time really capitalized upon that ranking drops to basically initiate a set of highly contested education policies at the time, one of which was to implement national achievement testing, scholastic testing. And Japan has such a long history of political contestation over scholastic achievement testing in the past, and introduction of national testing was considered to be highly, highly politically charged and contested. But he was able to use the “crisis atmosphere” in the immediate aftermath of the PISA 2003 release, in order to justify the introduction of this highly controversial policy. Then what happened was that in that test, his emphasis was much more on competition and basics. So Japanese students need to learn more academic basics and they need to compete with each other. And along with the national testing, which was introduced in 2007; even earlier than 2007, the government introduced this policy called “morning reading hours”. Basically, kids are asked to spend at least half an hour every day before class begins, spend half an hour reading books. So presumably, this was in response to the ranking drop in reading literacy. But the thing is that any of these measures that are introduced in response to the PISA result, in particular the ranking drop in reading literacy, did not seem to help students improve their scores in reading literacy. Because asking students to do more reading doesn’t really help them to do reading literacy score. For them to be able to do well on reading literacy in PISA, they are to be introduced to a different set of questions; very different from the conventional Japanese literacy practice.

And also forcing students to compete with each other and putting more emphasis on academic basics doesn’t really help them achieve better in PISA either because PISA is really about application of knowledge and skills, as opposed to factual recall. So not only policies that were immediately introduced in response to this PISA crisis in 2004, they did not seem to be closely aligned with the curricular orientation of PISA. So therefore, this is one example where what’s promoted by PISA isn’t really translated into the policy set of programs on a national education system level.

Will Brehm  22:47
It was the domestic actor who used PISA to advance the agenda that he or she wanted?

Keita Takayama  22:55
Yes, and those agenda are not closely aligned with PISA at all. And I could give you another example, again drawing upon a Japanese case here. So I looked at the way in which Finland became a national sensation in Japan. A lot of policy makers, the bureaucrats, and the education scholars and educators flocked to Helsinki to learn about the secret of Finnish success after the first two rounds of PISA. Not just in Japan, in many countries a lot of people basically did a pilgrimage to Helsinki to learn about the secret. And what happened in Japan, what I did was that I looked at the way in which Finnish PISA success was represented in the media. So I basically tried to read every single article – journal articles and newspaper articles – about Finland and explaining why Finland was so successful in PISA. And I paid close attention to different ways in which Finland education was represented along political lines. So first of all those on the center-left political line, they tend to talk about Finland a lot, but in a very specific way. And oftentimes they say, “Well the success from Finland is because they learn from Japan.” And they particularly highlighted two institutional features of Japan and they argued that that’s what Finland learned from us. And the first one is the fundamental law of education, which was under serious criticism at a time and which was in eventually revised in 2006. But at the time, some argued that Finland learned from the egalitarian and democratic natured principles of the fundamental law of education of Japan. And so the argument was that: “Well, the greatest education system in the world actually learned from this fundamental law of education so why are we trying to revise it? Why are we trying to change it?” So in a sense, the representation of Finland and Finnish PISA success was utilized to counter the conservative political movement at the time to revise this policy.

Another Japanese key feature that people on the political center-left talked about is 6-3-6-6 single track educational system, which was again under attack at the time. And presumably Finnish education learned from this 6-3-3 single track education system of Japan to create egalitarian education system. So that’s how center-left people articulate, or represent, Finland. But on the other hand, people on the center-right present a very different, almost contrasting representation of Finland PISA success. So in their mind, Finland is characterized by deregulation, standardized testing in a competition and decentralization. So this representation of Finland education obviously helped them legitimize the kind of education reform that they want to see in Japan. That is much more market-oriented competition and standardized testing, and so forth. So, as you can see, in the case of Japan where Finland became a national sensation, the Finnish way has become a projector screen upon which any desirable reform measures can be projected upon. So here again, how national media mediate the information and data that’s generated by an international organization like OECD and then in a way that is almost completely unexpected. And so, the argument that I’m making is that the data and information that OECD and PISA generate is always mediated, always acted upon, by domesticating agents such as education scholars and journalist and pundits and the media, most importantly, to generate effects that are not necessarily aligned with what’s intended by OECD and PISA.

Will Brehm  27:56
So the PISA power isn’t so straightforward and unproblematic.

Keita Takayama  28:00
No. I would even argue that PISA can only exert its influence only when its data and information are acted upon by the kind of domesticating agents that I mentioned.

Will Brehm  28:13
And then, therefore, the power will look different in different countries at different moments in time.

Keita Takayama  28:19
That’s correct. Yes.

Will Brehm  28:20
So let’s step back from the intricacies of the debate and talk more broadly. For instance, the open letter argues that we see a homogenizing effect in education policy around the world. Do you think this is true?

Keita Takayama  28:40
To a certain extent, there’s no doubt that PISA does exercise influence on education policy around the world. But the question is whether or not the term homogenizing is the right word to use. To me, that’s an overstatement. Certainly PISA has considerably influenced the way in which we talk about education, and PISA does help us with the use of data and now how data nowadays become so central in our decision making around education. But at the same time, it is highly questionable to what extent the impact of PISA can be so straightforwardly understood.
As I mentioned, the effect of PISA is always subject to political manipulation and contestation. So there’s always room for maneuvering by domesticating actors.

So on one hand, I think the way in which we talk about education has certainly shaped by PISA and other international agencies. But at the same time, political agents within a national context, such as the Ministry of Education, for example, does maintain considerable power over over the data and information that international agents create, and they are in a position to determine what sort of impact that those data and information might generate on a national debate in education system.

Will Brehm  30:26
Why do you think a group of US scholars or US-centric scholars wrote a letter about PISA in 2014 when the test has been around since the early 2000s and hasn’t really made that big of an impact on, at least in the media in the US? What happened in the USA, basically, in the last several years?

Keita Takayama  30:47
Thanks for asking the question. I think that’s the crucial question: Why now? As you pointed out, PISA has been around over 10, 15 years and the US has been part of the testing from the get go. And why suddenly? And many scholars have looked at the silence of US media, US academics about the PISA. Up until 2011 or 10, hardly any media reporting on US performance in PISA. So no one cared. No one paid attention to the American students’ performance in PISA. And it was all of a sudden in 2013 and 14. Suddenly, there’s a considerable rise of interest in the US, so one ought to be asking, “Why now?” And to me, this has a lot to do with what happened domestically in American context. What happened? So there was, in 2009, President Obama announced this Race to the Top. Basically, this is a federal funding for state initiatives. But in that, states are encouraged to develop their programs around common standards and assessment plans. And then we know that Pearson was involved in the Common Core State Standards at the time. And then 46 states, in the application to Race to the Top funding, signed a contract with Pearson to develop Common Core standards and develop their own assessment plans.

So 46 states, and the first one was New York State in 2013. So it was around that time that Pearson started playing an important role in American education reform at the state level, getting involved in the development of Common Core and relevant assessment tools and education services that enabled state governments to implement what they promised they would implement in the application to Race to the Top. And it was around the time that the connection between Pearson and OECD became much more clear to many American observers. For example, it has become known that Andreas Schleicher, the director of the OECD education division, sits on the advisory board of Pearson. And in 2013, in a PBS interview, he explicitly endorsed Common Core State Standards and immediately after the release of the PISA 2013 data OECD stated, here I quote, “A successful implementation of the Common Core standards would yield significant performance gains, also in PISA.” So what’s happening here is that this highly contested initiative, that is the Common Core State Standards Initiative, it’s highly contested in two senses.
First, as I mentioned, there’s a private corporation involved in the development of Common Core initiatives. So that’s highly problematic. But also this is a state and it’s a federal initiative. So oftentimes this is construed as federal imposition of education policy on to what is traditionally considered to be a state prerogative, that is a state education system. So on these two grounds, the Common Core State Standards Initiative has always been extremely contested and controversial. And in fact, there was a protest, a walk out protest movement in upstate New York in 2013. Many educators and parents and kids walked out of school as a form of protest. But the OECD, particularly Schleicher, was getting more and more involved, supporting this highly contested policy in the American context. So in the eyes of the US observers, there’s a close connection between Pearson, private corporation, and OECD, represented by Schleicher and highly contested Common Core State Standards Initiatives. So, if you go back to the open letter and the criticism of PISA as expressed in the open letter, it’s clear that the criticism was driven by their domestic education context at the time. The particular concerns that they had about the Common Core standards and assessment tools introduced at the time and how Pearson was increasingly controlling educational system around the country in the United States. So, what I’m highlighting here is the debate around PISA, which has been defined as a global conversation, global debate, is actually has been quite provincial in the sense that the debate actually strongly reflects the particular policy concerns and issues at the time in the United States. And it’s also quite relevant here, the fact that the open letter was authored by two individuals who were based in upstate New York at the time where, perhaps arguably, Common Core State Standards and Pearson’s involvement in education policy was highly contested and challenged at the time.

Will Brehm  37:09
So it’s the state politics, the local politics, that matter and can explain more about what’s going on in these letters?

Keita Takayama  37:18
That’s my take, yes.

Will Brehm  37:20
Well, Keita Takayama, thank you very much for joining FreshEd.

Keita Takayama  37:24
Thanks very much for your time and listening. 

Want to help translate this show? Please contact info@freshedpodcast.com

Have any useful resources related to this show? Please send them to info@freshedpodcast.com