Today we continue the mini-series on global learning metrics. Last week we heard from Eric Hanushek about the desirability of large scale international assessments such as PISA. He argued that cross-national tests offer ways for countries to see what is possible when it comes to student learning.
But what effect are large scale international assessments having on national governments? In my conversation today, I speak with Radhika Gorur about how PISA, and its embedded assumptions about education, are going a global.
In our conversation, Radhika unpacks what it means to “see like PISA.” She finds three major ways governments around the world have embraced PISA.
First, governments have assumed that the very purpose of education is to increase GDP, which is a cornerstone of PISA and the OECD. But of course education has many more values that are much harder to define.
Second governments have narrowed the field of vision of the meaning of education to be in line with what PISA has been able to test. In effect, we only talk about what we can actually measure on the test, missing so many other subjects and areas that are also important to education.
And the third issue she finds is that we now talk about an impersonal “Student” as represented by PISA. The many reports put out by the OECD talk about so-called “students”, but they are always abstracted and without color or context. Who is this so-called PISA “student” and why do states compare their young citizens to her?
Radhika Gorur is a Senior Lecturer at Deakin University, Australia, and a Director of the Laboratory for International Assessment Studies. She will speak at the inaugural CIES Symposium this November. The article discussed in this podcast can be found in the European Educational Research Journal.
Citation: Gorur, Radhika, Interview with Will Brehm, FreshEd, 44, Podcast audio, Oct 3, 2016 https://freshedpodcast.com/radhikagorur/
Will Brehm 0:46
Radhika Gorur, welcome to FreshEd.
Radhika Gorur 2:01
Thanks, Will.
Will Brehm 2:03
You have a new article in the European Educational Research Journal entitled: ‘Seeing like PISA’. What do you mean by the term ‘Seeing like PISA’?
Radhika Gorur 2:14
If you Google PISA, the first entry that shows up is not the Italian city, which has been in existence from the fifth century BC, but the OECD’s education survey, the Program for International Student Assessment, or PISA, which was started as recently as the late 1990s. When PISA rankings are released simultaneously around the world, like a Harry Potter book, it makes media headlines globally. PISA is the largest component of the OECD’s education budget. And countries like Australia have introduced the ambition of being in the top five in PISA rankings. So, that tells you a little bit about how influential and famous PISA has become. So, the simpler answer to the question, what is seeing like PISA is to say, PISA has become very influential and is shaping policy in many countries around the world, and policymakers are seeing through the eyes of PISA. But the more interesting answer, and I think the more important one has to do with what PISA epitomizes. PISA didn’t pop up from nowhere. And if it is flourishing, it’s because that way of thinking and seeing has become widely established in our governance practices. There are some key features of seeing like PISA. The first is that it’s a fiscal focus. That is, seeing education in economic terms as an investment, which is expected to yield good returns. So, this requires calculations. And you have to calculate to see if the returns are good enough.
The second feature is seeing the world in comparative terms. Once you start comparing, you become competitive. Education rankings have become very important. Everyone wants to be in the top five or top 10. Everyone at least wants to be above average, which is of course a mathematical impossibility. But this sits well with the OECD’s commitment to the market, and more generally to the neoliberal focus on choice and competition and marketization that characterizes governing styles in many countries today. So, to compare, you also need to standardize. You need to have some standard indicators against which to pitch different countries. So, when you try to standardize and render measurable, the quality and equity of school systems, you inevitably are going to get some reduction. It is not easy to render quantifiable, a lot of the qualities that we might consider important in education. The fourth thing is the international nature of such comparison. When the scale goes international, you get a synoptic view, a view from above. It’s a kind of aggregated system kind of view. You lose the detail, you lose the within country differences and you lose the minorities, and the outliers in such accounts. You see things at a glance. Today there are a number of such comparisons in every field: health at a glance, education at a glance, pensions at a glance even. This ‘at a glance way of thinking’ obscures a lot of really important detail. So, seeing like PISA is not really just about PISA, it is about making the world legible in a fiscal reductionist, synoptic, and abstract and comparative way. So, I want to be clear that PISA is actually not the cause of seeing like PISA. PISA itself is a product of the seeing, like PISA phenomenon. PISA depends on the presence of a large infrastructure of data collection, statistical expertise, and government agreements, all of which have developed over decades and across many fields. At the same time, PISA also perpetuates this way of ordering the world.
Will Brehm 5:35
So, in your article, you use the book by James Scott, Seeing like a State as a parable in a sense to think through PISA and the rankings that we know so well in the field of comparative education. Can you just talk a little bit about James Scott’s book, Seeing like a State, and how that helped you think through seeing like PISA?
Radhika Gorur 6:02
Yeah. Well, the subtitle of Scott’s book is: How certain schemes to improve the human condition have failed. Scott’s book is about large state sponsored social engineering projects, which were designed to do good things. In the instances he details, what happens is that the projects themselves succeed spectacularly but what they fail to do is to improve the human condition. In fact, they end up being disastrous for the human condition. The opening chapter of Scott’s book is about forestry management practices in the 1700’s in Prussia and Saxony, and he details his thesis in this chapter. Now, you might wonder what the 18th century Prussian forests have in common with 21st century education. But as I read the chapter, I was struck by the close parallel between the two. So, let me explain about this. Scott’s story of Prussian forestry is about the state’s project of making forests more politically and administratively convenient to govern. Forests were really important to the Crown’s finances and for its security. So, managing them and increasing yield was a very important activity. So, the effort was to harness nature for economic purposes. This was the translation of nature into a natural resource. The first thing they needed to do was to measure the worth of their forests. How much timber was there? This was very difficult to calculate when the forest was so biologically diverse. From trees and shrubs to birds and insects. It is teeming with life. Most of it might have been useful actually, even invaluable, to the communities nearby, but it had little fiscal value. To make measuring easier, the Prussians took an exclusively fiscal view focusing single mindedly on timber yielding trees. They devised ways to categorize trees into five different standard types based on how much timber they would yield. Soon, instead of trees, they only saw planks of wood. Eventually, they translated the forest in all its diversity and beauty and usefulness, into numbers on ledgers. All the standardization and abstraction allowed them to get a synoptic view. Sitting in their offices in the city, they could see the distant forest translated into neat rows and columns of numbers right on their desktops. Of course, their desktops would have looked a little different to ours, I guess. The picture of the forest reflected in the ledger books looked so logical and convincing and convenient in all its simplicity that they started to actually plant forests in neat rows and columns with all the shrubbery cleared. This is a powerful and dramatic move. They planted the same species of tree in each forest at the same time, making planting and harvesting much easier. They’re very easily able to predict the yield of the fiscal planning. This style of forestry management became hugely successful and began to be taught in universities and to spread to many countries around the world. In the long run, however, this was disastrous. The practice of monoculture had depleted the soil capital and the biodiversity of the forests and made the forest susceptible to a variety of threats. But it was only about 80 years later that the adverse effects of these practices became apparent. Now, I see the phenomenon of seeing like PISA working out in the same way. We have the same fiscal view. If they translated nature into natural resource, we are translating humans into human capital. We have standardized students and made them abstract in order to measure and compare. Students from Korea and Kazakhstan are seen in similar terms. We see the narrow focus on literacy and numeracy because they are easier to measure. Actually displacing, in many places, the unmeasured subjects like dance and the arts and even the social sciences. We see the use of these numbers for governing. Nikolas Rose has talked about governing by numbers. We see the same management practices of governing by numbers trending in many parts of the world. What we don’t know is the effect all this will have in the long run.
Will Brehm 9:46
And what’s amazing is that PISA is only, what is it, 16 years old at this stage, and yet it seems to have so much power globally. We see it in newspapers in Australia, or in Germany, or in Japan, or in countries like Cambodia where they’re preparing for PISA 2021. How did PISA get to this point? How did they kind of capture so much power on the international scene?
Radhika Gorur 10:21
Well, if you look at the origin stories, there are a few different types of origin stories. One of them is about the growing anxiety in the Western world, especially during the later Cold War years, and after Sputnik and the US. That the Soviet Union was surging ahead, and that America faced an education crisis. And this narrative of education system in crisis seems to kind of spread everywhere. And the idea of human capital was also, at the same time, growing. So, there was increased budgets allocated for education. And this meant that there was more interest in calculating the yield, the return on that investment. Until then, there was a lot of statistics in education, but it was more focused on gathering input data, the numbers of schools, numbers of teachers, the student-teacher ratio, and so on. But I’m told that in the kind of late 1990s, the US really pushed the OECD into thinking about measuring outcomes. And at that time, TIMSS was already doing some sophisticated internationally comparable studies and international comparisons, actually. And they were looking at mathematics and science. This was started by a group of academics with a focus on learning rather than economics. So, when the Americans pushed the OECD to develop some outcome measures, TIMSS was actually initially considered for this purpose. But the story goes that there was a personality clash between some important members of IEA and the OECD. And that OECD decided that it will produce its own test. And that’s how PISA was born. PISA was developed very quickly in just a couple of years. So, you can say that it came out due to American pressure to develop measures for education outcomes during the time of growing neoliberalism, and the growing economic focus on human capital. In terms of how it actually expanded, so it initially started with just its own member nations. And by the next round, a few other countries joined. It’s a voluntary thing, countries can ask to join PISA. And so now it has about 74 systems. I think there were some systems that it was keen to include. I think it was very keen to have India and China in its surveys. But in most other cases, in fact, initially, it was not even particularly interested. I remember interviewing one PISA official, and he was saying, “I don’t know why Peru joined, you know, what can they learn from PISA, they can learn nothing, so I don’t see why they wanted to join”. So, initially, they were not looking to expand. It was more about other countries wanting to join and see themselves on the world league tables. But I think that has changed now. And I think OECD is now actively looking to go to other places and to expand PISA. Sort of few other types of PISA that are coming up. So, you have PIAC, which is like PISA for adults, which has already been trialed a few times, actually been implemented a few times. We’ve had a few rounds of that. Second round of surveys, I think recently. Then we’ve got PISA for development, which is happening, it is going to be trialed next month in eight countries. You mentioned Cambodia earlier, including Cambodia, Honduras, Ecuador, and so on. And you are now hearing about PISA for babies, which apparently is a new initiative that’s coming out for five-year-olds. There’s also PISA for schools. I think Bob Lingard in a FreshEd podcast has talked about that earlier, which is where individual schools can be tested, and they can be compared against international rankings. So, there’s a lot of initiatives that are going on now.
Will Brehm 14:04
So, I mean, it sounds like PISA is ready to test anyone at any age from birth till death if they could.
Radhika Gorur 14:11
The term is cradle to grave.
Will Brehm 14:14
Cradle to grave. Yeah, I mean testing from cradle to grave. And so, does PISA do anything other than just provide testing?
Radhika Gorur 14:23
Yes, well, other than its tests in mathematics and science and reading, they’re actually called literacies. They talk about mathematical literacy, reading literacy and scientific literacy. Apart from that all students fill in a student survey, which is a background survey. So, they ask questions about their parents’ income, their parents’ education, and their home possessions. And these three groups of indicators actually give OECD some idea of their cultural kind of capital. And the schools also, the principals fill out a survey and that gives them an idea of the school’s administration, the schools kind of practices. And the OECD uses these two other measures as well in its analyses. The other things that some countries have used is, I think, the two new tests in problem solving and financial literacy, some countries have opted to use that as well. And some countries also include a parent survey. So, this is kind of the suite of surveys that are part of PISA. And what happens is that OECD can then do analysis like these particular school administration practices are linked with this kind of performance. So, that’s the kind of stuff they can do. They can also look to see whether in some countries, socio economic advantage is closely linked to performance or not and they can talk about whether systems are more or less equitable on that basis.
Will Brehm 16:02
So, this data that comes out from PISA, and some of the analyses that the OECD does about more or less correlations between different factors, do governments or do schools, how are they affected by the results of PISA and their participation in the test?
Radhika Gorur 16:28
Well, I don’t know too much about the PISA for schools because that’s actually a new initiative. And in fact, I’m not at all sure that there’s actually much. I’m not a statistics expert but I know that when I went to the OECD and I was interviewing people there, within OECD itself, there’s a lot of skepticism about being able to apply a test that was designed to do a large system kind of level assessment to be able to actually be sensitive enough to pick up things at a school level. So, let’s not worry about the schools because it’s a small initiative and only in the states. But otherwise, it’s actually very difficult to talk about effects of PISA. In fact, when I started my PhD, that was what I’d set out to examine, how is PISA affecting Australian policy. And it’s impossible to do that. You can’t really link something that Australia is doing directly to PISA. So, that’s why I actually prefer to talk about seeing like PISA, which provides me a different way to think about the collection of practices that actually characterize this particular way of seeing and knowing and governing, rather than thinking PISA made them do this. So, there’s been actually some research on the influence of PISA on national policies. Breakspear actually, I think that’s an OECD publication actually found that PISA results have an influence on policy reform in the majority of the participating countries and economies. And there’s another report, I think it’s by Hopkins and others and they talk about the policy impact of PISA, and they said 85% of policymakers, local government officials, academics and researchers report having a relatively high level of knowledge of PISA processes and impacts. So, it penetrates the policy kind of arena quite well. And they also asked them, in what areas do you think it is most likely to affect policies, and that’s quite a nice list. So, they say that it is most likely to result in the development of national standards, in the establishment of national institutes of evaluation, in changes to the curriculum, or the introduction of national curricula, and the introduction of educational programs targeting specific groups of students, and an increase in the allocation of resources to school. And also, they say that it might result in increased collaboration amongst key stakeholders of education systems within and across countries. So, these were some of the things that the people that they talked to actually reported as likely to be the effects of PISA. So, OECD also advocates increased accountability, as well as autonomy. So, this is leading to centralization of power and increased national monitoring and testing. So, in fact, that’s in the list as well, the development of national standards. We see that happened in Australia, not only national curriculum but also a national test. And a new body was set up called ACARA (Australian Curriculum, Assessment and Reporting Authority) which is like a central body. So, it used to be that states were responsible for education, and they still are, but there’s a lot of federal involvement in education now. That’s very apparent in Australia, and it’s not exclusive to Australia. So, this increased national testing also leads to this idea of ‘steering at a distance’, which is a term I think that Stephen Ball coined. And then new national schemes have been introduced in many places including exit exams because that’s one of the things that OECD says is important for better PISA results. And then there’s also this idea of learning from high scoring nations, which is fraught in many ways, actually. So, first, there’s nothing to say that the practices adopted in high scoring nations actually cause the nations to score highly, because those same practices actually could be happening somewhere else. It’s just that we go to Finland, or we go to Shanghai and look at the practices there. But some other countries might be having the same practices and getting very poor results. And in fact, we also don’t know whether those high scoring nations if they use other practices, they might have had even better results for all we know. So, it’s a very poor way of thinking about it but that’s quite a rampant phenomenon, this idea of learning from the best.
Will Brehm 21:00
And so, some of the school systems or countries that are adopting or adapting their educational systems based on these collections of practices that are being articulated by PISA, are some of these countries experiencing better results on PISA in the future after having adopted some of these new practices that we’re seeing as best practices by the OECD?
Radhika Gorur 21:47
Actually, my friend Margaret Wu, she has a wonderful graph, in which she shows how English-speaking nations are all experiencing declining PISA results. In Australia, there’s a lot of hype about slipping both in relative and absolute scores. We introduced our national test called NAPLAN, in 2008. And we’ve not been showing any growth at all, in those as recently. Over the last few weeks, our radio airwaves and press has been full of that. So, I can’t be sure that we can attribute changes to policies and practices directly to PISA but certainly national assessments in many parts of the world, especially in the US have resulted in the widely reported phenomenon of the narrowing of the curriculum. Berliner and others have talked extensively about that. And also, the children getting either bored or not getting the best education as teachers focus more and more on getting better results on the tested subjects. Because in the States, in particular finance is depended upon that, schools are getting closed or amalgamated and so on. And this also led to kind of, in some cases, a terrible tragedy where teachers started to cheat. And in fact, schools and whole school districts actually started to cheat. Such a travesty, that the very things that education should be giving a sense of ethics and so on, were actually being challenged because of test scores. So, OECD actually doesn’t link, if you think of how does OECD respond to this, it doesn’t associate itself in any way with this idea of narrowing of the curriculum. Because PISA sees itself as a rich application-based assessment. It says it’s not linked to the curriculum, it’s an application of what students know. And so, it says that if students can do well in PISA, it doesn’t mean that you have to narrow the curriculum. And unless they have a rich curriculum, they can’t do well on PISA is OECD’s argument. So, PISA is exploring how to expand it’s suite of tests. And so, in addition to mathematics, and science and reading, it’s adding new things like problem solving and financial literacy. And it’s also working on other so called 21st century skills, like teamwork. And I’ve heard about some initiatives, I think, to do with global capabilities, but I’m not too sure about that.
Will Brehm 24:15
So, I’d like to return to ‘Seeing like a state’ and the idea that you put forward seeing like PISA. In ‘Seeing like a state’, one of the main arguments was that high modernist central planners who tried to design society based on scientific laws ultimately failed. And I just want to know, do you think Andrea Schleicher, the head of the OECD PISA division, do you think he exhibits some of these high modernist ways of thinking that Scott critiques?
Radhika Gorur 24:57
Okay, so I think there’s some parallels and there are some worrying developments. The first is that with projects like -surprisingly, I’ll be shifting attention from OECD, actually to the big UNESCO Sustainable Development Goals that were instituted recently. So, after the Education for All initiative finished in 2015, they did some evaluations of the success of Education for All. And they found that, globally, there was still, I think, some 250 million or so children that still couldn’t read or write, and that half of them had been in school for four years or more. So, they are now shifting attention to the idea of actually monitoring the quality of schooling, which, earlier, their focus was also more on inputs, on the provision of schooling, on whether there were enough teachers, etc. So, this refocusing or shifting of attention to the quality of schooling has created all sorts of new initiatives now. First of all, it’s given OECD an opportunity to put up its hands and say, pick us we know how to do this stuff, we know how to do this outcome measurement stuff. So, earlier, OECD was mostly interested in its own member countries. But now, it’s really quite keen to enter all the low-income nations as well, or the so-called Global South. Earlier, UNESCO had a different set of regional assessments for the poorer nations of Africa, Asia, and Latin America. So, OECD was looking at the richer nations, UNESCO’s look at the poorer nations and there were two different kinds of approaches to this whole idea of measurement and in fact, governance as well.
Now OECD, UNESCO, UIS and agencies like Pearson and ETS, and Pratham, and the Brookings Institution, and you know, all sorts of very, very different institutions are coming together. They’re all consulting each other, they’re forming alliances, they’re working together. It’s really interesting. There are IGOs, NGOs, for-profit ventures, think-tanks, and they’re all working together on global metrics as part of the education 2030 program that UNESCO has instituted. The fact that these very disparate organizations are working together may mean that there are fewer checks and balances, and that there’s less critique. So that, you know, earlier, it used to be like two camps, and at least there was some kind of critique back and forth. And this kind of coming together into one really powerful kind of global amalgamation, that I find a little bit worrying. And there is a global agenda. There is in fact, a global agenda that is framed in Education 2030. And this is very much focused on measuring and monitoring. There’s a very strong -because I think there’s so much aid money that’s being poured into this project, they want to be sure to hold the countries accountable. So, they’re very strongly bringing in a lot of monitoring and accountability measures. And that is how OECD is also going in. So, PISA is trialing PISA for Development in eight low-income nations next month. And this is a modified version of PISA that can be linked to the main PISA for the purposes of comparison. And there is an idea that before long, there will be over 100 nations that will be doing PISA. And sometime back, PISA set up PISA for schools, as I told you earlier, and there’s also PIAC, and there’s PISA for babies now. So, I don’t like conspiracy theories, but it does seem like there’s an expansion of this, and the fact that so many organizations are supporting this, I think makes it quite a big project. A bit Seeing like a state type of project.
Will Brehm 28:53
Right. I mean, it seems like if we’re moving towards the development of one global matrix in education, or one way to measure educational test scores and outcomes. And all of these different actors are, in a sense, buying into this system of thinking. If we take Seeing like a state’s argument seriously, then it would suggest that we are kind of sowing the seeds of failure into this system.
Radhika Gorur 29:30
Well, it’s hard to say. With the Prussian forest, anyway, it took 80 years to know whether this was going to work or fail. And for the longest time, it was a great success. And, in fact, it’s so much of a success, that it spread everywhere in the world and became as I said, taught in universities. There are forests in India that are planted in that style. And it’s only after 80 years that they realized that this is not working very well. And, that the effects were actually irreversible. So, the Germans, they had to respond with all sorts of new measures to kind of compensate for that. So, we don’t know with PISA because the first batch of PISA test-takers would be in their early 30s now. And it would have taken countries a few years before introducing significant changes following their PISA results, and even longer for those changes to manifest in any kind of statistically significant ways in student performance. And even longer for those students who take up jobs and become senior enough to influence the world in significant ways. So, actually, we can’t tell what the effects are, whether they’re positive or negative. But that’s my point. We actually don’t know that it’s actually a good thing. And for us to be going wholesale into this thing without an alternative approach or thought. I think that is why I felt compelled actually, to write that paper. It’s a bit more pulpit-ish. That’s not my style usually. I don’t stand on a soapbox usually but with this, I really felt compelled to write that because I felt that you know that there is something very big and global going on. And we should actually not all blindly kind of shut our eyes to other possibilities or other ways of thinking or knowing.
Will Brehm 31:26
So, I guess we’ll have to wait another 80 years before we know what ends up happening here.
Radhika Gorur 31:31
Maybe not 80, but at least I think another 30 years.
Will Brehm 31:36
Well, Radhika Gorur, thank you so much for joining FreshEd today.
Radhika Gorur 31:39
My pleasure. Thank you, Will.
Coming soon!
