Will Brehm 1:27
Sam Sellar, welcome to FreshEd.

Sam Sellar 1:29
Hi Will. It’s good to be here.

Will Brehm 1:30
So, you have a new co-authored book that looks at what you call the “global education race”. What is the global education race?

Sam Sellar 1:38
So, the global education race is a metaphor that has emerged in newspaper headlines, political speeches, think tank reports. And it suggests that schooling has now become a global competition in which each country aims to surpass other countries in terms of their educational performance, and that they might worry about being left behind.

Will Brehm 2:00
And so, where is this race taking place? Like, who is hosting this race?

Sam Sellar 2:05
Well, we argue in our book that the OECD is now the most prominent host of that race, even if there are other events, like the IEA’s TIMSS assessment, for example. But it’s really the OECD assessments that have become the most publicly visible, and the rankings that they generate have really helped to consolidate this idea that there is a global race in education.

Will Brehm 2:25
And what about where these different countries are? Who’s winning this race, and who’s falling behind or losing this race?

Sam Sellar 2:32
That’s a complicated question without a straightforward answer. Countries like Finland, and countries in East Asia, have performed very well in the race, and a lot of attention has been directed towards them as a result. But I would argue that you can’t really win a race that you don’t want to run. So what matters is that you’re running in the right direction, that you’re in the right race. And so, being at the top of the rankings might be considered winning if you feel that having the best estimate of what 15-year-old students in school know, on a given day, about content that the OECD thinks is important in terms of economic growth. If you think that’s the most important goal in education and you’re at the top of the rankings, then maybe you’re winning that race. But if you hold broader aims for education, then this is not the only race, and it’s probably not the most important one.

Will Brehm 3:22
So, are there countries that have that specific goal of winning this very kind of narrowly defined race as operationalized by PISA?

Sam Sellar 3:30
I think it’s certainly an important goal for some countries in terms of the economic growth within the country and their economic competitiveness globally. But I think most countries, most school systems, would also hold broader aims for what they hope that students learn from school.

Will Brehm 3:48
I know, in a lot of the media that we see, it’s usually a rat race, and it usually comes out in these rankings, where in one very clear box we can see who’s winning and who’s losing. And so, it almost seems like we don’t really have a conversation about these broader aims of education.

Sam Sellar 4:07
Yes, that’s right. I think the race metaphor tends to focus us on the very simple issue of who’s ahead and who’s behind. And I guess that can narrow the conversation and take away oxygen from other kinds of debates that we might have about educational values that we hold – the purposes of education, and so on.

Will Brehm 4:28
So, what is this perceived benefit of joining the race, of participating in PISA testing, by each country?

Sam Sellar 4:36
I think countries participate in PISA and other large-scale assessments for a lot of reasons. So, the official justification – the OECD position – would be to emphasize a rational view of policymaking based on evidence, and then being able to borrow ideas from high-performing systems. So that’s the official line, I suppose. Now, I think it’s important to keep in mind that PISA is effectively a measure of human capital; that’s its kind of primary purpose. The OECD is an economic organization, and they think about education through that lens. And so, PISA really measures the flow of human capital at the end of compulsory schooling in most of the countries that participate. So, one perceived benefit for governments is finding out how prepared young people in their country are to be productive workers in a knowledge and service economy. Now that often gets conflated with the much broader question about the quality of schools, or the quality of teachers in a particular system, or the efficacy of particular policies. But countries also participate for other reasons, including simply to be part of an international community, to acquire expertise in how to conduct assessments of this kind, and there’s a range of other reasons as well.

Will Brehm 5:49
And what sort of impact does this have on students and teachers participating in the PISA tests?

Sam Sellar 5:55
It doesn’t have any direct impact, and that’s one reason why we would be less skeptical about PISA than we would be about other standardized testing. What happens with PISA though is that it really shapes the nature of educational debate, often through the way it’s reported in the media, the way it gets taken up by education ministers and other politicians. And so, it starts to shape the way that education gets talked about and that can play into debates about education reform agendas, other kinds of policy changes that might take place. There might be blaming of schools and teachers if there’s bad results. So, in a diffused way, PISA can actually have an impact on life in the classroom and what goes on in schools, but it’s not a direct one in the way that other kinds of standardized national assessments or school level assessments might have stakes for teachers and students.

Will Brehm 6:52
Do you have an example of how this debate was shaped in a particular national context that you think is emblematic of the issue here?

Sam Sellar 7:00
Yes. A really good example would be the province of Alberta in Canada. So, Canada has a federal system, and it’s one of the countries that oversamples on PISA; so, more students sit the test across Canada in order to break down the results by province. So, we can talk about the results in Alberta specifically. Now what happened in Alberta was mathematics performance in the early rounds of PISA was very good and it gradually declined over time. Now, when the results were reported recently, it sparked the whole debate about a math crisis – in Canada more broadly, but in Alberta, specifically. And this debate was really pushed by a parent activist who became concerned about the PISA results. And it got taken up in the media as part of a series of blog posts. And it really started to incubate this idea that there was a mathematics crisis. Now the concern about maths was really closely linked to debates about curriculum within the province, but it was PISA that helped to spur this. You could go and talk to teachers in Alberta now and say, “Have you heard that there’s a maths crisis in Alberta?” and most of them would be aware of this conversation that’s been going on, but they might not know about PISA itself.

Will Brehm 8:15
So, I want to actually turn to the specific test of PISA. In one of your chapters in your new cowritten book, you look at some of the technical limitations of PISA. Can you talk a little bit about what you mean by these technical limitations?

Sam Sellar 8:28
Yes. PISA is very good technically, as far as tests of this kind go. The methods are very well established. Some of the best psychometricians that there are working on PISA. So, it’s a very sound test of its kind. But even so, there’s always going to be limitations. So, some of those are linked to things like the sampling process and the way that scores are estimated. And also, I guess, the way that rankings are reported and interpreted by people as well. So, I could say a few things about that if you like.

Will Brehm 9:05
Please.

Sam Sellar 9:07
I mean the first point to make, I think, would be that the scores reflect the performance of the students who were tested. Now that may sound obvious, but what that means is we can only really say something about the students who are in the sampling frame for PISA. So, PISA, rather than testing all of the students in a school system, which would be an unreasonable burden on schools and on students themselves, PISA employs a sampling method that requires only a few thousand students to sit the test. And that allows people to draw inferences from that smaller sample to a larger overall population. But the scores can’t tell us about the types of students who weren’t included in that sampling frame. And so, differences or anomalies in sampling across different countries has to be taken into account when we interpret the scores. I guess another point I’d make on this is that scores aren’t accurate for individuals or schools. So, PISA not only does it sample students, it also requires a lot of items – about 10 hours of testing – to really build up a robust estimate of achievement. Now we wouldn’t ask students to sit down and take a 10-hour test. So, what PISA does is sample items as well. So, students only take a small subset of the overall set of items for PISA. And that means that we can only give students a set of estimated scores for their performance on their test. We can’t give an accurate score for individual students. So, the scores are for the whole population – they’re not for a student or for a school.

Will Brehm 10:38
So, they’re for the abstract student, not the specific.

Sam Sellar 10:42
They’re for the abstract 15-year-old student in your country who is enrolled at school, that’s right.

Will Brehm 10:48
So, using this sort of information, these findings that come out of PISA, is there anything that we can actually learn about school systems and teaching and learning and quality?

Sam Sellar 10:59
I think the potential usefulness of PISA depends on looking beyond the national scores and the rankings in the first instance. If that’s where the conversation starts and finishes, then there’s not a great deal to be learned. But if we start to unpack the data and look beyond those high-level scores and rankings, then I think there is something that can be learned from PISA.

Will Brehm 11:22
Such as?

Sam Sellar 11:24
Well, we’re a little bit hesitant in the book. We make the point early on that the book doesn’t offer any simple solutions. So, we don’t have a kind of a general answer to that question – “All countries can learn X from PISA, and they should look at this aspect of the test to find out something of relevance to them.” The point that we would make is that PISA is a useful place to start a conversation about what the data can tell us. It’s not very useful if it’s a report card and treated as the final word on the matter. But if it throws up some interesting differences between countries, or let’s take the case of Canada, again – a federal system – some differences between states or provinces in a system, then I think you can start to unpack the data and ask questions about why there might be differences between schools in different parts of the country, for example, and investigate that further. And if you can make a good argument about the factors that might have led to that difference, then I think you can learn something important.

Will Brehm 12:20
Are there any things you think we definitely cannot learn from PISA when it comes to schools and quality?

Sam Sellar 12:28
I think that’s an easier question to answer in a more general way. I think it becomes very difficult to learn about the differences between countries in terms of what students learn from school. So, PISA doesn’t aim to measure how well students have learned what school teaches. And so, we have to be very careful about looking at the rankings and jumping to an assumption about the quality of schools in different systems. Unfortunately, that’s one of the ways that PISA often gets read and reported in the media. I think that’s something we should be very careful not to do. So, there’s actually been a number of quite interesting studies that have shown students from very similar cultural backgrounds, for example, students from Chinese cultural background, who have studied in different countries – and I’m thinking of studies here that have looked at Chinese students in Shanghai, and Chinese students in Australian schools, who share a cultural background, but have gone through different school systems – perform comparably. So, what that tells us is that their performance might say more about cultural factors, family background, family values around education than it does about the quality of the school system in which they have participated.

Will Brehm 13:45
So, it seems that one of the values of PISA is the ability to do cross-national comparisons. What do you think PISA tells us about the ability, or perhaps some of the pitfalls, of doing these sorts of comparisons?

Sam Sellar 14:00
I think the question is, “What does PISA enable us to compare, and what should we be wary about comparing?” So, I think we’re on pretty safe ground if we claim that PISA lets us compare how well students in different countries are prepared to perform on this test, particularly if we understand preparation as including a whole range of factors that shapes students’ abilities. So not just what gets taught in schools, but the whole experience, I guess, of living in, and growing up in a particular society. So, students in one country perform really well in maths compared to another country. We can look around for reasons why that might be, and we could debate the educational, social merits of factors that we identify. So, I think it can help us to talk comparatively about students’ performance as defined by this test in different countries. But it gets pretty complicated from there. So, the main issue is that making valid comparisons depends on a shared definition of what’s being compared, and then being able to measure that thing consistently across different languages and cultures. And there’s a whole lot of debates about this, you know, ‘Is literacy the same thing in Anglo countries as it is in Japan?’, for example. Is the concept itself constant across those two different cultural contexts? When we translate items from one context to another, can we hold constant the construct that we’re trying to measure? There’s no evidence to suggest that that’s a problem with PISA, but there is certainly debate amongst psychometricians about whether this sort of thing is even possible.

Will Brehm 15:41
So, you’re saying that there’s no research right now, or there’s no evidence to suggest, that there are difficulties in establishing common definitions across the many countries that take the PISA test. Is that what you’re saying?

Sam Sellar 15:56
There’s a lot of evidence of difficulties, and evidence of test items performing differently in different contexts. But PISA puts in place a lot of mechanisms for trying to make sure that those differences don’t actually have an effect on the estimation of scores. PISA items are written in English and French, and then they get translated into the languages of the countries that participate in the test. And in some cases, that translation can make items more or less difficult in other contexts. So even if you get the translation technically correct, there could just be something about the nature of the language into which it’s translated that could confuse students and cause them to perform differently on that item than you would expect. So, the OECD takes care to remove those items if they function differently in order to reduce the effects that context or cultural translation would have on what’s being measured. But the difficulties of ensuring that the same thing is measured in different languages can’t always be easily overcome. And there would be people that argue that “Is it possible for us to say that we’re measuring exactly the same construct or trait in cultural contexts that are in some senses very different?”

Will Brehm 17:20
That seems to me to be a huge issue affecting the ability for comparison.

Sam Sellar 17:24
I guess it limits what we could say that we are comparing, and I guess that’s the point that I wouldn’t want to make. That comparison is complicated. We certainly shouldn’t just look at high-level scores and then jump to a conclusion that a particular school system is more effective at teaching science or mathematics, and that other countries should borrow those policies. I mean I think you need to be very sensitive to context, and comparisons need to be made with care with these kinds of limitations in mind.

Will Brehm 17:53
But that’s a lot harder, isn’t it?

Sam Sellar 17:55
Absolutely, absolutely.

Will Brehm 17:57
If you are a national policymaker, I mean that is much harder to do: To actually begin to understand another culture and context so sufficiently that it helps you make educational decisions at home. I mean that just seems like perhaps an impossible task to ask for policymakers. Whereas the alternative of just taking these high-level comparisons seems to be a lot easier. And obviously, I think a lot of policymakers are taking those on board in their decisions.

Sam Sellar 18:32
Yes, and I guess that’s one of the aims of our book: is to try and complicate the thinking about that a little bit. So, I would agree with you that it’s both, perhaps not feasible for policymakers to make those sensible sorts of comparisons and go and simply borrow policy ideas. We think PISA can start conversations and start debates, but it’s probably not helpful if it leads to quick solutions and simple solutions to policy problems. But PISA also gets used to put pressure on education ministers. This is one point that we do make that it’s not a high stakes test for students and teachers, but it has become high stakes for education ministers, who in some cases feel like it’s almost a report card on the work that they’ve done with the education portfolio. And I think by making clear some of these difficulties and limitations, we would hope to kind of reduce some of the pressure that might get put onto ministers by assessments like PISA to show them just because you’re ranked at a particular point doesn’t mean that you should jump to a conclusion about your performance as a minister or the performance of your schools, or that you should jump to a hasty conclusion about the kinds of policies that you should implement in your system.

Will Brehm 19:51
So, if you were to give advice to some ministers on how to use PISA – so, not to jump to conclusions, but what would be the more, in your sense, appropriate way, of using PISA for a particular context?

Sam Sellar 20:04
The first thing I’d say is don’t worry too much about it. Reject the suggestion that it’s a report card on your ministerial performance. The stories that get told about PISA in the media are not always the stories that the data itself tells. And so, I think ministers should resist the pressure of headlines about PISA setting the agenda to which they have to react. I think they could help to raise the standard of debate about assessments like this by rejecting invalid interpretations or uses of the data. You don’t need a lot of understanding of how the test works in order to be able to push back against some of the more simplistic stories that get told about this global education race. There are right and wrong ways to use the data. So, if the stories that get traction are based on wrong ways of using the data, then I think that should be called out, and I think education ministers are one group that has the ability to do that.

Will Brehm 21:05
Who interprets the PISA data? I mean it seems like if, for a minister to reject the race and saying that the media reports don’t necessarily match the findings of the data. But who is actually interpreting this data and feeding these findings to the minister?

Sam Sellar 21:24
That’s a good question. I guess the minister; there’s a number of channels through which education ministers hear about the results of PISA. So, countries have access to the results before they’re made public with the release of PISA. So, they have a little bit of time to think about how they’ve performed and how they might respond to the public release of results. So, they receive the information about the scores from the OECD, but then once the results are publicly released, it may create – it doesn’t always create, but it may create – a whole set of headlines and debate in the media. And it’s at that point that I think ministers will feel the pressure to have to respond to the results, particularly if they’re seen to be bad results. If the score has declined, or if you’ve performed particularly below other countries that you think you should be performing better than, or comparably with, then that can create pressure on ministers.

Will Brehm 22:27
It seems like the OECD has a huge amount of power. So, it’s the organization that gives the test and is also the organization that is interpreting the test for national ministers of education. I mean that seems like a huge concentration of power in an international body.

Sam Sellar 22:46
Yes. I mean the point that I always make is that the OECD is the countries that it comprises as an organization. So, the OECD is an intergovernmental organization; it’s made up of its members. When it comes to PISA, it has a PISA governing board that oversees the assessment. That includes all of the OECD member countries – representatives from those countries – as well as representatives from other countries that participate in the test. So, it should be the case that countries participating in PISA, particularly the OECD countries have the ability to set the agenda around the assessment. We would certainly argue that government should be taking back more of that agenda in terms of how they use the data and how they respond to the data. I think we’ve reached a point where there’s a very savvy strategy for releasing the results publicly. It’s a kind of an attractive sell to media because you have these rankings. The media put pressure on the OECD around the rankings as well, so I would make the point that the rankings are a very small part of the PISA report. And that’s a point also made by the OECD, that the rankings are not the most useful aspect of this. But of course, the OECD has built the PISA brand on the impact that it’s had. If we didn’t have this idea of a global education race, and rankings telling us who was winning or losing, PISA wouldn’t be as visible as it is, and it wouldn’t have had the impact that it’s had. So, it is the case that the OECD has a large degree of influence in terms of how it can shape the kind of global reporting of performance on the test.

Will Brehm 24:37
Do you think the OECD and PISA are here to stay for the foreseeable future?

Sam Sellar 24:43
I think the OECD is. I think PISA probably is too. It’s well established now. I don’t see any signs that PISA itself would disappear anytime soon.

Will Brehm 24:58
Do you think PISA itself would adapt to some of these critiques that the academic community has uncovered, and perhaps some of the limitations in the ability of … usefulness of PISA?

Sam Sellar 25:09
I think there’s signs that it has. I think the OECD does pay attention to good academic critiques of the work. The technical work of PISA, as I said, is done by very good psychometricians and experts in educational measurement. I think it’s sort of hard to fault the technical side of the test in many respects. But I think some of the critiques that have been made about sampling in particular countries, the OECD is sensitive to that, and responds to those critiques and tries to address the issues that are raised where that’s possible. Whether or not they actually address the critiques is another question, but I would say that they’re certainly sensitive to them in some cases.

Will Brehm 25:57
Right, so PISA is here to stay. And I guess, like you said, there is some value to what is happening, I mean what these tests can show. But it’s much more than the rankings. You have to dig a little bit deeper to really get a good understanding of what’s going on in these different countries.

Sam Sellar 26:15
Yes, that’s right. I mean a lot of the critiques of PISA I think really focus on the misuse of the PISA data. And that’s certainly something that happens, and it can have quite negative and perverse effects. So, I would certainly agree that PISA can have quite negative effects. But it’s a relatively low-cost, low-impact test, in the sense that it doesn’t take up a huge amount of time for students and teachers. It doesn’t have a direct impact on a lot of students and teachers. It can’t really be used for accountability purposes in kind of punitive ways, with teachers and with schools. And it does give us a set of data that we can use to have an international conversation about education, about its purposes, about differences between countries. And I think if it’s seen in that light, having a robust data set that can start these kinds of international conversations and doesn’t have a really large impact within systems is a useful thing. The problem begins when it gets amplified through the media. We get very simplistic stories about crisis, or about decline. These become detached from their basis in the evidence itself, and then get used to justify particular policies or particular reforms that might not have a strong basis in the data. Or there might not be a strong justification for the particular approach to reform that’s taken, and PISA just becomes the excuse to undertake that reform. Or governments just react to PISA and set targets like were set in Australia: To be in the top five in PISA by 2025, which is a sort of a nonsensical goal to have, because the point isn’t about where you rank in PISA, it’s what you might learn from the data, and how you might use that as a useful tool in your context for the particular purposes that you have for your schools and your school system.

Will Brehm 28:23
Well, Sam Sellar, thanks so much for joining FreshEd. It was really great to talk today.

Sam Sellar 28:27
Thanks for having me, Will. It was a pleasure.

Coming soon.

Coming soon.