Laura Engel & David Rutkowski
The costs of PISA

Will Brehm  1:52
Laura Engle and David Rutkowski, welcome to FreshEd.

Laura Engle  1:56
Thanks for having us.

David Rutkowski 1:57
Thanks Will, I really like the podcast and I’m happy to be here.

Will Brehm  2:00
How many countries participate in PISA, let’s say the latest round of PISA?

Laura Engle  2:06
Will, that’s a great question. Popularity in ILSES has grown over time. The numbers of countries participating in PISA have increased from 47 in the initial cycle in 2000 to over 70 economies in more recent cycles. Which is interesting is that entails about half OECD member countries and about half which are not member states of the OECD.

Will Brehm  2:30
And why would a country want to participate in PISA?

Laura Engle  2:33
I think countries have different motivations. I think some countries are motivated to participate as they want to be seen on a world stage. They want to be part of an assessment that’s taken by some of the world’s leading economies. Others might be compelled by international organizations like the World Bank, which are encouraging participation in international, large-scale assessments. Additionally, some systems may not have an advanced assessment mechanism at a national or sub-national level. So, international, large-scale assessments like PISA can become a viable, or in fact, even cost-effective alternative. One thing I think we know here is that PISA is different from other international, large scale assessments in terms of costs. So, for other international large-scale assessments, costs are one of the reasons why systems might not participate in the assessment. So, take, for example, the International Civic and Citizenship Study or the ICCS. The US did not recently participate in the latest round of ICCS due to budgetary constraints. But for PISA, OECD member countries, and those who want to become members of the OECD, they’re expected to participate in PISA. So, costs are not a factor for PISA.

Will Brehm  3:46
Okay, so before we get into some of these cost breakdowns and where the money is actually going, I want to ask sort of a technical question: How do countries actually participate in PISA? Like, what does participation actually entail? I mean, I understand it’s a test, but how do these countries actually go about administering this test? How do they do?

David Rutkowski 4:08
Well, first, I think it’s important to know the OECD self-governs the assessment. So, it’s OECD countries that make the assessment in the end, that vote on what is actually going to be on the assessment. Non-OECD countries, or the other countries, they don’t have that same say. And I think that’s a really important thing to think about. So, countries help develop the assessment and then it actually goes to the contractor who actually makes the assessment. But what do countries have to do? The countries have a few responsibilities. First, they have to draw the representative sample and that’s both in the field trial, which is about 1,500 students and do the main trial, which usually is about 6,000 students but that range really a lot depending on what the country wants to do. The countries also have to translate the assessment. And so, that’s going to be -the assessments written in either English or French- so, they have to do a translation from the assessment into their language. And then they have to actually give the assessment. If it’s a paper-based assessment, which was historically the case, they would have to take these paper booklets out to the schools, train the school leaders, or whoever’s administering the test, how to administer the assessment, and then collect that assessment. Now, it’s on computers so there’s a little bit different procedure to that. The last thing that countries have to do is they have to grade or score the open-ended questions. So, the fill in the blank questions, the multiple-choice questions, they’re automatically scored but the open-ended questions, it’s the country’s responsibility to score. All of that is checked and adjudicated and then you’ll see sometimes some countries they’ll find anomalies, they might not be included in the data set over open-ended test scoring.

Will Brehm  6:01
And is the analysis then done at the country level? Or does all the data go back to PISA who then does the labor of analysis?

David Rutkowski 6:09
Yeah, that’s a really good question. Two things, well, the initial analysis where the IRT scoring and everything like that, that’s done by a contractor of the OECD. That’s another interesting thing about PISA is most of the work is not done by the OECD, rather than by contractors. Scoring, for example, now that’s being done at ETS in the United States before it was done at ACER, Australia. That’s how that works.

Will Brehm  6:36
Okay, so the contractors of PISA, the ETS is a big one. Are there other ones that we know about?

David Rutkowski 6:44
Yeah, there’s a whole bunch of contractors. It’s been written quite a bit. Pearson has done some parts of the contract, although their role is less and less. And now, there’s going to be a new, every cycle or every few cycles, OECD puts out a new tender, and so different contractors will come in and have different parts of the assessment. As I said, historically, it was ACER in Australia who did the main part of the contract.

Will Brehm  7:13
And at the country level, so, you know, outside of the work that OECD has to contract, what about some of this work of, you know, coming up with a representative sample and doing the translation and then even giving the assessment and scoring the open-ended questions, do countries or do some countries, are there examples of countries that outsource that sort of work, that hire a private contractor to do it, or is some of this work being done by government civil servants?

David Rutkowski 7:45
Yeah, and that’s going to be really country dependent. For example, we could talk about how the samples done in the United States, and Laura’s going to talk about this more, I think in a little bit, but Westat does most of the sampling and that’s outside of our government, but they do most of the sampling and education for all of our government work. In Canada, most of the sampling is being done by StatsCan, and that’s a government institution that’s drawing their sample. So, it really differs between every country and the same would be who’s administering the assessment. When I lived in Norway, the University of Oslo was the one that was managing most of PISA. They were the ones going out and giving the assessment, for example, or overseeing that the assessment was given correctly.

Laura Engle  8:31
I do think it’s important to realize it is country specific. So, in a large country, like the US, as David mentioned, there are non-governmental actors which are contracted to do some of that work. David mentioned Westat, but then they might subcontract other entities. So, Pearson, for example, in some of the earlier cycles and others, but these organizations change, or the contractors change cycle to cycle and that’s based on an open tender process. But it is interesting because you see that there’s a whole range of different stakeholders, right? Governmental and non-governmental, involved in administering PISA and other assessments.

Will Brehm  9:08
And I guess it just occurs to me listening to how much work has to go into a single assessment and how many tenders have to be given that there must be so much money circulating sort of because of PISA? So, can you talk a little bit about the sort of costs to participate? Like, what how much does it actually cost a government to participate in PISA? And what are these different sort of line items?

Laura Engle  9:34
Yeah, it’s a great question. And, you know, we don’t talk much about costs of participation and I think that’s really worth underscoring. Prior to this article, I think costs were very much in a kind of a black box. You have statements about lavish budgets or millions and millions of dollars spent on PISA, but there wasn’t really any specific consideration of costs. And so, it’s really interesting because as you follow the money, so to speak, you follow the kind of architecture that’s built in and around PISA and other assessments. And so, another thing that’s pretty interesting when you start to think about the financial side of international, large scale assessments is that for most systems, we still don’t know how much they cost, nor are we likely to know, this information is really difficult to get. And I think that’s what’s so significant about this article. We talk about in the paper that the data related to costs. So, your question, you know, as to how much does PISA cost? We say that the data are publicly available, and they are, but I think what’s interesting about saying something is publicly available, that it really overshadows a long journey that it takes to get that publicly available data. So, I just want to mention that even though it was publicly available -and is publicly available- it doesn’t mean it’s easily accessible. And that actually surprised me a great deal because I initially entered into this study thinking this was a side question. I knew how much US states were paying to participate as adjudicated entities -all 50 states participate, of course, in national level PISA- but then some states were motivated to participate and get results specific for their state level. And we knew how much that costs. And so, I thought, well, how much does it cost for the US to participate? And thought it was going to be a very straightforward, simple answer. And what I found was that as I initially started interviewing different stakeholders is what got discussed was the overhead, meaning what a country, what the United States, pays the OECD to annually participate. But costs as we’re talking about in terms of these various contractors, and the tender process, you know, you can see that the cost of PISA go well beyond overhead and include national implementation of the assessment. And that’s where obtaining cost information was really remarkably challenging to find. Officials would indicate that the information about costs was public, but they didn’t know necessarily what it was, they couldn’t find the information, didn’t know necessarily even where to look. And so, after months of digging, we found that budget information as costs related to international, large-scale assessments only for the United States. As I said, we don’t know about other systems, those costs are submitted in documentation to the US Office of Management and Budget for public comment and approval. And so long-winded way of answering when you ask, what does it cost: For overhead, countries who participate pay an annual fee, and they pay that annual fee to the OECD, it’s determined by the size of the economy. US being the world’s largest economy pays the most about a million dollars a year. So that’s an annual assessment. And then countries are responsible for paying the cost of implementation. In the US, for previous cycles that was about 6.7 million. And then if a state wants to join separately, they would pay about $600,000-$630,000 to participate.

Will Brehm  13:05
Okay, so the $6.7 million for implementation, that’s being paid at the federal level by the Department of Education.

Laura Engle 13:11
Yes.

Will Brehm  13:13
And also, the annual fee to the OECD for overhead of a million dollars that’s being paid by the federal government in the US.

David Rutkowski 13:23
Yeah, and I think, you know, one thing that Laura mentioned there that’s important to remember is that OECD countries pay determined to the size of the economy. Non-OECD member states have a flat fee that they pay. So, the cost of overhead for non-OECD country, we just looked it up on the website was about it’s a $205,000 Euro over four years. So about 50,000 Euros a year.

Will Brehm  13:53
And then there is a state level cost for those states that want to obtain additional data from the test of about $600,000.

Laura Engle 14:04
That’s correct. And not many states have done that. So, in PISA 2012, the first states to do that were Florida, Connecticut, and Massachusetts. In 2015, Massachusetts, North Carolina. And to my knowledge, there are no states participating in 2018. Many states you can find online, many states will talk about the desire to participate in PISA, but they may not have the funds or there may be other reasons why they did not opt to participate.

Will Brehm  14:33
So, given these numbers, is PISA an expensive test, in your opinion?

Laura Engle 14:38
It’s a great question and it’s tough to answer when we don’t know what you know, compared to what? You know, are we comparing it to other international, large scale assessments? Are we comparing it, since we’re talking about the US to NAPE? Then we have to take into account relative uses, and you’re really getting into I think, you know, the real need, as I’ve argued, for a cost-benefit analysis. It’s really tough for me to say whether it’s simply a drop in the bucket or a very expensive endeavor. I’ve been unable to answer that without any kind of serious cost-benefit analysis.

Will Brehm  15:09
I also want to ask Laura, you were saying about this sort of “black box” and how difficult it was to even get the data. I mean, that’s pretty amazing in my mind, given that this is taxpayer money, and that it should be publicly available information, but you had trouble. So, how did you end up uncovering these numbers?

Laura Engle 15:31
Yeah, I mean, you know, they were always publicly available. But as they say, I mean, even talking to people who are doing the work of governing these assessments would say, well, yes, this is public information, taxpayer, you know, funding, but I’m not sure where it is. And so, you know, you really got into the process of how you administer this assessment from the very beginning, what kinds of information are needed for, you know, the National Center and Education Statistic to put together a package for the Office of Management and Budget to consider participation in each assessment. And so that is how, following that trail, and then the ultimate reporting of this documentation in a public site on the Office of Management and Budget, which is there for public comment is how I found these information. They’re all there, it just was a challenge to get pointed in the direction. Many, many searches online and stakeholder interviews, you know, didn’t necessarily point me there right away.

David Rutkowski 16:34
And the budgets are complex. And that’s one of the things I think that Laura found in that what’s interesting in the paper is that the money isn’t just going to one entity, there’s money going to many different entities. And as we talk about in the paper, for example, I mean, money goes to schools. And so, the budgets divided up a little bit on how much money is actually going to pay schools, to pay teachers, and oddly, in the United States, students get paid to take PISA.

Will Brehm  17:03
Wait a second, students get paid to take the test?! I missed out on that.

Laura Engle 17:07
They do, you know, they don’t get paid huge sums of money, but they do get paid as an incentive and it’s actually interesting to look at TIMSS versus PISA here. Schools get paid. They get paid, I think the initial amount offered is $200 and I find this really interesting that many schools in the US turn down PISA. This is different than other countries that don’t have that choice. And in the US schools are invited to participate and can say no, and so the documentation suggested that recruitment is really difficult. Schools do say no, and they tried to up the financial incentive, they increased it from $200 to $800 but even there, few of the schools that had initially turned them down agreed to participate based on the additional funds. But yes, students are offered a small fund of money or a small pot of money, and that is slightly higher if they take the assessment in non-school hours. And I don’t know how many would do that.

Will Brehm  18:03
That information isn’t available?

Laura Engle 18:05
So, not to my knowledge, because I think in this case, we would need to have a list of schools that were participating and to better understand the administration of the assessment in those individual schools. David, what do you think?

David Rutkowski 18:18
No. I mean, I don’t think we can get that information because we would have to have that list of schools and you would have to do it.

Will Brehm  18:24
And you don’t have the list of schools? Like, the list of schools who participate in PISA is not publicly available?

Laura Engle 18:29
No.

David Rutkowski 18:29
For reasons of confidentiality, you don’t want to know what schools, because this is a low-stakes test. But going back to students being paid, that was one of the things that really surprised me when we were working on the paper and Laura did most of the legwork -or all of the legwork- to really figure this out. But when it comes to students being paid, if they’re paid in the United States, and not paid in other countries, it really brings up a validity issue that I think it’s very important for a lot of people to understand when they’re comparing scores across countries, that some students are given a financial incentive to take the test, and other students aren’t.

Will Brehm  19:08
So, to stick with these costs, in the US how much of this money that we spend, maybe aside from that annual fee to the OECD, but just the amount of money that it takes to actually implement this test. How much of that money is going towards for-profit subcontractors, rather than say to, you know, public officials to do the administration of it, or to student incentives or school incentives?

Laura Engle 19:36
Well, that’s a great question and I think might take some further analysis to see if you can get a budget that’s further broken down. And what that means is, I think you need to get that largely from Westat because they were -and I’m thinking here of PISA 2012- where Westat is the contractor and we don’t have the information of how much of the funds for example, are given to Pearson in that cycle. Now, you could probably calculate, if students are paid $25, you could, you know, multiply the sample size and get some figures for that. Teachers are offered $20, school coordinators $200, so you can probably calculate some of that but off-hand, it’s hard to answer the proportion with any sort of definitive information.

Will Brehm  20:31
And we might know how much money goes to what’s it called West app?

David Rutkowski 20:35
Westat.

Laura Engle 20:37
Westat.

Will Brehm  20:37
Yeah, so Westat, if they get a big chunk of money, but then they do a subcontracting out to say, Pearson, that money we might not know about because Pearson doesn’t have to make that data publicly available. Is that correct? Is that how this sort of works?

Laura Engle 20:52
That’s my understanding, yes. That it wasn’t possible to know how much, let’s say of the $6.7 million was subcontracted to Pearson or some of the other entities that were named in other cycles? It would require further research.

David Rutkowski 21:10
And I mean, I don’t believe Westat has to give us that information.

Laura Engle 21:14
No, I do not think so. I mean, my understanding is that was proprietary. So, yeah, so it wasn’t, you know, there wasn’t a real concerted effort to get that information as part of this paper.

Will Brehm  21:26
Given the way these costs work and what you’ve uncovered in your sort of search in this black box, is it fair to say that PISA -in America at least- sort of contributes to the privatization of education because of all of these subcontracting out to for-profit companies?

David Rutkowski 21:46
I wouldn’t go as far as that. That’s tough to say. Because, again, we don’t really know all of the for-profit companies. Westat is not for-profit, for example. And in the United States, most of our testing is -just like the OECD- doesn’t actually do any of the testing. There’s not that many people that work on PISA in the OECD, a handful of people, all of that work is being given to outside organizations, non-profit and for-profit. And that’s the way we do things in the United States, as I said, before, other countries have different models. So, I don’t know if PISA is promoting that in any way in the sense that countries do what countries do and operate in that sense. The OECD doesn’t tell you, you have to use a private contractor.

Laura Engle 22:35
I mean, the interesting word, Will that you use was “contributing”. So, I’m pausing on this word of are they contributing. And I agree with David here, that these are really complex relationships and as we point out in the paper, the PISA architecture that’s built within the US is an architecture that, you know, it’s not built by the OECD, but rather by this sort of social network of new and really varying policy actors that, you know, cut across different, I guess we could say, different scales. And, you know, in doing some, some work, and in, you know, in other research comparing the US to other federal systems -take for example, Germany, Australia, Canada- what you see is the US education policy space is pretty unique in that it includes a wide variety of non-governmental actors, some public, but also some private entities, and the range of those organizations are really shaping the educational landscape. And so, it’s, I think it’s worth underscoring that this architecture and this sort of policy landscape is not necessarily created by the OECD. And so, what that means is when we attempt to understand the impacts or the uses of PISA within and across the US and other systems, I think we want to be really careful about suggesting any causation between the OECD and privatization. So, it’s not kind of this top-down, the OECD creates the conditions for privatization to take hold in a particular system, but I think it’s more complex than that because what we know about global educational governance -and I’m most familiar with federal systems- is, there’s this complex vertical and horizontal set of policy flows. And so, when we talk about contributing to privatization, that’s interesting. And so, I would be very interested in, you know, additional questions like, which Edu-businesses in the US have been championing PISA? Which have been promoting, say, PISA-for-schools or state level participation in it? Because that might actually help us answer that really interesting question about whether it’s “contributing”. I wouldn’t say causing but you know, contributing is a different question.

David Rutkowski 24:49
Yeah, PISA-for-schools is really another beast onto itself. But there, you could see a lot more, a closer relationship with the private sector in the Edu-business of trying to sell PISA-for-schools and coup some money. So, I think there’s, in that case, it really is more evident than in the larger PISA case.

Will Brehm  25:12
So, let’s now turn a little bit to the OECD itself. Because one of the arguments that you make in your paper is that the very way in which governments fund the OECD basically ensures PISA will continue into the future. Can you explain how you came to that conclusion?

Laura Engle  25:32
Yeah, um, you know, that’s part of uncovering or, you know, finding out a little bit more about the annual assessment. So, countries, in that they’re investing in PISA on an annual basis, it means that that annual assessments -it’s not per cycle- it’s an annual investment and so they’re supporting the past cycles and further analysis and reporting, they’re supporting the current cycle of course, and future cycles. So, what we’re arguing here is that in and of itself is ensuring PISA’s continuation by providing the OECD with what is a stable and invested participation base. At no time for the US -and something I said earlier- but at no time for the US is there a question of whether or not to participate. You know, participation is ongoing, and it’s paid for on an annual basis. So, the OECD here can advance in study designs, it can invest in those future cycles. It’s again, that stable and invested, I think, base.

Will Brehm  26:28
So, given this analysis. Given this, look at least the information that you were able to find, and perhaps thinking about, okay, in the future, we need, you know, other information, and we need to keep digging and do some more analysis if we can get the numbers. But so far with what you have learned, do you think that the benefits of participating in PISA outweigh these costs?

Laura Engle 26:51
Well, as I said, I think that’s a hugely important question. And without a cost-benefit analysis I think it’s very tough to say. When we had these data, we talked a lot about these questions and how to frame this paper. When you start talking about costs, much of your audience or, you know, people you’re speaking to want you to talk about whether it’s worth it and it’s really tough to say or answer definitively, is PISA merely a drop in a bucket given the size of, for example, the US federal budget for education? Is it a good deal? Is it a cost-effective assessment? And David could perhaps talk about this in other countries. You know, is it a pretty cheap alternative? Is it a good deal, given the robust cross-national information that can be gleaned that other national assessments simply can’t do, they’re not designed to do? But there’s other questions, do the risks, especially risks related to misuses of PISA outweigh the benefits? If we know these considerable costs, shouldn’t we be using results more deeply? So, there’s all kinds of questions that for me, are generated out of this inquiry into costs. And the question that inherently follows, which is, do we think benefits outweigh costs. So, I think, you know, part of the work of this article was to raise these critical questions to fill gaps in what we know, but also to argue for the need for a cost-benefit analysis, a serious deep dive into the benefits. And again, some of the costs.

David Rutkowski 28:20
I think, one of the few of the problems that you’re going to have. Will, you’ve had a few people come on your show and talk about PISA and some of the benefits and some of the downfalls of PISA but in the United States, to be honest, we largely ignore PISA and most international assessments. And that’s for, you know, valid reasons, a lot of times where country of over 300 million people, we have a really, really great national assessment, we have a lot of things to compare. So, $6.7 million out of, as Laura said, out of our federal budget, it’s really not a ton of money for the United States. Do they gain a lot from PISA? I think that’s the hard thing, to understand what the benefit in such a big country to participate with PISA is, and what it actually tells us. We don’t even have a federal system of education. Our states are very, very independent. So, it’s hard to know what the benefits of participating in PISA are, for the United States, besides some headlines, some extra analysis. Now, let’s move to a country, the Scandinavian countries, which I know a little bit about, or you’re in Japan. Norway, for example, they gain a lot from PISA, it helps them compare with countries that are similar to them, so they can compare themselves to Sweden, to Finland, to Denmark. So, there’s a lot of benefits for Norway in that sense, where the United States it doesn’t really benefit from comparing themselves to Norway, very much. You know, 5 million people, a system that’s pretty controlled by the federal government. It’s a hugely different, it’s makes it much more difficult to compare. So, I think the cost-benefit analysis really has to be done at each nations level. And also, to understand the benefits is really, really difficult. And then you go into what Laura was saying, of some of the downfalls of participating in PISA which there’s some negative points as people on your show have talked about before with participating in PISA, and then you would have to take that into consideration too.

Will Brehm  30:34
Yeah, I mean, it seems like there would be, you know, educational benefits and also economic benefits but obviously, I would imagine some sort of political benefit as well of participating in these international organization.

David Rutkowski 30:47
Well, you know, if you want to become part of the OECD, you have to participate in PISA, right? That’s to get into the OECD. So, there’s a huge benefit for countries that want to get into the OECD. They’re going to participate in PISA, just like you said, that’s clear economic benefit.

Will Brehm  31:03
Well, I kind of see a future paper for the two of you, something about the benefit of PISA in the USA. Laura Engle and David Rutkowski, thank you so much for joining FreshEd, it really was a pleasure of talking today.

Laura Engle 31:15
Thank you.

David Rutkowski 31:15
Thank you.

Want to help translate this show? Please contact info@freshedpodcast.com

Have any useful resources related to this show? Please send them to info@freshedpodcast.com