Will Brehm  0:15
Patricia Bromley, welcome to FreshEd.

Patricia Bromley  1:24
Hi, Will, thanks so much for having me on today.

Will Brehm  1:37
So, you are responsible -with a team I would imagine- for creating something called the World Education Reform Database. And I went on to the website, and I was blown away that it has over 10,000 policy changes in 189 countries since 1970. I mean, it’s pretty unbelievable. I just want to know sort of why on earth, you decided that this was even something that was needed in our field.

Patricia Bromley  3:15
Thank you so much for the kind words. It’s been a real journey creating this database. So, the idea for it came out of discussions with my co-principal investigator on the project, that’s Rie Kijima at the University of Toronto. And about five years ago, Rie and I were talking about her amazing work on the effects of international assessments on education systems. And she was trying to figure out if these assessments are shaping national education policies in systematic ways. So, we were looking around for data to help her research this, and we realized there were some targeted sort of datasets of cross-national reforms. So, that would be something like the year a country established compulsory schooling, but there was nothing comprehensive of the sort of range of effects that assessments might be having. When we were looking at what was out there, we noticed several studies that drew their information about targeted policies from international reports -so, reports to the World Bank, OECD, and so on. And when we were looking at the reports, we thought we could actually pull out a larger range of reforms, partly drawing on a method I had used in a previous body of research where I extracted information from textbooks. So, that was sort of what led to its emergence. And it grew and grew and grew from there over the past five years.

Will Brehm  4:40
So, It’s been five years that you’ve been putting this together.

Patricia Bromley  4:43
Yeah, that’s right. We released it publicly last year.

Will Brehm  4:46
Yeah, that’s right. I downloaded it in preparation for this interview and sort scrolled through these policy reforms and it was quite overwhelming, and I kept thinking it must have been quite difficult to compile this list. So, you know, what were some of the big challenges that you and your team faced when constructing a novel dataset to this magnitude?

Patricia Bromley  5:08
Yeah, it was challenging, also very fun. We have been working with a large team on it. So, it took us about a year -an academic year- to even figure out if it would be doable. So, we spent about a year piloting different kinds of like coding approaches. So, this is sort of ways of pulling information from the reports. And some of the challenges we faced were sort of coming up with a definition of reform was one of them or policy change, and coming up with a definition that across lots of different research assistants and different report types, people would pull more or less the same information. There’s always some difference but we try to aim for sort of a .8 on our interrater reliability tests, like standard social science, level of reliability. So, it took us months and months of testing out definitions and coding instructions to get to that level, and then deciding what reports to collect and gathering them.

Will Brehm  6:10
And it sounds like you worked with a lot of students, and students are transient. So, you’re probably dealing with this constantly changing team. I would just imagine it was rather a complex endeavor. I want to dig into this issue that you sort of just raised in this first year, as you were sort of conceptualizing, piloting and trying to figure out how to do this. And that’s over this notion of what do we even mean by education reform? So, you know, where did you land with your team? How do you in a sense conceptualize this term that on the one hand, seems so commonplace, but once you start unpicking it, it probably is hard to come up with a definition, as you said.

Patricia Bromley  6:47
Yeah, absolutely. It is actually a really fascinating conceptual issue, as well as a kind of technical problem we had to solve. So, we ended up landing on a definition of reform that has three parts. So, the three parts are that it’s systemic, non-routine, and purposeful or planned. We decided reform is something -so the systemic part- that suggests a change that’s linked to some kind of broader vision of education as a system, not an individual school, say operating on its own disconnected from the environment it’s linked to. So, systemic was part of it. Non-routine is some kind of change. There are lots of kinds of changes that are quite routine that even show up in reports, for example, as sort of annual budget reauthorizations. And this might show up as a legislative change every year. So, we didn’t count that as really a change, that’s a routine activity -personnel reauthorizations are another one.

Will Brehm  7:49
What would be a non-routine change, then?

Patricia Bromley  7:51
So, there would be anything that is different in the system. So, change maybe is a better way to think of it. So, it has to mark a change from what was coming before. Something we decided on the change front, is you can think of change as occurring at different levels. There can be sort of incremental change, or sort of transformational, paradigm shifting change, we opted to lean on the incremental side of this, because we were aiming to put together a pretty comprehensive list of possibilities for people. So, the idea was that if we include from the incremental through the transformational, then others can take the list and sort of if they only want to look at transformational paradigm shifting laws or something they can look through what we report and exclude the ones that don’t count. So, we went quite broad, I think so for some people, that wouldn’t count as reform, for example, they only think of reform as shifting a paradigm and maybe a policy is a small change.

Will Brehm  8:55
Right. So, can you give an example of that, like the difference between what might be considered incremental change, non-routine change in a system versus transformational change, non-routine change in a system? Sure.

Patricia Bromley  9:10
Sure. So, maybe something like, let’s say, learning differences that kids in elementary school are already mainstreamed. And then there’s a reform or policy change that’s passed to say, actually now kids in high school will also be mainstream. Kids with learning differences who might have been in a different system before that could be incremental, sort of one policy change that builds on some prior policy change. We would capture each of those as individual reforms or policy changes in our database. And then something like maybe the wave of neoliberal changes that went on in the world or new public management could use that label too. For some, that counts as a reform. So, a big reform is neoliberal changes to education. And there are dozens and dozens and dozens of policy changes linked to that whole shift in thinking of education systems as managed by market principles of something.

Right. So, like performance pay for teachers or bonuses, or something like that, which never happened before.

Yeah, by some definitions, the individual policy is not itself like a giant reform, the bigger picture is. So, for us, we just count them all as single policy changes, and don’t differentiate.

Will Brehm  10:32
And then the last one that you talk about for education reform is this notion of purpose.

Patricia Bromley  10:37
So, we think of reform or policy change as a planned administrative action. There’s lots and lots of change that happens in the world and to education systems and in schools that’s just change; just regular change. It’s not a sort of intentional administrative act. So, we wanted to distinguish sort of other kinds of change from the intentional administrative change goals of reform.

Will Brehm  11:04
Do these education reforms, then -like, what’s the relationship to policy here? In a sense, what were you looking at? Were you looking at specific documents, the written word, were you looking at websites? How do you tangibly hold an education reform that you could then analyze.

Patricia Bromley  11:22
So, I think you could use the word policy as interchangeable in my uses of the word, not by all definitions, but it would be fine to call it a database of policy changes. That said, I think depending on how you define policy, we probably have some things in there that might not fit. So, if a policy is like a legislative act, say, then we have some things that are frameworks of action, or vision plans, that we include that might not fit some definitions of policy?

Will Brehm  11:59
And what is the relationship between sort of education reform policies, and action, and practice? Because obviously, there’s usually a big divide between what say, a government might adopt and what ultimately goes on in classrooms. So, in other words, why is it important to look at the text of these policy documents over time?

Patricia Bromley  12:21
Yeah, it’s a big question. And I think about it in a couple of ways. One is that, like, the text of the policies, or discourse, is important in its own right because it tells us what sort of legitimate goals are. These are public facing, and especially internationally public facing reports of what countries intend to do. So, that’s giving us insight into what are countries saying is the right thing to do in education, regardless of whether they’re doing it or not? This second part, I think, is that it’s sort of the best we can do. We may not know for sure if reforms are implemented or not. But we still can look at, for example, if reforms that countries talk about expanding access, do in fact lead to expanded enrollment rates. It’s also very plausible if we think about sort of policy process that policy adoption, implementation and effects sort of have complicated relationships. So, you can imagine scenarios where the discourse actually does help shape the outcomes, even if implementation is weak, because it draws attention to a problem or sort of facilitates other actors to get involved.

Will Brehm  13:36
You can sort of understand how a discourse can almost frame the way we should see the world and understand practice. And even if the practice ends up looking slightly different, the discourse and the ideas that are circulating in that moment are hugely powerful. So, given those sorts of caveats of the concept of educational reform, and why looking at these texts is so important, what did you find? What are some of the big trends, when you look from the 1970s up till recent times, what do we see? What do you find when you look across these policies across the world?

Patricia Bromley  14:10
The big, most dramatic finding to us was a decline in levels of reported reform activity in recent decades, which in retrospect, I think, of course, there’s a decline. We sort of know that the neoliberal era, say like starting late 80s, through the 90s, and sort of through the 2000s was a period of massive global change. And we know that sort of set of policy prescriptions has come under a lot of attack, recently -a lot of pushbacks. And there isn’t really a new consolidated set of things that’s arisen. So, it makes sense that there is this decline, but somehow when we first plotted the trends, we really weren’t expecting to see this kind of mountain shaped curve with a big drop off in reported performs recently. And we think it does sort of reflect the drop off of global attention to a big sort of paradigm shift in education.

Will Brehm  15:11
Let’s unpack that a little bit. So, what you’re sort of saying this mountain-like shape is from the beginning of your data set, you sort of see this intensification of the sheer quantity of reforms that are happening in individual countries. And that sort of continues to rise up until about the 2000s. And then it sort of starts decreasing until today. That’s the idea?

Patricia Bromley  15:32
Yeah, exactly. And especially starting in sort of the late 2000s. So, 2008, 2009, 2010.

Will Brehm  15:39
Okay. So, after the global financial crisis, you start seeing this big shift. And so it sounds like one of the ways you’re reading this is that there was a paradigm shift from the neoliberal era that you were saying, which produced all sorts of reform ideas and efforts, which then of course, countries were quickly adopting and changing and putting into place, and now there’s sort of this move away from that neoliberal era, or at least -I guess, how do you read that decline a little bit more? That becomes a question, doesn’t it?

Patricia Bromley  16:08
Yeah, it does. So, I think that the sort of most direct reaction is exactly what I had said, and you were echoing there of it’s the kind of push back against some of the policies of neoliberalism that came under criticism. But I think an additional or deeper layer may be of what could be going on is sort of linked to what we can think of as the entire liberal order. So, this is the kind of international foundation of democracy and capitalism and human rights and rule of law. These principles as being the foundation of the global liberal order, especially since World War Two. And on many fronts, this whole order is under attack, or eroding in some way. So, there are lots of broad trends like declining levels of democracy worldwide, growing restrictions on civil society worldwide, increasing sort of rollbacks of rights for women and other various groups, rising populism. All these kinds of trends we hear about in the media, we can think of as attacks on the liberal order. And part of what a liberal order does is sort of believe in reform, like its rule of law as the way of achieving progress. So, if we undercut this whole world order, we would sort of expect less reform in general. Beyond education, it would be interesting to compare other sectors.

Will Brehm  17:38
Yeah, right. So, you’re sort of saying that this decline in the liberal order along these different lines that you’re articulating would then translate into policymakers producing less policy  because they have sort of less faith in what policy can do in the rule of law that might govern different countries?

Patricia Bromley  17:57
Yeah. I think that’s right. And the way we have sort of come to understand it as looking -this isn’t to say change wouldn’t happen. It just wouldn’t look like this necessarily, like formally codified sort of rule of law type practices.

Will Brehm  18:12
And the policies that are being created more recently, even if there’s fewer of them across time; Do we know if the content is different? Are the ideas that are sort of coming out different to really show the beginnings of what a paradigm shift might be? Or maybe it’s like this multipolar world and there’s all different ideas that are coming out and you wouldn’t be able to group them so easily under one paradigm yet.

Patricia Bromley  18:37
Yeah. I love where you’re going with this. So, a few things come to mind. I mean, one is, it’s really important to keep in mind that our source of reforms reports to international organizations. So, what we are sort of reporting here is the international discourse or discourse intended for an international audience, which is part of this whole, like liberal internationalism. So, it very well could be that that’s part of what’s eroding and maybe countries or regions have a lot going on, but they’re not sort of reporting as much or participating in a consolidated way at the international, global level. So, our data, when we look at the trends we see in terms of content, we don’t show very dramatic differences by region, for example, but you might find that if you looked within countries. Not in terms of what they’re reporting to the global community, just internally or in regional associations. So, I think what you’re saying could be happening -a kind of regionalism and undercutting of the international side of things. I will say one sort of striking thing we have seen; we have a new paper coming out that does topic modeling of the content -so, inductively using computational methods, looking at the trends in the reforms, and we find several themes that are various kinds of emphases on rights. So, right to education, anti-discrimination kinds of reforms. And there are three different sort of types of topics, I guess, reform themes around there. Those do decline in recent era, which would speak to that. Although -I don’t want to overstate things- there are many kinds of reforms about inclusion and access that continue to increase dramatically, sort of in line with the Education for All movement. So, those continue to grow. It’s just this rights language specifically, that seems to be sort of dampening.

Will Brehm  20:34
That’s interesting. Is that the part that’s most closely connected to this liberal world order, do you think? Like, in terms of reform policy, ideas and discourses; is that the main sort of proxy in a sense for the liberal world order? And access and quality, is it somehow connected, but sort of beyond the liberal world order as well?

Patricia Bromley  20:53
It could be. I mean, you can certainly think of access and inclusion as having lots of grounds other than rights, or individual empowerment. I think there’s some work talking about the expansion of mass schooling  -this is in political science- as partly driven by conflict within countries to help elites maintain power. So, it wasn’t a democratic reason for expanding education, it was to maintain control. We could think of lots of possibilities for that, but I don’t have the answers to these big questions. I love thinking about them, though.

Will Brehm  21:29
Yeah, I mean, you can just see where all of this empirical evidence still needs a lot of theory to begin to sort of make sense of what’s actually happening.

Patricia Bromley  21:37
Yes, I really hope people will make good use of it.

Will Brehm  21:40
I hope so too. I mean, I only looked at it for like 15 minutes, I was blown away, and just saying, Oh, my gosh, I sent it right to a PhD student, actually, and said, Have a look at this. You know, I guess like you said, you’re looking at the reports going to international organizations and trying to think about this international discourse that exists in education reform. Who are the actors here that you are looking at when it comes to the international organization? Or how do we make up the different actors in that space?

Patricia Bromley  21:40
We turn to ones that are commonly recognized as some of the main ones. So, this would be the World Bank, OECD, UNESCO for the different kinds of education reports they produce. And in the existing comparative education literature, they’re sort of identified as the key international organizations. I think, though, at the backdrop of this, we can think of academics and sort of universities or the expansion of science as being really central and cross cutting because the knowledge that academia sort of is producing often becomes the organizing principles around some reform efforts. So, things that come to mind right now are, for example, early childhood education, and that expansion or social and emotional wellbeing and in both of those, educational psychology has been very central in creating a kind of movement and evidence base that leads us to this road of reforms in these areas. So, we also included some encyclopedias produced by sort of experts of country reports. So, not just the international organizations but also education experts in general.

Will Brehm  23:24
So, you have sort of the World Bank, the OECD, that sort of group that has come to power and there’s been some really good work historically about how they’ve come to power over in that sort of post-World War Two period. You’ve also included these sorts of academic experts, and how the ideas that get pushed by academics, they have power within this international space as well. But you also include international nongovernmental organizations, NGOs, how do they fit into this picture as well?

Patricia Bromley  23:50
So, we’ve used them a little differently in our research, as in we haven’t used sort of NGO reports or documents as part of where we’re collecting the data from, but we do find that they are central actors in spreading reforms. So, in some of the analysis we’ve done, we show that memberships in international nongovernmental organizations sort of increase the numbers of reforms that countries do. We think of them as sort of carrying these global education principles to countries. They are carriers and enactors and facilitators of the global discourse. And we find a similar effect for World Bank education loans. So, at least for several decades, education loans from the World Bank were linked to increased levels of education reform in a country.

Will Brehm  24:41
Okay, right. And if you look at this over time, and we know that there’s a decline that’s been happening since the 2000s, in the number of education reforms; what can you say about say the World Bank and nongovernmental organizations, particularly international ones? Has their relationship to education reform changed over time?

Patricia Bromley  24:59
We did find an interesting shift in the sort of influence of these two actors over time, maybe related to our earlier discussion about neoliberalism, but we find that the influence of the World Bank was quite strong early on, say, in the 1970s, 1980s. World Bank lending was linked to increased numbers of reforms in a country. But in recent decades, there is not as strong an association. So, it declines, not necessarily the influence of the World Bank, but their direct effect in terms of triggering reforms. And it’s the opposite for NGOs. It’s possible these trends are even related that as the Bank came under some criticism and starts to use community organizations and NGOs, for example, more as the enactors of reform, so we see this sort of opposite trend of the two of them.

Will Brehm  25:53
Interesting. It’s quite a fascinating sort of insight. But what you’re not saying is that the amount of loans that the World Bank is providing that hasn’t decreased, right? They’re still lending money to education reform or education sectors, but it’s just the amount of education reform that comes out of these loans is decreasing?

Patricia Bromley  26:11
Yes, absolutely. It also could be the case that the Bank is sort of less trying to create national education reforms in countries. They position themselves more now as a knowledge bank. And maybe they’re trying to create ideas out there in the world and research and study things that are on the ground and less have this direct effect. They’re still doing education lending.

Will Brehm  26:35
And the NGOs are now sort of spreading reform and as they’ve grown in influence, in a way they are spreading reform, and maybe this goes into future research; do we know anything about the differences in content of reform that might be being promoted between say, the World Bank and some of these international nongovernmental organizations? Like, could we classify them as being different?

Patricia Bromley  27:01
Oh, that’s very interesting. I can’t speak to that from what we have done. But it would be interesting to look into to see if there’s a link there.

Will Brehm  27:10
I mean, I guess, what you’re saying is we’re sort of going through this moment of change. We don’t really know where we’re going or what’s going to happen. And we keep using this word neoliberalism, which often is sort of used as a negative term, right? It’s these bad ideas, these bad policies and if you’re sort of saying we’re moving beyond the paradigm of neoliberalism, we don’t know what’s coming next, I guess you know, this decline that you’re reporting, is this something that we should be worried about? Or is it something that we should rather be excited about because we’re sort of moving into potentially non neoliberal spaces of education?

Patricia Bromley  27:45
I think it really depends on where you stand. I do think you can read it both ways, though. I think for those who felt neoliberal reforms were harmful, then less reform is probably a good thing. And I think kind of anti-government stances, less reform is also a good thing. I think if you think that education reforms, sort of, in broad scale, have accomplished a lot of good things, greater numbers of kids in school, for example, then you might be more concerned.

Will Brehm  28:20
Exactly. And I think this is where the political debate will really come in. I think it’s not just neoliberalism, as you were saying. It’s also this larger notion of the liberal world order, which encompasses many other sorts of political ideas beyond seeing the world through sort of economic efficiency. And I think that’s going to be the big point of contention as to forcing people to really think, what is the value of the liberal world order? And do we want it to really decline? And I guess the final question that I would have is, you know, going back to some of this methodology that we started with. I guess I keep thinking about the causal arrow, like, how do we know that the decline of say democracy is sort of pointing and causing less education reform and not the reverse? Could it be that less education reform is pushing these anti liberal ideas?

Patricia Bromley  29:11
I think like any sort of study of schooling, I would never want to position school as only living on one side of that arrow. I think part of what’s so powerful about education as an institution is it does both. It lives on both directions. So, you can do things in your research designs, I think, to focus on one arrow or another, but both things would be happening. Schooling is important because it provides a window into broader social processes, and it reflects many social and political institutions, but it also helps create them down the road.

Will Brehm  29:53
Well, Patricia Bromley, thank you so much for joining FreshEd. It’s just such a fascinating study and I really look forward to reading you and your team’s many other publications that are likely to come out of this database.

Patricia Bromley  30:04
Thank you so much for having me on, Will.

Want to help translate this show? Please contact info@freshedpodcast.com

Related Guest Publications/Projects

Global determinants of education reform, 1960 to 2017

Global neoliberalism as a cultural order and its expansive educational effects

Education reform in the twenty-first century

Education in a postliberal world society

Education for global citizenship and sustainable development in social science textbooks

Recommended
World Education Reform Database

Global governance of education: …UNESCO, the OECD and the World Bank

Diffusing “destandardization” reforms across educational systems…: The case of World Bank

The World Bank’s saber: A critical analysis

Globalization, education and reforms

Have any useful resources related to this show? Please send them to info@freshedpodcast.com