The Globalization and Education Special Interest Group of the Comparative and International Education Society will be hosting a public webinar on December 12 entitled “Puncturing the Paradigm: Education Policy in a New Global Era.” The webinar will bring together the four co-editors of the newly published Handbook of Global Education Policy, Karen Mundy, Andy Green, Bob Lingard, and Toni Verger.

During the lead up to that event, FreshEd will interview the co-editors to set the stage for the webinar. Today I speak with Professor Andy Green about the global education policy of social cohesion.

Although we often think of education policy as primarily concerned with economic development, it also has been historically connected to the idea of creating a cohesive group of people who share certain norms and customs. Benedict Anderson called this “imagined communities.”

Andy Green has looked at the effect from education on social cohesion across the globe.

Andy Green is Professor of Comparative Social Science and Director of the Center on Learning and Life Chances at the Institute of Education, University College London.

Citation: Green, Andy, interview with Will Brehm, FreshEd, 52, podcast audio, November 21, 2016. https://freshedpodcast.com/andygreen/

Will Brehm 2:01
Andy Green, welcome to FreshEd.

Andy Green 2:04
Well, Will, it’s pleasure to be here to talk to you.

Will Brehm 2:07
What is social cohesion?

Andy Green 2:11
Well, social cohesion, fundamentally, is all those properties which bind societies together. So, it might well be common values, common lifestyles, common identities. It may simply be the rule of law. You can define it in quite a number of different ways, and I would say you need to define it quite broadly, though, as the set of attitudes, values, and behaviors that binds societies together. Bearing in mind that social cohesion actually works quite differently in different places. So, I wrote a book some years back called ‘Regimes of Social Cohesion’ with my co-author Jan Janmaat, and we were looking at countries across the developed world, basically, at what factors seem to be holding their societies together. What was the nature of social cohesion in those different societies? And it transpired there were groups of countries basically, in different regions which were quite distinctive. So, for instance, English-speaking countries -which aren’t a region, of course, but have a common cultural history- tended to have a core set of key values, not a strong emphasis on a broad set of common values, because they’re very diverse societies generally. And the key value tends to be about opportunity, and rewards for merit, if you like meritocracy, some people would say, which is a little bit different from what you’d find in countries with Republican systems like France, where there’s much less emphasis on social mobility, but more emphasis probably on equality of outcomes. And where a dominant lifestyle traditionally, with a broad set of values, has been rather more important. Although this is clearly under strain these days. And then again, you get Scandinavian countries where identity and social cohesion revolves very much around their particular forms of welfare state, and a very, very strong belief in those countries in equality of outcomes. And you can find different things again, if you go to East Asian countries, for instance, where you can provisionally sort of identify a form of social cohesion based very much on Confucian values. So, here it’s respect for elders, respect for the state, and so on, as well as a common sense of cultural identity are extremely important. So, it would seem that social cohesion does work somewhat differently in different countries. People do try to develop a single definition, which is okay, in my view if it’s a very broad definition. They also try to measure it with a single set of measures, which is rather difficult because it’s basically a cluster of different things. And the cluster of different properties may be different in different countries. And so, having a single scalar measurement is rather difficult. Very often, though, the key measure is taken to be levels of social trust. That’s to say how far we trust other people, and particularly how far we trust people we don’t know.

Will Brehm 5:43
What’s the connection between social cohesion and education?

Andy Green 5:48
Social cohesion has always been a primary aim along with economic development, and so on and so forth. It’s almost always been there somewhere in the big visions for education systems.

Will Brehm 5:59
So, governments, when they create policy, they would not only see their education system as producing future laborers in the national economy but also creating citizens and members of society that would come together in some form, broadly defined notion of social cohesion with these different values being emphasized in different countries.

Andy Green 6:25
They would. It’s been typical of all newly created public education systems from the early 19th century onwards, that forming of citizens was one of the main purposes of an education system as well as you know, developing skills and so on and knowledge for the economy. It tends to be particularly important in new states, which don’t have a common identity firmly established, or institutions firmly established. Young states always emphasize the forming of citizens is a primary role of education. In older democracies, it’s tended to take second seat to economic development. I suppose skills formation becomes more important than citizen formation, but it’s not disappeared, that’s the important thing. Almost every document that you’ll find coming out of national governments, and indeed, transnational organizations, will refer in some way to the importance of cultivating social cohesion and citizenship and so on through the education system.

Will Brehm 7:34
So, would you be able to say that social cohesion is, in a sense, a global policy of education?

Andy Green 7:43
Yes, I would, but it’s referred to in different ways, perhaps in different international organizations depending on their remit. So, the OECD, the European Commission will use the word social cohesion quite frequently. OECD likes the term social capital as well. If you look at the outputs of, say, the World Bank, or UNESCO, where they’re looking at a wider range of countries, including less developed countries, peace education might well be the main focus, or post-conflict education, education in conflict societies or post-conflict societies. They’re all talking, though, about different forms of social cohesion, really, with different emphasis.

Will Brehm 8:30
So, what sort of education initiatives exist for peace education, or citizenship education, or social capital? Like, concretely, what are these educational initiatives that are either being promoted at the global level or within the national level or perhaps both?

Andy Green 8:51
Well, at the national level, most countries have citizenship education as part of the curriculum. It may be cross-curricular, it may be a separate subject, but it’s there in almost all countries. There may be allied to that a number of procedural things around schools, say, which are designed to boost cooperation amongst children and common understandings and so and this may take the form of schools, councils or these kind of things -so structures will be part of it as well as the curriculum. Other initiatives have been tried in countries in the throes of conflict or in post-conflict societies. Common education across divides, say in Northern Ireland during the troubles, there was quite a strong initiative to build a set of schools which would cross the Catholic-Protestant divide and so on. This kind of thing has been tried in various different societies. Education about the values of peace and so on is something that’s often tried in many conflict-ridden and post-conflict societies. But you could add to that broader concerns about the social society, if you like, and how education could contribute and include environmental education. Notably, education on relationships, on birth control, and family planning, and so on in poorer countries, would be seen as part of education for social cohesion because of its proven very beneficial effects in raising the esteem of women and in reducing population growth, and so on, which it can have all sorts of, you know, social benefits. So, I would extend the range of social cohesion policies quite widely in that sense if you’re talking about developing countries.

Will Brehm 10:59
What has been the efficacy of some of these initiatives? Like, are they creating more social cohesion?

Andy Green 11:08
Well, this is where you have to start making this difficult distinction between what benefits individuals and benefits society as a whole. Politicians tend to start with the easy part, which is measuring the benefits to individuals, and certainly for developed countries -and it probably applies for developing countries as well, although we don’t always have such good data to show it- the general pattern is that more educated people tend to be more tolerant of other values and lifestyles, they tend to be more politically engaged, they tend to show better signs of health, and fewer signs of depression and negative aspects of health. They’re less likely to be overweight as children, and the list goes on and on and on. They tend to vote more in countries as well. So, at the individual level, higher levels of education, particularly up to degree levels, are certainly associated with social outcomes, which most people would consider good. But there are questions about that way of looking at it. There are two sorts of questions. One is ‘was it the education which actually caused them to have those social attributes, or did they have them already? ‘ Is it to do with who is selected what you call selection effects in statistical analysis, which is an important issue because it may be education is not adding that much, it’s the people themselves who go into higher education, for instance? The evidence, though, tends to suggest there is an effect from learning, as well as any effects from selection. So, at the individual level, yes. There are clear benefits to individuals from higher levels of education. They’re likely to be more engaged politically, they will trust more generally, they’re more likely to vote, and they’re likely to be healthier.

Will Brehm 13:21
So, how do we apply that from the individual to the social benefit? Can we aggregate? Can we just look at all of these individual benefits and say, we know if most people in society have these individual benefits, then we have some level of social cohesion? Is that possible? Is that how the policymakers do it?

Andy Green 13:43
That tends to be the start of the reasoning, but it rather quickly breaks down because many of these properties don’t aggregate for one reason or another. So, I mean, to take a quite commonly cited example: generally speaking, more educated people in most countries are more tolerant. They’re more tolerant of other sexualities, other lifestyles, other religions, and so on, so forth. That doesn’t apply in every country, but in most countries, there is a relationship there. However, better- educated countries are not necessarily more tolerant in the aggregate. And what we see in many countries now is increasingly educated societies judged on people’s qualification levels and declining levels of tolerance. So, something else is intervening…It’s not that education is not promoting tolerance; it probably is, but other things are working against it. So, you can have increasing levels of education at a societal level, but it’s not showing up in increasing levels of tolerance. So, you then have to start asking difficult questions about what else is involved in the context which may be working against it, and how far the individual’s tolerance anyway aggregates at the social level to more individuals being tolerantly to a more tolerant society, generally other things being equal. That’s where the difficult questions start to enter in.

Will Brehm 15:22
And so, what sort of theories have been put forward to explain social cohesion at the group level rather than at the individual level?

Andy Green 15:33
Well, you have to start talking about what social scientists call “mechanisms.” So that was the transmission mechanism for individual values, affecting societal values, and so on. And the research that’s done in this area more or less identifies four kinds of effects. The simplest will be what I’ve described already, for tolerance. It will be what we call absolute direct effects. So, a direct effect is where education through learning, or socialization, or whatever’s happening, is having a direct effect on somebody’s attitudes, which carries through into later life. That’s a direct effect. It would be an indirect effect if education was increasing people’s employability, got them better jobs in later life. And if it were the better job that were raising their levels of tolerance. So, an absolute direct effect is one that occurs without any mediation from anything else. And being absolute, it means that it should aggregate. So, most of the research on tolerance suggests that actually, in large part, it is an absolute direct effect. And the argument made by psychologist would be along the lines of higher education raises your cognitive abilities, which makes you better able to disentangle poor arguments, to see behind stereotypes, to challenge the logic of prejudicial kind of thinking. But the second part of it will be education simply, as they used to say, broadens your horizons, you get more experience of different types of people in different countries, different living situations, even if at a distance, you know. You have a wider experience, and that in itself is said to make you more sympathetic to difference and, therefore, you know, the other person. So, education there is making a direct contribution immediately. It may make further contributions later through helping people get jobs, which may make them even more tolerant. It may. But the complication there is that there are countervailing effects from other things. Which may mean that even though education is increasing amongst individuals, and at the societal level, generally, you don’t see rising levels of tolerance. But it’s still a direct effect. It doesn’t mean that education is not helping, it just means it’s not helping enough to counteract things which are having a negative effect. So, that would be the simplest kind of aggregation mechanism. So, you could say that education here is certainly contributing to an aspect of social cohesion. Now, of course, in some societies, such a high value isn’t placed on tolerance as in others, and social cohesion may not rest on it to such a degree. But in most Western societies, tolerance would be fairly key. Another situation would be where you have these direct effects, but in addition to that, you have other effects, which result further down the line from the impact of your education on employment. So, to take the tolerance example, again, you may have been socialized and learn towards to being and having more tolerant attitudes, your higher qualifications may also get you a higher-level job, which may in itself promote more tolerant attitudes through various psychological mechanisms, maybe you feel less threatened by others and so on and so forth. So, that would be a kind of a cumulative process. And then, rather different is what we call relative effects or positional effects. And this is where education is having an effect on something, but it is not the absolute level of education you achieve that has the effects. It’s the level of education of yourself in relation to others. So, it may be that some of the benefits of education result from the fact that you are better educated than others, you get a better job than others. And it’s that better job compared with others that has the social benefits for you. And in that kind of situation, it can be a zero-sum game. You can have more and more people being educated, but it’s only the best-educated who get the benefits. And the classic illustration of this would be work that’s been done by Nie and his collaborators for the US actually on political engagement. And their finding is really that the effects of education on being engaged politically in key activities, and they’re talking about belonging to parties, campaigning for parties, voting, of course, having particular influential roles in parties. They’re saying that actually, this is promoted by education, but it’s strictly positional, in as much as only some people can be at the center of the action, and only the people at the center of the action are going to have real influence. And that, again, is a positional matter. So, it’s only the very best educated, who will get those key roles in party campaigns, that will get to advise people, who will get the ear of important people, who will influence policy and because of all those opportunities they’re having, they’re more likely to be interested and engaged to do those things. And the argument these authors make is that it’s a zero-sum game, basically. There are only a certain number of what they call network central positions where you can have an influence. And twice as many people may have degrees, but it’s still only going to be that small percentage who are best connected and best educated, who will have the effect. So, this is how they would explain why in a country like the US, where people are more and more educated, actually levels of political engagement are going down in actual fact. Voting levels are going down, and so on and so forth. And what’s more, the younger generations, who are more educated than older generations, are less prone to be politically engaged. And that’s why it’s a relational thing, it’s positional. So, not everybody can be the best educated. So, in this case, the effects of education on individuals is not translating into societal benefits at all. It’s translating into some benefits for some people. That’s true probably have quite a lot of the things we think of in terms of the social benefits of education. They are probably of that nature, they are positional. And one other possibility, which is really quite different, actually. But it’s quite important nevertheless, is what you might call or what I would call distributional effects. And this is not to do with individuals. Individuals don’t have distributions. It’s to do with how education and skills, the outcomes of education, are distributed across society. And whether it’s that very distribution that affects social cohesion. And we have done some research on this, which follows a similar logic to a lot of the research done by people like Wilkinson and Pickett on the effects of income inequality on social outcomes. So, the very popular and widely disseminated work of Wilkinson shows that if you look across a range of societies, societies, where incomes are more equal, generally have better health, lower childhood obesity, lower suicide rates, lower mental health problems, they have higher levels of political engagement, and the list goes on and on and on. I mean, so many social benefits, some of which you’d associate with social cohesion, are related to lower levels of inequality. Trust, by the way, is primary amongst those. So, it would seem that it’s hard to explain exactly what’s happening. But it would seem that lower levels of inequality and income, anyway, is important. While in the same way, lower levels of inequality and skills may be important to achieving certain social benefits from learning. And there is a psychological theory behind this, which is quite plausible. It’s quite hard to prove statistically, but certainly, there’s a theoretical model of its believable, and that basically is about in societies with very unequal levels of education, very unequal levels of skill, the social distance between groups of people at different levels tends to be greater. Therefore, compensation tends to be more difficult and suspicion is higher, and trust is lower. But at the same time, in unequal societies, you have more high-stakes competition. There’s much more at stake in any given competition over resources, jobs, housing, whatever it is, there is more at stake quite simply because the top end is a long way away from the bottom end. And the argument that social psychologists make about these situations is that -and actually, you can find it in behavior of animals as well- if you put a lot of people in high stakes competitions, where there’s high levels of inequality, there’s high levels of stress and anxiety, and stress and anxiety is associated with all sorts of negative social outcomes, particularly negative health outcomes. So, those would be distributional kind of relationships which is education skills and positive social outcomes. And here, it’s not how much you educate any individual that matters, it’s really how you spread the education around.

Will Brehm 26:51
So, of these four approaches, the absolute direct effects, the cumulative effects, the relative effects, and the distributional effects, which approach do you normally think is the correct way to approach social cohesion?

Andy Green 27:09
I think it depends on the case. It’s horses for courses. You’re looking at particular values in each case, things you can measure. And in the case of tolerance, I think the argument is pretty strong that it is a direct effect of education, which can be affected by context, as well. But it is an effect of education. If you’re looking at, as I said, something like political engagement, it is pretty positional. And there may be quite a lot of other effects, which are in the same way positional. If you’re looking at trust, I will put that in amongst those things where you have to look at the distributions, how education is spread around. It seems just if not more important than the actual levels of education in the averages in a country in cross-country studies. And this is where policymakers lose touch slightly, I think, because they’re inclined to only look at individual direct effects. It’s easier to comprehend. They’re a bit suspicious of distributional matters even though the evidence is there.

Will Brehm 28:22
Talking about social trust, how is that even measured? How do you go about measuring social trust in a society?

Andy Green 28:30
Well, social trust has been measured, actually, for quite a long time. Going back -for Western societies- at least going back to the 1960s. There was a study by Almond and Verba about civic culture, which had some of the first measures. And it’s been measured, going through with the European value survey and the world value survey starting not long after that, right through to the present day. And they generally asked the same question, which is, how much do you think other people can be trusted? Or would you never be too careful? Or variance on that? So, there’s a pretty standard question they ask people, and it does seem to be tapping into the core aspect of what you’re looking at, which is ‘do people trust other people they don’t know?’ So, social trust is not about trusting members of your family. It’s not about trusting people you know well; it’s not about trusting institutions. It’s about trusting people you don’t know. And it can be, it’s been measured. And it varies across countries very substantially. And it changes over time rather slowly. But the differences are very marked. So, in Scandinavian countries, typically 70 or 80% of people will say they generally trust other people. In some Latin American countries, it goes down to 20%. And we can do various tests to see if they are answering the question in the way we are meaning it to be asked. And it does seem that they are talking about the same thing, which is do they trust strangers. And it turns out that this is fairly crucial to -certainly is central to social cohesion. It’s the only thing that everybody would agree probably is a measure of social cohesion. But it seems to be very important to economic life, GDP growth has been related to social trust, clearly, it’s much easier to conduct business if you can trust people, legal costs are low, and so on. It’s also been linked to innovation and economic life because people cooperate together better. And it’s linked to a whole range of positive social outcomes, not least good health, and, of course, low levels of conflict. It’s almost the inverse of conflict. So, trust is a definable and a measurable phenomena. You could also measure it ‘trust for institutions,’ political trust, which is slightly different. It is important, it has consequences, and it is a key part of what most people mean by social cohesion. And it varies a lot across countries.

Will Brehm 31:30
And so, what would be the connection of social trust to education? Do people learn social trust in schools, or are there other factors learning social trust in families or in religious institutions?

Andy Green 31:48
Well, that is a very good question actually and not easily answered. I mean, a lot of people who write on this will say that trust is a very fundamental attitude, it’s learned early in life, it’s about childhood socialization, you basically learn to trust through trusting your parents, and other members of your family. And I’ve no doubt there’s some truth in that. So, we’re talking about really fundamental socio-psychological properties of people. And in one respect, trust is almost synonymous in that sense with what you might just call normally optimism. You know, you generally expect to get the best out of any situation. Because you feel in control, because you trust others not to cheat, because you think they won’t cheat with you, which has a lot to do with your position and social status, but also your just levels of confidence. So, those childhood influences are extremely important. But the research also shows that circumstances in adulthood do change people’s levels of trust. So, you know, if they have bad experiences as adults, they can move quite a lot on the trust scale. And social contexts then are making a difference, whether it’s childhood contexts or adult contexts. And there is a remarkable and quite wide shifts going on. So that, for instance, in England, something like I think 50%, 60% of people used to say they trust other people, say when I was growing up in the 60s. This is now down to below 30%, which you could argue is a real culture shift. You know, we live in a different type of society now where people are much less trusting of other people. Why does education affect it? Well, that’s not so easy to answer either. It does seem to be true in most countries that more educated people are more trusting. It’s probably partly to do with the fact that they’re more confident and more optimistic. It may be because they’ve been able to rid themselves of some prejudices which might otherwise stop them trusting people. It may be because they’ve been through a school system that really puts a very strong emphasis on people cooperating. And one of the interesting things about the Nordic education systems is that they have children that that cooperation is absolutely a central purpose of learning with young children particularly. And they keep children in the same classes through the whole of school with the same sets of children, the same teachers. And the purpose of this is very much to promote a group of people who learn how to work with each other to share and to cooperate. So, probably it’s both the behavioral things you learn in education which affect it and also the more general effects of education on your levels of confidence. As well as maybe some cognitive things, some knowledge, which is beneficial. So, it’s quite complex, but there’s no doubt the more educated people in most countries are more trusting.

Will Brehm 35:11
So, if social cohesion is a global policy, and most national governments are thinking of something about social cohesion in their education policies. And if social trust is a good measure of social cohesion, what sort of advice would you give policymakers about accurately thinking about social trust and education?

Andy Green 35:41
Okay, well, it is, as we’ve discussed, a key aspect of social cohesion. It should be at the center of concerns and possibly more than it is in policymaking. I would say two things. I mean, behavior is important. Cooperation in schools is important. So, you know how children relate to each other in school, how they’re taught to relate to each other, is building a foundation of cooperation and trust. So, it’s not just about the syllabus, it’s about the hidden curriculum, and so on. It’s about the rules, and the way schools are organized, which is very important. But the other point I’ve made, which arises out of our research, is that you really have to address not only the average levels of education as a society, which is the obsession of most policymakers because of the PISA test and all the rest of it. You’ve got to look at how education is distributed. Countries with very, very unequal distributions of skills are going to be unequal societies, and they’re going to be less trusting societies. We know this almost certainly applies across the developed world in any case. So, you have to look at both. You have to try to raise the general standards of education and skill, but you have to look at how these skills are distributed and try to reduce the rather large gaps that are actually increasing in social gaps in skills in many countries.

Will Brehm 37:17
Well, Andy Green, thank you so much for joining FreshEd.

Andy Green 37:21
Thank you.

Want to help translate this show? Please contact info@freshedpodcast.com
Have useful resources related to this show? Please send them to info@freshedpodcast.com


This is the last installment of the FreshEd mini-series on global learning metrics. On Thursday, the CIES Symposium kicks off in Scottsdale, Arizona.

For this last show, I’ve invited Karen Mundy to talk about the Global Partnership for Education.

Karen offers interesting insight into learning metrics because she is both an academic and a development practitioner.

Karen Mundy is the Chief Technical Officer at the Global Partnership for Education. She came to the Global Partnership for Education in 2014 from the University of Toronto where she was Professor and Associate Dean of Research, International and Innovation.

She will present some of the ideas discussed in this podcast at the CIES Symposium in Scottsdale Arizona, which starts on Thursday.

Now it’s time for me to catch my flight! See you in Scottsdale!


We often think of international assessments as being synonymous with PISA, the OECD international assessment that has been the focus of many shows in FreshEd’s mini-series on global learning metrics.

But international assessments have a history far beyond PISA. In fact, it was the International Association for the Evaluation of Educational Achievement, known as the IEA, that first introduced large-scale comparative studies of educational systems in the late 1950s.

This history is important to consider when thinking about global learning metrics today.

My guest today is Dirk Hastedt, Executive Director of the IEA. He’s spent many years working with the IEA, seeing the development of assessments in new subjects, such as citizenship and computer literacies, and the emergence of league tables, which rank education systems and have become popular today. Drik offers valuable insight for any discussion on the feasibility or desirability of global learning metrics.

Citation: Hastedt, Dirk, interview with Will Brehm, FreshEd, 49, podcast audio, November 7, 2016. https://freshedpodcast.com/dirkhastedt/

Will Brehm 1:54
Dirk Hastedt, welcome to FreshEd.

Dirk Hastedt 1:56
Thank you, Will. It’s a pleasure for me.

Will Brehm 2:00
The history of international assessments reveals many different actors and debates about the role and purpose and presentation of assessments that we sometimes forget today. I mean, today, we often talk about PISA or TIMSS, but we don’t remember the history of those tests and international testing in general. Can you give us a brief kind of background to the history of international assessments?

Dirk Hastedt 2:26
Sure, Will. It’s a pleasure. I think it’s also very important to go back and think about what it’s all about. Well, the IEA’s origin dates back to 1958 already. And it started as a collaborative venture between measurement scholars, educational psychologists, sociologists, and psychometricians from different countries. They first gathered at the UNESCO Institute for Lifelong Learning in Hamburg, and then also in London, they met a couple of times. And the idea was to exchange ideas about educational processes in order to improve educational systems. And at that time, little was known about the process of education across the world. And the founders of the IEA proposed to engage in cross-national comparisons to student achievement, as they believe that the country’s educational systems can better be understood when comparing educational systems of different countries. And their opinion was that surveys must take into account not only inputs but both input and outcomes of educational processes. And outcomes were defined quite broadly, as achieved knowledge but also as attitudes and participation in education, which now is also another focus that we have in developing countries. And their interest was to collect data that would enable to better understand educational systems and to identify relevant factors that had an impact on student learning. So, the target of their analysis was notably the educational system, not the individual student. And to facilitate further cooperation of researchers, finally, the IEA became a legal entity in 1967. And that’s when all of this started.

Will Brehm 4:25
A legal entity of a particular country?

Dirk Hastedt 4:29
Well, that’s an interesting question. At that time, the researchers tried to found an umbrella organizations to do international research with international people from around the world. And so, they finally decided that this is only possible under the Belgium law. So, it first was emerged as a Belgium association with board members from different countries actually.

Will Brehm 4:59
And you said that originally it was focused on improving education systems, not the achievement of individual students. So, has that changed over time?

Dirk Hastedt 5:12
No, not really. The focus is still on the system level. But the focus always was on comparing systems, and not as we sometimes see today, the competition of systems. So, who is first in league tables? That was never the intention, and I think it still should not be. The question was, what can we learn from other countries’ practices? And in that sense, at that time, it was unclear if cross- cultural and cross language assessment at the beginning were feasible at all. So, they first conducted a study just to prove that this kind of assessment makes sense at all in general.

Will Brehm 6:05
Can you tell us a little bit about that first test? What exactly was uncovered and found?

Dirk Hastedt 6:11
Oh, that’s a good question. The first study was the pilot 12 country study. And that was done in the 1960s beginning. And the subjects included mathematics, reading comprehension, geography, science, and also non-verbal abilities. And the first study, however, started with mathematics because, at that time, the researchers also believed that the subject mathematics would be most language and cultural independent, and consequently, easiest to assess in international study. Later on, the six subject study that was conducted in 1970, and 71, expanded also the scope to science, reading comprehension, literature, education, English as a foreign language, and civic education. So, you can see that the target was very broadly defined in the beginning.

Will Brehm 7:18
And is that target still broadly defined today?

Dirk Hastedt 7:23
Well, there are different projections taking place. One is it’s always a question of what’s a focus in a particular time. So, for example, in the 1990s and late 80s, computers in education also became a focus. So, the IEA, at that time, started a ‘Computers in Education Study,’ COMPED, which was conducted in 1989 and 92, and provided data on the educational use of computers. This trend also was followed later on with our computer information study size and also the current International Computer Information Literacy study. And also, political changes in the world of the 1970s gave rise to the subject of citizenship. As a response, the IEA conducted, at that time, the civic education study, CIVED, and it investigated civic knowledge and engagement and policies and practices.

Will Brehm 8:39
So, different subjects or topics become of interest to the IEA depending on what’s going on in the world. So, today, what’s going on in the world? And what are the topics that are kind of trending?

Dirk Hastedt 8:56
Well, actually, there’s a little shift also in the 1990s, which we sometimes called the empirical shift also. And at that time, international large-scale assessment started to be used increasingly by policymakers, also outside the domain of education. And also, at that time, an extension to more developing countries took place. Before, these studies were more academic studies, and support from policy side was not that big. But at that time, this whole environment changed. And on one hand, this also had a change of the focus in skills like reading and numeracy, which are seen as preconditions for further learning of students. So, without reading abilities, textbooks and other areas can’t be read and understood. But also, economists were interested in relating educational outcomes to economic wealth. And this also resulted in a focus and change of interest of subjects to subjects that are useful for employability and regarded as preconditions for economic wealth. And here again, reading, numeracy, and also science became a focus of international assessments.

Will Brehm 10:32
So, it sounds like these assessments, they change for all sorts of reasons over time, and different actors maybe get more influence inside the testing agencies. And one of the moments that you referenced earlier was when league tables came about. And you were part of the IEA when this debate was happening. Can you give us a little insight about what were the different sides of the debate to either include or exclude league tables?

Dirk Hastedt 11:08
Oh, that’s a very good question. League tables were highly debated also in the studies, and that most of these debates took place in the 1980s, actually already, and beginning of 1990s. The IEA was always interested in understanding educational processes, and which education system is first or second or seventh doesn’t matter too much, actually. And it’s also not very informative for most policymakers. So, consequently, in the early studies, the IEA produced reports and related background instruments and background information to achievement but did not produce league tables. But then, the IEA realized that if they don’t do it, other people created league tables. And we found league tables in magazines, newspapers, and also in some academic journals. But these league tables were also then created in a wrong way. So, they were wrong. So, at that time –

Will Brehm 12:21
How were they wrong?

Dirk Hastedt 12:23
Well, we have a very complex system for assessing the achievement. For example, we use rotated booklet. That means that we want to cover different areas, and consequently, not all students are getting all items, because then every student would have to sit down for six or 10 hours to cover all the areas, which we, of course, do not do. So, that’s one thing. And secondly, also, we are taking a random sample of schools. And then, within the schools, we are sampling one or two classrooms with students. And the researchers call this a cluster sample. And a cluster sample is very different psychometrically from having a simple random sample. And we use different mathematical procedures to calculate, for example, standard errors, or if differences are statistically significant different. And when not considering this, you get wrong results, and you consider results different that are statistically not different. So, there are some parts that really require a good and deep understanding of the projects and its procedures.

Will Brehm 13:46
So, these league tables that weren’t necessarily considering all these statistical and methodological considerations as the IEA was considering. These league tables were basically being produced with potentially wrong information. And so, the IEA decided that it needed to jump in and actually produce accurate tables?

Dirk Hastedt 14:08
That is right. So, when we saw that other people are producing, obviously, these league tables anyway, because of interest. Then the IEA decided, well, if it’s done, then it should be done correctly. So, we decided to include this also in the report.

Will Brehm 14:25
So, over the last 30 years or so, when these league tables have been included in the report, what sort of effects or outcomes have you seen being used of these league tables? Because this is, for the most part in like the popular press, this is what we read about. We read about which school system or country is ahead of another country. And so, these league tables have really, in a way, have become the defining feature of these international assessments.

Dirk Hastedt 14:56
Yeah, well, I think there are different components and different things that our international large-scale assessments are good for. On one hand, of course, this international large-scale assessment describes the status quo in terms of how educational practice in a country is conducted. And this helps researchers and policymakers to compare a country to other countries. And then there’s a comparative perspective. So, identifying differences in ways in which education is organized and practiced across cultures and societies. Then this gives you an understanding of where you are actually. Secondly, I think international assessments like our Trends in International Mathematics and Science Study (TIMSS), or the Progress in International Reading Literacy Study (PIRLS), or the ICILS study, and the Civic and Citizenship Education study that I mentioned, they are measuring trends. And this helps policymakers and researchers to understand changes in educational outcomes and processes, also in comparison to other countries’ development. And especially when curricula or educational policies are changed, it is important to monitor changes. Imagine a captain of a big ship on the ocean. Which captain would not look at his instruments or look outside the window to see where his ship is heading to? And I think there is also the value of maybe also the league tables if you take them and interpret them correctly, in its context, and also looking at trends over time.

Will Brehm 16:42
So, looking at these trends and monitoring and describing the status quo, have these helped policymakers, do you think?

Dirk Hastedt 16:53
Oh, sure. I think policymakers also have seen different impacts from these studies, actually. And it depends very much on the country itself, actually. Since we are research organizations, we are conducting these studies, and we also help countries understanding and interpreting the data. But we do not give any policy recommendations from our side. But we help the countries to do that. And the focus in different countries is very different. We have some countries which, even before the actual assessment start, when we look at the curriculum, they realized that their curriculum is not in line too much with other countries curricula in terms of curriculum expectations. So, some countries, when they looked at this, changed the curriculum expectations and have now higher expectations for the students to be also international comparable. Other countries looked at different sub-populations. For example, immigrants, or they looked at boy-girl differences. So, there’s a lot of questions around equity in education. And policymakers, after looking at the results, took measures to provide more equal learning opportunities to students. One example that we can see is that there was always a discussion about differences of boys and girls in mathematics and science in particular. And 20 years ago, in most countries, boys were performing much better than girls actually. So, researchers and policymakers asked, is this a natural given that boys are better in these areas? And actually, it is not. But it depends very much on motivation, and what are the textbooks, and what are the examples that they used in textbooks? Are they also engaging for girls? So, in a lot of countries, there’s now a shift also to have more relevant and interesting materials that also engaged student girls more in science and mathematics education with the outcome that now the girls are doing as good as boys in most countries. And in a lot of countries, girls are even outperforming the boys in mathematics and science. So, there’s a lot of different conclusions that policymakers have drawn from looking at the results of these international large-scale assessments like the IEA studies.

Will Brehm 20:01
Have there ever been any misuses by policymakers of large-scale international assessments?

Dirk Hastedt 20:10
Misuses. Well, I think it’s not only maybe a question of misusing the data, but maybe also misunderstanding the data. What we do is we do statistically analysis. And there’s always a certain error margin around it. So, coming back to this league tables, again, some of the misunderstandings around these league tables by policymakers, or also the general public, are that if a country is one score point better on a scale with a mean of 500 and a standard deviation of 100, it is better. But we always have a measurement error and sampling error around it. So, there’s some variance around it. So, if one country’s achievement is one score point better than another one, or if one country’s achievement increased by one score point, that doesn’t mean anything, actually. But it might just be a question of the error term around it. So, actually, no change has happened, but it appears as if there would be an increase. But this is statistically not significant, and consequently, there are some misinterpretations of these results if you look only at league tables.

Will Brehm 21:48
So, in your experience, how many policymakers actually understand the statistics behind these studies?

Dirk Hastedt 21:55
Well, probably not many policymakers. And actually, I don’t think it’s something that they need to understand, actually. But I think it’s also something that researchers in the field have to explain to policymakers. And this is probably one of the crucial aspects to communication between researchers and policymakers. Policymakers have their perspective, and they want to have a clear and straight answer. But educational systems are quite complex. And if you look at the outcome of the studies and what researchers think about it, their answer is usually not that simple. And simplifying the results of educational studies in a way that it’s not over-interpreting the results of the studies is very difficult and challenging. So, I think we need more and more to pay attention about how to communicate results and a good communication between researchers who have the technical terms, and policymakers and the general public.

Will Brehm 23:17
But of course, we also have to be realistic that policymakers are navigating domestic politics and may use these international assessments to further their own interests, even if they use a finding in a wrong way or something like that. That will happen, and I think many researchers have shown that that happens.

Dirk Hastedt 23:41
I think you’re right. There’s also, maybe some misuse from policymaker side. But my experience, actually, is that this happens sometimes in a way that policymakers want to look good from the outside. So, they want to have good results for their educational systems, which I think is very natural. Everyone wants to have good results from the work that you do. But I think when it comes to making use of the results, there’s a strong focus from a lot of policymakers that they really want to improve their system. And this can be only done if you really interpret the results correctly. So, maneuvering blind and pretending you’re good might be good for that day, if there’s a day of election, but in a long-term process, you need to understand the educational system. And I think this is also what policymakers understand. That they use the data, and we call it “data-driven policies,” to look what are the strengths and weaknesses of their systems, and then, by that, also try to improve their systems. And if you look at policies today, they really make use of the data with the aim to improve educational systems. And this is, I think, what policymakers are mostly doing.

Will Brehm 25:20
What I find so interesting about the IEA and the history that you’ve recounted is that in the beginning in 1958, doing these cross-national assessments was very much an academic pursuit. You know, is it possible? Can we statistically compare education systems in different countries? And it seems, over time, it’s shifted to being not only an academic pursuit but also very much an issue of policy, and it seems like the role of the IEA, in a way, has slightly changed.

Dirk Hastedt 26:03
That’s a good point, I think. I don’t know if the role of IEA has changed. But it’s clear that we have member institutions, and these member institutions are sometimes research institutes from different countries, and we have member institutions from more than 60 countries. But it’s also people from ministries of education, or institutes that are connected directly to the ministries of education. And surely, they want to learn which policies can help improving achievement in their countries, which is very different from maybe the researchers point who for purely academic intents how … want to understand how educational processes are working and to understand these processes more in-depth.

Will Brehm 27:05
It’s also interesting that originally the test was -mathematics was the subject. Because, as you said, it was, in a sense, “easier” to control for context and local variation, and language variation, for that matter, around the world. But today, you’ve now talked all about all these different subject areas that are tested. How has the issue of context and variation been controlled in these various tests?

Dirk Hastedt 27:40
Oh, that’s a good question. What we are using in IEA is, we are looking at what we call the curriculum model. So, we are looking at the intended curriculum in the different countries. So, what they have written as policy documents, what should be taught in schools, and what are the aims. Then we look at the implemented curriculum. So, what teachers are actually doing in the different countries. And finally, we look at the attained curriculum. So, what have students learned. And this is what we measure with our assessments. But there are also these other fields like the intended and implemented curriculum, and we look at this whole process. And of course, when curricula are changing in participating countries, then also, there is a slight shift in what we are measuring in our international assessment. And I think what’s also very important is to understand that in these studies, we are not only having an assessment, but we also have a huge amount of background variables. So, when we are doing one of this assessment, we are looking at system- level information. So, the curriculum and other relevant information. We have a questionnaire to the school principals of the schools where the students are in about how education is organized, what’s the school’s size and background instruments on that. We have questionnaires to the teachers of the actual students so that we can directly relate teachers’ attitudes, perceptions, and background to student achievement. And then, we also have background questionnaires for all the students. And in grade four, we also have questionnaires to the students’ parents. So, we have huge amounts of background information to analyze what are the factors related to student outcome. And outcome are still not only achievement but also opinions, self-confidence, etc.

Will Brehm 30:14
And this wide range of measures allows IEA, or the tests, to compare across nations and account for the cultural variation?

Dirk Hastedt 30:34
Well, the cross-country cultural comparisons are, of course, always challenging. Education always takes place in a different culture, in different societies, with a history and with a background. But what we can see is that there are a lot of similarities across countries. Sometimes more than you would expect at first glance. Let me cite one of the results from one of our recent studies, which was the Computer Information Literacy study, ICILS, where we looked at what matters most for the usage of computers and computer information literacy of the students. And we had 22 educational systems taking part in that study. And we found that in all these educational systems, the crucial points were the teachers and the teacher education and what they think about computers, and if they want to use computers, and how self-confident they were in using computers in their teaching. And that was the same across all 22 different educational systems. And we had participants from Europe, from Latin America, from Asia. So, for more or less all around the world, and you see that you find the same patterns across different cultures, which was a surprise to us. But also, probably, to a lot of countries because we often hear that, “Well, our educational system is very different. And we have a problem in our country that’s probably very unique.” And when you look at these international assessments, you can see that a lot of countries are facing very similar problems.

Will Brehm 32:45
So, you’ve been involved with the IEA for quite some time, and you’ve mentioned all of these different shifts that have happened. So, in the 1970s, the focus became citizenship, and in the 1980s, it became computers, and in the 1990s, you talked about this empirical shift, I wanted to see -kind of reading the tea leaves- do you see any shifts occurring now? Or do you envision any future shifts about what the focus will be for these international assessments?

Dirk Hastedt 33:20
Well, on one hand, I think that the current process will continue. But on the other hand, I think a huge influence currently is with the UN and the Declaration for the Sustainable Development Goals, where the UN declarations set under target four is concerning with education. So, there’s also in all member countries also UN which means mostly all the world. There’s a common agreement that education plays a more and more important role. And when we look at the target four of the Sustainable Development Goals, one is universal primary and more and more secondary education. And here I see, for example, also a shift on the UN side, which before looked mostly on enrollment rates, now also target minimum standards of outcomes in literacy and numeracy for all countries. So, I see that there will be an expansion also to more developing countries, but still with a focus on numeracy and literacy abilities. But under the target 4.7, it also targets, for example, global citizenship and sustainable development. So, it’s also seen that this is an increasingly important aspect, which we see. And I think that’s a very, very important thing to understand that education is not only with means of educating the future workforce, but education is much broader and also helps us to live together in peace and to also understand the needs of future generations. And there I see also a shift on looking also at global citizenship competencies around the world.

Will Brehm 35:54
Well, Dirk Hastedt, thank you so much for joining FreshEd.

Dirk Hastedt 35:58
Thank you very much. It was a pleasure to meet.

Want to help translate this show? Please contact info@freshedpodcast.com
Have useful resources related to this show? Please send them to info@freshedpodcast.com