Video of the conference: https://education.asu.edu/global-learning-metrics
Position papers by the speakers: https://education.asu.edu/globalmetrics/speaker-position-statements
Posts
Setting the stage for the CIES Symposium on Global Learning Metrics
OverviewTranscriptTranslationResources
We often think of international assessments as being synonymous with PISA, the OECD international assessment that has been the focus of many shows in FreshEd’s mini-series on global learning metrics.
But international assessments have a history far beyond PISA. In fact, it was the International Association for the Evaluation of Educational Achievement, known as the IEA, that first introduced large-scale comparative studies of educational systems in the late 1950s.
This history is important to consider when thinking about global learning metrics today.
My guest today is Dirk Hastedt, Executive Director of the IEA. He’s spent many years working with the IEA, seeing the development of assessments in new subjects, such as citizenship and computer literacies, and the emergence of league tables, which rank education systems and have become popular today. Drik offers valuable insight for any discussion on the feasibility or desirability of global learning metrics.
Citation: Hastedt, Dirk, interview with Will Brehm, FreshEd, 49, podcast audio, November 7, 2016. https://freshedpodcast.com/dirkhastedt/
Will Brehm 1:54
Dirk Hastedt, welcome to FreshEd.
Dirk Hastedt 1:56
Thank you, Will. It’s a pleasure for me.
Will Brehm 2:00
The history of international assessments reveals many different actors and debates about the role and purpose and presentation of assessments that we sometimes forget today. I mean, today, we often talk about PISA or TIMSS, but we don’t remember the history of those tests and international testing in general. Can you give us a brief kind of background to the history of international assessments?
Dirk Hastedt 2:26
Sure, Will. It’s a pleasure. I think it’s also very important to go back and think about what it’s all about. Well, the IEA’s origin dates back to 1958 already. And it started as a collaborative venture between measurement scholars, educational psychologists, sociologists, and psychometricians from different countries. They first gathered at the UNESCO Institute for Lifelong Learning in Hamburg, and then also in London, they met a couple of times. And the idea was to exchange ideas about educational processes in order to improve educational systems. And at that time, little was known about the process of education across the world. And the founders of the IEA proposed to engage in cross-national comparisons to student achievement, as they believe that the country’s educational systems can better be understood when comparing educational systems of different countries. And their opinion was that surveys must take into account not only inputs but both input and outcomes of educational processes. And outcomes were defined quite broadly, as achieved knowledge but also as attitudes and participation in education, which now is also another focus that we have in developing countries. And their interest was to collect data that would enable to better understand educational systems and to identify relevant factors that had an impact on student learning. So, the target of their analysis was notably the educational system, not the individual student. And to facilitate further cooperation of researchers, finally, the IEA became a legal entity in 1967. And that’s when all of this started.
Will Brehm 4:25
A legal entity of a particular country?
Dirk Hastedt 4:29
Well, that’s an interesting question. At that time, the researchers tried to found an umbrella organizations to do international research with international people from around the world. And so, they finally decided that this is only possible under the Belgium law. So, it first was emerged as a Belgium association with board members from different countries actually.
Will Brehm 4:59
And you said that originally it was focused on improving education systems, not the achievement of individual students. So, has that changed over time?
Dirk Hastedt 5:12
No, not really. The focus is still on the system level. But the focus always was on comparing systems, and not as we sometimes see today, the competition of systems. So, who is first in league tables? That was never the intention, and I think it still should not be. The question was, what can we learn from other countries’ practices? And in that sense, at that time, it was unclear if cross- cultural and cross language assessment at the beginning were feasible at all. So, they first conducted a study just to prove that this kind of assessment makes sense at all in general.
Will Brehm 6:05
Can you tell us a little bit about that first test? What exactly was uncovered and found?
Dirk Hastedt 6:11
Oh, that’s a good question. The first study was the pilot 12 country study. And that was done in the 1960s beginning. And the subjects included mathematics, reading comprehension, geography, science, and also non-verbal abilities. And the first study, however, started with mathematics because, at that time, the researchers also believed that the subject mathematics would be most language and cultural independent, and consequently, easiest to assess in international study. Later on, the six subject study that was conducted in 1970, and 71, expanded also the scope to science, reading comprehension, literature, education, English as a foreign language, and civic education. So, you can see that the target was very broadly defined in the beginning.
Will Brehm 7:18
And is that target still broadly defined today?
Dirk Hastedt 7:23
Well, there are different projections taking place. One is it’s always a question of what’s a focus in a particular time. So, for example, in the 1990s and late 80s, computers in education also became a focus. So, the IEA, at that time, started a ‘Computers in Education Study,’ COMPED, which was conducted in 1989 and 92, and provided data on the educational use of computers. This trend also was followed later on with our computer information study size and also the current International Computer Information Literacy study. And also, political changes in the world of the 1970s gave rise to the subject of citizenship. As a response, the IEA conducted, at that time, the civic education study, CIVED, and it investigated civic knowledge and engagement and policies and practices.
Will Brehm 8:39
So, different subjects or topics become of interest to the IEA depending on what’s going on in the world. So, today, what’s going on in the world? And what are the topics that are kind of trending?
Dirk Hastedt 8:56
Well, actually, there’s a little shift also in the 1990s, which we sometimes called the empirical shift also. And at that time, international large-scale assessment started to be used increasingly by policymakers, also outside the domain of education. And also, at that time, an extension to more developing countries took place. Before, these studies were more academic studies, and support from policy side was not that big. But at that time, this whole environment changed. And on one hand, this also had a change of the focus in skills like reading and numeracy, which are seen as preconditions for further learning of students. So, without reading abilities, textbooks and other areas can’t be read and understood. But also, economists were interested in relating educational outcomes to economic wealth. And this also resulted in a focus and change of interest of subjects to subjects that are useful for employability and regarded as preconditions for economic wealth. And here again, reading, numeracy, and also science became a focus of international assessments.
Will Brehm 10:32
So, it sounds like these assessments, they change for all sorts of reasons over time, and different actors maybe get more influence inside the testing agencies. And one of the moments that you referenced earlier was when league tables came about. And you were part of the IEA when this debate was happening. Can you give us a little insight about what were the different sides of the debate to either include or exclude league tables?
Dirk Hastedt 11:08
Oh, that’s a very good question. League tables were highly debated also in the studies, and that most of these debates took place in the 1980s, actually already, and beginning of 1990s. The IEA was always interested in understanding educational processes, and which education system is first or second or seventh doesn’t matter too much, actually. And it’s also not very informative for most policymakers. So, consequently, in the early studies, the IEA produced reports and related background instruments and background information to achievement but did not produce league tables. But then, the IEA realized that if they don’t do it, other people created league tables. And we found league tables in magazines, newspapers, and also in some academic journals. But these league tables were also then created in a wrong way. So, they were wrong. So, at that time –
Will Brehm 12:21
How were they wrong?
Dirk Hastedt 12:23
Well, we have a very complex system for assessing the achievement. For example, we use rotated booklet. That means that we want to cover different areas, and consequently, not all students are getting all items, because then every student would have to sit down for six or 10 hours to cover all the areas, which we, of course, do not do. So, that’s one thing. And secondly, also, we are taking a random sample of schools. And then, within the schools, we are sampling one or two classrooms with students. And the researchers call this a cluster sample. And a cluster sample is very different psychometrically from having a simple random sample. And we use different mathematical procedures to calculate, for example, standard errors, or if differences are statistically significant different. And when not considering this, you get wrong results, and you consider results different that are statistically not different. So, there are some parts that really require a good and deep understanding of the projects and its procedures.
Will Brehm 13:46
So, these league tables that weren’t necessarily considering all these statistical and methodological considerations as the IEA was considering. These league tables were basically being produced with potentially wrong information. And so, the IEA decided that it needed to jump in and actually produce accurate tables?
Dirk Hastedt 14:08
That is right. So, when we saw that other people are producing, obviously, these league tables anyway, because of interest. Then the IEA decided, well, if it’s done, then it should be done correctly. So, we decided to include this also in the report.
Will Brehm 14:25
So, over the last 30 years or so, when these league tables have been included in the report, what sort of effects or outcomes have you seen being used of these league tables? Because this is, for the most part in like the popular press, this is what we read about. We read about which school system or country is ahead of another country. And so, these league tables have really, in a way, have become the defining feature of these international assessments.
Dirk Hastedt 14:56
Yeah, well, I think there are different components and different things that our international large-scale assessments are good for. On one hand, of course, this international large-scale assessment describes the status quo in terms of how educational practice in a country is conducted. And this helps researchers and policymakers to compare a country to other countries. And then there’s a comparative perspective. So, identifying differences in ways in which education is organized and practiced across cultures and societies. Then this gives you an understanding of where you are actually. Secondly, I think international assessments like our Trends in International Mathematics and Science Study (TIMSS), or the Progress in International Reading Literacy Study (PIRLS), or the ICILS study, and the Civic and Citizenship Education study that I mentioned, they are measuring trends. And this helps policymakers and researchers to understand changes in educational outcomes and processes, also in comparison to other countries’ development. And especially when curricula or educational policies are changed, it is important to monitor changes. Imagine a captain of a big ship on the ocean. Which captain would not look at his instruments or look outside the window to see where his ship is heading to? And I think there is also the value of maybe also the league tables if you take them and interpret them correctly, in its context, and also looking at trends over time.
Will Brehm 16:42
So, looking at these trends and monitoring and describing the status quo, have these helped policymakers, do you think?
Dirk Hastedt 16:53
Oh, sure. I think policymakers also have seen different impacts from these studies, actually. And it depends very much on the country itself, actually. Since we are research organizations, we are conducting these studies, and we also help countries understanding and interpreting the data. But we do not give any policy recommendations from our side. But we help the countries to do that. And the focus in different countries is very different. We have some countries which, even before the actual assessment start, when we look at the curriculum, they realized that their curriculum is not in line too much with other countries curricula in terms of curriculum expectations. So, some countries, when they looked at this, changed the curriculum expectations and have now higher expectations for the students to be also international comparable. Other countries looked at different sub-populations. For example, immigrants, or they looked at boy-girl differences. So, there’s a lot of questions around equity in education. And policymakers, after looking at the results, took measures to provide more equal learning opportunities to students. One example that we can see is that there was always a discussion about differences of boys and girls in mathematics and science in particular. And 20 years ago, in most countries, boys were performing much better than girls actually. So, researchers and policymakers asked, is this a natural given that boys are better in these areas? And actually, it is not. But it depends very much on motivation, and what are the textbooks, and what are the examples that they used in textbooks? Are they also engaging for girls? So, in a lot of countries, there’s now a shift also to have more relevant and interesting materials that also engaged student girls more in science and mathematics education with the outcome that now the girls are doing as good as boys in most countries. And in a lot of countries, girls are even outperforming the boys in mathematics and science. So, there’s a lot of different conclusions that policymakers have drawn from looking at the results of these international large-scale assessments like the IEA studies.
Will Brehm 20:01
Have there ever been any misuses by policymakers of large-scale international assessments?
Dirk Hastedt 20:10
Misuses. Well, I think it’s not only maybe a question of misusing the data, but maybe also misunderstanding the data. What we do is we do statistically analysis. And there’s always a certain error margin around it. So, coming back to this league tables, again, some of the misunderstandings around these league tables by policymakers, or also the general public, are that if a country is one score point better on a scale with a mean of 500 and a standard deviation of 100, it is better. But we always have a measurement error and sampling error around it. So, there’s some variance around it. So, if one country’s achievement is one score point better than another one, or if one country’s achievement increased by one score point, that doesn’t mean anything, actually. But it might just be a question of the error term around it. So, actually, no change has happened, but it appears as if there would be an increase. But this is statistically not significant, and consequently, there are some misinterpretations of these results if you look only at league tables.
Will Brehm 21:48
So, in your experience, how many policymakers actually understand the statistics behind these studies?
Dirk Hastedt 21:55
Well, probably not many policymakers. And actually, I don’t think it’s something that they need to understand, actually. But I think it’s also something that researchers in the field have to explain to policymakers. And this is probably one of the crucial aspects to communication between researchers and policymakers. Policymakers have their perspective, and they want to have a clear and straight answer. But educational systems are quite complex. And if you look at the outcome of the studies and what researchers think about it, their answer is usually not that simple. And simplifying the results of educational studies in a way that it’s not over-interpreting the results of the studies is very difficult and challenging. So, I think we need more and more to pay attention about how to communicate results and a good communication between researchers who have the technical terms, and policymakers and the general public.
Will Brehm 23:17
But of course, we also have to be realistic that policymakers are navigating domestic politics and may use these international assessments to further their own interests, even if they use a finding in a wrong way or something like that. That will happen, and I think many researchers have shown that that happens.
Dirk Hastedt 23:41
I think you’re right. There’s also, maybe some misuse from policymaker side. But my experience, actually, is that this happens sometimes in a way that policymakers want to look good from the outside. So, they want to have good results for their educational systems, which I think is very natural. Everyone wants to have good results from the work that you do. But I think when it comes to making use of the results, there’s a strong focus from a lot of policymakers that they really want to improve their system. And this can be only done if you really interpret the results correctly. So, maneuvering blind and pretending you’re good might be good for that day, if there’s a day of election, but in a long-term process, you need to understand the educational system. And I think this is also what policymakers understand. That they use the data, and we call it “data-driven policies,” to look what are the strengths and weaknesses of their systems, and then, by that, also try to improve their systems. And if you look at policies today, they really make use of the data with the aim to improve educational systems. And this is, I think, what policymakers are mostly doing.
Will Brehm 25:20
What I find so interesting about the IEA and the history that you’ve recounted is that in the beginning in 1958, doing these cross-national assessments was very much an academic pursuit. You know, is it possible? Can we statistically compare education systems in different countries? And it seems, over time, it’s shifted to being not only an academic pursuit but also very much an issue of policy, and it seems like the role of the IEA, in a way, has slightly changed.
Dirk Hastedt 26:03
That’s a good point, I think. I don’t know if the role of IEA has changed. But it’s clear that we have member institutions, and these member institutions are sometimes research institutes from different countries, and we have member institutions from more than 60 countries. But it’s also people from ministries of education, or institutes that are connected directly to the ministries of education. And surely, they want to learn which policies can help improving achievement in their countries, which is very different from maybe the researchers point who for purely academic intents how … want to understand how educational processes are working and to understand these processes more in-depth.
Will Brehm 27:05
It’s also interesting that originally the test was -mathematics was the subject. Because, as you said, it was, in a sense, “easier” to control for context and local variation, and language variation, for that matter, around the world. But today, you’ve now talked all about all these different subject areas that are tested. How has the issue of context and variation been controlled in these various tests?
Dirk Hastedt 27:40
Oh, that’s a good question. What we are using in IEA is, we are looking at what we call the curriculum model. So, we are looking at the intended curriculum in the different countries. So, what they have written as policy documents, what should be taught in schools, and what are the aims. Then we look at the implemented curriculum. So, what teachers are actually doing in the different countries. And finally, we look at the attained curriculum. So, what have students learned. And this is what we measure with our assessments. But there are also these other fields like the intended and implemented curriculum, and we look at this whole process. And of course, when curricula are changing in participating countries, then also, there is a slight shift in what we are measuring in our international assessment. And I think what’s also very important is to understand that in these studies, we are not only having an assessment, but we also have a huge amount of background variables. So, when we are doing one of this assessment, we are looking at system- level information. So, the curriculum and other relevant information. We have a questionnaire to the school principals of the schools where the students are in about how education is organized, what’s the school’s size and background instruments on that. We have questionnaires to the teachers of the actual students so that we can directly relate teachers’ attitudes, perceptions, and background to student achievement. And then, we also have background questionnaires for all the students. And in grade four, we also have questionnaires to the students’ parents. So, we have huge amounts of background information to analyze what are the factors related to student outcome. And outcome are still not only achievement but also opinions, self-confidence, etc.
Will Brehm 30:14
And this wide range of measures allows IEA, or the tests, to compare across nations and account for the cultural variation?
Dirk Hastedt 30:34
Well, the cross-country cultural comparisons are, of course, always challenging. Education always takes place in a different culture, in different societies, with a history and with a background. But what we can see is that there are a lot of similarities across countries. Sometimes more than you would expect at first glance. Let me cite one of the results from one of our recent studies, which was the Computer Information Literacy study, ICILS, where we looked at what matters most for the usage of computers and computer information literacy of the students. And we had 22 educational systems taking part in that study. And we found that in all these educational systems, the crucial points were the teachers and the teacher education and what they think about computers, and if they want to use computers, and how self-confident they were in using computers in their teaching. And that was the same across all 22 different educational systems. And we had participants from Europe, from Latin America, from Asia. So, for more or less all around the world, and you see that you find the same patterns across different cultures, which was a surprise to us. But also, probably, to a lot of countries because we often hear that, “Well, our educational system is very different. And we have a problem in our country that’s probably very unique.” And when you look at these international assessments, you can see that a lot of countries are facing very similar problems.
Will Brehm 32:45
So, you’ve been involved with the IEA for quite some time, and you’ve mentioned all of these different shifts that have happened. So, in the 1970s, the focus became citizenship, and in the 1980s, it became computers, and in the 1990s, you talked about this empirical shift, I wanted to see -kind of reading the tea leaves- do you see any shifts occurring now? Or do you envision any future shifts about what the focus will be for these international assessments?
Dirk Hastedt 33:20
Well, on one hand, I think that the current process will continue. But on the other hand, I think a huge influence currently is with the UN and the Declaration for the Sustainable Development Goals, where the UN declarations set under target four is concerning with education. So, there’s also in all member countries also UN which means mostly all the world. There’s a common agreement that education plays a more and more important role. And when we look at the target four of the Sustainable Development Goals, one is universal primary and more and more secondary education. And here I see, for example, also a shift on the UN side, which before looked mostly on enrollment rates, now also target minimum standards of outcomes in literacy and numeracy for all countries. So, I see that there will be an expansion also to more developing countries, but still with a focus on numeracy and literacy abilities. But under the target 4.7, it also targets, for example, global citizenship and sustainable development. So, it’s also seen that this is an increasingly important aspect, which we see. And I think that’s a very, very important thing to understand that education is not only with means of educating the future workforce, but education is much broader and also helps us to live together in peace and to also understand the needs of future generations. And there I see also a shift on looking also at global citizenship competencies around the world.
Will Brehm 35:54
Well, Dirk Hastedt, thank you so much for joining FreshEd.
Dirk Hastedt 35:58
Thank you very much. It was a pleasure to meet.
Want to help translate this show? Please contact info@freshedpodcast.com
Have useful resources related to this show? Please send them to info@freshedpodcast.com
History and development of international assessments
The meaning of “learning” in Global Learning Metrics
OverviewTranscriptTranslationResources
Today we continue our focus on global learning metrics during the lead up to the inaugural CIES symposium, which will take place in Scottsdale, AZ from November 10-11.
The past shows in this mini-series have focused broadly on global learning metrics: We’ve looked at the history and value of learning metrics for the perspective of national governments; we’ve examined the power of tests like PISA; and we’ve heard critiques of policy borrowing and outcome-based approaches to education that rely on learning metrics and their subsequent rankings.
But we haven’t yet looked at some of the questions on the tests that form the proxies for global learning metrics.
My guest today is Dr. Inés Dussel, Researcher and Professor at the Department of Educational Research, Center for Advanced Studies and Research (DIE-CINVESTAV) in Mexico.
She argues that global learning metrics are not culturally sensitive and uses examples from her work on digital literacy to show why.
Inés critiques PISA for taking a narrow focus of learning as only related to cognitive skills — the ability for students to read or write or problem solve. By contrast, she takes a broad view of learning, which encompasses not only cognitive skills but also a collection of interpersonal and social skills. Of course, these latter skill sets are nearly impossible to measure in one school let alone worldwide using universal metrics.
And this is the crux of the issue: how can global learning metrics measure any skill set across so many different contexts and cultures worldwide?
Photo credit: La Nación
Citation: Dussel, Inés, interview with Will Brehm, FreshEd, 47, podcast audio, October 25, 2016. https://freshedpodcast.com/inesdussel/
Will Brehm 2:00
Inés Dussel, welcome to FreshEd.
Inés Dussel 2:03
Hi, hello. Thank you for inviting me.
Will Brehm 2:07
You’ve done some research on digital literacies in Argentina, where one computer per student program was implemented universally throughout that country. Are there global learning metrics that try to measure digital learning, or digital literacy, in education?
Inés Dussel 2:29
Yes, there are. Although they were not used in Argentina or in some other countries that have implemented the one computer per student program. There have been correlations between what you would say traditional learning metrics about reading, mathematics, and the introduction of computers. So, one could say that there was an assumption that if you have more computers in the classroom, then children will improve their learnings. And that was not what happened. For example, in Uruguay, there were no significant improvements in standardized tests because of the introduction of computers or related to the introduction of computers. However, there are global learning studies about digital competencies or digital literacies. For example, one that was carried by the OECD, it was published in 2015 and based on PISA results from 2012. This, I think, to my knowledge, is the most serious study that has been done on the improvement of digital literacies because of the access to computers and the use of computers in schools. The findings of this report have been controversial because what they say is that there is no direct relationship between higher availability of computers in school and the improvement of digital literacies. So, for example, a country like Singapore where children don’t use computers so much in schools, however, they ranked first. And so, these digital literacies or digital literacy skills do not seem related to what is learned in school, but to what children do outside or inside schools, but to how cultures relate to computers too. Also, the widespread availability of computers. And for example, in many East Asian countries, there is wide access to computers, and children and teenagers use them extensively. So, apparently, it is this use at home, in schools, that makes them more fluent for doing better searches in the internet and being more critical about the information they read.
Will Brehm 5:32
Why do you think PISA is trying to develop these metrics on digital literacies?
Inés Dussel 5:41
Digital literacies seem to be a significant part of what is called the 21st-century skills. It’s very obvious to say that we now live in a digital culture. Digital media is everywhere. Our lives, one could say, are digitally mediated in many, many aspects of banking, consumption, health, of course, entertainment and culture. And so, these skills seem to be very crucial for our well-being and also for our social and economic participation in contemporary society. So, I think everybody’s concerned about what is going on with the skills? How do children and young people learn the skills? Is school any relevant for this learning? Can we do better? What are they learning on their own? So, I think, in this area of digital literacies, we have many, many questions. And PISA is trying to grasp some things, I would say not all but at least some things that are related to academic skills or things we could relate to academic performance.
Will Brehm 7:03
So, can you give an example of how PISA would test sort of these digital literacy skills for children?
Inés Dussel 7:12
Yeah. There are some examples. One could be they asked the students, I mean, these are tests that are taken online. So, they can see, for example, how well they do in order to obtain practical information from the internet. I don’t know. I’m trying to look for the information more precisely, but for example, how do they browse? How are their navigation skills? So, do they look around? Do they go more directly to what they want? If they’re given a task, how many clicks? How much time does it take it to get where they should get? But also, the quality of navigation steps is also important. So, there are, I would say, several measurements. However, and I think, of course, the people who do these tests are -I would say they’re very knowledgeable, and they’ve tried it extensively. However, many of these things could be, I would say, interpreted differently, because I don’t know, you browse. Are you efficient? In what terms? Is it time-related? Is it quantifiable? Do we really know what kids understood because of this browsing activity by how many clicks they made? So, I think we get, as probably with other tests, we get proxies of what they’re learning. But I am not sure that we really know what they are learning through this kind of quantifiable measurements. And I would say very general. In order to have massive data, you also tend to look at very general things. And these things say little. That would be my point about how children learn.
And I would relate this to the research I’ve done in schools. Because I think that if you look in- depth at some schools, and if you look in-depth at some children: What are they doing, how are they doing? Then you understand more things. For example, also a colleague of mine in Mexico, she has done very interesting research on in which sites do children trust. How do they search online? And she found that they generally don’t take many steps. They get tired very quickly. Their efficiency is related to going through well-known places, either Wikipedia, but if Wikipedia is not a consider as valuable or as trustable by their teachers, then Wikipedia is forbidden. Okay, then no. But then how should I orient myself? And for example, see, this colleague of mine posed them a problem. So, now you have two websites that have contradictory information about how the Egyptians mummified their deceased. One side says that they had to take away their heart. And the other one says that they didn’t. And so, who do you trust? Or which one do you trust? And what do you do with that? And so, she found that children had very loose ideas about which website to trust. For example, a blog is no good because I can produce a blog. So, that’s no good at all. Then “.com” is better because “.com” then is a fixed page. So, if it cannot be intervened, then that’s okay. And they didn’t know, for example, the academic sites. I mean, they knew in Mexico, UNAM, the National University. Oh, this might be good. But this knowledge did not come from the internet. This knowledge came from the families, from the teachers. So, what my colleague says is that we have to do much better. I mean, we don’t get to know that by how many clicks they performed. We have to look qualitatively who they trust, which are the steps they are doing, and why, what are they looking for? And I think the most important finding of her research was that children said, “You know, I really don’t care. If you hadn’t asked me, I really don’t care which website is right, because what difference does it make?” And it was not only related to the Egyptian mummies, one could say, yeah, who cares? I think the value of truth -and this is a philosophical, I would say, question, but also a political and ethical question- is in decline. And so, you have to be fast. It doesn’t matter whether you get it right, it doesn’t matter whether you are going in the right direction. And my point would be that there are some similarities between this idea of going fast and being efficient and some of the learning metrics that are not really concerned about what is going on and the depth of the challenges we face, and the ethical question we have about what are we learning?
Will Brehm 12:58
It’s interesting to hear that PISA uses a proxy measure of digital literacy of the speed of a search online by students and how many clicks they do. But like you said that it’s much more complex than that, and PISA might be using the wrong proxy measure for one of these 21st-century skills, but do you think it’s simply a matter of like could PISA come up with a better proxy measure or a better test item to use that could get us closer to this understanding digital literacy?
Inés Dussel 13:40
I think that at a massive scale, it is difficult to get through these other indicators that are much more complex. I also think there should be other conversations of the Global Learning Task Force that they’re producing. And people who understand that learning cannot be captured by metrics. I would say that all metrics are proxies, and I would say probably all measurements are proxies, not only the metrics. Because I have several colleagues who are very good psychologists, I can think of Emilia Ferreiro, for example, who trained with Jean Piaget and many others, newer generations say similar things that really, we are not sure about how children learn and what they learn. And maybe some of the things you see their effects in decades from now or years from now, or months from now. Performance in tests can be tricky because it has to do with many other things. I mean, the language the test is taken, the relationship you have with evaluation, with a school system. If you are shy and if you feel marginalized by the school system, it’s very likely that you will not perform to your best potential. So, I would say there are many things there are that, for me, make the test proxies. Do we have to have test? I would say yes, we have to have tests, but we have to be very careful about how we use that information, what effects they have, and also make very clear arguments about the limitations of the information we collect with these tests.
Will Brehm 15:32
So, like assuming that these sort of skills are universal?
Inés Dussel 15:40
I would say the skills are never universal. I would say they are cultural, and culture is not universal. And again, if one think of reading and I have some examples from my own research and from this comparative research done by close colleagues, in France, in Argentina, in Mexico. In France, for example, reading and writing is very central to the educational system. Young people, by the end of their secondary school, they are able to read novels or literature that is being submitted for national prizes. And the relationship of educational system towards consider high culture or literature is very close. In Argentina, that relationship is very distant, I would say. And perhaps Argentinian children are much better at arguing orally, they have very good oral skills. And these are nurtured, I would say, these are learned through many, many years of schooling in which children have to participate in group activities, they have to even contradict their teacher. And that’s very well regarded. I mean, if the students say, “Oh no, you are wrong. And I have this other opinion.” Whereas in Mexico, I would say these oral skills are not that developed. And I would say these oral skills are important as a 21st-century skill if you want to call them like that. Because there are many things that today go through our presentation, to our ability to persuade and communicate. So, oral skills should not be devalued. They’re not second to other skills, they’re important. And how do we measure them? So, again, now are these universal skills or are they related to cultures that value some kind of competency skills or language dispositions? Or do we think that they are universally processed or owned? And I don’t agree with the idea that skills are universal. And I would say we have to take a cultural approach to skills, understand these skills in cultures in the current debates that cultures have. I remember one of PISA’s question about ecology and pollution. And I was thinking, well, this question would be much easier to first understand and then to solve in a European country than in a Latin American country where the ecological discourse is not a public issue, or not as much, and it was not ten years ago. So, are we measuring what children know innately, or are we measuring what society’s value and how they think about problems, and how they think about how to solve problems? This has been widely discussed, the cultural biases of tests, but I think we have to make the case again because there are still people who think it is possible to get rid of this cultural bias. And I would say, no, it is not possible, then let’s work with that and make more humble claims about what we find.
Will Brehm 19:22
Do you remember the example of this ecology and pollution question?
Inés Dussel 19:27
Ah, I don’t exactly. It was a question about the Greek, the Athens Parthenon, and it was about the pollution that was eroding the statues. And they had to make a calculus variable thinking, well, for many countries, this is not a relevant question. And I don’t mean you can’t solve it. Yes, you can solve it at any rate. But no, it’s not the same. If this is something you hear all the time on the news, and you’re used to this kind of argument, and you are bringing in some other kind of knowledge. For example, I don’t know, does the air pollute, do carbon emissions pollute? I mean, these are things some people hear on the media all the time and on the newspapers and on the families and some other people don’t. So, we have to take that into account.
Will Brehm 20:24
So, with PISA privileging one particular perspective over others, would you see it as a form of like neo-colonialism? Like pushing one idea globally through these tests like PISA?
Inés Dussel 20:39
I think there is a risk of neo-colonialism. Yes. On the one hand, because there is an assumption of a problem-solving individual that is isolated and acts upon knowledge and information and choice. And I think that for many other children, this is not the normal life. I would say, for example, in Latin America, but also in India and Africa, many children who live in shanty towns, in marginalized populations. One thing that is very important is that they learn to go by. They learn to, for example, keep silent and feign ignorance because they know some things shouldn’t be told because they’re dangerous, there are risk, they might be illegal. This is a skill, a very important skill. Not necessarily one that I like, but how does it come into the school knowledge? I’m sure it is there, too. I’m sure it is there in relation to how they feel, for example, with invitation to speak up, to participate, to express themselves, if they know in their daily life that they shouldn’t, that it’s better for them, it’s safer for them not to express themselves. So, what are we measuring when we take them the same test, then we take to children who are raised in societies where this is the norm -speak up, express yourself, be creative, be autonomous? And I think there are lots of cultural biases that we have to take into account. So, yes, there is a risk of neocolonialism when we don’t understand how this norm that we think is universal is not universal, how it is being played in different contexts. And I think that the worst neocolonialism comes with the truth claims of PISA and many other tests that say, this is what it is, these are the best school systems, these are the worst school systems, and they don’t consider how they are performing in their own societies and to their own goals and problems.
Will Brehm 22:56
Do you think it would be possible to construct global learning metrics in a way that is culturally sensitive? It seems as if there are, on the one hand, global learning metrics need to be uniform, globally, to do this sort of comparisons. But at the same time, listening to all of the different examples you’ve given, it’s clear that a universal test question isn’t going to actually work in different contexts around the world. So, it poses a serious problem to global learning metrics that the very essence of them being universal. So, do you think it’s possible for global learning metrics to be culturally sensitive?
Inés Dussel 23:42
I hope it would be possible. I’m not sure it might be possible. One thing I would say is that we have to have more diverse agencies. And it’s not only a matter of representation, but people who work there are aware of all these issues that we are discussing. I would also say –
Will Brehm 24:07
So, you mean. I’m sorry. So, you mean like inside the OECD, we need people making these test questions that come from more countries than just where they currently come from. Is that what you’re saying?
Inés Dussel 24:19
That’s partly what I say. But I would say if people come from, for example, Latin America and share the same kind of understanding there, then that is not any good, I would say. Because it’s not a matter of the country of origin, but it’s can you really bring in different perspectives and discussions? Can you really bring in people who challenged some of these assumptions and who have done work on different kind of methodologies? And related to what we were saying about the comparison, I think the problem is that this would be much more expensive. But what if the reading test took into account cultural differences? I don’t think that’s impossible. Of course, that would make the comparison probably looser, more flexible. But yes, I can think that we can still make some claims about how children read in some countries and how children read in some others. They are not reading exactly the same question. They are, for example, or you could make subsets of countries with similar problems -in terms of violence or human rights or even ecological concerns- but that are not framed in the same language and way which it is framed in the Northern countries. So, I don’t know I’ll give it a try at least. I think we have to give it a try.
Will Brehm 25:51
So, you say that you want to see a shift from learning to schooling. What do you mean by that shift or that difference?
Inés Dussel 25:59
I think education cannot be reduced to learning. Also, because learning is very difficult to measure, really. I do believe that. So, many of the metrics I’ve seen are really not measuring learning. They are measuring what is being offered to children, the kind of access they have, the kind of exposure they have, the preparation of the teachers, but not necessarily what they are learning. And I think that’s because learning -even if you look at children’s notebooks, or if you make interviews with them individually, then it is very difficult to see what they have learned. And as I said before, maybe you know, and they know what they learned in ten years from now, and not now. So, I think that’s very complex. But also, education is not only about learning cognitive skills. There are many things we learn in schools that are important and not necessarily taught by the teacher. But we learn to live with others, we learn to listen, or I hope we learn to listen to others, we learn to be with ourselves, to be outside our homes, we learn that there are other people in society that think differently, we learn that we have to abide by certain rules, and these rules might be flexible, but not too much because that’s what societies do. And I think these other kinds of learnings are very important. School is a public space, and if you want, you can call it socialization, you can call it education in a broader sense. They provide us with other references, with other languages, tastes, and this is also a very important social function. And I would say today, in a world where the risk of isolationism is bigger, where there are many fundamentalisms threatening the possibility of a civilized or peaceful world, then learning to live together is a very important task for schools. And can we measure that? How do we measure that? I think we have to be careful about what we put in the test. And also, the focus of these. We make these the focus of schools while schools have many other functions that are very, very important.
Will Brehm 28:47
So, it’s like the global learning metrics, like PISA, have taken a very narrow focus of what learning is being just cognitive skills that can supposedly be measured. But you’re saying that schools actually have such a broad function for children in society -far beyond just cognitive skills.
Inés Dussel 29:08
Exactly. I think it’s very important that we keep these broader functions alive and as a focus of schools. And I think there are many studies I don’t know in the US, one, which caused much uproar was Diane Ravitch’s 2010 book on the decline of American schools and of the great American public school, and how all this test madness has made things really bad for teachers and children to work in schools, to be together, to decide for themselves what are their goals, what do they want to do, and again, learn some basic things about social life. And I think you can say the same thing. Maybe in Chile, sometimes in Mexico, but there has been very strong pressure on teachers to improve the performance in tests. And then they forget about other things. So, you just train to the test, and you forget that you are there to really educate a human being and to really contribute to a better life and a good society for all.
Will Brehm 30:27
Well, Inés Dussel, thank you so much for joining FreshEd.
Inés Dussel 30:31
Thank you for this conversation.
Want to help translate this show? Please contact info@freshedpodcast.com
Have useful resources related to this show? Please send them to info@freshedpodcast.com
The cultural insensitivity of global learning metrics
The problems with outcome-based approaches to education
PISA, policy referencing, and pantomime
Seeing Like PISA
Today marks the first installment of a seven-part miniseries on Global Learning Metrics. In effort to promote the inaugural Symposium of the Comparative and International Education Society, FreshEd will air interviews with some of the invited speakers.
Schools, skills and economic growth