Today we continue our focus on global learning metrics during the lead up to the inaugural CIES symposium, which will take place in Scottsdale, AZ from November 10-11.
The past shows in this mini-series have focused broadly on global learning metrics: We’ve looked at the history and value of learning metrics for the perspective of national governments; we’ve examined the power of tests like PISA; and we’ve heard critiques of policy borrowing and outcome-based approaches to education that rely on learning metrics and their subsequent rankings.
But we haven’t yet looked at some of the questions on the tests that form the proxies for global learning metrics.
My guest today is Dr. Inés Dussel, Researcher and Professor at the Department of Educational Research, Center for Advanced Studies and Research (DIE-CINVESTAV) in Mexico.
She argues that global learning metrics are not culturally sensitive and uses examples from her work on digital literacy to show why.
Inés critiques PISA for taking a narrow focus of learning as only related to cognitive skills — the ability for students to read or write or problem solve. By contrast, she takes a broad view of learning, which encompasses not only cognitive skills but also a collection of interpersonal and social skills. Of course, these latter skill sets are nearly impossible to measure in one school let alone worldwide using universal metrics.
And this is the crux of the issue: how can global learning metrics measure any skill set across so many different contexts and cultures worldwide?
Photo credit: La Nación
Citation: Dussel, Inés, interview with Will Brehm, FreshEd, 47, podcast audio, October 25, 2016. https://freshedpodcast.com/inesdussel/
Will Brehm 2:00
Inés Dussel, welcome to FreshEd.
Inés Dussel 2:03
Hi, hello. Thank you for inviting me.
Will Brehm 2:07
You’ve done some research on digital literacies in Argentina, where one computer per student program was implemented universally throughout that country. Are there global learning metrics that try to measure digital learning, or digital literacy, in education?
Inés Dussel 2:29
Yes, there are. Although they were not used in Argentina or in some other countries that have implemented the one computer per student program. There have been correlations between what you would say traditional learning metrics about reading, mathematics, and the introduction of computers. So, one could say that there was an assumption that if you have more computers in the classroom, then children will improve their learnings. And that was not what happened. For example, in Uruguay, there were no significant improvements in standardized tests because of the introduction of computers or related to the introduction of computers. However, there are global learning studies about digital competencies or digital literacies. For example, one that was carried by the OECD, it was published in 2015 and based on PISA results from 2012. This, I think, to my knowledge, is the most serious study that has been done on the improvement of digital literacies because of the access to computers and the use of computers in schools. The findings of this report have been controversial because what they say is that there is no direct relationship between higher availability of computers in school and the improvement of digital literacies. So, for example, a country like Singapore where children don’t use computers so much in schools, however, they ranked first. And so, these digital literacies or digital literacy skills do not seem related to what is learned in school, but to what children do outside or inside schools, but to how cultures relate to computers too. Also, the widespread availability of computers. And for example, in many East Asian countries, there is wide access to computers, and children and teenagers use them extensively. So, apparently, it is this use at home, in schools, that makes them more fluent for doing better searches in the internet and being more critical about the information they read.
Will Brehm 5:32
Why do you think PISA is trying to develop these metrics on digital literacies?
Inés Dussel 5:41
Digital literacies seem to be a significant part of what is called the 21st-century skills. It’s very obvious to say that we now live in a digital culture. Digital media is everywhere. Our lives, one could say, are digitally mediated in many, many aspects of banking, consumption, health, of course, entertainment and culture. And so, these skills seem to be very crucial for our well-being and also for our social and economic participation in contemporary society. So, I think everybody’s concerned about what is going on with the skills? How do children and young people learn the skills? Is school any relevant for this learning? Can we do better? What are they learning on their own? So, I think, in this area of digital literacies, we have many, many questions. And PISA is trying to grasp some things, I would say not all but at least some things that are related to academic skills or things we could relate to academic performance.
Will Brehm 7:03
So, can you give an example of how PISA would test sort of these digital literacy skills for children?
Inés Dussel 7:12
Yeah. There are some examples. One could be they asked the students, I mean, these are tests that are taken online. So, they can see, for example, how well they do in order to obtain practical information from the internet. I don’t know. I’m trying to look for the information more precisely, but for example, how do they browse? How are their navigation skills? So, do they look around? Do they go more directly to what they want? If they’re given a task, how many clicks? How much time does it take it to get where they should get? But also, the quality of navigation steps is also important. So, there are, I would say, several measurements. However, and I think, of course, the people who do these tests are -I would say they’re very knowledgeable, and they’ve tried it extensively. However, many of these things could be, I would say, interpreted differently, because I don’t know, you browse. Are you efficient? In what terms? Is it time-related? Is it quantifiable? Do we really know what kids understood because of this browsing activity by how many clicks they made? So, I think we get, as probably with other tests, we get proxies of what they’re learning. But I am not sure that we really know what they are learning through this kind of quantifiable measurements. And I would say very general. In order to have massive data, you also tend to look at very general things. And these things say little. That would be my point about how children learn.
And I would relate this to the research I’ve done in schools. Because I think that if you look in- depth at some schools, and if you look in-depth at some children: What are they doing, how are they doing? Then you understand more things. For example, also a colleague of mine in Mexico, she has done very interesting research on in which sites do children trust. How do they search online? And she found that they generally don’t take many steps. They get tired very quickly. Their efficiency is related to going through well-known places, either Wikipedia, but if Wikipedia is not a consider as valuable or as trustable by their teachers, then Wikipedia is forbidden. Okay, then no. But then how should I orient myself? And for example, see, this colleague of mine posed them a problem. So, now you have two websites that have contradictory information about how the Egyptians mummified their deceased. One side says that they had to take away their heart. And the other one says that they didn’t. And so, who do you trust? Or which one do you trust? And what do you do with that? And so, she found that children had very loose ideas about which website to trust. For example, a blog is no good because I can produce a blog. So, that’s no good at all. Then “.com” is better because “.com” then is a fixed page. So, if it cannot be intervened, then that’s okay. And they didn’t know, for example, the academic sites. I mean, they knew in Mexico, UNAM, the National University. Oh, this might be good. But this knowledge did not come from the internet. This knowledge came from the families, from the teachers. So, what my colleague says is that we have to do much better. I mean, we don’t get to know that by how many clicks they performed. We have to look qualitatively who they trust, which are the steps they are doing, and why, what are they looking for? And I think the most important finding of her research was that children said, “You know, I really don’t care. If you hadn’t asked me, I really don’t care which website is right, because what difference does it make?” And it was not only related to the Egyptian mummies, one could say, yeah, who cares? I think the value of truth -and this is a philosophical, I would say, question, but also a political and ethical question- is in decline. And so, you have to be fast. It doesn’t matter whether you get it right, it doesn’t matter whether you are going in the right direction. And my point would be that there are some similarities between this idea of going fast and being efficient and some of the learning metrics that are not really concerned about what is going on and the depth of the challenges we face, and the ethical question we have about what are we learning?
Will Brehm 12:58
It’s interesting to hear that PISA uses a proxy measure of digital literacy of the speed of a search online by students and how many clicks they do. But like you said that it’s much more complex than that, and PISA might be using the wrong proxy measure for one of these 21st-century skills, but do you think it’s simply a matter of like could PISA come up with a better proxy measure or a better test item to use that could get us closer to this understanding digital literacy?
Inés Dussel 13:40
I think that at a massive scale, it is difficult to get through these other indicators that are much more complex. I also think there should be other conversations of the Global Learning Task Force that they’re producing. And people who understand that learning cannot be captured by metrics. I would say that all metrics are proxies, and I would say probably all measurements are proxies, not only the metrics. Because I have several colleagues who are very good psychologists, I can think of Emilia Ferreiro, for example, who trained with Jean Piaget and many others, newer generations say similar things that really, we are not sure about how children learn and what they learn. And maybe some of the things you see their effects in decades from now or years from now, or months from now. Performance in tests can be tricky because it has to do with many other things. I mean, the language the test is taken, the relationship you have with evaluation, with a school system. If you are shy and if you feel marginalized by the school system, it’s very likely that you will not perform to your best potential. So, I would say there are many things there are that, for me, make the test proxies. Do we have to have test? I would say yes, we have to have tests, but we have to be very careful about how we use that information, what effects they have, and also make very clear arguments about the limitations of the information we collect with these tests.
Will Brehm 15:32
So, like assuming that these sort of skills are universal?
Inés Dussel 15:40
I would say the skills are never universal. I would say they are cultural, and culture is not universal. And again, if one think of reading and I have some examples from my own research and from this comparative research done by close colleagues, in France, in Argentina, in Mexico. In France, for example, reading and writing is very central to the educational system. Young people, by the end of their secondary school, they are able to read novels or literature that is being submitted for national prizes. And the relationship of educational system towards consider high culture or literature is very close. In Argentina, that relationship is very distant, I would say. And perhaps Argentinian children are much better at arguing orally, they have very good oral skills. And these are nurtured, I would say, these are learned through many, many years of schooling in which children have to participate in group activities, they have to even contradict their teacher. And that’s very well regarded. I mean, if the students say, “Oh no, you are wrong. And I have this other opinion.” Whereas in Mexico, I would say these oral skills are not that developed. And I would say these oral skills are important as a 21st-century skill if you want to call them like that. Because there are many things that today go through our presentation, to our ability to persuade and communicate. So, oral skills should not be devalued. They’re not second to other skills, they’re important. And how do we measure them? So, again, now are these universal skills or are they related to cultures that value some kind of competency skills or language dispositions? Or do we think that they are universally processed or owned? And I don’t agree with the idea that skills are universal. And I would say we have to take a cultural approach to skills, understand these skills in cultures in the current debates that cultures have. I remember one of PISA’s question about ecology and pollution. And I was thinking, well, this question would be much easier to first understand and then to solve in a European country than in a Latin American country where the ecological discourse is not a public issue, or not as much, and it was not ten years ago. So, are we measuring what children know innately, or are we measuring what society’s value and how they think about problems, and how they think about how to solve problems? This has been widely discussed, the cultural biases of tests, but I think we have to make the case again because there are still people who think it is possible to get rid of this cultural bias. And I would say, no, it is not possible, then let’s work with that and make more humble claims about what we find.
Will Brehm 19:22
Do you remember the example of this ecology and pollution question?
Inés Dussel 19:27
Ah, I don’t exactly. It was a question about the Greek, the Athens Parthenon, and it was about the pollution that was eroding the statues. And they had to make a calculus variable thinking, well, for many countries, this is not a relevant question. And I don’t mean you can’t solve it. Yes, you can solve it at any rate. But no, it’s not the same. If this is something you hear all the time on the news, and you’re used to this kind of argument, and you are bringing in some other kind of knowledge. For example, I don’t know, does the air pollute, do carbon emissions pollute? I mean, these are things some people hear on the media all the time and on the newspapers and on the families and some other people don’t. So, we have to take that into account.
Will Brehm 20:24
So, with PISA privileging one particular perspective over others, would you see it as a form of like neo-colonialism? Like pushing one idea globally through these tests like PISA?
Inés Dussel 20:39
I think there is a risk of neo-colonialism. Yes. On the one hand, because there is an assumption of a problem-solving individual that is isolated and acts upon knowledge and information and choice. And I think that for many other children, this is not the normal life. I would say, for example, in Latin America, but also in India and Africa, many children who live in shanty towns, in marginalized populations. One thing that is very important is that they learn to go by. They learn to, for example, keep silent and feign ignorance because they know some things shouldn’t be told because they’re dangerous, there are risk, they might be illegal. This is a skill, a very important skill. Not necessarily one that I like, but how does it come into the school knowledge? I’m sure it is there, too. I’m sure it is there in relation to how they feel, for example, with invitation to speak up, to participate, to express themselves, if they know in their daily life that they shouldn’t, that it’s better for them, it’s safer for them not to express themselves. So, what are we measuring when we take them the same test, then we take to children who are raised in societies where this is the norm -speak up, express yourself, be creative, be autonomous? And I think there are lots of cultural biases that we have to take into account. So, yes, there is a risk of neocolonialism when we don’t understand how this norm that we think is universal is not universal, how it is being played in different contexts. And I think that the worst neocolonialism comes with the truth claims of PISA and many other tests that say, this is what it is, these are the best school systems, these are the worst school systems, and they don’t consider how they are performing in their own societies and to their own goals and problems.
Will Brehm 22:56
Do you think it would be possible to construct global learning metrics in a way that is culturally sensitive? It seems as if there are, on the one hand, global learning metrics need to be uniform, globally, to do this sort of comparisons. But at the same time, listening to all of the different examples you’ve given, it’s clear that a universal test question isn’t going to actually work in different contexts around the world. So, it poses a serious problem to global learning metrics that the very essence of them being universal. So, do you think it’s possible for global learning metrics to be culturally sensitive?
Inés Dussel 23:42
I hope it would be possible. I’m not sure it might be possible. One thing I would say is that we have to have more diverse agencies. And it’s not only a matter of representation, but people who work there are aware of all these issues that we are discussing. I would also say –
Will Brehm 24:07
So, you mean. I’m sorry. So, you mean like inside the OECD, we need people making these test questions that come from more countries than just where they currently come from. Is that what you’re saying?
Inés Dussel 24:19
That’s partly what I say. But I would say if people come from, for example, Latin America and share the same kind of understanding there, then that is not any good, I would say. Because it’s not a matter of the country of origin, but it’s can you really bring in different perspectives and discussions? Can you really bring in people who challenged some of these assumptions and who have done work on different kind of methodologies? And related to what we were saying about the comparison, I think the problem is that this would be much more expensive. But what if the reading test took into account cultural differences? I don’t think that’s impossible. Of course, that would make the comparison probably looser, more flexible. But yes, I can think that we can still make some claims about how children read in some countries and how children read in some others. They are not reading exactly the same question. They are, for example, or you could make subsets of countries with similar problems -in terms of violence or human rights or even ecological concerns- but that are not framed in the same language and way which it is framed in the Northern countries. So, I don’t know I’ll give it a try at least. I think we have to give it a try.
Will Brehm 25:51
So, you say that you want to see a shift from learning to schooling. What do you mean by that shift or that difference?
Inés Dussel 25:59
I think education cannot be reduced to learning. Also, because learning is very difficult to measure, really. I do believe that. So, many of the metrics I’ve seen are really not measuring learning. They are measuring what is being offered to children, the kind of access they have, the kind of exposure they have, the preparation of the teachers, but not necessarily what they are learning. And I think that’s because learning -even if you look at children’s notebooks, or if you make interviews with them individually, then it is very difficult to see what they have learned. And as I said before, maybe you know, and they know what they learned in ten years from now, and not now. So, I think that’s very complex. But also, education is not only about learning cognitive skills. There are many things we learn in schools that are important and not necessarily taught by the teacher. But we learn to live with others, we learn to listen, or I hope we learn to listen to others, we learn to be with ourselves, to be outside our homes, we learn that there are other people in society that think differently, we learn that we have to abide by certain rules, and these rules might be flexible, but not too much because that’s what societies do. And I think these other kinds of learnings are very important. School is a public space, and if you want, you can call it socialization, you can call it education in a broader sense. They provide us with other references, with other languages, tastes, and this is also a very important social function. And I would say today, in a world where the risk of isolationism is bigger, where there are many fundamentalisms threatening the possibility of a civilized or peaceful world, then learning to live together is a very important task for schools. And can we measure that? How do we measure that? I think we have to be careful about what we put in the test. And also, the focus of these. We make these the focus of schools while schools have many other functions that are very, very important.
Will Brehm 28:47
So, it’s like the global learning metrics, like PISA, have taken a very narrow focus of what learning is being just cognitive skills that can supposedly be measured. But you’re saying that schools actually have such a broad function for children in society -far beyond just cognitive skills.
Inés Dussel 29:08
Exactly. I think it’s very important that we keep these broader functions alive and as a focus of schools. And I think there are many studies I don’t know in the US, one, which caused much uproar was Diane Ravitch’s 2010 book on the decline of American schools and of the great American public school, and how all this test madness has made things really bad for teachers and children to work in schools, to be together, to decide for themselves what are their goals, what do they want to do, and again, learn some basic things about social life. And I think you can say the same thing. Maybe in Chile, sometimes in Mexico, but there has been very strong pressure on teachers to improve the performance in tests. And then they forget about other things. So, you just train to the test, and you forget that you are there to really educate a human being and to really contribute to a better life and a good society for all.
Will Brehm 30:27
Well, Inés Dussel, thank you so much for joining FreshEd.
Inés Dussel 30:31
Thank you for this conversation.
The cultural insensitivity of global learning metrics
