Supriya Baily
The meaning of “learning” in Global Learning Metrics
Next week the CIES Symposium will take place where scholars and practitioners from around the world will come together to discuss and debate the desirability and feasibility of global learning metrics. I’ve had the honor of interviewing many of the speakers who will attend the symposium.
And one things that has struck me during my conversations about global learning metrics is that often a universal meaning of education is assumed by the tests and those who use it.
For instance, a 2013 OECD report that used PISA data was entitled “What makes Schools Successful?”
Implied in the very title of that report is an assumption that there is a universal definition of success, as if all schools around the world agreed on what it means to be successful. Moreover, the report implies that it is PISA data itself that can reveal the answer.
Perhaps more clearly, a 2013 report by the Learning Metrics Task Force, which is a multi-stakeholder collaboration organized by UNESCO Institute of Statistics and the Center for Universal Education at Brookings, was entitled Toward Universal Learning. The very goal of the task force seems to be reaching universal learning.
But can there actually be one definition of learning and success? Is it possible, in other words, to have a universal notion of “good” education?
This question has bugged me for some time, so I’ve invited Supriya Baily back to the show to discuss this idea of a “good” education in relation to global learning metrics. She points out how tests such as PISA are often culturally unresponsive and do not enable teachers to thrive. Although Supriya is hopeful that Global learning metrics can be meaningful with some revision, she cautions against universalizing concepts of learning or success.
Supriya Baily is an Associate Professor at George Mason University and the Associate Director for the Center for International Education. She will present some of the ideas discussed today at the CIES Symposium next week.
Citation: Baily, Supriya, interview with Will Brehm, FreshEd, 48, podcast audio, October 31, 2016.https://freshedpodcast.com/supriyabaily_2/
Will Brehm 2:06
Supriya Baily, welcome back to FreshEd.
Supriya Baily 2:09
Thanks Will, for having me here again.
Will Brehm 2:13
So, global learning metrics, you know, the goal is to really measure outcomes of education. To try and create this idea of like, what is “good education”. What is good education?
Supriya Baily 2:27
So, that’s a great question to start with because how you define good depends often on who you are, or where you live, what your goals are for education as well as where you exist, sort of, on this policy development, national continuum. So, for instance, a person who works for the Ministry of Education, or the Department of Education might have very different definitions based on the mandate of their job. On the other hand, a person living in rural Kentucky or rural Indonesia, or in the heart of New Delhi could also have very varying definitions of what education is. So, for me, the question really becomes, what is the end purpose of education? Is it to get a job? Is it to have an understanding of the world we live in? Is it to be able to be actively an agent towards achieving or accomplishing your own personal goals? Or is it to move the national economy forward? Or to move a particular group of people who’ve been otherwise marginalized forward? So, in this way, sort of, the definition of what a good education is depends on who, what, when, where and why you’re asking the question.
Will Brehm 3:43
So, a test like PISA, does that take a particular understanding or definition of good education?
Supriya Baily 3:51
Yeah, I would think so. Because one of the things about the PISA studies is that -I mean, you look at the TIMSS study, the PISA study, and the PIRLS, which is the program for Progress in International Reading Literacy Study- all of those have a particular focus on what they’re trying to measure. And in trying to measure something very specific, they’re defining what they think is a good education.
Will Brehm 4:23
So, I mean, it seemed as if you mentioned all of these different definitions of good education and how different actors would have different definitions because of their own positionality. So, I mean, in a way, it’s probably impossible to have a universal definition of good education.
Supriya Baily 4:45
Yes, and I think the people who are working on the Global Learning Metrics Task Force, I think they’re trying to actually accomplish that. I think they’re trying to find a balance between what we think people should know, what they want to know, as well as, sort of the cultural or local or community constraints or beliefs that might inform what it means to have this global good education. But one of the things that has been coming back to me over the last couple of weeks in thinking about the symposium and the conversations we’re going to have around it, is that even if you put multiple stakeholders at the table to try to define this global good education, even if those people come from diverse backgrounds, or different countries or different contexts, their agenda around education is usually pretty homogenous in terms of their class structure, where they all tend to come from which lends itself to thinking about pretty similar experiences around education or similar expectations for education. So, I think it is difficult to find a global definition of good even though I think people are trying to somehow conceptualize that.
Will Brehm 6:06
So, what is that agenda that these multiple stakeholders, trying to understand what education is for, what sort of agenda is that that’s homogenous, that you were saying?
Supriya Baily 6:19
I think the homogeneity comes out of our expectations of what the outcome of education really is. So, for instance, I was using some work by Zvi Bekerman in my class the other night, and it was talking about the central focus around education being about the preservation of the nation-state machinery and this need to conceptualize national consciousness and ideology. But also, one of the things that he talks about in terms of this goal of global education is for social mobility. And so, no matter where people are in the world, most parents see social mobility and advancement of their children through schooling being the predominant focus of education. And so, you have sort of that very fundamental local level where people want to see their children get better jobs. On the other hand, you have sort of institutions like the World Bank, or ministries of education, who are saying, well, if you are able to get a better job, that helps our economy and so therefore, access to good education means being part of this neoliberal ideology, in some ways, of education for work rather than education for purposes of personal transformation or personal agency. So, I would say one of the constraints for learning metrics to be effective is in some ways, those learning metrics are defined by expectations of economic rewards.
Will Brehm 8:20
Right. And that would be like the focus on skills, particular basic skills that are fundamental to learning like numeracy and literacy.
Supriya Baily 8:30
Correct.
Will Brehm 8:31
But having test questions or metrics that look at interpersonal skills or non-cognitive skills, those may be as important in education but might be missed.
Supriya Baily 8:48
Absolutely. So, you know, I’ve been listening to the other podcasts. And I was listening to the first one that you did with Dr. Eric Hanushek. And I mean, he talked in his podcast about how difficult it is to measure literacy or social studies, for instance. But he said numeracy skills might be the easiest thing to measure. But one of the examples that he used in that was the currency exchange. And as much as I respect his work, I have to respectfully disagree that even that requires a certain contextual understanding that is not often the same across income groups, across diverse groups, ability groups and across countries and regions within countries. And so, the assumption that there is a fundamental similar foundation of knowledge that exists across the board is one that is easy to sort of aspire to but it’s not one that actually currently exists the way our world is currently set up.
Will Brehm 10:09
So, in a sense, some of these metrics or test questions simply miss cultural differences. Is that what you’re saying?
Supriya Baily 10:18
Yes and no. Not just cultural differences but life experiences, awareness of language. So, for instance, I sort of think about myself, and I think about my own identity. As somebody who’s South Asian descent, somebody who identifies myself as a feminist, somebody who’s been educated in a school where the medium of instruction was the dominant world language, right -English. And yet, there are particular words, or phrases, or assumptions of capacity of language, or experience that might cut across certain classes of people globally. So, it’s not just cultural differences that need to be taken into account. It’s life experiences, it’s class experiences, it’s language experiences, it’s ability experiences. All of these things have to be reflected in the way you assess, or you look at the end product of what a group of young students know or don’t know.
Will Brehm 11:33
And to take those various differences into account in order to form meaningful comparisons.
Supriya Baily 11:41
Correct. And again, the question becomes, you know, just as I asked early on is: who’s defining the purpose of education. You have to say, what’s the purpose of these measurements? What are they going to tell you? What are you hoping that they’re going to tell you about the learning of students that also takes into account their life experiences, or the inequitable situations that might be constraining their ability to actually do these tests in the most effective way possible?
Will Brehm 12:20
So, what does it mean to actually be culturally unresponsive?
Supriya Baily 12:24
So, I really had fun thinking about this a couple weeks ago, to be culturally unresponsive doesn’t necessarily mean that one has to know about all the holidays and all the festivals and you know, all the ways in which people greet one another and things like that. It really is more about the fact that you aren’t able to see what you don’t know, and you value what you do know better. And so, one of the ways that I think about this is that when you’re exposed to something that is foreign or different, or not what you’re comfortable with, you aren’t able to work with it. And so, one of the examples I was thinking of was, if you’ve read Charlie and the Chocolate Factory, there’s a character in that named Violet Beauregarde. She loves chewing gum. And she’s always chewing gum, and she’s in the factory, and she finds that there’s this new gum and they tell her “No, no. You can’t have this. It’s not completely finished yet”. But she’s one of those greedy kids. She obviously doesn’t get the factory at the end, but she gets the gum, and she starts to eat it. The first thing that she tastes is this tomato soup. And she describes it as warm and creamy. And she really enjoys it. And then the second course comes along in her mouth, and it’s the roast beef, and she describes it as tender and juicy. And these are things she knows and she loves and she’s enjoying, and she’s really relishing it. And then she gets to the dessert course, which is where the formula is not 100% set, and she has the blueberry pie, and it doesn’t work for her. And it is awful and she starts to blow up like a blueberry. And the thing is, that’s how I see culturally unresponsive. I see it as a place where when you know something and you’re familiar with it, you enjoy it, you relish it. But the moment you try something, or you’re exposed to something that is unique or different or foreign depending on how you define it becomes something that you don’t relish and you’re challenged by it. And in some ways. you kind of blow up like this great big, giant blueberry.
Will Brehm 13:01
So, do you think it’s actually possible to come up with metrics that are responsive to all of these different differences?
Supriya Baily 14:57
You know, I’ve been reading a lot about what the Global Learning Metric Task Force is trying to do. And I have to say that I really do appreciate the types of areas that they’re hoping to aspire to be able to measure. So, they have six domains that they’re trying to look into and trying to evaluate a little bit more carefully. Benchmarking certain pieces of knowledge versus other pieces of knowledge. But on the other hand, within each of these areas there’s a lot of debate in terms of how you define, for instance, early childhood experiences. So, one of the domains they’re talking about measuring is early childhood experiences. And, again, to think about what Bekerman is talking about in terms of an equitable educational system. Are we assuming that there are a couple of different experiences that all children have to have across the globe? And how do you measure that? So, I think what you’re asking me is, are we able to develop metrics that are culturally responsive? Sure. I think we can try to do that. But on the other hand, we have to decide why are we comparing cross nationally? I mean in India comparing itself to Finland you may be able to benchmark those numbers. You may be able to look at those metrics. But are they telling you about the kind of inequities that students in Finland might face versus the students in India?
Will Brehm 15:13
I guess, so like Eric Hanushek in his interview with me, his answer to that question about why would we compare like, what’s the value of these global learning metrics? And for him, it’s about allowing national systems of education to understand what is possible in their education systems by looking abroad. Rather than saying, this is the quote, unquote, best practice. It’s simply about showing possibility.
Supriya Baily 17:28
Showing possibility as -and so this comes to sort of what we’ve been talking about in the draft of what I sent you about the symposium and sort of this intersection of a neoliberal ideology and patriarchy and colonialism. And so, progress towards what? Progress across what? So, the reality is that we can agree, I think that -or maybe we can’t- but global learning metrics don’t in and of themselves do anything. They may tell you a snapshot in time of how a particular cohort of students fare on a particular test on a particular day. And for a national policy level, absolutely! I would be the first person to say that there is a common core of knowledge. And I say that very carefully in the United States context but a common core of content across the world that is important to transfer as best we can to students. Because if we start to say that curriculum changes based on who you are, where you live, or what your aspirations are, then we go back to the way things were 30, 50, 100 years ago, depending on where you live. But on the other hand, if we’re measuring things just to measure them but we’re not actually changing the underlying foundations of inequity, then 20 years from now, we’re still going to be thinking about what we can measure to ensure that we are finding those pieces that we have to add to the curriculum.
Will Brehm 19:21
Right. I mean, it seems like the goal for global learning metrics is to be culturally responsive in order to be meaningful but then with academics, the lot of the focus is on how do we make these test questions correct? The psychometricians. And how do we do comparison in a meaningful way? But then there’s the other side, like you were saying, about policymakers of how do you actually use the data that comes out of the Global Learning Metrics in a way that increases equity, increases achievement levels of students and outcomes of education broadly defined. And it’s that sort of dual purpose of global learning metrics. One side that’s very academic and trying to be better methodologically. And then one side that’s very practical that’s about trying to really improve education systems in the nation state.
Supriya Baily 20:21
So, there’s actually this really interesting study that I found that was done in sort of 2009, maybe by Klenowsky and Gertz, which talks about assessment and equity in relation to indigenous students in Australia. One of the things that we can see across the board, and there are quite a few studies that do this, is that they do find that minority or indigenous or marginalized students have faced a lot of difficulties in understanding or faring well in these assessments. And one of the things they were able to hone in on is that both the test language and the graphics that were used in some of the Australian national tests were the reasons that these students didn’t perform very well. And so, recognizing that there’s patterns of underachievement, especially in national and international metrics or benchmark data, assumes that it’s more about, are we culturally appropriate? So, are we going to, and I know this sounds incredibly shallow, but are we going to change a hamburger to falafel to a Kati roll? You know, it’s not just those cultural assumptions that we’re making when we read the test questions and say, Oh, this is too Western, or this is too American, or this is too international. And so, it’s more than that. It is more about our own experiences that we bring to the table. It’s about the understanding and the comprehension and the ways in which we’ve been schooled or not schooled to be able to pick up on the nuances of these tests. And I mean, you can see that in the United States, the way more and more universities are stepping away from the SATs, or the ACTs as being good predictors of success or accurate representations of student learning.
Will Brehm 22:43
And do you think -I mean, like that move away from focusing on the SAT, what I find interesting is that it seems, for a long time now, what is defined as quality education is usually measured by a student’s ability to do well on a test like the SAT or other metrics. Other tests that either nation-states or can be given cross nationally like PISA. And so, this ability to do well on a test is like, the dominant way to say if there’s quality or not in the education system where that student came from. And of course, that kind of creates all of these negative outcomes like teaching to the test, everything is about the test, these ranking systems and this competition that now seems pervasive in the education systems around the world. But I wonder if there’s now in some universities, this move away from the SAT, if that trend might be broader, and we might start rethinking what tests are for and what that would mean for global learning metrics.
Supriya Baily 24:00
So, absolutely. And so, you used the critical word over there, which is quality of education, right. And so, one of the things that I’m grappling with, especially when I talk with my students is, what’s the difference between education and knowledge. And in some ways, I would argue that our experiment with education has, in fact, sort of failed the individual’s ability to be an agent of their own transformation. And so, I was at an international school a couple of years ago, and I was talking to a leader in the international school’s world. And I was looking at the curriculum that they were offering and I just said, you know, it’s really kind of interesting that this kind of curriculum shifts and changes depending on the privilege and access of students. And the gentlemen said to me, “Well, you know, we’re preparing leaders, not followers”. And to me in some ways, when we talk to teachers, when we talk about quality of education, there’s a real difference between what is the knowledge you’re gaining through your exposure to ideas and information and dialogue and debate versus knowledge or the education that you get to be able to show progress through a system. And there’s a really interesting quote by a scholar named Emma Goldman about schools today being the place that’s for the child -and this is very, very anarchist in some ways- what the prison is to the convict and the barrack is to the soldier, a place where you’re really seeking out discipline and uniformity. And so, if you’re going to seek out quality education, I think the question has to be in some ways, is the school the right place to do this?
Will Brehm 25:58
Where else would it be done?
Supriya Baily 26:13
Well, I don’t know. And that’s a great question for us to think about. But schools are, in fact, created in the model of what -mass schooling is an economic structure. It’s one that’s emerged out of an understanding of education that evolved from maybe the enlightenment, the reformation period, Western philosophical sort of orientation to the way knowledge is transferred. And so, again, I’m not sort of saying we blow up schools, or blow up the idea of schools. But I think that we’re looking at metrics as being tied to learning. But then we’re also looking at schools as places where kids will talk about the fact that all they learn on a standardized test, especially in the US, is how to fill out a bubble sheet. And this is a real frustration for teachers who return frequently to their frustration around preparing for tests that don’t, in fact, move knowledge forward. It might move education forward.
Will Brehm 27:51
And with teachers, another impact is that they’re often evaluated now based on these test scores.
Supriya Baily 27:58
Absolutely. And not just evaluated. So, the recruitment of teachers, the hiring of teachers, the promotion of teachers, their ability to make decisions about their own classroom, their agency really is oftentimes constrained by their tests. And the more that you teach in a challenging environment, or a low resourced environment, the more tests you’re expected to measure up to. So, in some ways, it’s almost backwards: The students who have the most resources and the best teachers tend to be the ones who have the least negative impact from these kinds of tests. Whereas the students who are most struggling and might be most in need of engagement with the ideas of education are oftentimes the ones who are bombarded with tests to prove that the teacher is in fact able to get them to learn.
Will Brehm 29:08
I keep coming back to this idea of like, is it possible for global learning metrics to be culturally responsive. And I’m thinking of it now in terms of teachers. Is there a way to make global learning metrics valuable for teachers to improve their own teaching rather than hold them accountable in some system that we’ve seen now, so many empirical examples of where the intended consequences are completely opposite of what actually happens?
Supriya Baily 29:40
Absolutely. You know, part of my work is in teacher education, and I don’t think I’ve met a teacher who doesn’t want a fair assessment of their progress from year to year. But saying that, I think also when these metrics come in and these tests come in that they are expected to show significant progress around. They often struggle with -so teachers talk a lot about what James Zelle calls prior knowledge, right. So, you can’t teach things where there’s no neural networks that have already created a pathway. So, the pathway has to exist for you to build upon. And oftentimes, when these tests are presented to teachers to administer in their schools and in their classrooms, they have to be prepared that the students understand the language of the test or the intent of the language of the test. And so, they have to be prepared that they have provided enough knowledge not only in the area of what the kids are studying but also enough to build a foundation. So, your question is, can we do that? Can we provide culturally responsive tests for teachers to be able to successfully administer to their students? And I think you can. But to what purpose again?
Will Brehm 31:28
What do you think? What sort of purposes could we do that for?
Supriya Baily 31:33
I think at the national policy level with aggregate data this could be very helpful. Understanding it helps to make decisions around equity and resource allocation. If you can find that students are unaware of certain -if you can see trends in what students are struggling with, it helps to build curriculum at the national level and it helps to build resources. But at the local level, for it to be something that the individual teacher is often held accountable for. And I know we’re talking about global metrics, and a lot of these bigger tests. But the Global Learning Metric Task Force says that we have to start with the teachers. That we have to be able to build the assessment system from what teachers teach in the classroom, to what happens in the schools, to what happens at the macro levels. And so, if the questions themselves are places where teachers are struggling because they don’t have the resources to be able to teach comprehensively then they do really struggle.
Will Brehm 32:45
So, are you hopeful about the future of global learning metrics being able to be culturally responsive when it comes to how we define good education and what it means to have quality education, and in particular for the work of teachers?
Supriya Baily 33:06
I would be more hopeful if we could ensure that the global learning metrics didn’t exist in a vacuum. So, I’m all for accountability and a strong platform of learning dimensions that we’d like to see students have access to across the globe. We have a lot of information in the world. And we have a lot that people can tap into to enhance their own agency and their own transformation. But on the other hand, if these global learning metrics are going to sit out there as benchmarks that we’re aspiring to but we don’t provide resources to give training or build curriculum or have professional development, and help teachers also understand that there are structures of inequity that affect their students. That’s a place that I bring a lot of my work into is around teacher education within the context of what are equitable and inequitable structures that surround students. So, we’re seeing these metrics -I don’t want to look at them as if, they’re the Emperor with no clothes so to speak. They’re admirable and important but they’re not going to do anything on their own. And so, if teachers are able to be prepared for diverse classrooms and they’re able to advocate for equitable environments, I think then these influences that are really about structural injustice may dissipate. And we might actually be able to let students across the globe be more successful in showing what they truly know.
Will Brehm 35:13
Well, Supriya Baily thanks again for joining FreshEd.
Supriya Baily 35:17
Thank you very much for having me, Will. I really enjoyed this conversation and look forward to seeing you in Arizona.
Coming soon.
Coming soon.
