Tests are part and parcel of the schooling experience. If a child goes to school, then I’m sure he or she will, at some point in time, have to take a test. But the nature and purpose testing has changed and seen a rapid expansion in the past thirty years. Tests have become increasingly standardized and connected to high-stake outcomes. Moreover, standardized testing has become the main tool by which policymakers measure education quality.
Standardized tests are both a national and international issue. The rise of international assessments, such as the Programme for International Student Assessment and the Progress in International Reading Literacy Study, have created a world in which governments at all levels rely on standardized testing. For students, testing — and the preparation for testing — has become commonplace.
My guest today, Will Smith, calls the worldwide phenomenon of standardized assessment the “Global testing culture.” Will is a senior associate with RESULTS Educational Fund, where he is developing the Right to Education Index. He completed his PhD in Educational Theory and Policy and Comparative International Education at Pennsylvania State University and has worked both as a US public school teacher and a fellow at the Organization for Economic Co-Operation and Development.
In his new edited collection, entitled, The Global Testing Culture shaping education policy, perceptions, and practice, which will be published this year by Symposium, Will argues that the reinforcing nature of a global testing culture leads to an environment where testing becomes synonymous with accountability, which becomes synonymous with education quality.
Citation: Will, Smith, interview with Will Brehm, FreshEd, 10, podcast audio, January 11, 2016. https://freshedpodcast.com/willsmith/
Will Brehm: 2:57
Will Smith, Welcome to FreshEd.
Will Smith: 3:01
Thank you. Thanks for having me. And I also want to thank Brent Edwards for getting us connected.
Will Brehm: 3:06
You have a new edited volume coming out in 2016 on the global testing culture. Can you give us an idea of just how far testing reaches around the world?
Will Smith: 3:17
Yes. So testing, I mean, no matter what school system you go into, you’ll see testing and testing is becoming more prominent and really becoming ingrained within the schooling and society. So for instance, testing is becoming a part of everyday life for students and parents and teachers. I was just this last November in a couple of different countries in Africa. And November is the testing season for all the countries that I was in. So in South Africa, I was a little surprised and a little saddened that I’m driving down around Johannesburg and basically on the radio is 24 hours of straight test prep and it was not test prep designed to create problem-solving skills or creative thinking skills. It was very direct test prep as in, here’s the question, when you get this question, here is the proper answer. Do not answer it this way. Make sure you answer it this way. And it was just hour after hour of test prep on the radio, which isn’t entirely surprising. One of the chapters in this volume is done by David Balwanz, who’s a postdoc at the University of Johannesburg. And he talks about kind of the Matric here and all the importance of the Matric exam, which is the exam for leaving secondary school and how really the Matric exam dominates everything to the point where other ideas on what education could be or should beare no longer existent. So sadly, I think you’re also going to see in the post-2015 agenda with more emphasis placed on quality, you’re going to see testing becoming more and more important. And so that’s actually for a couple reasons. One, because we have tend to have a very narrow idea of what quality means in education. And second, because we really have this desire to find something that is easily quantifiable and measurable. And in education that ends up being test scores.
Will Brehm: 5:20
The US Congress recently revised the decade-long policy of No Child Left Behind, which has been criticized for a temporary reliance on standardized testing. How does this new policy that they’ve enacted called the Every Student Succeeds Act fit into your notion of the global testing culture?
Will Smith: 5:44
Well, it’s actually what I consider a step back from the most intense idea of testing for accountability purposes. And so I’ve tried to understand national testing policies outside of this for my overall work on testing and accountability. And I’ve tried to identify punitive policies, which are policies that use test scores to really sanction educators and what I call evaluative policies, which are policies that use test scores to provide information to parents who then evaluate schools. And so what we’re seeing globally is initially a large push towards the end or extreme of accountability towards this punitive model, which was the No Child Left Behind model of low-test score schools are being sanctioned or closed down or the principal was being fired. Lots of different turnaround models and taking a step back towards more of an evaluative model, which is typified by kind of league tables or school report cards. And this actually has happened in a couple of different countries.
One of the interesting examples from the volume is the chapter on South Korea which is written by Pearl Chung and Hyeonwoo Chea who really talked about how South Korea is one of the top scoring countries in international assessments move towards a No Child Left Behind model just to get a lot of backlash, and then take a step back in 2013. So I think this is what is happening with the Every Student Succeeds Act. And when I look at the act, I’m not by any means an expert on it, it’s really new to come out, I’m just starting to get really engaged with it. But initial ideas are, I mean, I’m somewhat optimistic, I think there are a couple of things that they’re doing that are better, first, they’re not just linking everything so closely to test scores. So they’re encouraging the incorporation of graduation rates when states are setting up the accountability policies, they’re including and encouraging other more of what we consider subjective measures, whether that’s teacher observations or reviews. Additionally, I think they’re pushing to get rid of unnecessary tests. They didn’t really describe what unnecessary testing is, but they are looking to put a cap on the amount of time students are testing.
And I think the other good piece is that there is this emphasis and we’re seeing this all over the place, partially because of the failure of No Child Left Behind, is a push towards a brighter, a broader understanding of problem-solving, and creative thinking. And I think that emphasis will help reduce some of the teaching to the test that we see as these unintended, basically undesirable consequences. Because problem-solving is incorporated in all subjects, not just math, not just reading, and so we’re going to see less, hopefully narrowing the curriculum, less teaching to the test and more real deep understanding. However, there’s still a lot of concerns with this. I talked about, I’m not sure what an unnecessary test is, especially considering that the Act still maintains No Child Left Behind’s mandated federally-mandated test. So students in the US are required to take math and reading tests from grades three through eight, as well as one more time in high school, they’re also required to take science test three times from grades three through 12. And at the same time, the federal government is saying, oh, we need to remove unnecessary tests, they’re also maintaining these exams.
At the same time, one of the reasons No Child Left Behind came in was this idea for equity and education, we need to make sure the outcomes or the opportunities at least, are equal for all groups. So No Child Left Behind one of the, I guess, one of the best parts of it was this idea that we need to disaggregate data between different student groups, whether it’s based on disability, whether it’s based on ethnicity, whether it’s based on social-economic status, to really make sure that we are providing everyone with a great opportunity in education. And the most recent Act continues with that. But it also continues with aggregating the state at the school level, which doesn’t really change a lot of the motivations that educators and others have for really gaming the system.
Will Brehm: 10:14
So it seems like the desire for easy metrics is still there, while the narrow definition of quality may be broadening?
Will Smith: 10:23
Yeah, I would agree with that. I think the best part about this is an understanding that quality is really challenging to measure. And that it incorporates a lot more than just whatever you scored on your most recent math test.
Will Brehm: 10:42
You say that the global testing culture is comprised of high stakes testing and accountability. Can you give us a simple definition of first, high stakes testing?
Will Smith: 10:55
Yeah, I think basically, a test is a high stakes test if it has either real or perceived consequences for education actors. And this is taken in part from work done by Chapman and Snyder earlier, but it doesn’t have to be real, if it’s felt by students, or teachers or principals, then it’s considered high stakes. And I think the way you can figure out whether something is high stakes compared to relatively lower stakes is by, do the stakes attached to testing change their behavior? And oftentimes, there’s a change in their behavior without any real gains of learning, but just to make sure that they are scoring higher on the test. So if that’s the case, then I think we have tests that have too high stakes.
Will Brehm: 11:42
So that would be like the non stop radio shows in South Africa?
Will Smith: 11:49
Correct. I mean, they’re doing that. There’s no real learning to that. That’s just rote memorization and in a lot of the developing world you see that. So another interesting chapter in the volume by Anthony Somerset talked about how those types of really simplistic test and the pressure put on these more simplistic tests that are very common throughout the world really dictates the type of pedagogy that goes on in the classroom. And so he was exploring a lot of pedagogical techniques in the developing world and seeing that really, there’s a lot of simple rote memorization going on here. No real learning happening, just to make sure that we have the correct answer when it comes to that part of the test.
Will Brehm: 12:29
And what about accountability? How would you define accountability in simple terms as it relates to testing?
Will Smith: 12:37
I think accountability is the idea of holding someone responsible for something. And so testing is increasingly used as a form of accountability. And so it’s a way to, if we believe that teachers and educators and principals are responsible for student learning. And I guess in this sense, solely responsible for student learning, then testing is seen as this objective measure that is going to be able to help us understand whether or not they succeeded in their tasks. So if they’re responsible, then how do we measure whether or not they’ve been successful? And testing is often used for that. So an example of high stakes testing then, which is actually high stakes based on, towards the students, you can have high stakes towards the teacher, towards the principal, towards the school, towards the students.
And one of the examples once again, in this recent trip to Africa that I really learned a lot more about was some of the high stakes that are happening in Uganda. And I’m sure there are other examples out there as well. But Uganda has a primary school leaving exam, it’s not uncommon for, especially a lot of the developing countries that only have really made compulsory through primary education, a leaving exam. And so they have a primary school leaving exam. And at the end of that leaving exam, if you are a student, and you don’t score high enough on that primary school leaving exam, your schooling is over. So there is no opportunities in secondary education, no public school will take you and you are not eligible to sit for the secondary leaving exam. And so, that high stakes which decided how a student advances or if they advanced at all is really troublesome especially when we’re in a post-2015 world where we’re trying to expand access in a secondary education. I’ve got real concerns about whether or not some of these high stakes exams or entrance exams are going to leave some of the poorest students behind.
Will Brehm: 14:50
Can you quickly just explain what you mean by post-2015 world where we’re trying to expand access?
Will Smith: 14:56
Sure, sure. So from 2000-2015, we had the Millennium Development Goals and the Millennium Development Goals were all about focusing on primary school enrollment in the developing world. And so all we’re talking about really is trying to get more students into primary education and to varying degrees that was successful, we had a lot more students into primary education wasn’t fully successful. And we still have a lot of countries that are struggling to do that, as well as certain groups like students with disabilities and low income students really struggling to get into primary school. However, in 2015, the United Nations and really the world had adopted what was understood as Sustainable Development Goals. And Goal 4 of the Sustainable Development Goals focused on education. And it’s focused on providing basically free compulsory education, primary and secondary. And it’s looking at developing and the developed world. And so as we’re trying to expand into secondary, we’re going to have some issues with this. At the same time, there’s an emphasis on quality. And so as I talked about before, this is an interesting piece where, for instance, Uganda is going to have a hard time meeting quality requirements. If they lower, kind of, the passing requirements for their primary leaving exam, people are going to push back on quality and say oh we’re reducing quality but if they don’t do that, then they’re going to be limiting access to secondary education. So it’s going to be a really challenging post-2015 world.
Will Brehm: 16:30
You say that the two foundations for the global testing culture are positivism and individualism. Can you take both of those terms and perhaps just clearly, simply define them for the audience?
Will Smith: 16:47
I hope so. So to me, positivism just says that we can observe something and measure it. And so if we believe education, we should be able to understand what education quality is. We should be able to see education quality, high quality education. And we should be able to measure those that are higher quality schools, those that are medium quality schools, those that are lower quality schools. It’s something tangible, it’s something that can be objective. Individualism is basically a belief that we are all individuals, we all have freedom to act, and we all act in our own self interest. And so those are the two foundations, I think, for the global testing culture for the individuals acting in their own self interest in education. This means that parents are looking out for the interest of their children, which is a great thing. And so and for the global testing culture, this means that parents are trying to evaluate schools that they see as high quality and they’re doing this supposedly through looking at test results of those schools. And then they’re acting in the best interest of their child to make sure that they’ve chosen those high performing schools to send their child to. For students, this means that they’re properly prepared and motivated and looking out for their best interest. So they’re basically giving out a full honest effort on these tests. And for teachers and educators, this means that they’re looking out for their self interest. And sometimes, when the policy is more punitive in nature, this might mean that they’re teaching to the test because their self interest says I need to keep my job.
Will Brehm: 18:34
And would private tutoring fit into the self interest by preparing students more for these high-stakes tests?
Will Smith: 18:42
Yeah, I think there are a lot of advantages and self interest. So when you look at private tutoring, or shadow education, you’re going to see, you see an expansion of that. Generally, sometimes it’s focused more on higher-end or higher-class individuals, sometimes it’s focused on more remedial tutoring. But we’re seeing global expansion and out-of-school tutoring. And generally, just like we’re talking about with good information, and who’s basically acting on test scores orhigh-quality information, you’re seeing differences in social class. And so people that are having the ability to access this out-of-school tutoring are really the middle and upper classes, individuals that have access and understand the information that’s coming out, other than these very simplistic measures of this school got an A, this school got a B, are the upper middle classes. So I think what we see a lot is kind of a widening of inequity between upper middle class or upper middle and lower classes here.
Will Brehm: 19:51
Would that also have to do with the ability for people to have full knowledge when making decisions?
Will Smith: 19:57
I think it’s actually one of the flawed assumptions. I mean, when I’m talking about these two foundational assumptions, they’re assumptions. And I think a lot of people think, you know, self-interested actor has the ability and the information needed to act in their own self interest, in their best interest. But I think oftentimes, information is inadequate. And I think that, so, like I said previously, I think individuals that have the ability to access and understand the information that’s coming out tend to be in the middle and upper classes. But I also think that the idea that the information necessary is actually test scores is misleading. There’s a lot of research out there that shows that depending on the environment, depending on cultural preferences, that sometimes, what good information looks like is how far is the school or how safe is the school compared to other schools. So it has nothing to do with test scores. And a quality school might be a very safe place that I’m sure that my son or daughter can be in and learn something.
Will Brehm: 21:09
And can you problematize this notion of positivism in the sense that if we can see it, we can measure it? Is that actually possible in the world of education?
Will Smith: 21:20
I don’t entirely know, I don’t think so. So the idea that we can do it in a simple test measure is problematic. And I think, I think false. There’s obviously been attempts to do better testing to have, we talked about before going to more problem-solving based testing, it is always to refine the test. And I think in order to understand what really goes on, schools happen within a larger environment, you need to understand what the social and economic situation is. The environment, what other pressures are being put on kids? Did they eat that morning? There’s so much that goes into how schooling goes. And so the idea that we can objectively measure it in a single metric is really misleading.
Will Brehm: 22:14
Are there any examples that you found while looking at testing systems around the world that do not fit into your notion of the global testing culture?
Will Smith: 22:26
Well, I think there are. But it’s also important to understand that even if they’re not doing, you know, both the standardized testing and the high stakes, that even if you’re just doing national assessments, there does seem to be some type of reinforcing mechanism. So you’re still reinforcing the idea that test scores measure something objectively, or that oftentimes that math, science, this what I’ve considered academic knowledge, and what others have talked about is academic knowledge is what’s important to measure. And so even the high stakes may not be there, they’re still kind of reinforcing this overall global testing culture. However, I’ve done some looking across these. So I’ve looked at the 2009 participants and the OECD PISA, which is a large international assessment, there’re 69 participants in 2009. And out of those 69 participants, which are countries or I guess regions of countries, there were six that I didn’t, that I identified, it’s not having a national testing policy. And this is not having a national testing policy at basically primary or lower secondary or for America, that’s elementary and middle school. And so those six examples obviously are not lining up with a global testing culture. Some of those include Panama and Switzerland. Also, in the volume I think Hilla Auren and Devin Joshi give an interesting example of Finland, which gets a lot of publicity partially because of how well they do on assessments, as a supportive testing culture. And they contrast that to the global testing culture, which is more punitive in the way I presented it. And so some interesting comparisons there are Finland really using a sample-based assessment. So they do have assessments. But instead of requiring all students to take the assessment, so doing a census based assessment, they’re collecting a sample and then they’re aggregating their information at the national level and comparing, doing the same disaggregation we’re interested in, comparing by socio-economic status by region, so forth, but not comparing by school. And a large part of that is because they have sample-based stuff. And because they’re not comparing by school and making that information publicly available, a lot of the pressure that is put on schools through kind of, this market-based accountability or through sanctions is not there. And so those things, in addition to how they’re using test scores to support teachers, creates a very different culture that has laid out in the global testing culture.
Will Brehm: 25:12
You use the world culture theory to explain the global testing culture. Can you tell me what the world culture theory is and why you chose it over other theories to explain this phenomenon?
Will Smith: 25:29
So world culture theory is one of a strand that varies under the larger umbrella of neo-institutionalism. And basically, it’s very descriptive in nature. And it focuses on similarities that we are able to observe across very diverse environments. And so it was chosen really, because it emphasizes how the culture acts, and not how individuals act within the culture. And so once we identify the culture and how the culture is similar across different environments, then there’s this idea that individuals within the culture actually internalize those things. And certain practices or norms develop that are considered more legitimate than others. So one of the other reasons I chose the world culture theory is because I looked at the world culture theory, and a lot of great work has been done by John Meyer and Francisco Ramirez and David Baker, Alex Wiseman, and others, that I’ve identified key components and what they identify as the world culture. And I looked at that and realized that a lot of those components actually support a testing culture. And so the ideas of what they consider Western education, what I’ve talked about as human rights based education, where every child basically could grow up to be a mathematician, every child has a right to high quality education. Well, that’s reinforcing a testing culture, it says, oh, well, we need to make sure that happens. And the way to do that is to make sure that we have tests that are equivalent tasks that we can make sure that these individuals are learning. And so that’s somewhat, I guess, it is somewhat surprising to individual critics that a human rights approach might actually be supporting testing culture. But that’s the reality as I’m seeing it. And so that’s why I chose world culture theory.
However, world culture theory has a lot of critics out there. And most of the critics around world culture theory are criticizing the fact that it focuses on culture, and not individual actors. And that it doesn’t necessarily focus on the consequences of the culture. Because as we know, each culture decides what norms and appropriate values are, and then individuals basically don’t align with those norms and appropriate values, our game is more deviant. And so, world culture theory is very descriptive in nature and does a good job with that, but it’s not designed to talk about consequences. And so what I hope to do in this volume is really describe the culture using world culture theory as a framework, and then the rest of the contributions are really getting to who’s acting in this culture? What are our motivations for acting in this culture? And what are the consequences of this culture?
Will Brehm: 28:20
So, can you give us a few examples of the actors that are working within this culture and some of the outcomes as some of the authors in the edited volume find?
Will Smith: 28:31
Yeah, I think. So I think one of the interesting actors, this is actually Brent Edwards’ chapter on EDUCO in El Salvador. He spends a lot of time talking about the World Bank and how the World Bank uses this program in El Salvador as an example, and kind of the outcomes that came from it as an example, of why local accountability that puts local pressure on teachers and links them to student test outcomes is the correct accountability approach. And so the World Bank has done this, Brent did an excellent job of talking about how this is really done with some very questionable research. But these few studies done on EDUCO became so large and the World Bank is seen as such a legitimate player that it was taken up by organizations in the UN. It was taken up by even Save the Children and really praised as kind of the model approach for accountability. And so that’s part of it as the things that we talked about national governments. And so the chapter by Rie Kijima and Jane Leer really uses an interesting framework, compensatory legitimization framework to talk about how the Chilean government uses their participation and international assessments to show to their citizens that they are an effective and legitimate government. And so this is much different than a world culture perspective where the government is participating in international assessments because it’s deemed the correct thing for the government to do. This is saying the government is participating because of some type of local pressure and a consensus among citizens that quality effective governance participates in something like this, to participate in international assessments and is able to run an effective assessment program. And those are a couple examples of actors at the national and international level.
Will Brehm: 30:49
So the chapters in the book don’t all use world culture theory specifically, they use other theories that you then put under the umbrella in your introduction under the world culture theory is that?
Will Smith: 31:05
That is correct. I mean, most of them are using very different perspectives. So Balwanz’s chapter uses a critical international political economy perspective. Another very interesting approach is that done by the chapter from Kristine Kousholt and Bjorn Hamre who use Foucault’s approach on the dispositives. So they talked about the dispositives of discipline, security and optimization, and dispositive is basically a different lens to me to look at how testing works, and to actually problematize what’s happening. So depending on the lens that you use, different parts of testing are highlighted and really problematized. So most of the other things, they recognize world culture theory, a lot of the chapters do that. But then they move on to looking at something through a much different lens, which I think really adds to the richness of the volume.
Will Brehm: 32:00
I’d like to shift gears now and talk about you quickly. You have obviously been a student who took some tests, you also have been a teacher in different parts of America, and you now are a researcher looking at testing, have your opinions changed about testing in your different roles?
Will Smith: 32:24
I think as I’ve looked more in depth, and seen, kind of, the breadth of testing, not just in the US, but around the world, I become much more critical of it. And so as a student, you kind of do what you do. And I was, I guess, considered a good student. So I, you know, tried my hardest on the test, and was excited when I knew how to get good scores. And then as a teacher, that’s when I started to become really critical. But reflecting on my teaching is where what kind of really pushed the global testing culture and my dissertation around national testing policies. And I worked at the OECD around testing.And so in looking back at it, I was surprised. So I guess to frame this a little bit, I taught for six years in the middle of No Child Left Behind. So I taught from 2003-2009 and doing secondary social studies. I was looking back at it, I was surprised that we would create a policy that really encourages teachers to forget about whole sections of their students.
So an example of this, I’m doing professional development and it’s the Fall, we’ve just gotten back our test scores. So we have a new group of ninth graders, let’s say we’ve gotten their test from eighth grade. And we’re saying, here’s your students, here’s what they’re doing. We know that the way No Child Left Behind was set up, you needed to meet a proficiency mark. And so at the end of ninth grade, they were going to have to score, let’s say, 70% on a reading test. And we would look at basically who’s scoring around 70%, this is called teaching around the bubble or hugging the bubble. And so, who scored around 70%, we would say, Okay, if you’re scoring above 85% last year, we’re not going to worry about you because you’re going to score at least 70% on your own. And if you’re scoring under 50%, you’ve got so far to go that we can’t put all of our interest in you. And so we’re going to focus on these bubble kids that have a chance to score 70% or might slip below that 70% threshold. And looking back at it now, I never understood a policy that would actually encourage professional development, encourage teachers and educators to really forget about students. So especially, those that need the most support at the bottom.
Will Brehm: 34:59
Well, Will Smith, thank you very much for joining FreshEd today.
Will Smith: 35:03
So thank you. It’s been a pleasure, William.
Coming soon
Coming soon
