BLOG POST

CGD Podcast: Local Data for International Impact with Rakesh Rajani and Halsey Rogers

Development policy should use rigorous, representative evidence to improve people's lives. But where that evidence comes from, how it's collected, who is represented, and how it's shared are all key questions that have at times become contentious, generating important conversations in the field about the decolonization of research and policymaking in low- and middle-income countries (LMICs). 

The World Bank’s World Development Reports are the institution’s flagship reports and they typically focus on data shaped and produced internally within the institution. But in 2018, the World Bank's World Development Report on education made a striking departure from this norm in centering on data shaped and collected by two LMIC-based nonprofit organizations, India’s Pratham and Twaweza, an education-focused nonprofit in East Africa, through its Uwezo learning assessments. The 2018 World Development Report and, by extension, these data have gone on to influence global discourse, research, and policy in a way that few research reports have. 

For this episode of the CGD Podcast, I invited Rakesh Rajani, Tanzanian civil society leader and founder of Twaweza, and Halsey Rogers, a lead economist with the World Bank's Education Global Practice and a co-director of that 2018 World Development Report, to join me in discussing the importance and impact of centering local narratives and data for the quality and validity of development policymaking.

First, Rakesh reflects on the origins of Twaweza and Uwezo and how they became influential. Then, Halsey shares some of the Bank's motivations and considerations in anchoring the 2018 report in locally produced data. Together, I hope these conversations shed light on how partnerships between international actors and local institutions can have outsize impact on policy worldwide.

Resources 

EESHANI KANDPAL:
Hello and welcome to the CGD podcast. My name is Eeshani Kandpal, I'm a senior fellow here at CGD. At CGD, we spend a lot of time thinking about how to translate research into policy impact. At the end of the day, the goal of development economics is to make policy decisions that have been informed by the best evidence that is out there. But who gets to decide what the evidence is that is being taken into account? What is the evidence that is informing our policy decisions is a question that has at times been quite contentious and has, in fact, led to a large and increasingly influential movement in development economics that is about the so-called decolonization of research and of policy-making. Today, we're gonna hear from two people who have worked in opposite ends of the research into policy process. First, we're gonna hear from Rakesh Rajani, who is a Tanzanian civil society leader. Rakesh has worked and had influence in a number of roles over the course of a really lustrous career.

In 2001, he founded HakiElimu which is a nonprofit based in Tanzania that advocates for young people through education. He then went on to found Twaweza which is going to be a focus of a lot of our conversation today. Twaweza, also based in Tanzania, is a nonprofit that advocates for better education service delivery in East Africa - in Tanzania and Kenya and Uganda. And it promotes access to information to citizen agencies all with that aim of improving education service delivery. A striking policy impact of Twaweza's work has been the uptake of its major data collection effort which is called Uwezo, which is a large-scale survey of basic literacy and numeracy in Kenya, Uganda, and Tanzania, which went on to anchor some of the main findings of the World Bank's 2018 World Development Report which was on education, which in turn has influenced the strategy of the World Bank's education global practice and how it does its operations in its client countries. After we hear from Rakesh about his work with Twaweza and about how Uwezo came to be this citizen-driven and citizen-led effort to improve education service delivery, we'll hear from Halsey Rogers, who is a lead economist with the World Bank's Education Global Practice in the Latin America region, and was one of the co-directors of that 2018 World Development Report, which is formally titled Learning to Realize Education's Promise.

He helped lead many of the World Bank's initiatives in education that advanced the vision laid out in that World Development Report, including the Global Learning Poverty Estimates and the Global Education Policy Dashboard. And all of this is seeded in the work done by Rakesh and Rukmini Banerjee and her nonprofit called Pratham which is based in India. So we'll hear from Rakesh about how the data came into being and how they became influential locally. Then we'll go to Halsey, who will tell us about why they chose to anchor their findings in these locally produced data and how intentional of a decision that was, and how that in turn, led to the influence of this report. So let's start the conversation with Rakesh. Rakesh, thank you so much for joining us today. It's a real pleasure to have you on.

RAKESH RAJANI:
Thanks for having me on.

EESHANI KANDPAL:
I wonder if you would agree that a thread that connects your work through your various positions over time has been that pivotal role that is played by data in improving the quality of governance and of policy-making.

RAKESH RAJANI:
I am a big fan of data because I think data can help illuminate in ways that can get us beyond our ideological or political preferences or differences. It can help people come together and focus the mind on what matters. Data can help reveal new things that we are not aware of often. If you want to get beyond our assumptions, beyond our priors, beyond our preferences, we have a democratic influence, especially where there are power imbalances such as between citizens and state or gender imbalances or imbalances having to do with class and wealth and caste, it can be a way of having a common conversation rather than talking past each other. In terms of policymaking and governance, I think it can be a very useful way also of focusing on outcomes, because a lot of the time conversations naturally get focused on activities rather than outcomes. And so it's for the longest time we thought that the big challenge in education, for example, was to get kids into school because so many were out of school and we just assumed that once kids get into school, they will learn.

Very difficult to think about outcomes that are less tangible and that are more long term. So we look for buildings and desks and books, and the more tangible elements. And data can very helpfully allow that whole polity to focus on outcomes, and in that sense, bring a healthy accountability as to what is our collective purpose, what are we trying to achieve, and what progress are we making towards that purpose?

EESHANI KANDPAL:
Tell me a bit more about that last example you gave - we focus on chairs and textbooks rather than what's happening in the classroom.

RAKESH RAJANI:
So I think one of the most fantastic things that has happened over the last two decades across much of the global South is truly the expansion of primary and now secondary schooling so that all boys and girls can go to school. This was a serious problem if you looked at data and even just lived experience 2030 years ago, there are millions and millions of children who are simply not, and in many contexts disproportionately so women and girls. So that progress to truly universalize schooling is amazing and something that should be celebrated. And what we began to see is that it was necessary, but not sufficient, because these children who are in school were not in effect learning. Now, we knew this anecdotally. We knew this through small-scale surveys, but it was very hard to convince policymakers who are basking in their achievement, understandably so, to be able to say, yes, we made this great progress in terms of getting kids into school but we need to look further in terms of what is actually happening.

So this is where these large citizen surveys, first pioneered by Pratham in India, what are called ASER, the Annual State of Education Report which was a brilliant exercise using a very simple, accessible democratic tool to measure not perceptions but actual learning levels. And you could do it at scale so you could disaggregate the data down to the district level, for example. And what it showed in ways that were truly revealing was that large numbers, in many cases, majorities of children in school were not learning, were not able to read and count at the second-grade level, even though they might be in fourth grade or fifth grade or sixth grade. So it undermine the entire learning exercise. It demotivated the children, but it also demotivated teachers because teachers knew that they were going through the motions but it wasn't working. And yet the system and policymakers were not able to confront this. So the work that ASER did in India which we then adapted and replicated in Tanzania, Kenya, and Uganda through something called Uwezo, showed that there are these huge gaps.

Initially, there was a lot of resistance, but relatively soon, over time, not only the policymakers but other elites, community leaders, ordinary parents, in some cases, the children themselves, and the whole society came together to, one, understand that schooling isn't the same as learning, and to realise that what really matters is not schooling inputs but learning outcomes. It has had a huge impact on policy and practice and framing in India, in Kenya, for example, and across the world among funders and major institutions like the World Bank, the World Bank report on education a few years ago and the idea that financial literacy and numeracy, FLN, is core to getting learning right, confidence right, jobs is now well established in ways that it really wasn't even 15 years ago. And this really is the impact of these citizen surveys. This was a, I think, a wonderful example of South-South partnership where we work directly with Pratham in India. I led a team of people from Kenya, Uganda, and Tanzania to visit India.

We went to Delhi, we went to Gujarat, and we went right to the ground, we saw what this was in practice, how did it work very concretely, how was it organized, we looked at the relationships between the authorities and the organization as they did it. You can imagine the logistical issues are very complex when you're doing a large-scale survey, try to understand them. And it was a powerful experience where after doing this two-week visit, we as the East African group sat down and asked ourselves, is this relevant for us? Will it help us in East Africa? And there was resounding yes. As a result, Uwezo set up. And we did it an even larger scale than in India, like in just those three countries, for a while at the beginning, we were serving 200,000 households every year. So second to the census, it was the largest survey of its kind anywhere in the world. And it was profound in terms of its impact in reframing the debate and understanding.

EESHANI KANDPAL:
So you talked about some initial resistance to later on the World Bank's 2018 World Development Report in education really centered around this idea, which is that schooling isn't learning. Tell us about where that initial resistance came from.

RAKESH RAJANI:
Look, I think I can understand that these are very political issues. So the progress that governments had made in education was something they were very proud of. In this narrative punctured that story. And now, if I'm being candid, if I could rewind the clock, I would have probably approached it differently because the approach we took with this data, and this is a more general lesson I've picked up as well, is that while we shared it in the public with the goal of creating debate and understanding, we had a bit of a tin ear to the emotional lives and the political economy of government as to what this data and making it public would have. In addition to creating great debate and illuminating this story, it also created a humiliation for people in government. It put them on the defensive. It made them feel unprepared. It caught them a little bit off guard. And had we taken the time to exercise some empathy and curiosity, we might have realized that, in fact, the very champions of the information we were sharing would be undermined by this debate happening in public.

The impact it had on government is that it put them in a defensive crouch, and the reformers inside government had less space to operate to make progressive changes because now the entire government was under public assault. So they had to almost declare their allegiance to the government and that meant they had to discredit the data even though they knew better. Now this is tricky, because on one hand, you don't want to suppress evidence. You don't want to sugarcoat it. On the other hand, I do think if your goal is to actually change policy in practice, that it's important to be mindful of the political economy of how change happens inside government, what sorts of incentives and motivations, and power structures operate inside government, and what will it take that you often need a coalition of people inside government and outside who are able to thread this careful path, makes some judgment while they navigate this path.

EESHANI KANDPAL:
One thing that it brings to mind is work, for example, by Martina Bjorkman Nyqvist, and Jakob Svensson on citizens scorecards and how effective they can be at improving quality of health service delivery. In some work I've done in Nigeria, for instance, we tried to replicate that approach with a lighter touch, having citizen accountability. And in a smaller pilot we often find around the world, these sorts of pilots can have impact. And yet it's when you try to scale up that you hit a bit of a bottleneck. And I wonder if what you've just described is, to some extent, that bottleneck, reach to this idea of data sometimes puncturing government narratives. It almost makes data and evidence seem all-powerful. And yet I wonder if that's the case, if data and evidence aren't just as susceptible to elite capture, why we view them as unimpeachable when we know so well that they're not.

RAKESH RAJANI:
Yeah, I think data is not as powerful on its own to actually drive change, but data is very powerful in terms of media headlines and causing as far as government is concerned, putting it in hot water. But in many, many places of the world, the governments are just very reluctant to share data or often will go and cook the data. So then data gets released but nobody actually trusts its credibility, which is in some ways worse. So that's the first point that I think whoever releases the data can get into trouble. So often they don't, and there will be often a lot of self-censorship on their part. Now, we should also know that by the time we've got to a situation where a government agency is reluctant to release data, it should point us to the fact that there is a deeper underlying problem, and that deeper underlying problem is the basic institutions of how the rules of the game, of how this work have broken down. So it's almost like one needs to... Instead of being fixated on is this report gonna be be released or not, there should be others who try to do the quiet behind-the-scenes work that tries to say, look, what are we trying to achieve here?

What is our common purpose and what does it take to get there? So there is a way in which I think the hard work that needs to be done is to create, as I say, working relationships across difference and across oppositions. You did remind me a little bit of the great BBC show that I grew up on Yes Minister which talks exactly about this tongue in cheek way, of course, but exactly about that, the interactions between the bureaucrats and the politicians. We also have to somehow appreciate the politicians because in that show we poke fun at them. But the real-life politicians that I know, and not all of them, but they are having to balance many things. They have to win votes. They have their constituencies they have to deal with. They have to do their job and raise the money for the next election. I know ministers in government who their phone is ringing off the hook because a member of the constituency says the water pipe is broken.

EESHANI KANDPAL:
Tell us a little bit about how starting Twaweza shape you and what you did next, where you went from there next.

RAKESH RAJANI:
I was lucky enough to go to university in the US. After I did my first two degrees, I went back home very much feeling a existential crisis back home in Tanzania around the HIV AIDS pandemic, there were communities where the prevalence rate was something like 40%. And at that time, there was no cure basically. When I went back, I worked on HIV and worked on children. And it became very clear that if we wanted to have impact at scale on these matters of children's rights and rights around HIV AIDS and wellbeing, we had to think institutionally rather than in projects. And the most important institution by far was education. There are schools everywhere that reach that mandate and so on. So my work then focused on education through an organisation called HakiElimu, Tanzania's leading education rights organisation which is still thriving today. And our focus there was very much looking at institutional governance issues, rather than just the technical aspects of how do you improve pedagogy or examinations or teacher training or what have you.

And doing that work again made it very clear that at core, education reform was not about education, it's about governance around the political economy of policy making, and particularly practice and implementation. Not only were we doing work in East Africa, we are also helping shape global narratives. If you look at that World Development Report on education we referenced a few minutes ago, its first reference is Twaweza and the Open Government Partnership, which was launched, developed, and designed with the Obama White House, was very much an idea that emanated from East Africa, and that we work together with leaders across the world and the White House to develop and shape it. So he is a global initiative that is not received from up above, but co-created together. And I resist and resent the idea that you have funders and you have implementing organizations 'cause if the idea of an implementing organization suggests you're not doing the thinking, you're just doing the implementing.

This was a way of showing that we can do the thinking and the implementation.

EESHANI KANDPAL:
Development economists and certainly a card-carrying member of that club. And we are guilty, absolutely of what you've described. We collect a lot of data. We fly into remote communities, we survey them. We use the data to write our papers, maybe use the data to inform policy making although it seems to me that sometimes we use the data to justify policy decisions that have already been made. But in any case, there is a lot of data that's collected and it never gets reported back to the community from which we're collecting the data. So it seems like an inherently extractive process to me and not a just one. So I don't wonder if this is a just process and what we can do along the lines of decolonizing development to maybe make it a more participatory and inclusive process.

RAKESH RAJANI:
I think the issue is what role does that evidence play and who gets to make decisions 'cause the core issue is not only figuring out what is the right thing to do, but how do you get a system to adopt and adapt it and own it, and then as it rolls it out, how to make that work. And a lot of the time, the researchers who push this or the funders behind that, don't have this other part which is crucial. So it's evidence plus contextual understanding plus deep-rooted, respectful, and trustful relationships. Plus being able to read the moment. You have a number of these plus, plus, plus that are absolutely crucial to success. But if those are missing, then even a good idea or a good piece of evidence doesn't go far. So what I would put is a slight twist to your question, and to say that I think what is really important is to figure out in the everyday practice of government rather than just when a study is done, how do you create mechanisms and platforms where citizens can get the information they need, when and how they want it in ways that are useful to them?

How does it get just built in? As a matter of course, that becomes the new norm of how systems is really more important. Dan Honig has done really great work on this, and his new book that just came out that I highly recommend called 'Mission Driven Bureaucrats', I think is worth reading. And his whole concept of relational accountability, I think, is also very important because at core, it just says, let's see each other as human and understand not only incentives, which of course economists love, but also motivations. We can have multiple motivations. We can be motivated in different directions. Motivation, I think is a more complex term because it integrates incentives and interests and desires, and hopes. So I think being able to understand if one of the things I wish I knew better 20 years ago, because I certainly didn't then, and I wish I was more intellectually curious and emotionally empathetic to understanding what it means to be a politician or to be a senior civil servant, and to treat them as human beings, even the ones who are doing things that I really did not like or disagreed with.

But if I saw that, it would just open up ways of engaging with that person and understanding that person, that would then create avenues for change and partnership in ways that I think I just did not do. So the last thing I would say is this sounds corny, but I think is really important, is around the quality of attention. In the very busy lives that we live with social media and 1,000 emails and travel and internet, like our lives are full of stuff. The casualty there is that the quality of attention. And so we are only fleetingly present to each other, even when we are in the same room or on a call, because there's lots of other things going on in your mind. And I think when we show up with people, including people who are different than us, with the attention, a focus, openness, and grace, that can be transformative because it can allow us to get to new understandings and imagine new possibilities of collaboration and change rather than just putting you in a box that you are, and you're putting me in the box that I am.

And so for me, the quality of attention is crucial to being able to do the deep work that is needed in development.

EESHANI KANDPAL:
I agree with you wholeheartedly that the greatest gift we have to give is our presence, is our attention. And it is true that there are many competing demands on those two things. And so to have that intentionality at work, to show up with openness, with attention, with humility, there might be a transformative experience for both sides of that conversation. Going back to something that you mentioned at the very beginning, which is this defensiveness that your first accidentally engendered among the bureaucrats with Uwezo, I wonder if we should be thinking about relational accountability between the funders and the implementers, and to think about how sometimes the funders have the incentives to do something a certain way. But the intrinsic motivation that you described is often with the implementers much more so than with the funders.

RAKESH RAJANI:
I think that's really important, and I've had a chance to reflect on it because I've been working in civil society and with government for 25 years and as a funder for nine years. So in some ways, I've occupied all sides of this. I empathize with funders because I think a lot of the time when they make decisions or insist on conditions that are not as helpful, they are coming at it from a position of anxiety. They're giving away money and they don't have certainty for impact, they are worried, they are anxious that will this make a difference and how do I somehow provide protections. And so a lot of these conditions and strings and short term and all the sorts of things that are really antithetical to deep change. So we have to work with them to allay their sense of anxiety. And that requires patience, that requires conversation, that often requires people, other funders who've been around the block and have had experience, maybe speaking to them or speaking to their bosses because anxiety is a very powerful emotion.

But as you say, where there is trust and experience, you can take risks that results are not predictable and certain and linear. So that being the case, what your best assurance comes from are the people that are being entrusted with this money. Are they thinking and engaging in the right way? Do they bring the right values? Do they bring the right focus? And do they bring the right mindset and practices and muscles to be able to, for example, pivot when things are not working, knowing which questions to ask when something happens, like there's a regime change which will change everything.

EESHANI KANDPAL:
As someone who has done so many things, played so many different roles at so many different levels, if you could wave a magic wand and implement any one policy anywhere in the world, what would it be, where and why?

RAKESH RAJANI:
I think a lot of the world is in trouble because we believe things strongly, and then we simply stick to them and are not open to considering that we might be wrong, or that there might be some truth or value on the other side. And it fosters groupthink, it deepens polarization, and it gets us to take joy in tribal information rather than the achievement of something bigger and powerful and public purpose. So that would be my magic wand that everybody like researchers, think tankers, ministers, civil servants, NGOs stop being fundamentalists about whatever they are in. In terms of very concretely, I'll go back to the example of education because I think it is so foundational to all the other progress we seek in the world which is I really do think that while we have come to appreciate the centrality of learning outcomes as opposed to inputs. In practice, too much of our money, too much of our infrastructure, too much of the attention, and the human mind is not going to focusing on the outcomes that are needed.

EESHANI KANDPAL:
Thank you so much for your time. This was a real pleasure.

RAKESH RAJANI:
Thank you, Eeshani, for making the time and giving me this opportunity.

EESHANI KANDPAL:
During Rakesh talk about how the Uwezo and Pratham data influenced the World Development Report, which in turn went on to shift the paradigm, in his words, as he said, in global education policy to really center learning as opposed to school infrastructure, makes me wonder what it is about Uwezo and Pratham that was so notable or so striking to the team that worked on that World Development Report. So now let's turn to the conversation with Halsey Rogers where we'll ask him about why the World Development Report of 2018 made the choice to center, to really start with locally produced data, as opposed to data produced by the World Bank, by the UN, which is the norm, really, for World Development Reports. And whether he thinks that that was part of the enduring impact of this particular report. Halsey, thank you so much for joining us today.

HALSEY RODGERS:
Oh, it's my great pleasure, Eeshani. Thank you for the invitation.

EESHANI KANDPAL:
Before we get into the story of how this WDR was different, tell us a little bit more about the global education policy landscape going into that report.

HALSEY RODGERS:
So where were we when we started on this report which we actually started working in 2016? Now, the narrative in global education policy I think was quite inconsistent. First, we had this internal inconsistency within these education related policy discussions. I mean, on the one hand, by 2016 we had really made progress in putting learning on the agenda. So, as we'll discuss, the whole thrust of the WDR 2018 is to explore the consequences of the idea that schooling is not learning. So at the World Bank, we had launched our ten-year education strategy called Learning for All back in 2011. USAID, DFID, the former FCDO, the UK aid agency, also had strategies that they came out with emphasizing learning at around the same time. So that was great. Some of the key players were focusing in on learning and realizing that just getting kids into school was not enough. We had more and more evidence that kids weren't learning. We also, I think, in a huge factory, we got learning embedded in the Sustainable Development Goals.

That was a big achievement. I think many participants contributed to that. And it's not just the global actors, it's very much also groups like Pratham in India, like Twaweza in East Africa that were highlighting the learning crisis really effectively in their countries in terms that people could understand. So that helped in making sure that we had goal in SDG 4 that was focused in on making sure all children learn. But too often they seem like a rhetorical commitment to this goal without enough following up on it, without people thinking through, why aren't children learning? This agreement learning mattered, but not enough discussion of the actual barriers, not enough discussion of what it would take to lower them, not enough discussion of effective strategies. So that was this internal inconsistency. So second, we had this external inconsistency between this professed goal and then the practice on the ground in school systems. At the center, you had a lack of measurement and low awareness of low learning, so it was an invisible problem.

You saw children in school. Nobody was measuring, especially in primary education, which is when kids get all the most important foundations reading, numeracy, whatever. Number two, when you look at what drives learning, all those things, those school-based factors were breaking down. So we know what's important. You need children who are in school prepared to learn. You need teachers who are also prepared, who are well trained, who are motivated. You need materials, infrastructure, but you need them to be used for learning. And then you need school management, effective school principals, school directors, etc, that can bring all this together. All of those were breaking down in ways that we went into in the report. And then finally, the broader system, the political system, wasn't sending the signal to all these participants at the school level that learning mattered. So that really is what we were seeing, and that's why we thought it was so crucial to write a report that would focus in on learning, on what the barriers were, and on how we can tackle them.

EESHANI KANDPAL:
Great. Thank you. I wanna take a step back and go back to this insight that schooling isn't learning. What strikes me about this is in some sense it seems obvious that really what we care about are learning outcomes and not whether there are desks in school and textbooks. Obviously, those things are important parts of the production function, but the outcome seems so clear and so easily measurable that it has always struck me, I guess, as a bit strange that it really only occurred to us in the early 2010s if I understood your timeline correctly.

HALSEY RODGERS:
Yeah, no. Great point. We knew that for a long time, but there were just so few assessments other than sometimes as part of randomized evaluation. You'd do an assessment. You notice even in the baseline, you had very low levels of learning in the typical school. That was shocking, but often it wasn't representative. And so we really needed a revolution in getting assessments into schools, assessing representative samples. Your second point is an important one, too, about the inputs, about looking at whether schools have desks, whether there are schools, etc. This is important, but for too long, this was really the measure that many donors, many governments would use to say, are we making progress? And they would count the number of schools built. They would count the number of schools with desks, although often even then, you wouldn't really have a random sample. You wouldn't have people looking at the condition in a typical school, especially the more remote schools, etc. That was just as there is a bit of a mystical belief that a number of years in school would translate to a certain amount of learning.

There's probably also a little bit of a mystical belief that if you build it, if you provide it, they will learn. And that's been a parallel recognition because nobody's really focusing on the bottom line - are children learning? Are the children in the remote areas more disadvantaged areas? So those learning. The tests they tend to have in mind are the school leaving exams, perhaps at the end of lower secondary. In some cases, the university entrance exams, these get, say, widely covered in the media. People have an idea in some systems about this, many systems. The problem is you're missing a lot of the children who have already dropped out of school because they haven't been able to learn anything, or you're really looking at the elite. In many cases, you're looking at the top of the distribution in places like India, for example, where obviously the top of the distribution does extremely well by global standards. But we know at the same time that the median child in particularly those at bottom, are really struggling.

They're not acquiring the basic learning they should be having.

EESHANI KANDPAL:
So Halsey, a couple of times now you've referred to the implicitly a little bit, the challenges of staffing remote public facilities. I think it bears a bit. In many low and middle-income countries, public services for education, workforces for education, for health are centrally recruited. And then teachers or health workers, bureaucrats of all sorts are assigned. And there it becomes very difficult sometimes to get particularly the best teachers or the best health workers to go to the most vulnerable or marginalized facilities which tend to be in the remote areas. My mother worked in one of the early batches of the Indian Civil Service, and I do remember hearing the stories in the mid-'60s of going on field visits on horseback so she had to learn how to ride a horse while wearing a sari, which can be challenging, staying overnight in tents. So none of this is to downplay how difficult it can be to staff those remote locations, it is really important to do that. One thing that struck me about even the way you open the 2018 WDR is that you rely on the surveys that you just mentioned and Uwezo.

When I look at the introduction of many other WDRs, they rely on internally produced data. And yet you made this decision to rely on locally shaped surveys like ASER and Uwezo. Tell us about how these surveys influence your thinking going into that landscape that you just described. In that process of shaping the narrative for the 2018 WDR, was this decision to rely on locally shaped data and an intentional departure from the norm for WDRs and why.

HALSEY RODGERS:
Yeah. Great question. Let me step back and just say a little bit about WDR and data. Really what we wanna do is use the best evidence possible on a problem. As I said, it's often an invisible problem and we wanted to make the problem visible. And we saw from past WDR that research or data, just establishing very basic facts or basic metrics, not fancy econometrics. It's really the most powerful for changing people's worldviews, and that's what WDR tries to do. It tries to change the way that the broad development community thinks about a problem. And some examples from past WDR, as I was thinking about this one, is the data in the 1993 WDR and huge disparities in health outcomes across countries like measured in terms of Dalys that famously had a huge influence on Bill and Melinda Gates as they were starting up their foundation and helped convince them that this was a place they should be putting a lot of their efforts. Or you look at the 1997 WDR, where they had survey-based indicators of bribes being paid for firms.

That was really the first time we had seen that. That was at a time when we were shifting the narrative in the bank, Jim Wolfensohn was shifting the narrative to put a lot more focus on corruption as a cancer for development. So we wanted to really get these basic facts that people would remember, because people are never gonna remember a whole report. So where there was good data, we want to use it. As I said before, part of the problem is just governments weren't measuring learning in basic education. So we couldn't really use government data. But we were aware of the great work that was being done through the ASER and Uwezo initiatives by Pratham and by Twaweza in India and Pakistan, East Africa. And so we thought it would be great to open the WDR with data from them. And these are very memorable stats because they're just so straightforward and easy to understand. And we presented this WDR in some 60 countries around the world, always open with that. And just for the listeners, I mean, if I can just read the first three sentences so they can get an idea.

And so what we said was schooling is not the same as learning. In Kenya, Tanzania and Uganda, when grade three students were asked recently to read a sentence such as the name of the dog is puppy, three-quarters did not understand what it said. In rural India, just under three-quarters of students in grade three could not solve a two-digit subtraction, such as 46 -17, and by grade five, half could still not do so. Compare this to the way we normally express learning data which is like achievement of a certain percentage of curricular expectations. Or even PISA scores, the OECD PISA assessments, which get a lot of attention. But if you tell them, here's the score, relative to the average of 500 across OECD countries, that's not gonna mean anything to people. But these examples from Uwezo, from ASER, are just so easily understandable. We didn't just use these citizen-led assessments to dramatize the problem. They also helped inspire the strategy we laid out for improving. We had a three-part strategy for promoting learning for all.

So first was assess learning to make it a serious goal so that it wouldn't be invisible anymore, make sure you're measuring. Second, act on evidence to make schools work for learning. And that's in areas like early childhood development, effective teacher training, effective use of EdTech. But then third, what we called align actors to make the whole system work for learning. And how do you do that? Well, a big part of it is by building coalitions for learning that go beyond just the education sector. And part of that, we argued, is that you have to have clear information, clear metrics that people can understand. Again, if you try to rally support society-wide by saying our PISA score is 372, let's try to get it to 395, that's not gonna rally support, people don't understand. But if you tell them facts like the ones I just said, that half of grade five students can't do a two-digit subtraction correctly, everybody understands that's a problem. And so just seeing the power of what Pratham, Twaweza, others had done, really influenced our thinking on how we can tackle the learning crisis.

EESHANI KANDPAL:
Perhaps one distinction is that with the PISA assessments, for instance, which are done in OECD countries, as you said, there is a well-functioning accountability system. There are government investments in the right place in the education system, although you might wonder if that's the case here in the United States and to what extent. But I think broadly, there just isn't the need to convey the data as broad a section of the population, perhaps, where the intervention is really focused on the policymaker. And policymakers will understand, even finance ministers can be helped to understand a PISA score. But really, if you want community support, it is important to be able to present the data in a way that is relevant and salient to that community. And maybe this is one way in which local shaping is important because you understand your audience better and therefore you understand how to package the results better.

HALSEY RODGERS:
No, that's absolutely true. And I know Pratham in India better. Rukmini Banerjee created Pratham. I've been just very inspired over the years watching what she does in both working with communities to get this data, but then going back into the communities to let them know what they've learned from them, and then also what can be done about it, and really working with them in participatory ways to figure out the most effective methods to try to tackle this learning crisis and meet the other needs that they've identified in the education sector.

EESHANI KANDPAL:
Halsey, going back to the 2018 WDR, Rakesh described it as a paradigm shifter in the sense of Thomas Kuhn's coinage and use of that phrase. He is right that the WDR centered the learning crisis narrative in both the World Bank's education operations also went beyond the World Bank, I think quite broadly in education policy and research to really center that learning crisis. Looking back, what are three factors that you think were key to what I would call this exceptional impact on policy?

HALSEY RODGERS:
So I think there are three main keys. So one is we had a focused storyline and the right team to tell that story. Second, we had an explosion of high-quality empirical research on the problem and on the solutions that we were able to draw on. And then third, we developed new metrics that would help motivate progress after the WDR. And we were able to track the institutional support for using those metrics. We didn't mince words, we said there is a learning crisis. And at the time, that was not an uncontroversial decision to use that term. Interesting, one of the impacts we've seen is a great rise in the use of the term learning crisis, both in the academic literature and the media, etc, since then, we've been able to document that. But we called it a learning crisis, we kept it focused on foundational learning, and then we said let's really delve deep into that problem rather than trying to cover everything. Now, the storyline was inspired substantially by the great insights from local research and experience that I mentioned before.

This research from often local researchers, in particular national contexts, made the problems that were faced by children in low and middle-income countries much more concrete and really forced the global community to pay attention to what was going on in the classrooms, in schools, not just engaging in high-level debates about how much of a budget should go to education. An example of that that I thought was thinking of is the probe report. So the public report on basic education in India. This is a report that came out in 1999 that influenced me very early when I was just starting work in this line of education research because it really drove home the actual conditions in schools, the cases where teachers were away for weeks at a time, and didn't show up, or the cases where politicians showed up only in election cycles or otherwise didn't seem to care about what was going on in schools, etc. So that local research is really crucial. So we had a focus storyline inspired and one that was very important for policy.

And then we had the right team to tell this story, including people with a great deal of experience already. I'll mention, just for example, your former CGD colleague David Evans, one of our great WDR team members but we had numerous others. All of them had worked in-depth with countries, many of them coming from the regional departments of the World Bank. So that's first, this focus storyline. Secondly, as I mentioned, there was this explosion of empirical research over the previous decade or decade and a half that we could draw on. And this is crucial because otherwise we'd just be waving our arms and saying, learning is a problem, what do we do about it? Which to much of the discussion had been some of this research is homegrown. Like, as I said, ASER, Uwezo, as you mentioned, are very powerful examples. And I think great credit to Rukmini Banerjee to Rakesh Rajani and their colleagues for all the wonderful work they've done over many years. I think the emphasis on political economy of reform.

Obviously, you can't do that without really good local researchers. There's no way coming in from the outside we can do good research on that. And then third, I mentioned new metrics developed within our WDR or as a response to WDR. What these did is they helped build support and continuity and sustainability within and outside the institution to translate what we were saying into a sustained focus on this, especially outside the education sector.

EESHANI KANDPAL:
Halsey, we end our podcast here at CGD with a couple of questions typically. One of them is you had a magic wand that you could wave to implement any policy anywhere in the world, what would it be and why?

HALSEY RODGERS:
The change I'd like to see that's not exactly a policy would be to have what Lant Pritchett is called a social movement for learning, where everybody recognizes how crucial it is for all children to learn that everything else in society hinges from that. So that's really a sea change I'd like to see. But in terms of specific policies, if you wanna get more specific, I think what I would wanna see is that all teachers in struggling, low-income, middle-income systems receive practical teacher training and support through a method that actually works, like teaching at the right level, like structured pedagogy. We have evidence now that these approaches are often orders of magnitude more cost-effective and more effective than what systems typically do, just to increase the chances that all students can start their education with a firm foundation for future learning life. In the real world that's not gonna do it by itself, but that would be a big step in the right direction.

EESHANI KANDPAL:
The other question we ask is, is there a memorable story along Uwezo related to the conversation we've had that you'd like to share?

HALSEY RODGERS:
I mean, one thing that inspired this is unrelated to the WDR itself, but in the wake of it, when we talk about the institutional take-up, actually this is fun. Personal memory is, I remember at the HD week in 2019, so when all of our human development staff came together for a week from all around the world which is a very inspiring event in general 'cause you meet these dynamic, amazing, enthusiastic, smart colleagues from all over the world. But we have a talent show as part of that. And what's great fun is that we did, and this shows how much this had been institutionalized, as a friend of mine and I, Tim Johnson, have a tradition of writing parody songs for these talent shows, and we did one. We took the opening number from Hamilton and rewrote it as Building Human Capital. And it was an amazing, actually, one of my favorite moments of the World Bank, because we were able to sing about all these key issues on human development, human capital in a way that we hoped would be both funny and inspiring.

And personally was exciting because one of the other singers was my then 17 year old daughter. It was wonderful to be up there with her. But what was great was that everybody knew what we were talking about, because this agenda was now so well understood so quickly. And part of that grew out of the WDR, the narrative we were using for that.

EESHANI KANDPAL:
I was in the audience for that, actually. I remember that performance well, it was fantastic. That's actually a great way to end this because perhaps the bar should be that research report messages should be translatable into music. They should be so straightforward, we should be able to sing about them. Great. Thank you so much, Halsey.

HALSEY RODGERS:
No, thank you so much, Eeshani, this was great.

EESHANI KANDPAL:
So I found this conversation really illuminating of this one example, but a really powerful example of how locally rooted data and research have the potential to make an outsize impact on policy. Both Rakesh and Halsey highlighted how important it is to anchor in local context. How important it is to frame results in a way that is comprehensible to the population, not just to policymakers. Rakesh also highlighted how important it is to approach the relationships with partners in the field, whether they're researchers or policymakers, with openness, with humility, and with the intention to build trust. Underlying the conversation that we have today is a lot of research on education and education systems, on improving learning outcomes for children, and on global education policy reform. If this is a topic of interest to you, you might wanna check out the work done by our amazing education program here at CGD. Halsey referred to one of our great former colleagues, Dave Evans, who is now at the Inter-American Development Bank but has worked extensively on education systems around the world.

Other colleagues whose work I wanted to highlight include Justin Sandifer, who has looked at improving schooling quality in a number of different settings. More recently, Biniam Bedasso and Justin Sandifer have actually looked at the extent to which the World Bank's operational work, its lending work, actually implements that vision laid out in the 2018 WDR. Biniam Bedasso in other work, has looked at whether school feeding can be an important component of improving children's learning outcomes. If children are hungry, they can't pay attention to what's being taught to them. And in another complementary set of interventions that Halsey touched upon today, Pamela Jakiela, who is a former CGD colleague now a professor at Williams College has looked at, with Dave Evans and other co-authors, the importance of early childhood development interventions. And last but not the least, our new president, Rachel Glennerster, is a great authority on education systems and has worked extensively on global policy reform.

So if education-related research is your thing, go check these out. I hope you enjoyed listening to this conversation and you found it illuminating to some extent as well. Until next time.

Disclaimer

CGD blog posts reflect the views of the authors, drawing on prior research and experience in their areas of expertise. CGD is a nonpartisan, independent organization and does not take institutional positions.