[00:00:58] David Evans: Good morning, and welcome to today's event, our last Center for Global Development event during the World Bank IMF Annual Meetings Week. Today you're at the Big Returns to Early Investments in Foundational Literacy and Numeracy. My name is David Evans and I'm the Director of the Global Education and Child Wellbeing Program here at the Center for Global Development. If there's one thing that seems hard to argue with, it's the value of making sure that kids can read and do arithmetic. And so, let me give just a little bit of background for why this is so important right now. So for decades, governments and partners have worked on getting kids into school. And they've mostly succeeded at this. More than nine in 10 kids globally finished primary school, more than two thirds finished lower secondary school. There're still ways to go on these, but there's also a lot to celebrate. But at some point, countries and partners started realizing the kids weren't learning as much as they should be or could be. So, the most recent ASER survey from 2024 for children in rural India showed that only one in four kids in third grade could read and comprehend a second-grade text. That same number was one in 10 children in Uganda last year from the Uwezo survey. So as these kinds of results came to light in many countries, the international community started changing its rhetoric towards a focus on learning, particularly on helping children achieve mastery of basic skills like reading and arithmetic in the first two to three years of school. But even with that change, minds and the budgets that they control don't seem to be fully convinced. So, education policymakers in low- and middle-income countries consistently overestimate how well the kids in their countries are able to read and do basic arithmetic. Ministers of Finance often see education as a sinkhole for resources without clearly quantified results. Despite the shifts in rhetoric, a recent analysis of multilateral development bank projects showed no increase in projects targeting learning or quality. But that same analysis showed an increase in early childhood development projects, and this is probably because we have great evidence of the high benefits of these programs later in life. So, if these long-term benefits are so important to changing minds and budgets, what are the returns to investing in children and learning and mastering these fundamental skills of literacy and numeracy in those early years of primary education? So, to be clear, literacy has lots of benefits and a lot of them are difficult to quantify no matter how numerate you are, right? Lives are richer as a result of that. But a specific question is, do these early investments result in higher earnings in adulthood? Now, there's an intuition to it that's simple. If you can't read, you can't become an engineer. You cannot become a doctor. At least we hope that you can't. There are also a lot of correlations that adults who read well earn more. But evidence on the actual earnings gains from improving the quality of education and helping kids master these skills in the early years in low- and middle-income countries is remarkably sparse. So today we'll hear about two studies in two different countries that explore these returns with greater rigor than what we've seen before. Lee Crawfurd, a senior fellow here at the Center for Global Development will share new work on Indonesia. And Jishnu Das, a distinguished professor of public policy at Georgetown University will share new work from Pakistan. Then we'll hear from two esteemed discussants from the donor community, Jo Bourne, the Chief Technical Officer at the Global Partnership for Education, and Jennifer Swift Morgan, the Director of Foundational Learning at the Prevail Fund. So, what I invite you to consider over the course of this next hour is the following. What should donors and national governments be doing differently because of the evidence that we see today? What complementary actions and evidence would help change minds or change budgets to do things differently in order to reap the returns of mastery of these early skills. We'll have a few minutes for questions at the end, so I hope that you'll be thinking about the questions that you have along the way. And with that, I will turn the mic over to Lee Crawfurd, and then we'll go straight on to Jishnu.
[00:10:40] Lee Crawfurd: Thanks very much, Dave, for the introduction, and thank you all for coming. I'm excited to be able to present this paper. This is a working paper on the CGD website, which you can find. The Economic Returns to Foundational Literacy and Numeracy, evidence from Indonesia. And so Dave did a great job of setting the scene about why we should care about this question. This is just kind of a reminder of these are the sustainable development goals for education. There's 10 targets that were laid out, and none of them is about foundational learning. There's a sub indicator under one of these high-level targets that is specifically on foundational literacy and numeracy. But back when these were being developed 10, 15 years ago, it wasn't at the forefront of everyone's minds as it is, I think, today. And you know, in fairness, there are many outcomes in education that are important. All of these goals are important. But there's been a real sea change here. Why should we care about foundational literacy and numeracy? I think this is a really helpful graph for illustrating one reason to think it's important. This is a slide from Kartik Muralidharan's paper, and it shows learning for children from grades one through five, and it shows the different parts of the distribution. And so the very top line are children, the top 10% of children, they're on track, they're making good progress. The bottom 10%, the line is totally flat, they're making zero progress. And the median child is reaching grade one level by roughly grade four. So the idea here is the teaching, if you're on track from the start, and the teaching is sufficient, that the top 10% of children are making the expected progress, they are at the expected grade level. But if you fall behind, then you just fall further and further behind. If you haven't learned to read in those first grades, you're unable to read to learn. And so we would expect you to make much less progress and be more likely to drop out of school. That's the theory, but we've had little evidence on this. This is just a snapshot of some of the momentum we've seen in recent years. So, you know, 2015, when the Sustainable Development Goals were being developed, it really wasn't anyone's number one priority. I think that changed in part with the World Bank's World Development Report in 2018, which put this big spotlight on the global learning crisis. And then the year after, the Global Learning Poverty Indicator provided a kind of easy to understand metric for this learning situation. Then 2020, India's new educational policy put foundational literacy and numeracy really at the forefront, the center of that policy. And then we increasingly see national governments taking up this as a priority. So 39 countries signing the commitment to action on foundational learning in 2022. Ministers from 18 African countries signing the Adaya Declaration, which really had foundational learning at the forefront. And the president of Zambia hosting a two-day forum in 2023. So it's really risen up the policy agenda. But how about the evidence? So this is my literature review of studies on the wage gains. There are hundreds and hundreds of studies showing the wage gains from schooling. 705, according to one recent review. Studies estimating the wage gains from foundational literacy and numeracy. As I was writing this paper, I found one. To that, we can add this paper and Jishnu's paper, and hopefully some more in development. But it's really quite a sparse evidence base. And there's good reason for that. To estimate the returns to schooling, you could ask an adult, how many years of schooling did you complete? And then you can ask them, how much do you earn now? You can't ask an adult, how good was your reading when you were in grade three? Because they can't answer that. So you need to have tested it when they were in grade three. And that's obviously logistically challenging. So we have this one study from low and middle income countries. There are more studies, a few more studies, from the US and Norway. These are very different contexts, very different labor markets, so we might expect to see differences. And then there's a raft of studies showing the contemporaneous correlation between if we test your skills now as an adult and your earnings, and we see this correlation there, but this could be quite different to your foundational skills when you were in early primary school. There, roughly, we see a one standard deviation increase in skills, which is roughly moving from the 50th to the 84th percentile in ability, is associated with a 15% increase in earnings, so reasonably substantial, but we don't know if that's going to hold for early grades. So in this paper I searched through all of the panel surveys from low and middle-income countries that I could find. With these two criteria we need a cognitive test in early primary school, and we need measures of adult outcomes. And these panel surveys and many more, none of these surveys fit the exact criteria with the exception of this Gansu survey from China, which Paul Gloving co-author's uses, and the Indonesia Family Life Survey, which has been used by many, many studies, but not for this purpose yet. I believe the Young Lives Survey is still ongoing, and most children are nearly adults, So we may soon be able to add to this from other data sources. But at present, we have this Indonesia survey, which is not quite representative of Indonesia, but covers a broad swathe of Indonesia. The sample is just over 1,000 students who were tested between ages of 7 and 12 in 1997, and then resurveyed as adults in 2014, age between 24 and 29. We have quite rich covariates about the children. So what grade they were in, their age, their sex, their household wealth. So we think that's going to matter a lot for their educational outcomes and their later earnings. Their parental education and their nutritional status, whether they were stunted. So we have a fairly good set of background controls. And then in terms of end line measures, we have, again, quite a range. So we have employment, roughly 73% of the children were employed. Their earnings, their completed education, So it's around 11 years on average, their health, their fertility, and their life satisfaction. So we have earnings as a primary outcome that we're interested in, but we have a bit of a range. The assessment, this was a test in 1997 before we've made a lot of progress since then on developing good tests of foundational skills. But nonetheless, I think by our good fortune, this is a reasonably good assessment that measured similar concepts to the kind of things that are now in the UNESCO standard for global minimum proficiency at the end of lower primary school. So it's a 40 item maths assessment, mostly basic arithmetic, and a 35 to 40 item Bahasa assessment, which is mostly reading comprehension. So I think this, you know, roughly fits well with what we think about when we talk about foundational literacy and numeracy. So what do we see? This is the results here in a graph showing the correlation between foundational skills on the x-axis and the log of your salary on the y-axis as an adult. And you can see it's steeper before we make adjustments for all these background controls. Slightly less steep when we make these adjustments, but it's still clearly positive. And the magnitude is roughly a 10% increase in earnings for one standard deviation increase in early test scores. So again, that's quite a big increase in your ability, say from the 50th to the 84th percentile of the distribution. So it's slightly smaller than those studies looking at the test scores of adults, but it's still pretty substantive. Other outcomes, we look at these health outcomes. We don't really see anything that's statistically significant, whether you accessed care, whether you report being sick in the last month, whether you report generally being healthy. And we can look at height as a kind of cumulative measure of your nutritional status. I'm showing here the effects of early family wealth just as a comparison. But we don't really see big effects there. For the other outcomes, we look at early marriage, before 18, which is quite rare in this data, and we don't see effects there. We do see an effect on whether you have had any children. So this is, again, by age 24 to 27. You're about three percentage points less likely to have had a child, which is reasonably substantial. There's no statistically significant effect on subjective well-being, which is your life satisfaction measure, or on employment on average, but some impact on overall household income, which is not too surprising. There's one aspect of heterogeneity, so we look at heterogeneity across to see if effects differ by a whole range of different baseline variables, and nothing statistically significant with the exception of employment, where there is a difference for men and women. And that's essentially because 90 plus percent of the men are employed regardless of your early skill level. Whereas for women, the employment rate is lower and you do see this significant gradient here with skills. So women who had higher early foundational skills are more likely to be employed at all as adults here. Thinking about the mechanism through which foundational skills increase your earnings. So this is showing the effect on first adult skills, and then on your likelihood of completing different stages of school. And so this translates as around a 5 percentage point increase in your likelihood of completing secondary school if you had higher foundational skills. So this does provide evidence for that initial theory that you're more likely to stay in school if you're staying on track with keeping up with the curriculum. And that adds up to overall half a year of additional school that people are going to have completed. And that partly explains the effect on earnings, but not completely. So even after we control for, as an adult, your educational status and your adult skills, there's still this effect of your early skills. So it's partly about you helping you stay in school longer and schooling matters in itself but it's partly also you've just learned something which is going to be helpful in the labor market. So what does all of this mean for policy? So here I've taken another CGD working paper which evaluates the full set of USAID early grade reading programs and from for each study there's a standardized effect size on test scores and the cost of that program and so we can calculate the benefit-cost ratio for each of these programs using the coefficient here. So for every one standard deviation we're saying there's a 10% increase in adult earnings and so we can look at the lifetime value of that increase in adult earnings and compare that to the cost of the program and you can see there's quite a wide variation here but on the left-hand side the average benefit-cost ratio across these programs is 9.3, so for every dollar you're spending on the program you're getting nine dollars back in lifetime earnings for those children, so it's a really strikingly good investment. And on the right-hand side that shows the dollar value of this increase in earnings from being exposed to one of these programs, and this is hundreds of dollars, but you know in many of these that the poorest countries, the average spending on education is 50, 100 dollars per child, this data suggests we can justify spending hundreds and hundreds of dollars per child just purely on this benefit-cost investment, never mind the value of being able to read for all sorts of outcomes. So this is observational data, you can argue with whether this is truly a causal effect, and Jishnu's going to be able to say more about that, and hopefully more research coming from CGD is going to really nail down this causality question. But if you take these estimates at face value, they imply these really big returns on investment. So to conclude, these wage gains are, I think, slightly lower than some of these previous estimates, but still imply really high returns on investment, and hopefully more research is going to help us pin down precisely the relative cost-benefit ratios of these kind of programs compared to all the other things you could spend aid money on. But this evidence I think is enough to justify quite substantial investment already. And just as a coder in the current environment of the aid cuts, it's really sad speaking to people in working with governments where they've faced these withdrawal of these large USAID programs. And in many places, these gains weren't very visible. And so there hasn't been the same kind of crisis in the education sector as I think we've seen in health where it's literally life or death and people are dying. In education, so the kids are going to learn 5% less, there's going to be less teacher training, there's going to be fewer books, but there's less of a sense of crisis and urgency and people are kind of just moving on. So I think this research is important for showing that these gains, you know, are real and they do really add up to something important. Thank you.
[00:25:19] Jishnu Das: Thanks Dave, for having me here. So I'm going to talk about this program we've been working on in Pakistan, and Yash is the one doing all the work. I'll take blame for the stuff that's wrong, but really a lot of the work is coming from him. So you know about the learning crisis, just some numbers, it's really bad. 90% of 10-year-olds cannot read in Sub-Saharan Africa. Only 50% of rural grade five students in India can read a grade two text. Latin America, with COVID, things have become even worse, as they have in the US. And just one number that I keep telling people and they keep forgetting is 24% of American adults now cannot read. So it's just not a low-income country problem, it's an issue that's around the world, right? And I want to take you to the solvency problem of solving the learning crisis. Why has it been so hard to solve? And I think the fundamental issue is kind of fourfold, right? The first is you're investing in kids in early primary, and the first question you ask is, well, I'm investing then, is the return all going to fade out? In one year, are we going to see these thing stick? The second problem you're going to hit is even if they do not fade out, we don't know whether these kids will be able to continue, right? So they have all kinds of problems they face getting into high school, getting into college, getting on the labor market. Are these labor markets strong enough that they're going to recognize some improvement in test scores 10 years back? Then you have a third problem, which is these returns are going to be realized 10 to 15 years later. 5% interest rate for every dollar you put in, you're going to need $2 back. The last one is, well, what Lee already pointed out, the current evidence is correlational. What do we want to think about causal evidence? What do we want to think about whether, when you see these in experiments, are they actually going to survive that long? But what I want to also point out is there's a reason I put the solving in inverted quotes. And that's because Dave has a wonderful paper where he says, look, a really great intervention in this space doesn't push you up one standard deviation. it pushes you up 0.4 standard deviations. And the fairly good programs, all the work that Justin has done, the best effects they're getting is about 0.1 standard deviations, not one standard deviation. So let's see what that means. Here's our test. And here's theta at 0 standard deviations. 23% can read, listen to, and write the correct word in English. 0.4 standard deviation, that's going to go up to 39%, right? 17% can read, we'll have some reading comprehension at average. So theta zero is like the average kid. And 0.4 standard deviation is that's going to go up by 2%, right? Look across the board, it's the same. I mean, look at Urdu reading comprehension, which is even lower because it's complicated. It's going to rise by five percentage points, from 3% to 8%, right? With a great intervention, with an intervention that Dave tells us is great. We're going to have the same problem in math. Quantitative reasoning, for example, is going to go up by some small 18% to 25%. And we expect these are the kind of returns, the 0.4 is our go-to best intervention that we're going to find out there. So I'm going to come back and say, OK, given the 0.1, 0.4, what do we expect, right? And what I'm going to take you away from, from, I'm going to take you away from Indonesia and to the fields of Punjab, where in 2003 we did an experiment, right? And it is an experiment, so we have villages that were treated. I'll show you what the treatment is, and because it's an experiment, all the villages that were not treated with our experiment looked identical, right? In this case, because I copied and pasted the identical village, But the way we think about it is it looks identical because on average, they were randomly assigned, right? We published a paper showing the impact of this intervention where we distributed report cards to parents. We tested the children and then we said, hey, here's the subject. So that's Urdu, that's Angrazi, that's Riyaz, which is Urdu, English, and math. And here is your child's score. Here's the children in your school. And here are the children in your village. And then the report card had a reverse side, which was, again, the subjects. And here are all the schools in the village and how they're doing. And we showed that the test scores increased by, it's a good intervention, so it increased by 0.1 standard deviations to 0.2 standard deviations. How big is this improvement? it moves children by four percentile points, right? So if you're taking the SAT, everybody here is in the 90th percentile, it moved you to 92, 93, okay? What do we think will happen? Well, most of the literature finds that these effects are going to fade out. These are small effects to begin with. They're going to fade out, right? So we didn't have much hope. We went back 15 years later. We went back multiple times, but we went back 15 years later. And we tracked down the children who had been part of a survey of households. And we now look at, did that tiny increase there, did it have any effect on the increase in wages later on? Just to give you an idea of how long this took, this is where all the kids were. We had started in 112 villages. They had now spread out to 765 villages. 10% of the men had left the country. We waited two years, three years, for them to come back. We spent three years tracking each one down. And later, we have data on 94% of the kids from way back. This is what they're doing. The women don't work very much. Most of them are not in the labor market. And where they are, you can't see this. I thought for some reason you guys would have a big screen here. And that's the men, don't try and read it, but just remember the mean wages are about $140 a month, okay? Here's how much our report card cost, it cost $3, right? Here are the returns, $3,142. Okay, does anybody know what Monster Beverages is? Monster Beverages is that energy drink. The reason you should know what it is, is because it's the highest earning stock since we started the LEAPS report card to date. 121,000%, right? That number there, the way that calculation is done in the USAID paper, that's equivalent to this calculation. We are getting returns of 140,000%, right? That's not entirely the right way to do it, because it turns out that the biggest cost is actually because these kids are going to go to college, they're going to lose labor earnings. So that's the one that you really have to account for. But after you do all that, you get a benefit cost ratio of 6.4. You get an internal rate of return of 21%. Tell me which IRA Roth is going to give you 21%, right? And you get what we call a marginal value of public funds of infinity, which means the government makes more taxes on the returns than it costs to run this program, okay? So what if, you know, these are the gains of an intervention that improve learning by 0.1 standard deviation, right? What if we could do a lot better, right? What, you know, would the gains be greater? And how much should we be spending to improve learning by, say, 0.4 standard deviations. This is the graph that Lee was showing with the correlation. And the report card moves people from here to here. That's what it does. And it gives you the single biggest return we have ever seen in a development intervention in our lives. And you could start asking, what happens if we move kids from here to here? So that's what we started working on next. And what we use is an earlier paper we write where we go back to our villages, and we say it actually turns out that there is massive variation in the quality of schools within these villages. So this graph is every dot is a village, and the lines show you the variation in quality of schools within the same village. And we started asking, do we see differences between children who are going to the best school and the worst school? We spent most of the paper asking is this highly technical, how is this causal evidence, where's the identification coming from? We can talk about it, right? But that's where most of the paper is. And students in the best schools in a village will causally increase their test scores by 0.4 standard deviations, more on average than students in the worst, right? So here's what we find, and what's identical to Lee's correlations, and these are now causal estimates is the main story that's happening is these kids are going on to more schooling, right? So they've completed grade 8 a little bit more, they've completed grade 10 13 percentage points more on a base of 43 percent, so that's 30 percent more. They've completed grade 12 9 percentage more on a base of 19 percent, and they are 5 percentage points more likely to go to college on a on a base of 12%, so it's about 42% more, right? You can put all that together, and what we show is if you move a boy from the worst school in the village to the best school in the village, that increases years of schooling by about two years. And with 74 boys in the average school, the average lifetime gains in earnings will be $302,000. Okay, think about that number. That says the number which we should be willing to spend on improving a school that has 74 boys every five years in rural Pakistan, where the mean wage is $140 a month, is $300,000. We are not under-investing, we are under-investing by 1,000x, right? So these are the results from rural Pakistan. And just to think, we have now the results, Yash has been working on the results from the Young Lives, because the adult data we have access to, and they're identical, right? So we're getting now, we can do, now we have on correlations, we have Indonesia, Vietnam, Peru, Indonesia, India, Peru, Ethiopia, Pakistan, and then on Pakistan, we have the first causal results. right? But I don't want to leave you with just numbers, right? So this is, so she's one of the girls who did really well in the LEAPS exam, and we did find her. I actually did find her in the data, and we started looking. She's doing well. She went to college, you know, and she seems to be doing well, and if you look just as the correlations, we haven't done the causality yet. If you look at all of these, it seems like, across a range of different things, and especially for women, teen marriage, so there are all these papers looking at geniuses in the U.S. or the kids who were doing really well and what happened to them later on, and they don't compare them to others. And they say, well, they were kind of like everybody else. Here they're not. The women with the highest foundational learning scores, teen marriages were 7% compared to 23% among the lowest. Teen fertility and fertility in the early ages was lower, and years of schooling were almost doubled, right? So these are big effects. I mean, these are effects that you would have thought 0.1 standard deviation, that's not a lot, right? 0.4 standard deviations, that's not a lot. If we spend the money, well, people ask, how am I going to get that number? And I say, we have no clue because you're not spending the right amount of money. Give us $300,000 per small school for five years, I'll show you what you can do, right? But I don't want to leave you just confused with a lot of numbers. It's worth just putting, there's a human story behind all of this. And I'm a parent, a lot of you are parents. And the day your daughter or son comes back crying, because they've done really badly, and they have no option because the teacher's not good, right? That's the day you realize that there's something is really, really wrong. And this is a photograph, the two women at the back. The reason I put this up is we disseminated the report cards in the villages. And I used to go to all of the villages. I went to like 40 or 50 to disseminate them. And in one of them, the woman saw the numbers, And she was just like, this can't be right. And the woman next to her said, well, why don't you check with your kid whether they can read? And she said, but I'm illiterate, how am I going to check? So the other woman said, the way I do it, and she called her kid, and she said, the way I do it is I recite this poem I know. I ask him to write it down and then read it back. And the moment she said the first line, it was so clear the kid didn't know how to write. And this woman just broke down. Right in the meeting, she just broke down and started crying. These were the same two women. So then later, they set up their own school during COVID. And they somehow managed to find teachers to give tuition to these kids. And an incredible story from Pakistan is we see no COVID learning losses at all. It's come on the back of the woman. It's come on the back of the willingness of people to invest in learning. And I think everybody on the ground is doing it. We've got to hold our part. Thank you.
[00:40:50] David Evans: Thanks everyone for coming. I invite both Lee and Jishnu, and Jo and Jenn to join us up for the panel. Early investments, at least I found it compelling, and we'll hear from others from two different countries, from two very different models. And so, you know, we’ll turn to the donor community and the philanthropic community, I'd love to just hear, A, any reactions to this evidence and what complementary is there other evidence that we need in terms of sort of how to put this into place and then most important like what should the donor and philanthropic communities be doing differently in the light of this. Jo we'll start with you.
[00:42:17] Jo Bourne: Yeah well first of all thank you very much for the really engaging presentations there was so much that struck me I mean I didn't know 24 percent of the adults in America couldn't read I think your comments on taxes and the returns to investment, you know, were really compelling. And Lee, I was struck by, I was thinking about, and I'm going to show my age here, the conversation we had before the SDGs came into place, was very much around, we were fighting to even get learning into the SDGs at all. Some of you remember some of that sort of fight that we had. And now, over 10 years later, we're actually fighting to keep 4.1.1a, the basic literacy and numeracy, in place because the data availability is so bad. So, you know, there is an ongoing fight that we need to keep making that we're not going to, you're not going to be able to do this research if we don't have the data and if we don't have the data in a really convincing and comparable way. So it's a real privilege to hear this and, you know, it really tells us what we kind of know from experience in a way, but it puts rigorous evidence behind it, that investing in foundational learning, and when we talk about it, we're talking literacy, numeracy, social emotional skills, et cetera, it really does create lasting dividends across intergenerations I suspect as well. So thank you very much for that. I think for GPE, this is deeply relevant. Much of the way in which we work is asking our partner countries, what is it you want to achieve in the next few years? What reform do you really want to manage change over? And just under half of them are choosing foundational learning in some way or another and saying we want to do this and we want to do it at scale and we want to do it across the system. So you know many of our governments, our partner countries are wanting to put their own resources and our resources and crowd in other resources behind this effort. So I think the more that we can equip them with the arguments that they need to make in national planning and national investments is really, really important. So much of this research begins to validate some of those choices, but as you say, there's much more that still needs to be done to really make the case that this is a sound economic investment. And the country should be focusing their scarce resources on this as a first piece. Now, there are some questions I think we still need to continue to explore. I think one of the big ones, and you mentioned it, is how do you sustain gains over time? You know, we do see that systems are achieving short-term improvements in reading and numeracy. We do know that that progress sometimes stalls through the grades, and that sort of alignment and that through a curriculum teacher training assessment, really aligning those things so that those early gains are reinforced and not lost is really important. I think in other pieces around the equitable learning for all children, which you didn't touch on quite so much, is large gaps exist in foundational learning and skills by gender, geography, fragility, disability, and there are some studies that suggest that there are quite good returns to investing in some of these more difficult, that they're even greater than some of the standard effects. So we not only need to invest in the interventions, but also really ask serious questions about how those interventions are reaching the most marginalized and how they work alongside some of those other interventions around nutrition, language support, safe schools. I mean, there's been some great research done that really looks at the impact of violence on children and children's learning. And some of those effects are very difficult to get your heads around, but they will affect learning. and we know children in very traumatic situations do not learn so well. So there's a whole sort of ecosystem around this that we also need to take into account on. I think the thing that keeps us awake at night a little bit is how do we make these reforms kind of stick within systems? You know, systems are very complex. Even when the evidence is clear, this is the intervention coming from Dave. You know, how do you actually secure these interventions across a system that's got so many moving parts? and one of the things we've been looking at a lot is actually asking leaders, and we asked Indonesia, Ido State, and Nigeria, and a number of others, how did you affect change? Not just what did you do, but how did you affect change at a system-wise level? Okay, my last minute. What does it mean for the community, the education community and funders? Now, we don't consider ourselves like a donor in the traditional USAID sense. We are a fund, but we're a partnership. And I think you've pulled me straight out of writing GP's new strategy, I'm in the last mile. So I'm thinking about what does this new world mean for us as a partnership, which includes many people? I think one thing is that we need to move from fragmented projects to system enabling investments, and I would count this. And these returns are only realized when we're also working on things like fiscal space, teacher capacity, and data systems. So we really should be looking at how we can pool aligned support behind government investments and not running parallel projects. I think, and I mentioned this earlier, investing in data and evidence use, I mean, you wouldn't have been able to do this without the data. And that's been a long-term investment. So one of the things that we're putting in place is a results-based mechanism, specifically on 4.1.1a, where 10% of the allocations we give to money will be tied to governments making improvements in the availability of that learning outcome data along whatever pathway works best for them. And then I think the third and my final point is we need to look at new financing models for education, particularly how we use external financing, traditional grant aid, it's declining, even as these needs in education grow. So how can we sort of mobilize blended, innovative finance, concessional loans, guarantees philanthropic capital, really to stretch odor and catalyze and really incentivize domestic investments in the right thing. And that's something that we're looking at very much in our new strategy as well. So there you go.
[00:48:47] David Evans: Thank you so much, Jo. And I would just compliment this point about the shrinking aid budgets that there's a nice piece of analysis that was just posted on the CGD website recently from Susannah Hares and Jack Rossiter showing that across all the education aid, like the primary education aid is the one that's dropping the most, right? Exactly the piece where we just saw these enormous returns for. And so, figuring out both how to protect that and how to leverage what there is, I think, is really crucial.
[00:49:18] Jennifer Swift-Morgan: Thanks, Dave. You could have set me up better for that, thanks. And I'm just really thrilled to be here. Thanks for the invitation. We're just tremendously excited to see these results. The Prevail Fund is a new foundation, and we started with foundational learning as our first program because we think it matters. And we're learning today that we don't need to just think it anymore, and that we know it, and we really know it matters in longer-term ways, not just in immediate ways. So to your question about what donors should be doing more, I'm just going to take Lee and Jishnu’s advice here. I think we should be investing more. I think countries need to be investing more, and donors of all kinds need to be investing more. Let me just get a few things of these right. I heard 140,000% at one point in terms of returns. I heard that we are investing, we are under-investing by 1,000x. And I've heard Lee, who is a fairly, I would say reserved academic, saying this is a strikingly good investment. I would say if a British person who is an academic tells you that something is strikingly good, as an American, I would have been saying, this is awesome. We need to be listening to the data here. And as Dave just said, we are spending less on education, not more right now. We are spending less on basic education, compared to youth work and other things, not more. And in a world right now where we're trying to do more with less, we are in an aid crisis right now. And if we are trying to do more with less, then we should be doing the things that work. So for us in the philanthropic sector, traditionally foundations have given smaller amounts spread out, and we're thinking that we need to be doing fewer things, but doing for those big bets, and sticking in places until we get the job done. And the job done for us means getting to a new normal, so that not just access to school, but when a kid goes to school, the normal should be that she could come out reading and doing basic math. And so we can be doing more of that to link up with the bigger aid and to be focused on this in a system way. And the other thing that this evidence does is that there are different things that motivate different donors out there in terms of individual donors. Some of the folks that might be giving more off the sidelines would be looking at this bigger picture really compelled by that. So quickly to the question of what more questions can we ask about this. We're seeing that even a small improvement in FLN can get a good return, but there are three quick questions, quick because I don't have a lot of time. First is a question of what does good FLN programming look like? So if you go back to the CGD study of looking at a decade of USAID programming, we saw there is a wide diversity of actual impact that we're getting with these programs, right? We haven't seen quite so clearly why. Where is that diversity coming from? At Prevail, we're playing around with this hypothesis of a formula, akin a little bit to the simple view of reading, that design times implementation equals impact. Playing around with this a little bit, right? So design starts with the core inputs. Do you have the books? Do you have the training? Are you doing scorecards? And then are those things actually mobilizing the evidence base for the most cost-effective way of getting impact with those inputs? And are they all linked together? This is nothing more than the WDR report from 2018 of getting things aligned to learning and having them be cohesive, right? Secondly, implementation. Are the core inputs that you have actually being implemented, right? We know that, what is it, implementation that is designed for dinner or something or for lunch or for all the meals. So do we actually, are we actually getting that done? And do we have the data to tell us whether it's getting done or not? So if you go back to the equation, you've got great design, great. But if you don't have execution, you're not getting impact. If you have great execution, but the training that you're giving to the teachers isn't actually based on something that's going to get them read, you're also not going to get impact. So how do you get to those better results? Do the governments and the organizations that are helping them have what they need to develop that good design and to deliver the implementation? And going back to your ecosystem perspective, right? Who is it that's demanding that those results come? is that those parents are the mothers actually having that information and demanding of their own governments that we set good goals for learning and that we actually achieve them. So I think in terms of, again, what donors and countries alike can both do is not just increase the resources, but increase our expectations of what those resources should do and incentivize getting real impact. And finally, just going back to the study in terms of what if we could get more, right? So how much good FLN do you need to have to get a good result in order to get a good longer-term investment? So at Prevail, we have a goal right now of getting to at least two standard deviations better than the status quo. We picked that because at the time, the track record shows that that's actually fairly ambitious, right? And we're seeing today that even one standard deviation could get some knock-on effects. We're also having a deeper conversation, I think, as a general community and in our team about like, well, is that meaningful enough? Is 2SD actually going to get a kid that reads or can do basic math, right? Because there are immediate returns that we're also looking for in terms of the longer term also. And so if we actually use the most cost-effective approaches all the time and did that at a system level so that you get a whole classroom or nearly a whole classroom reading or a whole village reading, or a whole country reading, what would the returns be to an individual and to a country in terms of inclusive growth if we actually achieve that? What would it cost to do it? But then I think also the question is, what would it cost not to do it? What would it cost not to invest this, to an individual, to a mother, to a young girl, and then to an entire country that's, again, trying to achieve inclusive growth? So we're eager to learn about all of this going forward with the ambitious learning agenda that you all have, and just to see how much good can come from actual good FLNs in the end. Thank you.
[00:55:38] David Evans: Absolutely, thank you so much, Jenn. I think one of the things that is a challenge to hold these two ideas in our head is both that we need interventions, we need to invest interventions that deliver much bigger gains in foundational literacy and numeracy, but also holding the fact that even interventions that deliver modest gains in foundational literacy and numeracy deliver very large returns, right? The intervention that Jishnu showed was a modest increase, right? A 0.1, 0.15 standard deviation. What we think of is the median effect size across education interventions, right? Even so, we have these enormous impacts. So, keep both of these in our heads, right? We want to make bigger impacts, but even where we are, the returns are enormous. We have a few minutes for questions. And if you see me looking at my phone, I've been monitoring the online questions. Does anybody in the audience here have any questions for our speakers? Yeah, and we'll take a couple to start, and then I'll bring a couple of online ones.
[00:56:44] Alim: Hi, my name is Alim Walji from the Five and Five Impact Alliance. One of the questions I have is when it comes to investments in FLN, and I hear, for example, the Gates Foundation talk about structured pedagogy. It's not clear to me that we have a standardized way of talking about what an intervention in FLN is. Because you see things that I would describe more as point solutions versus systems approaches. They obviously are very different, both by design and what they cost. And so, when we talk about a return to investment in FLN, is there a standardized way of understanding what is that investment, what does it look like?
[00:57:29] David Evans: Great.
[00:57:32] David Evans: Yes, Brad.
[00:57:34] Brad: I just want to build on that a little bit. I'm Brad Wilson from the Brookings Institution. And this comes out of ignorance, really, because I'm a qualitative researcher here. But I'm very curious as to what is in the FLN that's showing the effects. And is there something else maybe going on? I think about research done in previous decades that found that families with books in their homes, their kids did better. So the solution was, let's have more books in the homes. But it turns out it wasn't about the books at all. It was about something else. Or those prior decades of research saying coffee was bad because we didn't disaggregate between smokers and coffee drinkers. I'm just wondering what's in the treatment that might be explaining some of these returns.
[00:58:15] David Evans: Great. And I'm going to bring in a couple of the online questions and I'm going to direct a little bit since we don't have an enormous amount of time. So Jishnu, I'd love for you to take a crack at Brad's question about what is going on here. And there's a related online question, which is how much of this is coming from keeping kids in school longer versus some other productivity. Lee already spoke to that a bit in the Indonesia case that we see some of it is through increased schooling, but not all of it. So I'd love to hear in the Pakistan case as well. Jenn, tell us what an intervention is. Jo, one of the online questions comes to, if we have these enormous returns, like what is the constraint? Is it mostly an information gap? Is that, hey, we have these new studies before we didn't have this information and we're going to go share that information and people are going to be like, wow, yes, We got to increase our education investments by 50 fold. Like what would you say in addition to this knowledge? Like what's the big constraint? And then Leah related question to what came out on the mechanisms, which is what about school dropouts? And related to that, one of the online questions was, what about, you know, we haven't completed the agenda of access to schooling. We still have lots of, we still have one in 10 kids worldwide who aren't finishing primary school. How much, and obviously those kids aren't learning very much, and so how do we think about the access agenda in the context of the foundational literacy and numeracy agenda? Why don't we start with Jenn and work our way down.
[00:59:55] Jennifer Swift-Morgan: So, I can talk about interventions. I won't be able to answer Brad's question about what of that, right, is actually getting to the effects because I'm not sure we've disentangled all of that yet. So, when we're talking about foundational learninliteracy,acy and numeracy interventions, we know that there are a number of core inputs you can get. But I want to start first with the learner. The question is, what does a learner need to actually learn to read and to do basic math? And we know a few things about that. We know that a learner generally needs a teacher, a good teacher that knows how to do those things. They need resources. Those materials could be books, they could be increasingly some digitally enhanced things, and they need parents. And so if you look at the research on the best buys on the cost-effective things, it's getting at those things. A scorecard gives a parent information. Now that parents are paying attention to that kid, getting back to the learner. In the classroom, the instructional core, what's happening? So some of the labels that we've been using, structured pedagogy, targeted instruction, one thing that we were really learning a lot at Prevail, in particular, when we look across our portfolio and when we looked across, again, the CGD report on USA programs, what is under the hood really matters. So you hear it's something structured pedagogy, but is it using systematic explicit phonics? And are kids getting enough time to practice those things? So I think we need to go beyond the labels and at the design level of the equation, really look carefully about, are they actually applying the evidence base? And then again, back to the implementation part of the question, if you have the design right, is that actually happening? Is the teacher actually using that book as intended? And do we have the information to tell us? So I think it comes back down to those things, but we're learning that details really matter.
[01:01:41] Jo Bourne: So you asked about constraints and I came into two, constraints to implementation or constraints to actually making the case for foundational learning. So I just want to build on what you've just said, but also acknowledge some of the work I know Brad's done around the different entry points into system-wide change, which was quite an influence on me. I had the privilege of co-facilitating a workshop in Nigeria a couple of years ago, and this is coming to constraints, and together the partners, and there were six states that we want to work on foundational learning. The first question was, what do we mean by foundational learning? So I just put that out there, that what we assume as common knowledge is not necessarily common knowledge, first of all, when you're talking to implementers and people or politicians who need to drive these things. But then as you dig down deeper as well, I think that many of the, you know, many of us can have access to descriptions of what the constraints are on implementation. But sometimes when you talk to people, some of those details are actually the constraints that are stopping things from happening. And so you do have to dig quite deep into what is stopping delivery and for whom and where. And that's not always something that you're going to get at without actually listening to people. So I think there's some, I really agree with what you say. it's about understanding the real implementation constraints in any given place. Now, when it comes to the financial, you know, the kind of, why aren't people making the investments, which is so damn obvious, right? Sorry, excuse my language. And yet, somehow, we're not cutting through with the argument that this is, you know, this is a lot of the investment. I can only postulate, I think we need to continue to actually make the case for education as an investment, not just an economic investment, but also for social gains, and for peace in the world that we're at here now. We don't always have all of that evidence. I think we struggle because of the long-term impact. I think our ability to be able to craft that more powerfully is incredibly important. I think we struggle with the results, that in an environment where people want to invest quick and get quick returns, we are struggling to articulate the results that will help fuel those investments. So I think there's quite a lot of constraints around the investment side that's not just about the foundational learning, but actually about education full stop.
[01:04:24] Jishnu Das: Okay, so let me give you why these are really hard questions to answer. So imagine that I want to say, was this going more to school or was this scoring, right? Then I have to say, I need to find children who went more to school but didn't learn more, because otherwise the two are infinitely confounded. A child who's gone to school more will have always learned more. So then I'm saying, but then what is that school where they're going, but they don't have higher test scores, right? So I'm looking for a very weird part of my sample in order to be able to answer a question of, what is the return to going to a school that doesn't help, right? So, I think that question is an incorrect question to ask. That's not an answerable question. The second question you asked was in a way very deep or very shallow. I think the examples you gave were very shallow, because they were not randomized. Those were not studies which said, we'll randomly provide books to parents and see what happens to their life outcomes, or we will randomly make people drink coffee. They were wrong precisely because they were correlational, not causal. This is causal evidence, right? We don't actually know whether 0.1 standard deviation changes your motivation, changes your perseverance, does it affect the way your parents viewed you, and therefore were willing to invest more in you. We don't know which of those pieces it is, but we are dang sure that this is causal. That much we are completely sure about. And I want to make one last point. Oh, I still have three minutes. I want to make one last point, which is the following. We discussed a lot of what is the precise intervention. If you want to figure out how to improve foundational learning, we should not go to the people who have not been able to do it for the last 50 years, which would be people in the U.S. and Europe. So I want us to discard every piece of what people in U.S. and Europe are telling low-income countries. The places that have done it is Syria and Brazil, right? Punjab, Pakistan, where we are now finding they managed to increase it by 0.6 standard deviations in 15 years, at the province level of 23 million people, while bringing in three million people more into school. How did they do it? They just ignored everything people were saying. So I want to take that view seriously, that this is really not about finding a product. It's about trusting people on the ground. The information we gave was, here's what's happening in your school. Go do something about it. Nothing more. We have to trust the parents. We've got to trust the people on the ground, which doesn't mean that we discard the evidence that's being created. We say, hey, here's an option. It turns out that when these guys tried the structured pedagogy stuff here, somewhere it worked, somewhere it didn't work. It's an option for you guys. It might work, might not. Exactly the same way teachers in Pakistan are looking up YouTube videos to figure out how to teach. So, I want that respect, that degree of localization, and that degree of democratization to come forward. I will not take evidence from the US and Europe where things are crumbling, and foundational learning is getting worse as a way to figure out what to do.
[01:08:46] David Evans: I would just compliment your point on, sometimes these work and not. I think the example of the evidence you presented of scorecards, right? Tell people what's going on in schools and then count. Obviously, a lot of us who've been in the field know there've been a lot of scorecard interventions, very mixed results. And so this goes back to Jenn's point about both the context, but also the implementation quality. So how is this designed? It's not just this label, but it's a particular kind of intervention that empowers households on the ground. Lee, last word.
[01:09:22] Lee Crawfurd: Yeah, I'd just like to add on this point what kind of intervention looks like that improves foundational literacy and numeracy. It could be a whole raft of things. It could be the Structured Pedagogy Program, or we're doing a lot of work here on lead exposure, and it’s roughly half of children in Africa and Asia who have elevated blood lead levels. And if you removed that, you would have this enormous improvement in foundational learning because their cognitive development is being stunted. And so that's a whole different set of interventions, but not something we typically think of as an FLN intervention, but something which could significantly boost foundational learning. And just to touch on Jo's point on the importance of data, we did some surveys with policymakers a couple of years ago asking officials and ministries of education, how many of the kids in your country can read, do you think? They all massively overestimated it compared to the data. With the exception of Uganda and Kenya, where they have this OASO surveys and so they were used to seeing the data and there they were much more accurate in their estimates. I think that's the same in Pakistan and India that it's a shock the first time that you keep on showing people this data and eventually it sinks in and people realize that they have a problem that they need to tackle.
[01:10:33] David Evans: Thank you very much. Okay, we're a little bit overtime. Thanks everybody for your patience. I just want to remind you of what we've seen today, which are enormous returns to investments in making sure children master these skills or get even better at these skills in these early grades. They're enormous. There's more work to come on this. Lee alluded to the Center for Global Development working with partners, Open Philanthropy and others to follow up on other randomized controlled trials where children have now reached adulthood and are in the job market. I know others are working in this space as well. At the same time, we need better interventions with bigger impacts. We need better implementation, more careful design after listening to people on the ground and really understanding what the constraints are. So there's still a lot to do in this space. Most importantly, I want to thank the people who made this event possible. So here at CGD, Mahima Gunapooti, Christelle Santis-Miller, Janet Hoder, and Shannon Hutchins, and then of course our excellent panelists, Jishnu Das, Lee Crawfurd, Jen Swift-Morgan, and Jo Bourne. Please give everybody a round of applause. Thank you very much for coming.
