A Report of the Center for Global Development Study Group on Measuring Learning Outcomes
The Center for Global Development is grateful for contributions from the UK Department for International Development and the William and Flora Hewlett Foundation in support of this work.
Copyright 2013. Center for Global Development. Some rights reserved under the Creative Commons Attribution-NonCommercial 3.0 Unported License.
The Center for Global Development (CGD) convened the CGD Study Group on Measuring Learning Outcomes to advise a staff team drafting a short report that would perform three functions. First, the report would lay out the development rationale behind learning metrics. Second, it would derive implications for assessment regimes that could have the largest impact on education outcomes in developing countries. Third, it would recommend policies and approaches that could support these assessment regimes.
The study group met in Washington DC on June 7, 2012, for initial consultations with the authors. Over the following months, the study group provided individual support and feedback. It reviewed an initial draft of the report between August 28 and September 14, 2012, and a subsequent draft from November 1 to November 20, 2012.
A CGD study group provides inputs, guidance, and reactions on drafts of major CGD staff reports. The final study group report reflects the opinions of the authors; it does not necessarily reflect the collective or individual opinions of any study group members or their employers.
Barbara Bruns (World Bank)
Chloe O’Gara (Hewlett Foundation)
Deon Filmer (World Bank)
Eric Hanushek (Stanford University)
Jeff Puryear (Inter-American Dialogue)
Justin Sandefur (Center for Global Development)
Lant Pritchett (Harvard University and Center for Global Development)
Ludger Woessmann (University of Munich)
Luis Crouch (Global Partnership for Education)
Lynn Murphy (Hewlett Foundation)
Miguel Szekely (Institute for Innovation in Education, Tecnológico de Monterrey)
Nancy Birdsall (Center for Global Development)
Rakesh Rajani (Twaweza)
Rebecca Winthrop (Brookings Institution)
Rukmini Banerji (Pratham and ASER Centre)
Santiago Cueto (GRADE, Peru)
Smita Singh (Center for Global Development Board of Directors)
Sylvia Schmelkes (Universidad Iberoamericana, Mexico)
For the last ten years, the major focus of the global education community has been on getting children into school. And that effort has been a success: most of the world’s children live in countries on track to meet the Millennium Development Goal of universal primary completion by 2015.
But behind that progress is a problem—one that grows with each additional child that walks through the classroom door. Some children in those classes are learning nothing. Many more are learning a small fraction of the syllabus. They complete primary school unable to read a paragraph, or do simple addition, or tell the time. They are hopelessly ill-equipped for secondary education or almost any formal employment. The crisis of learning is both deep and widespread. It is a crisis for children, too many of whom leave school believing they are failures. And it is a crisis for their communities and countries, because economic analysis suggests it is what workers know—not their time in school—that makes them more productive and their economies more prosperous.
School systems in many developing countries are chronically underfunded. Many are filled with undernourished children of illiterate parents and staffed by poorly trained teachers who lack mastery of the subjects they teach. But the crisis of learning is about far more than funding, training, or the socioeconomic status of students. It is about education ministries that have measured success on inputs such as budget, student numbers, teachers, and schools rather than outcomes such as students who can read. It is about parents and parliamentarians who demand schooling and simply assume learning will result.
Fixing the learning crisis will take systemic reform stretching beyond the education sector. It will take teachers, headmasters, and education officials with the mandate to focus on learning. And it will take those officials being held accountable for learning outcomes by informed stakeholders including parents, parliamentarians, and employers.
Assessment regimes are a central part of this reform effort. They can provide evidence on the scale of the learning crisis as a lever for reform. They can track progress on improvements and provide the evidence base for what works. They empower parents to demand better outcomes—or move their kids to where they can find them.
This report of the CGD Study Group on Measuring Learning Outcomes suggests that implementing assessment regimes that allow stakeholders to compare across schools, districts, and countries would cost a small fraction of current education budgets. It calls for international support for national assessment efforts including financial and technical assistance.
The Study Group includes funders, education experts, former education ministers, and pioneers of independent learning assessment in the developing world. The group’s work builds on CGD’s sustained engagement in education, including research on a Millennium Learning Goal, aid effectiveness in the sector, and a focus on educational inclusion for girls, in particular those from minority groups.
The report is being released as discussions about what should replace the Millennium Development Goals are highlighting education quality as an issue, and pressure is building from civil-society groups, employers, and parents worldwide to ensure schooling translates into learning. We hope the report adds urgency and direction to that pressure.
Center for Global Development
In the last decade, many countries have made heartening progress on enrolling children in schools. To be sure, millions remain out of school, especially children from disadvantaged groups. Enrollment in pre-primary and secondary institutions remains a privilege of the minority in many countries. Nonetheless, every 7 of 10 people live in a country that is on track to meet the Millennium Development Goal of universal primary education by 2015. And gross secondary enrollment in developing countries climbed from 50 percent to 66 percent between 1995 and 2010.
We want children to be in school because we think they will learn valuable lessons—from basic literacy and numeracy, to manners, workplace skills, and good citizenship. But learning requires more than a student’s physical presence in a classroom. Hunger and exhaustion can interfere with learning, and a lack of equipment, knowledge, or motivation can prevent effective teaching.
The development and education communities have long recognized the problem of failure to learn (see box). Even so, developing-country governments and donors continue to focus first on getting children to school, and second, on inputs such as staff numbers and materials. Rarely do the development and education communities focus on whether students are actually learning. Evidence is mounting, however, that learning is stagnating in developing-country school systems, as demonstrated by low levels of learning and small increments in learning across grades.
Learning stagnation is linked to educational institutions in developing countries: from teacher incentives, through syllabus design, to the structure of the education sector as a whole. Conditions outside of school, including household characteristics, remain by far the best single predictor of student test scores, but this fact itself signals a widespread failure of the school system to level outcomes. In fact, learning stagnation is primarily a problem of political economy, where senior education officials and staff focus on inputs rather than the output of learning.
The first step in systemic reform is to understand the problem: many students are learning close to nothing in school. And assessment can help shift the focus of citizens, businesses, parliamentarians, governments, and development agencies to maximizing outcomes rather than inputs.
Box: Learning as the goal for education
Article IV of the World Declaration on Education—an outcome of the 1990 World Conference on Education for All—recognizes that actual learning should be the focus of basic education. It states:
Whether or not expanded educational opportunities will translate into meaningful development—for an individual or for society—depends ultimately on whether people actually learn as a result of those opportunities, i.e., whether they incorporate useful knowledge, reasoning ability, skills, and values. The focus of basic education must, therefore, be on actual learning acquisition and outcome, rather than exclusively upon enrolment, continued participation in organized programmes and completion of certification requirements.
According to the United Nations Declaration of the Rights of the Child, education shall “promote [a child’s] general culture and enable him, on a basis of equal opportunity, to develop his abilities, his individual judgment, and his sense of moral and social responsibility, and to become a useful member of society.”
This report discusses evidence that learning—not schooling—drives broader development outcomes. It presents measures of the schooling-learning gap, notes the systemic issues underlying the gap, and outlines both the role of assessment and the limited reach of testing regimes today.
To ensure systemic change, engagement in reform must extend beyond education ministries and teachers’ unions to a broad range of stakeholders, including parents, employers, parliamentarians, and taxpayers. This report urges the education and development communities to greatly expand national and international assessments. Expanded assessments can not only measure progress, but can motivate broad engagement for reform to improve educational outcomes in the developing world.
Worldwide, education is a sizable economic sector. Public education expenditure accounts for 5.4 percent of gross domestic product in Brazil and the United Kingdom, and 5.5 percent in Ghana and the United States. In Kenya and Uganda, education accounts for around 15 percent of government expenditure. Even some of the poorest people worldwide spend a considerable proportion of their private income on education. The poorest urban households in Pakistan, which on average spend less than $1 per person per day in total, spend more than 6 percent of that on education. In Indonesia, the proportion reaches nearly 9 percent.
Parents and governments expect a return on this investment. Parents send their children to school in expectation of improved employment opportunities, income, status, and quality of life. Ministers and parliaments hope that expanded education will lead to economic growth, improved health outcomes, and nation building.
In some ways, schooling appears to meet these expectations. Micro studies often report high returns to years of schooling in developing countries—7 percent in Ghana, for example. Children with mothers who went to school are more likely both to survive childhood and to attend school themselves.
But at the same time, evidence suggests that expanded educational opportunities do not translate into improved economic performance. At the country level, the average Kenyan over the age of 15 in 2010 had more years of schooling than the average French person in 1985. Sadly, Kenya’s 2010 GDP per capita was only 7 percent of France’s GDP per capita in 1985. Kenya represents a trend: massively increased enrollments even where incomes have stagnated in recent decades. More broadly, the link between schooling and economic growth in cross-country analysis is fragile at best.
Drawing on Lant Pritchett’s analysis, figure 1 suggests three reasons why higher average wages for workers who have spent longer in school are not always associated with higher economic growth in countries where school enrollments have risen rapidly. First, educated workers may move into socially unproductive jobs. Second, there may be limited demand for educated workers. Third, education may produce little human capital because students acquire little useful knowledge.
Figure 1. Explaining the gap between higher wages for educated workers and the limited growth impact of education
Source: Authors’s construction based on Lant Pritchett, “Where Has All the Education Gone?” World Bank Economic Review 26 (3).
International evidence suggests that actual learning is more closely related to subsequent economic performance than school attendance. Eric Hanushek and Ludger Woessmann of Stanford University and the University of Munich suggest that average test scores on international assessments and initial gross domestic product per capita between them are associated with as much as 73 percent of the variation in economic growth outcomes across countries between 1960 and 2000. This association is not proof of a causal relationship, but it suggests a potential economic payoff to quality education. What explains the evidence of a learning payoff on the one hand and the limited evidence of a schooling payoff on the other hand? The schooling-learning gap.
Recent direct measures of the schooling-learning gap suggest that official literacy statistics significantly underestimate the scale of the problem. In several developing countries, many of the students who were enrolled in six full years of primary education were unable to answer questions about a simple paragraph or solve simple math problems. This suggests a dismal rate of return on years of school enrollment.
In India, national survey evidence reveals that only about one-third of children in grade 5 can perform long division, and one-third cannot perform two-digit subtraction. Nearly one-half of grade 5 students cannot read a grade 2 text and one in five cannot follow a grade 1 text. Sixty percent of Indian children enrolled in grade 8 cannot use a ruler to measure a pencil. Only 27 percent of Indian children who complete primary school can read a simple passage, perform division, tell time, and handle money, although students should master each of these skills by the end of the second year of school. These statistics compare starkly with the official 81 percent youth literacy rate reported by the United Nations Education, Scientific and Cultural Organization (UNESCO).
Similar findings have emerged elsewhere. Data from both Early Grade Reading Assessments and the Southern and Eastern Africa Consortium for Monitoring Educational Quality point to serious quality gaps across the region (figures 2 and 3). Uwezo surveys show that in Tanzania and Uganda, less than half of all 10- to 16-year-olds possess even basic literacy or numeracy skills.
Figure 2. Percentage of grade 6 pupils reaching reading proficiency levels—Southern and Eastern Africa Consortium for Monitoring Educational Quality
Source: Sacmeq 2010 data available at www.sacmeq.org/reports.htm .
Figure 3. Percentage of students who cannot read a single world of a one-paragraph story after two years in school, based on Early Grade Reading Assessments data
Source: A. Gove and P. Cvelich, Early Reading: Igniting Education for All. A Report by the Early Grade Learning Community of Practice (Research Triangle Park NC: Research Triangle Institute, 2010).
A flat learning trajectory through successive school grades is reflected in low test scores among older students. Using several sources of recent data from India, the Center for Global Development’s Lant Pritchett examined the number of repeat questions that fourth, sixth, and eighth graders answered correctly. For language, the percentage climbs from 51 to 57 percent between fourth and eighth grades. For math, it climbs from 36 to 53 percent. This suggests that it would take 32 years of schooling for 90 percent of all students to correctly answer a language question that more than half of all fourth graders already correctly answered. India is hardly unique in its flat learning trajectories. Studies of the impact of education on learning in Bangladesh in the 1990s found that three additional years of schooling had no appreciable impact on learning achievement.
At higher levels, results are perhaps even more worrying. Internationally comparable mathematics tests under the Trends in International Mathematics and Science Study (TIMSS) suggest that the average eighth grader in Ghana has a test score that would place her in the bottom 0.2 percent of US students. Even in considerably richer developing countries, the learning gap is large: the average Chilean student would be in the bottom 6.4 percent of US students, based on TIMSS scores. Figure 4 presents average TIMSS scores for a range of developing countries expressed as a percentile rank on Danish students’ TIMSS performance. Performance for disadvantaged groups within developing countries is even worse than these averages suggest. Deon Filmer of the World Bank reports that in Chile, the gap between students in the richest and poorest quintiles on Program for International Student Assessment (PISA) test scores is greater than 100 points, or one standard deviation.
Figure 4. Developing-country average Trends in International Mathematics and Science Study (TIMSS) score as a percentile rank on Denmark’s TIMSS score distribution
Source: Lant Pritchett, The Rebirth of Education: Schooling Ain’t Learning (Washington DC: Center for Global Development, forthcoming).
In an environment of limited learning opportunities (especially for disadvantaged students), continued schooling can be a net negative to household wellbeing. Deon Filmer and Norbert Schady’s analysis of a Cambodian scholarship program found that it increased school enrollment and attendance by 25 percent, but 18 months after the scholarships were awarded, children did no better on math and vocabulary tests than they would have without the program. Absent the scholarship, then, many parents and children made what appears to have been a rational choice to not go to school at all.
Until school systems can guarantee that students will learn while sitting in class, it may even be counterproductive to encourage longer periods of universal education. In fact, expanded enrollments can actually harm overall learning outcomes if quality cannot be broadly maintained. While grade 8 enrollment in India increased from 82 to 87 percent from 2006 to 2011, ASER tests suggests the fraction of grade 8 children who could do division fell from 70 percent to 57 percent. This suggests that fewer school-age children actually learned division, despite climbing enrollments.
The disconnect between schooling and learning highlights the wisdom of the 1990 World Declaration on Education for All and the Declaration of the Rights of the Child. Both declarations emphasize free and compulsory elementary education, not merely schooling. The right to education expressed in the Declaration of the Rights of the Child is more than the right to sit under a school roof for six or eight hours a day. It is a right to learn something while sitting there.
Why the gap between education inputs and learning outcomes? There are numerous causes and unknowns linked to the many prerequisites for learning. Nonschool factors play an important role, but even within schooling systems, several factors must coalesce for students to learn. Schools are needed, as are teaching materials, utility services, and other inputs. Students must be present, motivated, and able to learn. Teachers must be present, motivated, and able to instruct. That many of these requisites are often missing is a function of systemic issues in education delivery more than a simple lack of resources.
Lant Pritchett concludes that even considerable and costly improvements in inputs can often only marginally affect learning outcomes in developing countries. He examines the impact of physical school infrastructure; teacher-pupil ratios; teacher qualifications and training; and learning inputs such as chalk, pencils, paper, workbooks, and computers. Based on evidence from four studies of learning impact in the developing world, he calculates the impact on testing of increasing all of these input variables to the maximum plausible. The results are disappointing, ranging from 10 to 20 percent of a standard deviation of student test scores compared with a learning gap at least five times as large. The learning deficit, therefore, is perhaps one-fifth a lack of resources at the classroom level and four-fifths other issues. Evidence from India appears to confirm the relatively slight impact of increased resources on learning outcomes. Between 2007 and 2011, India increased expenditure on elementary education by 80 percent, but average learning outcomes reported by ASER surveys have slightly declined.
Inputs do have some effect on outcomes. Paul Glewwe and colleagues reviewed 9,000 studies of educational intervention impacts on learning outcomes in developing countries. They conclude that a number of basics do matter. Improved roofs, walls, and floors; school furnishings such as desks, tables, and chairs; and a school library all help students learn. However, inputs only lead to notably better outcomes if the broader environment is conducive to using those inputs effectively. In many countries, it appears that the binding constraint on learning is not resources, but other factors related to the broader environment.
For example, according to Glewwe and colleagues “the most consistent results” in improving learning outcomes “reflect having teachers with greater knowledge of the subjects they teach, having a longer school day, and providing tutoring.... It makes a difference if the teacher shows up for work.” Teacher absenteeism a significant problem; on an average school day, 11 percent of teachers are absent in Peru, 16 percent are absent in Bangladesh, and 27 percent are absent in Uganda. Even when they are present, teachers may make limited efforts to create a friendly learning environment. ASER’s observation of rural education practices in 1,075 classrooms across five Indian states reveals that in only about a quarter of classrooms was a student witnessed asking a question. Other child-friendly practices—joking with students, displaying student work, encouraging students to work in small groups—were even less common.
Direct methods to improve teacher attendance and effort have improved test scores with mixed results, as has the use of contract teachers. The importance of systemic issues is demonstrated by the fact that interventions that successfully improve teacher performance in one system can fail when applied elsewhere. For example, Paul Atherton and Geeta Kingdon show that students taught by contract teachers in public schools in Uttar Pradesh learned about twice as much per year as those taught by civil service teachers. At the same time, an attempt to scale contract teachers in Kenya revealed that contract teachers only influenced test scores when hired by a nongovernmental organization, not the Kenyan government. A randomized study in the Indian state of Andhra Pradesh suggests that bonuses for teachers based on exam results can improve outcomes. In Kenya, however, teacher performance incentives related to test scores increased student learning only in the short run.
Whether a school-level fix for the learning gap is effective depends on the broader environment for learning and the incentives offered to principals and government ministers. This may be why private schools (which respond to very different incentives) sometimes considerably outperform public schools in the developing world even if spending less per student. The Center for Global Development’s Justin Sandefur and colleagues found that in Kenya, private school students score an average of 290 on the Kenya Certificate of Primary Education exam, compared with a national average score of 247. The gap remains evident even when accounting for the fact that more able students are more likely to attend private schools. The difference in scores is considerably larger than the impact of any tested intervention on raising performance within public schools.
Publicly financed school choice has also made a significant difference in test scores in some settings. An analysis of the random distribution of vouchers covering half of the costs of private secondary schooling across all municipalities in Colombia found that voucher recipients were 10 percent more likely to have completed eighth grade and scored significantly higher on standardized tests.
This is not to argue that the solution to the schooling-learning gap is mass privatization of education or widespread voucher schemes. Free primary education is a right enshrined in the Universal Declaration of Human Rights, and there are cases where it appears that private or charter schools do not provide a dramatically better education once household characteristics are taken into account. There is insufficient evidence suggesting mass voucher programs would work in education sectors across different institutional environments and levels of development. Certainly the experience of school choice in Chile, where the average student PISA score has improved but remains at the 17.8th percentile for Denmark, suggests that choice is not enough on its own to significantly improve results.
Instead, experience with private education suggests that empowering motivated parents with information, choice, and responsibility can stimulate change when matched with school managers with the flexibility and incentive to respond to engaged and informed stakeholders. Achieving this match requires systemic reform that would enhance the long chains of accountability from education ministries to voters, and the shorter chains of accountability from headmasters and district education officials to parents.
Accountability reform would make teachers and headmasters more directly responsible for the quality of education in their classrooms, and would give them the freedom and tools to teach in ways that maximize learning. To achieve accountability, governments require the tools to monitor and reward quality and value added in teaching, and to intervene to fix failing schools. Furthermore, accountability would equip parents with knowledge of learning outcomes in schools and the tools to secure better results for their children. Accountability requires that parents and government know how much students are actually learning in school and can use that knowledge to change outcomes.
It is perhaps not surprising that there is a gap between schooling and learning, and that education reform is difficult. Schools and education systems are about a lot more than learning. For students, schools are also about signaling innate intelligence, status, and social networks. For parents, they are also a form of daycare. For teachers, they are a stable source of income. For governments, schools are also about socialization, employment, and rent generation. A complex story of political economy lies behind the schooling-learning gap.
Given the complex political economy of systemic reform, and the considerable diversity in existing educational systems around the world, solutions to learning stagnation will vary immensely across countries. As a starting point, however, countries should focus on the right problem—learning—so they can design effective solutions.
The role of classroom assessments as a tool for enhancing teacher performance is uncontroversial. Black and Wiliam note in their 1998 synthesis of over 250 studies on classroom assessment that “[t]he gains in achievement appear to be quite considerable, and ... amongst the largest ever reported for educational interventions.” Examinations also play an important role in selecting and certifying students both for employers and for higher levels of education.
But learning stagnation and its systemic underpinning suggest that assessment’s most important role may be as a tool for leveraging reform. Barbara Bruns and colleagues suggest three accountability channels through which information such as assessment could affect learning outcomes: choice (evidence for parents about the quality of education across schools), participation (a motivator for action), and voice (a tool to lobby for reform). When combined with a policy framework that allows for choice, engagement, and reform, testing can have a significant impact. Systemically, assessment can help stakeholders understand the scale of the schooling-learning gap. With this understanding, they can push for change, recalibrate expectations and syllabi to reasonable expectations of performance, allow targets and incentives for improvement, and facilitate experimentation.
In particular, assessment can move dialogue towards the right issue, from the well-understood act of going to school to the less widely appreciated question of learning while there. In the last 20 years, the push for universal access and enrollment has led to a widespread understanding of what going to school entails. Even illiterate rural mothers in India know what this means. The growth of enrollment is testament to the fact that governments knew how to improve access and parents knew how to improve enrollment.
In contrast, there is no universal understanding of “learning.” Assessments play a vital role in demonstrating to parents and governments the basic learning they should expect (in that tests should cover the skills of reading, writing, and arithmetic). Assessments also focus parental and political attention on the right problem (learning stagnation) rather than on education sector inputs and measures of enrollment that are at best only a partial solution. Assessments are a powerful tool that stakeholders outside the education ministry can use to improve outcomes. And given the systemic nature of the schooling-learning gap, outside actors will be vital for reform.
Absent assessment information, parents may be satisfied with very poor educational services. Figure 5 compares the percentage of adults in three East African countries who report that the government is addressing educational needs “fairly well” or “very well” with the proportion of children in standard 4 who passed standard 2 numeracy and English literacy tests. In all three countries in 2008, two-thirds or more of adults were broadly satisfied with the government’s performance in education. Yet the Uwezo results three years later suggest that less than half of standard 4 children in Kenya, less than a third in Tanzania, and less than a fifth in Uganda passed standard 2 tests in literacy and numeracy. The considerable majority of children in these three countries are not learning what they should be. Yet the considerable majority of adults profess satisfaction with the education system, perhaps because they do not fully grasp their children’s actual level of learning. Hopefully, by demonstrating the schooling-learning gap, public assessments such as Uwezo can foster demand for higher quality education.
Figure 5: Satisfaction and results in Education (East Africa)
Source: Uwezo, “Are Our Children Learning? Numeracy and Literacy Across East Africa,” mimeograph, Dar Es Salaam (Uwezo, 2011) and Afrobarometer surveys available at www.afrobarometer.org.
Assessments might also help recalibrate syllabi over time with realistic expectations of what the median student (or even the advanced student) in the average classroom actually knows. A national syllabus that demands teachers complete lessons in algebra or make students read long history texts places teachers in an impossible situation if the majority of their students still cannot do basic arithmetic or read a simple paragraph. Assessments can also focus attention on learning gaps within countries—between girls and boys, rich and poor, and minority language groups and others. Reformulated syllabi can ensure that all students benefit from completing basic education rather than the lucky few who can keep up with over-ambitious curricula.
Repeated assessment allows governments and the international community to set goals for improved learning outcomes to spur reform efforts. Although the worldwide rates of improvement in test scores suggest these goals should be modest and long term, assessments can help hold leaders to account. In the right institutional setting, they can also form the basis for incentive payments; schools, districts, or countries that meet goals might receive additional resources from national governments or donor agencies. In particular, specific goals and incentives might encourage progress in schools or districts furthest behind.
Finally, assessments allow for wider experimentation on what works in education, inside and outside the classroom, applicable to specific country circumstances. This experimentation is clearly required given the limited progress we have seen in improving outcomes to date.
There is some evidence that testing and assessment as tools to leverage systemic improvement can boost learning. Cross-country analysis suggests that students in school systems that have an external exit exam combined with local autonomy perform better on internationally comparable tests. However, performance is actually worse if there is autonomy without testing, which suggests that only informed demand from parents can make a real difference.
At the country level, Brazil has simultaneously expanded schooling coverage and learning over the past 15 years. The country’s national assessment, benchmarked to PISA, has shown considerable improvement since 2000. Gains are particularly impressive since 2005 when the national assessment began testing every student, and the government, media, and civil society groups began actively publicizing the results. The Basic Education Development Index, introduced in 2005, allows parents to compare school achievement and promotion on a 10-point scale aligned with the country’s PISA scores. Brazil’s average PISA scores in reading improved from 396 in 2000 to 412 in 2009. In mathematics, scores improved from 356 in 2003 to 386 in 2009. In science, Brazil’s scores improved from 390 in 2006 to 405 in 2009.
Alongside reforms in school funding, educational requirements, and teacher pay and bonuses, Brazil redesigned its curricula, promoted longer school days, and mandated 11 years of schooling. These reforms have driven greater accountability, especially in some of the poorest states. Ceara, in the Northeast, has implemented an incentive program that provides additional resources to the schools that make the most progress on the Basic Education Development Index. Ceara improved from one of Brazil’s worst performing states to 14th out of 27 states in the 2009 Index.
Despite continuing problems with unequal outcomes, Chile has seen similarly impressive gains; along with Brazil and Latvia, Chile improved student learning at the fastest rate worldwide between 1995 and 2011. The country has combined a robust system of publicly funded, privately managed voucher schools and teacher assessment matched by rapidly rising salaries, with its own national all-student assessment testing regime.
Although assessment can play a positive role in improved outcomes, it is far from a foolproof or stand-alone remedy. Mexico has had standardized national all-student testing since 2006, with considerable data published on the results. Nevertheless, opposition has grown and returns in terms of improved policies and student outcomes are limited, in part because results were not accompanied by guidance or additional school resources. Pratham’s ASER has documented declining scores since it was introduced. Similarly, the vast majority of participating countries have progressed slowly over time in PISA and TIMSS scores.
Assessments can only improve outcomes as part of a broader effort. Chile and Brazil, for example, have highlighted education as a national issue. Even so, testing and school reforms are likely to only narrow the gap between learning outcomes around the world. A range of factors that are completely beyond the control of education ministries—including family circumstances and home language—also help determine learning.
The many roles of assessment, and the need for pressure at the national, district, and school level for improved performance, point to the importance of an assessment system –repeated, independently verified national assessments of different grade levels. Assessments should be comparable across schools, districts, and time. They should also be comparable to internationally comparative tests. International assessments like PISA and TIMSS can play an important role in motivating national politicians and major employers to advocate for reform, but assessments that allow comparison at lower levels may be the key to engaging and informing parents and state and local representatives.
At a minimum, a representative sample of children in both public and private schools should be assessed every year. Preferably, all children should be tested regularly. And it should be possible for parents and stakeholders everywhere to compare school performance with results that allow for evaluation of value added.
Assessments are affordable. Reviews of the cost of testing and assessment programs in the United States and Latin America suggest that costs in no case amount to more than 0.3 percent of the national education budget. Across Latin America, the cost for all-student school testing programs ran from $2 to $25 per student in 2007. At $4.4 million, Chile’s grade 8 tests were the most expensive in the region. This suggests that the cost effectiveness of assessment in terms of improved learning, even absent systemic change, is many multiples of the returns to increased teacher pay or expenditure on inputs. If assessment plays a role in fostering systemic change as well, the ratio of benefits to costs of an assessment system would be multiples higher again.
The scarcest resource in many developing countries may be the people needed to manage the assessment process, not the funds to pay for it. In those circumstances, tests can be outsourced to private or civil society providers approved by government-appointed public/private committees.
Despite low costs and potentially high returns, few developing countries outside Latin America even approach a full assessment system. The ASER exercise in India and Pakistan, Uwezo assessments in East Africa, and emerging work in West Africa are welcome initiatives to help fill the assessment gap with national profiles and comparative data across schools and districts. The Early Grade Reading Assessment used in 44 countries is also a promising development (see figure 6). But testing to create school, regional, or national learning profiles remains the exception rather than the rule in many developing countries. Fewer than half of the countries in sub-Saharan Africa conducted a single national learning assessment between 1995 and 2008.
Figure 6. Number of developing countries covered by existing international assessment regimes
Source: J. van der Gaag and A. Adams, Where is the Learning? Measuring Schooling Efforts in Developing Countries (Washington DC: The Brookings Institution, 2010).
The internationally comparable statistics with the widest coverage may provide little insight into whether students are actually learning. Widely cited literacy statistics are based on years of schooling, not actual tests of learning. Some middle-income countries have progressed significantly in enrolling students into higher level international assessments. Nonetheless, only 28 developing economies administered the last round of the TIMSS grade 8 mathematics and science assessment, which is the most comprehensive international assessment in terms of country coverage at higher levels. (The next round of both PISA and TIMSS will cover more developing countries.)
Clearly there is a significant gap in measuring learning outcomes in developing countries. There is enough assessment data to suggest that learning stagnation is widespread in the developing world. But, except in a minority of developing countries, the level of assessment is too limited to allow for a detailed analysis of the nature and causes of that stagnation. Assessment data is insufficient to lever reform and experimentation to improve overall learning outcomes. Given the small cost and potential impact of improved assessment systems, this low level of assessment is a considerable failing of the global education community.
There are alternate uses for the resources governments spend on schooling—life-saving vaccination programs that cost tens of dollars per person saved, for example. If schooling does not deliver learning, it is hard to justify all the money that is spent on schools. The schooling-learning gap is, therefore, an issue not just for education ministries, teachers, parents, and students, but for finance ministries, parliamentarians, taxpayers, and aid agencies.
The immediate role for expanded assessment is to sound the alarm about a widespread stagnation of learning at very low levels despite a dramatic increase in school enrollments and education expenditures. It is clear that the main cause of the schooling-learning gap is not lack of classroom inputs, but the political economy of education. However, it is not clear that we know the answer to the learning challenge, or that there is one answer. The correct or practical responses to the learning stagnation will vary by country, and doubtless within countries. An expanded national and international regime of learning assessments will be invaluable in the search for those answers and in creating pressure for their implementation.
As part of a broader focus on improving learning outcomes—a focus that highlights learning for all—government, civil society, donors, and the UN system all play a unique role.
Governments, particularly in developing countries, should prioritize the development and rollout of assessments targeted to measure learning performance of cohorts of children from early grades through high school.
The nature of learning stagnation and its roots in a complex and country-specific dynamic of political economy suggests that the primary force for successful change will be domestic and grassroots, engaging parents and other stakeholders in the struggle for reform. This suggests a vital role for civil society.
The role for outsiders—and aid agencies in particular—will be secondary. Perhaps the greatest contribution the international community could make is Hippocratic: “doing no harm” by ending support for broken systems that focus on education inputs to the exclusion of learning outcomes.
Multilateral bodies and the UN system have an important role in setting global norms and standards.
Finally, efforts are underway to frame a concrete learning goal for all children, endorsed by UNESCO and potentially added to the new round of goals that will frame the development agenda after the 2015 deadline for the Millennium Development Goals. It is clear that an overall focus on learning rather than years of schooling is the correct approach for any such goal. A learning goal might spur improved data on educational outcomes, from basic literacy and numeracy to more advanced competencies related to successful participation in an increasingly globalized economy.
The post-2015 development agenda should prioritize global efforts to measure student learning comprehensively, in all countries and across key grades, skills, and competencies. Countries should carry out their measurement efforts systematicallyover time, so they can measure and analyze progress in a robust way. And measurement outcomes should be globally relevant, enabling even the smallest and poorest countries to map their progress against their regional neighbors and global leaders. An increased focus on worldwide assessment is key to ensuring that children everywhere enjoy their human right to education—a responsibility we all share.
Andrabi, T., J. Das, A. Ijaz Khwaja, and T. Zajonc. 2011. “Do Value-Added Estimates Add Value? Accounting for Learning Dynamics.” American Economic Journal: Applied Economics 3 (3): 29-54.
ASER Centre. 2010. Inside Primary Schools: Teaching and Learning in Rural India . New Delhi: ASER Centre.
Atherton, P. and G. Kingdon. 2010. “The relative effectiveness and costs of contract and regular teachers in India.” CSAE Working Paper Series 2010-15, Oxford United Kingdom: Centre for the Study of African Economies, Oxford University.
Banerjee, A. and E. Duflo. 2011. Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty . New York: Public Affairs.
Banerjee, A., R. Banerji, E. Duflo, R. Glennerster, and S. Khemani. 2010. “Pitfalls of Participatory Programs: Evidence from a Randomized Evaluation in Education in India.” American Economic Journal: Economic Policy , 2(1): 1-30.
Banerji, R. 2012. “Education Must Be More Goal-Oriented.” Hindustan Times , May 11.
Barrera-Osorio, F. 2007. “The impact of private provision of public education: empirical evidence from Bogota’s concession schools.” World Bank Policy Research Working Paper No. 4121, Washington DC: World Bank.
Beatty, A. and L. Pritchett. 2012. “The Negative Consequences of Overambitious Curricula in Developing Countries.” Center for Global Development Working Paper 293, Washington DC: Center for Global Development.
Birdsall, N. and R. Perakis. 2012. “Cash on Delivery Aid: Implementation of a Pilot in Ethiopia.” Center for Global Development mimeograph, Washington DC: Center for Global Development.
Birdsall, N. and W. Savedoff. 2010. Cash on Delivery: A New Approach to Foreign Aid . Washington DC: Center for Global Development.
Bold, T., M. Kimenyi, G. Mwabu, Alice Ng’ang’a, and Justin Sandefur. 2013. “Scaling Up What Works: Experimental Evidence on External Validity in Kenyan Education.” Center for Global Development Working Paper 321, Washington DC: Center for Global Development.
Bold, T., M. Kimenyi, G. Mwabu, and J. Sandefur . 2011. “The High Return to Private Schooling in a Low-Income Country.” Center for Global Development Working Paper 279, Washington DC: Center for Global Development.
Bruns, B., D. Filmer, and H. Patrinos. 2011. Making Schools Work: New Evidence on Accountability Reforms . Washington DC: World Bank.
Chaudhury, N., J. Hammer, M. Kremer, K. Muralidharan, and F. Rogers. 2006. “Missing in Action: Teacher and Health Worker Absence in Developing Countries.” Journal of Economic Perspectives 20(1): 91–116.
Clarke, M. 2012. What Matters Most for Student Assessment Systems . Washington DC: World Bank.
Das, J., P. Pandey, and T. Zajonc. 2006. “Learning Levels and Gaps in Pakistan.” World Bank Policy Research Working Paper No. 4067, Washington DC: World Bank.
Di Gropello, E. 2004. “Education Decentralization and Accountability Relationships in La t in America.” World Bank Policy Research Working Paper No. 3453, Washington DC: World Bank.
Duflo, E, R. Hanna, and S. Ryan. 2008. “Monitoring Works: Getting Teachers to Come to School.” Center for Ec o nomic Policy Research Discussion Paper No. DP6682, London: Center for Economic Policy Research.
Filmer, D. 2007. “If You Build It, Will They Come? School Availability and School Enrol l ment in 21 Poor Countries.” Journal of Development Studies 43 (5).
Filmer, D., A. Hasan, and L. Pritchett. 2006. “A Millennium Learning Goal: Measuring Real Progress in Education.” Center for Global Development Working Paper 97, Washington DC: Center for Global Development.
Glewwe P., E. Hanushek, S. Humpage, and R. Ravina. 2011. “School Resources and Educational Outcomes in Developing Countries: A Review of the Literature from 1990 to 2010.” NBER Working Papers 17554, Cambridge MA: National Bureau for Economic Research.
Gove, A. and P. Cvelich. 2010. Early Reading: Igniting Education for All. A Report by the Early Grade Learning Community of Practice. Research Triangle Park NC: Research Triangle Institute.
Hanushek, E. and L. Woessmann. 2008. “The Role of Cognitive Skills in Economic Development.” Journal of Economic Literature 46 (3).
Hanushek, E. and L. Woessmann. 2009. “Do Better Schools Lead to More Growth? Cognitive Skills, Economic Outcomes, and Causation.” NBER Working Paper 14633, Cambridge MA: National Bureau for Economic Research.
Hanushek, E. and L. Woessmann. 2010. “The Economics of International Differences in Educational Achievement.” IZA Discussion Paper 4925, Bonn: Institute for the Study of Labor.
Kenny, C. 2011. “Learning About Schools in Development.” Center for Global Development Working Paper 236, Washington DC: Center for Global Development.
Kremer, M. and E. Miguel. 2004. “Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities.” Econometrica 72 (1).
Muralidharan, K. and M. Kremer. 2008. “Public and Private Schools in Rural India.” In School Choice International , edited by P. Peterson and R. Chakrabarti, Boston: MIT Press.
Muralidharan, K. and V. Sundararaman. 2011. “Teacher Performance Pay: Experimental Evidence from India.” Journal of Political Economy 119 (1).
OECD. 2011. “Brazil: Encouraging Lessons from a Large Federal System.” In Lessons from PISA for the United States (OECD).
Pal, S. and G. Kingdon. 2010. “Can Private School Growth Foster Universal Literacy? Panel Evidence from Indian Districts.” IZA Discussion Papers 5274, Bonn: Institute for the Study of Labor (IZA).
Patrinos, H.A., F. Barrera-Osorio, and J. Guáqueta. 2009. The Role and Impact of Public-Private Partnerships in Education . Washington DC: World Bank.
Pratham and ASER. 2011. Annual Status of Education Report . New Delhi: ASER Centre.
Pritchett, L. 2001. “Where Has All the Education Gone?” World Bank Economic Review 26 (3).
Pritchett, L. 2009. “The Policy Irrelevance of the Economics of Education: Is Normative as Positive Just Useless, or Worse?” In What Works in Development: Thinking Big and Thinking Small , edited by J. Cohen and B. Easterly, Washington DC: The Brookings Institution.
Pritchett, L. Forthcoming. The Rebirth of Education: Schooling Ain’t Learning . Washington DC: Center for Global Development.
Psacharopoulos, G. and H. Patrinos. 2002. “Returns to Investment in Education: A Further Update.” World Bank Policy Research Working Paper No. 2881, Washington DC: World Bank.
Puryear, J. 2012. “Latin Countries Outpacing Peers in Education Advances” Latin American Advisor, August 8.
Rosenzweig, M. 2010. “Microeconomic Approaches to Development: Schooling, Learning and Growth.” Journal of Economic Perspectives 24 (3): 81-96.
Sapelli, C. and B. Vial. 2002. “The Performance of Private and Public Schools in the Chilean Voucher System.” Cuadernos de Economía 39(118): 423-454.
Schaffner, J. 2005. “Subjective and Objective Measures of Literacy: Implications for Current Results-Oriented Development Initiatives.” International Journal of Educational Development 25 (6).
Szekely, M. 2011. “Towards Results-Based Social Policy Design and Implementation.” Center for Global Development mimeograph, Washington DC: Center for Global Development.
Terryn, B. Undated. “Measuring Literacy in Developing Countries from an International Perspective.” Montreal Canada: UNESCO Institute for Statistics.
Uwezo. 2011. “Are Our Children Learning? Numeracy and Literacy Across East Africa.” Uwezo mimeograph, Dar Es Salaam: Uwezo.
Van der Gaag, J. and A. Adams. 2010. Where is the Learning? Measuring Schooling Efforts in Developing Countries . Washington DC: The Brookings Institution.
Wadhwa, W. 2009. “Are Private Schools Really Performing Better than Government Schools?” ASER Centre mimeograph, New Delhi: ASER Centre.
Wagner, D. 2005. “Monitoring and Measuring Literacy.” Background paper prepared for The Education for All Global Monitoring Report 2006 . Paris: UNESCO.
Wolff, L. 2007. “The Costs of Student Assessments in Latin America.” PREAL Working Paper 38, Washington DC: Inter-American Dialogue.
Thanks to school building programs over the past 50 years, school completion is no longer limited primarily by access. Indeed, Deon Filmer (2007) estimated that if a school was built right next to every household in 21 low-income countries, it would raise enrollments by an average of only 1 percentage point. In Ghana and Tanzania, 92 percent of 15- to 19-year-olds have completed at least one grade of school. In India, that number is 85 percent (Pritchett fortchoming).
Household conditions certainly contribute to the learning crisis. Data from the ASER Centre in India (2011) suggest that 46 percent—or close to half—of mothers with children currently enrolled in school in rural India have not been to school themselves. Eighty-two percent of children do not have any reading material at home other than textbooks. For a quarter of the children, the language of instruction at school was different than the language spoken at home. Together, these conditions lead to low or no learning support at home. Similarly, testing results in East Africa suggest that a mother’s education was a far more powerful predictor of learning outcomes than a school quality indicator based on class size, pupil-teacher ratios, and pupil-toilet ratios. Household wealth and private versus public schooling were also more important than school quality variables.
Development Initiatives, “Budget 2012: Is East Africa Spending Where It Matters?” June 20, 2012, www.devinit.org/budget-2012-is-east-africa-spending-where-it-matters.
Banerjee and Duflo (2011).
Psachariopoulos and Patrinos (2002); cf. Rosenzweig (2010).
Data is from the Barro-Lee Educational Attainment Dataset at www.barrolee.com .
Hanushek and Woessman (2008, 2009). The 2009 study includes openness and a measure of property rights.
Similarly, the relationship between simply sitting in school and other quality-of-life measures may be weaker than assumed. For example, the relationship between girls’ education and child health outcomes may reflect intrahousehold decision making as much as the impact of what girls learn in school. At the same time, a number of specific learning interventions around issues such as hand washing and open-field defecation do suggest a high payoff to health knowledge transfer (Kenny 2011).
Since 2007, the Pratham organization’s Indian Annual Status of Education Report has collected nationally representative survey data comparable over time on literacy, numeracy, critical thinking, and everyday calculations for schoolchildren across India. The literacy evaluation tests letter and word recognition as well as comprehension of first- and second-grade texts. The numeracy evaluation tests recognition of one- and two-digit numbers and two-digit subtraction and division.
Pritchett (fortchoming), using data from Educational Initiatives India at www.ei-india.com
This measure is the self-reported ability to read and write a simple paragraph.
In Togo, only 60 percent of adults who reached grade 5 could read and write with ease when tested (Terryn undated). Only one-quarter of surveyed 15- to 19-year-olds in Ghana could answer more than half of a set of math questions that involved four one-digit arithmetic questions, where questions and answers both involved only cardinal numbers between 1 and 10 (such as “5 – 2 = ?”) and four two-digit problems (such as “17 X 3 = ?”) (Filmer, Hasan, and Pritchett 2006).
See Uwezo (2012). Note this includes both children who have and have not attended primary school.
This uses ASER (2010) data. See also Wagner (2005). The percentage of children who can read a grade 2 story climbs from 31 to 44 percent in the Indian state of Uttar Pradesh between grade 4 and grade 5. It is worth noting that the fourth and fifth grade curricula are designed with the assumption that students have mastered basic literacy. This suggests that much of the material presented during the school year is incomprehensible to the majority of students who have not yet mastered reading. Again, less than half the students who have been in school through eight grades—up to high school—can perform division. This is a skill that should have been learned in grade 2, according to the curriculum (Pritchett fortchoming). Uwezo’s survey in East Africa suggests that at the most basic level of literacy and numeracy, a full course of primary schooling might deliver for a great majority of students. A representative survey of learning outcomes in Kenya, Tanzania, and Uganda focused on one- and two-digit math questions and comprehension of a simple paragraph in standard 2 tests. In Kenya, 94 percent of standard 7 children passed the standard 2 English test, 95 percent passed the Kiswahili test, and 88 percent passed the numeracy test. In Tanzania, progress was slower from a lower base: 51 percent of standard 7 children passed the English test, 68 percent passed the numeracy test, and 81 percent passed the Kiswahili test. For Tanzania, linear extrapolation of the data suggests that 11 years of schooling would be sufficient to ensure that 90 percent of students passed a standard 2 English test. The results are driven partially by attrition, where weaker students are likely to drop out. But in East Africa, enrollment rates remain around 80 percent for children who are 12 and 13 years old (Uwezo 2011).
Again, the average developing-country student tested by the Program for International Student Assessment (PISA) is below level 2 proficiency (lacking “the essential skills needed to participate effectively and productively in society”), and the average score of developing countries on PISA and TIMSS is typically 100 to 200 points (one to two standard deviations) behind the OECD norm of 500. Amanda Beatty and Lant Pritchett (2012) estimate that at current rates of progress, it would take a century for developing countries to reach current OECD learning levels measured by national PISA scores.
Cited in Pritchett (fortchoming). It is worth noting that the developing countries participating in PISA and TIMSS are not representative of the developing world as a whole; the sample is skewed towards richer countries almost certain to perform better than the average for low- or lower-middle-income countries or fragile states.
Surveys of girls in 33 developing countries who dropped out before completing secondary education indicate why they left school (Pritchett fortchoming). A median of only 2.4 percent of girls across countries dropped out because “school was not accessible.” Only 6.5 percent cited exam failure or “done enough” as their reason for dropping out. Sixteen percent answered that they “did not like school.” Twenty-six percent cited the need to earn money or the cost of fees. Twenty-five percent cited family concerns (pregnancy, the need for help, or child care). Two-thirds of the girls suggested that the changing balance of costs to benefits of school motivated them to drop out. Sixteen percent directly cited the limited benefits of school. But benefits surely play into the calculations of the others as well. For students who are not learning anything in school, earning money or looking after children is a better use of time. And it is certainly not worth paying fees if you are not learning.
Similarly, Miguel and Kremer’s (2004) analysis of a deworming program in Kenya found that it reduced school absenteeism by a quarter, but found no evidence that it improved academic performance (suggesting that reducing absenteeism is no guarantee of better learning outcomes in weak school systems).
Beatty and Pritchett (2012).
Banerji (2012). East African data also indicate that school inputs explain very little of the variation in test scores across the region. Local environmental and school input variables can account for at most 7 percent of the observed variation in literacy and numeracy scores across children. This compares with almost 50 percent of the variation that is accounted for by a full model, which includes information on family background (such as parental education and asset ownership).
Glewwe et al. (2011).
Glewwe et al. (2011).
Chaudhury et al. (2006).
Duflo et al. (2008); Muralidharan and Sundararaman (2011); Glewwe et al. (2003).
Muralidharan and Sundararaman (2011); Atherton and Kingdon (2010).
This is especially surprising given that the contract teachers had less training and formal education on average, and were paid as much as one-third less than their civil service colleagues (Atherton and Kingdon 2010).
Bold et al. (2013).
Muralidharan and Sundararaman (2011).
Teachers learned how to manipulate scores and the program did not increase teacher attendance, the quality of pedagogy, or homework assignment (Glewwe et al. 2011).
Muralidharan and Kremer (2008). The impact of moving from a public to a private school in Pakistan is not only larger than the impact of providing more inputs to the public schooling system, but also larger than the impact of household socioeconomic factors on learning outcomes. The gap in English test scores between government and private schools is 12 times the gap between children from rich and poor families, for example. This is all the more remarkable given that per student, expenditures are half as high in private as in public schools (Das et al. 2006). Markedly better learning outcomes in private schools may also help to explain why around half of all parents in Uttar Pradesh—where the average rural household spends 57 percent of its budget on food—send their children to private school rather than free public schools (Pritchett fortchoming).
Nor is it because private schools spend more money per child. Nearly two-thirds of pupils in the private system pay less in fees than the median per-pupil funding level in government schools (Bold et al. 2011).
See also Pal and Kingdon (2010); Barrera-Osorio (2007); and Sapelli and Vial (2002).
Di Gropello (2004).
See Patrinos et al. (2009) for a longer discussion.
At the same time, there is indirect evidence that fees are a comparatively minor constraint on enrollment even in poor countries. In Kenya, primary school fees were abolished in 2003. Net enrollment climbed from 75 to 81 percent between 1997 and 2006. But two factors suggest the role of fees in suppressing enrollment was limited. First, secondary enrollment over the same period expanded more rapidly—from 17 percent to 21 percent—even though fees remained in place. Second, net enrollment in government primary schools increased by only 0.4 percent. The net enrollment accounted for by private schools climbed from 3.8 to 8.9 percent over the period. In Pakistan, there is little evidence linking changes in household assets or wealth to switching children between public and private schools (Andrabi et al. 2011).
See, for example, Wadhwa (2009) using data for India from ASER.
The literature finds that decentralizing school management to the individual school level improves outcomes because parents are more likely to become involved in the schooling process (Di Gropello 2004).
Quoted in Clarke (2012).
Bruns, Filmer, and Patrinos (2011).
See Banerjee et al. (2010).
Over-ambitious national curricula leave many students struggling at the end of the first year and too far behind to catch up without immediate remedial help (Beatty and Pritchett 2012).
Hanushek and Woessmann (2010).
Bruns and colleagues (2011) note that test publication in Punjab, Pakistan was enough to considerably improve learning outcomes in local schools, but similar exercises in Jaunpur, India and Liberia found little impact from testing and publication alone. Adding interventions such as volunteer instructors in Jaunpur and teacher training in Liberia did have an impact, suggesting that testing can make a significant difference only in conducive environments.
Based on data from Hanushek and Woessmann (2010), the introduction of external exit exams; school autonomy in budget formulation, teacher hiring, starting salaries and course content; two hours of extra instruction each week; assessment-based student promotion; and lesson monitoring by principals and external inspectors together increased PISA test scores across students and countries by 42 points. While that makes these interventions perhaps the most cost-effective reforms in the education sector, it is less than the score difference between children from households with 11-25 books at home and those with 201-500 books at home (Hanushek and Woessmann 2010; Kenny 2011).
Wolff (2007). The most expensive national sample-based test reported was the $4.9 million Peruvian testing that covered four different grades and four subject areas for $70 per student tested. Internationally comparable sample testing ran as high as $117 per student tested. But in no case did the testing regime equal more than 0.33 percent of the total budget for the educational level assessed.
Van der Gaag and Adams (2010).
UNESCO has long collected data on whole adult populations and those between 15 and 24 years old who have the “ability to read and write, with understanding, a short simple sentence about one’s daily life.” Sadly, in very few countries is the data collected by surveys that involve actually testing literacy ability. In the great majority of cases, a person is declared literate on the basis of the number of years they have been in school or on self-reporting. (The interviewer asks “do you have the ability to read and write, with understanding, a short simple sentence about your daily life” and then asks the respondent about other household members.) Both approaches are subject to error. We have already discussed the evidence of the schooling-learning gap. With regard to survey questions, studies in rural Bangladesh found that half of those who said they could write were not recognized as literate by a minimum tested standard. Results from Morocco found that 46 percent of respondents declared themselves literate, but only 33 percent demonstrated basic competence (Terryn, undated; Schaffner, 2005).
Furthermore, many existing cross-country comparable literacy measures are designed with developed countries in mind and are likely to be too insensitive at the lower end of the proficiency scale for global use. Indeed, when used in developing countries, measurement instruments that were developed for industrialized countries find a large number test-takers score zero, because of test difficulty or lack of local relevance.
See, for example, the International Associations for the Evaluation of Educational Achievement’s civics education assessment at www.iea.nl/cived.html .
Birdsall and Savedoff (2010).
Birdsall and Perakis (2012).
Twaweza, “Three Experiments to Improve Learning,” June 26, 2012, http://twaweza.org/go/three-experiments-to-improve-learning .