With rigorous economic research and practical policy solutions, we focus on the issues and institutions that are critical to global development. Explore our core themes and topics to learn more about our work.
In timely and incisive analysis, our experts parse the latest development news and devise practical solutions to new and emerging challenges. Our events convene the top thinkers and doers in global development.
Each year billions of dollars are spent on development programs with relatively few rigorous studies of whether they actually work. In 2004, CGD set out to address this lack of good quality impact evaluations and our recommendations led to the creation of the International Initiative for Impact Evaluation (3ie) in 2009. The number and quality of impact evaluations has risen significantly, but there is still a long way to go to make sure future development interventions are based on evidence of what works.
The Center for Global Development (CGD) and the International Initiative for Impact Evaluation (3ie) will host a workshop at CGD on Tuesday, May 4 to discuss the health and education benefits of conditional cash transfer (CCT) programs. The event, Closing the Evaluation Gap: 3ie One Year On, will also include an introduction to a special issue of 3ie’s new Journal of Development Effectiveness.
CGD president Nancy Birdsall and 3ie executive director Howard White will deliver keynote speeches at the workshop, and high-level policymakers and researchers will discuss the implications of existing evidence and propose recommendations to improve impact evaluations of CCT programs in the future. Panelists and speakers include Ruth Levine, former Evaluation Gap Working Group co-chair and current director of evaluation, policy analysis and learning at USAID; William Savedoff, a Working Group co-chair and CGD senior fellow; Marie Gaarder, 3ie deputy director; Lyn Squire, former president of the Global Development Network; and Laura Rawlings, lead specialist for the World Bank’s Human Development Network.
The link between evidence and policymaking is not a simple one, according to Savedoff. “Impact evaluations often provide a lot of useful nuanced information but policymaking thrives on big messages while trying to accommodate political, social, and cultural pressures,” he wrote in a recent blog post that cautioned against the possible misuse of impact evaluations.
The participants will explore strategies for increasing the role of evidence in development policies. “We need development and policy leaders to call for reforms that focus development interventions in a more effective way,” said White. “This event provides an opportunity to learn what is the impact of conditional cash transfer programs on people’s health and education - when they have worked and why - and what are the lessons learned,” he said.
The recent surge of interest in impact evaluations and creation of 3ie were spurred in part by the activities of the CGD Evaluation Gap Working Group which began in 2004 and drew upon input from 100 development policymakers and practitioners. In 2006 the group’s final report recommended strengthening evaluation efforts within major development organizations as well as establishing an independent organization to push for more and better evaluations designed to be able to attribute development outcomes to specific interventions. After two years of preparatory work, and with support from more than a dozen members and more than a score of associate members, the 3ie was formally launched in 2009.
The May 4 workshop will run from 9:00 a.m. to 5:00 p.m. and will be held at the Center for Global Development, 1800 Massachusetts Avenue NW, in Room 1004/1006 on the lobby-level. Reservations required.
By Ben Edwards
Focusing on the role of rigorous evaluation in policy interventions and highlighting lessons learned from conditional cash transfer programs in the health and education sectors, the event will be an opportunity for dialogue between high-level policymakers and researchers to discuss the implications of available evidence and propose recommendations for moving forward. The event will include the introduction to a special issue of the newly-launched Journal of Development Effectiveness focusing on CCT programs, along with a series of panel discussions focused on the capacity for social interventions to improve outcomes such as educational achievement, prenatal care, and newborn health in developing countries.
Nancy Birdsall gave the closing remarks at the 2004 Conference of the World Bank's Operations and Evaluations Department. The conference focused on the effectiveness of the Bank's policies and reforms. Birdsall discussed her views on reforming donor activities in order to improve foreign aid to poor countries.
Access the speaker slides
These slides are from a presentation given by Nancy Birdsall on October 27, 2004. The presentation is based on a CGD Working Paper of the same title.
Access the presentation slides (PDF).
A brief summary of the working paper:
In the face of continuing development challenges in the world's poorest countries, there have been new calls throughout the donor community to increase the volume of development aid. Equal attention is needed to reform of the aid business itself, that is, the practices and processes and procedures and politics of aid. This paper sets out the shortcomings of that business on which new research has recently shed light, but which have not been adequately or explicitly incorporated into the donor community's reform agenda. It outlines seven of the worst "sins" or failings of donors, including impatience with institution building, collusion and coordination failures, failure to evaluate the results of their support, and financing that is volatile and unpredictable. It suggests possible short-term practical fixes and notes the need ultimately for more ambitious and structural changes in the overall aid architecture.
In December CGD announced that Howard White had been selected as the first director of the International Initiative for Impact Evaluation or 3IE ("Triple I E"). The announcement marked a milestone in the creation of a new international entity designed to dramatically increase the number of rigorous impact evaluations in areas such as health and education, thereby providing the global development community with more knowledge about what works when and where--and what doesn't.
The announcement came just 20 months after CGD released a working group report, "When Will We Ever Learn? Improving Lives through Impact Evaluation" that offered recommendations for how the international community could ensure that the hundreds of billions of dollars spent annually on human development efforts also result in rigorous studies to help guide the design and implementation of future programs. The working group was co-chaired by CGD vice president Ruth Levine and William Savedoff, a consultant who is an expert in the financing and delivery of social services in developing countries.
The 3IE is a membership organization, with fees set on a sliding scale. Current members are: the Mexican Ministries of Health and Education, Ugandan Ministry of Finance, UK Department for International Development, Netherlands Ministry of Foreign Affairs, Canadian International Development Agency, IRC, African Development Bank, Bill & Melinda Gates Foundation, William and Flora Hewlett Foundation, Google, CARE USA, and Save the Children (United States).
White, who has extensive evaluation experience across Africa and Asia working with a number of different governments and agencies, is currently based in Cairo. He has set an ambitious agenda for getting the 3IE up and running. In a Q&A with Lawrence MacDonald, CGD's director of communications and policy, White described his plans for making the 3IE a reality:
Q: What will be your first actions in setting up the 3IE?
A: 3IE’s purpose is to deliver rigorous evidence on what works and why, focusing on questions that are relevant both now and in the long-term. My first task is to develop an initial set of “enduring questions” through a broad consultative process. This will establish 3IE’s identity, but also, and more importantly, give voices from developing countries a prominent role in shaping 3IE’s agenda.
3IE’s job is not just to commission rigorous impact evaluations, but commission influential impact evaluations. A second task in setting up 3IE will be to develop an influence strategy, drawing on international best practice, to guide us in producing materials that will make our findings influential and easy to apply in ways that will make a real difference in poor people’s lives.
Part of ensuring influence from our work is to broaden our membership, especially amongst Southern-based governments and institutions. And the larger our membership, the greater the scope of our work, and the greater our impact will be. I will be taking steps to broaden 3IE’s membership from my first days in the job.
Finally, the current members of 3IE are working together to identify a suitable host institution to act as a base for 3IE. We hope to be able to announce where this will be in the next couple of months.
Q: How will you decide what types of programs or projects to evaluate first?
A: This will be decided by the enduring questions identified through the consultative process I just mentioned.
Q: How much do you think a typical impact evaluation supported by the 3IE will cost, and how long will it take?
A: There is no one answer to that question. The high-end scenario is to start an impact study now for an intervention that will run for several years, collecting primary data for both treatment and control groups. Such a study could cost as much as a US$1 million if carried out in a high-cost country, though I’d expect most budgets to be well under half that. But there are frequently under-exploited data sets that allow for far cheaper and quicker studies—the Inter-American Development Bank has conducted a number of quality impact evaluations at a cost of less than US$50,000, each relying on existing data sources. 3IE will be commissioning work across this whole range. We do hope to identify some possibilities using existing data early on, so as to produce results in our first year or two of operation.
Q: Who will conduct these impact evaluations?
A: 3IE will publish the enduring questions, inviting proposals from any organization with the technical skills to conduct quality impact evaluations. But in general, the teams conducting these studies should be led by individuals and institutions based in developing countries. At present, the balance of skills is in developed countries, so we will encourage collaborative arrangements with a strong capacity-building element.
Q: Will all 3IE impact evaluations use Randomized Controlled Trials (RCT)?
A: 3IE evaluations will use whatever method is most appropriate to address the question under examination. In some cases this method will be a RCT, but this will likely not be possible in the majority of studies.
Q: Will evaluations be subject to public comment before they are finalized?
A: 3IE-commissioned evaluations will be subject to several levels of review. First, by 3IE’s own staff and an expert panel of reviewers selected for each study. Second, by the staff of the intervention being reviewed and the relevant government ministry or NGO. These latter consultations will include other concerned stakeholders from the wider public.
Q: How will the findings be disseminated?
A: 3IE’s purpose is to ensure its studies have influence, not only on the particular intervention under examination but also in other situations where the lessons are deemed relevant. Evaluation teams will be expected to work closely with staff implementing the intervention to draw out policy conclusions. The studies will also identify conclusions of general policy relevance for a wider audience. 3IE will organize thematic workshops for the presentations of findings from clusters of studies, and encourage study authors to present at relevant conferences and workshops. All this will be supported by cluster-specific websites and publicity for 3IE’s work amongst its members and supporters. Publication of results in book or journal form will also be encouraged.
From the meta-analysis work carried out by 3IE itself we will identify common issues about what kinds of interventions should be promoted, or how they should be implemented, and conduct our own dissemination campaigns around these findings.
Q: How many evaluations do you expect to commission in the first year of operation? In the first three years?
A: I hope we can identify 12-15 studies in the first year, at least half of which can report initial findings within 12-18 months. After three years 3IE will be commissioning new studies at the rate of 30-50 a year depending on the scale of financial support we receive.
Q: How long will it take until we know if 3IE is a success, and what are some of the signs of success or failure that you would expect to see?
A: Our success will be measured by greater support for interventions that have a demonstrable and cost-effective impact on poor people’s lives. 3IE will have its own monitoring and evaluation system to capture such impact. We will have failed if the studies we finance do not have policy relevant conclusions, or if no one acts on those conclusions.
Q: How did the 3IE come into existence? What was CGD’s role, and will CGD be involved going forward?
A: 3IE has its general origins in the growing demand for evidence of development effectiveness. This was spurred by the so-called “results agenda” of recent years which links judgments about development agency performance to the achievement of measurable improvements. Against this setting, the catalyst was the work of the Evaluation Gap working group organized by the Center for Global Development. The report of that group, When Will We Ever Learn? Improving Lives Through Impact Evaluation led to a series of meetings of various stakeholders. They became the founding members listed above, undertaking the initial tasks for its creation, facilitated by CGD, and providing a sound foundation with their financial commitments. CGD has been 3IE’s midwife -- and as befitting this role, will step back from 3IE once it is set up so that it can develop in its own way. CGD will not be a member of 3IE or receive any funding from 3IE.
Q: Will the 3IE itself be subject to an independent impact evaluation? How much time do you think would be needed before such an evaluation would be useful?
A: Yes, after the first three or four years depending on the volume of work. In addition there will be an annual performance review by 3IE's members.
- - - - - - - - -
Note: For more information on the purposes and formation of the 3IE, see the CGD initiative: When Will We Ever Learn: Closing the Evaluation Gap. To contact Howard White, write to 3IE@3ieimpact.org.
The sudden death of World Health Organization director general Lee Jong-wook at the start of the World Health Assembly has created a leadership vacuum at a time when the WHO faces immense challenges. Ruth Levine, who heads CGD's Global Health Policy Research Network, argues that the WHO leadership must become more independent so that science can shape public health policies and practice.
Q: What is the most pressing challenge facing the WHO today?
A: The WHO needs to figure out its place within an increasingly complicated group of funders, technical agencies, and advocacy voices in global health. The WHO is far from being the single authoritative voice or influence in the field and it weakens itself when it tries to do too much. It should focus on strengthening technical expertise around what might be called regional and global public goods: surveillance of infectious diseases, with transparent global reporting; stimulating investment in neglected diseases; and generating sound scientific knowledge about how to implement public health programs with broad social benefits.
Q: In your blog post on Lee Jong-wook's sudden death you urged greater independence in the leadership of the WHO. What do you mean by independence, and how can it be assured?
A: The WHO is severely hampered by a budget inadequate to its mandate and a staffing arrangement in which key positions are year-to-year appointments. Governance of the organization is fundamentally and explicitly political: the organization is responsible to ministers of health, who tend to be politicians, not people with deep technical expertise. Of course, there are notable exceptions, but by and large the organization is governed by a political body. Because of this, WHO seems to be in a perpetual cycle of trying to raise resources for programs that might be attractive to one or another interest group, but might not really be the most important to achieve better health. The WHO also faces intense scrutiny about some of its decision making – for example, how it recommends and pre-qualifies pharmaceutical products for use in developing countries - and has not always been able to be clear about the rules of the game. It’s almost as if the U.S. Centers Disease Control and Prevention or the FDA were directly managed by the U.S. Congress. The WHO would be stronger with a more arms-length relationship between the governing bodies and the technical work of the organization.
Q: How will the new director general of the WHO be selected?
A: According to the WHO Constitution, the director general is appointed by the World Health Assembly on the nomination of a 32-member Executive Board. The member states vote and in the past there has been lots of politics and horse-trading. As in the other international organizations where governance and leadership have been criticized, the process would be better if it were more truly merit-based rather then reflecting which region’s turn it is and other political considerations.
Q: What personal characteristics do you think are most important in the future leader of the WHO?
A: A strong commitment to science and evidence, and an ability to be politic without being political; that is, somebody who understands issues of national sovereignty and the sensitivities of different interest groups without subjugating technical decisions to such considerations.
Q: What can ordinary people do to help ensure that Dr. Lee’s successor will be adequate to these challenges?
A: The selection of the next director general will be decided by member countries' ministers of health. In the U.S. context this means Secretary of Health and Human Services, Mike Leavitt, who is currently in Geneva for the World Health Assembly. (Note: Staff in Sec. Leavitt’s office told CGD they are not authorized to give out his e-mail. They suggested that Americans interested in the U.S. role in the selection of the next director general of the WHO fax him at (202) 690-7203.)
Each year, donors spend more than $30 billion and developing countries spend hundreds of billions more on programs to improve health, education and other social outcomes. But few programs are evaluated to learn whether they make a difference in people's lives. This shortfall in evaluation wastes money and means that many decisions about social sector spending are made on political grounds.
CGD's Evaluation Gap Initiative aims to address this problem by highlighting the need for more and better impact evaluations, and proposing ways to increase the supply of knowledge about “what works.” Ruth Levine, CGD director of programs and a co-author of CGD’s Evaluation Gap Working Group draft report, recently traveled to Mexico to hear from senior Latin American officials their views about closing the evaluation gap.
Read the Mexico communiqué (pdf)
(Gonzalo Hernandez, head of Mexico's National Council for the Evaluation of Social Programs, confers with CGD’s Ruth Levine.)
Q: What is the single most striking aspect of your discussions in Mexico?
A: Two points stand out: First, I learned about the very impressive impact evaluation work being undertaken in Mexico, Chile and Argentina. Mexico is a real pioneer in this area. The legislature has mandated that impact evaluations be conducted, and the country is developing a track record of looking carefully at very important questions: how well is public spending reaching the poor; how are services being utilized; and--remarkably--what are the real-world effects on measures such child nutrition, school completion and household income? They are taking a strategic view: what do we need to know so that big anti-poverty, food supplement, housing and other programs work better in the future? Second, I was struck how the champions for good impact evaluation must fight daily battles--the same battles being fought by those who work on evaluation within development agencies and NGOs. Budgets are inadequate; it's hard to connect with and learn from technical colleagues outside the country; program managers feel threatened because they think of evaluation as a sort of policing function; the media focuses on the "bad news"; and evaluation results--whether positive or negative--are sometimes discredited by being labeled as part of a political agenda.
Q: Who did you meet?
A: CGD co-hosted the meeting with SEDESOL, Mexico's federal agency that manages major social programs like Oportunidades, to get feedback on the ideas generated by the Evaluation Gap Working Group. We met with the officials who lead and run the evaluation office of SEDESOL, as well as individuals who have been active in the design and evaluation of social programs in Argentina and Chile, and several people from research institutes, NGOs, USAID and philanthropic foundations.
Q: Do you see demand for better impact evaluation coming from other quarters?
A: I think the right question is not whether there's demand for evaluation, but whether there's demand for knowledge. From what we've seen and heard, there's lots of demand for genuine, credible knowledge about what works. Politicians want to hold ministries accountable; program designers want to learn from others; even widely dispersed “beneficiaries” have a stake in knowing what governments and donor agencies have actually accomplished, and how programs can be improved. While it's naïve to think that there could or should ever be a mechanistic application of evaluation results in the highly political domain of social sector spending, we are in an era of greater access to information, movement toward more evidence-based policy making, and more demands for accountability and transparency. This is the right moment for a big push.
But knowledge is a public good--a global public good, in fact--so individual countries, programs, and agencies don't have the incentives to put in the resources for adequate evaluation. The “knowledge agenda” is also hampered by political, bureaucratic and technical difficulties of conducting evaluations to generate that knowledge. We need to figure out how to get those who want the knowledge to recognize that the way to get it is by working with others to support impact evaluations.
Q: Is impact evaluation the only kind of evaluation that matters?
A: Absolutely not. I think about evaluation questions in terms of "are we doing things right?" and "are we doing the right things?" It's clearly essential to look at that first question: to understand the complex social processes affecting the implementation of a program; and to look closely at how well or haltingly a given program is rolled-out. It really does matter whether funds are being disbursed smoothly, people are being hired and retained, schools are being built and equipped. All of that is absolutely vital to feedback to program managers and designers, so that adjustments can be made. And I think for that sort of evaluation, it’s important to have a close link between evaluation and implementation to promote real-time learning.
But at the same time, we have to be thinking about whether we are doing the right things. When we build the schools, train the teachers, and introduce an innovation like computers in the classroom, are children going to school, staying in school and learning more than they would otherwise? And the way to figure that out is with impact evaluation.
Q: What is the biggest obstacle you see to improving social sector impact evaluations?
A: I think the biggest obstacle is that the "doing it" and the "learning whether it works" functions of organizations like social sector ministries, NGOs and development agencies have a hard time co-existing. To "do"--that is, to design a grant, to convince a government to take a loan, to implement a program--often requires being convinced that your approach is the best one. "I'm testing an idea" is not nearly as compelling as "This is going to work." In contrast, learning about whether the program is achieving the anticipated impact requires distance from the doing; you need to be able to look in an impartial way at what actually happened. So far, each organization has tried to solve this problem by setting up separate and sometimes explicitly independent evaluation units. But it is inevitable that except under extraordinarily visionary leadership the organization's priority will go toward the doing, leaving the learning function undervalued, under-resourced and, sometimes, undermined.
These are problems that can be solved if we get out of the mode of thinking about each organization as an isolated unit. If national governments, NGOs and donor agencies share a demand for knowledge, then it seems quite possible for them to participate in and benefit from a collective approach to generate that knowledge.
Q: What are the next steps for this initiative?
A: We are continuing to consult with a broad set of individuals about why there is a relatively weak base of evidence in the social sectors, and what can be done about it. We are particularly seeking feedback on the idea of establishing an independent international facility to provide flexible funding to support evaluation opportunities, to collaborate with countries and agencies in building an agenda of learning around some of the enduring questions in international development, and to widely share impact evaluation methods and findings. The next developing country consultation is in India in early April, at a meeting co-hosted by Rajat Gupta, Senior Partner Worldwide of McKinsey and Company, and Suman Bery, Director-General of the National Council for Applied Economic Research.
We are getting very thoughtful and constructive input from these discussions, as well as meeting champions in the field, which is really inspiring. We will be finalizing the report of the working group in the next couple of months, with specific, practical recommendations for the international community. I am optimistic that we can achieve a genuine breakthrough.
CGD senior fellow and director of programs Ruth Levine has urged the U.S. Congress to push for independent evaluation of development assistance. In testimony before the Senate Foreign Relations Committee, Levine said that independent impact evaluation is crucial for ensuring that the billions of dollars spent on development actually helps poor people.