In December CGD announced that Howard White had been selected as the first director of the International Initiative for Impact Evaluation or 3IE ("Triple I E"). The announcement marked a milestone in the creation of a new international entity designed to dramatically increase the number of rigorous impact evaluations in areas such as health and education, thereby providing the global development community with more knowledge about what works when and where--and what doesn't.
The announcement came just 20 months after CGD released a working group report, "When Will We Ever Learn? Improving Lives through Impact Evaluation" that offered recommendations for how the international community could ensure that the hundreds of billions of dollars spent annually on human development efforts also result in rigorous studies to help guide the design and implementation of future programs. The working group was co-chaired by CGD vice president Ruth Levine and William Savedoff, a consultant who is an expert in the financing and delivery of social services in developing countries.
The 3IE is a membership organization, with fees set on a sliding scale. Current members are: the Mexican Ministries of Health and Education, Ugandan Ministry of Finance, UK Department for International Development, Netherlands Ministry of Foreign Affairs, Canadian International Development Agency, IRC, African Development Bank, Bill & Melinda Gates Foundation, William and Flora Hewlett Foundation, Google, CARE USA, and Save the Children (United States).
White, who has extensive evaluation experience across Africa and Asia working with a number of different governments and agencies, is currently based in Cairo. He has set an ambitious agenda for getting the 3IE up and running. In a Q&A with Lawrence MacDonald, CGD's director of communications and policy, White described his plans for making the 3IE a reality:
Q: What will be your first actions in setting up the 3IE?
A: 3IE’s purpose is to deliver rigorous evidence on what works and why, focusing on questions that are relevant both now and in the long-term. My first task is to develop an initial set of “enduring questions” through a broad consultative process. This will establish 3IE’s identity, but also, and more importantly, give voices from developing countries a prominent role in shaping 3IE’s agenda.
3IE’s job is not just to commission rigorous impact evaluations, but commission influential impact evaluations. A second task in setting up 3IE will be to develop an influence strategy, drawing on international best practice, to guide us in producing materials that will make our findings influential and easy to apply in ways that will make a real difference in poor people’s lives.
Part of ensuring influence from our work is to broaden our membership, especially amongst Southern-based governments and institutions. And the larger our membership, the greater the scope of our work, and the greater our impact will be. I will be taking steps to broaden 3IE’s membership from my first days in the job.
Finally, the current members of 3IE are working together to identify a suitable host institution to act as a base for 3IE. We hope to be able to announce where this will be in the next couple of months.
Q: How will you decide what types of programs or projects to evaluate first?
A: This will be decided by the enduring questions identified through the consultative process I just mentioned.
Q: How much do you think a typical impact evaluation supported by the 3IE will cost, and how long will it take?
A: There is no one answer to that question. The high-end scenario is to start an impact study now for an intervention that will run for several years, collecting primary data for both treatment and control groups. Such a study could cost as much as a US$1 million if carried out in a high-cost country, though I’d expect most budgets to be well under half that. But there are frequently under-exploited data sets that allow for far cheaper and quicker studies—the Inter-American Development Bank has conducted a number of quality impact evaluations at a cost of less than US$50,000, each relying on existing data sources. 3IE will be commissioning work across this whole range. We do hope to identify some possibilities using existing data early on, so as to produce results in our first year or two of operation.
Q: Who will conduct these impact evaluations?
A: 3IE will publish the enduring questions, inviting proposals from any organization with the technical skills to conduct quality impact evaluations. But in general, the teams conducting these studies should be led by individuals and institutions based in developing countries. At present, the balance of skills is in developed countries, so we will encourage collaborative arrangements with a strong capacity-building element.
Q: Will all 3IE impact evaluations use Randomized Controlled Trials (RCT)?
A: 3IE evaluations will use whatever method is most appropriate to address the question under examination. In some cases this method will be a RCT, but this will likely not be possible in the majority of studies.
Q: Will evaluations be subject to public comment before they are finalized?
A: 3IE-commissioned evaluations will be subject to several levels of review. First, by 3IE’s own staff and an expert panel of reviewers selected for each study. Second, by the staff of the intervention being reviewed and the relevant government ministry or NGO. These latter consultations will include other concerned stakeholders from the wider public.
Q: How will the findings be disseminated?
A: 3IE’s purpose is to ensure its studies have influence, not only on the particular intervention under examination but also in other situations where the lessons are deemed relevant. Evaluation teams will be expected to work closely with staff implementing the intervention to draw out policy conclusions. The studies will also identify conclusions of general policy relevance for a wider audience. 3IE will organize thematic workshops for the presentations of findings from clusters of studies, and encourage study authors to present at relevant conferences and workshops. All this will be supported by cluster-specific websites and publicity for 3IE’s work amongst its members and supporters. Publication of results in book or journal form will also be encouraged.
From the meta-analysis work carried out by 3IE itself we will identify common issues about what kinds of interventions should be promoted, or how they should be implemented, and conduct our own dissemination campaigns around these findings.
Q: How many evaluations do you expect to commission in the first year of operation? In the first three years?
A: I hope we can identify 12-15 studies in the first year, at least half of which can report initial findings within 12-18 months. After three years 3IE will be commissioning new studies at the rate of 30-50 a year depending on the scale of financial support we receive.
Q: How long will it take until we know if 3IE is a success, and what are some of the signs of success or failure that you would expect to see?
A: Our success will be measured by greater support for interventions that have a demonstrable and cost-effective impact on poor people’s lives. 3IE will have its own monitoring and evaluation system to capture such impact. We will have failed if the studies we finance do not have policy relevant conclusions, or if no one acts on those conclusions.
Q: How did the 3IE come into existence? What was CGD’s role, and will CGD be involved going forward?
A: 3IE has its general origins in the growing demand for evidence of development effectiveness. This was spurred by the so-called “results agenda” of recent years which links judgments about development agency performance to the achievement of measurable improvements. Against this setting, the catalyst was the work of the Evaluation Gap working group organized by the Center for Global Development. The report of that group, When Will We Ever Learn? Improving Lives Through Impact Evaluation led to a series of meetings of various stakeholders. They became the founding members listed above, undertaking the initial tasks for its creation, facilitated by CGD, and providing a sound foundation with their financial commitments. CGD has been 3IE’s midwife -- and as befitting this role, will step back from 3IE once it is set up so that it can develop in its own way. CGD will not be a member of 3IE or receive any funding from 3IE.
Q: Will the 3IE itself be subject to an independent impact evaluation? How much time do you think would be needed before such an evaluation would be useful?
A: Yes, after the first three or four years depending on the volume of work. In addition there will be an annual performance review by 3IE's members.
- - - - - - - - -
Note: For more information on the purposes and formation of the 3IE, see the CGD initiative: When Will We Ever Learn: Closing the Evaluation Gap. To contact Howard White, write to 3IE@3ieimpact.org.