May 2010

OPENING BLACK BOXES:
EVIDENCE ON CONDITIONAL CASH TRANSFERS

Policymakers, researchers, and development experts gathered at CGD on May 4 to discuss recently published research on conditional cash transfer (CCT) programs and propose recommendations to improve impact evaluations of future interventions.   The workshop, Closing the Evaluation Gap: 3ie One Year On (Are Conditional Cash Transfer Programs Improving Human Capital?), was jointly hosted by CGD and the International Initiative for Impact Evaluation (3ie), and focused on the effects of CCT programs on outcomes such as educational attainment, grade progression, birth weight, and access to family planning methods. As explained by Marie Gaarder (deputy director of 3ie), the articles included in a special, newly-launched issue of the Journal of Development Effectiveness seek to “open the black box” by going beyond the measurement of average effects of program interventions. 

The event’s final session addressed the ways in which evidence and policymaking interact to define questions and establish the credibility and usefulness of evidence. In this panel, Ruth Levine (director of evaluation, policy analysis and learning at USAID) recommended that researchers follow a “3 Ps” strategy – Predicting likely problems to inform research; Prescribing clear core messages; and drawing Pictures, such as using GIS maps, to provide compelling visual demonstrations of information. Miguel Szekely (former undersecretary of education in Mexico) and Lyn Squire (former president of the Global Development Network) both stressed that bridging the gap between research and policy would require radical thinking, underscoring the importance of devising incentives for policymakers themselves to demand good evidence for developing more effective programs.

IF YOU KNOW A PROJECT WORTH EVALUATING...

Check out 3ie’s New Policy Window. A key part of the mandate for the International Initiative for Impact Evaluation (3ie) is to promote impact evaluations that are relevant to policy. 3ie has already encouraged this in the criteria it applies in reviewing grant proposals; but now, it is also experimenting with another approach. 3ie is currently soliciting proposals in a 2-step process. The first step asks for policymakers and program managers from developing countries to propose interventions to be evaluated. The interventions that are chosen by a panel of internationally recognized policymakers will then be the subject of a request for proposals from teams of researchers and implementing agencies who will conduct the studies. The deadline is June 11, so if you have an idea, put together a proposal right away.

Today, 3ie has also announced a request for proposals to conduct impact evaluations through its existing Open Window grants. Interested researchers should submit their expressions of interest by July 5 with 3ie’s new online application system.

MICROFINANCE GRAPPLING WITH EVIDENCE

More studies of microfinance programs are increasing our understanding of the ways such programs do and do not work in different places. But the nuanced results aren’t always welcome news to people who are strongly committed to the approach. In Microfinance Groups, Feeling Misunderstood, Misunderstand Research, CGD’s David Roodman discusses reactions to new impact evaluations and troubling confusions over the interpretation and use of evidence. Chris Dunford, head of Freedom from Hunger, also assesses the debate.

THE CASE OF THE MISSING BASELINE DATA

In June, the Executive Board of the World Food Program (WFP) will be reviewing: “Summary Evaluation Report of the Impact Evaluation of WFP School Feeding Programmes in Kenya.” I hope, they’ll note this key sentence (emphasis added): “In the absence of baseline data prior to the introduction of school feeding, a control group was assembled from randomly selected schools in the semi-arid districts that had offered only occasional or no school meals over the previous two school terms.” The researchers did a good job given the constraints, but this constraint can be and should be remedied for future programs. It represents another case of shortsightedness and further evidence of the need to plan evaluations right from the start.

LOOKING FORWARD

The large and energetic turnout for our May 4 event confirmed, once again, the strong interest and commitment to using impact evaluations to improve public policy. It was a great opportunity to celebrate the first year of 3ie - fully independent and going strong. But there's more to do. 3ie receives far more good proposals than it can support at current funding levels, and promoting constructive relationships between researchers and officials is an ongoing task that is critical to making evaluations relevant and policy responsive.

Thanks also to Katherine Douglas for her work on the May 4 event and contributions to this newsletter.

Regards,


William D. Savedoff
Senior Fellow
Center for Global Development

ADDITIONAL RESOURCES

• The European Evaluation Society (EES) will hold its Biennial Conference “Evaluation in the Public Interest: Participation, Politics and Policy” in Prague, October 6-8, 2010.
  
  
• In a recent blog, Michael Clemens argues not only that the Millenium Villages Project can be rigorously evaluated but explains that such an evaluation is imperative.
  
  
• In March, Lattimore and Visher published a summary of a careful impact evaluation of initiatives to reduce recidivism in the U.S. among released prisoners and to improve their outcomes for jobs, education, health, and housing.
  
  
• For a lighthearted and acerbic list of defensive maneuvers against evaluation findings, see Larry Dershem’s blog in which he quotes “Handy Hints for Program Managers” by John Scott Bayley.