July 2010

REVEALED DEMAND FOR IMPACT EVALUATION

For those who are skeptical that demand for impact evaluations exists, I am pleased to report that 3ie received 35 applications from 27 countries for its innovative Policy Window. You may recall from our last newsletter that this Policy Window first seeks proposals from policymakers regarding the projects they would like to see evaluated. By the end of July, a 3ie panel will announce which applications were selected for the second phase, in which researchers can apply for grants to evaluate the winning entries. The deadline for submissions to the second call will be at the end of August.

MCC SETS HIGH BAR BY DISCLOSING DATA

The Millennium Challenge Corporation is setting a high standard for other organizations that finance impact evaluations by making data from such studies available on its open micro data website. For example, last October, we reported on an impact evaluation of an education program in Burkina Faso called “BRIGHT” that found the program improved overall school enrollment by about 15-20 percentage points (depending on the data source and statistical model used). Normally, that would be the last you would hear about such a study, except perhaps to see it cited in synthetic reviews or “lessons learned.” By publicly disclosing data like this, the MCC is setting a standard for other organizations that is critical to scientific progress. Open data makes it possible for other researchers to test the interpretation of results and derive new findings.

GOOD IMPACT EVALUATIONS REQUIRE MORE THAN GOOD DESIGNS

The number and quality of impact evaluations is increasing and the sophistication of methodological debates is growing. However, it is rare to find documentation of the complex relationship between researchers, policymakers and institutions that affects which studies are done, with what resources, and whether the findings affect policy decisions. “Toward Closing the Evaluation Gap: Lessons Learned from Three Recent Impact Evaluations of Social Programs in Latin America and the Caribbean,” in the journal Well-Being and Social Policy, addresses this issue, identifying how impact evaluations are encouraged and resisted in Latin America and the Caribbean. Drawing on case studies from work in El Salvador, Jamaica, and Mexico, the article outlines best practices for successfully designing and implementing a high quality impact evaluation.

ADDITIONAL RESOURCES

• The Impact Evaluation Network (IEN) of the Latin American and the Caribbean Economic Association (LACEA) is calling for papers for its meeting on October 8-9, 2010 at the University of Miami in Florida, USA. Last day to submit papers is July 31st. Information on registration and training courses is on the IEN conference site.

• The World Bank is seeking field coordinators for impact evaluations of a distance teacher training program in Malawi and of a teacher incentive program in Lesotho.

• A 3ie synthetic review of “Community-Based Intervention Packages for Preventing Maternal Morbidity and Mortality and Improving Neonatal Outcomes” is also the basis for an online wiki – “Access to health - How to reduce child and maternal mortality?” – to which you can contribute.

• The U.S.Department of Education has published a study of comprehensive teacher induction programs, finding no impact on teacher retention and mixed results on student achievement.

• “India’s Janani Suraksha Yojana, a conditional cash transfer programme to increase births in health facilities: an impact evaluation” documents wide regional variation in implementation of this program along with effective reductions in mortality.

• IIS and AIID are offering a course in impact evaluation on Oct. 4-5 followed by a conference (Oct. 6-7) at the Royal Tropical Institute in Amsterdam. Confirmed speakers include Sulley Gariba, Paul Gertler, Ruerd Reubens and Howard White. Deadline for registration is Sept. 10.

• In the latest Journal of Economic Literature, David Lee and Thomas Lemieux look in-depth at regression discontinuity methods. The paper is followed by a forum with contributions from James Heckman, Guido Imbens, and Angus Deaton, who provide enlightening perspectives on the strengths and limitations of econometric modeling and statistical techniques.

• Stories vs. Statistics: The Impact of Anecdotal Data on Accounting Decision Making documents how accounting decisionmakers give greater credence to anecdotes than to statistical data. The study also shows how judgment orientation and counterargument can redress this bias. If this is an issue for accounting decisions, imagine how it might apply to development policies!