Evaluation Gap Updates

The U.S. government shows renewed interest in using evaluation to improve public policy, with significant steps by its foreign policy agencies. In China, a new study shows the importance of baseline data due to systematic bias in evaluations that rely on recall. And even movie stars can play a role in evaluation, providing data that helps explain statistical bias.

I’d like to take a moment to acknowledge contributions to these newsletters from the supportive team of researchers here at CGD who regularly produce and use impact evaluations, including Michael Clemens, Victoria Fan, Amanda Glassman, David Roodman, and Justin Sandefur. My thanks also go to Ted Collins for his assistance in finding items and pulling the newsletter together.

Regards,


William D. Savedoff
Senior Fellow
Center for Global Development

Evaluation policies move forward at USAID and the U.S. State Department

“USAID Evaluation Policy: Year One” assesses progress on the new evaluation policy that the agency approved last year. Only so much can be done in a year, but USAID has already posted its evaluations online for public use and has initiated training and other activities to implement the policy. At the same time, the U.S. State Department has moved toward a more systematic, rigorous and open evaluation process with its newly approved “Department of State Program Evaluation Policy.” To its credit, the State Department appears to have learned from USAID’s approach in terms of clarifying the different types of evaluation; distinguishing accountability from learning; and promoting principles of independence and integrity.

Uncovering bias in recall-based evaluations

Why go to the effort of doing rigorous impact evaluation with counterfactuals and baseline surveys? Why not just ask beneficiaries about how much a project benefited them? Martin Ravallion has a new paper that uses both approaches for the same project, a village-level antipoverty intervention in southwest China, and compares them. One evaluation repeatedly measured income and consumption in both treated and untreated villages. A second evaluation asked villagers in both groups, after the project was over, how income and consumption had changed in the village. Ravallion shows that the second, recall-based evaluation gives weak and biased signals of how the project actually changed economic conditions. He offers nuanced evidence of how this bias arises: the same unobserved traits of different villagers can influence both their subjective assessments of their own welfare and their degree of participation in the project.

Image: Flickr user Yoshimai / cc

Innovation Grants Available from USAID

Aid agencies are not monolithic and sometimes use special offices or programs to promote experimentation. This is the case for Development Innovation Ventures (DIV), an office created by USAID in 2010 to fund innovative development initiatives with a strong emphasis on rigorous evaluation, learning and dissemination with a range of grants covering conceptual, pilot and scale-up phases. It has awarded 20 development grants so far, including an inexpensive tamponade to stop postpartum hemorrhaging, a messaging campaign to reduce road accidents in Kenya, and hermetic grain storage bags that eliminate losses from insects mold and mildew in Afghanistan. The office accepts applications on a rolling basis, with the next window of review on April 16, 2012. Applicants do not have to be from the United States.

Image: Kendra Helmer / USAID