|
July 2012
Dear Colleague,
I find it particularly hard to think clearly when the temperature rises above 85oF/30oC … though I admit to considerable recall bias and measurement error (for both my productivity and the temperature). Oh, and I have no appropriate counterfactual. Darn. Isn’t it nice to know that researchers use systematic methods to deal with such problems when addressing more serious public policy questions?
As this month’s Update shows, though, we’re still asking the question “When Will We Ever Learn?” because certain basics of impact evaluation are being ignored in highly touted programs. Fortunately, researchers continue to expand the range of policies that can be studied with experimental methods; while others grapple with questions of attribution and proof for evaluation questions that don’t seem amenable to quantitative techniques. With these kinds of ongoing debates, things may never cool down.
Regards,
William D. Savedoff
Senior Fellow
Center for Global Development
|
When Will We Ever Learn? |
A debate over evaluating the Millennium Villages Project (MVP) illustrates the embarrassments that policy advocates could avoid if they would listen – early – to impact evaluation experts and welcome their perspectives. Authors of a paper evaluating the MVP in The Lancet published a retraction of some of their findings within two weeks of its online publication. The Economist describes several technical faults with the study referring in part to work by Michael Clemens and Gabriel Demombynes. If the MVP team, and its chief proponent Jeffrey Sachs, had been willing to specify their theory of change, collect appropriate baseline data, and include independent evaluators – as they were advised to do as early as 2006 – the retraction would have been unnecessary and we would have learned something from the MVP experiment. Unfortunately, The Lancet’s peer review process failed to detect the study’s problems despite public critiques of earlier MVP evaluations (such as here and here).
Image: Ted Collins |
|
|
Large n |
The central feature of many impact evaluations is to infer causality by comparing otherwise similar individuals between those who participate in a program and those who do not. Two recent studies show how researchers are pushing the envelope, extending these approaches to analysis of groups. A recent study of block grants in Indonesia, “Should Aid Reward Performance?” analyzes how entire villages respond to incentives related to health and education performance indicators. Another study, “Can Institutions be Reformed from Within?” measured responses by Rajasthani police stations to different strategies for improving their performance. Both papers demonstrate how much more we can learn about incentives and institutions for groups with well-designed studies.
Image: United Nations
|  |
Small n |
In a recent 3ie working paper, Howard White and Daniel Phillips inquire about the core features required for credible evaluations when the evaluation question cannot be answered by a sample with sufficient power (“small n”). This problem arises, for example, when the object of study is unique (such as a national policy in a single country), the population is small, or the units of analysis are simply too heterogeneous. The paper summarizes qualitative methods such as general elimination, process tracing, and outcome mapping, seeking to identify a common set of core methodological principles. “Broadening the range of designs and methods for impact evaluations,” also asks what methods to apply when experimental and statistical techniques are infeasible. Commissioned by the United Kingdom’s DFID, the study concentrates on theory based, case-based and participatory methods for evaluating aid projects. Lawrence Haddad comments on both papers in his blog.
Image: Ted Collins |
 |
|
| Resources |
|
|
