If impact evaluations are feasible and useful, why are so few conducted? One important reason is that good impact evaluations have to be designed as part of a new program; yet, in these early program phases, politicians and project managers are primarily focused on designing and implementing their programs. The costs of starting an impact evaluation at this early stage are real and present, while the benefits of measuring the impact will only materialize in the future. Paradoxically, the same people who would like to have good evidence today about the impact of earlier social programs are unlikely to make the efforts required to design and implement studies that could benefit others in the future.
In addition, the outcome of an impact evaluation cannot be known in advance. Hence, those in charge of social programs in developing countries may be unwilling to face the risk of proving their program “failed”; while those involved in international assistance may fear that a few “negative” studies could undermine domestic support for foreign aid more broadly.
Consequently, those who are best positioned to conduct impact evaluations have very few positive incentives to do so, and face many costs and obstacles when they try.
Demand, Money and Incentives[i]
The main obstacles or barriers to conducting impact evaluations appear to be: diffuse demand for the knowledge, funding that is not available in a timely fashion, and weak incentives for actors to seek knowledge of program impact.
Demand for the knowledge produced by impact evaluations tends to be spread out across many actors and across time. It happens every time someone in a government, multilateral development bank or bilateral agency asks the question: “What programs are effective at … ?” This sometimes occurs at the beginning of designing a new program. Other times it arises when producing a document to lobby for additional funding. It also emerges when doing internal reviews of institutional performance.
But it is only at the moment of designing a new program that anything can be effectively done to start an impact evaluation. At that exact moment, program designers want the benefit of prior research, yet have few incentives to invest in starting a new study. Ironically, if they do not invest in a new study, the same program designers will find themselves in the exact same position four or five years later because the opportunity to learn whether or not the intervention has an impact was missed (O’Donoghue and Rubin 1999). Since information from impact evaluations is a public good, other institutions and governments that might have learned from the experience also lose when these investments in learning about impact are neglected.
It is in such circumstances, that timely availability of funding can make a big difference. (See Box 5). Despite the lack of incentives to conduct impact evaluations, many program designers and managers still have an interest in measuring the impact of their programs. When funding for impact evaluation studies is not readily available, it makes it more difficult to act on their interest. If funding were readily available, it might make the difference between doing or not doing a rigorous study.
[i] For a complementary discussion of the obstacles to good impact evaluations, see Levine, D.I. 2005.
| Box 5: Timeliness of funds can be critical to good impact evaluations
"A 'rapid-response' fund to support program preparation is a much needed initiative. The lack of baseline data is critical. For large projects, often the process of preparing the project for approval requires so much time that there is an unwillingness to delay for baseline data collection once the official approval is announced. For example, in the "Familias en Acción" project in Colombia, the project began implementing in some of the communities before the baseline information was collected. A ‘rapid-response’ trust fund could allow data activities to advanced independent of project approval." Source: Comments by Suzanne Duryea, Senior Research Economist, Inter-American Development Bank. Note: for further information on "Familias en Accion" see http://www.ifs.org.uk/edepo./wps/familias_accion.pdf |
Other incentives exist at the institutional level to discourage conducting impact evaluations. Government agencies involved in social development programs or international assistance need to generate support from taxpayers and donors. Since impact evaluations can go both ways – demonstrating positive or negative impact – any government or organization that conducts such research runs the risk of findings that undercut its ability to raise funds (Pritchett 2002). Policymakers and managers also have more discretion to pick and choose strategic directions when less is known about what does or does not work. This can even lead organizations to pressure researchers to alter, soften or modify unfavorable studies, or to simply repress the results – despite the fact that knowledge of what doesn’t work is as useful as learning what does.
When such pressures hold sway, a noticeable bias appears in the body of published findings. Studies that demonstrate programs are successful are more likely to be publicized by the participating institutions and also are more likely to be published in academic journals.[i] A "publication bias" emerges that provides an unfairly positive assessment of social programs. One way to counter this "publication bias" is to establish a prospective registry of impact evaluations, that is, record impact evaluations when they start. Then, future literature reviews can better assess whether published findings are or are not representative.
Reasons for optimism
Despite these barriers, a number of factors favor conducting impact evaluations. The first is personal commitment from people who recognize the value of impact evaluations, many of whom are employed by governments and international agencies today. A second factor is the growing capacity to collect data and do research around the world. More people are trained in impact evaluation methods and in fields that allow them to interpret the findings of such studies, and technological advances have reduced the costs and time of collecting and processing data. Third, there appears to be growing recognition among different agencies of the need for measuring results and that this cannot be done well without complementary studies that get at the issue of attribution. Finally, skepticism about the use of funds also puts pressure on agencies to measure impact. When the overriding risk is closure of a program, then the downside risk of negative findings is less problematic and both managers and project designers see greater benefit in measuring the impact of their programs in the hopes that they can demonstrate that the programs should continue.
[i] There is a wide literature on publication bias. With regard to publication bias in clinical trials, see Dickersin, K and Y.I. Min. 1993. “Publication Bias: The Problem that Won’t Go Away” Annals of the New York Academy of Sciences, 703(1):135-146.