|
September 2012
Dear Colleague,
Here in the north, orchards are full, and fields are being harvested before impending frosts. This month, I feel like the cycle of impact evaluations seems to be tied to the seasons. For many countries, researchers, officials and NGO staff are just returning from August vacations and are beginning their push toward the end of the year. Yet in other ways, impact evaluation knows no seasons. The cycle of a research project follows its own logic, related to design, collaboration with practitioners, logistics of data collection and analysis which may, or may not, respect the sun, moon, and weather.
This month’s Update includes a resource questioning why impact evaluations should follow unnecessary rules regarding fiscal spending cycles. The update also addresses how NGOs are trying to do impact evaluation without the resources or scale of large development agencies. And it refers to studies – systematic reviews – which (systematically) suggest the Evaluation Gap is alive and kicking. There’s still more planting to do.
Regards,
William D. Savedoff
Senior Fellow
Center for Global Development
|
|
If you think we’ve got too much evaluation, take a moment to read a few systematic reviews. While they are highly variable in quality, they regularly conclude that there are too few rigorous studies on any particular policy question. Here are some recent examples.
A Canadian systematic review on the "social determinants of health and health equity" qualified its conclusions by noting "the lack of available high quality, rigorously evaluated evidence." A BMJ systematic review on agricultural interventions that aim to improve nutritional status of children includes a statement that "[t]he question posed by the review cannot be answered with any level of confidence." The WHO Bulletin published a study of the factors that influence implementation of e-health systems in which the authors do a systematic review of systematic reviews (!), explicitly recognizing that the "… review data is two steps removed from primary data, and the quality of the primary research may not be properly assessed in reviews of substandard quality." |
|
|
Small organizations are often daunted by the growing demands for rigorous evaluation. In a recent blog, Bill Savedoff describes a range of approaches to evaluation at Oxfam GB, the Inter-American Foundation, and GlobalGiving. He underscores the importance of contributing to the growing body of knowledge regarding impact of different kinds of interventions. But he also points out that the fundamental starting point for all these efforts is a willingness to prove yourself wrong and therefore finding convincing ways to put your beliefs about “what works” to the test.
|  |
