Evaluation Gap Update

July 16, 2014

Plenty of debate to report over the "power" of Fairtrade coffee and the relevance of impact evaluations to public policy. At least now we know where to go to find all the relevant studies that have been published in the last 30 years … or could you be the person to win the prize for finding the one study that’s still missing? Just be sure you’re not suffering from medical conditions that affect your statistical judgment.

Regards,


William D. Savedoff
Senior Fellow
Center for Global Development

Is Fairtrade Fair?

When a recent study from the Fair Trade, Employment, and Poverty Reduction (FTEPR) project found little impact on rural workers’ wages, it sparked debate over issues critical to good impact evaluations: relevance, adequate controls, and statistical power. Kim Elliott asked whether the study focused on a relevant question, since Fairtrade aims primarily at smallholder farms, not rural laborers. A blog post by Matt Collin and Theo Talbot argued that the study did not control adequately for differences between non-Fairtrade and Fairtrade communities. Collin and Talbot also observe the study lacked statistical power - that is, the sample is too small to detect an effect.

3ie challenges you to find a missing impact evaluation ... and win a prize!

Aim for high quality research, but don’t forget that impact evaluations can only be influential if people read them. If you search for “health impact evaluations” via Google, you will get more than 53 million results – most irrelevant to your search. Knowing this, 3ie has created an Impact Evaluation Repository with 2400 studies spanning 3 decades and 120 countries. But to assure it is comprehensive, 3ie has issued a challenge: if you can find an impact evaluation that meets their selection criteria and isn’t already in the database you could win a prize. But hurry, the deadline is July 31st.

Is there a "right" method? A "wrong" one?

Debates over impact evaluation methodologies appear to be endless, especially when the word “random” is involved. The European Journal of Development Research features a special debate on the usefulness of impact evaluations. Lant Pritchett argues that rigorous evidence isn’t – for lack of external validity. Philipp Krause agrees and says Randomized Control Trials are irrelevant to the real process of policy design and implementation. Bill Savedoff responds with a plea to end the evaluation wars, arguing that no method can be declared better or worse without reference to the questions being addressed and the context. Ruth Levine makes an eloquent case that we risk both throwing out the baby (genuine efforts to find out what policies do) and drowning in the bathwater (of polarized debates over methodological superiority).

Wikimedia Commons

Resources

• Don’t underpower your next study. Read this recent systematic review of medical and non-medical literature about specifying target differences for sample size calculations.
• Evidence-Informed Public Health: Opportunities and Challenges is the agenda for the 22nd Cochrane Colloquium in Hyderabad, India, Sept. 21-26.
• Boosting the Life Chances of Young Men of Color is a good model for systematic reviews, courtesy of MDRC.  
• Find out if you suffer from Apophenia or related conditions like Patternicity and Randomania … but watch out for false positives.
• The Impacts of Microcredit: Evidence from Ethiopia finds few impacts if any.
• The London International Development Centre (LIDC) is offering a short course on Evaluation for Development Programmes, from November 10-14th.
• You can still register to attend "Making Impact Evaluation Matter," a conference co-hosted by the ADB and 3ie that will take place Sept. 1-5 in Manila.
• Submit your paper to present at the Social Protection in Africa conference in Nairobi, Nov. 12-13. Deadline July 22nd.
• The Campbell Collaboration has given preliminary approval to an adapted version of reporting standards for systematic reviews and conduct standards. Comments welcome before July 31.

If there is anything you think we should know about or would like to recommend for the next Evaluation Gap Update, please send it to [email protected].

Earlier issues of this newsletter are also available on our website for your perusal.

Thanks to Albert Alwang for his assistance in preparing this newsletter and to Michael Clemens, Ruth Levine and Justin Sandefur for their suggestions.