The Millennium Villages Evaluation Debate Heats Up, Boils Over

The Millennium Villages Project, now underway in villages across Africa, is a keystone of United Nations efforts against global poverty. For years there has been a largely behind-the-scenes debate about how that project is evaluating its impacts. In the past week that debate suddenly heated up. A lot.First came the project’s report on its first five years, and announcement that it had secured an additional $72 million to continue and expand its intervention. That new report claimed a long list of quantified “results” and “achievements” of the intervention so far, such as 72% decreases in malaria prevalence at the project sites and 55% increases in skilled birth attendance at the sites.My co-author Gabriel Demombynes and I pointed out that those new claims continue a pattern of baseless claims of “impact” by this Columbia University-based project. For example, malaria prevalence has been falling in many of the provinces and countries where the small project intervention sites are located. And skilled birth attendance has been rising in many of the same provinces and countries. This makes it inappropriate and unscientific to simply claim that all change seen at the project sites—much of which would likely have happened without the project due to broader trends—as a “result” of the project.In new critiques, Lawrence Haddad, director of the Institute of Development Studies, argued that such simple before-and-after analysis is inadequate. And Madeleine Bunting of The Guardian raised questions about the project’s claims that its impacts are sustainable:

[T]he question is what has been left behind that will be sustainable and how long the impacts of the project will last.

Jeffrey Sachs, a professor in Columbia University’s economics department who runs the Millennium Villages Project, issued stern responses to Bunting and to Haddad.In the response to Bunting, Sachs asserted that the project is “enormously successful” and that its claims of impact are backed by 27 articles of “peer-reviewed science.” If you actually read those articles, you’ll see that only three of them involve any impact evaluation, all of which is short-term. You’ll also find that those three articles only measure two of the long list of impacts claimed by the project (child stunting and crop yields), and that one of those articles (about child stunting) is based on the same simplistic before-and-after analysis mentioned above. Peer-review or no peer-review, before-and-after comparisons are not scientific. For example, child stunting at the Bonsaaso, Ghana Millennium Village has declined, but child stunting across the large region where Bonsaaso is located has declined at the same rate over the same period.In the response to Haddad, Sachs defended the project’s claims of impact based on before-and-after analysis. He and Prabhjot Singh wrote that one cannot compare trends at the intervention sites to trends elsewhere:

The logic is also flawed. In a single-intervention study at the individual level (e.g. for a new medicine) one can have true controls (one group gets the medicine, the other gets a placebo or some other medicine). With communities, there are no true controls. Life changes everywhere, in the MVs and outside of them.

In other words, they claim, comparing trends at the intervention sites to trends in other areas is illegitimate, because things are changing everywhere.This is where it gets ugly, because the above is just bizarrely wrong.Serious economists who saw Sachs’ statements could not believe their eyes. Professor Justin Wolfers of the University of Pennsylvania called Sachs’ post “indefensible”. David McKenzie of the World Bank, one of the world’s leading impact evaluation experts, called Sachs’ statements “stunning” and “baffling”. David's colleague Berk Özler, also a top expert on development impact evaluation, “couldn’t take it anymore” and called Sachs’ statement “absurd” and the project’s evaluation a “mess”.What has all of these people so upset? Suppose I open a flu-shot clinic on a street in New York City, and I want to show my funders what my clinic’s impact was. I observe that over the following year, fewer people who live in the neighborhood got the flu. Suppose also that flu infections declined at the same rate across all of New York City, where lots of people were getting flu shots from lots of other clinics.There are two, separate questions you could ask about the impact of flu-shots in this scenario. The first is, “What was the impact of my clinic on flu infections?” It’s likely to be small, since flu rates are going down all over the city, and many of those who got their shots at my clinic would have gotten them somewhere else if I had never showed up. A different question you could ask is, “What is the impact of flu shots in general on flu infections?” The city-wide decline supports the evidence that flu-shots in general have an impact on flu, but it would contradict my claim that the flu infection decline near my clinic was the “result” or “achievement” of my clinic in particular.What Sachs and Singh are saying, above, is that you can’t compare the results of the project’s activities at the intervention sites to results elsewhere, because other projects are doing similar things—like promoting the use of fertilizer—elsewhere. But the project’s reports do not claim impacts of fertilizer promotion in general. They claim impacts of the Millennium Villages Project in particular. That’s what all these experts find indefensible, baffling, and absurd. Funders of a project want to know what the impact of that project is, and this is how the project portrays its claims of impact to its funders.Of course, if I could find a way to claim that the entire, city-wide decline in flu was somehow caused by my little clinic, then I would be on firmer ground. This is, in essence, what Sachs’ defender Charity Ngilu has done in a statement today:

The Millennium Villages Project, and Professor Sachs individually, had a huge effect in enabling Kenya to pursue a policy of mass distribution of bed nets and the shift to community-based treatment of malaria.

I invite readers to judge for themselves the plausibility of the claim that “Professor Sachs individually” has caused the huge increases in school enrollment, vast increases in cell phone ownership, and huge declines in malaria prevalence that have occurred all across Africa over the past decade—all changes that the Millennium Villages Project has uncritically claimed in large part as its own “impacts” and “achievements” with its before-and-after evaluation. If that is true, Africans themselves would not have made the enormous efforts and sacrifices they have made to accomplish those sweeping improvements if not for the arrival of Professor Sachs from New York. On that, I am speechless.