Kenyan Economist Offers First Independent Evaluation of Millennium Villages Project

A remarkable study reached the public last week. It is the first independent, rigorous, firsthand evaluation of the Millennium Villages Project (MVP), an effort by the United Nations and Columbia University whose admirable goal was to show that “the poorest regions of rural Africa can lift themselves out of extreme poverty in five year’s time.” The new study shows that the MVP is far from reaching that goal at its flagship site.Working on her own, without the collaboration or endorsement of the MVP, Kenyan economist Bernadette Wanjala of Tilburg University collected data on households in or near the site at Sauri, Kenya, where the project was launched in 2005. She interviewed 236 randomly-selected households that had been exposed to the MVP’s large package of agriculture projects, education programs, infrastructure improvements, and health/sanitation works. She also interviewed 175 randomly-selected households from an area of the same district (called Gem) that was not exposed to the intervention. She wanted to compare the two groups to see for herself whether or not the project had done what it promised: to lift the treated households out of poverty in a few years’ time and spark “self-sustaining economic growth”.In their just-released paper, Wanjala and her colleague Roldan Muradian of Radboud University use the new survey data to measure the project’s impact on poverty. They carefully compare treated and untreated households that were otherwise similar in many ways—such as household composition, adults’ education, fertility, economic sector, and land holdings. Because this project is large and intensive, spending on the order of 100% of local income per capita, it is reasonable to hope that it might substantially raise recipients’ incomes, at least in the short term.Wanjala and Muradian find that the project had no significant impact on recipients’ incomes.How is this possible? While Wanjala and Muradian find that the project caused a 70% increase in agricultural productivity among the treated households, tending to increase household income, it also caused less diversification of household economic activity into profitable non-farm employment, tending to decrease household income. These countervailing effects are precisely what one might expect from a large and intensive subsidy to agricultural activity. On balance, households that received this large and intensive intervention have no more income today than households that did not receive the intervention.Wanjala and Muradian’s independent results contrast sharply with the MVP’s own impact evaluation, which is carried out strictly internally and is based on confidential data.The director of the MVP, economics professor Jeffrey Sachs of Columbia University, states that a top priority of the project is to “raise community incomes” and to meet the Millennium Development Goals, the first of which is to cut income poverty in half. Sachs asserts that “incomes are rising” and that this “enormously successful” effort is “achieving its goals”. But these statements are not supported by scientific impact evaluation. To date the project has not released any numerical data on the impact of the project on recipients’ incomes. It has released extensive data on the sites apart from income, and is collecting data on incomes, so it is noteworthy that the project releases no analysis of impacts on incomes.The project’s internal impact evaluation also describes very large positive effects on agricultural productivity, stating that crop yields “doubled or even tripled across the sites”, while Wanjala and Muradian independently find more modest effects on agricultural incomes, which are directly linked to agricultural productivity. Part of this difference can be attributed to the fact that the MVP’s evaluation measures one thing and describes it as something else. Although the MVP describes the impact it calculates as occurring “across the sites”, it is in fact calculated by comparing farm plots that received inputs to “plots within the MVP area but where inputs were not applied.” In other words, the doubling or tripling of yields happens when farms receive the intervention and fully cooperate with and correctly implement the intervention. (In economics this is called the “treatment-on-treated” effect.) Wanjala and Muradian’s lower estimate compares farm productivity across entire villages where the intervention was attempted to entire villages where it was not attempted, accounting for all limitations to the de-facto degree of cooperation and implementation by individual farmers. (In economics this is called the “intent-to-treat” effect.) Wanjala and Muradian’s method is a more meaningful measure of the impact of the project. The two different effects would only be equal if farmers were interested, willing, and able to do exactly what is wanted by outside technical experts.The results of Wanjala and Muradian are even more striking for another reason. The Millennium Villages’ intervention sites were chosen specifically because the project’s designers thought the project would work better in those villages than in other villages. The project’s evaluation protocol states, “Issues of feasibility, political buy-in, community ownership and ethics also featured prominently in village selection.” This selection bias alone might have caused incomes at the treated sites to be higher, many years into the project, than incomes at the untreated sites—even if the project itself hadn’t caused that difference. Instead, incomes today are typically the same in the two groups.My colleague Gabriel Demombynes and I have described several failings of the MVP’s internal impact evaluation over the past year (see here, here, and here), and our concerns are widely shared in the development community (such as here, here, and here). The project has categorically rejected the need for any change at all. Wanjala and Muradian’s new study highlights the critical importance of independent and transparent impact evaluation in development work. It also calls into question whether the Millennium Villages Project, the latest in a decades-old tradition of village-level package antipoverty interventions that have failed to reduce poverty in the long run, is capable of reducing poverty even in the short run.