MCC Success Story Is a Failure (spoiler alert: n=1)

Don’t get me wrong. I’m a big fan of MCC. I once declared the organization the biggest experiment in impact evaluation and praised them for their commitment to independent high quality evaluations.

So when I received a note from a colleague with the subject heading “n=1” in reference to an MCC press release, “One herder’s plump cows signal improvement in Burkina Faso”, I was initially inclined to treat it as an aberration. Indeed, he anticipated my reaction:

"I know what you are going to say, Bill: ‘Don't have a cow.’ Or maybe, ‘Don't let MCC's public relations folk get your goat.’"

I read the piece and was relieved to find a few paragraphs that indicated the single anecdote seemed to have the backing of some broader evidence. For example, since the agricultural training program began, the average weight of cows increased “from 213 pounds in 2008 to 549 pounds in 2013.” This sounded like rather dramatic evidence of success… until I got another email from a colleague who grew up in the Far West and has actually seen a cow. She wrote:

"Is that really right? the baseline cow weight was 213 lb??  That is a tiny cow."

 To which my first colleague responded:

"… cows in poor developing countries can sometimes be little bigger than a Great Dane. But perhaps the cows of today were calves in 2008!"

And it looks like he may be right.  My second colleague took advantage of MCC’s generally informative website to track down data that might explain this possible anomaly. She wrote back:

"No current data reported. I did confirm the baseline figure of 97 kg (or 214 lbs).  But… the M&E report says the baseline was from 2012 (not 2008) for a cohort of calves whose weight gain would be monitored.  Sounds like an adolescent growth spurt."

Sadly, I added this story to my file on “Examples Demonstrating Importance of Counterfactual” and tagged it “Good Case for Explaining Difference-in-Differences.” Then I looked one more time at the press release for information to restore my faith in MCC’s standards of rigor.

There! Information about income gains:

"… because of the MCC-funded training, improved techniques and receiving a cow with better genetics, Tall [the farmer with the plump cow] is expecting more than double his daily milk production—a boost that would net Tall another $6 each day, he said."

Tall also collects the cow manure and sells it to his neighbors for fertilizer—a practice that has earned him almost $550 over the past 18 months.

I emailed this to my friends who wrote back:

"Doesn’t sound like the cows got fatter, but the farmers got new and improved cows. They replaced their Great Danes with Herefords!"

And:

"For the $142 million program to be cost effective, the farmers need to sell more manure than MCC!"

Now this seemed totally unfair. After all, a net gain of $6 per day from increased milk production plus $360 more in fertilizer revenues over the same 12 months comes to about $2,500 per year more in milk profits and manure sales. I emailed back: Is it possible that a farmer in a rural area of a country that has a per capita income of $300 got an increase in annual income of $2,500 as a result of the program?

First response:

"Maybe, but only if Mr. Tall is an unusually successful farmer. If MCC had monitoring data to show average annual income gains of something like $1,000 per farmer, it might justify the $36.5 million price tag on this component. That would be way more useful to know than the story about the exceptional Mr. Tall."  

Second response:

"The numbers still sound fishy (improved cows are often fed fishmeal …)."

Enough said. We had our fun toying with the article and punning around.

Kidding aside, MCC should know that its reputation as a different kind of aid agency rests not on the compelling nature of the special cases, but rather on the experience across all farmers. Anecdotes are generally misleading. They tend to trumpet individual success stories regardless of the associated costs and draw attention to changes that might have happened even without the program. When MCC claims success on the basis of such stories, it erodes the distinctive nature of MCC’s commitment to cost-effective impact.

If MCC has documented this project the way it claims it does, then it should know the average increase in income across all farmers required to achieve the minimum level of impact needed for success. Further, their monitoring data should reveal whether the program is on track or not. Then readers could understand if Mr. Tall is representative of farmers in the program or an exceptionally successful one. It might take an extra paragraph or two to explain, but MCC fans expect analysis to back up the anecdotes. As one colleague wrote:

  "… that is the long and the (Mr.) Tall of it."

Thanks to my colleagues who regularly toss me tough challenges and analytical insights in funny packages.