MCC Says Data Is in Its DNA—Show Us You Mean It by Using It Right

Though it’s still a couple of months until the Millennium Challenge Corporation (MCC) will pick countries to be eligible for funding in the coming year, it’s already clear the agency’s board of directors will face some tough decisions when it meets in December. The country scorecards that serve as the basis for MCC country eligibility decisions aren’t complete, but the data for the particularly weighty indicators—including the must-pass Control of Corruption hurdle—is now available.

I ran the numbers to get a sneak peek at some of the issues the agency and its board will grapple with over the next few months. Some of what emerged from this number crunching is encouraging—most current partner countries surpass MCC’s standards and some interesting new prospects for partnership emerge. More troubling is that two of the countries currently developing compacts—Kosovo and Mongolia—don’t pass the corruption hurdle. Both must be reselected in December to continue progression to compact signing.

Now, I don’t find this troubling because I’m concerned MCC might be working with countries that are “too corrupt." In fact, for neither of these countries does a “failing” score at all suggest that there has been an actual deterioration in the anticorruption environment. Instead, my concern is that MCC finds itself once again in the unfortunate and awkward situation in which the fate of a country partnership hangs in the balance, teetering between smart use of data and a dogmatic interpretation of criteria that were never meant to be enforced as ironclad rules.

Though it hasn’t always been this way (see p.4), in the recent past, dogma won out over nuance when the board did not reselect two countries that failed the Control of Corruption indicator and stated that MCC wouldn’t sign a compact with either until they passed—even though there was no evidence of an actual policy decline. The board will face a similar decision again this year, and this time I hope the result is different. For MCC to retain its credibility as an evidence-based agency it must show that it understands that the key to being evidence-based is knowing your evidence, including its limitations.

Let’s pause for some brief background. MCC was established to work only with countries that demonstrate a commitment to good governance and growth-friendly policies. To identify these countries, the agency relies on country scorecards, made up of a series of 20 quantitative, third-party indicators of policy performance. To be considered for compact eligibility, a country must score above a certain threshold on at least half the indicators. In addition, there are two “hard hurdles” that a country must pass to meet the scorecard criteria. One of these is the Control of Corruption indicator. A country must score better than half its income level peers (i.e., the median score) in order to pass. Though the scorecards are the public face of the selection process, the board also considers supplemental information on policy performance and whether MCC could effectively pursue its mission in a country.

Once a country is initially selected for compact eligibility, MCC generally requires it to be reselected each year during program development until the compact is signed. This typically means that a country must be selected 2-4 years in a row before getting to the finish line. This is where problems can emerge since the scorecard is not well-suited for monitoring policy performance over such short periods of time. The indicators are imprecise proxies of policy performance and most of the time—as MCC itself clearly states—small year-on-year changes in score reflect indicator noise rather than actual policy changes. This is precisely what we see happening with both Kosovo and Mongolia.

Let’s look at Kosovo. First of all, even though it “fails” MCC’s criteria this year, we can see that its score remains higher than almost any other year since its 2008 independence. But what about that dip from last year? Should we be concerned?

Source: Worldwide Governance Indicators, 2016

 
To answer this question, you have to get into the weeds, so bear with me for a minute. The Control of Corruption indicator is an index of several different corruption assessments, so it’s possible to dive into the data underlying the composite score to see what’s driving the change. In Kosovo’s case, only one of the 9 underlying sources rates the country any differently at all compared to last year (see below). And that one source? The Gallup World Poll that asks citizens whether or not they feel like corruption is widespread in their country. It’s ludicrous that the board might consider pulling back MCC’s compact engagement with Kosovo simply because a few more Kosovars said, “yeah, I feel like there’s a lot of corruption”—especially when you bear in mind that there tends to be a large gap between surveyed perceptions of corruption and surveyed experiences with corruption.

Data sources that make up Kosovo’s composite Control of Corruption score

Source: Worldwide Governance Indicators, 2016 (country data view)

 
Mongolia’s situation is somewhat different. Here we see that its Control of Corruption score is essentially unchanged over the last several years; there is certainly no evidence of decline. Indeed, a look at Mongolia’s scores over the last 10 years shows that assessments of the country’s anti-corruption environment remain at historically favorable levels.

Source: Worldwide Governance Indicators, 2016

 
Mongolia’s main problem is that there has been an increase in median performance on the Control of Corruption indicator in its peer group of lower middle income countries (LMICs). This is partly due to changes in the composition of the LMIC pool—some high performers joined the group this year, some lower performers moved out. In fact, had this year’s median pass-fail threshold remained the same as last year, Mongolia would pass.

Excerpt of FY17 lower middle income country rankings on the Control of Corruption indicator

 
Trying to ensure MCC funds go to relatively less corrupt countries is a worthy objective. Employing rigid rules around an imprecise indicator is not the best way to do it. This is not to say that a data-based eligibility system should be jettisoned. The indicators help weed out the poorest performers and protect against greater intrusion of geopolitical considerations—something MCC’s eligibility system was explicitly intended to confront. What’s damaging is the ironclad interpretation of data rules.

Recognizing this, my colleagues, Charles Kenny and Casey Dunning, have advocated dropping the hard hurdle on the Control of Corruption indicator. I agree completely with their justification but also see room for a more moderated stance. In my view, applying concrete decision rules to imprecise data is less egregious for the initial selection decision (i.e., the first time a country is picked to begin developing a compact). It at least provides a transparent basis for eligibility decisions, even if being in or out comes down to little more than luck for a number of middle-performing countries. On the other hand, rigid interpretation of imprecise data is inappropriate for determining whether to continue a pre-existing partnership. If a country up for reselection fails the scorecard, the board should not, based on this fact alone, determine that MCC should curtail the relationship. If MCC can point to a concrete and concerning deterioration in actual policy performance, the agency should of course take action (and it does). But absent that, curtailing a partnership with a country (or threatening to) is capricious.

Since its inception, MCC’s supporters have rightly credited the agency with its smart use of data to inform decision-making. For eligibility decisions this has mostly (though not always) been the case. Unfortunately, a (poor) precedent has been set recently with respect to the situation we’ll see this year with Kosovo and Moldova. This year’s selection process is an opportunity for the board to prove MCC’s supporters right. Let’s let good data sense rather than data fundamentalism rule the day and make MCC—and the USG—a better development partner for it.