Quasi-experimental Thai Microfinance Evaluation (my vacation reading)

Turns out my list of significant new papers on the impacts of microfinance is arguably short an entry. In April, Joseph Kaboski and Robert Townsend released a more-final version of their study of the impacts of the "Million Baht Village Fund" Program in Thailand. In fairness to myself, this paper is in a different category. It is "quasi-experimental," meaning that no researcher worked with the program implementers to roll the dice to decide who was offered loans when. Rather, the evaluated program was shaped by an ordinary mix of administrative convenience, politics, and genuine government commitment to economic development. Yet, K&T submit, the particular way in which it was executed contains elements of arbitrariness which can substitute for artificial randomness as a way of generating "clean" estimates of the impact of microcredit on such outcomes as household income and investment.My list consisted of three randomized experiment-based studies (not quasi-experiments) as well as my own work with Jonathan Morduch, which replicated and questioned an important quasi-experimental study and concluded:

The sudden swell of randomized trials 30 years after the birth of microcredit of course reflects a broader trend in the social sciences. As such, it also leads to a broader question, about the value of non-randomized studies. Our prior is that exclusive reliance on one type of study is not optimal. But the present analysis suggests that for non-randomized studies to contribute to the study of causation in social systems where endogeneity is pervasive, the quality of the natural experiments must be very high. And it must be demonstrated.

In other words, we'd never say never, but we're reluctant to believe quasi-experimental studies unless they powerfully demonstrate that the arbitrariness that they exploit really is arbitrary enough to substitute for randomization.That's a lot of abstraction. I'll illustrate the ideas is a moment with the K&T study. And in fact I'm happy to report that the study in my opinion bids fair to meet our standard.In January 2001, Thaksin Shinawatra and his populist Thai Rak Thai party won a national election in a landslide. The party's platform included a promise to give each of the 77,000 villages in Thailand one million baht (about $24,000) for a revolving credit fund. (Urban neighborhoods were considered "villages" for this purpose.) The promise was kept, through the state-run Bank for Agriculture and Agricultural Cooperatives (BAAC). Write K&T, "In order to receive funds, villages needed to form committees, develop policies, submit an application/proposal for the village fund, and have the proposal evaluated and accepted. The vast majority of village households became members of the village funds and village funds averaged 94 members. The committees were selected democratically by the villagers at a village meeting, with regulations set up to ensure fairness of these elections." The committees thus exercised a lot of control over who got the funds, for how long, and at what interest rate. Brett Coleman found that earlier funds run this way in Northeast Thailand put most of the credit in the hands of influential committee members. Whether that happened this time is unclear.Let me give you Kaboski and Townsend’s bottom line before my own. Using data on 960 households in 64 villages, some rural some “semi-urban,” over the years 1997--2003, they find that the one-shot Million Baht program of 2001 increased:

• total short-term borrowing as opposed to, say, just substituting for moneylenders;
• loan defaults---not in 2002, but in 2003
• the frequency, though not the average amount, of investing in farming;
• spending on home and vehicle repairs, and to a smaller extent alcohol;
• net income, especially from wages and salaries;
• going wages.

It mattered little for such results whether households were headed by men or women, except that female-headed households saw no increase in spending on alcohol consumption outside the home. Women were not more likely to invest in their children’s education.Should you believe the results?Central to K&T’s argument is that the program was implemented in two ways that were arbitrary (speaking statistically, not morally). First, in the timing: the program did not come about, for example, as a quick response to the financial crisis of 1997, which from the researchers' point of view would have comingled the effects of the crisis and a contemporaneous response to it. Its timing was tied to the Thai political cycle (if one can use that term for the country's coup-punctuated governance). Second, every village got 1 million baht regardless of size. So families that happened to live in smaller villages got more credit each on average. K&T argue that the villages are geographic units of administrative convenience that are often split or redrawn ad hoc. "These decisions are fairly arbitrary and unpredictable, since the decision processes are driven by conflicting goals of multiple government agencies." K&T also generate maps and run statistical tests to show that villages of a given size are not systematically clustered in one part of the country, nor near rivers, nor even near each other. (Though they note that villages near forests tend to be a bit smaller.) In other words, small villages, which received more credit per person, are not obviously so different from large ones in how poor or rich or urban or rural or agricultural or industrial they are. I admire this thoroughness as a perfect example of the idea that "the quality of the natural experiments...must be demonstrated." I also applaud how they go to great lengths to check which of their results are generated by “outliers” such as a few rich families making big investments that throw the averages for the whole sample. K&T marshal these observations to buttress the core assumption of their paper: any systematic correlation between village size and changes in outcomes of interest such as average household earnings can only be explained by the Million Baht program, in particular the way it gave more credit per person to smaller villages.Recall the ostentatiously named Roodman’s Law of Instrumental Value: for a study to be useful, the assumptions on which it rests must be more credible than the ones it tests. Ultimately, credibility is in the eye of the beholder. The authors understand that the assumption on which their study rests is, while plausible, not ironclad. For example, that the drawing of the borders of villages is often arbitrary does not make them purely arbitrary. It might still be the case, say, that among urban “villages” (districts) those with fewer people gained more economically in the 2002--03 for reasons other than microcredit. Maybe they were low-density, affluent areas at a time when the rich got richer for broad economic reasons. How likely that is, I cannot judge. The point is that one can never stamp out such second-guessing with quasi-experiments.On the other hand, sometimes reality serves up quirky events that look like experiments. Should researchers ignore them all in the name of methodological purity? Kaboski and Townsend make the most of this singular microfinance venture, and it is for us to take their work in the appropriate spirit, as a hard-won, imperfect, and intelligent contribution to knowledge. It is probably the best ever non-experimental quantitative impact analysis of microcredit (albeit of an unusual kind of credit). I am certain of the high quality of the effort, less so about its conclusions.