Why I Doubt Most Microfinance Impact Studies

Asif Dowla quotes my First Randomized Trial of Microcredit post and then asks a good question:

So in my view it was for decades essentially correct to say that we have zero solid studies of whether microfinance makes clients better off on average

Isn't this sweeping as well? What about all those journal articles published in JPE, Econ Dev & cultural change and other journals.

(I think Asif is referring to these two papers.)Yes, it is sweeping. It seems to fly in the face of this literature survey and this one. I stand by my statement but wonder whether it was a mistake to write it in passing without justifying it. I appreciate the comment because it made me realize I haven't been completely forthcoming on this point. So let me explain better. I was trained as a mathematician. That education exposed me to certain 20th century ideas that left me with intuitions (biases?) about what you can and cannot do with numbers. Gödel's famous incompleteness theorem showed that there are true statements about the integers (non-fractional numbers like --1, 0, 1, 2...) that cannot be proved and false ones that cannot be disproved. Despite the power of modern math, there are unknowable truths. In physics, Heisenberg's uncertainty principle states that one cannot simultaneously observe a particle's location and velocity with perfect accuracy. Despite the power of modern instruments, there are unseeable things.Early in my career at CGD, I became involved in statistical studies of whether foreign aid raises economic growth in receiving countries. Many studies seemed to show that it does, at least under certain circumstances, such as having good economic policies in the recipient nation. But the closer I looked, the more problems I found, as I have written in my jargon-free Guide for the Perplexed. These studies were not randomized: no government has ever experimentally given lots of aid to some countries while randomly withholding it from others. One major shortcoming in these papers, which I defined in the post on which Asif commented, is data mining. Non-randomized studies are more prone to it. Another is the difficulty of going beyond correlation to prove causation, as I also blogged. Recognizing this difficulty, researchers often adopt complex techniques to try to make the leap, the very complexity of which often hides their inadequacies even from the researchers using them. That's the black box problem. Another problem is poor data. (Remember when the number of HIV cases worldwide suddenly dropped?)All of these concerns apply to the academic studies done over the last two decades of the impacts of microcredit on families and businesses. In leveling this charge, I am not just hand-waving. I have spent a lot of time examining some leading studies (mentioned here). My paper on this work should be out soon, and then I will say more. [Update: It is out and I said more.]For now, consider the review of this literature in the textbook Economics of Microfinance by Beatriz Armendáriz de Aghion and Jonathan Morduch (limited preview). Here it is summarized by Pablo Cotler and Christopher Woodruff in a journal Asif mentioned, Economic Development and Cultural Change:

Armendariz de Aghion and Morduch...note that “the number of careful impact studies is small but growing, and their conclusions, so far, are [measured].” Armendariz de Aghion and Morduch’s review of the existing studies suggests that those showing the strongest impacts are also those with the largest methodological flaws, while those that are cleanest methodologically generally show little or no positive impact.

Last year's World Bank report Finance for All? says:

...the evidence from microstudies of favorable impacts from direct access of the poor to credit is not especially strong.

I quote to prove not the correctness of my opinion but the reasonableness.