April 14, 2009
I just made this up:
For a study to teach us about the world, the assumptions on which it rests must be more credible than the assumptions that it tests.A big challenge in the social sciences is to go beyond merely observing correlations to showing causation---e.g., that microcredit borrowing is not merely correlated with the well-being of households but increases it, on average. A common statistical technique for ferreting out causation is to use instruments, which are factors that are assumed to affect outcomes of interest only through a determinant of interest (caveat for experts: "...after linearly controlling for observed covariates"). For example, the Pitt and Khandker study sets up this picture:
landholdings => microcredit borrowing => household well-beingThe first arrow says that how much land a household owns before it starts with microcredit affects how much it borrows; in fact, in 1990s Bangladesh (Pitt and Khandker's study setting) owning enough land formally disqualified one from borrowing altogether. The second arrow embodies the hope that microcredit makes households better off, as measured, say, by their spending on food and other needs and wants. But by assumption no arrow runs from land directly to well-being. Landholdings are held to affect consumption only through microcredit. So if we observe in the data that the things on the two ends of the diagram are correlated---moving up and down together---then both the arrows in between must be at work. In particular, microcredit is making a difference. Here, we say that land "instruments" for credit; and having the first arrow, running from the instrument, lets us study the second arrow.(I don't mean to pick on Pitt and Khandker. Reading a similar microfinance evaluation by Joseph Kaboski and Robert Townsend last night made me think of this, but theirs is harder to explain.)Notice the reasoning here. We assume:
A. Landholding affects household well-being only through microcredit.That plus the data leads to:
B. Microcredit affects household well-being.A few comments about this structure:
- Just about all reasoning works this way. You have to assume something to conclude something. Think of Euclid's classic text on geometry, The Elements, which begins with a handful of axioms, such as that for any two points, a straight line can be drawn to connect them.
- This logical structure is often buried. It is the rare social science abstract that reads, "If you assume A, you can conclude B" even when that is actually what the paper shows. More often, B is highlighted while the dependence on A is deemphasized, even left implicit.
- It is often not clear that we should believe A more easily than B. If I am ready to make one assumption about causality in a society I hardly understand---landholdings only affect well-being through microcredit---why stop there? Why don't I just assume B---that microcredit raises household well-being on average? It would save me a lot of time. The answer has to be that A is easier to believe than B, just as Euclid's axioms are easier to believe than what Euclid proves with them, such as the Pythagorean Theorem (a2+b2=c2).
A. The random number generator affects household welfare only through microcredit.This is as easy to buy as Euclid's axioms, which is what makes RCTs powerful. And it illustrates Roodman's Law on the Instrumental Value of Instrumental Variables. Studies have "instrumental value"---they serve greater ends---when the assumptions on which they rest are easier to believe than the assumptions that they test.
Disclaimer
CGD blog posts reflect the views of the authors, drawing on prior research and experience in their areas of expertise. CGD is a nonpartisan, independent organization and does not take institutional positions.