The Rapid Rise of the Randomistas and the Trouble with the RCTs

As I mentioned, "randomized control trials" (RCTs) are proliferating in development economics, being used to study such questions as whether microcredit puts more girls in school.I have spent much time challenging non-experimental studies of the causes of economic growth. Trained as a mathematician, I am less skeptical of the proposition that foreign aid or financial system development typically speeds national progress than I am of the number-crunching economists have done to back such claims. I'm more of an aid regression skeptic than an aid skeptic. (Though my skepticism of the evidence does lead me toward agnosticism on the substance.) The biggest problem is that technical efforts to rule out reverse causality usually fail, so that we can't be sure about what is causing what.By contrast, randomized studies compel respect. A researcher flips a coin to determine who gets a service; a year later, those who got it are happier, or healthier, or more stressed, or whatever, than those who didn't. Short of the supernatural, the only explanation for the correlation is that the intervention caused the outcome. What's to argue with?So I like RCTs, and I think it is good that randomized trials of microfinance are underway.Still, the rapid rise of the "randomistas" feels like a fad. Will a healthy movement overshoot? Already, grand men of economics such as Nobelist James Heckman and Angus Deaton are asking tough questions. Such as: Are RCT researchers doing science if they treat people and households as black boxes---things to be experimented on and observed---without modeling or studying what goes on inside the black boxes? If you learn that pushing this button turns on that light, what have you really learned about electricity? The concern is practical because it gets to whether researchers gain insight into human behavior that can lead to improvements in programs meant to help people.There are other impediments and disadvantages, which I want to think through. Can you add to my list of Troubles with the RCTs (apologies)?

• Arbitrarily (randomly) offering a service to some people and not others is immoral and goes against the professional grain of the people providing the service. The randomistas accommodate this concern by only randomizing in less-fraught ways. Karlan and Zinman randomly offered credit to South Africans who would not otherwise qualify. Banerjee, Duflo, and Glennerster worked with Spandana to randomize which neighborhoods get microcredit first as the group expands across Hyderabad, India.
• The placement of randomized trials is non-random. You can't perform one on microcredit in Bangladesh, where arguably it's been most successful, because just about everyone already has access to it. Only certain groups will allow researchers to "interfere" with their decision-making. (See above.) So interventions only get studied in certain contexts, which may not be globally representative.
• Any study is a test of both an intervention and an intervenor, so how do you attribute success or failure to the intervention per se? Two groups might do, say, after-school tutoring that looks the same on paper put works out quite differently in practice. And there is variation over time too: Maybe Spandana won't get the kinks out of its Hyderabad operations until after the researchers have finished evaluating them.
• RCTs only tell us the average effect. Whether you're talking about loans or pills, the same "treatment" affects different people in different ways, and that diversity around the average matters as much as the average itself. But you can't get at it with RCTs. You cannot randomize whether subjects in a randomized trial of penicillin are allergic to it.

Update 1: After I clicked "Publish" on this post, I walked into a talk given by Jeremy Weinstein, in which he did what I had just written was impossible. In fact, if you know who is allergic to penicillin, then you can do a control-treatment comparison just among them. To correct myself, RCTs falter on unobserved dimensions of diversity. If we can't tell who is entrepreneurial, then even an RCT cannot tell us how entrepreneurial ability mediates the impact of microcredit.Update 2: See Should Randomistas Rule? in The Economists' Voice (free with registration) by Martin Ravallion, director of the World Bank's research department, and another respected figure questioning the enthusiasm for RCTs.