Which Studies Should Someone Be Paid to Reexamine?

Probably you agree that actions meant to help poor people should be guided by the best science about what works. (Or perhaps you also have a problem with motherhood and apple pie.) And probably you'd concede that part of what makes science science is replicability. Cold fusion is a scientific joke, not a scientific advance, because the experiments seeming to generate evidence of fusion at room temperature could not be independently reproduced.In this way, replicability is at the heart of the grand project to give everyone a shot at a decent life.But in economics, and the social sciences generally, replication doesn't happen much. Why? No one ever won a Nobel Prize by copying others. And as I think John Kenneth Galbraith once observed about competition, people like replication until it happens to them.But times are changing. The World Wide Web was invented in 1990 to help researchers share data, and it is slowly shifting norms about openness and collaboration. Communities converge faster to agreement, dare I say the "truth," when analysis is public, collective, and iterative. Back when data were stored in punchcards and scientific discourse took place via ungainly bound volumes, statisticians could easily bar outside observers from their private digital laboratories. Now expectations of transparency are higher.So here's a cool thing. The International Initiative for Impact Evaluation (3ie) is seeking nominations by May 31 for impact studies that it should pay to have replicated. 3ie was born out of a recommendation in a CGD report led by Ruth Levine, Nancy Birdsall, and Bill Savedoff; its mission is to support new, high-quality research on what works in development, and at what cost. The just-launched replication program, on whose advisory board I serve, embodies an astute recognition that 3ie can also contribute by rigorously reexamining research done by others.Are there studies in your field that you think are influential enough to deserve reexamination---and confirmation, contradiction, or something in between? How strong is the evidence base for the idea that girls' education is an investment with particularly high returns? How sure should we be that, roughly speaking, men spend all their money on beer while women invest in the children? Or that financial sector expansion increases economic growth?"Replication" has a range of meanings. I could replicate the Hyderabad microcredit impact study by performing a similar experiment in Mumbai, what 3ie calls "external replication." That is important, but not what 3ie has in mind here. Or I could field my own survey team in Hyderabad to collect and analyze new data on the impacts of the microcredit experiment there, what psychologist John Hunter called statistical replication. Or, since I am lazy and lack managerial prowess, I could download the original study's data and reanalyze it, for a "pure replication." 3ie looks to support work of these latter kinds.Actually, I'd draw a further distinction within the "pure replication" category. There is what I consider true, pure replication, which strives to exactly reproduce a published analysis in order to scrutinize it. Partly for lack of imagination, that's what I usually do, as in my work with Jonathan Morduch on the Pitt & Khandker microcredit impact study. And there is what I could call reanalysis, which applies different methods to same data, as Jonathan did for Pitt & Khandker in 1998 and as Maren Duvendack and Richard Palmer-Jones (who also advises the 3ie program) did in 2011.Perhaps the biggest challenge will be persuading the authors of the chosen studies to share their data. Should that problem arise, I hope it will be solved by the public exposure generated by this exercise.Importantly, the expectation is that results will be posted regardless of outcome. Everyone loves a conflict, but confirmations matter as much as contradictions.Send your nominations to replication@3ieimpact.org. But surely the list should be crowdsourced, so I encourage you to post your ideas below as well. And if you want to achieve riches by replicating (any starving grad students out there?), stay tuned.