This is a joint post with Michael Clemens.
Michael Clemens recently wrote me, saying that he gets asked this question a lot. I do, too. So I was interested when he brought my attention to a
2007 article in Forbes that discusses a number of companies that do use randomized studies. I wasn’t surprised to see Google in the list, but I never imagined that all the junk mail that I receive from Capital One might be guided by sophisticated research (though it hasn’t convinced me to sign up yet!). Progressive Insurance apparently discovered profitable lines of business (middle-aged motorcycle drivers) by randomly accepting a portion of applicants who would normally be rejected and studying their claims behavior.
According to Hunt Alcott, other companies that have used randomized studies include H&R Block, PNC Bank, Amazon, Subway and Harrah’s Casino.
Businesses have an advantage in the evaluation arena because they get very effective feedback from sales data – but they still have the basic problem of comparing their performance against an appropriate counterfactual. If ice cream sales go up after a change in strategy was it due to the new approach or to an especially hot summer? And the key to answering these questions usefully means thinking in terms of testing assumptions with information that can prove your hypotheses wrong as well as confirm when you’re right. This kind of “testing mind-set,” as argued by
Thomas Davenport in the Harvard Business Review (2009), is used by companies all the time. A lot of Davenport’s points could apply equally well to improving the way aid agencies or public policies are assessed and improved.
So if companies are doing so much sophisticated evaluation work, why don’t we hear about it more often? As Michael pointed out, they have incentives to keep what they learn secret. Companies that have found a new profitable niche or effective marketing strategy don’t want to share the news – which is why you’ll never know when the iPhone 5 is coming out even if your best friend works at Apple. This is where public policy really diverges from the for-profit world. Companies are accountable to their shareholders; governments and non-profits are supposed to be accountable to the public through
transparency and by widely disseminating knowledge from good evaluations.
CGD blog posts reflect the views of the authors, drawing on prior research and experience in their areas of expertise.
CGD is a nonpartisan, independent organization and does not take institutional positions.