Who Will Index the Indexers? Think Tank Ranking in Foreign Policy Magazine

On January 5, the University of Pennsylvania's International Relations Program and Foreign Policy magazine released a first-ever ranking of think tanks. Penn's James McGann, the author, calls it the “The Global Go-To Think Tank” index and Foreign Policy, simply the "The Think Tank Index." A think tank, in case you were wondering, is a "a group or an institution organized for intensive research and solving of problems, especially in the areas of technology, social or political strategy, or armament." The irony did not escape us at CGD: our own Commitment to Development Index was born in the pages of FP; now FP would rate us.Think tanks exist to influence through ideas. But quality of ideas and degree of influence are hard to measure. How then does this think tank index work? Does it count Google hits? Or the number of times a tank's scholars testify before legislatures or are quoted in the newspaper? It turns out that the rankings are based on a perception survey of knowledgeable persons:

...extensive research was conducted to develop a comprehensive list of all think tanks in the world....5,465 institutions worldwide were identified for inclusion in the study.To create a more manageable list...an international group of scholars, think tank executives, public and private donors, and policymakers were then asked to nominate think tanks they consider to be the best in the world. These experts used selection criteria such as a think tank's ability to retain elite scholars and analysts; access to elites in policymaking, media, and academia; media reputation; reputation with policymakers; scholarly output; and usefulness of the organization's information, among others. This effort resulted in a list of approximately 400 think tanks worldwide that were then included in the Think Tank Impact Survey.The Think Tank Impact Survey was then sent to hundreds of think tank scholars, think tank executives, and government and NGO personnel who fund think tanks. More than 150 responses from this panel of experts were received.

As the long-time manager of CGD's Commitment to Development Index--which was jointly conceived and published with Foreign Policy in 2003--I know well that every index is easy to criticize. In designing an index, one must balance the need for conceptual integrity with the value of communicating a message. For what it is, the think tank index seems to have been done well, with several rounds of culling and polling. And indexing via perception surveys is an established tradition: think of Transparency International's famous Corruption Perceptions Index. However, the think tank index's methods ought to be more apparent, for my taste. A careful reading of the University of Pennsylvania press release could not rule out the hypothesis that the rankings fell from the sky. At the FP site, one must peruse a lot of results before reaching an explanation of what they mean.We are of course pleased to see CGD in 15th place on the overall list--15th out of a universe of 5,465. Not bad, considering that CGD is easily the youngest of the top 15 and, except for the Peterson Institute for International Economics next door, well the smallest in budget. If our overall score were divided by our budget, we would rank much higher, on perceived productivity.I do wonder whether the index's methods somewhat tilt the results in CGD's favor. CGD broadcasts to many audiences, in many countries. We study more subjects than many think tanks: aid, trade, global climate, migration, many aspects of third world development, and more. So perhaps more people have come to know and respect our work than that of, say, the Center for Studying Health System Change, where my wife works, which prides itself on providing quietly objective information and analysis on U.S. healthcare to the 535 members of Congress. Moreover, because so many think tanks are American (1,777 out of the 5,465), the respondents were probably disproportionately American, and were more familiar with U.S. tanks. The Danish Institute for International Studies may be doing great work, but how many respondents would know?On the other hand, several institutions in the top 15, such as the Brookings Institution and American Enterprise Institute, make CGD's development-focussed agenda look narrow. And the fact remains that a lot of influential people know our work---and value our impact.