Share

I happened to hit the home page of FINCA, a signer of last month's statement on the impacts of microfinance. Among the fly-down menus at the top is a phrase that caught my eye: client research. 2009 was a banner year for microfinance impact studies. I blogged five and, it turns out, missed one, which FINCA published in October under the name, Bottom-Up, Free Enterprise Solutions for Ending Poverty. Actually, it is a project report to the Templeton Foundation, which funded the work with a cool $1.337 million.

The report reads like a caricature of bad research. As far as studying impacts goes, it's $1.337 million wasted. Here's some running commentary:

Report text

Comment

This project proposed that free enterprise from the bottom-up---represented by the global microfinance movement---has already created the largest and most ubiquitous enterprise-based solution to poverty in human history. Since the late 1980’s, hundreds of microcredit programs, collectively channeling several billions of dollars in loan capital, have already doubled the incomes of over 100 million of the world’s poorest families....This is precisely the...story that FINCA wants to help the Templeton Foundation

tell in order to convincingly make its case for the transformative potential of free enterprise to end poverty.

"Double the incomes"?? According to whom?

In other words: We already know microcredit reduces poverty. Now we just need to prove it.

FINCA has worked to overcome the many barriers to measuring the effects of "bottom-up free enterprise" by creating a "clearing house for microfinance research"---a specialized entity designed to catalyze a strategic breakthrough in public awareness of, and funding for, the global microfinance

movement.

The party doing the research has a strong vested interest in the results.

Over the course of this project, over 16,000 client interviews were conducted by more than 100 Client Assessment Research Fellows in 33 countries throughout the world, consisting of over four million data points collected that depict the social and financial status of our clients and their families. [emphasis in original]

FINCA interviewed thousands of clients, asking hundreds of questions. In a few countries, the report explains elsewhere, they resurveyed the people so that they could measure changes in clients' lives after starting with microcredit. There was no attempt to construct a control group by interviewing non-clients. The assumption is that all measured changes in the lives of FINCA borrowers in Mexico, Uganda, and other countries since 2007 are caused by FINCA microfinance. No doubt FINCA asked people how old they are; and presumably they aged between survey rounds; so by the logic of the study, microfinance causes aging. More to the point, the study cannot rule out aging or general economic developments, such as changes in crop prices, as sources of whatever "impacts" are found. The fundamental problem is the confusion of observed outcomes with impacts, the latter being differences between observed outcomes and (unobserved) outcomes in the absence of FINCA.

The Structured Query Language (SQL) data warehouse (SQL Server 2005) currently holds millions of data points from thousands of FINCA Client Assessment Tool client interviews realized from 2003 to 2009, along with social performance data from additional research projects. The architecture includes hundreds of tables that allow for the easy creation of static reports, in addition to key documentation such as a business requirements guide and a data dictionary.

Sounds impressive. I use SQL Server 2005 too. I bet the motorcycles or SUVs driven by the FINCA staff are complicated too.

FINCA is reaching the poor and very poor

...

Clients spend most of their income on food

No serious quibble here, though I was surprised that only a third of subjects live below $4/day.

FINCA is reaching the poor and very poor

...

For example, with data from one year only, there are a number of potential reasons that clients’ daily per capita expenditures (DPCE) may have increased after receiving a loan from FINCA....This increase may be attributed to the loan (Relationship 2), or to a good harvest (Relationship 1) or an increase in demand (Relationship 3). However, by tracking the same clients over time, we are better able to identify (pinpoint) the exact relationship between the receipt of a FINCA loan and change in DPCE....This is a major step forward for both FINCA and the microfinance industry because for the first time we can narrow or hone the possible outcomes of the data collected. We can progress from three likely outcomes to one definitive outcome.

We can do nothing of the sort: see above. And actually it's not the first time.

Longitudinal data (from surveying the same people were more than once, in order to study changes) were collected in 15 countries. But this, FINCA's official report of "results of a four-year project," only gives results from Mexico. Why Mexico was chosen is not explained.

It seems that the people who designed and funded this project didn't understand how difficult it is to infer causality from this kind of data. The reliable way around the problem is randomization---not of who is interviewed, but who is offered microfinance. No matter how many people you interview, how many questions you ask them, how powerful your database...none of that of that can make up for bad research design. If you don't randomize the treatment (or use some other highly credible design), it's garbage-in-garbage-out. This study is like the non-randomized studies in the U.S. that erroneously led millions of women to take hormone-replacement drugs to reduce breast cancer and heart disease. Actually the FINCA analysis is far more naive.

So the Templeton Foundation spent more than $1 million on a study that any decent researcher could have explained was, from the point of view of creating knowledge, foredoomed. And the project doesn't seem to have succeeded on its own terms: in part because of credible research, the public doubts the benefits of microcredit more than it did a year ago. Go figure.

To be fair, I'd hate to see academics try to run a microcredit program. It'd be like watching some microcredit managers do research.

 

CGD blog posts reflect the views of the authors drawing on prior research and experience in their areas of expertise. CGD does not take institutional positions.

X

David Roodman's Microfinance Open Book Blog

Feed

CGD Experts