Learning by Measuring in Practice

At the heart of our work on Development Impact Bonds is the idea that to solve complex social problems it is necessary to test interventions, measure the effects, and then learn and adapt. 

This is why, unlike Duncan Green, I think that the growing recognition that we are dealing with complex adaptive systems means that we need to collect more data, not less.  The mantra for dealing with complexity is probe – sense – respond. Effective use of data – learning by measuring - is at the heart of how we should manage complexity.

But what does “learning by measuring” mean in practice?

My interest in how data can improve complex service delivery did not originate in some conceptual, think-tanky discussion or academic paper. I learned this from the real world experience of the organisations who are working together on the Social Impact Bond to reduce re-offending by former inmates of Peterborough Prison.

The Social Impact Bond has significantly changed their understanding of which interventions are working, and how they need to combine to achieve results.  Here is Evan Jones at St Giles Trust quoted in the Guardian:

We had a bit of a finger-in-the-air approach to understanding how clients engaged with us after our initial intensive work with them on release. A lot were drifting off but we didn't understand why. Through the Peterborough bond, we've been able to see really clear trends for the first time. We can see which areas of crime young people are returning to, why people stop engaging with us, how we can stay in contact in a way that's right for them, and be there for them to turn to if they are heading towards re-offending, as well what work and training are having the most positive effect. We've been getting results with some of the most prolific locals who are known for repeat offences.

Development Impact Bonds are not primarily a financing model: they are a business model. They enable a group of organisations to come together around a well-defined social problem, and work together to find solutions through a process of testing interventions, measuring the effects, adapting and learning.  This is possible in the case of a SIB or a DIB, but not in a conventionally structured project, because private investors provide flexible funding for the interventions and public sector agencies only have to pay if they work. These investors have the incentives to put in place high-quality data management systems (in the case of the Peterborough Prison SIB, managed by Social Finance) to ensure that interventions are adapting to information about what is working as it is gathered.  Evan Jones adds:

Normally, by the time we've realised what is working or isn't working on a contract, the funding has run out. This time we have space to say, 'Let's do more of this and less of this.’

We think that NGOs and other service delivery organisations working in development face these challenges in much the same way as their counterparts working in domestic service delivery. By design, DIBs put the rigorous collection and use of data at the heart of the partnership, informing and improving decision-making in real time. A striking lesson from the Social Impact Bonds in that using this data in this way is hugely powerful for improving performance, and that the service delivery organisations – contrary to feeling “over-managed” - find it empowering and valuable.