Water Water Everywhere? A World Water Week Q&A with Louis Boorstin

One privilege of compiling Millions Saved was learning more about the work of people like Louis Boorstin, currently Managing Director of the Osprey Foundation. In his previous role as head of water, sanitation, and hygiene (WASH) at the Bill & Melinda Gates Foundation, Louis worked to co-invest with the World Bank in at-scale programs that would be rigorously evaluated. The unique result: lives saved in the real world, measured rigorously, with lessons about what did and didn’t work so well. As experts, practitioners, and advocates gather in Stockholm to celebrate World Water Week, here’s a Q&A with Louis—in his own words—on what he’s learned from his experience:

You founded the WASH program at the Bill & Melinda Gates Foundation, and you now lead the WASH program as Managing Director of the Osprey Foundation. Why should these and other global health funders invest in WASH programs?

Helping households and communities to improve their access to, and sustained use of, safe water, sanitation and hygiene services brings multiple benefits, starting with basic human dignity (who likes having to worry that they can’t get a clean drink of water or use a proper toilet?). Indeed, the WASH program started at the Gates Foundation because when the foundation’s leaders would visit poor communities and ask about people’s health, the residents often raised the challenges they had accessing safe water and decent toilets. Bill Clarke, the President of the Osprey Foundation, decided to make WASH one of Osprey’s focal areas for similar reasons borne of his experience in Latin America and Africa.

Of course effective WASH programs offer many other advantages: better health, particularly for children under 5 who can die or suffer profound illness from water-borne diseases (e.g., poor sanitation can cause physical and cognitive stunting in young children); better lives for girls and women, who will not have to travel long distances to collect water or run the risk of attack when they don’t have a latrine at home and must defecate in the open; and increased income due to fewer days of work or school lost to water-borne diseases (a World Bank study estimated that in the year the program was launched Indonesia lost the equivalent of 2.3% of GDP due to poor sanitation and hygiene). Improved water, sanitation and hygiene are also connected to many other development challenges, from reducing transmission of the polio virus to making AIDS treatment more effective to better nutritional uptake, so making progress in this area has substantial indirect benefits too.

Millions Saved features two WASH programs with funding from the Gates Foundation: a sanitation program in Indonesia and a handwashing program in Peru. Why did the Gates Foundation decide to fund these two particular programs? What results were they hoping to see—and what lessons were they trying to learn?

These were two of the earliest initiatives funded by the WASH program at the Gates Foundation. We were encouraged to take "big bets" (in keeping with the foundation’s "impatient optimist" spirit), so after studying the challenges and opportunities in the WASH sector and consulting with a number of experts, we saw the chance for making real progress in these two areas. The driving force behind the rural sanitation initiative was the apparent success of a new approach to getting whole communities to actually use toilets. For handwashing, we were inspired by progress in wide-scale behavior change in other fields (e.g., condom use) and by the simple fact that if you can succeed in getting people to wash their hands (admittedly a big “if”), it’s one of the least expensive ways to reduce disease and death. We also had found a partner, the Water and Sanitation Program (WSP) at the World Bank, which was capable of managing large scale projects in multiple countries while also learning from them at the same time.

From a broader perspective, both of the initiatives were designed to test a hypothesis we had formed: that the concept of "scaling up" was a myth. We saw lots and lots of pilots in the WASH sector, some of which were successful… but very, very few of which actually scaled up. So it occurred to us that a key stage was missing in the development cycle between "pilot project" and "widespread adoption"—what we came to call "testing at scale." Simply assuming that a pilot would scale up was like saying that because I can build a bike in my garage, I can run a bicycle factory. More to the point, just because an NGO succeeds at getting several villages to build and use latrines doesn’t mean that the approach can be replicated at scale to reach thousands of villages across several districts. If you want to work at scale, you need to test and plan at scale. And at the Gates Foundation, we were fortunate enough to have the resources—which were substantial but still constrained—to test at scale.

Speaking of learning—both of these programs were structured to facilitate rigorous impact evaluation. Why was that important? And what design choices helped make that possible?

The WASH program at the Gates Foundation aimed to achieve three goals at the same time: i) demonstrable impact, meaning there is hard evidence that people’s lives are getting better through improved health, economic and social indicators; ii) true sustainability, meaning the financial, technical, managerial and physical resources are available to keep things running for a long, long time; and, iii) scalability, meaning the approach being supported has the potential to reach millions of people. Impact matters because it’s not enough to pay for latrines or to assume that something works; toilets actually have to be used and programs need to change people’s lives. Sustainability matters because the WASH sector is littered with projects that failed within a few months or years (which often comes from focusing on installing hardware instead of enabling long term service delivery). And scale matters because the problem is so huge—about 4.5 billion people, or 2 out of every 3 people in the developing world, lack access to safe or sustainable sanitation. (Interestingly, these three goals are very similar to the criteria used to assess candidates for Millions Saved: i) attributable health impact; ii) duration longer than 5 years; and, iii) more than 100,000 beneficiaries.)

So the demand for rigorous impact evaluation was built into the program right from the start. At the time these two initiatives were conceived, this was relatively new thinking for the WASH sector: rigorous methods were rarely used, and programmatic decisions were (and often still are) largely driven by anecdotal data and practitioners’ informed, but unproven, views. It seemed to us that the absence of good evidence was one reason progress in WASH was very slow despite significant spending. The lack of data also made it hard for the WASH sector to compete for funding against other sectors, such as global health, which were building solid evidence on effectiveness and cost-effectiveness.

Over time, we realized that rigorous evaluation could contribute to sector progress in two main ways: by measuring impact and by assessing which particular approaches work (and which don’t). We developed a WASH sub-program with multiple grants aimed at rigorous learning on critical questions, effectively deconstructing the broader challenges faced by these two early grants into separate, more manageable topics. (We also realized the limitations of approaches like randomized controlled trials, which typically answer a relatively narrow question for a specific context; whether such results can be generalized and applied at scale may call for further evaluations in a range of contexts.) In retrospect, our ambition to learn in these two initiatives was well intentioned but also unrealistic in what could reasonably be achieved, hence putting a burden on the grantee and its implementing and evaluation partners. The WASH program evolved to use a more flexible, demand-driven—yet still rigorous—approach to evaluation.

In Indonesia, the program led to significant improvements in sanitation access and a 30% drop-off in the prevalence of diarrhea—a tremendous accomplishment. What approach did you take to achieve this system-wide change, and why?

That’s the key to success here (and across the WASH sector)—system-wide change—and figuring out how to do that effectively was the result of the deep experience and persistent hard work of our grantee, WSP, its close collaboration with the Indonesian government, and its relations with many other partners. WSP approached the rural sanitation crisis not as a social or charitable problem that could be solved by giving away toilets but rather as an economic and political challenge that required active engagement with government systems and markets. Indeed, WSP’s hypothesis was that for this type of program to be effective at scale, it had to be led and "owned" by the Indonesian government, both at local and national levels. WSP saw its role as a coach for government staff, recognizing both the leverage and the limitations of this approach.

To that end, WSP developed and implemented a 3-pronged approach of: i) stimulating demand for sanitation through community-wide behavior change; ii) increasing the supply of affordable, appropriate products and services by actively engaging with the private sector; and, iii) strengthening the enabling environment for sustainable rural sanitation by working closely with government on regulations, financing, capacity building, incentives and other policies. This approach was designed to work at large scale and, while obviously tailored to the needs of Indonesia, it was meant to be replicable elsewhere (and was, in fact, implemented at the same time by other WSP teams in two states of India and in Tanzania). Of course, doing this in practice—while also managing a rigorous M&E program—was very challenging given the range of activities and partners involved… but WSP knew that this was necessary to achieve the systems change required for sustainable progress at large scale. (For those who want to know more about this, I recommend the case study in the Millions Saved, which provides a clear, concise description of the approach.)

One other thought here: I actually believe the impact of this program was much greater than has been estimated by the Millions Saved team. This is for two reasons. First, it’s because the independent evaluators analyzed Phase 2 of the initiative. As noted in the case study, the results of this phase, based on the program’s own M&E indicators, were considerably worse than the two other phases: 17% of the communities achieved open defecation free status in Phase 2 as compared to 58% in Phase 1 and 54% in Phase 3. So the overall program is likely to have had a much greater impact, even though that wasn’t captured by the independent evaluation. (As much as I believe in rigorous evaluation, I also believe there are situations where it’s appropriate to apply judgement.) Second, the indirect impact of the initiative has been significant: Indonesia now has a government-led and government-funded sanitation program that is active in every province across the nation. While the benefits of that program cannot be counted under the Millions Saved methodology, and while I can’t be sure that it wouldn’t have happened anyway without the Gates Foundation grant, it’s reasonable to say that we contributed to this long-term, systemic change which has added real value to the lives of millions of Indonesians.

In Peru, the program increased the frequency of handwashing but did not lead to measureable health improvements. What challenges did the program encounter? Based on what you’ve learned, is there anything you would do differently if you could do it all over again? 

I wish I had a definitive answer to these questions. The challenges could have been in the basic program design, such that the range of activities intended to influence people’s handwashing behaviors at scale just weren’t effective enough. We knew this was a risk going into the project, as there was no accepted approach for changing handwashing habits… let alone doing so consistently at large scale. Indeed, the experts leading the behavior change effort recognized mid-stream that they should have focused more on habit formation, as that could have been more effective than motivation and opportunity. Or the challenges could have been in the execution of the program—for example, the program might have invested more in human centered design and brokering private sector partnerships. The Case Study in Millions Saved describes some of those obstacles (again, I recommend the Case Study to those who want to get good picture of the initiative). Most likely, it was a combination of these elements. Another factor to bear in mind is that Peru had a national handwashing initiative that predated and continued during the WSP program (which was designed to complement and strengthen the national initiative). This was an attraction in selecting the country as one of the four program sites (the others were Senegal, Tanzania and Vietnam), but it also may have made it harder to achieve incremental progress on reducing diarrhea.

On the plus side, the initiative generated considerable learning. And, while the Gates Foundation did not to fund further large-scale hygiene initiatives (because of the decision in 2009 to focus the WASH program on sanitation), others—such as the London School of Hygiene and Tropical Medicine—have continued to seek out more effective ways to influence handwashing habits at scale. All that being said, if I could do it over again knowing what we know now, I would certainly take a more methodical, staged approach where we first tested a range of behavior change interventions in various contexts and combinations to see what actually worked. Only then would I consider testing at wide scale.

As we celebrate World Water Week and look to achieve SDG 6, what is the most important lesson we should take away from this experience?

Here are three key lessons from this experience:

1. Change the system. Real progress on WASH—high impact, truly sustainable, at large scale—only happens if you have the courage, insight and patience to work with the permanent systems that deliver WASH services to the poor. WSP understood this and designed both the rural sanitation and handwashing initiatives to achieve this… one with, as it turned out, better results than the other. That’s why they took the 3-pronged approach I mentioned earlier: supporting demand, supply, and the enabling environment. Beyond the direct impact of the programs, the learning from this and similar experiences has informed the strategies of many others in the sector, including an initiative that the Osprey Foundation currently supports called the Agenda for Change. Still, systems change isn’t easy—if you want "easy" then just dig a well or pay for a latrine—but it’s the only way we have a shot at reaching the SDGs.

2. Influence motivations. To catalyze and sustain real change over time, you need first to understand, and then to influence, the motivations and incentives of key players in the system. This can range from decisions by households (why should we install, maintain and actually use a toilet when our forebears have always defecated in the open?) to the actions of government officials (why and how should we reward villages and district that make real progress on sanitation?). Systems are dynamic, not static, and sustained change can only occur when key players are motivated to make different decisions. Although sometimes that can result from evidence of health impacts and other benefits, it’s usually much more effective to appeal to social pressures or institutional interests.

3. Test and learn at scale. Working at scale is not just a matter of expanding what worked in a pilot project. Rather, it means designing the implementation approach to engage with large-scale players in ways that are readily replicable… and then testing and refining the approach based on rigorous learning. Building learning into a large-scale project can be challenging as most governments and official funders just want cost-effective results and may balk at the concept of testing and iterating at scale. But that’s what it takes to make real, durable progress.