Give the Global Fund a Gold Star for Their Hard-Hitting Evaluation...Now Comes the Hard Part

In 2002, the Global Fund (GF) was established to be a “new and improved” model for health aid. Founding head, Richard Feachem coined the pithy phrase “Raise it. Spend it. Prove it.” to capture their raison d’etre. A hard-hitting evaluation of their first five years has just been published. It gives them: an A – for “raising it”; a B – for “spending it”; and, a D minus, for “proving it”.Much to their credit, the evaluation assessed not just the grants, but also how the Fund’s structure, and modus operandi, influences how the grant activities are identified and implemented. GF funders and board members are now in a position to make informed decisions about changes that could make the GF work better. By any measure, hard work awaits.The evaluation synthesis describes the GF model for supporting countries as a “work in progress”. This is code, in the fine diplomatic tradition, with details in the detailed reports telling a more candid story. The reality described in the second evaluation report is a funding entity that does not yet have in place a workable model, neither for “doing the right things” nor “doing things right.” That is, the Fund is not yet able to ensure that it is funding grants that are pursuing the best strategy or strategies to achieve program goals; nor does it have in place a system to ensure that that the activities are implemented efficiently and that problems in implementation are identified and resolved in a timely fashion.The GF model for supporting disease programs had higher ambitions, and presented itself as an alternative to past failures. Prior to its creation, the main vehicles for supporting health initiatives in developing countries were bilateral aid agencies and development banks. These mechanisms did fairly well on impact and capacity building, with the exception of low capacity countries in Sub-saharan Africa. (see the 1999 report by the World Bank’s Independent Evaluation Group on the development effectiveness of health projects). However, they were seen as too slow, and not enabling enough country ownership. The government-centric focus of these projects was also seen as constraining impact and sustainability (see 1997 World Bank Health Strategy).The GF was intended to keep the virtues but not the vices of the “old model”: impact and capacity building elements, but with faster and more flexible money. And it was intended to bring a broader set of actors into the process of grant preparation and implementation, including specifically both the commercial sector and civil society.In the early days of GF activities, job one was to turn on the tap. The challenge of quickly starting up an agency with such a large and complex mandate was met however, and they successfully mobilized and allocated substantial new funding for the three diseases. Together with PEPFAR and PMI, they have mobilized “game changing” volumes of funding for AIDS, malaria and, somewhat, for TB. From 2000, donor funding for AIDS increased by an average of 24% annually, to reach, in 2007, US$6.6 billion, with US$1.2 billion (18%) being channeled through the GF. (See Evaluation - Study Area 3 Report.)At the country-level though, the evaluation found problems. The evaluation did find more country ownership, mostly a result of their Country Coordinating Mechanisms (CCMs) model for grant preparation. And the participation of civil society organizations has expanded. The commercial private sector however has not, neither as supporters or co-investors, nor as participants in implementation.But the new money and the new model was intended to be a game-changer for results, not just processes. And therein lays the big problem.The GF was created as a “financing only” entity. In contrast to “old model” funders, the GF was set up to rely on the contributions of “partners” at the global and country level to ensure that grant proposals are well prepared (that is, the most effective strategies are identified, and the best means of implementing them are identified and planned for); and well implemented (this means that implementation has to be supervised and problems identified and resolved quickly).The GF model was designed to ensure more country ownership, but explicitly did not deal with the “old model” weaknesses related to project preparation or implementation. Many of those weaknesses were well known long before the Global Fund came along. In 1999, the development effectiveness review, found that weak performance in health projects was strongly undermined by having too little analytical work to inform project content and too little country presence during implementation of project activities. The report proposed that the Bank substantially ramp up these activities to improve the impact of their health projects, especially in low capacity countries. During the discussions of these recommendations, participants noted that sometimes there is a direct trade off between “doing the right project” (in the sense of using the best strategies for pursuing project goals) and country ownership – another area where the report suggested the Bank needed to improve.Relative to the “old” development bank model, the Fund went in the other direction; the GF model leaves the technical content of grant preparation to ad hoc arrangements and contributions of partners in each country. It also leaves oversight of implementation to the CCM. Neither is found to be predictably working.

"The Global Fund's policies regarding country level oversight responsibilities often require capacities that do not exist. For example, the Fund's expectations that CCMs could coordinate with Principal Recipients, and work with partners, and the GF Secretariat, to identify grant implementation bottlenecks, is very rarely met." (p 66 Study Area Two)

The evaluation finds these partners (usually bi-lateral agencies, the WHO and the development banks) are occasionally able to step-in and provide needed technical support and input; often they aren’t. And they all complain that their contributions, insufficient as they are, constitute an unsustainable “unfunded mandate” that is integral to effectiveness of GF funded activities, but not acknowledged and not provided for with formal support. And they note, they are all pretty stretched just getting their own work done.The global community increasingly is aware of this weakness in the GF model of support. And actors like PEPFAR and Stop TB are taking steps to address it through provision of on-demand TA or program reviews. But the evaluation makes clear these arrangements are not nearly adequate to ensure grants are well prepared and implemented. Some participants in these discussions suggest that the weak arrangements in this arena may be compensated for by the strong performance pressures created for GF supported activities by the Performance-Based-Funding (PBF) framework.Unfortunately, the evaluation finds the PBF framework does not yet function to predictably provide these performance pressures for programs and GF funded activities. The evaluation rightly commends the PBF for its aspiration. Who could argue with focusing on delivering results? But for now, the evaluation finds the results measured and linked to funding are process indicators (e.g. staff trained) and outputs (e.g. bednets distributed).

“Though plans include impact measures, reports reported on numbers of people trained, numbers of materials produced, numbers of supervisory visits conducted, to demonstrate performance, that is far removed from the outcome-level data originally anticipated in the PBF model”. (p 71, Study Area Two)

The “old model” funders have used this kind of production-based funding for decades.The fundamental problem is that performance for continued funding is assessed 2 years into grant activities, while outcomes and impact are only measured from the 3rd year. That is, decisions about continued funding are made BEFORE information on outcomes or impact is available, effectively nullifying any incentives to achieve results in order to ensure continued funding. Ready, fire, aim.It is not a “brave new model” for health development assistance to spend money and get people trained, or products distributed. The old models delivered that too (see 1999 World Bank development effectiveness review). The sticking point for health development assistance is, and always has been, ensuring those trained health workers actually deliver services that people use; and that those products get used properly, not just handed out. And most importantly, the challenge has always been to ensure that the services and products actually contribute to health improvements, especially for the poor. The evaluation finds that the GF hasn’t gone farther than the “old model” aid providers in cracking this nut. The evaluation found that the system “as a whole does not sufficiently demonstrate linkages between measured grant performance and financing decisions". (p 143, Study Area Two)Efficiency, sustainability, quality and equity are still missing-in-action. The Study Area 2 report presents findings from 16 detailed country studies that assessed how the RBF is working on the ground. They found no performance indicators linked to funding which measured service quality, efficiency, sustainability, or equity (in terms of gender, income, or vulnerability). Yes, we know a lot of stuff was delivered (e.g. ART treatment, nets) and that is encouraging; but we have no way of judging whether the strategies used were good value for money, whether they reached the poor or those most in need, or whether the services were of the quality required to achieve disease impact results. Nor do we know whether the non-medical items were used for their intended purposes. It’s not uncommon for bednets to be used for fishing nets or in other ways (see Minikawa et al 2008).The evaluation found (see Study Area Two report, pp72-73) that monitoring of service quality and gender and income equity are major gaps in GF supported activities. Although the Fund clearly articulates the principle that grants should improve service quality, improve gender equity, and target vulnerable groups, performance monitoring is not explicitly linked to any of these principles. A review of 93 country grant proposals showed that monitoring of service quality is a particular gap: although 44% and 55% of grants had gender and vulnerable group indicators, respectively, only 5% had any indicators for service quality.This review shows that the majority of proposals are approved for funding without inclusion of even a single service quality indicator. The lack of service quality indicators included in grant proposals is linked to the GF’s M&E Toolkit, which is the main guide for selecting and measuring indicators. In the Toolkit, none of the top ten indicators for routine reporting address issues of service quality, gender equity or targeting of the poor; neither do any of the top ten indicators for medium-term outcome and impact.I’m not suggesting these things are easy; they aren’t. But until the GF model is changed so that these shortcomings can be explicitly addressed, there is simply no way of knowing if the funds flowing through it are well spent.The evaluation team recommended three actions which struck me as being so critical that additional funding to the GF should be conditional on their implementation:

1. Ensure an adequate level of technical input into grant proposal preparation (including analysis of relevant aspects of health system, consideration of alternative strategies, identification of likely barriers to implementation, and a logical framework linking the actions taken to impact). The GF evaluation strongly recommends that action be taken to “fill the gap” in grant proposal preparation and supervision that is done, ineffectively on a “partnership basis.”
2. Change the timing of decisions on continuation of funding, to allow the decisions to be based on outcome and impact information (e.g. delay until year 3 of grant activities).
3. Amend the Performance-Based Funding framework to link funding to outcome and impact indicators for PBF; this necessitates consistently taking baseline measures of indicators, and linking funding to improvements for which data will be available from population and independent facility surveys.

The Global Fund is a young organization. This evaluation confirms a serious intent to be a “learning organization.” The priority actions recommended by the evaluation team will require some major changes, but they are well-substantiated by the analysis. The GF, its funders and board members have some serious work ahead if they are to turn the existing arrangements into a real new model for health development assistance. Let’s wish them luck, and be generous with the hard won lessons of other development assistance efforts that have also fallen short of lofty goals.