With rigorous economic research and practical policy solutions, we focus on the issues and institutions that are critical to global development. Explore our core themes and topics to learn more about our work.
In timely and incisive analysis, our experts parse the latest development news and devise practical solutions to new and emerging challenges. Our events convene the top thinkers and doers in global development.
With shifting disease burdens, growing populations, and rising expectations comes a greater focus on value for money. International health funders and agencies want to know how to make the most of money spent by focusing on the highest impact interventions among the most affected populations. Whether through better procurement systems for health commodities, results-based financing, or more detailed assessments of the effective ness of health technology, CGD’s work aims to make health funding go further to save, prolong and improve more lives.
Earlier this month, the African Development Bank held its first capacity building program on value for money in social service delivery in Dar es Salaam, Tanzania, following the Tunis Declaration on value for money signed by some 50 African Ministers of Finance and health on July 5, 2012.
For this capacity-building program, I was privileged to participate as a speaker in a session on trade-offs in allocation – by virtual pre-recorded webinar (see video here and slides here).
My presentation reflected the chapter on Planning Allocations in CGD’s recent report, More Health for the Money. While our report’s recommendations were addressed to the Global Fund, they could be adopted by any funding agency including a national government.
The summary points of our chapter, addressed to a global-health funder, are to:
(1) Choose the most effective and cost-effective interventions and commodities;
(2) Focus on hot spots and key modes of transmission of an epidemic;
(3) Improve budgeting and its transparency to reduce duplication among actors; and
(4) Optimize the mix of interventions subnationally using epidemiologic and economic intelligence (see also this blog post on the importance of disease modeling).
Frankly it was my first webinar, and I feel a bit shy revealing my disembodied voice. What do you think? Do you agree or disagree with our recommendations? Are more pre-recorded webinars like this worthwhile, and should we do more of them for the other chapters of our report? Feedback welcomed!
Victoria Fanis a research fellow and health economist at the Center for Global Development.She thanks Nejmudin Bilal of AfDB for this opportunity, Amanda Glassman for helpful comments and Jenny Ottenhoff, Aaron King, and the CGD IT team for making the webinar possible. You can follow Victoria Fan on Twitter at@FanVictoria.
This blog is the first in a series of three on the quality of PEPFAR’s HIV treatment programs.
Counting the number of patients on treatment is no longer enough. For years even the friendliest critics of the global struggle against AIDS have pointed out that this metric unfairly neglects the people who are not put on treatment and then die, largely because their deaths are uncounted except in so far as they increase the treatment “coverage rate.” This diverts attention from the challenge of assuring that patients are retained on treatment and remain alive and healthy, rather than failing treatment and dying, sometimes after only a few months.
By adopting the “HIV treatment cascade” as its theme for the October 2 meeting of its Scientific Advisory Board meeting, PEFPAR has signaled its willingness to be judged by a much more comprehensive metric of its on-the-ground success. A vivid metaphor, the “AIDS treatment cascade” provides a snapshot assessment of the quality of AIDS treatment across the stages of care, and shows that there is room for improvement in Africa. (Also see WHO’s discussion of the “cascade” in Chapter 3 of their Global update on HIV treatment 2013.)
The following figure compares the AIDS treatment cascade in the United States (as it was presented and explained in this blog from the Centers for Disease Control) to estimates of the cascade in Africa that were synthesized for the SAB by Elvin Geng and Thomas Odeny.
Figure 1. Cumulative loss of patients in the AIDS treatment cascade in the United States and in Africa. In Africa, the proportion of patients with viral suppression is unknown.
Source:US data is from CDC’s June 2013 “Today’s HIV/AIDS epidemic”. Africa estimates are derived by the authors from Elvin Geng’s PEPFAR SAB presentation (October, 2012) which credits Rosen S, Fox MP (2011) Retention in HIV Care between Testing and Treatment in Sub-Saharan Africa: A Systematic Review. PLoS Med 8(7): e1001056. , Fox MP, Rosen S (2010) Patient retention in antiretroviral therapy programs up to three years on treatment in sub-Saharan Africa, 2007-2009: systematic review. Trop Med Int Health 2010-Jun:15 Suppl 1:1-15.
A deeper look at the figure shows that most of the difference between the US and Africa occurs at the first stage in the cascade, that of diagnosis. The US health system is estimated to detect and diagnose 80% of its HIV infected population of approximately 1.1 million, while Elvin Geng estimates that in the recent past the 40-some African countries have managed to detect only about 40% of the continent’s much larger mass of 26 million HIV positive people. (In view of PEPFAR’s recent dramatic scale-up of HIV testing, this first stage of the cascade may see a rapid improvement in coming years.)
Given the difficulty of testing the entire African population for HIV, a useful alternative perspective to the cumulative approach depicted in Figure 1 is to independently judge each stage shift, by calculating the proportion of patients from each stage who don’t make it to the next one (See Figure 2).
Figure 2. Incremental loss of patients between one stage and the next in the AIDS treatment cascade in the United States and in Africa. Sources: Same as Figure 1.
Source: US data is from CDC’s June 2013 “Today’s HIV/AIDS epidemic”. Africa estimates are derived by the authors from Elvin Geng’s PEPFAR SAB presentation (October, 2012) which credits Rosen S, Fox MP (2011) Retention in HIV Care between Testing and Treatment in Sub-Saharan Africa: A Systematic Review. PLoS Med 8(7): e1001056. , Fox MP, Rosen S (2010) Patient retention in antiretroviral therapy programs up to three years on treatment in sub-Saharan Africa, 2007-2009: systematic review. Trop Med Int Health 2010-Jun:15 Suppl 1:1-15.
Still, Figure 2 makes clear that Africa’s treatment cascade faces severe challenges at two other important stages, that from diagnosis to care, where Africa loses 41% compared to the US’s 23%, and that from initiation to retention, where Africa loses 30% compared to the US’s 9%.
As we absorb the implication that HIV/AIDS treatment is losing vast numbers of patients, one immediate question elicited from this data is if the data is accurate enough to warrant alarm.
For example, another presentation at the SAB shows that patient retention in Africa may actually be better than estimated, because some of the patients who have not returned to an individual facility may have continued their care at a different facility. The researchers sent out health personnel on motorcycles to look for the patients who were unaccounted for. This effort confirmed that facility’s retention rates are indeed biased downward by the inability of patient records systems to track transferring patients. In facilities whose own records showed a retention rate of 60% after three years, the corrected estimate was about 80% retention or about one-third better [ (80%-60%)/60% = 1/3 ]. (See Table 1 below.)
Source: Geng EH, Glidden DV, Bwana MB, Musinguzi N, Emenyonu N, et al. (2011) Retention in Care and Connection to Care among HIV-Infected Patients on Antiretroviral Therapy in Africa: Estimation via a Sampling-Based Approach. PLoS ONE 6(7): e21797.
Still, there is great variation in treatment retention across facilities. Figure 3 shows the two year retention in each of five specific facilities spread over three study countries. Here, the retention one year after antiretroviral therapy (ART) initiation is best at Site 2 (slight over 90%) and worst at Site 5 (roughly 70%). At two years after ART initiation, Site 4 (slightly over 90%) retains its proportion over the course of a year, while Site 5’s proportion declines to lower than 65%.
Source: Presentation by Odeny at PEPFAR SAB (October 2-3, 2013) on behalf of Elvin Geng , Odeny and other investigators in the East Africa IeDEA consortium.
On the other hand, Vivek Jain of UCSF showed that in some instances the retention rate at six and twelve months could be in the high-nineties. So a patient’s chance of successful treatment appears to vary dramatically and unacceptably according to the facility that patient attends.
We must bear in mind how extraordinarily well Africa is doing now compared to ten years ago when the US launched the PEPFAR program. At that time virtually 100% of HIV infected Africans went undiagnosed, un-linked, un-initiated and un-retained. So the international campaign to expand treatment in Africa has achieved a virtual miracle in providing the opportunity for treatment to so many.
And it is a welcome sign of the maturity of the United States PEPFAR program that its management has chosen to shine a spotlight on the broken links in the continuum of AIDS care in Africa.
Clearly, a 11% cumulative success rate is unacceptable if it means that 89% of HIV infected people fail to benefit from proffered treatment.
Even if a proper measure of retention in treatment yields a less pessimistic estimate of the attrition in the treatment cascade, the enormous loss of patients throughout the treatment cascade remains a glaring reminders of the need to better understand the causes of patient loss at every stage in the cascade and how these losses can be stemmed.
So how can retention be improved throughout the cascade? Stay tuned. In my next blog I will report on other sessions from the October 2-3 SAB meeting that addressed this question.
More than ever, global health funding agencies must get better value for money from their investment portfolios; to do so, each agency must know the interventions it supports and the sub-populations targeted by those interventions in each country. In this study we examine the interventions supported by two major international AIDS funders: the Global Fund to Fight AIDS, Tuberculosis, and Malaria (‘Global Fund’) and the President’s Emergency Plan for AIDS Relief (PEPFAR).
It appears that the worst kept secret in Washington is out: Ambassador Goosby is expected to step down as Global AIDS Coordinator later this year. As CGD has done for similar leadership transitions, we are working on a report to examine the future direction of PEPFAR and consider which tasks PEPFAR’s next leader should put near the top of the program’s list of priorities. One preliminary conclusion: Goosby’s successor will certainly face programmatic challenges, but the political ones may prove to be more difficult.
On the programmatic side, Ambassador Goosby did a phenomenal job strengthening the evidence base behind PEPFAR’s work, in part by scaling up cost-effectiveness studies and instituting the first Scientific Advisory Board (both of which should continue). There’s more to do. Goosby’s successor should make more progress on data management and dissemination and take steps to better integrate PEPFAR’s programs with other donor- and country-led activities. He or she should also deploy innovative tools (like performance-based financing) to make programs more efficient, effective, and sustainable.
On the political side, PEPFAR, like all development projects, will continue to face budget pressures. That reality demands getting more value for money out of PEPFAR’s investments. But it also requires maintaining political support in a landscape very different from what previous leaders of PEPFAR have faced.
Take the figure below: The first two graphs show votes in the House of Representatives during PEPFAR’s authorization in 2003 and 2008. The third shows the current composition of the House based on those who have voted in the past. Nearly half of the current members of congress weren’t around for PEPFAR’s past two authorizations and thus don’t have the same knowledge of or commitment to the program.
With so much change in the House, PEPFAR may soon no longer benefit from the congressional authorization that has set the program apart from other US development programs for the past decade and has been responsible, at least in part, for its robust funding. Even if pending legislation to extend the programs authorization another five years is passed, it won’t generate the same level of congressional commitment that a full reauthorization process would.
Predicting the exact effects of changing politics on PEPFAR’s future is not possible, but it is clear that the next Global AIDS Coordinator will have to make the case for PEPFAR and be able to exercise considerable political muscle to maintain the kind of support that has served the program well for so long. It’s a tall order in need of ideas and dialogue. That’s why we have convened members of the policy community to inform a short report that outlines essential and actionable priorities for the next US Global AIDS Coordinator. Watch this space for more in the coming weeks and months.
PEPFAR deserves to be commended for its efforts to define key measurable outcomes for its orphans and vulnerable children (OVC) portfolio. Approximately 17 million children worldwide have lost one or both parents due to HIV/AIDS. In response, PEPFAR has earmarked 10% of its annual program funds to help mitigate the psychological and developmental effects this loss can have on children. This is serious money – totaling $2 billion over the last decade or so. While PEPFAR reports that its services have reached over 4.5 million children in the last year alone, less is known about what results those scarce resources have actually yielded. Until recently, in fact, there has been little consensus on just what results OVC programs are trying to achieve.
As such, PEPFAR’s efforts to define outcomes of interest for its OVC programming are a big step forward for increasing learning and accountability. It now falls to USAID, which administers the vast majority of PEPFAR’s OVC funds, to evaluate how well its programs are achieving those outcomes. Doing so will 1) help those working to support HIV-affected kids and families know more about what kinds of interventions work, and 2) help stakeholders understand what they are getting for their money.
To date, PEPFAR has mainly relied on an output measure to track its OVC programs (the number of OVC reached with services like psychological/spiritual support, access to health care, and education/vocational training, to name a few). Output measures like these are useful for monitoring project implementation on an ongoing basis (i.e. ensuring, at a minimum, that implementing partners are doing the work they’re contracted to do). But there are limits to what it can tell us, such as:
What About Service Quality? Under pressure to achieve output targets, there is a risk that implementing partners focus more on simply “reaching” clients than they do on the quality of service provision.
Referrals Don’t Necessarily Mean Services Delivered. OVC service providers tend to do a lot of referrals (to health facilities, to legal services, etc.). These referrals are important and are counted as services provided in and of themselves, but they’re a pretty small part of the story. True benefits only occur if the referral is actually completed. However, tracking whether a client follows through is often challenging since the service provider has little incentive (and usually lacks an established system) to provide evidence of service delivery.
Are OVC better off as a result of services provided? This is the fundamental issue and is tightly linked to the two preceding points. It’s especially important for OVC care and support, where the evidence base on the link between services and outcomes is mixed. (PEPFAR’s most recent guidance on OVC programming provides a nice review of the areas in which there are specific interventions with stronger and weaker evidence of results. And there have been several evaluations of PEPFAR OVC programs themselves, though of varying methodological strength).
All of this suggests that PEPFAR should measure outcomes more systematically and robustly. As a necessary first step, it should clearly define the outcomes that its OVC programming is trying to achieve. This isn’t new or earth shattering. The Institute of Medicine (IOM) made exactly this point in its PEPFAR evaluation released earlier this year:
“To improve the implementation and assessment of nonclinical care and support programs for adults and children, including programs for orphans and vulnerable children, the Office of the U.S. Global AIDS Coordinator should shift its guidance from specifying allowable activities to instead specifying a limited number of key outcomes.”
CGD’s recently released report, More Health for the Money, made the same point. Indeed, a more intentional shift from defining and measuring services toward defining and measuring outcomes could allow greater flexibility for implementing partners to experiment with different approaches to achieve results. This could include things like cash transfers, which some studies show are associated with delayed sexual debut and increased secondary school enrollment for OVC beneficiaries.
As it turns out, PEPFAR is well on the way toward implementing this recommendation. Earlier this year—PEPFAR developed 12 “Core Indicators for OVC Program Evaluation” to measure child and household outcomes, along with methodological guidance on collecting them. Establishing this focused set of outcomes – combined with a call for increased attention to robust evaluation design and adequate funding for evaluation efforts – is a solid step toward more systematically understanding PEPFAR’s return on investment for its OVC programming.
Of course, these indicators are only useful if used. At this time, they’re not required reportable indicators like PEPFAR’s service provision indicator. And they’re the kind of indicator that can only be collected through special studies, not through ongoing monitoring. So the onus is now on USAID to redouble its efforts to make sure the right studies—focusing on these core outcomes—are done. Fortunately, there are reasons to believe we may see more of these soon: USAID now requires that its larger projects are subject to evaluations, and I understand that PEPFAR is exploring the idea of making some of these outcomes required reporting. All of this points to a real opportunity for USAID to bolster the evidence base for OVC programming. At the end of the day, if PEPFAR is to spend 10% of its large pot of money in this area, then it should evaluate whether its programs are achieving the desired results, publish the findings to increase accountability, and apply learning to improve future programming. PEPFAR and USAID are headed in this direction—let’s see them keep it up.
Our recent report, More Health for the Money, aims to answer the question: How can the Global Fund save more lives with the billions it spends each year to combat AIDS, TB and malaria? So at our recent launch event, we put this question to a panel of experts (and a room full of practitioners and policy makers) to highlight where progress on value for money is being made, and where room for improvement remains.
The event took place on the sidelines of the UN General Assembly in New York City and featured remarks from Christoph Benn and Shu-Shu Tekle-Haimanot from the Global Fund, Stefano Bertozzi from UC Berkeley, Julia Martin from PEPFAR, and Mphu Ramatlapeng from the Clinton Health Access Initiative. The discussion covered a lot of ground and tracked closely to the four main chapters of our report – allocation, contracts, costing and spending, and verification. So I’ve curated a few of my favorite sound bites that highlight the progress and potential in each area.
Allocation: Julia Martin discussed how increased transparency around the allocation of funds is critical to country ownership:
“Allocative efficiency is a very good place for talking about how country ownership can fit. It’s in the transparency on how you’re allocating all of the available resources to promote a good response for HIV…You need to look at donor resources and countries own resources to determine where the different funding streams are maximally used for the best value...but to intentionally have that discussion…you must have a forum and you must have a commitment to using that forum and not just as a check box that says ‘we’ve spoken to everyone in the country’. It needs to be real. And it needs to be transparent.”
Contracts: Stefano Bertozzi discussed the need for financing mechanisms that create incentives for results:
“One of the things sketched out in your cartoon, is that it would be nice to have funding mechanisms that have inbuilt incentives to drive efficiency. Unfortunately, it’s not clear to someone who is about to make a contract with a country what that mechanism is and how it should work. So we need investment in the design, testing, and diffusion of mechanisms that have better incentives build in.”
Christoph Benn also mentioned that the Global Fund is piloting the Cash-on-Delivery aid model in some of its grants (which we’re thrilled to hear, and look forward to seeing more on):
“We are exploring COD, we have one pilot going on in Mesoamerica, it’s an interesting one where you really pay based on the results achieved, not on the input. We’re exploring currently if we can introduce that in Rwanda on a much bigger scale.”
Costing and Spending: Christoph Benn discusses changes at the Global Fund to help reduce the costs of bed nets:
“We had a very passive approach – it was part of our philosophy of country ownership…– and it meant that we did not leverage at all our significant purchasing power. That is changing now...we are working very closely with all the other major players – PMI, DIFID, UNICEF, the World Bank and others – and…tendering for these commodities together and simplifying the process… to drive the prices down with bed nets. We’ll see similar developments with drugs and other commodities. So that’s something that we’ll focus even more on in the future, but the first steps have been taken.”
To this point I’d add my hope that they focus on the most cost effective commodities, not just the lowest cost.
Verification: Christoph Benn highlighted the Global Fund’s new results, and acknowledged that they only capture program inputs rather than health outcomes:
“Now you can ask yourself, ‘did this guy not listen to us? We don’t want these kinds of input indicators, we don’t want to hear about how many bed nets you’ve distributed, we want to hear how many children are sleeping under bed nets…’ We did get that message...We’re not shifting that quickly….But you’re absolutely right. In the future we’re going into more outcomes indicators and we’ll report on that.”
And the final word goes to Julia Martin, who said "We aren't finished stretching our dollars - not even close." I was particularly glad to hear that, and will be watching (and chiming in with advise) as the Global Fund and their partners continue working to improve their return on investment in health around the world.