Seeking Results from PEPFAR’s Orphans and Vulnerable Children Programs

PEPFAR deserves to be commended for its efforts to define key measurable outcomes for its orphans and vulnerable children (OVC) portfolio. Approximately 17 million children worldwide have lost one or both parents due to HIV/AIDS. In response, PEPFAR has earmarked 10% of its annual program funds to help mitigate the psychological and developmental effects this loss can have on children. This is serious money – totaling $2 billion over the last decade or so. While PEPFAR reports that its services have reached over 4.5 million children in the last year alone, less is known about what results those scarce resources have actually yielded. Until recently, in fact, there has been little consensus on just what results OVC programs are trying to achieve.

As such, PEPFAR’s efforts to define outcomes of interest for its OVC programming are a big step forward for increasing learning and accountability. It now falls to USAID, which administers the vast majority of PEPFAR’s OVC funds, to evaluate how well its programs are achieving those outcomes.  Doing so will 1) help those working to support HIV-affected kids and families know more about what kinds of interventions work, and 2) help stakeholders understand what they are getting for their money. 

To date, PEPFAR has mainly relied on an output measure to track its OVC programs (the number of OVC reached with services like psychological/spiritual support, access to health care, and education/vocational training, to name a few).  Output measures like these are useful for monitoring project implementation on an ongoing basis (i.e. ensuring, at a minimum, that implementing partners are doing the work they’re contracted to do).  But there are limits to what it can tell us, such as:

1. What About Service Quality? Under pressure to achieve output targets, there is a risk that implementing partners focus more on simply “reaching” clients than they do on the quality of service provision.
    
2. Referrals Don’t Necessarily Mean Services Delivered. OVC service providers tend to do a lot of referrals (to health facilities, to legal services, etc.). These referrals are important and are counted as services provided in and of themselves, but they’re a pretty small part of the story. True benefits only occur if the referral is actually completed. However, tracking whether a client follows through is often challenging since the service provider has little incentive (and usually lacks an established system) to provide evidence of service delivery. 
    
3. Are OVC better off as a result of services provided? This is the fundamental issue and is tightly linked to the two preceding points. It’s especially important for OVC care and support, where the evidence base on the link between services and outcomes is mixed.  (PEPFAR’s most recent guidance on OVC programming provides a nice review of the areas in which there are specific interventions with stronger and weaker evidence of results. And there have been several evaluations of PEPFAR OVC programs themselves, though of varying methodological strength). 

All of this suggests that PEPFAR should measure outcomes more systematically and robustly. As a necessary first step, it should clearly define the outcomes that its OVC programming is trying to achieve. This isn’t new or earth shattering. The Institute of Medicine (IOM) made exactly this point in its PEPFAR evaluation released earlier this year:

“To improve the implementation and assessment of nonclinical care and support programs for adults and children, including programs for orphans and vulnerable children, the Office of the U.S. Global AIDS Coordinator should shift its guidance from specifying allowable activities to instead specifying a limited number of key outcomes.”

CGD’s recently released report, More Health for the Money, made the same point. Indeed, a more intentional shift from defining and measuring services toward defining and measuring outcomes could allow greater flexibility for implementing partners to experiment with different approaches to achieve results. This could include things like cash transfers, which some studies show are associated with delayed sexual debut and increased secondary school enrollment for OVC beneficiaries.

As it turns out, PEPFAR is well on the way toward implementing this recommendation. Earlier this year—PEPFAR developed 12 “Core Indicators for OVC Program Evaluation” to measure child and household outcomes, along with methodological guidance on collecting them.  Establishing this focused set of outcomes – combined with a call for increased attention to robust evaluation design and adequate funding for evaluation efforts – is a solid step toward more systematically understanding PEPFAR’s return on investment for its OVC programming.

Of course, these indicators are only useful if used.  At this time, they’re not required reportable indicators like PEPFAR’s service provision indicator.  And they’re the kind of indicator that can only be collected through special studies, not through ongoing monitoring. So the onus is now on USAID to redouble its efforts to make sure the right studies—focusing on these core outcomes—are done.  Fortunately, there are reasons to believe we may see more of these soon: USAID now requires that its larger projects are subject to evaluations, and I understand that PEPFAR is exploring the idea of making some of these outcomes required reporting.  All of this points to a real opportunity for USAID to bolster the evidence base for OVC programming.  At the end of the day, if PEPFAR is to spend 10% of its large pot of money in this area, then it should evaluate whether its programs are achieving the desired results, publish the findings to increase accountability, and apply learning to improve future programming.  PEPFAR and USAID are headed in this direction—let’s see them keep it up.