Institute of Medicine Pushes PEPFAR on Data Collection, Disclosure

The Institute of Medicine, the prestigious health arm of the National Academy of Sciences, has weighed in with a massive report on the President’s Emergency Plan for AIDS Relief (PEPFAR), the multibillion dollar US effort to confront the epidemic in the developing world. The evaluation validates PEPFAR’s enormous reach during its first 10 years and identifies concrete actions that Congress and PEPFAR should take for the program to become more sustainable moving forward. Like other PEPFAR-watchers, I’ve spent many hours digesting the landmark 700+ page evaluation and I’m pleased to see the detailed and rigorous attention (96 pages!) that the IOM has paid to PEPFAR’s data collection and knowledge management.  I am happy that the IOMs recommendations align with those previously put forth by the Data Working Group (DWG) of PEPFAR’s Scientific Advisory Board (SAB), on which I serve. Could this consensus lead to action? Having conducted a large-scale evaluation of this scope, the IOM is  uniquely positioned to understand both the strengths and gaps in PEPFAR’s data systems – after all, the authors can only offer “evaluation” to the extent that there is appropriate data to support their analysis and conclusions. Indeed, the IOM is up front about the implications of limited data throughout the report, particularly with respect to financial (p. 92), monitoring (p. 523) and outcome/impact (p. 64 & p. 527) data – data limitations which motivated the chapter on knowledge management, an area originally outside of the evaluation’s Congressionally-define scope (see p. 514). The DWG observed similar challenges with knowledge management and presented preliminary recommendations to Ambassador Goosby at an October 2012 SAB meeting.  It’s worth noting up front that PEPFAR has already begun to take meaningful steps toward improved financial data collection – like institutionalizing expenditure analysis into routine annual reporting.  But of course, there is more to be done. So with an eye towards both reports, here are four recommendations where there seems to be agreement and feasible opportunities to improve PEPFAR’s collection, utilization, and dissemination of data and knowledge. 1.       Establish and maintain a PEPFAR public access knowledge portal While PEPFAR’s current website is an invaluable resource for those interested in the USG contribution to the fight against AIDS, public accessibility of PEPFAR data and documents does not compare favorably to those of its partners and foreign counterparts (think DFID, the Global Fund, and the World Bank) – nor has it lived up to the spirit of the Obama administration’s Open Government Initiative. In fact, since 2007 the scope of PEPFAR’s annual public reporting has declined substantially – from a detailed 216 page report in 2007 to a bare-bones 5-page report for 2012 (shorter than 2007’s executive summary!).  Still, as the IOM points out, “PEPFAR has the opportunity to play a significant leadership role in making HIV/AIDS monitoring, evaluation, and research data available to other researchers, evaluators, and the public, so that these data are used for the greatest public health impact and to accelerate the pace of new knowledge creation” (p. 586). Based on the principles that data are a public good (that allows learning by others) and that data should strengthen transparency and accountability, both the IOM and DWG recommend that PEPFAR establish and maintain a public access portal which offers PEPFAR-funded reports, publications, and financial and programmatic data. Further, to the joy of researchers everywhere, both reports echoed the importance of making data available in a timely manner for use by external evaluators and researchers to facilitate analysis at a more granular level. 2.       Strengthen and streamline HIV/AIDS program indicators In examining PEPFAR’s data systems, the DWG found that huge quantities of data are being collected, at a high cost and burden to OGAC and implementers alike (to get an idea, see the picture below which shows the incredible complexity of the data flow process). Yet, it appears that that indicators are often of limited utility, either due to low epidemiological or programmatic relevance (i.e. inputs or “trainings”), or because the reported data is incomplete or unreliable.  Similarly, the IOM report finds that “PEPFAR’s current indicators do not capture sufficient information on its stated prioritized goals and activities and are focused primarily on inputs and outputs. As a result, the program monitoring system has limited utility for determining the effectiveness of PEPFAR’s effort” (p. 508).

Source: Slide 21

Based on these findings, both reports recommend that PEPFAR streamline its indicator list, and data collection and reporting process. This could be done by commissioning an external study on its indicators system, including a cost/benefit analysis of the value and cost of each indicator, and then using that report to identify a small set of “Tier 1” indicators that should be reported up to headquarters (as the IOM report recommends). Tier 1 indicators would stay consistent over time, be harmonized with existing global and/or national indicators, and form the basis of Congressional reporting on PEPFAR’s portfolio-wide achievements and epidemiological trends. 3.       Require each future grantee and contractor to submit a standardized, realistic, and contractually binding “Data Management Plan” that conforms to specific PEPFAR guidance. Currently, most contractors are required to submit some sort of “performance monitoring plan” or a similar document as part of the grant proposal process. Typically, these documents outline a definition of each performance indicator; the source, method, frequency and schedule of data collection; and targets for each performance indicator. But because PEPFAR has not historically provided centralized guidance for the preparation of such documents, they have been limited in standardization, scope, enforceability, and utility beyond “indicators.” For example, such documents rarely account for the collection, archiving, and sharing of patient or facility-level data, such as on patient retention and outcomes. Accordingly, the IOM report recommends that PEPFAR “develop systems and processes for routine, active transfer and dissemination of knowledge both within and external to PEPFAR” (p. 594). One way to do this (as the DWG recommends) is for PEPFAR to produce clear, centralized guidance for Data Management Plans, which would focus on 1) high-quality data collection and reporting for PEPFAR’s essential indicators, and 2) implementers’ internal collection and management of patient-level data for their own purposes, including the expectations for public access to such data. 4.       Expand the collection, application, and sharing of budget, expenditure and cost data through the Expenditure Analysis initiative The lack of available financial data for PEPFAR has been a longstanding frustration for those who want to understand the distribution and use of PEPFAR funds (see here, for example). But it’s not just a transparency issue – there’s some evidence that even internally, PEPFAR has not historically been able to track and manage its cost-structure below the prime partner level. In recent years, PEPFAR has worked proactively to better understand and improve the cost structure and cost-effectiveness of its programs, most ambitiously through the Expenditure Analysis (EA) initiative, which released a pilot report from its first six countries in July 2012. While EA itself was not a focus of the IOM report, the need for better financial data was repeatedly emphasized. The authors’ analysis itself was limited by data gaps in this area, for example on annual expenditures (p. 107) and PEPFAR’s prime and sub- implementing partners (p. 123). And the IOM reported implications of this limited data for program management and accountability:  “partner country governments have been frustrated by not knowing where the PEPFAR money was going. Some interviewees wondered how the government could be expected to hold implementers accountable if they did not know where the money was going” (p. 399). Accordingly, the IOM recommends that PEPFAR collect and report financial data that is more closely aligned with programmatic data and program implementation (p. 92). Further, the DWG urged PEPFAR to share the relevant (albeit anonymized) underlying EA data more widely, particularly with partners and country governments, who must understand the financial implications and cost structure of HIV/AIDS programs if they hope to assume direct responsibility for program implementation. To be clear, this is not a comprehensive list of all IOM recommendations for knowledge management; beyond what’s discussed above, the IOM also considers issues such as PEPFAR’s evaluation policy and permitted research methodologies.  Nonetheless, these four focus areas provide perhaps the most feasible, practical, and meaningful opportunities to improve PEPFAR’s collection, utilization, and dissemination of data and knowledge, ultimately helping us to better understand the HIV/AIDS epidemic and improve our programs. I am optimistic that this apparent consensus between the IOM and the DWG will lead to real action. Let’s get started!