GAVI Moves on Better Data Verification

This is a joint post with Kate McQueston. 

Data quality and rigorous measurement is important for any funder using performance-based or results-based aid. Poorly measured or self-reported data are often subject to major biases. Indeed, recent CGD research by Justin Sandefur and Amanda Glassman found a clear increase in over-reporting of DTP3 vaccination after GAVI introduced a now defunct pay-for-performance program (its immunization services support program) in the early 2000s.  Thus, strengthening systems to verify data is important and increasingly feasible; recent experience from the World Bank’s HRITF suggests that independent verification of data isn’t overly expensive.

Image

So we’re encouraged by the GAVI Alliance’s new application guidelines, which outline strengthened requirements for data verification of the immunization outcomes used for performance payments as part of its Health System Strengthening (HSS) support. Per the new guidelines, countries must meet both performance goals and “checks and balances for data verification based on WHO/UNICEF estimates, independent assessments of the quality of administrative data, and periodic household surveys.”

Specifically, all GAVI grants will require that countries have a mechanism to independently assess the quality of administrative data and to monitor data quality over time. In addition, GAVI will require household surveys to be administered at certain frequencies (two surveys every five years). If a country doesn’t have independent data quality assessment mechanisms, parts of GAVI’s HSS grants can help cover these activities. Discrepancies between coverage estimates greater than 5% require an explanation form the recipient and a plan for improving M&E systems. 

These new guidelines are an improvement on GAVI’s previous M&E requirements from 2013, which are not specific and state only general intentions, such as: “Performance payments will be based on... administrative data, with…estimates and surveys used for data verification” and “Countries with discrepancies are encouraged to invest in strengthening data quality and routine information”. By contrast, the new guidelines specify the frequency of independent surveys and thresholds for coverage estimates discrepancies. This is one promising step forward to help strengthen national administrative data systems.

However, much more work on data verification is still needed, both by GAVI and the global health aid community writ large. While GAVI appears to be moving towards increased use of survey-based estimates for future grants, it remains unclear how countries with historical grants will be transitioned to this new requirement—or whether improved data quality will estimate lower coverage values, which in turn has ramifications for GAVI’s performance payments. Further, it remains to be seen if these additional checks and balances actually lead to improved accuracy and consistency of administrative data, or if they will simply uncover over-reporting. Similar issues were faced by GAVI’s Data Quality Audit, which conducted audits in 2002-3.

While more rigorous verification is important for more effective results-based aid, additional verification alone provides only limited information on how M&E systems should be improved and the underlying drivers of discrepancies, raising broader questions about the need to strengthen statistical systems (check out CGD’s Data for Development Working Group here and Amanda Glassman’s recent blog post here). Of course, GAVI’s guidelines alone won’t be sufficient to address low national statistical capacities, which have affected quality and timely data use and production by other global health agencies as well. The Global Fund, for example, faces poor (if not worse) quality data, representing a major obstacle to improving value for money (see here). There is a mechanism called the Health Systems Funding Platform in which donors can pool funds to pay for things like verification. But without major donor participation in that platform, it remains irrelevant, unfortunately (see here), and verification is still mainly done donor-by-donor. It remains to be seen how far down the road towards improved administrative data GAVI’s valuable but lonely policy will take us.

Victoria Fan is a research fellow and health economist and Kate McQueston is a program coordinator at the Center for Global Development. The authors thank Jenny Ottenhoff for helpful comments. You can follow Victoria Fan at @FanVictoria and Kate McQueston at @kate_mcq on Twitter.