Published last week in The Lancet, a new study by the Institute for Health Metrics and Evaluation (IHME) finds that there were 1.2 million deaths from malaria in 2010, not 655 thousand as estimated by the WHO. Following its release, headlines began splashing uncritically: “Malaria kills twice as many as previously thought, study finds” (The Guardian) “Malaria deaths hugely underestimated” (BBC). Which set of estimates is correct? Or at least which is less biased? Given the 1.2 billion dollars by donors to malaria in 2010, is it unreasonable to demand to know with more certainty, how many people are dying from malaria?
In an effort to gain clarity on these questions, we’ve ventured into the sausage-making of both malaria estimates (see below, if you dare venture into the methods of the sausage machine). In essence, both the IHME and WHO start with ‘raw ingredients’ (some data), then apply different ‘cooking’ techniques (some methods). From both of their ‘recipes’, it is not obvious to us that IHME’s estimates are necessarily less biased (or more correct) than WHO’s. Both IHME and WHO are doing the best they can, given shockingly unavailable and bad data. In Africa where 90% of malaria deaths occur, almost all mortality data come from verbal autopsy (VA) studies, where family are asked to report symptoms and diagnose the cause of a person’s death well after the death occurred. Malaria mortality estimates are taken from sub-national samples of this VA data and then extrapolated to the broader population. For example, one study in a Nigerian village with 227 malaria deaths was used as the main basis to extrapolate all malaria deaths in all of Nigeria.
One distinguishing feature of the IHME study is that it nobly tries to correct these VA studies, but even the validity of these corrections for estimates of Africa is questionable (see below). The other main factor contributing to the differences between IHME and WHO estimates is the extent to which the burden of malaria falls on children under age 5 compared to those older (see below). These two differences probably explain much of the discrepancy between IHME and WHO in counting malaria deaths in Africa. Beyond that, both IHME and WHO take fairly similar methods in Africa to extrapolating and modeling bad and non-existent data (although we’re not too certain since it’s a bit harder to find out what WHO did).
It would help if both IHME and WHO (WHO probably more so) were more transparent and made their data and methods available for interested parties to replicate their results. In any case, if we want to say anything reasonably straightforward without having to make sausages every year from VA data, better quality vital registration systems and data are needed. Data-production and sausage-making enterprises, fancier or simpler, are ultimately temporary coping mechanisms until countries themselves have better information and can correct it on their own. The long-term goal should be to not rely on people sitting in Seattle and Geneva for the ‘correct’ numbers. To realize this goal, we will need to value this investment of knowledge and country capacity as essential to public health. But until then, will donors continue to spend million or billions of dollars on malaria without really knowing how many people malaria kills each year?
The Method of the Sausage Machine: How to Count Malaria Deaths IHME’s sausage recipe: what did IHME do?
IHME researchers compiled vital registration (VR) and verbal autopsy (VA) studies on malaria deaths of all ages globally. In Africa alone, the researchers found 106 VR and VA studies since 1980, which were almost entirely subnational VA studies. As an example of such ‘subnational’ studies, look at Nigeria (which according to IHME had 380,632 malaria deaths and the highest in Africa). For all of Nigeria the IHME paper used 1 subnational VA study in 1 Nigerian village which had 227 malaria/fever deaths among infants in 1995 (see Appendix, p. 100).
Using this VR/VA data as the main ‘ingredient’, IHME processed the data in two main steps. First, they attempted to correct the VA estimates given that malaria was likely misclassified. Why correct for misclassification? Malaria is notoriously known for being incorrectly diagnosed: a fever may be incorrectly classified as malaria when it is simply a fever, for example. Here is where the sausage-making starts. In the VA studies, IHME used a “validation” study from a “multi-site” study – and by multi-site they mean 6 sites in Mexico, Tanzania, India, and the Philippines which had a total of only 217 malaria deaths – to correct their VR/VA database to correct the proportion of malaria deaths. From this exercise, they shift estimates up by 21%. At this point you may be asking, “So, for all of Africa which accounted 90% of all malaria deaths – 1.1 million (IHME) or 596 thousand (WHO), they used 1 validation study in Tanzania (and perhaps the other regions) to correct the array of VA/VR data points?” It is tough to imagine that these sites can be considered representative of all Africa.
In the second main ‘processing’ step IHME ran statistical models to determine associations on malaria death rate and cause of death fraction and an array of predictor variables including malaria transmission rates, rainfall, bed net coverage, etc. Running literally hundreds of these models, they test the “predictive validity” in order to choose the best-fitting model and extrapolated in time and within region where they did not have data.
WHO’s sausage recipe: what did WHO do?
The WHO relies on two main sources of data. The first set are administrative data on the number of cases multiplied by a case fatality rate, estimated from previous studies, for each country and voila, the numbers of malaria deaths (p. 73 in WMR 2011); WHO used this data for low-transmission countries in Africa plus countries outside of Africa.
The second set of data involved compiling vital registration (VR), verbal autopsy (VA), and other mortality studies to estimate the percentage of deaths under age 5 attributable to malaria; WHO used this data for the remaining countries in Africa plus Somalia and Sudan. Sound familiar? Like IHME, WHO had no other choice but to mainly rely on VA data because of the lack of VR data in Africa. In addition, it’s not clear whether, like IHME, WHO also adjusted for VA misclassification (we suspect not). Next using the VR/VA data, WHO processed them by using statistical models to find associations between the fraction of child deaths from malaria and various country predictors (not clear what variables they use). As for those aged 5 and older, it seems that WHO deduced mortality rates using previous studies on age-specific malaria mortality rates and the intensity of malaria transmission (p. 140 in WMR 2008).
How do the results between WHO and IHME differ?
We compared the numbers by IHME and WHO by WHO Region (see Table 1 below). Both IHME and WHO agree that Africa accounts for 90% of all malaria deaths. There is agreement between IHME and WHO in the numbers for the Americas, Western Pacific, and Europe (regions which have better vital registration data). However, the numbers differ greatly for Africa, Eastern Mediterranean, and South-East Asia regions. This difference in South-East Asia is likely explained by WHO non-use of the latest VA estimates for the South-East Asia region (and similarly for the Eastern Mediterranean). But why do estimates in Africa vary so much if both IHME and WHO compiled and processed VA/VR data?
Source: Compiled by authors from IHME and WHO
One distinguishing feature is that IHME seemed to correct for misclassification. The other major question is how much of the malaria burden falls on children under 5 compared to people aged 5 and older. Will the IHME study overturn years of clinical observation that, by far, most of malaria’s victims are children? IHME says that in 2010 children under 5 accounted for 58% of the world’s malaria deaths (our calculations from IHME data), whereas WHO says it is 86% (p. 74). Yet when we looked by region at the percentage of malaria deaths aged 5 and older (see Table 2), we found that indeed both IHME and WHO agree that age distribution of malaria deaths differ by region and that at least in the Americas, South-East Asia, and Western Pacific, most malaria deaths occur in people older than age 5.
Source: Compiled by authors from IHME and WHO
Where IHME and WHO differ the most, in both the numbers and the age profile of death, is for Africa. How valid is either group’s estimates in Africa? It would depend on how much you believe each stage of the sausage-making process:
- Are WHO’s VA/VR/mortality study for child deaths estimates combined with review of other mortality studies for adult deaths sufficient compared to IHME’s VA/VR studies of deaths for all ages? (And how exactly did WHO calculate its adult deaths? Why is it hard to figure out what WHO did?)
- How valid is IHME’s “multi-site” validation study (with its 214 malaria deaths in 4 countries, of which only 1 is in Africa) to “correct” the VA data? Did WHO correct for misclassification?
- How sensitive are results to the choice of predictors in the models by both WHO and IHME?
- More importantly, how much does each stage of the sausage-making process contribute to the estimates?
Disclaimer
CGD blog posts reflect the views of the authors, drawing on prior research and experience in their areas of expertise. CGD is a nonpartisan, independent organization and does not take institutional positions.
