This is a joint post with Victoria Fan.

The New England Journal of Medicine recently published the results of “the Oregon experiment” based on the 2008 US Medicaid program expansion in Oregon. The study is one of very few randomized control trials on publicly-subsidized health insurance that exists to guide health policy, and found what some commentators considered a disappointing result: while health care utilization increased and households were protected from financial hardship, expanding Medicaid coverage had “no significant impact on measured physical health outcomes over a 2-year period.”

Should we be surprised? To date, there are few countries that have demonstrated a causal link between health insurance/coverage and physical health status using a randomized trial, and many of the studies that find a linkage used observational, or non-experimental, methods (see here and here for comprehensive reviews of the literature). For example, a 2012 observational study found that the Thai health insurance program reduced infant mortality by 6.5 per 1000 live births.

But in many studies there are no detected associations between expanded health insurance or risk pooling with population health outcomes. For example, an update on an experiment in Mexico on the Seguro Popular public insurance program finds out-of-pocket spending dramatically reduced, but impact on health status remains elusive.

For studies with null results, the absence of evidence does not mean an evidence of absence. There are many reasons why detecting a causal effect between insurance and health outcomes is complex and challenging. Methodologically, detecting changes in health status in a short time period provided from an academic study is very difficult. Mortality is a rare event, requiring large sample sizes to detect change (which is probably why the Oregon experiment did not measure it), while morbidity is still relatively infrequent (which the study did measure) – but also not so easy to change even with health insurance (just think about the last time you tried to lose weight, stop smoking, or reduce your blood pressure!)

In addition to methodological difficulties, the mechanism between health insurance and health status operates through increasing the utilization of health care, holding individual, household, and social and community factors constant. If insurance does indeed increase utilization, but does not have an impact on health outcomes amenable to efficacious clinical interventions, then there are many other factors that could be at play: poor clinical quality on the provider side; poor adherence or compliance of medicines by patients of treatment; or other unobserved behavioral variables on both sides.

Moreover, while this may seem obvious, changes in utilization and associated health outcomes need to be directly related to the benefits offered under an insurance scheme. In other words, changes in physical health may have more to do with the scope and content of benefits plans, the structure of and incentives associated with provider payment and quality oversight mechanisms, and the amounts of premiums and co-pays – and much less to do with the mere accessibility and availability of services through insurance. Hence it holds that insurance may be a necessary but not a sufficient condition for improving health outcomes.

Indeed, these various possibilities are mentioned by the Oregon experiment authors. Compliance and treatment-seeking behaviors were likely critical; for example, the study found that there was no effect on the use of hypertension medications and hence, not surprisingly, no statistically significant change in blood pressure in the experimental group.

Physical health measures aside, one particularly encouraging finding in the Oregon experiment is that within just two short years there was a detected impact on mental health and several measures of self-reported health. While it is easy to disregard self-reported health measures, there are many studies out there that show a strong correlation between self-reported health and long-run mortality (see here, here, and here for example) – and these studies are the reason why such mental health measures are often used when it is impossible to detect mortality changes.

So what are the lessons for global health? In the drive to universal health coverage, as championed by the UN and others, we should remember the evidence base is still developing, and that key interventions delivered with quality as well as people’s own behavior and incentives are important factors that will drive health status improvements in the short-term. Countries should not shy away from embarking on robust experimental and quasi-experimental studies (e.g. through a lottery) as they expand pooled funding for health including through insurance – and in doing so they can learn a great deal to improve their health programs.