Following the Best Science Does Not Always Make the Best Vaccine Investment

The current vaccine we have against tuberculosis (TB)—the BCG vaccine—whilst valuable, has relatively low efficacy, and as a result TB remains the deadliest infectious disease in the world, killing about 1.2 million people every year. Discovering a better vaccine would be one of the most important medical breakthroughs this century, so funders are investing considerable resources in this area. Their goal is to create a highly effective vaccine that can be used to inoculate the vast majority of people who are at risk of contracting a disease; we term a vaccine that achieves this a success. Intuitively, it might seem likely each funder backing what they regard as the most promising technology would be the best way to do this, but that is not always the case.

Imagine the choice between two candidates that both need funding to start clinical trials in humans. One is a protein subunit with adjuvant (PSA), a well-established vaccine platform that is thought to be one of the most likely to succeed in this space. Indeed five of the 13 vaccines in clinical trials at the time of writing are PSAs, including two that are in the final Phase III trials. Other vaccines with the same target and platform and similar adjuvants have looked very promising in early stages of development. The second candidate has some promising pre-clinical studies in animal models and the lab, but is based on a much less-tested technology that does not have a single candidate in clinical development for TB.

In this hypothetical scenario, the PSA vaccine is much more likely of the two to succeed, so this choice might seem obvious, but it is not. There are five vaccines that use the same vaccine platform as the PSA candidate all further ahead in development. If all five of these vaccines fail, then it increases the probability that the PSA candidate will fail for a similar reason, and if any of them succeed, they will reach people long before the additional PSA candidate is ready. The second candidate looks far less promising on its own, but because it is based on a technology that is fundamentally different to everything else in the portfolio, its chance of succeeding and failing are far less correlated with other vaccines in development. This means it is possible that it increases the probability that at least one vaccine against TB is discovered by more than the additional PSA vaccine does. We define this probability of at least one vaccine succeeding as the probability of portfolio success (PoPS). In practice, success might not be final, as a valuable vaccine that is developed may be later displaced by another with better properties.

This marginal contribution to the portfolio can be calculated by using a tool that simultaneously models all of the candidates in the portfolio. This would include estimates of how likely they were to succeed on their own, and how correlated failures between them would be before assessing how much each candidate increases the overall PoPS. Though previous work has investigated this issue, it has not generated a ready-to-use tool for application to novel vaccine portfolios, which is why researchers at CGD are planning to build one.

The problem with backing the leading technology

The maths behind this is striking. If you have five candidates, each with a 50 percent probability of success (PoS), then when risk is completely uncorrelated, the PoPS is 97 percent. However, if outcomes are heavily correlated, then the PoPS might not be much greater than 50 percent for the portfolio. A second portfolio which has a candidate with a 50 percent PoS and four more have a 20 percent PoS, would have an 80 percent PoPS if the candidates are uncorrelated. One way to model correlated risk is the approach taken in CGD’s 2020 analysis of COVID-19 vaccines. When a candidate succeeded or failed, the model updated the PoS of other candidates using the same platform, with the size of the adjustment determined by the assumed correlation. Put simply, if two vaccines each had a 50 percent PoS and the correlation strength was 20 percent, success by one would increase the relevant PoS for the other to 60 percent, while a failure would reduce it to 40 percent.

In the example of the five vaccines with a PoS of 50 percent compared to an individually weaker uncorrelated portfolio, the first portfolio only needs a correlation of 34.3 percent between candidates to become less likely to succeed overall. Whether the first or the second portfolio of candidates is better depends at least as much on how correlated the risks are across candidates as it does to the fact the candidate’s underlying strength (Figure 1).

Figure 1. How correlated risk changes which portfolio is better, using two illustrative portfolios of vaccines

Image

Instead of looking at the global vaccine pipeline as a whole, funders often rely on tools built for the pharmaceutical industry, where the goal is to invest in a product that will reach the market and maximise the expected return on investment for the company. This differs from the goals of many public and philanthropic funders who want to increase the probability that any product succeeds.

Even when people are working in a not-for-profit environment, scientific and social incentives can push people away from trying to maximise the PoPS. If a funder backs a novel candidate and it fails, they might have difficult questions to answer from their stakeholders about why they funded a candidate that looked less likely to succeed. If they invest in a leading technology and it fails, they have a much stronger defence: they invested in a technology that was widely seen as one of the most promising in the space.

There are numerous areas where investment decisions might have been improved by tools that assess the whole portfolio:

In HIV vaccines, where nine efficacy trials have been run and none produced a licensed vaccine, the candidates shared two weaknesses: (1) they did not elicit antibodies neutralising a broad range of circulating strains, and (2) they did not elicit responses that persisted in an active state rather than as immunological memory. It may have been possible to have made this conclusion without funding all nine of these large trials, which each cost up to $100 million.

In Alzheimer’s treatments, research on amyloid was prioritised over all other research avenues. The same underlying hypotheses failed in numerous clinical trials. Some authors argue this has delayed development in this space by 15 to 30 years. There have been some breakthroughs in this field, but their clinical value is contested and there is a growing consensus that a more diversified approach would probably have brought more success.

In malaria vaccines, a 2018 review advocated for building on existing work by embracing newer technologies and diversifying approaches going forward.

Prior work to build on

This problem has already been recognised. During the COVID-19 pandemic, CEPI explicitly tried to organise its investments as a diversified portfolio to spread the risk across technologies. CGD researchers built a model and published a portfolio analysis in October 2020 that combined expert elicitation with a Monte Carlo simulation to assess how well the COVID-19 portfolio was diversified. Similar work was done by Ahuja et al., and was undertaken privately by governments. However, bespoke models had to be built do to this, and interview questions optimised at the same time. This meant that evidence generation was slower and less robust than it would have been if there were existing tools for assessing the PoPS. Having high-quality assessment tools ready to go would have supported decision-making and investments earlier in the pandemic.

We are going to build a model to assess portfolios of vaccines

Over the next year, we will build and test a deployable framework and model that will use a combination of historical data, information on vaccines in the portfolio, and structured expert interviews that can be used to generate evidence on any portfolio of vaccines. For a portfolio, this approach should be able to estimate:

• The probability that the portfolio will generate at least one effective vaccine, or probability of portfolio success (PoPS)
• How much the portfolio would benefit from greater diversification
• How much each candidate in the portfolio contributes to the PoPS
• What is the most cost-effective way to increase the PoPS

The mathematics involved in this kind of modelling is not very complicated, but getting disease-specific inputs is. The main contribution of this work will be to produce a standardised approach that can be rolled out relatively quickly, for both new and well-understood threats.

After we build this tool, we will then apply this methodology to the portfolio of TB vaccines. This will be done both as a demonstration project to improve the model, as well as something that we hope will improve future investment decisions on TB.

While we are developing this tool, we are not experts in the science of vaccines, immunology, or microbiology. Our goal is to design a methodology that takes information from a given vaccine portfolio, alongside assessments from relevant experts, and so estimate the probabilities of outcomes if those experts’ assessments are broadly correct.

Our aim is to support health research funders to estimate a candidate’s contribution to the global portfolio, as well as the candidate itself. We will look to provide more information for people making investment decisions, so that they can better weigh the value of novelty against stronger underlying science, as well as justify any decision to back an outside candidate.