Covid-19 vaccine efficacy results are not enough

The recent results of Covid-19 vaccine clinical trials from teams at the University of Oxford and AstraZeneca, Moderna, and Pfizer and BioNTech have been one of the most exciting and promising developments in the pandemic to date.

All three vaccine candidates appear to be highly effective against Covid-19, the research teams say. Pfizer and BioNTech say their vaccine had an efficacy of 95 percent against Covid-19, meaning 95 percent of people who received the vaccine were protected against the disease. Moderna reported that its product was 94.5 percent effective. Oxford and AstraZeneca reported that theirs was 62 percent effective in one dosing regimen and 90 percent effective in another, averaging to 70 percent.

Determining efficacy is a crucial step, and these results are far better than many scientists expected. They’re also based on what may seem like a small number of cases, ranging from 131 detected Covid-19 cases to 170, out of a pool of tens of thousands of participants in the trials. But researchers say these numbers are enough to establish how well a vaccine prevents the disease.

Then why haven’t regulatory authorities like the Food and Drug Administration given the green light to a Covid-19 vaccine?

Clinical trials are designed to answer multiple questions, and efficacy is only one of them. Vaccines also have to meet a high bar for safety, much higher than conventional drugs. Since vaccines are given to millions of people — most of whom are healthy and some who have preexisiting conditions — complications have to be rare events.

Oxford and AstraZeneca, Moderna, and Pfizer and BioNTech reported minimal side effects for their vaccines, which is encouraging. Pfizer and BioNTech have already applied for emergency approval for their vaccine, which would likely make it available to people in high-risk positions like health workers at the outset. Full approval, however, would require the companies to produce a lot more safety data than what they summarized in their press releases. At that point, average, generally healthy people can start getting injections.

Health officials will also need more information about how the vaccines performed in different age brackets and ethnic groups, and among people with preexisting conditions, before they recommend how these vaccines should be administered, though experimenters have noted that they went to great lengths to recruit a diverse pool of volunteers.

Overall, their results are only preliminary findings, and more validation is needed to answer critical questions before the billions of people who remain vulnerable to Covid-19 can gain protection from a vaccine. It’s worth unpacking, then, how these clinical trials reached their preliminary conclusions, and why they have to continue despite these early promising results.

How phase 3 clinical trials showed that these Covid-19 vaccines are effective

Vaccines, like many other drugs, are tested in stages to gauge how well they work and how safe they are. In phase 1 and phase 2 clinical trials, vaccines are tested in hundreds and up to several thousand healthy volunteers who are monitored to figure out what doses are effective, to see if any problems arise, and to see if their immune systems start to generate a response, an early sign that a vaccine may offer protection.

But to truly know if a vaccine works, it has to be tested against the actual virus, in the real world, in a broad swath of the population, including people with some preexisiting health issues. That’s what happens in phase 3, the largest, most complicated, and often slowest phase of clinical trials prior to approval. There are currently 12 Covid-19 vaccines in phase 3 clinical trials.

In phase 3, the vaccines are tested in tens of thousands of people. Researchers divide the pool of participants into a group that receives a vaccine and a group that receives a placebo or a comparison vaccine. Simply enrolling enough volunteers can take months.

Moderna enrolled more than 30,000 participants in its phase 3 trial. Pfizer and BioNTech recruited more than 43,000 people for their trial. Oxford and AstraZeneca’s recent results came from a pool of more than 11,000 volunteers. After giving everyone the requisite doses of the vaccine or a placebo/comparison (all three are two-dose regimens, with doses spaced weeks apart), the companies then wait to see how many people end up getting sick with Covid-19 as they go about their daily lives. These cases, also called “events,” have to be confirmed with laboratory testing.

So how many Covid-19 events are enough to draw conclusions about the efficacy of a vaccine? Fewer than one might guess. In some cases, just dozens.

What companies — and regulators — are looking for is fewer cases of confirmed Covid-19 in participants who received the vaccine than in those who just got the placebo. If there is no difference in the split of infections between the placebo and vaccine treatment groups, the trial could potentially end early and be declared futile. If there is a moderate difference, the trial may continue further. And if there is a big difference, it could serve as the basis for emergency approval.

Moderna says its trial efficacy endpoint is 151 confirmed Covid-19 cases. Pfizer and BioNTech set an endpoint of 164 cases. There are also interim checkpoints where the vaccine companies can meet with independent trial monitors known as the Data Safety Monitoring Board (DSMB) to gauge progress. Since these trials are double-blind, where neither the experimenters nor the participants know who is slated to receive a vaccine or placebo, the DSMB regulates when experimenters can peek behind the curtain.

Companies typically tell the DSMB how many cases they are looking for at the start of the trial and what checkpoints they will use to assess progress, a safeguard to prevent experimenters from moving the goalposts.

Oxford and AstraZeneca reported their efficacy results based on 131 cases. Moderna reported its efficacy based on an interim number of 95 cases, and Pfizer and BioNTech reached an even higher benchmark with 170 cases.

Moderna’s interim analysis found that of their 95 events, 90 of them were in the placebo group and five were in the treatment group. Of Pfizer and BioNTech’s 170 Covid-19 cases, 162 were in their placebo group and eight in the group that received their vaccine. Oxford and AstraZeneca did not report the exact split between the group that received their vaccine and the comparison group.

There are some important nuances to the recent vaccine efficacy announcements. Beyond the fact that the results were announced in press releases rather than peer-reviewed papers (although companies say they will publish in scientific journals after the trials are complete), the reported efficacies here are mainly against disease — i.e., people getting sick — and not infection, i.e., people carrying the virus. One of the frustrating things about SARS-CoV-2, the virus that causes Covid-19, is that it can spread between people while causing few or no symptoms. Though the Oxford-AstraZeneca trial did regularly screen volunteers for infection, how these vaccines perform at preventing infection rather than just disease remains unclear at the moment.

Holly Janes, a professor of biostatistics at the Fred Hutchinson Cancer Research Center, explained that, statistically, these findings do show that these vaccines are highly effective in preventing Covid-19 disease, even though they are a tiny fraction of the overall pool of volunteers.

“If we had a trial that enrolled 100,000 people, that accrued 164 events versus a trial that has 2,000 people and accrued 164 events, the amount of information we would have would be the same in terms of efficacy,” she said.

While Covid-19 is running rampant around the world, it’s still only infecting a small fraction of the population at a given moment. So a confirmed case of Covid-19 disease is a rare enough event that less than a couple hundred cases are enough to make a statistical conclusion.

And right now, phase 3 trials in the United States are accruing cases at a rapid clip, a grim consequence of the uncontrolled spread of Covid-19 in the country.

What we still need to find out from Covid-19 vaccine trials

So if several dozen cases are enough to gauge efficacy, why did vaccine developers have to run such a big clinical trial?

One reason is practical. Having more volunteers can speed up the rate at which they accumulate disease events in the trial pool.

“Unlike human challenge trials, where people are intentionally infected, in standard efficacy trials participants are exposed in their communities, in the same way that people not enrolled in a trial would be,” said Natalie Dean, a professor of biostatistics at the University of Florida, in an email. “So we need large numbers of participants to observe relatively few events, and this is exactly why trials number in the thousands.”

The other big reason is that efficacy is not the only parameter in a phase 3 clinical trial. Safety is a huge consideration, and experimenters have to look for any potential complications and mitigate them. “Those [rare events] would only be captured in a very large trial, and you wouldn’t capture those in a very small study,” Janes said.

For example, Guillain-Barré syndrome is an uncommon autoimmune disorder that is associated with the influenza vaccine, but ever since researchers discovered the link, they worked to reduce its frequency from 1 in 100,000 vaccinations to roughly 1 in 1 million. That means the rate of this complication from getting the vaccine is lower than the likelihood of getting Guillain-Barré syndrome from an influenza infection itself, making the vaccine safer than getting the illness based on that outcome alone.

Pfizer and BioNTech, Moderna, and Oxford and AstraZeneca have all reported that their vaccines have so far reported no serious safety concerns from their respective DSMBs. The Oxford-AstraZeneca trial was paused twice to investigate neurological complications among two volunteers, but the trials resumed once researchers reported they found no link between the issues and vaccines.

To gain emergency approval in the US, the companies need two months of monitoring their volunteers to have enough safety data before applying for an Emergency Use Authorization. The Pfizer-BioNTech team reported that they have met this benchmark and filed for an EUA last week. The other vaccine developers expect to file for emergency approvals within a few weeks.

But experimenters need to collect long-term safety data too. Pfizer-BioNTech and Moderna have committed to watching their pool for at least two years, while Oxford and AstraZeneca committed to at least one year, actively checking volunteers for any potential health concerns that might emerge later. Companies will also have to keep tabs on the broader population that receives their respective vaccines after they’re licensed.

These trials also need to reveal more information about how well these vaccines protect against both very mild Covid-19 cases and very severe cases. Pfizer and BioNTech reported nine severe Covid-19 cases in their placebo group and one in their vaccine group, while Moderna reported that all 11 severe Covid-19 cases were in their placebo group. Oxford and AstraZeneca also reported no severe illness in the group that received the vaccine. While this hints that these vaccines make a difference against severe disease, the data is not as robust as it is for Covid-19 cases in general.

“We can more reliably estimate vaccine efficacy against disease than we can estimate efficacy against severe disease, because, again, it all comes down to how many events of each type that have occurred,” Janes said.

On the other hand, it is also possible that people who received the vaccine may have been infected with the virus but had only mild or no symptoms, so they might not have been captured in the number of confirmed Covid-19 cases so far. Oxford-AstraZeneca conducted weekly swab tests of volunteers as part of its trial, which may have detected mild infections. But the Moderna trial and the Pfizer-BioNTech trial only reported people who experienced symptoms and were later confirmed to be infected.

“There are likely many more infections than there are disease events, and even more than that, there are many more exposures,” Janes said. “We will ultimately capture the infections that accrue in the trial, but it will be a while longer before we know anything about those.”

That could have big implications for public health and ending the pandemic as we know it. If people can get a vaccine that is quite effective in preventing Covid-19 symptoms but still carry the virus, they could still potentially spread SARS-CoV-2. This is an important issue because there will only be very limited doses of the vaccine available once it’s cleared for emergency use — and it will likely be many months after that before a sizable amount of the US population will be able to be vaccinated. That means people who have received the injections might need to maintain precautions like wearing masks and social distancing to prevent further spread of the virus before the vast majority of people have been vaccinated.

Outside of the vaccine clinical trials for Oxford-AstraZeneca, Moderna, and Pfizer-BioNTech, health officials will also be paying attention to the outcomes of the other vaccines in development. Johnson & Johnson, for instance, is developing a Covid-19 vaccine that requires just a single dose. That could solve the administrative challenges of deploying a two-dose vaccine. Pfizer and BioNTech’s vaccine also has some of the most stringent cold storage requirements of any Covid-19 vaccine, adding to the challenge of deploying it to areas without ultra-cold freezers.

The vaccination strategy for Covid-19 will have to balance all these variables, which are also changing rapidly as clinical trials progress and more information is revealed. With a variety of Covid-19 vaccines likely to enter the market, each with its own upsides and drawbacks, the next major task is figuring out who gets what and when. That’s why even with promising early efficacy results, vaccine developers must continue to forge ahead with their clinical trials so that we have robust data to answer those questions.