Studies show that the benefits of COVID-19 vaccines outweigh their risks; preprint claiming to show otherwise is flawed

Key takeaway

Multiple peer-reviewed published studies have shown that the benefits of COVID-19 vaccination outweigh its risks. While some COVID-19 vaccines are associated with a risk of heart inflammation while others are associated with a risk of blood clots, the same risks are much higher following SARS-CoV-2 infection.

Reviewed content

Verdict:

Inaccurate

Claim:

“Covid Vaccines More Likely to Put You in Hospital Than Keep You Out”

Verdict detail

Misrepresents source: The preprint compared the risk of serious adverse events with the risk of COVID-19 hospitalization. However, several of these events, such as rash and diarrhea, don’t typically require hospitalization. Claiming that the preprint showed COVID-19 vaccines are more likely to cause hospitalization than prevent it is inaccurate and misrepresents the analysis.
Cherry-picking: The list of serious adverse events assessed included hyperglycemia (high blood sugar), but excluded hypoglycemia (low blood sugar). It included gastrointestinal hemorrhage, but excluded duodenal ulcer hemorrhage (a subtype of gastrointestinal hemorrhage). No explanation was given for why these exclusions were made.

See our method of evaluation

Full Claim

“Covid Vaccines More Likely to Put You in Hospital Than Keep You Out”

An article published by the website The Daily Sceptic, formerly known as Lockdown Sceptics, claimed that “Covid Vaccines More Likely to Put You in Hospital Than Keep You Out, BMJ Editor’s Analysis of Pfizer and Moderna Trial Data Finds”. The claim is based on a preprint—a study that hasn’t yet been peer-reviewed—co-authored by Peter Doshi, an associate professor at the University of Maryland and an associate editor at the BMJ, a medical journal. [Editor’s note: the preprint has since been published in the peer-reviewed journal Vaccine.]

The preprint went viral, receiving more than 440,000 abstract views and more than 22,000 engagements across various social media platforms, according to social media analytics tool Buzzsumo. The preprint was also shared by influential accounts, lending it credibility among many users. For example, clinical psychologist Jordan Peterson, who currently has more than 2.8 million followers on Twitter, tweeted about the preprint, commenting that “This is not good at all”. Claudio Borghi, a member of the Italian Chamber of Deputies (part of the Italian Parliament), also shared the study on Twitter.

Unsurprisingly, individuals who have published misinformation about vaccines, like tech entrepreneur Steve Kirsch, seized on the preprint to claim that the COVID-19 vaccines are dangerous. However, the study’s methodology is flawed and the study doesn’t provide evidence for the claim that it’s more dangerous to get the COVID-19 vaccine than to remain unvaccinated. In this YouTube video, Susan Oliver, a scientist who specializes in materials for medical uses, explained these methodological flaws in detail.

The primary flaw in the preprint is the use of P-hacking. This occurs when scientists manipulate data, such as by leaving out certain datasets, in order to obtain statistically significant results in their analysis. Statistical significance means that an observed difference between two groups, like a control group and a treatment group, is unlikely to be due to chance and is therefore likely a true effect from the treatment. P-hacking is a significant problem in scientific research, owing in part to scientific journals’ preference for publishing studies that report effects over studies that don’t report any effects. This PLOS One article discussed the problem of P-hacking and its negative effects on scientific research and progress.

In the preprint, the authors explained that they analyzed adverse events of special interest (AESI), using data from the Phase III randomized clinical trials for the Pfizer-BioNTech and Moderna COVID-19 vaccines. The exact data sources are listed in the preprint’s Table 1, on page 14. The AESI analyzed in the preprint were selected from a priority list of adverse events to assess in relation to COVID-19 vaccines. This priority list was defined by the Brighton Collaboration, a non-profit that works to improve vaccine safety, based on three criteria: “known association with immunization or a specific vaccine platform; theoretical association based on animal models; occurrence during wild-type disease as a result of viral replication and/or immunopathogenesis”.

As the preprint stated, unlike the analyses conducted by the U.S. Food and Drug Administration (FDA), the preprint authors counted the total number of adverse events, rather than the total number of individuals affected by adverse events. For example, if a person experienced three AESIs, they would be counted thrice. The FDA analyses would have counted this case as one instance instead. This distinction is important, as we will see later.

The authors compared the risk of AESIs to the risk of COVID-19 hospitalization, and concluded that “The excess risk of serious adverse events of special interest surpassed the risk reduction for COVID-19 hospitalization relative to the placebo group in both Pfizer and Moderna trials”. That is, the risk of serious AESIs was greater than that of COVID-19 hospitalization.

But as Oliver pointed out, the problem with the analysis is that people who were hospitalized for COVID-19 were counted only once, but AESIs were counted on a per-event basis, as we explained above. This would have enlarged the number of AESIs counted, making it more likely for the authors to detect a higher AESI risk as compared to the risk of COVID-19 hospitalization.

Jeffrey Morris, professor of biostatistics at the University of Pennsylvania, also pointed out the same problem in this tweet: “Another glaring methodological flaw is comparing events per 10k (including multiple events per individual) with hospitalizations per 10k (counting each individual only once). Should compare at individual not event level for this”.

There are other ways in which the analysis is biased in favor of the conclusion that the risk of serious adverse events are greater than the risk of serious COVID-19 outcomes. For instance, death is the most serious outcome of COVID-19, but was excluded from consideration in the analysis, which only considered hospitalization. To date, more than six million people have died from COVID-19, according to the Johns Hopkins University Coronavirus Resource Center. And among these deaths, over a million were in the U.S. alone. There is also long COVID, which exerts negative health effects on individuals even months after the initial infection, that also wasn’t considered.

The comparison of the included AESIs with COVID-19 hospitalization also raises some questions. For instance, conditions such as rash, diarrhea, and arthritis are by definition AESIs, but aren’t typically severe enough to require hospitalization. Their inclusion in the comparison with COVID-19 hospitalization is therefore one of “apples and oranges”, as Oliver noted.

There were also inconsistencies with the kinds of AESIs the authors chose to include and exclude. For example, the authors included diarrhea, but excluded vomiting; they included hyperglycemia (high blood sugar), but excluded hypoglycemia (low blood sugar); they included gastrointestinal hemorrhage, but excluded duodenal ulcer hemorrhage (the duodenum is part of the large intestine). The authors didn’t explain these inclusion and exclusion decisions.

For these reasons, the preprint’s findings aren’t reliable and the conclusion that the risk of hospitalization from vaccine-related adverse events is greater than that of COVID-19 hospitalization isn’t supported by the analysis that the authors performed. The claim that the preprint showed the COVID-19 vaccines are more likely to cause hospitalization than prevent it is inaccurate and misrepresents the preprint’s analysis.

Following the publication of our review, the preprint was later published in the peer-reviewed journal Vaccine. After the study was published, the authors contacted Health Feedback by email, criticizing the findings in this review pertaining to the preprint, stating that “the allegations against our analysis are serious, but they are demonstrably false”. You can read the full response to our review by Fraiman et al. here. The response didn’t contain evidence demonstrating that our reporting of the flaws in the preprint—and their repercussions—was inaccurate.

We already have plenty of data showing that the benefits of COVID-19 vaccination outweigh its risks. For example, some COVID-19 vaccines are indeed associated with a risk of heart inflammation. However, a study by researchers at the U.S. Centers for Disease Control and Prevention, which included patient data from 40 health centers in the U.S., found that SARS-CoV-2 infection is more likely to lead to heart inflammation than COVID-19 vaccination^[1]. Specifically, teenage boys (12 to 17 years old) were two to seven times more likely to develop heart inflammation after infection than after vaccination. The risk was seven to eight times higher in young men aged 18 to 29 years old.

There are also concerns about the risk of blood clots associated with certain COVID-19 vaccines. But a study led by researchers at Oxford University, published in the BMJ, found that the risk of blood clots after SARS-CoV-2 infection was higher compared to the same risk following COVID-19 vaccination^[2].

A study published in The Lancet Infectious Diseases in June 2022 used mathematical modeling to estimate the number of deaths averted by COVID-19 vaccination in the first year after the first COVID-19 vaccine was administered post-trial (between 8 December 2020 and 8 December 2021)^[3]. The researchers estimated that more than 14 million deaths were averted through COVID-19 vaccination, a finding that further emphasizes the benefits of the COVID-19 vaccines.

READ MORE

Science-Based Medicine published two articles on the analysis by Fraiman et al. which explain the study’s limitations and why it’s a flawed basis for the claim that the risks of COVID-19 vaccines outweigh their benefits.

UPDATE (21 October 2022):

This review was updated to report that the preprint has since been published in the peer-reviewed journal Vaccine. We also included a link to the full response to our review by Fraiman et al.

REFERENCES

• 1 – Block et al. (2022) Cardiac Complications After SARS-CoV-2 Infection and mRNA COVID-19 Vaccination — PCORnet, United States, January 2021–January 2022. Mortality and Morbidity Weekly Report.
• 2 – Hippisley-Cox et al. (2021) Risk of thrombocytopenia and thromboembolism after covid-19 vaccination and SARS-CoV-2 positive testing: self-controlled case series study. BMJ.
• 3 – Watson et al. (2022) Global impact of the first year of COVID-19 vaccination: a mathematical modelling study. The Lancet Infectious Diseases.