Claims that a “Johns Hopkins study” showed lockdowns are ineffective at reducing COVID-19 mortality are based on a working paper with questionable methods

In early February 2022, multiple social media posts and articles published by various outlets, including Fox News (see here and here) and the Daily Mail, widely touted a “Johns Hopkins study” purportedly showing that lockdowns only reduced COVID-19 mortality by 0.2% and that lockdowns should be rejected as a pandemic measure. These articles and posts went highly viral on social media; the Fox News and Daily Mail articles received a combined engagement exceeding 130,000. Political commentator Tomi Lahren also posted a video citing the study, in which she called lockdowns “useless”, drawing more than 160,000 views on Facebook.

The alleged study refers to a working paper posted by three economists, Jonas Herby, Lars Jonung, and Steve Hanke, titled “A Literature Review and Meta-Analysis of the Effects of Lockdowns on COVID-19 Mortality”. Herby is a special advisor at Center for Political Studies in Copenhagen, Denmark, Jonung is professor emeritus in economics at Lund University, and Hanke is a professor of applied economics at the Johns Hopkins Institute for Applied Economics, Global Health, and the Study of Business Enterprise.

As the title of the paper explained, the authors undertook a literature review and meta-analysis. In simple terms, a literature review surveys the existing scientific work on a subject. It can be used to provide an overview of developments in a field and/or offer new insights based on previous studies, among other things. A meta-analysis combines data from multiple studies to answer a specific question, for example, whether sunscreen reduces the risk of skin cancer. One of the key advantages of a meta-analysis is that it provides greater statistical power, as it involves many more subjects in the analysis than a single study could provide.

However, several reports about the paper described it in inaccurate and misleading terms. More importantly, public health experts have drawn attention to issues in the authors’ methodology that call the validity of the findings into question, but aren’t acknowledged in certain reports.

The study is a working paper that isn’t peer-reviewed; authors of the paper are economists with no expertise in epidemiology

The paper was described by several outlets as a “Johns Hopkins study”. This description is problematic for several reasons. One reason is that only one of the three authors, Steve Hanke, is affiliated with Johns Hopkins University. The second is that the study wasn’t endorsed by the university. Finally, describing the paper as a “study” could mislead readers into believing that the paper is a peer-reviewed published study, which it is not. Even the authors themselves described it as a working paper, meaning that it is still a work in progress and hasn’t been peer-reviewed by other scientists.

Nevertheless, the label “Johns Hopkins study” stuck, likely because it burnished the paper’s credibility in the eyes of the public.

Secondly, many articles and social media posts promoting the study failed to inform readers that the authors are economists and lack expertise in epidemiology and infectious diseases, which is pertinent to this subject, given that lockdowns were used to combat the spread of an infectious disease.

The paper contained an unusual definition of a lockdown and methodological issues

Experts told the Science Media Center in the U.K. that the paper contained important methodological flaws.

One issue with the paper is that the meta-analysis it undertook “systematically excluded from consideration any study based on the science of disease transmission”, said Seth Flaxman, an associate professor at Oxford University who specializes in statistics.

“[T]he only studies looked at in the analysis are studies using the methods of economics. These do not include key facts about disease transmission such as: later lockdowns are less effective than earlier lockdowns, because many people are already infected; lockdowns do not immediately save lives, because there’s a lag from infection to death, so to see the effect of lockdowns on Covid deaths we need to wait about two or three weeks,” Flaxman explained.

Another point of contention was the paper’s definition of a lockdown. A lockdown is typically defined as a measure that requires people to stay at home and avoid activity outside the home involving public contact. However, the authors defined a lockdown as “the imposition of at least one compulsory, non-pharmaceutical intervention (NPI)”. Non-pharmaceutical interventions are measures apart from taking medicine and vaccination. This means that simply making isolation for infected people mandatory or imposing a mask mandate would count as a lockdown.

Neil Ferguson, an epidemiologist and professor at Imperial College London, pointed out that “by that definition, the UK has been in permanent lockdown since 16th of March 2021, and remains in lockdown – given it [remains] compulsory for people with diagnosed COVID-19 to self-isolate for at least 5 days.”

Samir Bhatt, a professor of statistics and public health at Imperial College London, likewise expressed similar concerns about the definition used in the paper, saying, “The most inconsistent aspect is the reinterpreting of what a lockdown is […] For a meta-analysis using a definition that is at odds with the dictionary definition (a state of isolation or restricted access instituted as a security measure) is strange.”

That being said, when it came to their analysis, the authors applied the usual definition of a lockdown, as Bhatt pointed out: “The authors then further confuse matters when in Table 7 they revert to the more common definition of lockdown.”

Epidemiologist Gideon Meyerowitz-Katz, who discussed the paper in a Twitter thread, also noted the confusion in the paper, saying, “The authors of this review define a ‘lockdown’ as LITERALLY ANY INTERVENTION”. But he also observed “this is certainly not how the analysis is actually done. In most cases, it seems the authors have taken estimates for stay-at-home orders as their practical definition of ‘lockdown’”.

But if we were to go with the authors’ initial definition of a lockdown, which is the application of any NPI, then “the reality is that virtually all research shows a (short-term) mortality benefit from at least some restrictions,” concluded Meyerowitz-Katz.

In fact, the working paper itself demonstrated this in their review:

26/n You can even see this IN THE REVIEW. The authors found benefits for compulsory facemasks, business closures, border closures/quarantine, and school closures. Some of the benefits were very large! pic.twitter.com/ZPzWnHz6Oo

— Health Nerd (@GidMK) February 4, 2022

In addition, Meyerowitz-Katz noted the paper’s anomalous results which were at odds with the studies that made up their meta-analysis. For example, one study^[1] by Chisadza et al. included in the meta-analysis, which also contributed the greatest weight in the analysis, reported that lockdowns were beneficial with mitigating COVID-19-related deaths:

“This study investigates the effectiveness of current government policy responses to COVID-19-related deaths. We find that the overall government response index has a non-linear association with the number of deaths—driven by the containment and health interventions—for the aggregated sample of countries. The number of deaths increases with partially relaxed lockdown restrictions, but decreases with severe restrictions.”

Yet the working paper arrived at the opposite conclusion:

13/n But it gets even weirder. If you look at the model, almost the entire weighting is based on this paper, Chisadza et al

But Chisadza et al found a BENEFIT for lockdowns pic.twitter.com/yS9CIedzFX

— Health Nerd (@GidMK) February 4, 2022

In fact, an author of the study by Chisadza et al. told science journalist Tom Whipple that the working paper’s authors had simply set out to prove a predetermined conclusion:

I spoke to the author of the paper on whose research this entire meta-analysis was based, but who reached a diff conclusion.

She said: "They already had their hypothesis. They think that lockdown had no effect on mortality, and that’s what they set out to show in their paper." https://t.co/oLc9HncolB

— Tom Whipple (@whippletom) February 3, 2022

The same issue cropped up with another study^[2], which found “strong evidence consistent with the idea that employee mask policies, mask mandates for the general population, restaurant and bar closures, gym closures, and high-risk business closures reduce future fatality growth”. But the study was somehow “included in this review as estimating a 13.1% INCREASE in fatalities”:

17/n Another included paper found that significant restrictions were effective, but is included in this review as estimating a 13.1% INCREASE in fatalities. The maths used to derive this is pretty opaque pic.twitter.com/WolYRPzi88

— Health Nerd (@GidMK) February 4, 2022

Finally, results from published studies contradict the findings of the paper, finding that lockdowns were beneficial in reducing the spread of COVID-19^[3-7] and saving lives^[2,8]. For example, Lurie et al. reported that “Among states without stay-at-home orders, the median increase in [epidemic] doubling time was 60% (95% CI, 9.2–223.3), compared with 269% (95% CI, 277.0–394.0) for states with stay-at-home orders”, suggesting that “stay-at-home orders combined with varied levels of implementing practices of testing, tracing, and isolation … as well as travel restrictions, likely played a key role in significantly reducing the epidemic growth rate”^[3]. Sethman et al. estimated that across 11 countries in Europe, lockdowns helped prevent more than three million deaths^[8].

If we consider the nature of infectious diseases, it is only to be expected that fewer cases and deaths follow a lockdown. Given that close contact with other people facilitates virus transmission, measures that limit such contact, like a stay-at-home order, would inevitably reduce infection and in turn deaths. An assertion to the contrary would require strong evidence, which the working paper doesn’t provide.

At least two of the three authors are known to have publicly expressed opposition to or skepticism of lockdown measures, suggesting that they already held certain preconceived notions about lockdowns and that their working paper was influenced by their preexisting beliefs. Readers should be made aware of this initial bias. Jonung co-authored an article titled “A Message from Sweden: Mandatory Lockdowns Do more Harm than Good” in Svenska Dagbladet, a Swedish newspaper. The first author Hanke, in particular, equated lockdown measures to fascism and falsely claimed that Vietnam didn’t report COVID-19 data (see screenshots below).

Screenshots of tweets by Hanke in June 2020 (left) and December 2021 (right) (archives: here and here).

Following his tweet about Vietnam, 285 “public health researchers and professionals and concerned citizens” penned an open letter to Johns Hopkins University demanding an apology from Hanke; Hanke later deleted the tweet.

In summary, there are several issues with the working paper that call into question the validity of its findings. Firstly, the paper’s definition of a lockdown (imposition of any NPI) differs from the dictionary definition of a lockdown. This distinction made in the working paper is lost in many articles and social media posts. Secondly, it’s unclear how the paper arrives at some of the figures it reported. As explained above, there was more than one instance where the paper provided an interpretation of a study included in their analysis that was at odds with what the study actually reported. However, certain outlets that reported on the working paper, such as Breitbart, the New York Post, and The Blaze, didn’t provide readers with context about the paper’s limitations and the issues raised about its methods, potentially leaving readers with a false impression about the reliability of the paper’s conclusions.

Overall, the claim that lockdowns are “useless” or ineffective at reducing COVID-19 mortality using this working paper as scientific justification is, in reality, unfounded, and contradicts the conclusions from published studies.

UPDATE (14 February 2022):

We added context regarding two of the working paper authors’ views on lockdown measures.

REFERENCES

• 1 – Chisadza et al. (2021) Government Effectiveness and the COVID-19 Pandemic. Sustainability.
• 2 – Spiegel and Tookes (2021) Business Restrictions and COVID-19 Fatalities. The Review of Financial Studies.
• 3 – Lurie et al. (2020) Coronavirus Disease 2019 Epidemic Doubling Time in the United States Before and During Stay-at-Home Restrictions. Journal of Infectious Diseases.
• 4 – Hsiang et al. (2020) The effect of large-scale anti-contagion policies on the COVID-19 pandemic. Nature.
• 5 – Brauner et al. (2020) Inferring the effectiveness of government interventions against COVID-19. Science.
• 6 – Chernozhukov et al. (2020) Causal impact of masks, policies, behavior on early covid-19 pandemic in the U.S. Journal of Econometrics.
• 7 – Salje et al. (2020) Estimating the burden of SARS-CoV-2 in France. Science.
• 8 – Flaxman et al. (2020) Estimating the effects of non-pharmaceutical interventions on COVID-19 in Europe. Nature.