Ivermectin study in the city of Itajaí contains several methodological weaknesses, resulting in questionable conclusions

The drug ivermectin has become a breeding ground for conflict during the COVID-19 pandemic. Ivermectin is a derivative of avermectin, a naturally-occurring compound discovered by William Campbell and Satoshi Omura, which won them the Nobel Prize for Physiology or Medicine in 2015. Used to treat parasitic infections in humans and animals, an in vitro study and a study in hamsters in 2020 suggested that the drug could have beneficial effects in fighting SARS-CoV-2 infection^[1,2], prompting scientists to study the drug for its potential as a COVID-19 treatment.

Since that time, a body of clinical research has grown around the subject. At the moment, public health authorities like the World Health Organization (WHO) and regulatory bodies like the U.S. Food and Drug Administration don’t recommend ivermectin as a COVID-19 treatment except in trials assessing its effectiveness. The available evidence so far, including clinical trials in humans, haven’t turned up evidence indicating that the drug is effective against the disease, as a Cochrane Systematic Review reported^[3].

While there are some studies reporting positive results, which were cited by proponents of ivermectin as evidence that the drug is effective against COVID-19, the BBC reported in October 2021 that “more than a third of 26 major trials of the drug for use on Covid have serious errors or signs of potential fraud. None of the rest show convincing evidence of ivermectin’s effectiveness”.

Similarly, the latest WHO guideline on the use of ivermectin, which was formed through a discussion of the existing data by a panel of experts, didn’t find conclusive evidence of ivermectin’s effectiveness. The experts analyzed 16 randomized controlled trials, which are considered the gold standard for evaluating a drug’s safety and effectiveness. However, only five of the trials directly compared ivermectin to standard treatment (standard of care), and two of the five were at a high risk of bias due to inadequate randomization. In this review, scientists told Health Feedback that any beneficial effect observed in the group treated with ivermectin disappeared once the studies at a high risk of bias were removed.

In early December 2021, another preprint about ivermectin emerged, which concluded that “Prophylactic use of ivermectin showed significantly reduced COVID-19 infection rate, mortality rate and chance of dying from COVID-19”. A preprint is a report of a study that hasn’t been peer-reviewed by other scientists or published yet. The study was touted in articles by Gateway Pundit, Zero Hedge, and The Blaze as evidence of the drug’s effectiveness against COVID-19. All in all, these articles received a total of more than 8,300 user engagements on Facebook to date, including more than 5,300 shares. [Editor’s note: The preprint has since been published in the scientific journal Cureus (see here).]

However, the study contains multiple methodological flaws that call into question the reliability of its conclusions, as explained by epidemiologist Gideon Meyerowitz-Katz in detail on Twitter.

Briefly, the preprint in question was a retrospective analysis of data from July to December 2020 in a clinical database for a COVID-19 prevention program implemented in the Brazilian city of Itajaí. The aim of the study was to determine the effects of prophylactic (preventative) ivermectin on the risk of infection, hospitalization, and mortality. To do this, the authors compared the disease outcomes (infection and mortality) in more than 133,000 people who used ivermectin with that of more than 87,000 people who didn’t. A person was considered to be an ivermectin user if they opted into the program.

For example, despite the authors’ claim to have controlled for all relevant confounding factors in the study, Meyerowitz-Katz pointed out that they didn’t account for confounding factors influencing risk of infection, like occupation and income, despite being a study examining ivermectin’s effect on risk of infection. 

Confounding factors are variables that affect the outcome of an experiment, but aren’t the variables being studied in the experiment. For example, scientists may wish to study the causes of heart disease. They may observe that red meat consumption is associated with a higher risk of heart disease. But it is possible that people who tend to eat more red meat might also be more likely to smoke cigarettes. In such a case, one possible explanation for the observation is that cigarette smoking, rather than red meat consumption, is the true influence behind that risk. This makes cigarette smoking a confounding factor.

The above example illustrates how failing to account for such confounding factors can lead scientists to draw erroneous conclusions about the cause of an effect that they observe in the study.

10/n For example, there's no control for occupation, nothing about income, no analysis of the results looking at many well-known risk factors for COVID-19 infection and death

— Health Nerd (@GidMK) December 16, 2021

In addition, it’s unknown how many people in both groups had already been taking ivermectin before the study. It’s also unknown whether people who were part of the ivermectin treatment group had consistently taken the drug throughout the study. In fact, there are indications that many who had initially started the ivermectin treatment didn’t continue the treatment. This raises questions about whether any effects seen in the ivermectin treatment group can be reliably attributed to ivermectin in the first place. Indeed, it’s unknown whether the improvement observed in the treatment group would still remain if people who didn’t actually take ivermectin were removed from that group and the data re-analyzed.

15/n There's also quite strong evidence that many of the "ivermectin" group did not pick up the medication, and stopped taking it almost immediately https://t.co/sKmC2N0kwD

— Health Nerd (@GidMK) December 16, 2021

In other words, attributing causality is challenging, given that the authors didn’t monitor patients’ adherence to ivermectin treatment. As such, they didn’t have a reliable way of knowing who was and wasn’t taking the drug and whether the drug was being taken in the correct dose. Consequently, there could be people who didn’t adhere to the treatment that were nevertheless assigned to the group of ivermectin users. Conversely, there could have been others who were taking ivermectin but were assigned to the group of non-users instead. 

16/n This means that there may have been a large proportion of people in the control group taking ivermectin, and a similar proportion in the intervention group NOT taking ivermectin. There's no analysis of this issue in the paper that I can see

— Health Nerd (@GidMK) December 16, 2021

Interestingly, the authors acknowledged this issue in the Discussion: “Due to the large number of participants, this citywide program was unable to supervise whether ivermectin users were using ivermectin regularly, in the correct dose and interval proposed.” But they didn’t fully account for its potential effects on their findings, instead simply suggesting that their findings underestimated ivermectin’s beneficial effects, implying that the true benefit was even larger. While that is one possibility, the authors didn’t appear to consider how the problems above may have also obscured a potential absence of benefit or even harm from ivermectin treatment. There isn’t evidence to support the a priori assumption that ivermectin treatment can only lead to improvement.

In his concluding remarks, Meyerowitz-Katz called it “a very weak observational study” that gave us “no useful information at this point in the ivermectin literature”:

18/n But overall, it's just not a very convincing paper. There is a huge potential for uncontrolled confounding, there are issues with the delineation of intervention vs control groups, and it's generally just a very weak observational study

— Health Nerd (@GidMK) December 16, 2021

It’s also of note that Itajaí actually saw the highest COVID-19 mortality rate in 2020 among the major cities in the state of Santa Catarina, another observation that casts doubt over whether the COVID-19 prevention program using ivermectin was actually effective.

Finally, at least two of the authors, Flavio Cadegiani and Juan Chamie, are members of a group called the Front Line COVID-19 Critical Care Alliance (FLCCC). This group, led by physicians Pierre Kory and Paul Marik, propagated unsupported claims about ivermectin as an effective COVID-19 treatment. Cadegiani is also the principal investigator of a clinical trial that is being investigated for potential violations of medical ethics and human rights by Brazil’s National Health Council, as reported by the British Medical Journal. Cadegiani and Lucy Kerr also have financial conflicts of interest from their work for a company that produces ivermectin.

The misinformation about an unproven treatment, arising in part as a result of unreliable studies, has the potential to place people’s lives at risk. From Scientific American:

“The notion that ivermectin is a miracle medicine gives people who reject vaccines a false sense of security, says Daniel Griffin, a physician and infectious disease researcher at Columbia University and Chief of the Division of Infectious Disease at the company ProHEALTH […] the hype over the antiparasitic drug may keep some people from getting vaccinated but also that sick people taking it at home might delay going to a hospital and miss the efficacy window for evidence-based COVID treatments.”

In summary, the current pool of reliable evidence on ivermectin’s effectiveness against COVID-19 doesn’t support its use as a treatment, except in clinical trials. Scientists are conducting more studies to acquire more data that will help us better answer the question of whether ivermectin is an effective COVID-19 treatment.

UPDATE (25 Jan. 2022):

We updated the review to indicate that the preprint by Kerr et al. has since been published in the scientific journal Cureus (see here).

REFERENCES

• 1 – Caly et al. (2020) The FDA-approved drug ivermectin inhibits the replication of SARS-CoV-2 in vitro. Antiviral Research.
• 2 – Dias de Melo et al. (2020) Anti-COVID-19 efficacy of ivermectin in the golden hamster. bioRxiv. [This preprint has since been published in EMBO Molecular Medicine.]
• 3 – Popp et al. (2021) Ivermectin for preventing and treating COVID‐19. Cochrane Systematic Reviews.