clock menu more-arrow no yes

Filed under:

Is the rush to understand COVID-19 making us more confused about COVID-19?

A growing number of studies are being published without peer review. Some are reliable and some are not.

Lab technicians speak with each other while researching COVID-19 at Johnson & Johnson subsidiary Janssen Pharmaceutical in Beerse, Belgium, on Wednesday, June 17, 2020. Janssen Pharmaceutical hopes to begin clinical trials on a potential vaccine for COVID-19 in the middle of the summer.
Virginia Mayo, Associated Press

SALT LAKE CITY — The day after the World Health Organization declared COVID-19 a global emergency, a group of scientists from India published a research article with startling conclusions: the coronavirus had “uncanny” and “unlikely” genomic similarities to HIV.

The article was published in January on bioRxiv, a premier biology-themed preprint server, or website for research articles that have not yet been peer-reviewed. As soon as the article came out, it was attacked by researchers around the world for shoddy work. The article was retracted in a matter of days.

But that wasn’t fast enough, according to Jevin West, a data scientist at the University of Washington and director of the Center for an Informed Public. The article became one of “the most viral ever in the history of science,” he said, based on altmetrics that measure shares on social media and links in news articles. Conspiracy theorists latched on, claiming in blogs and on Twitter that the now-debunked study was evidence that the virus is a bioweapon man-made to be highly infectious.

The rush to understand the novel coronavirus has led to an outpouring of new research from epidemiologists and scientists. Preprint servers like bioRxiv have surged in popularity as scientists feel an urgency to share any information that might help save lives, West said.

But the troves of unvetted studies could be why there is so much confusion about the pandemic, according to West, and provide an explanation for the spread of some conspiracy theories about the virus. While the existence of bad science is nothing new, he said, people desperate for answers are more likely to pay attention to flawed studies about the coronavirus.

“I do think this flood of research and this flood of information in science is great, but it makes it harder to discern which is the stuff to really get behind, and which is the stuff we should toss out,” said West.

Together, bioRxiv and a similar site called medRxiv, which is run by the same nonprofit institution, Cold Spring Harbor Laboratory, have published more than 7,800 preprint articles about COVID-19.

According to Harlan Krumholz, a founder of medRxiv and a professor of medicine at Yale University, papers on the site undergo a rapid screening process to make sure they are not plagiarized, that the subjects of trials and experiments are not identified and that the studies are not likely to cause harm. But while the vast majority of papers on the site are written by qualified scientists, ultimately anyone with or without training can publish on medRxiv, Krumholz said. He calls it “citizen science.”

Preprint servers give people all over the world access to early research and help scientists improve their work by letting others offer feedback in real time, Krumholz said. “It is people who are irresponsibly promoting bad science who are confusing the public.”

He admits that some things get through that are “made up” and thus need to be withdrawn, like the “uncanny similarities” article on bioRxiv, which went through a separate screening process. But withdrawals are extremely rare and happen relatively quickly, he said.

“Science is progressive and self-correcting,” said Krumholz.

Ultimately, preprint servers are a useful tool if the media makes it clear they represent preliminary findings that have not been scrutinized by peer review, said West. “Preprints can help accelerate science, as long as we’re cautious and we know the limitations.”

Confusion

An increasing number of “armchair epidemiologists” and scientists who aren’t necessarily experts in that field are getting involved in studies about the coronavirus, said Kate Grabowski, an infectious disease epidemiologist at Johns Hopkins University who is leading an effort to create a curated set of pandemic papers called the 2019 Novel Coronavirus Research Compendium.

“Everyone is hungry for any information that will get us out of this pandemic sooner, myself included,” said Grabowski. “This is unprecedented in our time.”

But the people doing research in areas they are unfamiliar with are drowning out the “true experts,” West said.

In addition to misleading papers being published without peer review, some faulty studies have been published in peer-reviewed journals that were negligent or that expedited the publication process because of the coronavirus.

In March, President Donald Trump referenced a peer-reviewed study from France that was based on 20 patients and indicated hydroxychloroquine could be a treatment for the coronavirus. Trump called it “one of the biggest game changers in the history of medicine.” But after widespread scrutiny from the scientific community, the study was retracted. The president of the International Society of Antimicrobial Chemotherapy, which published the research, released a statement saying it did not meet expected standards.

Peer review is vital and essential, according to Grabowski, but currently, the process is being rushed. To some extent, there is confusion about the coronavirus because it’s only been circulating for less than a year, whereas it’s taken decades to acquire knowledge on other viruses, she said.

“I think we all need to step back and take a breather. It’s urgent that we know this information, but we have to be willing to evolve on our stances. We have to let the data lead us where it leads us,” said Grabowski.

“Science is typically slow, but it has been pushed into the fast lane, and it’s not comfortable in the fast lane,” said West.

The limits of peer review

While preprint publications come with the risk of spreading bad science, the traditional peer review process isn’t perfect either, said Daniel Himmelstein, a data scientist at Related Sciences, a disease research and drug development company.

“Peer reviews at journals don’t weed out all bad science,” said Himmelstein. “There are bad articles that are preprint and bad articles published in journals. ... We’ve known this for a while.”

The 1998 Andrew Wakefield study linking the MMR vaccine to autism was famously retracted from the Lancet, one of the world’s most respected general medical journals, after 12 years. But there are countless other examples of retractions from established publications, said Theodora Bloom, executive editor of the BMJ (British Medical Journal).

According to Himmelstein, who has published and reviewed numerous articles himself, beyond a sense of duty, scientists have few incentives to accept peer review requests and do a thorough job. It’s not a fun task, they are not paid and their names are usually kept anonymous so their reputation is not at stake either, said Himmelstein.

Peer review also takes a long time. Himmelstein calculated that one round of peer review takes on average 100 days, but that doesn’t include the extra time it takes when a paper is rejected by a journal and the authors have to reapply.

“That whole process could take years, and that research is growing more stale as time goes on,” said Himmelstein. “With COVID, it doesn’t make sense. It really slows down science.”

“Given there are hundreds of thousands of people dying, you want to know about a treatment as soon as possible,” said Theodora Bloom. That’s where preprint servers are beneficial, she said. Based on preprint research that so far has not been published in a peer-reviewed journal, England’s chief medical adviser recommended that Dexamethasone be used as a coronavirus treatment to save the lives of the severely ill, said Bloom.

“With Ebola, SARS and MERS, mostly those outbreaks had passed before the research on them was published,” said Bloom, who helped found medRxiv along with Krumholz. “That was how things played out.”

Himmelstein added that because comments on preprint servers are public, scientists can see and respond to feedback immediately, whereas most journal websites don’t have space for comments at all.

“I think public feedback is the key there. If we can get scientists discussing, commenting and reviewing in public, we can get a much better public assessment,” Himmelstein said.

Finding the truth

In the end, no single study will prove that something is true or not, said Jessica Polka, executive director of ASAPbio, a nonprofit promoting transparent and innovative science communication based in Cambridge, Massachusetts.

Scientific discovery is a process that takes multiple minds working together, she said.

“There’s a lot of potential for misinformation in this pandemic,” said Polka. “I think the antidote for that in my opinion is a system of commentary, displayed alongside the initial claim with proper contextualization.”

She is happy to see more preprint servers adding banners and disclaimers to their sites, explaining that the research is preliminary.

“It is very good to see a lot of people devoting their energies to working on this problem,” she added.

“Science is a building on lots of little bricks. Some are wobbly and some are very small, it’s not until you see them all in the same general direction that you start to believe it,” Bloom said.

Like the faulty hydroxychloroquine study that was based on a small number of people, one initial report saying that loss of taste and smell could be a symptom of the coronavirus was looked at just six patients in Italy, said Bloom. The difference between the two, is that numerous other studies came out supporting the loss of taste and smell claim, she said.

“The risk is when people who aren’t trained in reading the literature are trying to pick any one paper and say it proves something,” said Bloom. “The best science comes to the top.”