BMJ 2014;348:g2263 doi: 10.1136/bmj.

g2263 (Published 9 April 2014) Page 1 of 3

Analysis

ANALYSIS

Multisystem failure: the story of anti-influenza drugs

Last year the Cochrane team, with the help of the BMJ’s open data campaign, finally got access to
full clinical study reports on neuraminidase inhibitors. Tom Jefferson and Peter Doshi explain what
the new systematic review found and how a series of failures meant that decisions about these
drugs were made without the full evidence

1 2
Tom Jefferson reviewer , Peter Doshi assistant professor

Cochrane Acute Respiratory Infections Group, 00187, Roma, Italy; 2Department of Pharmaceutical Health Services Research, University of Maryland
1

School of Pharmacy, Baltimore, USA

For the past decade decisions makers worldwide have endorsed pledge to do so made during the H1N1 “swine flu” outbreak of
the use of neuraminidase inhibitors. They spent billions of 2009.7 8
pounds stockpiling the two anti-influenza drugs oseltamivir and Clinical study reports, which are used in regulatory submissions,
zanamivir from the mid-2000s as part of a global effort to be are comprehensive structured reports of industry sponsored
prepared for an influenza pandemic. When the H1N1 pandemic trials that can run to hundreds or thousands of pages.9 They
emerged in 2009 the drugs were rolled out around the globe for include the study protocol, statistical analysis plan, blank case
treatment and prevention of influenza and its complications. report forms, and other appendices that provide important
Under this spotlight, we were asked to conduct a systematic contextual information, such as certificates of analysis describing
review for Cochrane to update evidence on their efficacy. What the content and physical appearance of the intervention and
should have been a routine review got complicated as the placebo.
validity of a key study that underpinned the evidence on efficacy
Research for our 2009 review also highlighted inconsistencies
was unclear. Our three and half year battle for data has resulted
in decision making. The Food and Drug Administration (FDA),
in the drug manufacturers providing us with full clinical study
which had access to the full clinical study reports, concluded
reports and unveiled a story in which no party has taken full
on the product label that “Tamiflu has not been shown to prevent
responsibility for ensuring the validity of the evidence
such complications [serious bacterial infections].” The European
underlying its decisions. We hope that the publication of our
Medicines Agency (EMA), which had only partial reports, and
systematic review of the trials, alongside all the source clinical
another prominent US agency, the Centers for Disease Control
study reports,1 will change the way such decisions are made.
and Prevention (CDC), came to the exact opposite
Emergence of problems conclusion—all apparently based on the same trials.10
Officials have not unambiguously documented their reasons for In 2013, following a long running campaign by the BMJ and
stockpiling oseltamivir and zanamivir, but the decision seems ourselves, Roche released full clinical study reports for what
to be based on the assumptions that the drugs would reduce we believe are close to all of the oseltamivir studies it sponsored
hospital admissions and serious complications of influenza such without any limitations on their use or access. GlaxoSmithKline
as pneumonia by half and slow down the spread of the virus.2-4 also released the reports for studies of zanamivir, and we have
Some of these assumptions were supported by a peer reviewed now published our analysis of them.1
pooled analysis of 10 randomised trials of oseltamivir published
in the Archives of Internal Medicine in 2003 by Kaiser and Unimpressive results
colleagues.5 Although this analysis seemed to be high quality
science and formed a powerful scientific rationale for The results of our systematic review challenge some of the
stockpiling,6 during our review in 2009 it became apparent that assumptions about these drugs. Although prophylactic use does
the data underlying it were largely unpublished and inaccessible reduce the risk of developing symptomatic influenza, because
to independent scrutiny. Roche, the manufacturer of oseltamivir, virus culture was not performed on all trial participants it is not
funded the Kaiser review, employed some of its authors, and clear whether this is because participants were not infected or
had also sponsored the 10 trials. But for three and half years it because they had an asymptomatic infection. This is important
refused to release the full clinical study reports despite a public because it is thought that infection is also spread by people with

Correspondence to: T Jefferson [email protected]

For personal use only: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe
BMJ 2014;348:g2263 doi: 10.1136/bmj.g2263 (Published 9 April 2014) Page 2 of 3

ANALYSIS

asymptomatic infection (www.accessdata.fda.gov/drugsatfda_ Like complications, adverse event outcomes were not defined
docs/nda/2000/21-087SE1-002_review.pdf). in the trials, so the quality of the data is variable. However, the
There is no trial evidence to show that the drugs had any effect prophylaxis study population included people of all ages with
on mortality: only 13 deaths occurred among the over 24 000 no influenza-like symptoms, providing a far better testing ground
participants enrolled in the clinical trials (about 0.05%). Nor for learning about harms than treatment trials.
was there evidence that treatment with oseltamivir reduced the
risk of hospital admission in adults (relative risk 0.92, 95% Multiple failures
confidence interval 0.57 to 1.50) or children (1.92, 0.70 to 5.23);
It seems that every major player could have acted differently to
no data were collected for zanamivir. Like mortality, hospital
ensure that the true picture was available sooner. If evidence
admission was a rare event, and no trial protocol included either
played a role in government decisions to stockpile these
as an outcome of interest.
drugs—and we would like to think it did—we need a rapid
Although the Kaiser analysis claimed that oseltamivir reduces accounting of the judgments, evidence, and considerations that
the risk of complications, the trials do not settle this question underlie the original decision as well as the present continued
completely, particularly for pneumonia. Pneumonia is difficult stockpiling.
to diagnose clinically, and no oseltamivir study report included
a definition of pneumonia or any other complication, meaning Regulatory questions
that we (and possibly the trialists themselves) could not verify
that the records of pneumonia or bronchitis were accurate. Most The FDA, which was the only body to re-run the analyses and
trials relied on participant self reporting. The lack of a definition thoroughly review some clinical study reports, considered that
is unsurprising because complications were not a specified the benefits of these drugs were “modest” 15 years ago (this
outcome in the protocol for half of the trials. For the other half, adjective appears six times in a four page oseltamivir medical
complications were a secondary or tertiary outcome. Our officer review document11). However, despite the statements to
meta-analysis of unverified pneumonia events suggests that this effect in the product labels, the drugs were adopted
oseltamivir reduces the risk in adults (relative risk 0.55, 0.33 to enthusiastically. Today, one US public health agency, CDC,
0.99 (number needed to treat (NNT)=100, 95% CI 67 to 451)) continues to suggest that neuraminidase inhibitors “may reduce”
but not in children (relative risk 1.06, 0.62 to 1.83). But as there complications and death,12 although the FDA still makes only
were only 66 cases of pneumonia among 4452 participants, the conservative conclusions.13
results are not robust. To gauge oseltamivir’s possible effect We might also ask whether regulators should approve drugs
against serious unverified pneumonia and other complications, that they conclude are only modestly effective against placebo.
we carried out an additional analysis of all secondary illnesses Our reading of the FDA’s files on zanamivir suggests that the
that led to hospital admission or withdrawal from the trial. With drug was approved not because of its performance but because
even fewer events (24), the result was not significant (0.91, 0.4 it offered “an alternative therapeutic approach for an important
to 2.06). public health problem” in a market where “current influenza
Similar problems were present in the zanamivir trials, but we treatment options [were] limited.”14
found no significant effect on unverified pneumonia (0.90, 0.58 The European Medicines Agency approved oseltamivir without
to 1.40). Zanamivir reduced the risk of unverified bronchitis in the the full dataset from Roche, which is not required as part of
adults (NNT=56, 36 to 155) but the reduction with oseltamivir the European licensing process. The EMA also uses external
was non-significant (relative risk 0.75, 0.56 to 1.01). Neither experts to assess submissions for market approval, and the 2009
drug had a significant benefit on bronchitis in children. BMJ investigation found that two of the experts the EMA
Both drugs reduce the time to first alleviation of influenza consulted for advice were featured in Roche promotional
symptoms by around half a day, but this was not the reason that material. It is not known whether they disclosed this to the
they were stockpiled. Furthermore, in some zanamivir trials, EMA.15 Competing interests should always be declared, but
there was no analysis according to whether infected participants such problems could be avoided if the EMA was funded
took other drugs to relieve symptoms (mainly paracetamol), sufficiently to allow it to evaluate the evidence internally without
and the data are not reported in detail in the oseltamivir clinical relying on external experts.
study reports. Therefore the relative contribution of
neuraminidase inhibitors versus rescue medications in alleviation Research methods and funding
of symptoms is unclear. Our investigation calls into question whether credible evidence
In short, the benefits of both drugs appear modest, and these synthesis should rely on peer reviewed publications. Current
need to be weighed against possible harms. timelines and funding structures do not give sufficient resources
for systematic reviewers to analyse detailed clinical study reports
Data on harms rather than short journal articles. We were fortunate to have
received funding for our four year review, but there are many
Concerns about harms have largely been absent from public other therapeutic areas where in-depth, credible evidence
discussions on oseltamivir, and the Kaiser analysis did not report synthesis is needed. Given the increasing public availability of
the risk of harms. Our results show no excess rates of observed clinical study reports,9 funders need to reconsider how they
harms in adults treated with zanamivir. However, when used to allocate resources towards evidence based medicine.
treat influenza, oseltamivir increased the risk of nausea (relative In 2013, Roche began funding the Multiparty Group for Advice
risk 1.57, 1.14 to 2.15; number needed to harm (NNH)=28, 14 on Science (MUGAS) to reanalyse the oseltamivir dataset, but
to 112) and vomiting (2.43, 1.75 to 3.38; NNH=22, 14 to 42). industry funding raises questions about objectivity. At the first
Prophylactic use increased the risk of headache (1.18, 1.05 to MUGAS meeting, Roche downplayed the importance of its trial
1.33; NNH=32, 18 to 115) and psychiatric adverse events over data in settling the question about complications: “We didn’t
the duration of follow-up (1.80, 1.05 to 3.08; NNH=94, 36 to ask physicians to actively look for complications … They simply
1538).1 reported them if they thought patients had, for example, sinusitis,

For personal use only: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe
BMJ 2014;348:g2263 doi: 10.1136/bmj.g2263 (Published 9 April 2014) Page 3 of 3

ANALYSIS

otitis media, bronchitis, pneumonia, or other chest infections. 1 Jefferson T, Jones MA, Doshi P, Del Mar CB, Hama R, Thompson MJ, et al. Neuraminidase
inhibitors for preventing and treating influenza in healthy adults and children. Cochrane
… To be honest, we weren’t that stringent at the time.”16 Yet Database Syst Rev 2014;4:CD008965.
the Roche funded Kaiser analysis concluded, without 2 UK Department of Health. UK influenza pandemic contingency plan. 2005. www.dh.gov.
uk/prod_consum_dh/groups/dh_digitalassets/@dh/@en/documents/digitalasset/dh_
qualification, that oseltamivir reduces such complications. 4121744.Pdf.

Lastly, the journals that published the studies have yet to correct 3 US Department of Health and Human Services. HHS Pandemic Influenza Plan. 2005
www.pandemicflu.gov/planning-preparedness/federal/hhspandemicinfluenzaplan.pdf.
the reporting biases that we have documented,17 and the editorial 4 World Health Organization. WHO interim protocol: rapid operations to contain the initial
world has yet to tackle the failure of peer review to detect them. emergence of pandemic influenza. www.who.int/entity/csr/disease/avian_influenza/
guidelines/RapidContProtOct15.pdf.
5 Kaiser L, Wat C, Mills T, Mahoney P, Ward P, Hayden F. Impact of oseltamivir treatment

Final word on influenza antivirals? on influenza-related lower respiratory tract complications and hospitalizations. Arch Intern
Med 2003;63:1667-72.
6 Jefferson T, Demicheli V, Rivetti D, Jones M, Di Pietrantonj C, Rivetti A. Antivirals for
We hope public health bodies such as CDC and the World influenza in healthy adults: systematic review. Lancet 2006;367:303-13.
Health Organisation will consider our review’s conclusions and 7 Doshi P, Jefferson T, Del Mar C. The imperative to share clinical study reports:
recommendations from the Tamiflu experience. PLoS Med 2012;9:e1001201.
revise their recommendations. CDC’s endorsement of 8 Thomas K. Breaking the seal on drug research. New York Times 2013 Jun 29 www.
neuraminidase inhibitors is now based on analyses that include nytimes.com/2013/06/30/business/breaking-the-seal-on-drug-research.html.
9 Doshi P, Jefferson T. Clinical study reports of randomised controlled trials: an exploratory
retrospective observational studies, many of which have review of previously confidential industry reports. BMJ Open 2013;3:e002496.
substantial amounts of missing data and did not adjust for 10 Doshi P. Neuraminidase inhibitors—the story behind the Cochrane review. BMJ

survivor bias (where those who die earlier have less opportunity 11
2009;339:b5164.
Centers for Drug Evaluation and Research. Application for oseltamivir. 1999. www.
to receive treatment).18 WHO considers oseltamivir important accessdata.fda.gov/drugsatfda_docs/nda/99/21087_Tamiflu_medr_P1.pdf
enough to place it on the list of essential medicines that should 12 US Centers for Disease Control and Prevention. Influenza antiviral medications: summary
for clinicians. 2012. www.cdc.gov/flu/professionals/antivirals/summary-clinicians.htm.
be universally available,19 20 and the many unproved assumptions 13 Hoffman-La Roche. Product label. Tamiflu (oseltamivir phosphate) capsules and for oral
about antiviral performance that supported pandemic plans suspension. 2014. www.accessdata.fda.gov/drugsatfda_docs/label/2013/021087s061,
021246s044lbl.pdf.
largely remain unchanged. We need to act to make sure that 14 FDA. Division director memorandum on zanamivir. 1999. www.accessdata.fda.gov/
future decisions are not made on incomplete data. drugsatfda_docs/nda/99/021036-medreview9.pdf.
15 Cohen D. Complications: tracking down the data on oseltamivir. BMJ 2009;339:b5387.
16 Groves T. What does oseltamivir do, and how will we know? BMJ 2013;347:f4687.
Competing interests: Both authors have read and understood the BMJ 17 Doshi P, Jones MA, Jefferson T. Rethinking credible evidence synthesis. BMJ
2012;344:d7898.
Group policy on declaration of interests and declare that they have 18 US Centers for Disease Control and Prevention. CDC recommendations for influenza
applied for and received competitive research grants. Both authors are antiviral medications remain unchanged. CDC Newsroom, 7 February 2012. www.cdc.
gov/media/haveyouheard/stories/Influenza_antiviral.html.
co-recipients of the NIHR grant to carry out the linked Cochrane review.
19 World Health Organization. WHO model list of essential medicines. ADULTS, 18th ed.
TJ receives royalties from his books published by Blackwells and Il 2013. http://apps.who.int/iris/bitstream/10665/93142/1/EML_18_eng.pdf.
Pensiero Scientifico Editore and is occasionally interviewed by market 20 World Health Organization. WHO model list of essential medicines for children. 2013.
http://apps.who.int/iris/bitstream/10665/93143/1/EMLc_4_eng.pdf.
research companies about phase 1 or 2 pharmaceutical products. In 21 Chan M. Swine influenza. 2009 www.who.int/mediacentre/news/statements/2009/h1n1_
2011-2013 TJ was an expert witness in a litigation case related to 20090425/en/index.html.
22 Nicholson KG, Aoki FY, Osterhaus AD, Trottier S, Carewicz O, Mercier CH, et al. Efficacy
oseltamivir and in a labour case on influenza vaccines in health care and safety of oseltamivir in treatment of acute influenza: a randomised controlled trial.
workers in Canada. He has acted as consultant for Roche, GSK, and Lancet 2000;355:1845-50.
23 Treanor JJ, Hayden FG, Vrooman PS, Barbarash R, Bettis R, Riff D, et al. Efficacy and
Sanofi-Synthelabo and was a consultant for IMS Health in 2013. PD
safety of the oral neuraminidase inhibitor oseltamivir in treating acute influenza: a
received €1500 from the European Respiratory Society in support of randomized controlled trial. JAMA 2000;283:1016-24.
his travel to the society’s September 2012 annual congress in Vienna, 24 Jefferson T, Jones M, Doshi P, Del Mar C. Neuraminidase inhibitors for preventing and
treating influenza in healthy adults: systematic review and meta-analysis. BMJ
where he gave an invited talk on oseltamivir. He is an associate editor 2009;339:b5106.
of the BMJ. 25 Smith J, on behalf of Roche. Point-by-point response from Roche to BMJ questions. BMJ
2009;339:b5374.
Provenance and peer review: Commissioned; externally peer reviewed.
Cite this as: BMJ 2014;348:g2263
© BMJ Publishing Group Ltd 2014

For personal use only: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe
BMJ 2014;348:g2263 doi: 10.1136/bmj.g2263 (Published 9 April 2014) Page 4 of 3

ANALYSIS

Story of neuraminidase inhibitors

Mid-late 1990s: Around 40 Roche sponsored randomised clinical trials of oseltamivir and 25 Glaxo-Wellcome sponsored randomised trials of zanamivir are
completed
July 1999: FDA approves zanamivir for treatment of influenza
October 1999: FDA approves oseltamivir for treatment of influenza
November 2000: FDA approves oseltamivir for prophylaxis of influenza
June 2002: EMA approves oseltamivir for prophylaxis and treatment
November 2000: FDA tells Roche that its promotional material claiming a 45% reduction in secondary infections is “false or misleading”
2003: Kaiser and colleagues’ pooled analysis of 10 randomised clinical trials conducted before licensing in 1999 is published in Archives of Internal Medicine
and concludes that oseltamivir reduces the risk of lower respiratory tract infections resulting in antibiotic use (55%; 4.6% v 10.3% for placebo) and hospital
admissions (59% 1.7% v 0.7%) in adults aged 13 to 97 years5
2004-5: Governments around the world begin stockpiling oseltamivir spurred by fears of avian influenza H5N1
2005: UK and US pandemic contingency plans highlight the importance of antivirals in reducing the impact of a pandemic: Both state treatment will reduce hospital
admissions by about 50% and decrease mortality, based on the Kaiser analysis2 3
January 2006: Cochrane review concludes that oseltamivir reduces complications such as pneumonia. The Kaiser 2003 paper drove the result in meta-analysis
March 2006: FDA approves zanamivir for prophylaxis of influenza
2009: Novel A/H1N1 influenza virus discovered to be spreading in North America. In June 2009 WHO declares A/H1N1 influenza a “pandemic”21
2009: Australian and UK governments commission rapid update of Cochrane reviews related to influenza
2009: Tip from a reader alerts the Cochrane team that only two of the 10 trials in the Kaiser analysis were published
2009: We request data from the authors of the Kaiser pooled analysis and the oseltamivir “pivotal” studies.22 23 The authors respond that they do not have the
data and refer us to Roche
December 2009: Unable to (and unwilling to sign a confidentiality agreement with a secrecy clause necessary to) obtain the 10 trials’ raw data, we conclude that
we no longer are sure whether oseltamivir reduces complications of influenza.24 A joint investigation by BMJ and Channel 4 News shows that one of the published
trials had been ghostwritten and that the largest treatment trial of oseltamivir conducted (M76001) was presented as a conference abstract carrying the name of
a professor who did not recollect being involved; it was never published in full15
December 2009: Roche promises to release full clinical study reports to legitimate investigators “within the coming days.”25 At the end of the month it releases
3195 pages of study report but none are complete
2010-2012: Cochrane team repeatedly requests the full clinical study reports
2011: A freedom of information request to the European Medicines Agency provides the Cochrane team with over 20 000 pages from 16 Roche oseltamivir
clinical study reports. EMA has no data on zanamivir (the drug was approved at the national regulatory agency level). All but one oseltamivir report was incomplete
2012: We publish the interim version of our Cochrane review based on EMA’s incomplete clinical study reports and regulatory comments from the FDA. Our
conclusions that there is no evidence the drugs reduce hospital admission and the evidence for or against a possible effect on complications is insufficient are
dismissed by the CDC, WHO, and European Centre for Disease Prevention
February 2012: WHO refuses to answer our questions on the review process that led to the inclusion of oseltamivir on the essential medicines list (www.bmj.
com/tamiflu/who)
February 2012: CDC refuses to answer our requests for clarification on what data its continued promotion of oseltamivir is based on (www.bmj.com/tamiflu/cdc)
October 2012: BMJ begins publishing Cochrane correspondence with Roche, EMA, CDC, and WHO at bmj.com/tamiflu as part of its Open Data campaign
2013: GSK suddenly decides that a contract on data use is no longer necessary to access the zanamivir studies and send the full reports for the 30 trials we
requested
2013: Roche subsequently releases 77 full clinical study reports of Roche oseltamivir trials
2013: Roche begins funding the Multiparty Group for Advice on Science (MUGAS) to reanalyse the oseltamivir dataset
April 2014: We publish our updated review based solely on full clinical study reports and regulatory documents

For personal use only: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe