Peer Review
Peer Review
Published by Sense About Science 60 Cambridge Street London SW1V 4QQ Tel: +44 (0)1795 591975 Email: Web: [email protected] www.senseaboutscience.org
Registered Charity No. 1101114 ISBN 0-9547974-0-X May 2004 2004 Sense About Science Reproduction of short parts of this text for the purposes of research and comment is permitted. No part of the publication may be reproduced for any other purpose, nor stored in a retrieval system, nor transmitted in any form or by any means, without the prior permission of the copyright owners. Online version The full text of this discussion paper is available to download as a pdf document, free of charge, from http://www.senseaboutscience.org Publication orders Printed copies of this report are available for 10 to cover production and postage (all countries). A limited number of copies are available at no charge. To order, and to request review copies, please contact [email protected] A leaflet version of Section 2, A short guide to peer review, will be available for distribution in October 2004. Please contact Ellen Raphael at [email protected] Printing and design Printing costs kindly paid by the Medical Research Council as a contribution on behalf of all of the Research Councils. Printed in the UK by Short Run Press Ltd, Exeter. Cover design kindly contributed by Andrew Giaquinto, Institute of Physics Publishing.
PEER REVIEW
November 2002 May 2004
Contents
Page Foreword & Acknowledgements The Working Party Executive Summary Section 1: A cultural challenge for peer review?
Debates about scientific research Science in a challenging environment Can peer review help society navigate the constant stream of science news? Why should the public be interested in peer review?
vii viii ix 1
1 2 4 5
7
7 7 9 9 11 11 13 13
14
Should referees be anonymous? Are some scientists results suppressed through rejection of their papers? Is maverick science rejected through peer review? Does peer review reinforce scientific resistance to change? Why cant there just be a technical checking system, rather than peer review, to make sure the researchers have abided by good practice? Should research be judged on the basis of who funds it? The limits: when is peer review not really peer review? iv
15 16 16 17 18 18 19
But is it true? Error and correction Fraud and misconduct Some challenges for scientific publishing
The volume of research papers Electronic publishing and Open Access
19 20 21
22
23
23 25
28
Peer review as part of science education Equipping a wider public with an understanding of peer review
The social implications of research claims Clarifiying the contribution of peer review to scientific publishing
32 35
Concluding comments
37
Appendices
Appendix 1: References and sources Appendix 2: Conflict of interest form Appendix 3: Guidance on price sensitive information Appendix 4: Other projects and sources of information about peer review Appendix 5: Working Party members Appendix 6: Sense About Science 39 42 44 45 46 49
List of illustrations
Box 1: Referees usually comment and make recommendations on some of the following Box 2: Typical examples of referees comments Box 3: The scientific publishing scene Box 4: PPL Therapeutics and British Biotech Box 5: Scientific literacy Box 6: Flow chart indicating the typical process of peer review 8 10 12 28 33 34
vi
Foreword
Few outside the scientific world can be expected to know about the immense effort, and even pain, that is experienced before research is ready to be published as a peer-reviewed paper in a scientific journal. The peer-review process is almost as complicated and unpredictable as the birth of a baby. Yet, as with delivery, it involves analysis, judgement and evaluation or at least it should. We believe it is important for the public to understand more about the critical engagement and the checks and balances that help to determine the plausibility of new knowledge (Section 1). This report therefore seeks to provide an outline of the complex way that science enters the public arena (Section 2) and why the time-honoured peer-review process is crucial, though it could be improved and better understood (Section 3). This report also endeavours to show how things can go wrong. Delivery of results may be premature and a claim can reach the airwaves before adequate testing has been performed. A consequence is that the unsuspecting are presented with untested opinions rather than peer-reviewed conclusions. Donald Kennedy in a recent editorial in Science1 pointed out that peer review of a scientific paper involves the addition of qualifications and limitations on conclusions. Other forms of communication, by contrast, such as intelligence, news reporting and campaigning, often delete qualifying language and caveats so that scientific conclusions are strengthened and simplified. This tension between the description of experimental findings and interpretation arises for all who aim for clarity and urgency. Yet it is the pursuit of truth that remains fundamental to the scientific endeavour. This report is for scientists and for all who grapple with the barriers and difficulties arising from new knowledge as societies come to terms with the latest scientific and technological news. It is a discussion paper, written in a cultural context, that makes a more determined case for peer review from a social standpoint than we envisaged at the outset. It is the outcome of a vigorous debate within the Working Party and is published with its full support. May I take this opportunity to thank all the members of the Working Party for their immensely valuable contributions, many other colleagues who also gave most generously of their time and energies, and most of all Tracey Brown, who has worked tirelessly over the past 18 months in bringing the report to a successful conclusion. She has directed and inspired our wide-ranging deliberations and has brought them together in a comprehensive and comprehensible document. We are further indebted to Dr Bridget Ogilvie and Professor Onora ONeill for their review of the document. Finally, we are deeply appreciative of the help of Dr Irene Hames for her thorough and thoughtful editing over several drafts.
vii
viii
Executive Summary
Background
Public discussion is sometimes dominated by debates about the implications of scientific research: what causes Sudden Infant Death? Will genetically modified crops create superweeds? Is the Measles Mumps Rubella (MMR) vaccine linked to autism? Will transgenic pigs help to solve a shortage of organs for transplants? Do mobile phones damage childrens brains? What is an optimal dose of fluoride? Over one million papers about scientific research are published in scientific journals worldwide annually. To get a paper published, scientists submit their research findings to a journal, which sends them out to be assessed for competence, significance and originality, by independent qualified experts who are researching and publishing work in the same field (peers). This is known as peer review. Despite its extensive use and recognition among scientists in assessing the plausibility of research claims, in the rest of society very little is known about the existence of the peer-review process or what it involves. A Working Party was established by Sense About Science in November 2002 to consider how an understanding of peer review might help the public to weigh the relative merits of different research claims. This report of its discussions3 is for scientists and for the many groups who mediate and comment on scientific information. It also contains a guide to peer review (Section 2) for everyone who is interested in science.4
3 4
Recommendations are indicated in bold text. A short version of Section 2, A guide to peer review, will be available for separate distribution in October 2004.
ix
broader social interest in the reliability and quality of research, rather than identified with the preoccupations of particular scientific groups that their messages are not getting through. The public, in its widest sense, should be encouraged to ask questions about peer review when listening to claims about scientific findings in an interview, press release, or news report. Has the work been evaluated by experts in the field, or is the report based on opinion or unsubstantiated extrapolation? Is it acknowledged by other scientists as a contribution to the field, or dismissed because it is flawed? Has it been replicated? (p.22) While many concerns were raised with the Working Party about the serious effects on society of misleading and conflicting research claims from creating unnecessary parental anxieties to wasting research funds there is little empirical data about these effects. This paper recommends further collaborative research between scientists and social scientists to extend existing accounts of how science stories are reported and the questions that are asked about new research by different social groups. (p.36)
The peer review of scientific papers submitted to journals for publication has a widely proven record as a means to test the plausibility of new findings. However, scientists never regard peerreviewed research as beyond criticism. Peer review of a paper is just the first stage: a hypothesis that survives this first test must go on to be re-tested, and judged for its coherence with work in related areas. As with any system that is dependent on human judgement, such as jury trials and doctors diagnoses, mistakes are sometimes made by referees. These can result in valuable papers being overlooked by the higher quality journals, and also in weak or flawed papers occasionally being published. But if the findings are very significant, any flaws are likely to be discovered quite quickly because the paper will be widely read and discussed and other scientists will attempt to repeat the work. (This rapid discovery of mistakes is often referred to as science being selfcorrecting.) In short, the most basic problem with peer review is that so few citizens are made aware of it, at a time when people have become very concerned about how to weigh scientific research claims meaningfully. Scientific bodies should make systematic attempts to explain peer review and to communicate what it is to a wider public, especially when there is controversy about particular claims. Section 2 describes the peer-review process and addresses some of the questions about how and why it is used.
II.
III.
xi
results; but that they distinguish between this and matters of taste and style in how others choose to discuss their work. (p.27) IV. Scientific results are often discussed at conferences and there is growing media interest in these as a source of news. With talks covering a combination of new and old work, conference organisers would find it almost impossible to indicate systematically what has been peer reviewed. However, it is recommended that conference organisers try to put information about the peer-review status of claims into their promotional literature, and encourage presenters to communicate with them about this when (i) a talk is clearly likely to cause wider controversy; or (ii) new findings are being widely promoted to draw attention to a conference. (p.26) Scientists also need to be aware of the context of informal discussions about their work. Regulatory obligations require many commercially generated scientific findings to be announced immediately (for unavoidable reasons such as preventing insider trading in shares). Unlike peer-reviewed publications, Stock Exchange notifications (usually a press release) do not require sufficient information for other scientists to be able to evaluate the research, so at this point peer review of any sort can be impossible. A best practice guide should therefore be developed by companies that are obliged immediately to report R&D results to the financial markets and to product licensing authorities. It is also recommended that the use of an open access Web-based resource be explored, where organisations can provide supporting scientific data simultaneously with any press release. (p.31) Peer review has too low a profile in science education in the context of rapidly escalating sources of scientific information and the need to equip students and pupils with an understanding of how scientific material is generated. With the increasing use of the World Wide Web, students encounter material on scientific topics with great diversity in its status. It has become more difficult to assess the information sources used or to predict the material that student research will generate. It is recommended that bodies concerned with devising curricula, producing teaching materials and promoting science education, produce teaching resources on peer review for educators for all age groups. (p.33) In further and higher education, all courses covering risk assessment and the philosophy of science should include some education about the process of peer review.
V.
VI.
Many different groups of people comment on scientific issues and very few of them refer to whether work has been peer reviewed. There is little pressure for them to do so while scientists themselves rarely explain peer review to the public and sometimes fail to demonstrate regard for the distinction. If scientists regularly draw attention to whether work has been scrutinised by peers, and to whether results have been replicated, it will become easier for everyone to be more demanding about the quality of information that informs social discussions about science. The social uncertainty and scepticism of our times undoubtedly make the tasks of conveying scientific evidence and weighing scientific claims more challenging. In such circumstances, the fact that the development of science has at its centre a trust culture and deference to knowledge, codified in peer review, is potentially very significant. There is an opportunity to share its benefits with wider society within the debates about scientific evidence. This discussion paper encourages scientists, and others, to take that opportunity and to explore how an understanding of peer review can contribute to societys judgements about the results of scientific research.
xii
Section 1
A cultural challenge for peer review?
Debates about scientific research
1.1 The dissident, so-called whistleblower, however dodgy the research on which his or her evidence is based, is afforded massive attention; it is taken as axiomatic that the mainstream, evidence-based government-endorsed view will be self-serving and wrong. More than half of us believe the medical profession is divided over the health risks of MMR; in fact, it is more or less united that there is no risk.5 This was how the current popular treatment of scientific evidence was summed up by the commentator Will Hutton in a British newspaper in 2003. 1.2 In recent years, public discussion has frequently been dominated by debates about the findings of scientific research, particularly when research has a policy implication. Claims that the Measles, Mumps, Rubella (MMR) vaccine could cause autism have generated several years of debate, during which scientists, clinicians and health officials have presented many studies indicating that there is no discernible risk of autism from the MMR vaccine. Over the same period, other scientists have been caught up in contesting claims about research into alleged risks posed by genetically modified (GM) crops and food. Physicists have found themselves trying to respond to claims that nonionising radiation from mobile phones caused brain tumours and disturbed sleep. Endocrinologists and toxicologists have been engaged in responding to announcements that research has demonstrated a link between pesticides in food and damage to human health. These are just some of the more sustained debates that have circulated against a rapidly moving background of research claims about health and environmental risks, including plastic softeners, hormone replacement therapy, and silicone-gel-filled breast implants. Bad news assertions have not been the only source of publicly contested claims about research findings. Announcements about the imminent production of pig organs for human transplantation were seen by many scientists as premature and unsupported by research. Scientists were also almost unanimously sceptical about claims that human reproductive cloning was about to succeed, which provoked concerned reactions from politicians despite the lack of evidence to support the claims. As stories about environmental risks, health risks, and new research findings accumulate, scientists and medical practitioners have expressed frustration that unsubstantiated claims are treated with the same seriousness as more reliable studies, and that in some cases evidence is apparently ignored altogether. People who rely on the media and public bodies to interpret findings describe themselves as confused about what to believe. Some have become very sceptical of new claims, particularly those concerning health risks.6 Doctors experience the opposite phenomenon: they complain of surgeries full of the worried well7, resulting in part from scare stories and unfounded claims. Organisations charged with providing public health services and developing environmental and health policies are also frustrated about conflicting claims and the influence of unreliable research.
1.3
1.4
5 6
Hutton, Will (2003) Facts are free, opinion is sacred. The Observer, 17 August. See Philo, G. (ed.) (1999) Message received: Glasgow Media Group research 1993-1998. Cambridge: Longman. 7 Le Fanu, J. (1999) The rise and fall of modern medicine. London: Little Brown & Co. p.xix.
1.5
What is often missed in public discussions of these kinds is the fact that scientists subject their work to a system called peer review, to determine which research papers qualify for publication in scientific journals.8 Formalised peer review began with some journals in the 18th century and scientists have used it as a systematised method of quality control for the last 100 years. To succeed in getting a paper published, scientists must present their findings clearly for review by experts in their field, chosen by a knowledgeable, neutral journal editor. This process is the accepted route for making findings public: only once a paper has been reviewed, revised and published does the wider scientific community take it seriously, examine it and evaluate its contribution. For new work to be incorporated into the body of scientific knowledge, researchers must first convince those knowledgeable in the same field about the plausibility of their claims and the appropriateness of the research methods and evaluation techniques they use. The peer-review system means that statements made by scientists in published papers are unlike other kinds of statements or claims. For example, claims made by politicians, newspaper columnists, think tanks or campaign groups are not systematically subjected to independent quality review beforehand. Consequently, scientists usually make a distinction between claims that have been peer reviewed (and published) and those that have not. Peer review ensures that the research has been evaluated by other scientists with appropriate knowledge. There has been an opportunity to spot mistakes and omissions for example, as well as to clarify what the findings show. It also means that the results are then available to the wider scientific community, so that others in the field can consider the work and try to replicate the findings, or use them, in conjunction with other work or results, to reach further conclusions. Scientists never regard peer-reviewed research as beyond criticism. Over one million papers are published in peer-reviewed scientific journals worldwide each year. Peer review of a paper is just the first stage: a hypothesis that survives this first test must go on to be re-tested, and judged against other work in the same area, and for its coherence with work in related areas. Some of a papers conclusions will be hotly disputed or further research will show that they need to be revised as more data are acquired. The quality of peer review can also vary, so scientists treat work in some publications as more significant than others. Scientists rarely draw firm conclusions from just one paper or set of results, but consider the contribution it makes in the context of other work and their own experience. However, because peer-reviewed results can be treated as plausible and scientifically accountable, peer review is an essential dividing line for scientists to judge what is scientific fact and what is speculation.
1.6
1.7
1.8
1.9
Peer review is also used by many scientific funding bodies to assess proposals for future research. However, the focus of this report is solely on the use of peer review to determine which research papers should be published. See Reilly (1999) on public understanding of the BSE crisis. See also: MORI (2001) The role of scientists in public debate, research conducted by MORI for the Wellcome Trust.
often treated as more significant than whether their work has been peer reviewed and at what level. This is particularly the case when research is funded to any degree by corporations. Just under half of the UKs research and development work is funded by private enterprise.10 1.12 The confusion about what scientific evidence tells us and about what is credible in the light of available evidence may be exacerbated by non-reviewed claims that are presented as scientific, whether by scientists going beyond their reviewed findings, or by others who advance claims that appear to be scientific in the thought that these may have more public impact. 1.13 For example, one of the first claims that mobile phone emissions are unsafe was made in 1998 by Mr Roger Coghill, a self-employed researcher, who had previously argued that mobile phones cause headaches and memory loss. In 1998, he said that the waves that phones produce could damage the activity of lymphocytes in the body's immune system. Coghill published these claims himself and released them to the media, rather than submitting them first for peer review. Many other studies have failed to point to damage specifically of the body's immune system as a result of mobile phone usage, before or since. Despite the lack of corroboration, Coghills claims were widely reported, and fuelled discussion about mobile phone safety. Between 1998 and 2003, he was cited in 119 printed news publications in the UK, most of which made no reference to the lack of peer review of the research or to the fact that other, peer-reviewed research did not corroborate the hypothesis. 1.14 A similar attitude to scientific expertise was displayed in claims about the MMR vaccine and autism by Mr Paul Shattock, a pharmacist who set up the Autism Research Unit at the University of Sunderland, which advocates the view that autism is a metabolic disorder. In June 2002, he claimed to have identified a group of children whose autism resulted from the MMR vaccine. The research, based on the claim that children with bowel disease have abnormal levels of indolyl acryloyl glycine in their urine, was not published in a scientific journal but made headlines and Shattock was cited 41 times in 2002 in newspaper articles about the safety of the MMR vaccine. 1.15 In April 2002, the world media reported research results from Stockholm University and the Swedish National Food Administration that suggested that people were at risk of cancer from ingesting acrylamide from heated fatty foods. The reports provoked serious concern from the World Health Organisation, the UK Food Standards Agency and cancer charities, and reports of this concern in turn added to speculation about dangers, for example to children from eating crisps. According to a BBC report, the research was deemed so important that scientists took the unusual step of going public with their findings before the details had been officially published in an academic journal.11 Later studies found no relationship between acrylamide-rich food consumed and incidence of kidney or bladder cancer, and possible beneficial effects on bowel cancer rates.12 It is likely that there will be further scientific contributions to understanding of acrylamide. However, playing out such debates about early findings in the form of public announcements and health warnings has promoted confusion that reduces the effectiveness of public health information. 1.16 The demand for pre-scientific news seems to encourage a culture that erodes the distinction between expertise and subjective experience. As a public and a news readership, we seem to be attracted to what has not been endorsed by experts, perhaps wanting to believe that authentic information is that which does not conform to accepted ideas. At a popular level, this cultural move might be captured by the refrain of many campaigners and advocates of alternative therapies and theories: I dont care what the experts think, I know that my child has been affected. The publication of claims without reference to whether they have been reviewed suggests that there is little recognition of the significance of peer review, and that some believe that going straight to the public underlines the importance of claims.
Office for National Statistics (2003). Bread and crisps in cancer risk scare. BBC News Online, 25 April 2002. 12 Mucci et al.. (2003) Dietary acrylamide and cancer of the large bowel, kidney and bladder: absence of an association in a population-based study in Sweden. British Journal of Cancer, 88, 84-89.
11
10
Can peer review help society navigate the constant stream of science news?
1.17 Since many of the claims about the findings of scientific research have serious implications for matters ranging from global policy to family health, it is regrettable that scant attention is paid to the widespread use of peer review by scientists, with all the invested hours of expert scrutiny that it represents. That so little is known about it suggests that there has been a missed opportunity on the part of scientists, both to explain how scientific findings are shared and advanced, and to equip a wider public with a more reliable tool for assessing the claims put before them. 1.18 In November 2002, a Working Party on Peer Review of Scientific Papers was convened by Sense About Science13, a charitable trust promoting public use and awareness of scientific evidence. The Working Partys aim was to explore ways of equipping a wider public with an understanding of peer review and the relative merits of research claims in the public domain. 1.19 From the outset, the Working Party recognised that scientific peer review has, traditionally, not been a subject of general public interest. It has been seen as a technical topic or a set of procedural issues for organisations concerned with science publishing. 1.20 There are concerns within the scientific world about the challenges of managing peer review. However, peer review is not only a technical issue. It is central to establishing which scientific claims should be trusted. Some contributors to these debates are directly critical of the peer-review system in scientific publishing, arguing that it is used by the scientific establishment to screen out unorthodox work or troublesome findings. Some campaigners on issues as diverse as microwave emissions from mobile phones, genetically engineered crops, homeopathic treatments, chemical testing and the use of illicit narcotics see peer review as a cover for, rather than a corrective of, poor science. 1.21 More widely, the appeal of questioning conventional wisdom often encourages commentators to overlook the distinction between research that has been peer reviewed and that which has not. For example, amid controversy about the alleged links referred to previously between the MMR vaccine and autism and between mobile phone emissions and cancer, many contributors to public debate ignored the very different levels of scientific peer review and credibility of competing claims. Yet the significance of the issues clearly warranted a comment. Without it, the public at large was denied important means for evaluating what was being said. At least two social surveys have indicated that there is a public demand for greater discrimination in the quality of scientific information that is reported.14 1.22 However, the Working Party acknowledged that very little effort has been made to draw out the social implications of peer review. Specifically, these were identified as: how peer review influences the quality of scientific claims reaching the public domain; what the social effects are of publicising unscrutinised scientific claims, for example about risks to health or the environment; the insights that peer review provides into the way that scientific knowledge advances, a subject for which there is a growing audience given the popular interest in scientific controversies; peer reviews potential value for a wider public as an indicator of the plausibility of research claims; an explanation of how and why peer review is conducted.
13
14
1.27 Discussions about the need for scientists to recover public trust often emphasise more detailed regulation of scientific work. However, while it is true to say that science now develops in a climate of public scepticism about established sources of information, there is not a spontaneous and focused public demand for specific reforms to which scientists can respond. The Working Party judged that attempts to address suspicion by seeking changes in the practice of peer review might be misplaced or premature. Such action would neglect the opportunity to explain the process, and could overgeneralise accusations about peer review raised in the heat of particular controversies. 1.28 The challenge for the Working Party was to consider peer review from a social perspective and to engage the groups identified above in explanations of how peer review works for science and society. In this sense, it took as its starting point the idea that wider public interests might be served by the scientific interest in peer review, rather than the more prevalent assumption that they necessarily conflict. 1.29 Members agreed that the best aim for a discussion about peer review would be a cultural shift in the treatment of scientific claims. The Working Party has looked for ways to promote a culture where people bringing research claims to the public domain feel obliged to pay greater heed to expertise and evidence. In doing this they can encourage the public to ask more effective questions about the scientific information put before them.
Specifically, the Working Party has sought measures that: help to explain the practice of peer review to a wider public; explain how scientific findings of sufficient quality, validated by peer review, are essential for the policy and decision-making process; popularise the benefits of ensuring that work is judged by other scientists before it is made public; contribute to clearer understanding of the status of research claims made in the media, on the Internet and in other public domains; build awareness among scientists and commentators about the social consequences and costs of disregarding peer review.
1.30 Deliberations were therefore focused around these social aims, rather than around refining scientific or technical processes15. Themes discussed by the Working Party over the past 18 months are listed below. What critics of peer review have said and written. Transparency and democracy. Attitudes to expertise. Editorial pre-screening, or triage, at wide-spectrum journals. How scientists might communicate peer review via the media (from proceedings of a session held by the Science Media Centre as part of its How Science Works series). The requirements of the financial markets that publicly listed companies make some research results public immediately, and good practice in doing so. The social consequences and costs of publicising misleading scientific claims. Increased social and political interest and contestation about scientific claims. How to explain the science publishing scene to a non-scientific audience. Evidence of grantsmanship, institutional promotion and other motives in research reporting. Claims that peer review is inherently anti-orthodox or suppressive. Recognition of peer review in the formal education system. Interplay between peer review and editorial judgement.
1.31 Finally, while the experience and specialisation of the Working Party have been diverse, its members have shared a common view: peer review is a valuable thing and there is a considerable gap between the principles and energetic commitment it represents, and the way it is treated or understood in wider controversies about science. As explained in the following pages, peer review is not simply the best quality checking system. It is more accurately understood as the process of science itself and has by far the longest track record. As one useful account describes it, science is cumulative, often collective, and comprises a body of knowledge that is logically consistent, testable, and self-corrective. Peer review is fundamental to this as the process through which scientists test one anothers theories and evaluate and criticise one anothers research.16 The following report draws attention to the need to establish how an understanding of peer review can contribute to societys judgements on which scientific research to trust.
15
The Working Party confined itself to the issue of peer review of scientific papers in the public domain. This is distinct from peer review of grant applications, which, while the subject of similar discussions, is bound up with funding policy, so more usefully addressed elsewhere. It is also distinct from the scientific analysis of confidential material, such as commercial or military-related papers. Other projects addressing peer review are listed in Appendix 4. 16 Murray et al.. (2002) p.149.
Section 2
A guide to peer review and scientific publishing
What is scientific peer review?
2.1 Scientific peer review is the evaluation of scientific research findings or proposals for competence, significance and originality, by qualified experts who research and submit work for publication in the same field (peers). Most commonly, peer review is used by the editors of scientific17 journals, who ask well-qualified experts to provide written opinions about research papers that have been submitted for publication. On completing a project or stage of work, researchers write up their results into a paper presenting their experiments, findings and conclusions, and send the paper to a journal to be published. Scientific papers are sometimes written by individual scientists, but frequently the authors are groups of scientists who have worked as a team on the research. The journals editorial staff selects experts in the same field of work who are qualified to judge the scientific merits of the paper its competence, significance and originality and who are themselves involved in research and publication and subjected to the same discipline (peers). For some journals the editorial staff is employed by the journal and for others the work is done by professional scientists who act as editors as an additional activity. The selected experts, known as referees (or reviewers) review the paper and judge such things as whether the design and methodology of the research were appropriate, the data are plausible and the paper is written clearly. The referees are asked whether the paper acknowledges prior work, whether it is suitable for the journals scientific readership and whether it should be published in its current form or with revisions. (See Boxes 1 and 2.) Sometimes peer review is used to decide which papers should be delivered at scientific conferences. Many funding bodies ask scientific peers to assess whether proposed research is likely to contribute something new and significant, and whether it uses suitable expertise and methods. Peer review helps to keep funding decisions objective.18 However, our focus here is on the use of peer review to provide corrective feedback on papers describing research results submitted for publication. This is the process through which research findings become formally public.
2.2
2.3
2.4
2.6
17
Peer review is also used by many other academic journals, for example in the social sciences and humanities, to determine whether work is sufficiently competent, significant and original to merit publication. 18 See the peer review guidelines of the Association of Medical Research Charities (1993). 19 Science Media Centre (2003) Communicating peer review in a soundbite, p.1.
publishing requires very specific and substantive feedback about each paper, not just a yes or no decision. Referees might notice mistakes in calculations, or the absence of sufficient safeguards for checking results, for example, or inappropriate statistical tests. Whether research has been conducted by distinguished scientists in an eminent laboratory or by less established teams, it is subject to this scientific scrutiny. A useful summary of peer review has been provided by a group of social scientists: Researchers can make mistakes that render their conclusions worthless; and even when they conduct their research properly, they are also all too likely to exaggerate its importance. A review by scientists familiar with the subject matter is likely to detect mistakes and to qualify exaggerated claims. Thus peer review is important because it helps determine whether a studys substantive conclusion follows logically from the procedures used to arrive at it and whether the conclusion makes a significant contribution to our knowledge.20 2.7 Papers are sent to scientists to review because of their abilities to make a scientific assessment. Without the peer-review system, which research findings come to prominence would be arbitrary. Personal attributes and social influence or power would be more likely to play a role. Papers would be published regardless of whether experiments are poorly constructed, control groups inaccurately devised or the data insufficient. Every scientist would have to navigate so much unfiltered material that they would have time for little else. Scientists would have no choice but to resolve this arbitrariness because they depend on published results they can trust, in the same way that people on the sixtieth floor depend on the lift working scientists would reinvent peer review.
BOX 1 Referees usually comment and make recommendations on some of the following:
1. Significance: Are the findings original? Is the paper suitable for the subject focus of this journal? Is it sufficiently significant? (Is it a me too paper; is it salami slicing?21) Is the paper clear, logical and understandable? Does it take into account relevant current and past research on the topic? Are the methodology, data and analyses sound? Is the statistical design and analysis appropriate? Are there sufficient data to support the conclusions? Are the logic, arguments, inferences and interpretations sound? Are there counter-arguments or contrary evidence to be taken into account? Is the theory sufficiently sound, and supported by the evidence? Is it testable? Is it preferable to competing theories? Does the article justify its length? In papers describing work on animals or humans, is the work covered by appropriate licensing or ethical approval? (Many biological and medical journals have their own published guidelines for such research.)
5. Reasoning:
6. Theory:
7. Length: 8. Ethics:
20 21
Murray et al. (2002) pp.148-149. Me too papers are those that are predominantly repetitious of previous work, albeit reporting different experiments. Salami slicing refers to dividing a corpus of research work between several minimal papers at the threshold of acceptability, rather than presenting it in one.
2.8
In wider society, we also depend on peer-reviewed work but in a way that is not so obvious. If a close relative is seriously ill, we assume that they will be treated according to expert knowledge. We would consider it unacceptable if it transpired that they were treated on the basis of an arbitrary free-for-all, where the distinction between reliable expertise and ignorance, incompetence or charlatanism had been left to individual clinicians to assess, on a paper-by-paper basis.22
2.10 Scientific referees generally do not get paid. A few journals use small incentives to reward referees who return comments quickly. Reviewing can involve a lot of work, especially if scientists are referees for several journals, and this work is often done in their spare time. Scientists accept this as part of their scientific activities. They often have a sense of commitment to the journals output quality or to the learned society that publishes the journal. 2.11 Confidentiality is an essential part of the undertaking in reviewing scientific papers. Referees should not keep copies of the papers they are sent for review, nor reveal them to anyone, nor use any part of their content, without prior permission.
2.13 Some journals send referees each others comments together with details of their decision about publishing the paper. This is a relatively recent development, facilitated by email. It provides useful feedback and extends accountability among peers; all journals should adopt the practice.
22
Harnad (2000).
10
Rejection
2.14 Some journals reject a proportion of papers before sending them out to referees because they do not fit the emphasis of the journal. This is called editorial pre-screening. Journals that cover a range of subject areas receive many papers and use pre-screening more than others. Rejection at this stage does not imply that papers are of poor quality; many go on to be published in other, high-quality journals. 2.15 If a paper is rejected following peer review, it may be submitted to a different journal, perhaps one with a more specialised readership or a stronger interest in the papers topic. If the paper has been rejected because of the low importance or quality of the work, the author might publish it in a journal with lower quality standards, and hence also less influence in the field. 2.16 In the vast majority of cases, when a paper is rejected, authors do submit and publish the work in some form. Particularly with the rapid growth of online publishing, the range of journals in some fields of research is very wide and most papers find an outlet at some level. In order to understand this process, it helps to have a map of the scientific publishing scene. (See Box 3.)
For example, to charge authors a small submission fee, refundable if the paper is accepted, to discourage sloppy papers and multiple submissions. 24 See Appendix 4.
23
11
Staff
At high-circulation weekly journals, editors and editorial staff are usually paid, whereas many basic research journals that publish less frequently are run by academic scientists who receive little or no payment.
Figures compiled by Yvette Diven for Ulrich's Periodicals Directory 29 August 2003. Vanity-press refers to low-cost publishers with low standards of editorial control. They serve the needs of authors wanting to get their work into print rather than the needs of any readers. 27 Produced by the Institute of Scientific Information in the US and available on the Journal Citation Reports database. 28 Citation counts can reflect the size of a research field and bad papers may be cited while being criticised. For a range of views, see Seglen (1997); Lachmann and Rowlinson (1997); Lee et al. (2002); and Ray et al. (2000). 29 There is a concern among some scientists that publishing poor but controversial papers could be used by competitive wide-spectrum journals to increase their citations ratings. 30 A typical science and technology journals revenue breakdown is: 5% advertising, 85% subscriptions, 6% back copies, 1% offprints/reprints, 1% permissions and 2% from other sources. Pira International (2002) p.37. 31 Pira International (2002).
26
25
12
2.20 There is a different issue of time delay in relation to papers that have been reviewed and accepted for publication, but that have to be held back by the journal until there is a suitable space for them. Authors often discuss the findings in these papers publicly before the paper appears because the final version has been agreed at that point.
Referees motivation
2.21 To people who do not work in scientific research, it may seem strange that scientists are willing to spend hours reading through papers in great detail, without payment! But while there may not be a financial reward, referees feel strongly inclined to do their bit and this seems to be for a number of reasons. 2.22 It is a marker of a scientists own scientific contribution, as a researcher and author, to be considered suitably expert and the peer of others publishing research in the field. For the majority of people involved in research and discovery, sharing knowledge and expertise is important. Scientific journals are, by and large, a dialogue between scientists, to share ideas and debate new findings. Considering the findings written up in a paper, and pondering the questions it raises, can be as interesting and challenging as any other aspect of scientific endeavour. Reviews are one way that scientists contribute to this exchange and ensure that it is kept at a high enough quality to be genuinely useful. Journals would quickly become of little use to any scientist if the material in them was full of mistakes or hard to follow. More specifically, this is what is meant by scientific publication being public and self-correcting: if a journals standards drop, it loses both its authors and its readers to competing journals with higher standards. 2.23 Referees depend on others being willing to review their own papers attentively and objectively, and understand how valuable that critical scrutiny can be in making sense of their own results and improving their presentation for others.
13
use a referee and to weigh their comments accordingly or to seek another or an additional referee. Editors of specialised journals often know a lot about the referees that they use, having used them over time and formed opinions about their reliability and standards. 2.24.5 Authors are often given the opportunity to name anyone to whom they feel their papers should not be sent for review because of conflicts of various sorts. They are told what referees have said. They can challenge an editor if they feel a publication decision is biased. It is worth noting the comment of more than one experienced editor that, in such cases, authors often guess the identity of referees incorrectly! Abuse of privileged information 2.24.6 Papers are sent to referees in confidence. They may not use any part of the paper, retain copies or show it to anyone else, without permission. In some research areas, such as microbiology, the speed at which new insights can be adopted by other scientists is very fast and research papers may contain information that puts the reader immediately at a potential commercial or research advantage. 2.24.7 If a paper has commercial implications, these are for the author and associates to assess when deciding on a suitable time to publish in relation to protecting new discoveries by a patent or licence. Research publication is, after all, about publicly reporting findings for the use of fellow researchers. 2.24.8 Papers may provide a research advantage to the referee. (Referees have to seek permission for any use of the content, and failure to do so is considered serious misconduct by journals and by many employers.) In the main, authors publish because they want other experts to read and comment on their work, so long as they are acknowledged as the author of the paper. Personal prejudices 2.24.9 Referees might recommend rejection of a paper because they do not agree with it or because of personal prejudices, rather than because the work is weak or clearly flawed. Alternatively, they might recommend acceptance of a paper because it backs up their own approach, even if there are questions about its quality or originality. Competent editors minimise the chance of this affecting publication decisions by selecting referees carefully and using more than one. Editorial correspondence with referees is a matter of record and can be challenged. 2.25 It is worth remembering that the mere observation that scientists could misuse peer review does not establish the factual claims that scientists do misuse peer review. Authors are committed to publishing their work and are liable to complain if they feel that criticisms are unreasonable, or to go elsewhere if there are unjustified delays in reviewing their papers. These realities are a further discipline on the behaviour of referees. 2.26 In general, the system of peer review does not facilitate, but rather discourages or exposes, abuses of trust. The fact that referees are obliged to provide a review that will be scrutinised by an editor, usually shown to a second referee, and relayed to the author is a considerable discipline. Editors have an interest in ensuring that their referees use agreed criteria (see Box 1) for assessment, both to decide what is worthy of publication and to avoid future embarrassment.
Committee on Publication Ethics (1999) Guidelines on good publication practice, The COPE Report, Section 8.
14
Negative results 2.27.1 It is often said that journals turn down research if the results are negative. There is some confusion here about what makes a result negative in scientific terms. Critics have taken it to mean those results showing adverse effects of particular technologies, for example that a chemical is harmful to health. By contrast, when scientists say that negative results are rarely published, they mean that many studies that have no clear result, that fail to confirm previously demonstrated effects, or that find no effect, are less likely to be published (which does present some problems, for example for developing overviews of clinical trial data). They do not mean that papers are rejected because their findings have negative social or business implications, such as indicating that some process or product is harmful. In fact, these kinds of findings are regularly published and there are whole fields, like toxicology, that are devoted to reporting such effects. Scientific peer review helps to ensure that plausible results are published and that editorial decisions are not prejudiced by social or commercial aims. Seeking to make news 2.27.2 Some journals promote the studies that they publish to the general media. Science commentators have raised concerns that, in a competitive publishing world, news journalism considerations may trump the criteria competence, significance, originality on which peer review is based.34 It is to be expected that when journals publish articles about new discoveries these are reported more widely. For a handful of wide-spectrum journals, though, competition for news attention does appear to be influential. They sometimes tell authors not to promote their work in other ways before the paper is published and they use embargoes to encourage the timing of media discussions to suit the date of the journals publication. Some scientists suspect that papers are pre-screened for newsworthiness but this is more difficult to determine, because originality and newsworthiness may often coincide. 2.27.3 However, at the reviewing stage, newsworthiness considerations are marginalised. There is no evidence, or even anecdote, to suggest that anticipation that a paper might be of wider news interest influences referees judgements. Referees are usually anonymous: they have no public association with, or credit for, the work they review and they are not involved with journals promotional strategies. It is important to remember that these concerns are only directed at a small part of the work of a handful of journals. The editors of these journals insist that any desire they have to publish exciting new developments is kept in check by the recommendations of referees who are concerned only with the science.35 They cannot disregard those recommendations in any but a tiny minority of special cases without damaging the reputation of the journal with authors, referees and readers.
34 35
See, for example, Stewart (2003). Adam and Knight (2002) p.774.
15
2.30 Some scientists prefer to disclose their names when they referee papers. Editors usually give referees the option of waiving anonymity. Some journals also offer the option of waiving anonymity among referees. There does not appear to be any clear evidence available to show how anonymity, or waiving anonymity, affects the standard or range of scientific papers published. 2.31 Some commentators on scientific practices have suggested that the names of referees should always be published with the paper unless they provide a compelling reason not to, and that unsigned reviews should be given less influence.36 However, weighting reviews according to the referees preferences for anonymity would treat the identity of the referees as more significant than the content of the paper and could lead to very uneven criteria in editorial decisions. 2.32 The anonymity issue really comes down to a practical matter of establishing what works. What encourages scientists to give time to being referees? What encourages them to write frank, attentive and fair reviews that help editors to make accurate and objective assessments? This is the shared goal in scientific publishing. Although editors vary in their attitudes to the significance of anonymity, they generally think that it elicits candid reviews and it is common practice.
16
breaking scientific work is beyond conventional judgement? If so, peer review would be incapable of offering proper assessment of such work. Yet it is hard to imagine what other practices might be used in place of drawing on the expertise of others. 2.38 Sensitivity about misunderstood genius draws upon a very individualistic notion of genius. Moreover, since great leaps forward in science often arise from thorough and brilliant appropriation and transformation of existing knowledge, original science cannot dispense with the background of normal peer-reviewed work.37 Many developments in scientific understanding are the result of an interactive and cumulative process, as occurred in the series of results that led to the sequencing of the human genome.38 2.39 In some cases, referees or editors have been slow to spot the significance of a paper submitted, and a journal has missed the opportunity to publish ground-breaking work. This is arguably more damaging for individual journals than for the advance of science since the paper will almost certainly be published elsewhere. While the full implications of such work may be beyond the comprehension of referees and even beyond that of the discoverers themselves this does not suggest that peer review is dispensable or damaging. Subjecting work to the scrutiny of scientific peers can help to bring discoveries forward in a number of ways. The peer-review system imposes discipline on researchers to check results and to cross-reference their material with others. This itself can push scientists to think more about their findings and can be a source of discovery.
37 38
Kuhn (1970) p.23. "Did we realise the significance of our discovery? Yes we did ... . Did we foresee the sequencing of the human genome? No we didnt. We saw as far as the genetic code but we did not foresee either introns or RNA editing. We thought then that sequencing DNA would be very difficult and time consuming. Nor did we foresee recombinant DNA. But I think that this is a rather general rule, that one can seldom predict correctly more than about 10 or 15 years ahead. Unexpected discoveries can often change the picture completely." Francis Crick, The Biology of DNA, Cold Spring Harbor Laboratory, NY. 26 February 2003. http://meetings.cshl.org/2003helix_online.htm.
17
published in The Lancet.39 The hypothesis ran counter to established medical science, which maintained that high levels of hydrochloric acid in the stomach killed all bacteria. Within a year, Marshall had produced clear evidence and convinced his peers of the connection, leading to treatments of peptic ulcer and gastritis and lower rates of stomach cancer.40 2.43 Far from limiting recognition of new ideas, the peer-review system ensures that scientists, as referees, read many new papers in their field and evaluate them for competence, significance and originality.
Why cant there just be a technical checking system, rather than peer review, to make sure the researchers have abided by good practice?
2.44 Referees are sometimes given guidance about what to look for in a paper (see Box 1) but ultimately they are being asked to make an informed judgement. A technical checking system may work for marking arithmetic tests, but any original piece of research has its own unique features, requiring the application of expert judgement in ways that no one yet has been able to codify in a check-list. It would be no easier to devise a checklist to replace referees judgements than it would be to prepare one to replace medical diagnoses or juries judgements of guilt.
Marshall et al. (1984). Marshall and Warren (1985). 41 A US campaign, Center for Science in the Public Interest, refers to well known cases of industry seeking to prevent the publication of research results that are critical of its products. It aims to raise awareness about the role that corporate interests play in scientific research, oversight, and publication. www.cspinet.org/integrity/index.html. Media broadcasts have raised the same question: Some scientists question if commercial and political interests tied to biotechnology can tolerate scientific dissent. The Today programme, BBC Radio 4, 19 September 2003. Nicholas Regush, ABC news medical producer, has similarly complained that heavily funded scientific claims, such as the HIV-AIDS model, are rarely questioned. Regush (2000). 42 See, for example, Association of Medical Research Charities (1993).
40
39
18
2.49 Once papers are written and submitted, if they are published in the peer-reviewed scientific literature we can at least be sure that they have been judged independently to have scientific merit to be competent, significant and original no matter who funded the research.43 The findings are also then available to be replicated and debated. For society, scientific peer review is something of a safeguard against special interests, be they policy, commercial or campaign group led, that might distort the reporting of scientific findings. The problem with using funding as a guide to plausibility is that it generally relies on guesswork and rumours. The peer-reviewed literature offers a more accountable, reliable guide to what is scientifically plausible.
19
and further critical assessment. Other scientists try to repeat the experiments, checking the results and considering alternative explanations. 2.55 All of this careful deliberation, repetition and commentary are an essential part of scientific enquiry. Scientific research is written to be used in further research, so tends to be self-correcting. Errors are caught because it is not possible to build further research on a foundation that will not bear the weight. Error is a normal part of science, and uncovering flaws in scientific observations or reasoning is the everyday work of scientists.45 Peer review helps to avoid a lot of error and unnecessary wasting of time by asking authors to rectify defects in their papers before they are published, and by indicating to the wider scientific community which papers are most worthy of their attention. 2.56 Very occasionally, there are serious flaws in a paper that should have been apparent to the referees but which they missed in recommending the paper for publication. The cases where this has happened in recent years are quite varied and too few to suggest that there is a pattern to mistakes. It is a feature of any system of judgement, however expert, that mistakes are occasionally made, and scientists are aware of this possibility. If the findings are very significant, any flaws are likely to be discovered quite quickly because the paper will be widely read and discussed and other scientists will attempt to repeat the work. 2.57 Dealing with those major flaws that come to light when a paper is already published is a challenge for scientific publishing. Editors are expected to take responsibility for correcting the record prominently and promptly,46 but there is no single, accepted route for doing this. Authors often adjust their own findings, sometimes by writing to the journal that published their paper to retract some or all of their results, but often by submitting a further paper. Sometimes editors correct the record by printing a paper from another scientist. Editors rarely take the step of distancing themselves from the papers they publish, unless a mistake is very serious or there has been a breach of trust.47
45 46
Park (2000) p.9. COPE (1999). 47 In November 2001, Nature published a paper by David Quist and Ignacio Chapela, of the University of California, stating that genetically engineered DNA had found its way into wild Mexican corn. The paper drew complaints from scientists about apparent flaws in the methodology. In April 2002, the editor of Nature published a statement, saying, Nature has concluded that the evidence available is not sufficient to justify the publication of the original paper. Nature 416, p.601. 48 In October 2002, Jan Hendrik Schon, of Bell Laboratories in New Jersey, was exposed by other scientists as constructing fraudulent data to claim ground-breaking discoveries in nanotechnology. He had written (and had published) 25 papers by this time, most in the high-impact journals Nature and Science. Sixteen of these were deemed entirely false and the journals retracted many of the papers. With so few laboratories as well equipped as Bell Laboratories, other scientists had been unable to test the claims. The fraud was discovered as a result of other scientists comparing the published papers, which indicated a pattern between each set of data where there should have been random events.
20
2.59 Scientific bodies typically classify scientific misconduct under the following headings: Piracy deliberate exploitation of ideas from others without proper acknowledgement. Plagiarism copying ideas, text, or data without permission and due acknowledgement. Misrepresentation deliberate attempts to represent falsely the ideas or work of others. Fraud deliberate deception, which may include the fabrication of data.49 2.60 Serious scientific fraud appears to be very rare.50 Cases that come to light receive considerable attention and condemnation,51 as one would expect when trust and integrity are so fundamental to the scientific system of sharing ideas.
Edited text from Kings College Londons Guidelines on good practice in academic research www.kcl.ac.uk/depsta/kcle/research/resdocs/Ethics.pdf. 50 Goodstein (2002). 51 For example, the Lawrence Berkeley National Laboratory in California admitted in 2002 that its scientist, Victor Ninov, had fabricated the discovery of two new chemical elements. This admission was reported in many scientific journals and consequently in major newspapers across the world. 52 Royal Society response to the House of Commons Science and Technology Committee Inquiry into scientific publications, February 2004, policy document 04/04, p.1. http://www.royalsoc.ac.uk/files/statfiles/document252.pdf. 53 Some models of Open Access envisage charging authors a fee for processing the articles they submit, whether or not they are published.
49
21
their published, peer-reviewed journal articles on their own institutional websites to make them available to everyone.54 2.61.4 The scientists involved in all sides of the debate about the way forward are committed to good peer review. Open Access may even increase the extent to which science is self-corrective because all qualified experts will be able to access all published papers. Whatever model is eventually adopted, it will be organised around the needs of the scientific community and the publishing houses to maintain a peer-reviewed literature.
54 55
See http://www.eprints.org/self-faq/. The scientific establishments obsession with the peer review means important science that raises risks of GM technology is side-lined. Rowell, A. (2003) Safe science is not always good science. The Guardian 19 August.
22
Section 3
Raising the public profile of peer review
Introduction
3.1 The discussions of the Working Party brought forth a distinction between the practical problems that are faced by scientific publishing, which concern practitioners, and the task of explaining the peer-review process in response to the lack of wider awareness that scientific claims have been scrutinised in this way. Most criticisms of peer review are overstated. It is regrettable that they have not been diminished by a more accurate account of the peer-review process and what it offers both to science and to society. That this opportunity has not yet been seized perhaps reflects the general mood of defensiveness about scientific authority that has arisen in recent years, particularly in Britain. Peer review is very rarely contested as a matter of sustained principle. The promotion of alternative sources of authority and information to peer-reviewed research seems to come about because some journalists and opinion formers are drawn to stories that minister to a growing cultural ambivalence about established expertise. The specific attributes of shocking research claims about genetic modification, stem cell research, cloning, the MMR vaccine, new variant CJD and mobile phone radiation, for example, are presented to an increasingly common formula: when public stories begin to take shape, there is an almost instinctive search for a wronged scientist, suppressed research, apparently unco-ordinated official denials of risk, and so on. Under such conditions, peer review is treated as irrelevant and so the scientific merits of the claims become particularly hard to judge for everyone looking on. That scientists need to act with greater emphasis to challenge indifference to expert judgement is also underlined by the political reaction to some recent science scare stories. It is increasingly recognised that we now live in a climate where political actors and state bodies are very anxious and reactive on issues of science and risk. They seem less inclined to take responsibility for judgements based on scientific expertise. One consequence of this seems to be the potentially irresponsible practice by policy officials of putting scientific material into the public domain for conclusions to be drawn, without any indication as to its purpose or judgement on how it should be viewed.56 While the greater ambivalence about expert opinion means that less social emphasis is placed on expert review, it does not amount to a general critique. Likewise, people objecting to the conclusions of peer review complain predominantly about their own exclusion from the traditional vehicles of expert authority, scientific journals. This was apparent in the examples of dissenting scientists on subjects such as mobile phones and alternative health therapies that were considered by the Working Party. Some science commentators have tended to present complaints about peer review without first explaining why it is used. In 1999, Richard Horton, editor of The Lancet, described peer review as
3.2
3.3
3.4
3.5
3.6
56
The publication of specific energy absorption rates (SAR) for mobile phones, for example, quite predictably was met with indifference in the absence of any meaningful interpretation or judgement. Department of Health (2000) Mobile phones and health, leaflet, 8th December, London: HMSO.
23
usually ignorant and frequently wrong57 after the journal was criticised for publishing research on the effects of feeding GM potatoes to rats, which had been turned down by another leading journal. He did not, though, explain why The Lancet would continue to base its publication decisions on reviews that are usually ignorant. Horton also argued that BSE had made the public deeply sceptical about science and that The Lancet was encouraging a more open debate by publishing the GM paper. Being vague as to whether decisions about scientific publication should be based on peer review or on editorial ideas about the needs of public discussion is unlikely to diminish public scepticism, or to promote well-informed debate, and indeed it does not appear to have done either. 3.7 Even if some people are minded to criticise peer review more systematically, the limitations are immediately obvious. It is only possible to question the use of scientific expertise up to a certain point without eroding ones own grounds for reporting and commenting, or without sounding unconvincing to a society that, while more open to alternative views, does still expect healthcare rather than quackery, and science not witchcraft. The arguments and material reviewed and the discussions pursued by the Working Party have failed to indicate that there is anything systematically wrong with the conduct of peer review. What did become clear is the absence of wider recognition that scientific papers are peer-reviewed papers. There is a very confused picture of what peer review is and a lack of sensitivity on the part of scientists about explaining that. In short, the most basic problem with peer review is that so few citizens are made aware of it, at a time when people have become very concerned about how to weigh different claims meaningfully. In the context of the need for a more vigorous and clearer explanation of peer review, the Working Party considered four matters which were thought to be significant to promoting knowledge about peer review and eliminating causes for reservation among the scientific community. These were: The supply chain for promotion of research findings. Commercial publishing of research outside of the peer-reviewed literature. Peer review as part of science education. The need to find out more about the impact of different research claims on wider society. These are discussed in the following pages. Recommendations are highlighted in bold.
3.8
3.9
57
Horton (2000) Genetically modified food: consternation, confusion and crack-up. The Medical Journal of Australia, 172: 148-149.
24
25
that in some instances scientists may be overruled by an institution that wants publicity and in other instances the scientists publicity needs may not match the research data. It is recommended that scientists work with press officers to ensure that publicity is correct and sign it off. At the same time, press officers (including those working in other areas of science promotion such as conference organisers) should be aware of the potential impact of editing press releases and other communications about research and should seek to clarify what has been peer reviewed. It is recommended that universities and other scientific bodies endeavour to make press officers without a scientific background more aware of the peer-review process, for example as outlined in this report.
Talks at conferences
3.16 Many contributors to the Working Partys discussions have pointed out that confusion about research findings and their scientific status often arises from the reporting of conference proceedings. Conference organisers now regularly promote the talks to be given at their events to the media, and the greater demand for news stories about science makes these events attractive to journalists and commentators. There are different kinds of scientific conferences. Some are held to showcase areas of work to a wider audience, sometimes to non-scientists. These are usually concerned with published work. Others have traditionally been quite closed events where peers people working on similar problems at an advanced level can discuss and criticise one anothers work, usually at a more preliminary, pre-publication stage. 3.17 It is not feasible, nor desirable, to discourage interest in scientific conferences. After all, it is far more likely that there will be accurate, evidence-based discussions about subjects like the SARS virus if reporters and policy advisers attend a microbiology conference on the issue. 3.18 With talks covering a combination of new and old work, and information about what is peer reviewed changing in the run-up to an event, conference organisers would find it almost impossible to indicate systematically what has been peer reviewed. However, it is recommended that conference organisers try to put information about the peer-review status of claims into their promotional literature, and encourage presenters to communicate with them about this when (i) a talk is clearly likely to cause wider controversy; or (ii) new findings are being widely promoted to draw attention to a conference. 3.19 Reporters do not have time, nor the specialised knowledge, to read and interpret even a small selection of research findings, and so the contents of a press release or promotional material could well be taken to represent the findings. These materials are usually prepared by conference organising committees or by staff at a professional or learned society. While many of the people involved understand peer review, it is recommended that non-scientists employed in conference promotion are informed about the peer-review process. 3.20 It is also recommended that scientists pay greater attention to the context of informal discussions about their work, such as conferences that may now have a media orientation. This does not mean that scientists should retreat from discussions of their work, either with peers at conferences or with others attracted to those events. Rather, it is sometimes appropriate to provide more information, for example to state clearly which findings have been peer reviewed. One can assume that others in the field know the distinction from reading the same journals, but this is not so for a wider audience.
26
do not seem to be such a problem. With stories that are clearly personal accounts we are given enough information about the source to deploy our scepticism and see clearly that it is an individuals perception. It is when we are told wrongly that scientists are divided, or x study proves a link, that deception is really a problem. 3.22 When reporters choose wilfully to withhold information that would clearly influence the conclusions that readers or listeners will draw about the authority of a study, that is deceptive.60 This is not the same thing as journalists more subtle choices of presentation to make a subject more dramatic or interesting. That may be frustrating to scientists, but it seems a rather necessary spoonful of sugar for non-specialists reading such reports. 3.23 It is recommended that scientists follow the presentation of their work in wider media, and endeavour to correct unfounded claims that deviate substantially from their peer-reviewed work; but that they distinguish between this and matters of taste and style in how others choose to discuss their work. While it is reasonable to expect accuracy in reports, it is unfair to charge journalists with the same responsibilities as public health officials. News and public information are not the same thing and the latter is the responsibility of public bodies rather than the news and entertainment industry, for good reason. 3.24 Journals are the main source of science stories for the wider media, but journalists have sufficient contacts with scientists to know when new developments are expected, rather than simply relying on the journals. Increasingly, scientific journals make some papers available electronically to prevent delays and speculation.61 However, there continue to be tensions between journals (particularly the widely read ones), authors and journalists about the appropriate time for a paper to be discussed. Journals that attempt to hold back discussion of new work until the paper appears run the risk that it will happen anyway, without reference to the paper at all. On the other hand, they are free to try to maximise their own publicity in this way. There is no unifying process to deal with the problems that these tensions create for each party. From the perspective of clarifying the use of peer review, though, it makes little difference whether peer-reviewed findings are discussed before or after the publication date, so long as the fact that they have been peer reviewed and accepted is made clear. 3.25 The relatively few scientists who choose to publicise their findings before peer review, on the other hand, sometimes add to the confusion and potential for misleading claims that have been described in this report. Those whose expertise depends on promoting stories rather than convincing peers also put themselves at the mercy of other non-scientific priorities. This was clearly illustrated in February 2004, when many of the British news agencies that had courted Dr Andrew Wakefield for stories about the risks of the MMR vaccine beyond the findings reported in his papers, were able, very quickly, to destroy his position of expert comment. It is strongly recommended that scientists make it very clear whether their findings are peer reviewed and avoid speculation if it is liable to be treated more seriously than their actual findings.
60 61
ONeill (2002). Authors sometimes make pre-print copies available for other scientists.
27
BOX 4
PPL Therapeutics cloned pigs On 2 January 2002, PPL Therapeutics plc issued a statement to the London Stock Exchange that five cloned genetically modified pigs had been born in December 2001, with a single copy of a gene knockedout so that their organs and tissue would not trigger an immune reaction in humans and would therefore be suitable for transplants.63 No corroboration of the claims was available; the work was not peer reviewed at that time. The announcement naturally made news headlines. Two days later, the peer-reviewed results of work by Immerge BioTherapeutics in the United States, also resulting in knock-out pigs born three months previously, were published in Science, but this was barely reported. Reporters commented that PPLs announcement might have been intended to overshadow its competitors imminent publication, under pressure to maintain a competitive edge in the financial markets. The company argued that Stock Exchange Rules had obliged it to publish the information.64 British Biotech In 1997, British Biotechs former head of clinical research trials publicly accused the company of misleading investors about the effectiveness and prospects of its anti-pancreatitis drug Zacutex and exaggerating slender results from research into the anti-cancer drug Marimistat. The companys share price fell dramatically, and it was investigated by the Stock Exchange and the House of Commons.65
3.27 Press announcements about results put scientists in a difficult position because they are unable to comment on claims about breakthroughs and implications of the work due to an absence of accessible data. There is some concern among scientists outside of the commercial sector, and science commentators, about the practice of announcing commercial scientific results through press releases. 3.28 The Working Party looked at the prospects for encouraging greater use of peer review or peer scrutiny for commercial findings. However, it became clear that there is indeed a tension between the demands of financial reporting and the time involved in having results peer reviewed. For pharmaceutical companies, the situation is further complicated by regulations governing the supply of product information in different countries and the duty to inform licensing bodies.
Office for National Statistics (2003). N.B. Some of this work is conducted in public institutions. Worlds first announcement of cloned knock-out pigs: Christmas-born pigs are a major step towards successful production of animal organs and cells for human transplant use, PPL Therapeutics plc, Press Release, 2 January 2002. http://www.revivicor.com/KOrelease.htm. 64 Firn, David (2002) Cloned pigs raise hackles of scientists. Financial Times, 6 January. 65 BBC News 15 June 1998 and 17 August 1998. http://news.bbc.co.uk/1/hi/business/the_company_file/133740.stm.
63
62
28
29
Comment
3.42 It is in the interests of science and society, and of course companies themselves, if those involved in research in the science and technology sectors conduct themselves in a way that scientists find defensible. The range of approaches adopted by companies making announcements is quite broad. Some provide much greater supporting detail than others, particularly when announcing good news. Some companies make a simple announcement to the Stock Exchange, while others combine it with a press release to the general media, news agencies and medical and scientific press. Improvements should be focused on the information that is made available, and the potential for scientific scrutiny, rather than on trying to curtail the use of announcements as promotional opportunities. 3.43 Many larger companies already have policies or good practice guides in place for the publication of their research. These deal with peer review, the style of announcement and the stakeholder groups that should be informed, such as patient groups in human trials, as well as the financial and legal obligations. Some aspects of best practice might be applicable to smaller companies. 3.44 All companies are able to submit their work for peer review at the same time as an announcement; and any guideline, cultural pressure or good practice model that encourages this is to be welcomed. However, some improvement in the availability of data for expert comment and scrutiny at the point of announcement is desirable. This might overcome the fact that some small companies, due to their time constraints and short-term outlook, are unlikely to write up papers for peer review. 3.45 Scientific journals have become more flexible about authors making aspects of their findings public before a paper appears in the journal.66 However, it is still widely believed that any kind of publicity about results will discourage journals from considering a paper for publication. It urgently needs to
66
In the 1970s, because of a surplus of manuscripts, journals were able to be very selective. Dr Franz Ingelfinger, editor of the New England Journal of Medicine, demanded exclusivity and refused to publish any paper that had been reported in detail elsewhere. This became known as the Ingelfinger Rule and was imposed by many other journals.
30
be clarified with the editors of journals: (a) whether they would refuse to consider a paper if the research had been the subject of an announcement; and (b) if so, whether there are some kinds of announcement that would avoid this problem, for example whether a distinction would be drawn between an announcement made to the financial markets and a press release to the national news media. (The latter is still forbidden, in advance of a papers publication, by many journals). 3.46 In respect of guidance about the publication of scientific results, the Financial Services Authority has drawn attention to its current review of the listing regime. It is anticipated that new rules and guidance will be issued in 2004. The new rules will ask listed companies to review their own practices and this suggests a timely opportunity to launch guidance on the publication of scientific results. 3.47 It is recommended that guidance on good practice in announcing the results of scientific research by companies should be produced to coincide with the general guidance issued by the Financial Services Authority. This should include a template to accompany announcements against which companies can provide information about their data and research that is relevant for wider scientific expert scrutiny, for example details of trial sizes and stage of work. This should be drawn up in collaboration with scientific and learned bodies, large and small companies, scientific research publishers and the relevant financial, trade and corporate regulatory bodies. Organisations that should be involved in the development and adoption of guidance on the publication of scientific results to the Stock Exchange, in order to ensure that they are taken up, include: the 100 Group; the Investor Relations Society; the Association of the British Pharmaceutical Industry; the BioIndustry Association; and scientific institutes with corporate members. A range of companies need to be canvassed about their own in-house standards and practices concerning making announcements about scientific research. It is recommended that a collaborative network be established, with the potential to see through the proposed guidance. 3.48 It is recommended that the guidance should also provide information about the requirements of publishers concerning how announcements might affect the opportunity to submit research results for peer review. The Royal Society, as a learned society and publisher and in view of its recent consultation about peer review, the Academy of Medical Sciences, and the Royal Academy of Engineering, among others, are well placed to contribute to, or adopt, a project of this nature. In particular, to consult journal editors and to clarify how the type and scope of a recommended announcement protocol might be shaped to avoid disqualifying the research from publication. Also to assist in establishing good-practice guidance concerning commercial announcements of research results and clinical trials. 3.49 It is recommended that the viability of a Web-based resource for submitting announcements of corporate scientific research against a recommended template be investigated. This would need to be maintained by a suitable trade or regulatory body and would be accessible to scientists, journalists and other interested parties.
31
3.53 Currently, education about the process of peer review is not formally reflected in the curriculum. However, one initiative is developing packs for teachers, offering guidance and teaching materials, about the scientific process. This includes a new set of materials called Ideas, evidence and argument in science, which is aimed at putting sciences commitment to evidence to the fore of teaching: Essentially, we believe (and we have evidence to support this belief) that knowing why the wrong idea is wrong is as important to your learning of science as knowing why the scientific idea is correct [Teachers should] lead pupils through historic evidence, so they understand how each idea was rigorously tested. 70 3.54 The purpose of these materials is to help students understand that scientific knowledge that we regard as facts, such as the earth revolving around the sun, are actually the result of many years of academic argument and of evidence gathering. In this way, students are encouraged to consider new research critically and to consider its evidence base, not just to believe new theories because they appear to make sense. 3.55 A new science curriculum, 21st Century Science,71 is currently being piloted in schools in Britain and also aims to encourage students to understand the processes of science. The curriculums objective is to increase scientific literacy (as defined in Box 5) and to teach the kind of science that everyone needs to understand as citizens.72 3.56 The Environmental Inquiry, a website and curriculum series developed at Cornell University in the United States, has developed teaching resources that aim to help high-school students conduct science research and to participate as a community with other student scientists.73 One of the
67
Department for Education and Skills (2002) Secondary schools curriculum and staffing survey: provisional. November. http://www.dfes.gov.uk/. 68 Lessons taught by non-specialists. BBC News Online, 25 September 2003. 69 Similar bodies fulfil this function for Wales, Northern Ireland and Scotland. 70 Osborne (2004) Behind the Big Bang. Times Education Supplement, 2 January. 71 Co-ordinated by the Nuffield Foundation and the University of York: http://www.21stcenturyscience.org/home/ 72 http://www.21stcenturyscience.org/home/. 73 http://ei.cornell.edu/toxicology/peerreview/prtutorial/scientists.asp.
32
modules focuses on the peer-review process and suggests ideas and activities for school classes to engender an understanding of its importance.
appreciate and understand the impact of science and technology on everyday life; take informed personal decisions about things that involve science, such as health, diet, use of energy resources; read and understand the essential points of media reports about matters that involve science; reflect critically on the information included in, and (often more important) omitted from, such reports; and take part confidently in discussions with others about issues involving science.
Consumers not producers of science Most people are unlikely ever to be producers of new scientific knowledge. But all of us, as citizens, need to be informed users and consumers of scientific knowledge. For this, we need to have some understanding of two quite distinct kinds of thing:
3.57 Each of these educational resources aims to improve secondary school students understanding of the scientific process, and with it the peer-review process, but currently the use of these resources is not widespread and it will take time for these measures to be used in the classroom. In addition, these tools are aimed only at secondary school students: how to educate primary and degree-level students in these issues has not been addressed. 3.58 Tools and classroom materials for teachers are a valuable resource for building an understanding of peer review. Important steps have already been made toward providing this support in the UK by the Nuffield Foundation, the University of York and Kings College, London. It is recommended that the available resources are collated as a resource list, together with some discussion about the significance of educating students and pupils about peer review. This would need to be repeated for the teaching of different age groups. It might best be done by the Association for Science Education. It is also recommended that a similar summary is developed for students of other subjects that are concerned with evidence, risk assessment or the sociology and philosophy of science. 3.59 It is recommended that the following materials are produced: first-hand descriptions of roles played by scientists in peer review: author, referee and editor; explicit description of the way in which students and pupils reviews of one anothers experimental outcomes develops an understanding of the role of peer review;
74
http://www.21stcenturyscience.org/newmodel/literacy.asp.
33
a programme of familiarisation with scientific publishing, including typical visual aids (see Box 6).
Scientist/ author
Published?
Yes
No
34
In relation to the proportion who answered correctly, Michele Corrado, Head of Medicine and Science Research at MORI, noted, Because the question actually describes the process, some respondents may have demonstrated an understanding of the words, and/or logically deduced an answer, rather than having a prior familiarity or an understanding of what is involved in peer review in any detail. One could therefore conclude that the proportion to communicate the meaning of this phrase with is at least seven in ten British adults. MORI correspondence, March 2004.
35
preoccupations of particular scientific groups that their messages are not getting through. The aim is to strengthen and support the publics ability to assess warnings and research claims. While it has been possible to identify significant opportunities to extend knowledge of peer review, there is a need for social research into the way that perceptions about research results are developed. It is recommended that more extensive research is undertaken, collaboratively between scientists and social scientists, to develop accounts of how science stories are reported, the questions that are asked about research by different groups, and the resulting perception about the relative merits of different claims.
76
36
Concluding comments
3.69 This report argues that the current cultural challenges to science, and particularly the frustrations experienced by scientists and the public about how to weigh different claims, make it imperative to see past the narrow experiences of science publishing to the essence of peer review as the culture of science. It is recommended that systematic attempts are made to develop effective explanations of peer review and to communicate these to a wider public. 3.70 Peer review is systematised accountability to expert judgement. The issue for scientists has traditionally been how best to organise this and ensure a reasonable degree of fairness from the review process. It has rarely been noted that other organisations do not have this kind of quality control. Charities, for example, are accountable through whether people continue to donate money to their causes, journalists through whether people read their articles and opinion formers through popularity and cultural recognition. It seems ironic that in a culture that emphasises the need for continual audit and rules of governance, the arguably more engaged system of accountability that is central to the practice of science, namely peer review, receives little attention or celebration. 3.71 The social uncertainty and free-floating scepticism of our times undoubtedly make the tasks of conveying scientific evidence and weighing scientific claims more challenging. In such circumstances, the fact that the development of science has at its centre a trust culture and deference to knowledge, codified in peer review, is potentially very significant. There is an opportunity to share its benefits with wider society within the debates about scientific evidence. This report encourages scientists, and others, to take that opportunity.
37
38
Appendices
Appendix 1: References and sources
Adam, D., Knight, J. (2002) Publish, and be damned Nature, 419, pp.772-6. Agres, T. (2003) Cloning Crackdown. The Scientist, 3rd January. Angell, M. (1996) Science on trial: the clash of medical evidence and the law in the breast implant case. New York: W W Norton & Co. Association of Medical Research Charities (1993) Peer review guidelines. London: AMRC. Association of Learned and Professional Society Publishers (2002) Learned journal publishing in the UK. http://www.alpsp.org/jnlpubuk.pdf. Bauer, H. (2001) Fatal Attractions: The Troubles with Science. New York: Paraview Press. Bauer, H. (1994) Scientific Literacy and the Myth of the Scientific Method. Illinois: University of Illinois Press. Bazalgette, P. (2003) Why we must all mistrust the experts. Evening Standard, 13th August. Borger, J. (2002) Cult scientists claim first human cloning. The Guardian, 28th December. Boseley, S. (2003) Doctors turn on each other as MMR debate rages again. The Guardian, 1 November. Cintas, P. (2004) Confidential reports may improve peer review. Nature, 428, 255, correspondence. Cochrane Collaboration (2003) Editorial Peer Review for Improving the Quality of Reports Of Biomedical Studies. Oxford: Cochrane Collaboration. Collins, H. M. (1996) Some comments on peer review in the journals. PSA Political Science Specialist Group, Faculty Seminar, University of Southampton. Committee on Publication Ethics (1999) Guidelines on good publication practice. The COPE Report. Committee on Publication Ethics (2003) The COPE Report. December 2003. Dalrymple, T. (2001) An intelligent persons guide to medicine. London: Duckworth & Co. Diven, Y. (2003) Ulrichs periodicals directory. New Jersey: R R Bowker. Financial Services Authority (2003) Handbook of rules and guidance. London: FSA. Financial Services Authority (1996, July) Guidance on the dissemination of price sensitive information. London: FSA. Fraser, R. (2004) Open access publishing: is it the future for scientific journals? Microbiology Today, 31: 31. Friedlander, M. (1995) At the Fringes of Science. Boulder, CO: Westview Press. Godlee, F. (2002) Making reviewers visible: openness, accountability, and credit. JAMA, 287: 21 (reprinted), 27624. Goodstein, D. (2002) Scientific misconduct. Academe (publication of the American Association of University Professors), 88: 1. Gregory, J., Miller, S. (1998) Science in public: communication, culture and credibility. Cambridge: Perseus Publishing. Gura, T. (2002) Peer review, unmasked. Nature, 416: 258-60. Hargreaves, I., Lewis, J. and Speers T. (2003) Towards a better map: Science, the public and the media. Cardiff: ESRC. Hellman, H. (1999) Great Feuds in Science: Ten of the liveliest disputes ever. New York: John Wiley & Sons. Henneberg, M. (1997) Peer review: the Holy Office of modern science. Natural Science, 1. Highfield, R. (2000) Selling science to the public. Science, 289: 59. Horton, R. (2000) Genetically modified food: consternation, confusion and crack-up. The Medical Journal of Australia, 172: 148-149. Horton, R. (2003) Second opinion: doctors, diseases and decisions in modern medicine. London: Granta Books.
39
Hutton, W. (2003) Facts are free, opinion is sacred. The Observer, 18th August. Kassirer, J. P., Campion, E. W. (1994) Peer review: crude and understudied, but indispensable, JAMA, 272: 96-97. Kinsella, J. (1989) Covering the plague: AIDS and the American media. Rutgers University Press. Kirby, A. (2002) Doubts over Mexican GM maize. BBC Online, 14th April, news.bbc.co.uk/hi/english/sci/ tech/newsid_1911000/1911535.stm Kuhn, T.S. (1970) The structure of scientific revolutions, International Encyclopedia of Unified Science, 2, 2. Chicago: University of Chicago Press. Lachmann, P. (2002) The Research Integrity Initiative: Progress Report. The COPE Report, November. Lachmann, P., Rowlinson, J. (1997) Its not where you publish that matters. Science and Public Affairs, Winter Issue, 8. Lawrence, P. A. (2003) The politics of publication, Nature, 422: 259-261. Lee, K.P., Schotland, M., Bacchetti, P., Bero, L.A. (2002) Association of journal quality indicators with methodological quality of clinical research articles. JAMA, 287: 2805-2808. Le Fanu, J. (1999) The rise and fall of modern medicine. London: Little Brown & Co. Marshall, B.J., Armstrong, J.A., McGechie, D.B., Clancy, R.J. (1985) Attempt to fulfil Kochs postulates for pyloric campylobacter Medical Journal of Australia, 142: 436-39. Marshall, B.J., Warren, J.R. (1984) Unidentified curved bacilli in the stomach of patients with gastritis and peptic ulceration. The Lancet 1: 1311-14. Martin, B. (2002) Strategies for dissenting scientists. http:/www.scientificexploration.org. Media Medics (2001) Managing media hype! Guidelines for Primary Care Practice. London. MORI (2003) Scientific peer review (Research conducted by MORI for the Science Media Centre, funded by Nature). London. MORI (2001) The role of scientists in public debate. (Research conducted by MORI for the Wellcome Trust) London. Morris, S. (1998) Learned journals and the communication of research. Learned Publishing, 11, 4: 253-8. Murray, D., Schwartz, J., Lichter, S.R. (2002) It aint necessarily so: how the media remake our picture of reality. New York: Penguin Books. Nussbaum, B. (1990) Good intentions: how big business and the medical establishment are corrupting the fight against AIDS. Boston: Atlantic Monthly Press. Office for National Statistics (2003) Expenditure on research and development in the UK by sector of funding 19812002. London: HMSO. ONeill, O. (2002) A question of trust: the BBC Reith Lectures 2002. Cambridge University Press. Park, R. (2000) Voodoo science: the road from foolishness to fraud. New York: Oxford University Press Parliamentary Office of Science and Technology (2002) Peer review. Postnote, 182, September. Pira International (2002) Publishing in the knowledge economy: competitiveness analysis of the UK publishing media sector. London: DTI & UK Publishing Media. Popper, K. (1972) Objective knowledge: an evolutionary approach. Oxford: Clarendon Press. Quist, D. and Chapela, I.H. (2001) Transgenic DNA introgressed into traditional maize landraces in Oaxaca, Mexico. Nature 414: 541-543. Ray, J., Berkwits, M., Davidoff, F. (2000) The fate of manuscripts rejected by a general medical journal. American Journal of Medicine, 109, 2: 131-135. Reilly, J. (1999) Just another food scare? Public understanding and the BSE crisis. In Philo, G. (ed.) Message Received: Glasgow Media Group research 1993-1998. Essex: Longman. Rensberger, B. (2000) The nature of evidence. Science, 289: 5476, p61. Seglen, P. O. (1997) Why the impact factor of journals should not be used for evaluating research. BMJ 314, 497. Silvers, R. (ed) (2003) Hidden histories of science. New York: New York Review of Books. Stewart, C. N. Jr. (2003) Press before paper when media and science collide, Nature Biotechnology, 21, 353. The Herald (2003) More confusion as MMR doctors clash. 1st November.
40
Wanjek, C. (2003) Bad Medicine: misconceptions and misuses revealed, from distance healing to Vitamin O. New Jersey: John Wiley & Sons. Weinberg, S. (2001) Facing up: science and its cultural adversaries. Cambridge: Harvard University Press. Wildman, L. (2002) Health scares that worry you sick. Readers Digest, May, pp.88-93. Williamson, A. (2003) What will happen to peer review? Learned Publishing, 16: 1: 15-20. Wright, R. (1999) It took an ex-physicist and a former ornithology student along with some unwitting help from a competitor to crack the secret of life' Time Magazine, 29 March.
41
42
Paid Consulting: Within the last 3 years, receipt of consulting fees, honoraria, speaking fees, or expert testimony fees from entities that have a financial interest in the results and materials of this study. Please enumerate. None I have a paid consulting relationship, as described below Patents: A planned, pending, or awarded patent on this work by you or your institution. Please explain. None I or my institution has a patent related to this work, as described below We declare that we have read Sciences full Conflict of Interest Policy and have disclosed all declarable relationships as defined therein, if any. Manuscript Number: _____________________Title:_____________________________________________________ First Author: ______________________________________________ Signature: ________________________________________________ Date: _______________________
43
1 The regulatory framework seeks to secure as fair a distribution of information as is practicable. The Listing Rules place a general obligation on companies to disclose certain information which is not public knowledge and which may lead to a substantial movement in the price of its securities. Such information will include major new developments, changes in the companys financial condition or business performance or changes in the companys expectation of its performance. Information must always be given to the market as a whole, by an announcement to the Company Announcements Office. Companies are free to use additional media, but selective disclosure of price sensitive information, without an announcement, is never acceptable. 4 It is not feasible to define any theoretical percentage movement in a share price which will make a piece of information price sensitive. Attempts at a precise definition of price sensitive are not possible, since it is generally necessary to take into account a number of factors specific to the particular case, in addition to the information itself, which cannot be captured in a mechanistic formula. These include the price and volatility of the share and prevailing market conditions. No such definition is included in the relevant legislation. 5 However, price sensitive information will potentially have a significant effect on a companys share price. In particular, a company should be able to assess whether an event or information known to the company would have a significant effect on future reported earnings per share, pre-tax profits, borrowings or other potential determinants of the companys share price. The Listing Rules indicate many events which have to be announced to the market because they may be price sensitive. These include dividend announcements, board appointments or departures, profit warnings, share dealings by directors or substantial shareholders, acquisitions and disposals above a certain size, annual and interim results, preliminary results, rights issues and other offers of securities. In other areas judgement will necessarily be required. This guidance seeks to assist in these judgements by conveying the spirit within which investor communications are to be conducted. Handling of confidential information 16 Companies are sometimes confronted with the problem of how long to keep an issue confidential and what constitutes the proper time for its release. There are many processes which are inherently price sensitive where it is essential to maintain confidentiality until the major elements have been finalised and where premature release of information would be more misleading than informative. Such processes include, for example, the development of a new product, the planning of a major redundancy programme, the negotiation of significant financing arrangements, or the preparation of a take-over or partial disposal. Once these issues have been finalised an announcement should be made, unless a dispensation has been granted by the Exchange to avoid prejudicing a companys legitimate interests. However, if, during negotiations, the circle of parties involved becomes too large to ensure the confidentiality of the information, or there is a danger that information has leaked to parties not directly involved, an announcement should be made. Extracts from chapter 9 Continuing Obligations of The Listing Rules 4 A company need not notify to the Company Announcements Office information about impending developments or matters in the course of negotiation, and may give such information in confidence to recipients within the categories described in paragraph five. If the company has reason to believe that a breach of such confidence has occurred or is likely to occur, and, in either case, the development or matter in question is such that knowledge of it would be likely to lead to substantial movement in the price of its listed securities, the company must without delay notify to the Company Announcements Office at least a warning announcement to the effect that the company expects shortly to release information which may lead to such a movement.
44
45
Dr Derek Bell
Dr Derek Bell is currently Chief Executive of The Association for Science Education, having been involved in science education as teacher, lecturer and researcher for over 25 years. Throughout his career, Derek has maintained a strong and active interest in the enhancement of teaching and learning, and approaches to helping children develop their understanding of the world around them. He was a member of the SPACE (Science Processes and Concept Exploration) Project team and went on to co-ordinate the Nuffield Primary Science Project. Derek has published widely and is currently Chair of the Wellcome Trust's Society Awards Panel and its advisory group for public engagement in science.
Ms Tracey Brown
Tracey Brown has been the Director of Sense About Science, a charitable trust for the advancement of public knowledge about scientific evidence, since it was established in April 2002. She has a background in social research, particularly on the social dynamics of risk, together with experience of organising research projects in the social sciences. She previously worked on the establishment of an EC-funded regional research and academic teaching centre in Kazan, Russia, while based at the University of Kent; she spent a year in a more commercial research environment as an analyst, setting up a research unit for crisis management specialists. She is a regular contributor to public and media debates about science and progress.
Dr Peter Cotgreave
Peter Cotgreave is the Director of Save British Science, an independent campaign for effective science policies. He previously worked as a research ecologist at the Universit Claude Bernard in France, as Lecturer in Ornithology and Human Sciences at the University of Oxford, and as a conservation biologist at the Zoological Society of London. He is the author of many scientific papers and popular science articles, and of two books, including Science for Survival, an exploration of the links between science and society. He has peer reviewed many papers for international journals.
46
Lord Drayson
As co-founder and Chief Executive of PowderJect Pharmaceuticals plc, Lord Drayson built the company from a technology start-up into the worlds leading independent vaccines company. Following PowderJects acquisition by Chiron, he has explored new projects in biotechnology and pharmaceuticals. His additional roles are as Science Entrepreneur in Residence at the Said Business School and as Chairman of the fund-raising campaign for Oxford's Childrens Hospital. He took a seat in the House of Lords in June 2004.
Ms Fiona Fox
Fiona Fox is the Head of the Science Media Centre, an independent venture working to promote the voices, stories and views of the scientific community to the news media when science is in the headlines. She has a degree in journalism and 20 years experience in media relations. She was previously Head of Media at CAFOD, one of the UKs leading aid agencies. The Science Media Centre has produced a media training guide for Communicating Peer Review in a Soundbite, which was launched with a MORI poll on public attitudes to peer review.
Mr Tony Gilland
Tony Gilland is the Science and Society Director at the Institute of Ideas. He has organised and directed several major public engagement activities related to the controversies surrounding science today. These have included the Institute's "Genes and Society Festival" in London (2003), the Institute of Ideas' and New School University's "Science, Knowledge and Humanity" conference in New York (2001) and the Institute of Ideas' and Royal Institution's "Interrogating the Precautionary Principle" conference in London (2000). Tony is the editor of a number of publications, including Science: can we trust the experts? and a frequent contributor to journals, radio programmes, and public events on issues related to risk, the environment and scientific expertise. He holds a degree in Philosophy, Politics and Economics from the University of Oxford.
47
Professor Peter Main (Deputised by Dr Philip Diamond, Manager Higher Education and Research, Institute of Physics.)
Peter Main is the Director of Education and Science at the Institute of Physics, a learned society which supports physics at all levels. Previously, he was Head of Department and Professor of Physics at Nottingham University. He has been instrumental in developing an ethical code of conduct at the Institute of Physics.
48
Sense About Science 60 Cambridge Street London SW1V 4QQ Tel: +44 (0)1795 591975 Email: [email protected] Web: www.senseaboutscience.org Registered Charity No. 1101114
49