Peer Review – How to…

Robert Platt
My Background
• Associate editor since 2009 (JECH, Stat Med, Int J Biostat)
• Editor, Am J Epid (one of ~20) since 2018
• Editor in Chief, Statistics in Medicine (one of 4) since 2019

2
 INTRODUCTION

This worksheet has been compiled from the advice of a number of journals
and publications. The aim of the worksheet is to give less-experienced

Stiller-Reeve – what should peer review do?

peer reviewers a concrete workflow of questions and tasks to follow when
they first peer-review. Please note that this is a suggested framework
for reviewers to follow. Depending on the journal or subject-field, some
elements may be lacking. Users should adapt the worksheet to suit their
needs, their personal review style, and the journal’s guidelines.

EVERY PEER-REVIEW PROCESS SHOULD AIM TO (HAMES, 2008):

• Prevent the publication of bad work – filter out studies that have been poorly
conceived, designed or executed

• Check that the research reported has been carried out well and there are no
flaws in the design or methodology

• Ensure that the work is reported correctly and unambiguously, with

acknowledgement to the existing body of work

• Ensure that the results presented have been interpreted correctly and all
possible interpretations considered

• Ensure that the results are not too preliminary or too speculative, but at the
same time do not block innovative new research and theories

• Select work that will be of the greatest interest to the readership

• Provide editors with evidence to make judgments as to whether articles meet

the selection criteria for their particular publications

• Generally improve the quality and readability of a publication (although this is

more a by-product of peer review)

Mathew Stiller-Reeve PhD, NORCE Klima, Bergen, Norway In collaboration with Prof. Geraint Vaughan, University of Manchester
1
EGU2018, 8-13 April 2018 and Bronwyn Wake PhD., Nature Climate Change
Typical workflow
1. Editor in Chief receives paper
• “desk-reject” – 20-50% of papers are rejected at this stage
2. Papers are sent to Editor/Associate Editor
• Another ~20% of papers desk-rejected at this stage
3. Send to reviewers.
• At most journals (excluding NEJM, JAMA, etc) this is a good sign that the
editors want the paper published. % success now much higher
• Revise/resubmit/etc.
Should you review?
• Topic – CAN you review?
• Need to be comfortable/interested (would you read the paper if it comes out)
• DON’T need to be an expert in the area.
• Time – should you review?
• Review at least as much as you submit, ideally more
• Every paper that goes to 2 journals requires 4-6 reviewers plus editor time
• Some recommend reviewing 3 papers for every paper you submit
• Bear in mind concurrent reviews
• Don’t overdo it, though – it’s OK to say no!
So you’ve agreed to review
• COMPLETE THE REVIEW ON TIME!
• Better to say “no I can’t do it” than to commit and turn in a review a month
late.
• If you were the author, how do you feel about overdue reviews?
• Check the journal review formats
• Some have specific requirements for formatting etc.
What next?
• Read the paper!
• Start with one relaxed easy reading.
• Initial thoughts:
• Is the question clear?
• Is the question important?
• Does the paper answer the question?
• If the answer is NO to any of these, the paper is probably a “Reject”
• A brief review, highlighting (gently) the major deficiencies should usually suffice.
• If the answer is YES – the paper is publishable in principle in the journal
• Move on to a more complete review
• Think about your final recommendation – major/minor revision?
Completing the review
• Read section by section looking for
• Major issues – things that really need fixing or are unclear.
• Be clear – necessary fixes or not?
• Minor issues – small things that the authors could do
• Helpful but not necessary
• Presentation issues –
• Are the sections well organized?
• Are tables/figures clear?
• Is the right stuff in the paper vs. the appendix? The main paper should stand alone.
Warnings
• Don’t do a detailed review of the language/grammar
• Many journals copy-edit
• OK to state “the text is unclear/may benefit from proofreading”
• Don’t be rude
• Avoid personal attacks
• Be constructive, even if the paper is terrible
Write your review
• Very brief summary of paper and of your thoughts (NOT your
recommendation)
• Major points
• Number them!
• Minor points
• Same
Comments to the Editor (private)
• Is the paper scientifically sound?
• Is it publishable?
• In this journal? OK to say it’s good but maybe doesn’t fit.
• Be direct
• If it is fatally flawed, say so.
• If it’s good science but a minor advance, say so.
Revisions
• You may get to review a revision
• OK to say “my questions were all answered well, nice job”!
• Point out where the answers are unsatisfactory
• Watch out for settings where your review and the other reviewer(s) were
discordant. How did the authors manage?
• Try not to find new problems with the paper
Concluding thoughts
• Peer review is a service. There’s lots to complain about (ahem,
Elsevier), but it is what makes our world move.
• Academics need to contribute
• It’s an opportunity to learn about others’ research
• Don’t overburden yourself with reviews, but do the ones that you’re
good at
• Be confident in your recommendations