Reviewer's quick guide to common statistical errors in scientific papers

Design errors hospital number, or simply Obscure statistical tests should be

alternating between treatments justified and referenced.
Sample size for human subjects
is therefore inappropriate.
Many studies are too small to detect
Central randomisation is ideal. Comparing P-values between subgroups
even large effects (Table 1).
instead of carrying out tests of interaction
Unblinded assessment of is incorrect. Some may wrongly conclude
Table 1: Guide to sample size
outcomes may be influenced by from these results that:
Expected Total sample
knowledge of the treatment
difference size required*
group.
(p1-p2) Subgroup B P>0.05

5% 1450-3200 Look for:

10% 440-820 • Appreciation and Subgroup A P<0.05
20% 140-210 measures taken to reduce
30% 80-100 bias through study design
1
40% 50-60 • Selection of patients,
* 5% significance level, 80% power. Smaller numbers
may be justified for rare outcomes (p1 <.1)
collection of data, definition Treatment effect with 95% CI

and assessment of the subgroup affects response to

Look for: outcome and, for clinical treatment, based on comparing P-values.
• Clinical trials should always report trials, method of A test of interaction would show no
sample size calculations randomisation should be evidence of any effect of the grouping on
• Authors with 'negative' results (i.e. clearly described response.
found no difference) should not • Number and reasons for
report equivalence unless withdrawal should be Correlating time series: any two variables
sufficiently powered -"absence of reported by treatment that consistently rise, fall or remain
evidence is not evidence of group constant over time will be correlated.
absence" • Appropriate analytic 'Detrended' series should be compared
methods such as multiple instead.
Bias regression should be used
Randomisation is the best way of to adjust for differences Method comparison studies
avoiding bias but it is not always possible between groups in Correlation ≠ agreement
or appropriate. observational studies
• Authors should discuss
Perfect agreement
Some biases affecting observational likely biases and potential

Method B
studies: impact on their results
Treatment-by-indication bias: different
Method comparison studies
treatments are given to different groups Perfect correlation
If different methods are
of patients because of differences in their evaluated by different observers
clinical condition. Method A
then the method differences are
confounded with observer Higher correlation can be induced by
Historical controls: will tend to differences. The study must be including patients with extreme
exaggerate treatment effect as recent
repeated with each observer measurements. Limits of agreement
patients benefit from improvements in
using all methods. should be calculated according to
health care over time and special
method of Bland and Altman. Adequate
attention as a study participant. Recent
patients are also likely to be more Analysis errors agreement between methods is a clinical
not a statistical judgement.
restrictively selected. Failure to use a test for trend on
ordered categories (e.g. age-
Multiple testing
Retrospective data collection: availability group).
Conclusions should only be drawn from
and recording of events and patient
Dichotomizing continuous appropriate analyses of a small number
characteristics may be related to the
of clear, pre-defined hypotheses. Results
groups being compared. variables in the analysis
from post-hoc subgroup or risk-factor
(acceptable for descriptive
purposes). analyses should be treated as
Ecological fallacy: an association
speculative. If many such tests have
observed between variables on an
been carried out adjustment for multiple
aggregate level does not necessarily Using methods for independent
samples on paired or repeated testing should be considered.
represent the association that exists at
the individual level. measures data. An example is
Comparing groups at multiple time points
using both arms or legs of the
same patient as if they were two should be avoided – a summary statistics
Some biases affecting observational
approach or more complex statistical
studies and clinical trials: independent observations.
methods should be used instead.
Selection bias: low response rate or high
refusal rate – were patients that Using parametric methods (e.g.
Further reading:
participated different to those that did t-test, ANOVA or linear CONSORT: http://www.consort-statement.org
not? regression) when the outcome Greenhalgh T. How to read a paper: Statistics for the
or residuals have not been non-statistician. I: Different types of data need different
Informative dropout – was follow-up verified as normally distributed. statistical tests. BMJ 1997;315:364-366
Bland JM, Altman DG. Statistical methods for
curtailed for reasons connected to the assessing agreement between two methods of clinical
primary outcome? If so, imbalance in Over using hypothesis tests (P- measurement. Lancet 1986;1:307-310. Available online
dropout rates between the groups being values) in preference to at http://www-users.york.ac.uk/~mb55/meas/ba.htm
compared will introduce bias. confidence intervals. BMJ Statistics Notes: http://www-
users.york.ac.uk/~mb55/pubs/pbstnote.htm

Bias in clinical trials: One-tailed tests are very rarely

appropriate.
No-one should know what the next Produced by Tony Brady
random allocation is going to be as this Sealed Envelope Ltd
Failing to analyse clinical trials
may affect whether or when the patient is http://www.sealedenvelope.com
by intention-to-treat.
entered into the trial. Using date of birth,