METHODOLOGY

Assessing the risk of bias of quantitative analytical studies:

introducing the vision for critical appraisal within JBI
Downloaded from http://journals.lww.com/jbisrir by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4XMi0hCywCX

systematic reviews
1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC1y0abggQZXdgGj2MwlZLeI= on 06/03/2024

Zachary Munn1  Jennifer C. Stone1  Edoardo Aromataris1  Miloslav Klugar2  Kim Sears3  Jo Leonardi-Bee4 
Timothy Hugh Barker1
1
JBI, Faculty of Health and Medical Sciences, The University of Adelaide, Adelaide, SA, Australia, 2Czech National Centre for Evidence-Based
Healthcare and Knowledge Translation (Cochrane Czech Republic, Czech Republic [Middle European] Centre for Evidence-Based Healthcare: A JBI
Centre of Excellence, Masaryk University GRADE Centre), Faculty of Medicine, Institute of Biostatistics and Analyses, Masaryk University, Brno,
Czech Republic, 3Queen’s Collaboration for Health Care Quality: A JBI Centre of Excellence, Queen’s University, Kingston, ON, Canada, and 4The
Nottingham Centre for Evidence Based Healthcare: A JBI Centre of Excellence, School of Medicine, University of Nottingham, Nottingham, UK

ABSTRACT

A key step in the systematic review process is the assessment of the methodological quality (or risk of bias) of the
included studies. At JBI, we have developed several tools to assist with this evaluation. As evidence synthesis
methods continue to evolve, it has been necessary to revise and reflect on JBI’s current approach to critical
appraisal and to plan a strategy for the future. In this first paper of a series focusing on risk of bias assessment, we
introduce our vision for risk of bias assessment for JBI. In future papers in this series, the methodological approach
taken for this revision process will be discussed, along with the revised tools and guidance for using these tools.
Keywords: critical appraisal; methodological quality; risk of bias; systematic reviews
JBI Evid Synth 2023; 21(3):467–471.

Introduction protocol) need to be subjected to rigorous appraisal

by 2 independent reviewers (in duplicate) using
A defining feature of systematic reviews is the
process of critique or appraisal of the included
research evidence.1–10 This fundamental review
an appropriate critical appraisal tool. The results of
this appraisal can then be taken into consideration
process is called by different names in the literature, in the analysis, synthesis, and interpretation of the
and includes terms such as risk of bias assessment, results within the systematic review. In most cases,
critical appraisal, assessment of study validity, as- the primary purpose of this assessment is to allow
sessment of methodological quality, or assessment of reviewers to answer the overarching question of
methodological limitations.11 The purpose of this how well a study was designed and performed with
appraisal is to assess the methodological conduct of regard to avoiding systematic error (bias).
a study and to determine the extent to which a study Over the nearly 3 decades of JBI’s ongoing invest-
has addressed the possibility (or risk) of bias in its ment in evidence synthesis,12–15 there have been
design, conduct, or analysis. All papers selected for many different iterations of JBI critical appraisal
inclusion in a systematic review (ie, those that meet tools. These tools have been developed by JBI and
the inclusion eligibility criteria described in the collaborators, and approved by the JBI Scientific
Committee following extensive consultation.16,17
Correspondence: Zachary Munn, [email protected] Although these tools have been specifically designed
ZM, JCS, EA, and THB are paid employees of JBI, The University of for application within a systematic review process,
Adelaide. ZM, JS, EA, and THB are members of the JBI Scientific Com- JBI critical appraisal tools can also be used for var-
mittee. All authors are members of the JBI Effectiveness Methodology
Group. EA is editor in chief, MK is an associate editor, and JLB is a senior ious educational and clinical purposes, such as creat-
associate editor of JBI Evidence Synthesis. No authors were involved in ing critically appraised topics, reading the literature,
the editorial processing of this manuscript. during the peer review process, and in journal clubs.
DOI: 10.11124/JBIES-22-00224 The suite of critical appraisal tools is largely based

JBI Evidence Synthesis ß 2022 JBI 467

 METHODOLOGY Z. Munn et al.

on study design and exists as a checklist, with tar- for our current tools to bring them into alignment
Downloaded from http://journals.lww.com/jbisrir by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4XMi0hCywCX

geted questions that address key methodological with current methodological developments in this
limitations of the study design and the safeguards field. The proposal also outlined the ideal character-
that authors may have implemented to minimize the istics of a future JBI tool to assess risk of bias in
1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC1y0abggQZXdgGj2MwlZLeI= on 06/03/2024

impact of bias in the results of their study. analytical studies, and how we might move towards
There are many critical appraisal and risk of bias our ideal vision in a phased approach. This new
tools available for use in systematic reviews.18,19 At approach will then be included in our guidance,27
JBI, we have had our own suite of critical appraisal education programs, and software.20
tools (the first proposed in 2002) available for use by Following the approval of this proposal, a signif-
our Collaboration and other review authors, and icant amount of work has been conducted both in
included in our JBI Manual for Evidence Synthesis revising our current tools and investigating key prin-
for many years. These tools have been endorsed and ciples and concepts related to risk of bias assessment,
ratified by the JBI Scientific Committee as the ideal including the ideal features of risk of bias tools from
tools to use across JBI’s toolkit for evidence synthe- the JBI perspective. To better communicate these
sis.16 JBI has developed its own set of tools and developments, a new series in JBI Evidence Synthesis
does not simply recommend use of those developed has been launched to discuss in detail the advance-
by other groups or used in publication by other ments in this field. This paper aims to introduce JBI’s
authors for a number of reasons. Firstly, no other short-, medium-, and long-term objectives regarding
set of tools is broad enough to encompass all of the the future of risk of bias assessment. The ideal prin-
JBI-endorsed approaches to evidence synthesis. If JBI ciples of risk of bias are also established to set the
were to take an endorsement approach, it would foundation for 2 additional series of articles to
lead to a multitude of different tools (all of which be published in this journal: the first being a series
may differ in design, structure, and application) of revised tools for the assessment of risk of bias; the
in use across JBI reviews. This would inevitably second, a series of companion papers to introduce,
lead to issues regarding consistency, publication for- discuss, and propose concepts, principles, and
mats, and steeper learning curves, among others. In advancements in the field of risk of bias assessment
addition, across the suite of methodologies and to direct future tool development at JBI.
methods,16 there are still gaps where perhaps no
alternative tool exists. Another benefit is that by Principles for an ideal risk of bias approach
developing these tools ourselves, we are able to read- for analytical studies for JBI
ily embed them in our systematic review software20 After discussion among the JBI working group and
and our educational training programs.21 further input and ratification by the JBI Scientific
As the field of evidence synthesis continues to Committee, the following have been identified as the
evolve,22,23 there have been ongoing discussions ideal characteristics of a tool to assess method-
within JBI regarding our approach to critical apprai- ological limitations for JBI systematic reviews. In
sal, particularly given the advances in the method- this paper, we provide the initial list of these princi-
ological literature (regarding the concept of risk of ples and concepts, which will be elaborated on in
bias as opposed to critical appraisal)11,24 and the further papers in this series. The ideal JBI risk of bias
development of new tools to appraise studies25 (espe- approach will:
cially non-randomized studies).26 As such, a working i) Focus only on issues related to risk of bias (ie,
party consisting of members of the JBI Effectiveness systematic error or internal validity) and use
Methodology Group has been considering the ideal consistent terminology for risk of bias.
way forward for critical appraisal and risk of bias ii) Be sophisticated enough to consider not only
assessment within JBI quantitative systematic re- the presence or absence of methodological
views. In 2020, a proposal was put forward to the safeguards but also the feasibility of these
JBI Scientific Committee to embark on a process to safeguards in research and whether they are
evaluate our current tools and produce recommen- likely to increase the risk of bias in a study
dations for assessing the risk of bias of quantitative (eg, lack of blinding of outcome assessors for
analytical studies within the context of JBI reviews objective outcomes such as mortality), and will
moving forward. This proposal outlined a strategy require clear alignment between safeguards/

JBI Evidence Synthesis ß 2022 JBI 468

 METHODOLOGY Z. Munn et al.

conduct/signaling questions and their relevant iii) JBI will review all current tools and categorize
Downloaded from http://journals.lww.com/jbisrir by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4XMi0hCywCX

bias domain. Ideally, the tool would enable current checklist questions into risk of bias cri-
any approach to risk of bias assessment, in- teria domains (eg, a “selection bias” domain or
cluding checklist approaches, judgments within “attrition bias” domain) so that assessment can
1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC1y0abggQZXdgGj2MwlZLeI= on 06/03/2024

domains/methodological standards or consid- occur at the domain level, if desired, including

eration of safeguards independently, quality nuanced guidance regarding whether safeguard
counts, relative ranks, or other schemes. implementation was feasible (eg, blinding feasi-
iii) Map clearly to a comprehensive framework/ bility for hard outcomes). This will allow the
hierarchy/taxonomy of bias structured into dif- tools to be flexible in how they can be applied,
ferent levels, which can be used as a support either as checklists, scales, or domain-based as-
resource or for educational purposes (ie, a sessments. JBI will also continue to accept the
framework of safeguards under methodological use of the Cochrane RoB 2.0 tool and ROBINS-I
standards or domains) and facilitate compari- (and other tools) for JBI reviews, with justi-
sons across different analytic study designs fication.
using a common scale. iv) JBI should strongly endorse risk of bias assess-
iv) Be user-friendly (to an extent), timely, widely ments to be carried out at the result or outcome
applicable, and compatible with the Grading of level, and disallow study-level judgments. For
Recommendations, Assessment, Development context, risk of bias may change depending on
and Evaluations (GRADE). the outcome or result within a single study – for
v) Be valid, evidence-based (ideally on meta- example, although issues related to selection
epidemiological studies), and, where evidence bias (randomization and allocation conceal-
is lacking, theoretically sound. ment approaches) should apply to all out-
comes/results in a study, for other types of bias
The proposal for the future of JBI risk of bias (such as attrition, detection, or measurement
assessment bias), the risk of bias may vary depending on
the individual outcome and/or result.
The proposal for the future of JBI risk of bias assess-
ment contained a set of key recommendations,
which were discussed and approved by the JBI Scien- Medium-term recommendation
tific Committee.16,17 The justification for these rec- A working group will create an “overview of bias”
ommendations will also be expanded on in future framework, including a map, with a clear and
papers within this series. Transitioning to a new risk comprehensive hierarchy of bias structured by dif-
of bias framework for JBI reviews will likely take a ferent levels, which can be used as a support re-
significant amount of energy and resources, and will source and for educational purposes; for example,
be a multiyear project. As such, short-, medium-, a framework of safeguards and their multi-
and long-term recommendations were submitted level categorization (ie, domains or methodologi-
and approved by the JBI Scientific Committee. cal standards - subdomains - safeguards) across
common quantitative study designs seen in health
care. For context, this will ideally be useful for tool
Short-term recommendations
developers and students to clearly see how items
i) JBI should move away from using the term
“assessment of methodological quality” or “cri- relate to domains of bias and may be common
tical appraisal” and use the term “risk of bias” across different study designs, and it will provide
a clear framework for how study design elements
assessment (for all tools for quantitative designs).
map to different types of bias.
ii) JBI will review all current tools for quantitative
designs and move to focus only on internal
validity, or “risk of bias” assessment, rather Long-term recommendation
than other issues related to reporting, external JBI will adopt, adapt, or create a new tool that meets
validity, imprecision, etc. These items can be all the characteristics we consider appropriate for an
removed from tools or at least clearly separated “ideal” tool, informed by a comprehensive frame-
from internal validity questions. work of bias.

JBI Evidence Synthesis ß 2022 JBI 469

 METHODOLOGY Z. Munn et al.

Conclusion cumulative incidence data. Int J Evid Based Healthc 2015;

Downloaded from http://journals.lww.com/jbisrir by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4XMi0hCywCX

13(3):147–53.
JBI has a pragmatic vision to identify the best avail-
7. Stone JC, Doi SAR. Moving towards a standards-based
able evidence to inform decision-making. As such,
methodological quality assessment scheme for clinical re-
we provide methodologies for several diverse evi- search. Int J Evid Based Healthc 2019;17(2):72–3.
1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC1y0abggQZXdgGj2MwlZLeI= on 06/03/2024

dence synthesis approaches and promote the use of 8. Stone JC, Glass K, Clark J, Munn Z, Tugwell P, Doi SAR. A
the best available evidence to answer systematic unified framework for bias assessment in clinical research.
review questions. For this reason, we consider many Int J Evid Based Healthc 2019;17(2):106–20.
different types of analytical study designs across and 9. Stone JC, Glass K, Clark J, Ritskes-Hoitinga M, Munn Z,
within review types, and it is not uncommon for JBI Tugwell P, et al. The MethodologicAl STandards for Epide-
reviews to include experimental, quasi-experimental, miological Research (MASTER) scale demonstrated a uni-
and observational studies within the one review. fied framework for bias assessment. J Clin Epidemiol 2021;
Currently, there are some limitations within the JBI 134:52–64.
10. Stone JC, Gurunathan U, Aromataris E, Glass K, Tugwell P,
critical appraisal toolkit. To address these shortfalls,
Munn Z, et al. Bias assessment in outcomes research: the
the current toolkit will be revised, and planning is
role of relative versus absolute approaches. Value Health
underway for a future approach that will enable JBI 2021;24(8):1145–9.
to achieve an approach to risk of bias that will align 11. Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Krebs
with our proposed key principles. As this approach Seida J, et al. Risk of bias versus quality assessment of
evolves, our revised tools, principles, guidance, and randomised controlled trials: cross sectional study. BMJ
concepts will be further discussed within this new 2009;339:b4012.
series of papers for JBI Evidence Synthesis. 12. Jordan Z, Lockwood C, Aromataris E, Pilla B, Porritt K, Klugar
M, et al. JBI series paper 1: introducing JBI and the JBI
model of EHBC. J Clin Epidemiol 2022;150:191–5.
Funding 13. Jordan Z, Lockwood C, Munn Z, Aromataris E. The
ZM is supported by an NHMRC Investigator Grant, updated Joanna Briggs Institute model of evidence-
APP1195676. MK is supported by the INTER-EX- based healthcare. Int J Evid Based Healthc 2019;17(1):
CELLENCE grant number LTC20031—Towards an 58–71.
International Network for Evidence-based Research 14. Pearson A, Wiechula R, Court A, Lockwood C. The JBI model
in Clinical Health Research in the Czech Republic. of evidence-based healthcare. Int J Evid Based Healthc
2005;3(8):207–15.
15. Jordan Z, Munn Z, Aromataris E, Lockwood C. Now that
References we’re here, where are we? The JBI approach to evidence-
1. Straus SE, Glasziou P, Richardson WS, Haynes RV. Evidence- based healthcare 20 years on. Int J Evid Based Healthc
based medicine: how to practice and teach EBM, 5th ed. 2015;13(3):117–20.
Churchill Livingstone: Elsevier; 2018. 16. Aromataris E, Stern C, Lockwood C, Barker TH, Klugar M,
2. Porritt K, Gomersall J, Lockwood C. JBI’s systematic reviews: Jadotte Y, et al. JBI series paper 2: tailored evidence syn-
study selection and critical appraisal. Am J Nurs 2014;114 thesis approaches are required to answer diverse ques-
(6):47–52. tions: a pragmatic evidence synthesis toolkit from JBI.
3. Tufanaru C, Munn Z, Stephenson M, Aromataris E. Fixed or J Clin Epidemiol 2022;150:196–202.
random effects meta-analysis? Common methodological 17. Pilla B, Jordan Z, Christian R, Kynoch K, McInerney P, Cooper
issues in systematic reviews of effectiveness. Int J Evid K, et al. JBI series paper 4: the role of collaborative evidence
Based Healthc 2015;13(3):196–207. networks in promoting and supporting evidence-based
4. Moola S, Munn Z, Sears K, Sfetcu R, Currie M, Lisy K, et al. healthcare globally: reflections from 25 years across 38
Conducting systematic reviews of association (etiology): countries. J Clin Epidemiol 2022;150:210–15.
The Joanna Briggs Institute’s approach. Int J Evid Based 18. Quigley JM, Thompson JC, Halfpenny NJ, Scott DA. Critical
Healthc 2015;13(3):163–9. appraisal of nonrandomized studies—a review of recom-
5. Munn Z, Barker TH, Moola S, Tufanaru C, Stern C, McArthur mended and commonly used tools. J Eval Clin Pract 2019;
A, et al. Methodological quality of case series studies: an 25(1):44–52.
introduction to the JBI critical appraisal tool. JBI Evid Synth 19. Zeng X, Zhang Y, Kwong JS, Zhang C, Li S, Sun F, et al. The
2020;18(10):2127–33. methodological quality assessment tools for preclinical and
6. Munn Z, Moola S, Lisy K, Riitano D, Tufanaru C. Method- clinical studies, systematic review and meta-analysis, and
ological guidance for systematic reviews of observa- clinical practice guideline: a systematic review. J Evid Based
tional epidemiological studies reporting prevalence and Med 2015;8(1):2–10.

JBI Evidence Synthesis ß 2022 JBI 470

 METHODOLOGY Z. Munn et al.

20. Munn Z, Aromataris E, Tufanaru C, Stern C, Porritt K, Farrow medical and health sciences. BMC Med Res Methodol
Downloaded from http://journals.lww.com/jbisrir by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4XMi0hCywCX

J, et al. The development of software to support multiple 2018;18(1):5.

systematic review types: the Joanna Briggs Institute System 24. Furuya-Kanamori L, Xu C, Hasan SS, Doi SA. Quality versus
for the Unified Management, Assessment and Review of risk-of-bias assessment in clinical research. J Clin Epidemiol
Information (JBI SUMARI). Int J Evid Based Healthc 2019;17 2021;129:172–5.
1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC1y0abggQZXdgGj2MwlZLeI= on 06/03/2024

(1):36–43. 25. Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS,
21. Stern C, Munn Z, Porritt K, Lockwood C, Peters MD, Bellman Boutron I, et al. RoB 2: a revised tool for assessing risk of
S, et al. An international educational training course for bias in randomised trials. BMJ 2019;366:l4898.
conducting systematic reviews in health care: the Joanna 26. Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND,
Briggs Institute’s comprehensive systematic review training Viswanathan M, et al. ROBINS-I: a tool for assessing risk of
program. Worldviews Evid Based Nurs 2018;15(5):401–8. bias in non-randomised studies of interventions. BMJ 2016;
22. Tricco AC, Tetzlaff J, Moher D. The art and science of 355:i4919.
knowledge synthesis. J Clin Epidemiol 2011;64(1):11–20. 27. Aromataris E, Munn Z. Chapter 1: JBI systematic reviews. In:
23. Munn Z, Stern C, Aromataris E, Lockwood C, Jordan Z. What Aromataris E, Munn Z, editors. JBI Manual for Evidence
kind of systematic review should I conduct? A proposed Synthesis [internet]. Adelaide, JBI; 2020 [cited 2022
typology and guidance for systematic reviewers in the Aug 1]. Available from: https://synthesismanual.jbi.global.

JBI Evidence Synthesis ß 2022 JBI 471