RAPID REVIEWS

TO STRENGTHEN
HEALTH POLICY
AND SYSTEMS:
A PRACTICAL
GUIDE
EDITED BY:
ANDREA C. TRICCO
ETIENNE V. LANGLOIS
SHARON E. STRAUS
RAPID REVIEWS
TO STRENGTHEN
HEALTH POLICY
AND SYSTEMS:
A PRACTICAL
GUIDE
EDITED BY:
ANDREA C. TRICCO
ETIENNE V. LANGLOIS
SHARON E. STRAUS
Rapid reviews to strengthen health policy and systems: a practical guide / Andrea C. Tricco, Etienne V. Langlois
and Sharon E. Straus, editors

ISBN 978-92-4-151276-3

© World Health Organization 2017

Some rights reserved. This work is available under the Creative Commons Attribution-NonCommercial-ShareAlike
3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo).

Under the terms of this licence, you may copy, redistribute and adapt the work for non-commercial purposes,
provided the work is appropriately cited, as indicated below. In any use of this work, there should be no
suggestion that WHO endorses any specific organization, products or services. The use of the WHO logo is
not permitted. If you adapt the work, then you must license your work under the same or equivalent Creative
Commons licence. If you create a translation of this work, you should add the following disclaimer along with
the suggested citation: “This translation was not created by the World Health Organization (WHO). WHO is
not responsible for the content or accuracy of this translation. The original English edition shall be the binding
and authentic edition”.

Any mediation relating to disputes arising under the licence shall be conducted in accordance with the mediation
rules of the World Intellectual Property Organization.

Suggested citation. Tricco AC, Langlois EV, Straus SE, editors. Rapid reviews to strengthen health policy and
systems: a practical guide. Geneva: World Health Organization; 2017. Licence: CC BY-NC-SA 3.0 IGO.

Cataloguing-in-Publication (CIP) data. CIP data are available at http://apps.who.int/iris.

Sales, rights and licensing. To purchase WHO publications, see http://apps.who.int/bookorders. To submit
requests for commercial use and queries on rights and licensing, see http://www.who.int/about/licensing.

Third-party materials. If you wish to reuse material from this work that is attributed to a third party, such as
tables, figures or images, it is your responsibility to determine whether permission is needed for that reuse
and to obtain permission from the copyright holder. The risk of claims resulting from infringement of any third-
party-owned component in the work rests solely with the user.

General disclaimers. The designations employed and the presentation of the material in this publication do not
imply the expression of any opinion whatsoever on the part of WHO concerning the legal status of any country,
territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted
and dashed lines on maps represent approximate border lines for which there may not yet be full agreement.

The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed
or recommended by WHO in preference to others of a similar nature that are not mentioned. Errors and
omissions excepted, the names of proprietary products are distinguished by initial capital letters.

All reasonable precautions have been taken by WHO to verify the information contained in this publication.
However, the published material is being distributed without warranty of any kind, either expressed or implied.
The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO be
liable for damages arising from its use.

The named editors alone are responsible for the views expressed in this publication.

Design and layout by Paprika, France.

Printed in Switzerland.
 CONTENTS

Chapter 1: The need for rapid reviews to inform health policy and systems 1

1.1 Introduction������������������������������������������������������������������������������������������������������������������������������������������3

1.2 The rapid review approach��������������������������������������������������������������������������������������������������������5

1.3 The need to swiftly inform health policy and systems decisions������������������� 6

1.4 Health policy and systems research�����������������������������������������������������������������������������������9

1.5 Ways to expedite reviews on health policy and systems research������������� 10

1.6 Challenges in rapid reviews for health policy and systems������������������������������ 11

1.7 Practical considerations to expedite reviews �������������������������������������������������������������12

1.8 Conclusion������������������������������������������������������������������������������������������������������������������������������������������15

Chapter 2: Performing rapid reviews 21

2.1 Introduction��������������������������������������������������������������������������������������������������������������������������������������� 23

2.2 Needs assessment, topic selection, and topic refinement�������������������������������26

2.3 Protocol development���������������������������������������������������������������������������������������������������������������26

2.4 Literature search���������������������������������������������������������������������������������������������������������������������������27

2.5 Screening and study selection��������������������������������������������������������������������������������������������27

2.6 Data extraction�������������������������������������������������������������������������������������������������������������������������������27

2.7 Risk-of-bias assessment����������������������������������������������������������������������������������������������������������27

2.8 Knowledge synthesis�����������������������������������������������������������������������������������������������������������������28

2.9 Report production and dissemination����������������������������������������������������������������������������28

2.10 Operational considerations�������������������������������������������������������������������������������������������������28

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide III
 2.11 Information technology for rapid reviews������������������������������������������������������������������29

2.12 Suggested approaches to rapid reviews���������������������������������������������������������������������31

2.13 Conclusion�������������������������������������������������������������������������������������������������������������������������������������� 34

Chapter 3: Improving quality and efficiency in selecting, abstracting, and

appraising studies for rapid reviews 39

3.1 Introduction����������������������������������������������������������������������������������������������������������������������������������������41

3.2 Study selection�������������������������������������������������������������������������������������������������������������������������������45

3.3 Data abstraction�����������������������������������������������������������������������������������������������������������������������������46

3.4 Assessment of the methodological quality of included studies�������������������48

3.5 Allocating resources for selecting, abstracting, and assessing studies��49

3.6 Other considerations������������������������������������������������������������������������������������������������������������������49

3.7 Conclusion�����������������������������������������������������������������������������������������������������������������������������������������50

Chapter 4: Selecting rapid review methods for complex questions related to

health policy and system improvements 55

4.1 Introduction���������������������������������������������������������������������������������������������������������������������������������������57

4.2 Strategic decisions for rapid review���������������������������������������������������������������������������������57

4.2.1 Project management choices�������������������������������������������������������������������������������57

4.2.2 Choosing the scale and focus of a rapid review: the two-stage
process��������������������������������������������������������������������������������������������������������������������������������������������59

4.2.3 Transdisciplinary working��������������������������������������������������������������������������������������� 60

4.2.4 Methodological choices�������������������������������������������������������������������������������������������62

4.2.5 Rapid reviews as a social and methodological enterprise����������������65

4.3 Conclusion�����������������������������������������������������������������������������������������������������������������������������������������67

IV Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

Chapter 5: Engaging policy-makers and health systems managers in the
conduct of rapid reviews 71

5.1 Introduction���������������������������������������������������������������������������������������������������������������������������������������73

5.2 Objectives of engagement�����������������������������������������������������������������������������������������������������73

5.3 Points of engagement �������������������������������������������������������������������������������������������������������������� 74

5.4 Processes of engagement������������������������������������������������������������������������������������������������������ 75

5.5 Ongoing engagement of decision-makers������������������������������������������������������������������77

5.6 Supportive structures and mechanisms for engagement in rapid reviews����78

5.7 Benefits and challenges of engagement����������������������������������������������������������������������78

5.8 Conclusion�����������������������������������������������������������������������������������������������������������������������������������������79

Chapter 6: Fostering the conduct and use of rapid reviews of health policy and
systems research in low- and middle-income countries 83

6.1 Introduction �������������������������������������������������������������������������������������������������������������������������������������85

6.2 The potential for rapid reviews in LMICs ���������������������������������������������������������������������85

6.3 Challenges of conducting and using rapid reviews in LMICs������������������������ 86

6.4 Strategies to improve the conduct and use of rapid reviews in LMICs����87

6.5 Conclusion�����������������������������������������������������������������������������������������������������������������������������������������92

Chapter 7: Reporting and disseminating rapid review findings 95

7.1 Introduction���������������������������������������������������������������������������������������������������������������������������������������97

7.1.1 Goals of research reporting and dissemination ���������������������������������������97

7.2 Guidance and methods for reporting rapid reviews����������������������������������������������97

7.2.1 Core principles of reporting knowledge syntheses�������������������������������97

7.2.2 Reporting guidelines and checklists����������������������������������������������������������������98

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide V

 7.3 Dissemination of review findings������������������������������������������������������������������������������������� 100

7.3.1 Overview of available research dissemination frameworks�����������101

7.3.2 Engagement meetings and dialogue������������������������������������������������������������101

7.3.3 Dissemination activities and tools ������������������������������������������������������������������ 102

7.3.4 Special considerations for rapid reviews of health policy and

systems research ������������������������������������������������������������������������������������������������������������������ 105

7.4 Conclusion�������������������������������������������������������������������������������������������������������������������������������������� 105

Chapter 8: Improving the uptake of rapid reviews 109

8.1 Introduction��������������������������������������������������������������������������������������������������������������������������������������111

8.2 Barriers and facilitators to the uptake of rapid reviews�������������������������������������111

8.2.1 Attitudes ��������������������������������������������������������������������������������������������������������������������������� 112

8.2.2 Knowledge���������������������������������������������������������������������������������������������������������������������� 112

8.2.3 Skills������������������������������������������������������������������������������������������������������������������������������������� 113

8.2.4 Behaviours����������������������������������������������������������������������������������������������������������������������� 113

8.3 Considerations in writing the rapid review report������������������������������������������������� 113

8.4 Conclusion���������������������������������������������������������������������������������������������������������������������������������������116

VI Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

List of Figures

Figure 1.1. Interface of health policy and systems research ��������������������������������������9

Figure 1.2. Intensifying the scoping review process ������������������������������������������������������13

Figure 4.1. Two-stage review conducted in discussion with stakeholders�����59

Figure 4.2. Diagram showing how stakeholder input influences

methodological choices������������������������������������������������������������������������������������������������������������������� 61

Figure 5.1. A practical example of an integrated approach to engage policy-

makers and health systems managers throughout the review process �������� 74

Figure 5.2. Frequency and Intensity of Engagement here���������������������������������������� 75

Figure 7.1 Core principles of rapid reviews��������������������������������������������������������������������������97

Figure 8.1. Research brief: Rapid scoping review of medical malpractice

policies/models/frameworks��������������������������������������������������������������������������������������������������� 115

List of Tables

Table 1.1. Categorization of rapid evidence products, according to extent of

knowledge synthesis���������������������������������������������������������������������������������������������������������������������������� 6

Table 1.2. Rapid reviews to support health policy-making ������������������������������������������7

Table 1.3. Examples of rapid reviews for health policy and systemsa������������������8

Table 2.1. Common methods, approaches, and key considerations for the
steps in a rapid review����������������������������������������������������������������������������������������������������������������������� 24

Table 2.2. Information technology for rapid reviews�����������������������������������������������������29

Table 3.1. Consensus-rankinga of rapid review approaches relative to

systematic review approach��������������������������������������������������������������������������������������������������������� 42

Table 4.1. Scoping of medical malpractice policies in obstetrics��������������������������58

Table 4.2 Theory generation for no-fault compensation schemes����������������������58

Table 4.3. Accelerating the early work in a two-stage reviewa����������������������������� 60

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide VII
 Table 4.4. Outline of what a rapid review can achieve, according to three
different time frames (days, weeks, months)�������������������������������������������������������������������� 66

Table 5.1. Case examples of decision-maker engagement in rapid review

programmes���������������������������������������������������������������������������������������������������������������������������������������������76

Table 5.2. Case examples of various levels of engagement for systematic

reviews of health policy and systems research ��������������������������������������������������������������77

Table 6.1. Institutions in low- and middle-income countries that are involved
in preparation of rapid reviews����������������������������������������������������������������������������������������������������87

Table 7.1. Suggested minimum reporting items for rapid reviews of health
policy and systems research����������������������������������������������������������������������������������������������������� 100

List of Boxes

Box 1.1. Case example: Rapid review to understand communicable disease

surveillance and control in conflict‑affected Syria������������������������������������������������������������4

Box 1.2. Scope of health policy and systems research������������������������������������������������ 10

Box 1.3. Case example: Use of a rapid realist review to assess integration of
mental health care into primary care��������������������������������������������������������������������������������������� 11

Box 1.4. Rapid scoping review of medical malpractice policies������������������������������13

Box 1.5. Health policy and systems decision-makers����������������������������������������������������14

Box 2.1. The Center for Systematic Reviews on Health Policy and Systems
Research (SPARK) rapid review programme to support health policy in the
Eastern Mediterranean region������������������������������������������������������������������������������������������������������31

Box 2.2. Rapid production of evidence summaries at the Ottawa Hospital

Research Institute using an 8-step approach�������������������������������������������������������������������� 32

Box 2.3. Interim guidance for the conduct of rapid reviews������������������������������������� 33

Box 3.1. Generally accepted standards for study selection, data abstraction,
and quality assessment for systematic reviews ��������������������������������������������������������������41

Box 3.2. Evidence supporting decisions regarding streamlined methods for

rapid reviews��������������������������������������������������������������������������������������������������������������������������������������������44

Box 3.3. Reproducibility of systematic reviews�����������������������������������������������������������������45

VIII Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide
Box 3.4. Example of methodological decisions to allow a review to be
conducted rapidly��������������������������������������������������������������������������������������������������������������������������������49

Box 4.1. Approaches to framing a rapid review�����������������������������������������������������������������63

Box 4.2. Sources of prior systematic evidence and their application in a rapid
review�����������������������������������������������������������������������������������������������������������������������������������������������������������64

Box 5.1. Conducting a rapid review with maximal engagement in mind����������79

Box 7.1. Essential questions for developing a research dissemination plan103

Box 8.1. Barriers and facilitators to the uptake of rapid reviews for health
care decision‑making�����������������������������������������������������������������������������������������������������������������������111

Box 8.2. Methods to increase the uptake of rapid reviews������������������������������������� 114

Box 8.3. Conducting a rapid review with maximum uptake in mind�����������������116

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide IX

 CONTRIBUTORS
Editors
Andrea C. Tricco, Etienne V. Langlois, Sharon E. Straus

Editorial Support Team

Jesmin Antony, Huda M. Ashoor, Melissa Courvoisier, Susan Le, Peggy Robinson, Reid Robson

Management Committee
Sandy Oliver, Rhona Mijumbi-Deve, Andrea C. Tricco, Sharon E. Straus, Etienne V. Langlois

Scientific Advisory Board

Daniel Phillips, Tomas Pantoja, Suzanne Kiwanuka

Authors
Chapter 1 - The need for rapid reviews to inform health policy and systems
Etienne V. Langlois, Sharon E. Straus, Rhona Mijumbi-Deve, Simon Lewin, Andrea C. Tricco.

Chapter 2 - Performing rapid reviews

Valerie J. King, Chantelle Garritty, Adrienne Stevens, Barbara Nussbaumer-Streit, Lisa Hartling, Curtis S.
Harrod, Jeanne-Marie Guise, Chris Kamel.

Chapter 3 - Improving quality and efficiency in selecting, abstracting, and appraising studies for rapid reviews
Ba’ Pham, Reid C. Robson, Sonia M. Thomas, Jeremiah Hwee, Matthew J. Page, Andrea C. Tricco.

Chapter 4 - Selecting rapid review methods for complex questions related to health policy and system improvements
Sandy Oliver, Michael Wilson, G. J. Melendez-Torres, Mukdarut Bangpan, Kelly Dickson, Carol Vigurs.

Chapter 5 - Engaging policy-makers and health systems managers in the conduct of rapid reviews
Andrea C. Tricco, Wasifa Zarin, Vera Nincic, Patricia Rios, Paul A. Khan, Marco Ghassemi, Sanober S. Motiwala,
Ba’ Pham, Sandy Oliver, Sharon E. Straus, Etienne V. Langlois.

Chapter 6 - Fostering the conduct and use of rapid reviews of health policy and systems research
in low- and middle-income countries
Rhona Mijumbi-Deve, Fadi El-Jardali.

Chapter 7 - Reporting and disseminating rapid review findings

Shannon E. Kelly, Sharon E. Straus, Jessie McGowan, Kim Barnhardt.

Chapter 8 - Improving the uptake of rapid reviews

Andrea C. Tricco, Roberta Cardoso, Sonia M. Thomas, Sanober S. Motiwala, Shannon Sullivan, Michael
(Ryan) Kealey, Brenda Hemmelgarn, Mathieu Ouimet, Laure Perrier, Sharon E. Straus.

X Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

 ACKNOWLEDGEMENTS
The editors wish to thank the authors, advisors, peer reviewers, and editorial support team, whose dedication
and expertise made this publication possible. Special thanks to Abdul Ghaffar for his guidance in developing
the publication, Gail Klein for the management of the beginning of the project, and Michelle Thulkanam who
supported the design and printing phases. We wish to acknowledge Shirley Ho and Gefra Gustavo Fulane
for assisting with background research for Chapter 1, and Sasha Shepperd and Michael Hillmer for their
contributions to the publication that Chapter 5 was based upon.

In addition, we acknowledge with thanks the following peer reviewers for their valuable feedback: Elie Akl,
Thomas W. Concannon, Janet Crain, Racha Fadlallah, Robin Featherstone, Gerald Gartlehner, Dena Javadi,
Kiera Keown, Sara Khangura, Laurenz Langer, Tianjing Li, Edoardo Masset, Nancy Santesso, Birte Snilstveit,
Ruth Stewart, and Britta Tendal. We thank Simon Lewin and David H. Peters for reviewing the development
process and final content of the Guide. We also wish to acknowledge Ismail Sharif, Roberta Cardoso and Clara
Tam for their contributions to this publication.

This publication was funded by Alliance for Health Policy and Systems Research, an international partnership
hosted by the World Health Organization, with support from the Norwegian Government Agency for Development
Cooperation (Norad), the Swedish International Development Cooperation Agency (Sida) and the UK Department
for International Development (DFID).

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide XI

 PREFACE
As the global health community is pushing for ambitious reforms towards universal health coverage and
health equity in the era of Sustainable Development Goals (SDGS), there is increasing demand for relevant,
contextualized evidence to strengthen health policy and systems.

Governments worldwide increasingly recognize the need for knowledge synthesis to inform health policymaking
and health systems decision-making in routine, as well as emergency contexts. Rapid reviews are an efficient
solution to support health policy and systems decision-making by providing high-quality evidence in a timely
and cost-effective manner.

Rapid reviews are also increasingly recognized as an optimal approach to generate the necessary contextualized
knowledge relevant to different health systems settings, thus promoting their applicability for decision-making.
Rapid reviews often stem directly from requests by end-users, including policymakers and health system
decision-makers. This demand-driven feature also contributes to their usability to strengthen local health
systems and respond to pressing policy decisions.

Furthermore, there is increasing experience globally in establishing “rapid response services”, whereby
researchers respond to queries from policy-makers or health systems managers through rapid evidence
products. This is a promising avenue to support evidence-informed policy-making globally.

Yet, there is a paucity of guidance on the conduct, contextualization, and use of rapid reviews, particularly
in relation to complex health policy and systems evidence. There is also a need for capacity strengthening
in low- and middle-income countries in the field of evidence synthesis and rapid reviews more specifically.

Rapid Reviews to Strengthen Health Policy and Systems: a Practical Guide aims to address this gap by providing
guidance on how to conduct rapid reviews and support their use to inform health policy and systems decisions.
The guide also aims to provide practical recommendations on the conduct of rapid reviews to facilitate their
use in decision-making. At the same time, key challenges in fast-tracking knowledge synthesis processes
and applying them to complex issues pertaining to health policy-making and health system strengthening are
described.

This Practical Guide was developed as a global public good of relevance to both the research and policy
communities. I anticipate it will provide useful guidance to support knowledge synthesis and evidence-informed
policy- and decision-making worldwide.

Marie-Paule Kieny
Assistant Director-General
Health Systems and Innovation Cluster
World Health Organization

XII Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide
 EXECUTIVE SUMMARY
Health systems worldwide face increasingly complex challenges that require the generation and synthesis of
knowledge in limited amounts of time. Policy-makers require valid evidence to support time-sensitive decisions
regarding the coverage, quality, efficiency and equity of health systems. Systematic reviews and other types
of evidence syntheses are increasingly employed to inform policy-making and produce guidance for health
systems. However, the time and cost to produce a systematic review is often a barrier to its use in health
policy and systems decision-making.

Rapid reviews have emerged as a useful approach to provide actionable and relevant evidence in a timely
and cost-effective manner. Rapid reviews are a type of knowledge synthesis for which the steps of the
systematic review are streamlined or accelerated to produce evidence in a shortened timeframe. In a range
of circumstances, there is value in accelerating the review process and fast-tracking knowledge synthesis for
pressing policy and systems decisions. In times of emergency and crisis for instance, rapid reviews can provide
strategic evidence to make crucial decisions about health systems response. Expediting evidence synthesis is
also essential for health systems strengthening beyond emergencies, in different routine situations in which
policy-makers and managers need to make informed decisions about health systems.

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide offers a rationale for the conduct
and uptake of rapid reviews to support health policy and systems decisions. The publication provides guidance
on how to plan, conduct, and promote the use of rapid reviews to strengthen health policy and systems.
The Guide explores different approaches and methods for expedited synthesis of health policy and systems
research, and highlights key challenges for this emerging field, including its application in low- and middle-
income countries. This publication does not provide a one-size-fits-all approach to rapid reviews of health
systems evidence, but rather a reflection on their usefulness, and key insights into applied methods to swiftly
conduct knowledge syntheses and foster their use in policy and practice.

Chapter 1: The need for rapid reviews to inform health policy and systems provides the rationale for
the emerging use of rapid reviews for health policy-making and health systems strengthening. This chapter
presents the rapid review approach and its application in the field of health policy and systems research.
Some of the methods introduced to enhance the timeliness of reviews include knowledge synthesis shortcuts,
automation and intensification of review steps, as well as practical considerations to expedite reviews. We also
acknowledge the challenges and limitations in developing and using rapid reviews to strengthen complex
health policy and systems.

Chapter 2: Performing rapid reviews is the first of three chapters on how to conduct rapid reviews of health
policy and systems research. This chapter presents an overview of methods used to streamline the systematic
review process at various stages, from searching the literature to synthesizing the results. As the methods
used can vary from one review to the next, we emphasize the transparency of methodological choices and
encourage constant collaboration with stakeholders to ensure the review fulfills its intended purpose.

Chapter 3: Improving quality and efficiency in selecting, abstracting, and appraising studies for rapid
reviews provides recommendations on how to maintain the scientific rigor of these three steps of the review
process, while using streamlined approaches to increase efficiency. Some of the strategies highlighted include
the use of well-defined eligibility criteria, explanation and elaboration forms, training and calibration exercises,
and the involvement of content experts and experienced reviewers.

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide XIII
Chapter 4: Selecting rapid review methods for complex questions related to health policy and system
improvements describes how to select effective streamlined methods for rapid reviews by considering how
the project will be managed, the scale and scope of the work to be completed, and the existing knowledge
available. Strategies to conduct rapid reviews are provided, such as using a two-stage process of first scoping
the literature, then selecting a focus; use of a transdisciplinary team to speed and enhance the review; use of
a framework to organize the concept under study; as well as conducting a search for existing reviews to allow
reviewers to summarize and integrate the review findings, resynthesize primary studies, or update the search
and reanalyse one or more of the systematic reviews.

Chapter 5: Engaging policy-makers and health systems managers in the conduct of rapid reviews
expands on the importance of facilitating an effective partnership between researchers and decision-makers.
This chapter discusses the importance of involving policy-makers and health systems managers in the rapid
review process to increase relevance and applicability. We present potential points and levels of stakeholder
engagement, to be tailored to each review.

Chapter 6: Fostering the conduct and use of rapid reviews of health policy and systems research in
low- and middle-income countries outlines the specific challenges of conducting rapid reviews in low-
and middle-income countries, while highlighting the need to develop supportive systems and structures to
overcome these challenges. Strategies described to ensure rapid reviews are utilized to their full potential
include addressing methodological concerns, mobilizing sustainable resources, and raising the profile of rapid
reviews in these countries.

Chapter 7: Reporting and disseminating rapid review findings focuses on knowledge translation and
dissemination of rapid reviews. This chapter describes how to report findings of health policy and systems
reviews by prioritizing the practical needs of the knowledge user, and recommends the use of reporting
guidelines when developing rapid review reports. To assist the reader in the development of a dissemination
plan, a checklist of essential questions is provided.

Chapter 8: Improving the uptake of rapid reviews identifies barriers to the use of rapid reviews, and suggests
several methods to help facilitate increased uptake by decision-makers. Although rapid reviews can be helpful,
policy-makers do not always use rapid review evidence to inform their decisions. This chapter suggests that
promotion of the validity and usefulness of rapid reviews, improved formatting of evidence reports, and the
development of connections with health systems managers and policy-makers can help promote the uptake
and use of rapid reviews.

As a whole, the chapters of this Guide can be used to inform both researchers and policy-makers on the utility
of rapid reviews to support health policy and systems decisions. The Guide also identifies key priorities for
additional research on the conduct and application of rapid reviews for health policy and systems. This strategic
research agenda includes, but is not limited to: robustness and transparency of rapid reviews methods;
potential risk of biases introduced by rapid review methods; rapid synthesis and analysis of complex health
policies and health systems interventions and reforms; external validity and context-sensitivity as applied to
rapid reviews and the broader field of health systems research synthesis; and good practices in strengthening
individual and institutional capacities for the generation and use of rapid reviews, especially in low- and middle-
income countries.

XIV Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide
 PUBLICATION PROCESS
Authors with expertise in each of the chapter subject areas were approached and selected by the editors to
contribute to the Guide. Lead authors then selected co-authors to assist them with the development of content
and the presentation of their respective chapters. Each chapter was drafted by the authors and underwent 4
rounds of revisions by the editors, scientific advisory board, copy-editors, and peer reviewers.

The Guide management committee was comprised of the three editors along with two of the lead authors Rhona
Mijumbi-Deve and Sandy Oliver, who participated in the early development of the protocol and Guide outline.

We invited individuals with expertise in evidence synthesis and health policy and systems research to join
our independent scientific advisory board. Daniel Phillips, Tomas Pantoja, and Suzanne Kiwanuka provided
suggestions for chapter authors, comments on the annotated outline of the chapters, and high-level feedback
on the fully drafted chapters.

All chapters were then sent for medical copy-editing to Peggy Robinson (medical writer) and Reid Robson
(medical editor) who proofread the chapters for consistency and clarity, overlap in content and adherence to
the WHO publication style guide.

An international group of peer reviewers were approached based on their experience and interest in the areas
of rapid reviews and health policy and systems research. Each chapter was reviewed by two peer reviewers,
who independently provided feedback for the authors to consider. The authors provided a point-by-point
response to each reviewer comment and incorporated appropriate changes.

The project manager, Jesmin Antony, developed a plan for the Guide, managed overall timelines and
communication, and drafted the chapter objectives and other relevant documents to inform the development
of the Guide. The project manager and editorial support team (Huda Ashoor, Melissa Courvoisier, and Susan
Le) coordinated the review of chapters with the lead authors, management committee, scientific advisory
board, copy-editors and peer reviewers, and conducted a review of all chapters.

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide XV

 TARGET AUDIENCE
Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide will appeal to those interested in
learning how to plan, conduct or promote the use of rapid reviews to strengthen health policy and systems.
As such, the intended audiences include researchers, decision-makers (e.g. policy-makers and health systems
managers), knowledge brokers, journal editors and peer reviewers, as well as commissioners, funders and
agencies supporting the use of rapid reviews.

For researchers, the Guide provides practical guidance on how to conduct rapid reviews; as such, its use can
help to build capacity among junior and intermediate reviewers. Decision-makers will gain a broad understanding
of rapid reviews and how they can better collaborate with the researchers conducting the reviews. Journal
editors and peer reviewers can use our Guide to determine whether authors of rapid reviews of health
policy and systems research submitted for publication have used appropriate, streamlined methods. Finally,
commissioning and funding agencies can use the Guide as a reference providing information on rapid reviews,
health policy systems research, and the conduct of research in low- and middle-income countries.

XVI Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide
 ACRONYMS
AHPSR: Alliance for Health Policy and Systems Research

AUB: American University of Beirut

COM-B: capability, opportunity, motivation, and behaviour model

EQUATOR: Enhancing the QUAlity and Transparency Of health Research) Network

GRADE: Grading of Recommendations, Assessment, Development and Evaluation

HPSR: Health Policy and Systems Research

IECS: Institute for Clinical Effectiveness and Health Policy

K2P: Knowledge to Policy Center

KSG: Knowledge Synthesis Group

LMIC: Low- and middle-income countries

MakCHS: School of Medicine and School of Public Health, Makerere University College of Health Sciences

NICE: UK’s National Institute for Health and Care Excellence

OHRI: Ottawa Hospital Research Institute

PICO: Population, Intervention, Comparator, and Outcome

PRESS: Peer Review of Electronic Search Strategies

PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement

PRISMA-P: Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols

RCT: Randomized controlled trial

SPARK: Center for Systematic Reviews on Health Policy and Systems Research

StaRI: Standards for Reporting Implementation Studies

SUMARI: System for the Unified Management, Assessment and Review of Information

SURE: Supporting the Use of Research Evidence project

TB: Tuberculosis

TRASI: Tool for Recording and Accounting for Stakeholder Involvement in Systematic Reviews

WHO: World Health Organization

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide XVII
 NAVIGATION MAP

CHAPTER Researchers

1 The need for rapid reviews to inform health policy and systems
Objectives: Presents the rapid review approach and its application for health policy‑making
and health systems strengthening
• p.3 to 5
• p.10 to 15

2 Performing rapid reviews

Objectives: Describes different methods that can be used for rapid reviews of health
policy and systems research
• p.23 to 34

3
Improving quality and efficiency in selecting, abstracting, and appraising studies
for rapid reviews
• p.41 to 50
Objectives: Outlines how to tailor study selection, data abstraction, and risk of bias
appraisal for rapid reviews

4
Selecting rapid review methods for complex questions related to health policy
and system improvements
• p.57 to 67
Objectives: Provides strategic decisions for the conduct of rapid reviews of health
policy and systems research

5
Engaging policy-makers and health systems managers in the conduct of
rapid reviews
• p.73 to 78
Objectives: Discusses the importance of involving policy-makers and health systems
managers in the rapid review process

6
Fostering the conduct and use of rapid reviews of health policy and systems
research in low- and middle-income countries
• p.85 to 92
Objectives: Describes challenges and strategies to increase the use of rapid reviews
in low- and middle-income countries

7 Reporting and disseminating rapid review findings

Objectives: Explains how to optimize the reporting of rapid reviews and disseminate findings
• p.97 to 105

8
Improving the uptake of rapid reviews
Objectives: Presents barriers to the use of rapid reviews and methods to facilitate • p.111 to 116
increased uptake by policy-makers and health systems managers

These are the pages the editors deemed most relevant to each target audience. However, other pages might also be
relevant, depending on the reader’s interest.

XVIII Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide
 MOST RELEVANT PAGES FOR THE GUIDE’S TARGET AUDIENCES:

Decision-makers Journal editors/ Commissioners/ People/groups/

(policy-makers and Knowledge brokers peer reviewers funders of agencies supporting
health care managers) rapid reviews rapid reviews

• p.3 to 5 • p.3 to 5
• p.3 to 9 • p.3 to 9 • p.3 to 9
• p.10 • p.10
• p.15 • p.15 • p.15
• p.12 to 15 • p.15

• p.26 • p.23 to 28
• p.26 • p.23 to 26
• p.28 • p.29 • p.23 to 34
• p.34 • p.34
• p.34 • p.29 to 34

N/A N/A • p.41 to 50 N/A • p.41 to 50

N/A N/A • p.57 to 67 N/A • p.57 to 67

• p.73 to 78 • p.73 to 78 N/A • p.73 to 78 N/A

• p.85 to 92 • p.85 to 92 N/A • p.85 to 92 • p.85 to 92

• p.97 to 105 • p.97 to 105 • p.97 to 105 • p.97 to 105 • p.97 to 105

• p.111
• p.111 to 116 • p.111 • p.111 to 116 • p.111
• p.116

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide XIX
1
THE NEED FOR RAPID
REVIEWS TO INFORM
HEALTH POLICY AND
SYSTEMS

Etienne V. Langlois, Sharon E. Straus,

Rhona Mijumbi-Deve, Simon Lewin, Andrea C. Tricco
KEY POINTS

• Policy-makers often need and request evidence to plan, develop, and

implement health policies in a timely fashion.

• Systematic reviews are increasingly used to inform policy decisions and

produce guidance for health systems, yet the production of systematic
reviews is often protracted and misaligned with decision timelines.

• Rapid reviews are a useful approach to swiftly provide actionable and

relevant evidence to make informed decisions about health systems in
routine as well as emergency contexts.

• Rapid reviews are generated through a transparent, scientific, and reproducible

method that respects the key principles of knowledge synthesis.

• Policy-makers and health systems managers require rapid reviews that

address a range of issues, including the effectiveness of health systems
interventions and policies, how and in what settings these interventions
work, and their cost‑effectiveness.

• Various methods exist to expedite the conduct of reviews to inform health

policy and systems decisions; the main challenge lies in accelerating review
methods while maintaining robustness and transparency.

• The complexity of health systems decision-making is both an important

challenge and a key opportunity for developing the field of rapid reviews
of health policy and systems evidence.
1.1 INTRODUCTION reviews takes an average of 12–24 months (5),
and this protracted timeline is often misaligned
Riad Teriaqi, a bag-maker by trade, has lost with policy- and decision-making cycles. Lack of
track of the number of doors he has knocked timeliness in the production of reviews therefore
on. This is Mr Teriaqi’s eighth house-to-house remains a strong barrier to the use of knowledge
polio immunization campaign since Syria’s synthesis in health policy-making (6).
2013 outbreak, which left 35 children paralysed.
Polio was thought to have been eradicated in Rapid reviews have emerged as a useful approach
Syria, where cases had not been reported for to provide actionable and relevant evidence in
almost two decades. Volunteers like Riad are the a timely and cost-effective manner (5). For the
backbone of Syria’s efforts to prevent another purpose of this Guide, we define a rapid review
outbreak, but after years of civil war and with as a type of knowledge synthesis in which
over 11 million people displaced, the challenge systematic review processes are accelerated
is daunting (1). and methods are streamlined to complete the
review more quickly than is the case for typical
The conflicts in Syria, and those like it, create the systematic reviews (7). Rapid reviews take an
potential for the rapid spread of communicable average of 5–12 weeks to complete, thus providing
diseases because of the evidence within a shorter
resultant breakdown in critical time frame required for some
infrastructure including water health policy and systems
and sanitation systems, scale Timely reviews are of decisions (5).
of population displacement,
loss of trained healthcare
the utmost importance Rapid reviews are common in
wo r ke r s , e q u i p m e n t to inform health health technology assessment,
shortages and overall reduced clinical care, and comparative
health system functionality.
policy and systems effectiveness research,
To support health interventions recommendations, and they are also increasingly
in complex emergencies used in health policy-making
like Syria, there is a need
including rapid and the development of health
to understand evolving advice guidelines. programmes globally (7, 8).
health systems challenges, In a range of circumstances,
for instance the disruption there is value in accelerating
of communicable disease the review process and fast-
surveillance and control measures. Understanding tracking knowledge synthesis for pressing policy
these challenges requires in turn the generation and systems decisions. In times of emergency
and synthesis of context-sensitive knowledge, and crisis, for instance, rapid reviews can provide
often in limited time frames. strategic evidence to allow crucial decisions
to be made about health systems responses.
This is the case not only in war-torn settings, Emerging disease outbreaks are examples of
like Syria, but in conflict-free health systems such public health emergencies in which health
settings worldwide, as these systems become systems are pressured for a rapid response.
increasingly complex. Policy-makers require valid In these circumstances, decision-makers may
evidence to support time-sensitive decisions be confronted with an absence of reviews on
regarding the coverage, quality, efficiency, specific health policy challenges, or existing
and equity of health systems. reviews may lack context specificity to inform
health system decisions (9).
Systematic reviews and other types of knowledge
syntheses are increasingly employed to inform As such, timely reviews are of the utmost
policy decisions and to produce guidance for importance to inform health policy and systems
health systems (2-4). Yet conducting systematic recommendations, including rapid advice

3 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

guidelines (10, 11). Supporting the health system and provide policy guidance on disease control
in Syria in the context of conflict and mass in this conflict‑afflicted setting (Box 1.1) (12).
displacement is a case in point, whereby a rapid Rapid reviews such as this can also be useful
review was conducted to describe trends in major for identifying evidence gaps and areas where
communicable diseases, assess the infectious primary research should be targeted (13).
disease surveillance and response systems,

BOX 1.1. Case example: Rapid review to understand communicable disease surveillance and
control in conflict‑affected Syria

Since the start of the war in 2011, systematic surveillance systems in Syria have been
dysfunctional. The mass mobilization and displacement of individuals compound the issue by
making it difficult to collect accurate and timely data when they are needed most. Nonetheless,
if diseases like polio are to be contained in Syria, and the wider theatre to which millions of
people have fled, the technical challenges to communicable disease prevention and control
must be understood.

Failures and fragmentation of communicable disease surveillance systems have been identified
as important challenges to outbreak and infectious disease management by the Syria Public
Health Network, a collaboration of researchers and practitioners established to address
various aspects of the health response to the Syria crisis. Through consultations focusing on
the health system responses to the crisis in Syria and surrounding countries, the Network
identified a critical need for up-to-date and context-sensitive evidence on communicable
diseases prevention and control measures in the country.

To address this knowledge gap and inform realistic recommendations, Ismail and colleagues
(12) performed a rapid review with the objective of describing trends in major communicable
diseases during the on-going conflict in Syria, and the challenges to disease surveillance and
control in the context of dynamic, large-scale population displacement, unplanned mass
gatherings, and disruption to critical infrastructure. The review focused on the published
peer-reviewed and grey literature, supported by secondary analysis of monitoring data from
two disease early warning systems currently operational in Syria, focusing on three diseases:
tuberculosis (TB), measles, and polio.

The rapid review was completed in seven weeks, and numerous means were used to
accelerate the research process, including:
• using a clearly defined conceptual framework to guide the review, in this case the WHO
framework for assessing capacity for implementation of the International Health Regulations
at the national level;

• limiting the time period for the literature searches from 2005 to 2015, and restricting the
grey literature search to specific agencies working on communicable disease surveillance
and control in Syria. These institutions were identified through an expert consultation
managed by members of the Syria Public Health Network;

• deploying a large review team with varied skill sets, enabling parallelization of review tasks.

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 4

BOX 1.1. Case example: Rapid review to understand communicable disease surveillance and
control in conflict‑affected Syria (continued)

The rapid review provided the current landscape of surveillance mechanisms, an understanding
of preparedness and response capacity, an analysis of coverage through immunization
programmes, and an understanding of current gaps and challenges in infectious disease
management. For tuberculosis, disruption of all aspects of the control programme, including
prevention, case finding, diagnosis, and management, has led to an increase in cases among
displaced populations. The review also identified the lack of information on the health status
of prisoners in Syria as a particular concern for the spread of TB, building on previous evidence
of high incidence of the disease in incarcerated populations. In addition, the rapid review
findings highlighted that few of the public health facilities still functional in Syria have the
capacity to perform the specialized tests required to confirm communicable disease cases.

In this context, Ismail and colleagues (12) identify the need for innovative approaches to ensure
that early case detection, treatment initiation, contact tracing, and follow-up is implemented,
which in turn would contribute to reducing the risk of treatment interruption and subsequent
drug resistance. The review also stresses the need to develop basic or mobile laboratory
capabilities linked with the surveillance mechanisms, to increase the accuracy and timeliness
of case identification.

This is an apt example of the role of rapid reviews in synthesizing key information to assess
needs and pave the way for strategic health system intervention in times of crisis.

Source: Ismail et al., 2016 (12)

Expediting research synthesis is also essential and context specificity of research is a strong
for health systems strengthening beyond determinant of its uptake by policy-makers and
emergencies, in various routine situations in other health systems decision-makers (6).
which policy-makers and managers need to make
informed decisions about health systems quickly. 1.2 THE RAPID REVIEW APPROACH
For instance, rapid reviews may be useful where
policy-makers have given a department of health Rapid reviews have been described as falling
a very short time frame in which to identify policy “within the family” of systematic reviews,
options in relation to a topical health systems as their methodology was established to provide a
issue, such as developing strategies to expand transparent, scientific method that is detailed and
health insurance or to scale up the implementation reported in advance and that will be reproducible
of a key health intervention. Rapid reviews are by others (15). Rapid reviews are intended to
also considered a cost-saving strategy for health respect the key principles of knowledge synthesis,
system decision-makers and other commissioners including a clear statement of review objectives,
faced with limited resources. Rapid reviews predefinition of eligibility criteria, assessment of
are thus emerging as an efficient approach to the validity of findings (e.g. through assessing risk
generating the necessary context-sensitive of bias), and systematic presentation and synthesis
knowledge needed to inform decisions on health of results. The term “rapid review” incorporates an
systems questions (14), thus promoting their array of products that vary greatly in their purpose,
applicability for decision-making. The usefulness methodological rigour, comprehensiveness,
of contextualized rapid reviews is supported by resources used, transparency, and time spent
previous experience showing that the relevance for their production (10, 16). This wide spectrum

5 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

of products reflects differences in how agencies our Guide does not directly address the production
and stakeholders commissioning and producing of rapid evidence, policy briefs, or other policy-
knowledge syntheses define the review topic, friendly summaries of research. Guidance to
select streamlined methods, and customize support these latter types of outputs has been
the timeline, reporting, and dissemination of developed by the Supporting the Use of Research
reviews (7). Evidence (SURE) project, including the SURE
Guides for Preparing and Using Evidence-Based
This Guide focuses mainly on the swift generation Policy Briefs (17). Table 1.1 categorizes the various
of new knowledge through rapid reviews, or what types of rapid evidence summaries, to highlight
Hartling and colleagues (16) call “true” rapid our conception of rapid reviews and how they
reviews, i.e. those that use reduced or accelerated differ from other rapid response products.
forms of systematic review methodology. As such,

TABLE 1.1. Categorization of rapid evidence products, according to extent of knowledge

synthesis

Category Description

Inventories Inventories only list the evidence that is available on a

given topic. There is no attempt to appraise, summarize
or synthesize the evidence for further use, nor is there an
attempt to present conclusions or recommendations to the
knowledge user.

Rapid response briefs Rapid response briefs present a summary of the best
available evidence in a synthesized and contextualized
manner, in direct response to a decision-maker’s question.
They are knowledge translation products created through
formal methods to synthesize and appraise the evidence.
They do not generate new knowledge but use findings that
are already available, especially from existing systematic
reviews.

Rapid reviews Rapid reviews represent a knowledge generation strategy.

They synthesize findings and assess the validity of research
evidence using “abbreviated” systematic review methods,
modifying these methods to generate evidence in a short
time.

Source: Adapted from Hartling et al., 2015 (16)

1.3 THE NEED TO SWIFTLY policy-making and strategies for health systems
INFORM HEALTH POLICY AND strengthening (18). Syntheses of research can
SYSTEMS DECISIONS support policy-makers by providing state-of-
the-art knowledge and actionable evidence at
Rapid reviews are garnering interest, numerous steps in the policy-making process
as governments worldwide recognize the (3, 19). Rapid reviews can inform health policy-
need for this type of evidence to inform health making in a number of ways, as shown in in Table

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 6

1.2. Such evidence can be combined with local strategic approach to address a range of barriers
evidence on modifying factors, values, and the to the uptake of research evidence. These
availability of resources to make judgements barriers go beyond timeliness and include the
about the anticipated benefits, harms, and costs engagement of decision-makers, incentives for
of policy options in a particular context, thereby demand-driven research, and the relevance of
informing health policy decisions (2). As such, scientific findings to local health systems (6).
rapid reviews are increasingly recognized as a

TABLE 1.2. Rapid reviews to support health policy-making

Policy step Description Example of rapid review

Priority-setting Identifying and conceptualizing priority Rapid review of the evidence

issues for the policy agenda (20) on prevention and control of
vector-borne diseases in urban
areas of low- and middle-
income countries, with a view to
informing policy priorities (21)

Policy Assessing options to develop policies. Rapid review of international

formulation Here, policy-makers can make the models of primary care provision
most of rapid reviews that focus and primary care policies (22)
on different questions, including
but not limited to rapid reviews of
effectiveness to identify the benefits
and harms of policy options, and rapid
reviews of economic evaluations to
explore the cost-effectiveness of
different policy interventions (19)

Policy Mobilizing resources by governments Rapid review of barriers to and

Implementation and implementers. At this stage, facilitators of the implementation
informative rapid reviews could of e-health systems in rural
include qualitative evidence syntheses communities (23)
to assess factors influencing the
implementation and scalability of a
policy (19)

Experience from developing rapid response interventions can be useful in providing decision-
mechanisms in low-income settings shows makers with key evidence to strengthen the
that policy-makers have evidence requests that performance of health systems or reform their
need to be addressed within relatively short core elements (e.g. human resources or financing).
time frames, including requests for evidence Table 1.3 presents examples of rapid reviews
about health systems arrangements, such as that have been conducted to inform policy and
delivery of services and governance (24). In these systems decisions. Rapid reviews can also be
circumstances, rapid reviews that synthesize used to scope existing health policy and systems
knowledge on the effectiveness, implementation, evidence and to identify gaps that might require
and efficiency of health policy and systems additional research (13).

7 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

TABLE 1.3. Examples of rapid reviews for health policy and systemsa

Health Population Concept Context

system
challenge

Prevention Urban Rapid review of scoping Limited to cities and urban

and control of population and systematic reviews to areas in LMICs; socioeconomic
vector-borne examine the evidence on contexts vulnerable to vector-
diseases (21) urban health interventions borne diseases
for prevention and control of
vector-borne diseases

Integration of Patients Rapid review of the evidence No geographical limitations;

e-mental health receiving on digital interventions for special focus on geographically
interventions in mental mental health (including hard-to-reach populations
health systems health their applications, strengths, and socioeconomic barriers
(25) services and limitations) in relation to mental health services
to integration in health-care accessibility
systems

Physician Health-care Rapid review to explore the No geographical limitations;

payment providers in impact of physician payment focus on specialist health-care
schemes in oncology methods on system costs, systems (in oncology) facing
cancer care (26) quality of care, and health rising population incidence of
outcomes, with a specific cancer and increasing health
focus on cancer control system costs

Prevention and General Rapid review on the aspects Focus on socioeconomic

management of population of primary care that are determinants, including poverty
mental health (adults, effective in preventing, and unemployment; specific
disorders in children) recognizing, and managing challenges of young and elderly
primary health with mental mental health issues patients, as well as those with
care (14) health across the lifespan: the post-traumatic stress disorder
disorders people for whom these
interventions work, in what
circumstances, and for what
reasons

Demand-side Women and Rapid review of the impact LMIC contexts of high
policies and newborn of demand-side intervention maternal and early neonatal
interventions populations on utilization of services mortality, low perceived
for maternal and in LMIC and health outcomes for quality of health-care services,
neonatal health settings mothers and neonates direct and indirect costs,
in LMICs (27) discrimination (religious,
political, ethnic), and dearth of
information about maternal and
neonatal health services

LMIC, low- and middle-income countries.

a Reported using the Population, Concept and Context approach (28). With this approach, “concept” refers to interventions, phenomena of
interest, and outcomes, and “context” refers to external and internal influences such as geographical location and cultural factors, as well as
health policy and systems determinants.

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 8

1.4 HEALTH POLICY AND policy-making (32, 33). As such, the science of
SYSTEMS RESEARCH health policy and systems research addresses
the multiple interactions and synergies between
Health policy and systems research is a health policies and systems, reflecting the fact
multidisciplinary field studying how various that systems dynamics directly inform policy-
stakeholders, institutions, and interests interact making, and vice versa (Figure 1.1). Health policy
in policy development and implementation and systems research recognizes that health
processes, in order to contribute to policy systems are constituted by the “hardware”
outcomes (29, 30). Empirically, health systems components or building blocks, as much as the
research addresses the building blocks of health “software” consisting of interests, values, norms
systems: governance, information, financing, and power dynamics (32). The field of health
service delivery, human resources, and medicines policy and systems research also recognizes
and technologies, as well as their interlinkages that the health system encompasses both the
and influences on health systems performance, suppliers of policy, services, and interventions,
responsiveness, and people-centredness (31). and the communities and households intended
While the building blocks framework provides a to benefit from them who, as citizens, also play
simplified approach, health systems are in reality important roles in policy change (31). The scope
more complex and comprehensive by nature, of health policy and systems research is further
and their functioning is intertwined with health described in Box 1.2.

FIGURE 1.1. Interface of health policy and systems research

Global & National Forces

Health Systems Health Policy

Hardware:
Content &
Structure;
System Instruments
Organization;
Technology; functioning Actors, Power
Resourcing
& & Politics
Software: Policy
Institutions, Interests
Values; Change & Ideas
Norms;
Actions & Relationships

Source: Gilson, 2012 (32)

Reproduced with permission from World Health Organization (2012)

9 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

BOX 1.2. Scope of health policy and systems research

Health policy and systems research encompasses research on the policies, organizations,
programmes, and people that make up health systems, as well as the interactions among these
elements and the socioeconomic influences over decision-making practices within a health
system. The ultimate goal of health policy and systems research is to generate knowledge
that will enable societies to strengthen health systems and achieve health goals. Health policy
and systems research is concerned with the system-level factors and forces that cut across
actions dedicated to tackling particular health problems, as well as those that underpin and
shape the performance of health programmes. Health policy and systems research does not
address clinical management of patients or basic scientific research (e.g. research into cell
or molecular structures). Health policy and systems research is characterized by the type of
problems that it addresses, rather than by any particular disciplinary underpinnings. As such,
most health policy and systems research is multidisciplinary by nature.

Sources: Gilson, 2012 (32); Alliance for Health Policy and Systems Research, 2007 (30)

1.5 WAYS TO EXPEDITE REVIEWS • increasing the intensity of work on review

ON HEALTH POLICY AND processes, by intensifying the efforts of
SYSTEMS RESEARCH multiple reviewers to simultaneously complete
review steps (i.e. parallelization of tasks), e.g.
There is no consensus to date on the timeline eligibility screening, data abstraction, and risk-
that would qualify a review as “rapid” (34), but it of-bias assessment;
has been suggested that most rapid reviews are • using review shortcuts, whereby one or more
conducted within 12 weeks (35). In addition to systematic review steps may be reduced
timeliness, rapid reviews vary in their purpose and or omitted;
format and in the methods used for knowledge • automating review steps, by developing,
synthesis (7). This variation is related in part to adapting, and using new technologies to fast-
the novelty of rapid review methods (including track the standard systematic review steps,
lack of agreement on optimal methods) as well as e.g. screening or data abstraction (10).
the tailoring of the timing and scope of reviews
to decision-makers’ needs (7). Various methods Chapters 2 and 3 of this Guide provide further
are available to expedite the conduct of reviews information on various approaches to streamline
informing health policy and systems. However, review processes. In addition, rapid reviews
no methodological “one-size-fits-all” approach may need to draw on a range of synthesis
exists to support rapid reviews. Rapid reviews methods, as policy-makers and health systems
are produced for decision-makers working in managers often ask questions that go beyond
an array of health system settings, in response the effectiveness of policies and programmes,
to different objectives, under different time including “how and in what settings programmes
constraints, and with different financial and human work” and “how to promote the implementation
resources available. As such, rapid reviews can or scale-up of effective strategies”. Qualitative
be considered as fit-for-purpose research outputs evidence or mixed-methods syntheses may
of an iterative process between knowledge be needed to understand factors affecting the
users (such as policy-makers and health systems implementation, scalability, and sustainability of
managers) and rapid review producers (10). Various health programmes (36). Consequently, there is
mechanisms exist to enhance the timeliness of a broad spectrum of rapid review methods and
reviews, and these mechanisms can be used outputs applicable to health policy and systems
independently or concurrently: research (37), including rapid realist review

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 10

(Box 1.3) (14). A realist review is a synthesis answering the question “What works for whom
of a wide range of evidence that seeks to under what circumstances?” rather than only
identify underlying causal mechanisms and “What works?” (38).
explore how they work under what conditions,

BOX 1.3. Case example: Use of a rapid realist review to assess integration of mental health
care into primary care

Primary care systems have a crucial role to play in ensuring continuity of care for vulnerable
populations, including individuals with mental health disorders. As the point of entry into
health-care systems, it is imperative that primary care functions effectively in the recognition
and management of mental health disorders, as well as engaging in preventive interventions.
With survey results revealing one of the highest levels of mental illness in the world (14),
and with the accompanying challenges for primary care, Northern Ireland embarked on an
initiative to modernize and improve law, policy, and services for mental health and intellectual
disabilities.

As background, the Department of Health, Social Services and Public Safety commissioned a
set of reviews of the evidence related to health promotion and protection for those affected
by mental health disorders. One of these syntheses, completed by Bunting and colleagues
(14), was a rapid realist review, undertaken to understand how well mental health programmes
and services were integrated into primary care in the country, why interventions worked (or
did not work), for whom, and under what conditions. The authors’ choice to use a rapid realist
review was strategic, as this approach aims to create a deeper understanding of why and how
something works, and the underlying pathways of the implementation and effectiveness of
an intervention. A realist review aids in this type of analysis by emphasizing the importance
of context and the interactions of interventions with the health system. As such, this rapid
review on primary care services considered the levels of need, risk factors, and profile of
service use in the population, as well as important contextual factors that affect and interact
with these phenomena.

To accelerate the review process, the authors used a variety of shortcuts, including a limited
rather than exhaustive range of search terms, restriction of the search for grey literature to
key websites, and only considering studies published since 2000. By using a rapid realist
approach, they were able to produce, in a timely manner, context-relevant evidence for
primary care integration, including support for the development of collaborative care models
for managing mental health disorders in primary care. Beyond its implications for practice and
policy, the rapid review also identified gaps for further research, including a lack of knowledge
of factors that would facilitate collaboration between service providers and users with regard
to treatment decisions.

Sources: Bunting et al., 2011 (14); Pawson, 2006 (38); Rycroft-Malone et al., 2012 (39)

1.6 CHALLENGES IN RAPID the robustness and transparency of the review

REVIEWS FOR HEALTH POLICY methods (40). Those conducting rapid reviews
AND SYSTEMS relevant to health policy and systems research thus
need to strike a balance between “abbreviating”
It is crucial to remember that conducting rapid or “accelerating” systematic review methods
reviews poses specific challenges in relation to and maintaining the methodological rigour and

11 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

transparency of typical reviews. The quantity of between those requesting and those producing
studies retrieved and the quality of the evidence reviews to address key elements of rapid reviews
might pose important challenges that review of health policy and systems research.
teams must address in order to reduce the risk
of bias and ensure the validity of review findings. > 1.7.1 Scope of the synthesis
Chapters 2 and 3 of this Guide offer practical
insights to tackle these issues. Health policy and systems challenges are often
framed by policy-makers and other decision-
Reviews of health policy and systems research makers as broad questions, which might not be
are characterized by complexity in health systems amenable to rapid review. This is a key challenge,
settings and heterogeneity in policy-making as policy issues as expressed by policy-makers
processes. In addition, many reviews in the often need to be refined and translated into a
field concern complex interventions, including “reviewable” research question. Defining the
comprehensive health policies and programmes scope of the review question is therefore an
or health systems reforms. Assessing and important step and requires a dialogue between
understanding this complexity presents an policy-makers and researchers.
important challenge and a potential caveat to
the swift conduct of health policy and systems > 1.7.2 Type of review
research reviews. For instance, reviews of health
systems interventions might be challenged by the Policy and systems decisions require different
time required to conceptualize, appraise, and make types of review, based on the nature of the
sense of heterogeneous and manifold evidence. evidence requested:

Then again, rapid reviews can be useful in studying • rapid scoping reviews to understand and map
the complexity of health systems per se, helping out existing health policies and programmes.
to make sense of underlying frameworks and Once such example is a rapid scoping review
health systems underpinnings. One example is conducted to understand medical malpractice
a rapid review conducted on the evidence for policies in obstetrics (R. Cardoso, unpublished
successful and sustainable large-scale changes data, 2017), as described in Box 1.4;
in complex health systems (41), which aimed • rapid effectiveness reviews to understand
to understand the enablers of and barriers to whether a health system intervention works,
systems change and the frameworks to guide including its intended and unintended effects
the change process. Another example is a rapid (e.g. rapid review of the effectiveness of
review conducted to appraise effective strategies interventions to improve the health of or
for reducing complex health inequalities in priority health-care utilization by homeless people (43)
public health conditions (42). As they are mostly or rapid mixed methods reviews to assess how
tailored to the needs of end-users, rapid reviews health systems interventions work, or how
are also a valuable approach to study context- to sustain or expand interventions);
sensitive evidence and generate knowledge that • rapid overviews of systematic reviews to
is relevant to complex decision-making in local synthesize recent evidence relevant to health
health systems. policy and systems, with the overviews
providing a “map” of the policy questions
1.7 PRACTICAL CONSIDERATIONS addressed by systematic reviews and the
TO EXPEDITE REVIEWS insights derived from them (19) (e.g. rapid
overview of knowledge syntheses on the
Rapid reviews are often commissioned by policy- benefits and costs of nursing and midwifery,
makers themselves, and it is important to establish both within the health-care system and wider
from the outset a clear and realistic mandate society (44)).
and time frame for completion of the synthesis.
As such, there should be continuing dialogue

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 12

BOX 1.4. Rapid scoping review of medical malpractice policies

To inform recommendations and guidance documents in various fields of global health,

the World Health Organization (WHO) often supports the generation and update of systematic
reviews on key policy-relevant challenges. For instance, there is general agreement that the
current medical malpractice systems are becoming costly and inefficient, with litigation costs
ranging from 2.4% to 10% of health-care spending in some settings. In addition, litigation
can also have positive effects, if it results in improvements in policies and practices in areas
such as pregnancy and childbirth. To better understand the problem, WHO supported the
conduct of a rapid scoping review of worldwide policies on medical malpractice in obstetrics,
and the short-term and long-term consequences of these policies, taking into account the
presence of multiple stakeholders, including patients, clinicians, health systems managers,
and policy-makers (R. Cardoso, unpublished data, 2017).

Purposes of a scoping review:

Scoping reviews are used to map the concepts underpinning a research area and the main
sources and types of evidence available (45). Scoping reviews can be used to develop a
research agenda by identifying gaps in the literature where future primary studies are required,
as well as areas that may require a systematic review. Scoping reviews can also be used to
identify the implications of policy or practice recommendations.

The rapid scoping review was performed by a team at the Li Ka Shing Knowledge Institute, St.
Michael’s Hospital (Toronto, Canada), who aimed to identify studies evaluating the effectiveness
of medical malpractice models, as well as frameworks and policies available to improve
litigation-related outcomes in obstetrics. To produce the review in a timely manner, Cardoso
and colleagues (R. Cardoso, unpublished data, 2017), used two strategies to accelerate the
scoping review process. The first strategy entailed intensifying the research by working with
nine reviewers to conduct the screening and data abstraction phases in duplicate (Figure 1.2).

Figure 1.2. Intensifying the scoping review process

Number of
Review stage Time (weeks)
reviewers

ALL LEVELS OF
SCREENING 9 4

DATA
ABSTRACTION 9 2

SYNTHESIZING
FINDINGS 3 2

REPORT WRITING 3 2

13 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

BOX 1.4. Rapid scoping review of medical malpractice policies (continued)

The second strategy entailed simplifying some components of the scoping review process.
While all types of study designs and reviews evaluating or comparing different policies were
included, the publication date was limited from 2004 to 2015 and the rapid review was restricted
to published documents written in English. References lists of relevant studies were not
scanned and the review team did not contact authors for further potentially relevant studies.

Several initiatives for improving the medical malpractice litigation system were identified,
including no-fault approaches, i.e. medical injuries compensated without proof of fault; policy
initiatives related to patient safety; communication and resolution, i.e. mutual agreement
between physicians and patients outside the court setting to resolve the dispute and achieve
fair compensation; caps on compensation and attorney fees, i.e. models to limit the amount of
non-economic or punitive damages that may be awarded for a case; and alternative payment
system and liabilities, i.e. strategies that reduce the burden of liability pressure and financial
burden of claims payment. The results of the review were requested by the government of
South Africa with the aim to implement policies to improve litigation in obstetrics.

Source: (R. Cardoso, unpublished data, 2017)

> 1.7.3 Stakeholders might then be facilitated by knowledge brokers,

who are increasingly focusing on rapid reviews to
As with many types of review, rapid reviews have a answer such questions (46). Other stakeholders
variety of stakeholders, the parties who will engage include research funders, who might be interested
in, benefit from or be affected by the process of in rapid reviews as a way to improve the impact
a faster review. These stakeholders must be kept of knowledge generation. Finally, rapid reviews
in mind during the rapid review process. The main might be useful to other agents whose activities
knowledge user stakeholders are the policy- and pertain to health systems strengthening, including
decision-makers who will benefit from easy access to nongovernmental and multilateral organizations,
evidence to aid a decision-making process (Box 1.5). media, patients’ associations and communities.
In fact, most often, it is requests by decision- Processes underpinning the interaction and
makers that prompt researchers and research collaboration of stakeholders who demand, conduct
institutions to generate rapid reviews addressing and use rapid reviews are explored in Chapters 5
health policy and systems questions. The process and 8 of this Guide.

BOX 1.5. Health policy and systems decision-makers

For the purpose of this Guide, we define three categories of decision-makers:

• policy-makers: individuals at some level of government or decision-making institution, including
but not limited to international organizations, non-governmental agencies or professional
associations, who have responsibility for making recommendations to others. Policy-makers
who use evidence from rapid reviews may be elected or nonelected individuals, depending
on the context;

• health systems managers: individuals in a managerial or supervisory role in a health system

with management or supervisory mandates, including implementers and public health officials;

• policy analysts: individuals (nonelected) at some level of government or decision-making

institution, responsible for analysing data and informing decisions and recommendations.

Source: Adapted from Tricco et al., 2016 (47)

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 14

> 1.7.4 Timeliness reviews also have great potential to address
emerging needs for contextualised evidence
A timeline should also be developed and agreed to inform pressing health system decisions,
upon, to ensure realistic expectations from as exemplified by the prevention and control
policy-makers or commissioners of reviews. of communicable diseases in Syria. However,
This aspect is particularly important given that low- and middle-income countries face important
the time needed to produce rapid reviews varies barriers related to the limited capacity and
greatly, and the time available may influence the resources of individuals, teams, organisations,
methods used to streamline review processes. and knowledge systems to support the production
A key challenge faced by rapid review producers and use of rapid reviews. Swiftly reviewing
is how to meet the time-sensitive needs of evidence relevant to health policy and systems
decision-makers while upholding methodological also poses challenges related to the complex
robustness and ensuring the validity of review and diverse knowledge at stake. At the same
findings. time, this complexity is a key opportunity for
developing the field of rapid reviews of health
1.8 CONCLUSION policy and systems research. The following
chapters offer guidance on addressing these
Rapid reviews are an efficient method to provide challenges and adopting methodologically sound
policy-makers and health system stakeholders approaches to conducting and using rapid reviews
with relevant and state-of-the-art evidence on in evidence-informed policy-making and health
health policy and systems challenges. Rapid system strengthening.

15 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

REFERENCES

1. Keeping Syrian children free from polio at home and across the border. Geneva, Switzerland,
World Health Organization, 2015 (http://www.who.int/features/2015/polio-immunization-syria/
en/, accessed 22 May 2017).

2. Oxman AD et al. SUPPORT Tools for evidence-informed health Policymaking (STP) 1: What is
evidence-informed policymaking? Health Research and Policy Systems, 2009, 7 Suppl 1:S1.

3. Bosch-Capblanch X et al. Guidance for evidence-informed policies about health systems: rationale
for and challenges of guidance development. PLoS Medicine, 2012, 9:e1001185.

4. WHO recommendations on antenatal care for a positive pregnancy experience. Geneva,

Switzerland, World Health Organization, 2016 (http://www.who.int/reproductivehealth/publications/
maternal_perinatal_health/anc-positive-pregnancy-experience/en/, accessed 22 May 2017).

5. Tricco AC et al. A scoping review of rapid review methods. BMC Medicine, 2015, 13:224.

6. Oliver K et al. A systematic review of barriers to and facilitators of the use of evidence by
policymakers. BMC Health Services Research, 2014, 14:2.

7. Polisena J et al. Rapid review programs to support health care and policy decision making: a
descriptive analysis of processes and methods. Systematic Reviews, 2015, 4:26.

8. Harker J, Kleijnen J. What is a rapid review? A methodological exploration of rapid reviews in Health
Technology Assessments. International journal of evidence-based healthcare, 2012, 10:397-410.

9. Oliver S, Dickson K, Bangpan M. Systematic reviews: making them policy relevant. A briefing
for policy makers and systematic reviewers. London, UK, EPPI-Centre, Social Science Research
Unit, UCL Institute of Education, University College London, 2015 (http://www.who.int/alliance-
hpsr/projects/eppibrief15.pdf, accessed 3 February 2017).

10. Tsertsvadze A et al. How to conduct systematic reviews more expeditiously? Systematic Reviews,
2015, 4:160.

11. WHO Handbook for Guideline Development, 2nd ed. Geneva, Switzerland, World Health
Organization, 2014.

12. Ismail SA et al. Communicable disease surveillance and control in the context of conflict and
mass displacement in Syria. International Journal of Infectious Diseases, 2016, 47:15-22.

13. Haby MM et al. What are the best methodologies for rapid reviews of the research evidence for
evidence-informed decision making in health policy and practice: a rapid review. Health Research
Policy and Systems, 2016, 14:83.

14. Bunting B et al. Rapid review of mental health in primary care. What aspects of primary care
are effective in the prevention, recognition and management of mental health issues across the
lifespan; for whom do they work, in what circumstances and why? Ulster, UK, University of Ulster

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 16

 and HSC Public Health Agency, (http://www.research.hscni.net/sites/default/files/Primary%20
Care.pdf, accessed 26 January 2017).

15. Moher D, Stewart L, Shekelle P. All in the Family: systematic reviews, rapid reviews, scoping
reviews, realist reviews, and more. Systematic Reviews, 2015, 4:183.

16. Hartling L et al. EPC Methods: An Exploration of Methods and Context for the Production of
Rapid Reviews [research white paper]. Rockville, (MD), US, Agency for Healthcare Research and
Quality (US), 2015 (15-EHC008-EF).

17. SURE guides for preparing and using evidence based policy briefs. Geneva, Switzerland,
SURE (Supporting the Use of Research Evidence) Collaboration, World Health Organization,
2011 (http://www.who.int/evidence/sure/guides/en/, accessed 30 January 2017).

18. Ghaffar A et al. Health policy and systems research: building momentum and community. Bulletin
of the World Health Organization, 2014, 92:851.

19. Lavis JN. How can we support the use of systematic reviews in policymaking? PLoS Medicine,
2009, 6:e1000141.

20. Uneke CJ et al. Promotion of evidence-informed health policymaking in Nigeria: bridging the gap
between researchers and policymakers. Global Public Health, 2012, 7:750-765.

21. Bermudez-Tamayo C et al. Priorities and needs for research on urban interventions targeting
vector-borne diseases: rapid review of scoping and systematic reviews. Infectious diseases of
poverty, 2016, 5:104.

22. Thomas K et al. Rapid review of primary care models and policy. Discussion paper for Wales.
Cardiff, UK, Public Health Wales, 2014 (http://www.gpone.wales.nhs.uk/sitesplus/documents/1000/
Primary%20care%20models%20and%20policies-%20rapid%20review%20July%202014%20
FINAL.pdf, accessed 2 February 2017).

23. Banbury A, Roots A, Nancarrow S. Rapid review of applications of e-health and remote monitoring
for rural residents. Australian Journal of Rural Health, 2014, 22:211-222.

24. Mijumbi RM et al. Feasibility of a rapid response mechanism to meet policymakers’ urgent needs
for research evidence about health systems in a low income country: a case study. Implementation
Science, 2014, 9:114.

25. Lal S, Adair CE. E-mental health: a rapid review of the literature. Psychiatric Services, 2014, 65:24-32.

26. McPherson E, Hedden L, Regier DA. Impact of oncologist payment method on health care
outcomes, costs, quality: a rapid review. Systematic Reviews, 2016, 5:160.

27. Hurst TE et al. Demand-side interventions for maternal care: evidence of more use, not better
outcomes. BMC Pregnancy and Childbirth, 2015, 15:297.

17 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

28. The Joanna Briggs Institute Reviewers’ Manual 2015: Methodology for JBI Scoping Reviews.
Adelaide, Australia, The Joanna Briggs Institute, 2015 (https://joannabriggs.org/assets/docs/
sumari/Reviewers-Manual_Methodology-for-JBI-Scoping-Reviews_2015_v2.pdf, accessed 27
January 2017).

29. Walt G et al. ‘Doing’ health policy analysis: methodological and conceptual reflections and
challenges. Health Policy and Planning, 2008, 23:308-317.

30. What is health policy and systems research and why does it matter? Geneva, Switzerland,
Alliance for Health Policy and Systems Research, World Health Organization, 2007 (http://www.
who.int/alliance-hpsr/resources/Alliance%20HPSR%20-%20Briefing%20Note%201.pdf?ua=1,
accessed 15 January 2017).

31. Sheikh K et al. Building the field of health policy and systems research: framing the questions.
PLoS Medicine, 2011, 8:e1001073.

32. Gilson L, ed. Health policy and systems research: a methodology reader. Geneva, Switzerland,
Alliance for Health Policy and Systems Research, World Health Organization, 2012.

33. De Savigny D, Adam T. Systems thinking for health systems strengthening. Geneva, Switzerland,
Alliance for Health Policy and Systems Research, World Health Organization, 2009 (http://apps.
who.int/iris/bitstream/10665/44204/1/9789241563895_eng.pdf?ua=1, accessed 27 January 2017).

34. Khangura S et al. Evidence summaries: the evolution of a rapid review approach. Systematic
Reviews, 2012, 1:10.

35. Tricco AC et al. An international survey and modified Delphi approach revealed numerous rapid
review methods. Journal of Clinical Epidemiology, 2016, 70:61-67.

36. Glenton C, Lewin S, Gulmezoglu AM. Expanding the evidence base for global recommendations on
health systems: strengths and challenges of the OptimizeMNH guidance process. Implementation
Science, 2016, 11:98.

37. Kastner M et al. Conceptual recommendations for selecting the most appropriate knowledge
synthesis method to answer research questions related to complex evidence. Journal of Clinical
Epidemiology, 2016, 73:43-49.

38. Pawson R, ed. Evidence-based policy: a realist perspective. London, UK, Sage Publications
Ltd., 2006.

39. Rycroft-Malone J et al. Realist synthesis: illustrating the method for implementation research.
Implementation Science, 2012, 7:33.

40. Featherstone RM et al. Advancing knowledge of rapid reviews: an analysis of results, conclusions
and recommendations from published review articles examining rapid reviews. Systematic
Reviews, 2015, 4:50.

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 18

41. Atkinson J et al. Drivers of large-scale change in complex health systems: a rapid review [executive
summary]. Ultimo, Australia, Sax Institute, 2013 (https://www.aci.health.nsw.gov.au/__data/assets/
pdf_file/0004/202396/LgeSysTransf-exec.pdf, accessed 2 February 2017).

42. Bambra C et al. Reducing health inequalities in priority public health conditions: using rapid review
to develop proposals for evidence-based policy. Journal of Public Health (Oxford), 2010, 32:496-505.

43. Fitzpatrick-Lewis D et al. Effectiveness of interventions to improve the health and housing status
of homeless people: a rapid systematic review. BMC Public Health, 2011, 11:638.

44. Caird J et al. The socioeconomic value of nursing and midwifery: a rapid systematic review
of reviews. London, UK, EPPI-Centre, Social Science Research Unit, Institute of Education,
University College London, 2010 (http://eppi.ioe.ac.uk/cms/Portals/0/Socioeconomic%20value%20
report_WEB.pdf?ver=2010-03-03-141647-487, accessed 30 January 2017).

45. Tricco AC et al. A scoping review on the conduct and reporting of scoping reviews. BMC Medical
Research Methodology, 2016, 16:15.

46. Moore G et al. Does knowledge brokering improve the quality of rapid review proposals? A before
and after study. Systematic Reviews, 2017, 6:23.

47. Tricco AC et al. Barriers and facilitators to uptake of systematic reviews by policy makers and
health care managers: a scoping review. Implementation Science, 2016, 11:4.

19 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 20
2
PERFORMING RAPID
REVIEWS

Valerie J. King, Chantelle Garritty, Adrienne Stevens,

Barbara Nussbaumer-Streit, Lisa Hartling,
Curtis S. Harrod, Jeanne-Marie Guise, Chris Kamel
KEY POINTS

• Early and continuing engagement with the research requester is essential

for focusing the rapid review and ensuring that it is appropriate to the needs
of stakeholders. The protocol serves as the starting point for the review,
although methodological decisions for rapid reviews are often iterative,
involving the stakeholder, and any changes to the protocol should also be
reflected in the final report.

• Methods can be streamlined at all stages of the review process, from

search to synthesis, by limiting the search in terms of dates and language;
limiting the number of electronic databases searched; using one reviewer
to perform study selection, risk-of-bias assessment, and data abstraction
(often with verification by another reviewer); and using a narrative synthesis
rather than a quantitative summary.

• Researchers need to make transparent methodological choices, informed

by stakeholder input, to ensure that the evidence review is fit for its
intended purpose. It is not yet clear how these choices can bias a review,
so transparency is essential.

• Information technologies can assist researchers in the conduct of rapid

reviews by making various steps in the process more efficient.
2.1 INTRODUCTION programmes, are displayed in Table 2.1, along with
key considerations for each methodological step.
Health policy-makers and other stakeholders need
evidence to inform their decisions. However, their As much as policy-makers may desire faster and
decision time frames are often short, and they more efficient information syntheses, there is a
may have other resource constraints, including need for more research to understand whether
financial ones (1-4). Rapid reviews are increasingly rapid reviews can inform policy in the same
being used and are increasingly influential in way as do systematic reviews. Only a few
the health policy and system arena (2, 5, 6). empirical studies have compared the findings
A recent needs assessment (7) showed that of rapid reviews and systematic reviews on the
policy-makers want evidence reviews to have same topic, and their results are conflicting and
the following characteristics: inconclusive, leaving questions about the level
of bias that may be introduced into the results
• responsive—answering the right question of a review with use of a rapid review method
• timely—completed in days to weeks rather (2, 8, 18, 19). At this point, the consequences
than months or years of various streamlining choices for the validity of
• credible—accurate and reproducible conclusions from a rapid review are uncertain,
so transparent documentation of the methods
To date, a standardized or commonly agreed used is critical.
upon set of methods for conducting rapid reviews
does not exist (1, 6, 8-10), unlike the situation for
systematic reviews (11, 12). However, a minimum
set of standards is being developed by the Researchers need to make
Cochrane Rapid Reviews Methods Group (1),
and other researchers have proposed methods and transparent methodological
approaches to guide rapid reviews (4, 10, 13-17). choices, informed by
This chapter gives an overview of potential stakeholder input.
ways to streamline systematic review methods
to produce a rapid review, while maintaining a
synthesis process that is sufficiently rigorous
to support health policy-making. We also Readers should also consult Chapter 4 of this
detail additional or expanded methods items Guide, which focuses on methods for rapid
gathered from the growing body of research on reviews of more complex questions in health
rapid review processes. Options for common policy and systems research.
methods choices, summarized from descriptions
and evaluations of rapid review products and

23 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

TABLE 2.1. Common methods, approaches, and key considerations for the steps in a
rapid review

Review step Commonly employed methods Key considerations

and approaches

Needs Most use standard intake processes, Work with requester to ascertain
assessment, involving the requester, to refine the intended purpose, scope and
topic selection, topic, obtain clarity on purpose(s), timeline, and ensure the proposed
and topic and determine whether rapid review approach fits the intended purpose
refinement is a suitable method
A preliminary literature search can
Total production timeline generally 1 help to inform conversations with
to 4 months requester and to scope the review
Map the mandate to timeline and
deliverables

Protocol A protocol is commonly prepared, Consider registering the protocol

development serving as a point of reference to with PROSPERO (20) and include
avoid (or document) deviations, but “rapid review” or a similar term in
is usually not formally registered the title
Producers typically use a PICO Use PRISMA (21) reporting items
format and develop key questions to guide protocol development and
iteratively with requesters review reporting, and to track the
overall process and information flow

Literature Many rapid reviews are based on Tailor the selection of literature
search searches of the PubMed/MEDLINE, databases to the topic. Addition of
Cochrane Library, and Embase a grey literature search depends on
databases the topic, purpose, and timeline
Most entail a search of two or more Use a staged search to first identify
databases, with common limits existing systematic reviews, then
being date, language (generally studies with other designs that will
English only), and study design; provide the most rigorous evidence
geographical limits may be used to to answer the question
enhance applicability
Peer review of the search strategy,
Some level of grey literature using a tool such as the PRESS
searching is common, but contact checklist can help to optimize the
with authors is uncommon search strategy (22)

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 24

 Review step Commonly employed methods Key considerations
and approaches

Screening and Approaches are highly variable, with Choose the approach for study
study selection about half of rapid reviews using screening and selection according
a single reviewer, with or without to requirements of the review and
verification by a second reviewer resources available
In lieu of dual screening and selection,
reasonable approaches involve using
a single experienced reviewer for
application of inclusion criteria and two
reviewers for application of exclusion
criteria, or using one person for
screening with verification of a subset
of records by another

Data extraction Approaches vary, but data extraction Similar to the situation for screening,
by a single reviewer, with or without the number of independent reviewers
verification, is the most common varies, but a reasonable approach is
method to use a single reviewer to extract
data, with a second reviewer checking
at least a 10% random sample of
extractions for accuracy. Use of
dual performance or checking may
be needed more for extraction of
quantitative results than for extraction
of descriptive study information
Limit extraction to key study
characteristics and outcomes

Risk-of-bias For most rapid reviews, some risk-of- The choice of appraisal instrument
assessment bias or quality assessment of included varies, with both standard and
studies is conducted by a single customized approaches in use
reviewer, with or without verification
An approach similar to that for data
extraction can be used (i.e. single
reviewer, with verification by a second
reviewer)

Knowledge Narrative summaries are common, An iterative approach to the synthesis

synthesis with meta-analysis performed only process can involve post hoc protocol
infrequently adjustments
Final reports often include The quality of the body of
implications, recommendations for evidence and the strength of any
policy, and discussion of research recommendations can be assessed
limitations using an approach such as the GRADE
system (23)
The limitations of the review
should be discussed and cautious
conclusions provided

25 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

 Review step Commonly employed methods Key considerations
and approaches

Report Peer review is common, but is often Software tools can help to automate
production and performed internally and track review steps
dissemination
Reports are often disseminated Standardization of processes and
beyond the original requester, but are templates aids in production of the
infrequently published in the peer- report and enhances transparency of
reviewed literature the review

GRADE, Grading of Recommendations Assessment, Development and Evaluation; PICO, population, intervention, comparator, outcome;
PRESS, Peer Review of Electronic Search Strategies; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses

2.2 NEEDS ASSESSMENT, requester the possibility of conducting a full

TOPIC SELECTION, systematic review, either in parallel or serially
AND TOPIC REFINEMENT with the rapid review.

Rapid reviews are typically conducted at the 2.3 PROTOCOL DEVELOPMENT

request of a particular decision-maker, who has
a key role in posing the question, setting the A research protocol clearly lays out the scope of the
parameters of the review, and defining the timeline review, including the research questions and the
(24). The most common strategy for completing approaches that will be used to conduct the review.
a rapid review within a limited time frame is to Most reviewers use the PICO format (population,
narrow its scope. This can be accomplished by intervention, comparator, outcome), with some
limiting the number of questions, interventions, adding elements for time frame, setting, and study
and outcomes considered in the review (18, 19). design. The PICO elements help to define the
Early and continuing engagement of the requester research questions, and the initial development
and any other relevant stakeholders, in order to of questions can point to needed changes in the
understand their needs, the intended use of the PICO elements. For some types of research
review, and the expected timeline and deliverables, questions or data, other framework variations
is critical (4, 14, 19, 24). Policy-makers and other may be used, although the PICO framework
requesters may have vaguely defined questions can generally be adapted. Health services and
or unrealistic expectations about what any type policy research questions may call for more
of review can accomplish. A probing needs complex frameworks and readers are referred
assessment is therefore the critical first step to Chapter 4 of this Guide for more information.
in any knowledge synthesis approach, with the This initial approach assists both researchers and
goals of determining the scope of the request knowledge users to know what is planned and
and the intended purpose for the completed enables documentation of any protocol deviations;
review, and also obtaining a commitment for however, the customized and iterative nature
collaboration over the duration of the project of rapid reviews means that some flexibility
(14, 15). Once the request and its context are may be required. Some rapid review producers
understood, researchers should fully develop the include the concept of methods adjustment in
question(s), including any needed refinement the protocol itself (25, 26). However, changes
with the requester or other stakeholders, before made beyond the protocol stage, as well as the
starting the project. This process can be iterative rationale for making them, must be transparent
and may require multiple contacts between the and documented in the final report.
reviewers and the requester to ensure that the
final rapid review is fit for its intended purpose. PROSPERO (20), the international prospective
In situations where a definitive review might register of systematic reviews, accepts registration
be needed (e.g. for a problem that is likely to of protocols that include at least one clinically or
persist), it may be useful to discuss with the patient-relevant outcome. Researchers are advised

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 26

to include “rapid review” or another similar term 2.5 SCREENING AND STUDY
in the registered title, as this will assist tracking SELECTION
the use, validity, and value of rapid reviews (1).
More generally, registration helps to decrease Methodological standards for systematic reviews
research waste and allows both requesters and generally require independent screening of
review authors to avoid duplication. Currently, citations and abstracts by at least two reviewers
most rapid review producers report using a to arrive at a set of potentially eligible references,
protocol, but few register their protocols (18). which are in turn subjected to dual review in
full-text format to arrive at a final inclusion set.
2.4 LITERATURE SEARCH Rapid reviews often streamline this process,
with up to 40% using a single reviewer at
Multiple authors have conducted inventories each stage (9, 18, 19, 27). Some rapid reviews
of the characteristics of and methods used for report verification of a sample of the articles by
rapid reviews, including the broad categories of a second reviewer or, occasionally, the use of
literature search, study selection, data extraction, two reviewers (9, 18, 27). We recommend that
and synthesis steps (9, 18, 19, 27). dual screening be used to minimize the risk of
selection bias through inappropriate exclusion
PRISMA standards call for documentation of the of relevant studies (26).
full search strategy for at least one electronic
database (21). Most published rapid reviews search 2.6 DATA EXTRACTION
two or more databases, with PubMed, Embase,
and the Cochrane Library mentioned frequently As for screening and study selection, the number
(9, 18, 27). Rapid reviews often streamline of independent reviewers who extract study data
systematic review methods by limiting the search for a rapid review can vary. The most common
by date, language, geographical area, or study approach is single-reviewer extraction (41%),
design, and some rapid reviews search only for although another 25% report verification of a
existing systematic reviews (9, 18, 19, 27). Other sample by a second reviewer and nearly as
rapid reviews use a layered searching approach, many use dual extraction (18). Data abstraction
identifying existing systematic reviews and then generally includes PICO elements, although
updating them with a summary of more recent data abstraction was often limited by the scope
eligible primary studies (9, 18, 19). Searching the of the review, and authors were contacted for
reference lists of eligible studies (sometimes missing data very infrequently (18).
known as the “snowballing” technique) and
searching the grey literature (i.e. reports that 2.7 RISK-OF-BIAS ASSESSMENT
are difficult to locate or unpublished) are done in
about half of published rapid reviews, and may Risk-of-bias assessment, sometimes called critical
be essential for certain topics (9, 18, 19, 27). appraisal or methodological quality appraisal,
However, rapid reviews seldom report contact with examines the quality of the methods employed
authors and other experts to identify additional for each included study and is a standard element
unpublished studies (9, 18, 19, 27). One study of systematic reviews (11). The vast majority of
found that peer review of the search strategy, rapid review producers perform critical appraisal
using a tool such as the PRESS (Peer Review or do it selectively (9). Similar to the situation for
of Electronic Search Strategies) checklist (22) other steps, some rapid reviews report the use
was reported in 38% of rapid reviews, but that of a single assessor with verification of a sample
it was usually performed internally rather than of study assessments by another assessor (27).
by external peer reviewers (18). There is no consensus as to which risk-of-bias
assessment tools to use, although most reviews
use study design–specific instruments intended
for assessing internal validity (9, 18). When the

27 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

purpose of the review is to scope the available 2.9 REPORT PRODUCTION
literature, rather than to evaluate specific effects, AND DISSEMINATION
this step may not be needed.
Standard templates for each stage of the review,
2.8 KNOWLEDGE SYNTHESIS from protocol development to report production
can assist the review team to perform each step
Nearly all rapid review producers conduct narrative efficiently. Use of a report template, with minimum
knowledge syntheses, but a few perform meta- methodological standards, reporting requirements,
analysis or economic analysis (9, 18). Narrative and/or standard report sections, can assist
syntheses may be limited to a basic descriptive the producer in streamlining production of
summary of studies, but should not resort to “vote the report and can also enhance transparency
counting” (or simply tallying up the number of (9, 14, 19, 24). An extension of the Preferred
studies with results that do and do not support the Reporting Items for Systematic Reviews and
intervention), an approach that can be misleading Meta-Analyses (PRISMA) statement has not yet
(12). If meta-analyses with combined estimates been created for rapid reviews, although one
of effect are not available, reviewers should be is under development and has been registered
cautious in concluding that there is a lack of effect; with the EQUATOR Network ((28); A. Stevens,
in this situation, there may simply be a lack of personal communication, 2017). Nonetheless,
evidence or a lack of statistical power to detect the PRISMA checklist can serve as a reporting
an effect. When possible, a narrative synthesis template to increase the transparency of rapid
should report the results of included studies reviews (5, 24, 29).
and should discuss the reasons for differences
among studies, such as heterogeneity of the Research about review formatting and presentation
PICO elements, study design, or methodological is now being conducted, but it is likely that
quality. Most rapid reviews present conclusions, the forms employed will need to be adapted
recommendations, or implications for policy as to the individual requester and stakeholder
another component of the synthesis, underlining audiences (26). Khangura and colleagues (14)
the role of rapid reviews in the development of present a figure showing formatted sections of
health policies (18, 19). Multiple experts also a sample report, and many other rapid review
recommend that rapid reviews should clearly producers have examples of reports online that
describe and discuss the potential limitations can serve as formatting samples. Most rapid
arising from methodological choices (6, 18, 19). review producers conduct some form of peer
The Grading of Recommendations Assessment, review for the resulting reports, but such review
Development and Evaluation (GRADE) system is often internal and may include feedback from
is used by many systematic review producers the requester (18). Most rapid review producers
to rate the certainty of the evidence about disseminate their reports beyond the requester,
health outcomes (23). Guideline developers, but dissemination varies by the sensitivity
and others who make recommendations or or proprietary nature of the product (9, 18).
policy decisions, use GRADE to rate the strength When reports are disseminated, it is common
of recommendations based on that evidence. for them to be posted online, for example, at an
Rapid review authors can also employ GRADE organizational website (9, 18). Chapter 7 of
to rate the certainty of synthesized evidence and this manual contains more information on the
develop policy implications for decision-makers. reporting and dissemination of rapid reviews.
However, the GRADE system works best for
interventions that have been subject to trials 2.10
and where there is at least one meta-analysis OPERATIONAL CONSIDERATIONS
with a single estimate of effect.
Evaluations and descriptions of research
programmes that produce rapid reviews typically

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 28

include some helpful pragmatic and operational a meta‑analysis, the knowledge user must
considerations for undertaking a rapid review be aware of the implications of streamlining
or developing a rapid review programme (4, decisions (13, 16, 19). Some programmes also
13, 14, 16, 19, 24, 30, 31). Highly experienced, emphasize the need for follow-up with review
permanent staff with the right skill mix, including requesters, both to develop the relationship and
systematic reviewers, information specialists, to continuously improve knowledge products
methodologists, and content experts (15, 19, (14, 31).
24, 27), are essential. It is time-consuming to
assemble staff on a per-project basis, so the 2.11 INFORMATION TECHNOLOGY
presence of an existing team (which may only do FOR RAPID REVIEWS
rapid reviews or may also do systematic reviews
or other research) allows projects to get off to a Another method of conducting reviews rapidly
quick start. The existence of a dedicated team involves the use of information technologies
also creates the potential to build relationships (19, 32, 33). Essentially all reviews make use of
with requesters and to cultivate mutual trust. information technology, by virtue of the electronic
Staff with experience conducting systematic database searching that is employed to locate
reviews will be familiar with standard methods relevant studies. A time-consuming step in any
and may be alert to any needed protocol changes review involves finding and screening citations.
as the review proceeds (27). The rapid review Basic reference management software can be
team must understand the methodological used for multiple tasks in a review, such as
implications of decisions taken, and must convey downloading references from a search engine,
these implications to the requesters, to allow finding full-text articles, removing duplicate
them to understand the caveats and potential references, tracking references, and documenting
limitations. Continuing relationships and longer- citations in the review. Some authors create
term contracting with requesters, to allow for custom fields to track dual reviewer selection or
a quick start and “good faith” initiation of work quality assessment. However, other technologies
before a contract is in place, can speed the early can help to make the review process even more
development stages (16, 24). It is important efficient. While not an exhaustive inventory,
for rapid review producers to confirm that the the remainder of this section outlines the various
choices they make to streamline the review types of technologies, where they fit in the
are acceptable to the requester. Whether it review production process, and how readers
is a decision to limit the scope to a single can find more information about them. Table 2.2
intervention or outcome, restrict the literature summarizes common software tools for tasks
search to existing systematic reviews, or forgo such as screening, data extraction, and synthesis.

TABLE 2.2. Information technology for rapid reviews

Product name (cost) Characteristics

Abstrackr, OpenMeta[Analyst] (34) Suite of products on the website of the Brown University
School of Public Health
(open source, freely available)
Abstrackr is a semi-automated citation screening software
program
OpenMeta[Analyst] is software for performing meta-
analysis of continuous, binary, or diagnostic test accuracy
data

29 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

 Product name (cost) Characteristics

Covidence (35) Primary screening and data-extraction tool for Cochrane

authors
(first review free; subscription
required for subsequent reviews) Full text can be highlighted and linked to prepare a risk-of-
bias table
Data can be exported into various analytic packages

DistillerSR (36) Tool for citation import and tracking for inclusion and
exclusion
(purchase of licence required)
Customizable data-extraction tables
Data can be exported into various analytic packages

EPPI-Reviewer (37) Supports development of all types of systematic reviews,

including complex reviews
(available to Cochrane authors
free of charge; subscription fee for Includes reference management, screening, data
others) extraction, and risk-of-bias assessment
Contains quantitative and qualitative analysis functions.
Allows coding of text and generation of keywords

GRADEpro GDT (38) Software for generating evidence profiles and summary-
of-findings tables for systematic reviews and supporting
(freely available)
development of guideline recommendations

Rayyan (39) Software for semi-automated screening titles and abstracts

(freely available, web-based,
including mobile applications)

Review Manager (RevMan) (40) Contains Cochrane review template, including tables of
study characteristics, comparisons, charts for risk-of-bias
(purchase of licence required for
assessment, and templates for graphical display of results
non-Cochrane review use)
Integrates meta-analysis software

System for the Unified Suite of modules for systematic reviews produced by the
Management, Assessment and Joanna Briggs Institute and available to systematic review
Review of Information (SUMARI) researchers. Includes tools for data extraction and critical
(41) appraisal for multiple study designs
(free, but registration required) Can import and manage citations

The SR ToolBox is a searchable, web-based Tsafnat and colleagues (32) reviewed the research
catalogue of systematic review support tools tasks involved in systematic reviews and the
(42). Researchers can search for tools that use, automation potential of each step, along with
for example, text-mining or machine-learning to research gaps. They identified the potential for
support various stages of a review. The catalogue automation to improve the speed and accuracy
is searchable for tools that are available for free of several review steps. Of note, automated
or for purchase (33, 43). full-text screening (based on machine-learning
algorithms) and some level of automated data

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 30

extraction are currently possible and improving 2.12 SUGGESTED APPROACHES TO
rapidly (32, 33). However, as is the case for rapid RAPID REVIEWS
reviews themselves, there is little empirical
evidence about the implications of innovative The previous sections have summarized the
technologies for review validity (33). For example, numerous approaches to conducting rapid
some steps, like study screening, may be easier reviews and the programmes that use them.
to automate than the more nuanced decisions Similarly, Abrami and colleagues (13) summarized
involved in assessing risk of bias. Tsertsvadze several methods of conducting rapid reviews and
and colleagues (33) recognized the application developed a brief review checklist of considerations
of innovative technologies, including machine- and recommendations, which may serve as a
learning approaches, as having the potential to useful parallel to Table 2.1 in this chapter. Boxes
speed the review process and reduce costs, 2.1 and 2.2 present some practical examples from
but at some risk of increasing the risk of bias. research centres in Lebanon and Canada that
These are the same benefits and risks associated perform rapid reviews and how they are working
with streamlining systematic review methods, to support policy decisions in their regions.
and both may be heightened with a combination
of these two approaches (33).

BOX 2.1. The Center for Systematic Reviews on Health Policy and Systems Research (SPARK)
rapid review programme to support health policy in the Eastern Mediterranean region

The Center for Systematic Reviews on Health Policy and Systems Research (SPARK) was
established in 2013 at the American University of Beirut in Lebanon through funding from the
Alliance for Health Policy and Systems Research at the World Health Organization (WHO).
The Center produces systematic reviews responding to health policy and systems priorities,
and builds capacity to conduct systematic reviews at the individual, team, institutional,
and national levels. It also developed and conducted initial validation of the SPARK tool for
prioritizing questions for systematic reviews in health policy and systems research, along
with a user manual.

Building on these successes, the Center further established a rapid response service to
address requests from health policy-makers and stakeholders at the national and Eastern
Mediterranean region levels. The service builds on processes for managing demand,
conducting rapid reviews, and delivering rapid response products. It also takes advantage
of its close collaboration with the Knowledge to Policy (K2P) Center, a WHO Collaborating
Center for Evidence-Informed Policy and Practice. SPARK has collaborated with K2P to rapidly
inform key policy decisions in Lebanon, such as those related to antibiotic resistance and to
implementing the salt fluoridation and iodization law.

The SPARK Center and K2P were invited to support and contribute to the Lancet-American
University of Beirut Commission on Syria: Health in Conflict (44). The Commission aims to
raise the profile of the Syrian crisis in global health and to mobilize a stronger international
response through its work. SPARK has already conducted a rapid scoping review to inform
the Commission’s first policy paper, addressing policies to protect and support health-care
workers in the setting of armed conflict zones (45). The centres will build both experience
and expertise as they respond to the requests of policy-makers in Lebanon and the region.

31 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

BOX 2.2. Rapid production of evidence summaries at the Ottawa Hospital Research Institute
using an 8-step approach

The Knowledge Synthesis Group (KSG) at the Ottawa Hospital Research Institute is an
academic group that works closely with a variety of decision-makers to provide timely,
evidence-based answers to help direct policy, implementation, and practice decisions. In 2012,
the KSG outlined a formal approach to conducting rapid evidence summaries (14), a process
that emphasizes incorporating “off-the-shelf” evidence such as existing systematic reviews,
and then including primary studies if warranted. This rapid review approach evolved iteratively
over time, and is based upon widely accepted systematic review standards. Khangura and
colleagues described the development of the KSG’s approach as they produced 11 evidence
summaries (14), including one on the timing of elective repeat Cesarean birth.

Importantly, the KSG approach involves continuing engagement with decision-makers to

ensure that the project scope is defined appropriately, which in turn ensures that the research
questions posed will generate useful answers. Therefore, the KSG engages with end-users
from the beginning, making certain that they have a clear understanding of the rapid review
process, managing their expectations, and conveying the limitations of this approach. Built
into this process is an internal assessment as to the suitability of the rapid review approach
for each question under consideration. Further topic refinement and protocol development
is undertaken using a PICO (population, intervention, comparator, outcome) framework to
focus the review on what will meet the requester’s needs and to ensure that the project will
be manageable within a condensed time frame of 4–16 weeks. End-users are also engaged
throughout the process of conducting the rapid review to answer questions and to be involved
in decision-making, should post hoc changes be needed in light of the nature and volume of
evidence. To accommodate the information needs of various requesters, the KSG employs a
rapid review approach that is tailored across the various stages of conducting the review to
best meet the specific needs of the end-users (25). For example, the KSG researchers may
conduct a meta-analysis if it is needed, or they may elect to consider all published literature,
regardless of study design, for narrow questions where there is a lack of high-quality studies.

A “one-size-fits-all” approach may not be suitable literature and those involving complex subjects
to cover the variety of topics and requester needs (4). However, rapid review producers should
put forward. Watt and colleagues (6) observed remain aware that streamlined methods may
nearly a decade ago that “It may not be possible to not be appropriate for all questions, settings,
validate methodological strategies for conducting or stakeholder needs, and they should be honest
rapid reviews and apply them to every subject. with requesters about what can and cannot be
Rather, each topic must be evaluated by thorough accomplished within the timelines and resources
scoping, and appropriate methodology defined”. available (16). For example, a rapid review would
Thomas, Newman, & Oliver (4) noted that it may likely be inappropriate as the foundation for a
be more difficult to apply rapid approaches to national guideline on cancer treatment due to
questions of social policy than to technology be launched three years in the future.
assessment, in part because of the complexity
of the topics, underlying studies, and uses of Tricco and colleagues (10) conducted an international
these reviews. The application of mixed methods, survey of rapid review producers, using a modified
such as key informant interviews, stakeholder Delphi ranking to solicit opinions about the
surveys, primary data, and policy analysis, may be feasibility, timeliness, comprehensiveness,
required for questions with a paucity of published and risk of bias of six different rapid review

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 32

approaches. Ranked best in terms of both risk various approaches ranked in this survey. Although
of bias and feasibility was Approach 1, which the survey report was based on expert opinion
included published literature only, based on a (10), it did not provide empirical evidence about
search of one or more electronic databases, the implications of each streamlined approach.
limited in terms of both date and language. However, in the absence of empirical evidence,
With this approach, study screening is conducted it may serve as a resource for rapid review
by a single reviewer, while both data extraction producers looking to optimize one of these
and risk-of-bias assessment involve a single review characteristics.
reviewer with verification by a second reviewer.
Other approaches were ranked best in terms Given that empirical evidence regarding the
of timeliness and comprehensiveness (10), implications of methodological decisions for rapid
representing trade-offs that review producers reviews is not yet available, we have developed
and knowledge users may want to consider. interim guidance for those conducting rapid
Table 3.1 in Chapter 3 gives details about the reviews (Box 2.3).

BOX 2.3. Interim guidance for the conduct of rapid reviews

• Engage with the review requester early and throughout the review process to understand
needs and expectations, and collaborate with the requester in making decisions about
how to approach the review.

• Use a team experienced in doing systematic reviews to conduct the rapid review.

• Develop a protocol, including PICO (population, intervention, comparator, and outcome)

elements, key questions, and the planned approach, to guide the review and to track any
changes that are made as the review progresses (and their rationale). Protocol registration
is strongly encouraged.

• Search at least two electronic databases for most topics; use a targeted grey literature
search if the topic is not well addressed in published articles.

• If timeline and resources allow, use two reviewers for study selection.

• Perform data extraction and risk-of-bias assessment using one researcher; if time and
resources allow, a sample of articles should be checked by a second one.

• Consider the use of innovative technologies that can help to make particular review steps
more efficient.

• In conducting the knowledge synthesis, include both a typical results component (with
description of included studies, their results, reasons for any differences in results across
studies, and the quality of the evidence from those studies, perhaps with GRADE (Grading
of Recommendations Assessment, Development and Evaluation) rating for the overall quality
of evidence) and a discussion component describing limitations of the evidence and the
review, overall conclusions, recommendations, and implications for policy- and decision-
makers.

• When possible, obtain peer review, and use feedback from the requester and other
stakeholders to inform and improve future knowledge synthesis.

• Consult with the requester about the best report format and presentation that will support
the use of the review and subsequent decision-making.

33 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

2.13 CONCLUSION policy-making. Interim guidance suggestions
are outlined in Box 2.3. The keys to success
This chapter has summarized the rapid review are early and continuing engagement, careful
methods that can be used to balance timeliness streamlining of decisions for each review step,
and resource constraints with a rigorous and transparency of these decisions through a
knowledge synthesis process to inform health clearly written protocol and report.

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 34

REFERENCES

1. Garritty C et al. Cochrane Rapid Reviews Methods Group to play a leading role in guiding the
production of informed high-quality, timely research evidence syntheses. Systematic Reviews,
2016, 5:184.

2. Hite J, Gluck ME. Rapid Evidence Reviews for Health Policy and Practice.
AcademyHealth, 2016 (http://www.academyhealth.org/files/publications/files/publications/
AH%255FRapid%2520Evidence%2520Reviews%2520Brief.pdf, accessed 20 May 2017).

3. Peterson K et al. User survey finds rapid evidence reviews increased uptake of evidence by
Veterans Health Administration leadership to inform fast-paced health-system decision-making.
Systematic Reviews, 2016, 5:132.

4. Thomas J, Newman M, Oliver S. Rapid evidence assessments of research to inform social

policy: taking stock and moving forward. Evidence & Policy: A Journal of Research, Debate and
Practice, 2013, 9:5-27.

5. Moore GM et al. Rapid reviews in health policy: a study of intended use in the New South Wales’
Evidence Check programme. Evidence & Policy: A Journal of Research, Debate and Practice,
2016, 12:505-519.

6. Watt A et al. Rapid reviews versus full systematic reviews: an inventory of current methods and
practice in health technology assessment. International Journal of Technology Assessment in
Health Care, 2008, 24:133-139.

7. Gluck M, ed. Can Evidence Reviews Be Made More Responsive to Policymakers? Fourth Global
Symposium on Health Systems Research: Resiliant and Responsive Health Systems for a Changing
World; 2016; Vancouver, (BC), Canada.

8. Haby MM et al. What are the best methodologies for rapid reviews of the research evidence for
evidence-informed decision making in health policy and practice: a rapid review. Health Research
Policy and Systems, 2016, 14:83.

9. Polisena J et al. Rapid review programs to support health care and policy decision making: a
descriptive analysis of processes and methods. Systematic Reviews, 2015, 4:26.

10. Tricco AC et al. An international survey and modified Delphi approach revealed numerous rapid
review methods. Journal of Clinical Epidemiology, 2016, 70:61-67.

11. Higgins JPT et al. Methodological Expectations of Cochrane Intervention Reviews. London, UK,
Cochrane, 2016 (http://methods.cochrane.org/mecir, accessed 20 May 2017).

12. Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions Version
5.1.0 [updated March 2011], The Cochrane Collaboration, 2011.

13. Abrami PC et al. Issues in conducting and disseminating brief reviews of evidence. Evidence &
Policy: A Journal of Research, Debate and Practice, 2010, 6:371-389.

35 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

14. Khangura S et al. Evidence summaries: the evolution of a rapid review approach. Systematic
Reviews, 2012, 1:10.

15. Varker T et al. Rapid evidence assessment: increasing the transparency of an emerging methodology.
Journal of Evaluation in Clinical Practice, 2015, 21:1199-1204.

16. Wilson MG, Lavis JN, Gauvin FP. Developing a rapid-response program for health system
decision-makers in Canada: findings from an issue brief and stakeholder dialogue. Systematic
Reviews, 2015, 4:25.

17. Featherstone RM et al. Advancing knowledge of rapid reviews: an analysis of results, conclusions
and recommendations from published review articles examining rapid reviews. Systematic
Reviews, 2015, 4:50.

18. Abou-Setta AM et al. Methods for Developing Evidence Reviews in Short Periods of Time: A
Scoping Review. PloS One, 2016, 11:e0165903.

19. Hartling L et al. A taxonomy of rapid reviews links report types and methods to specific decision-
making contexts. Journal of Clinical Epidemiology, 2015, 68:1451-1462.e1453.

20. PROSPERO: International prospective register of systematic reviews. York, UK, Centre for
Reviews and Dissemination, University of York, (https://www.crd.york.ac.uk/PROSPERO/#index.
php, accessed 24 May 2017).

21. Moher D et al. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The
PRISMA Statement. PLoS Medicine, 2009, 6:e1000097.

22. McGowan J et al. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement.
Journal of Clinical Epidemiology, 2016, 75:40-46.

23. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working
Group. 2000 (http://www.gradeworkinggroup.org/, accessed 24 May 2017).

24. Haby MM et al. Designing a rapid response program to support evidence-informed decision-
making in the Americas region: using the best available evidence and case studies. Implementation
Science, 2016, 11:117.

25. Garritty C, Stevens A, eds. Putting Evidence into Practice (PEP) workshop – Rapid Review Course.
2015; Edmonton, Alberta.

26. Garritty C et al., eds. Rapid Review Workshop: Timely Evidence Synthesis for Decision Makers.
Cochrane Colloquium; 2016; Seoul, South Korea.

27. Tricco AC et al. A scoping review of rapid review methods. BMC Medicine, 2015, 13:224.

28. The EQUATOR Network. Oxford, UK, UK EQUATOR Centre, (http://www.equator-network.org/,

accessed 13 June 2017).

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 36

29. Kelly SE, Moher D, Clifford TJ. Quality of conduct and reporting in rapid reviews: an exploration
of compliance with PRISMA and AMSTAR guidelines. Systematic Reviews, 2016, 5:79.

30. McIntosh HM et al. The Healthcare Improvement Scotland evidence note rapid review process:
providing timely, reliable evidence to inform imperative decisions on healthcare. International
journal of evidence-based healthcare, 2016, 14:95-101.

31. Gibson M et al. Methods and processes to select and prioritize research topics and report
design in a public health insurance programme (Medicaid) in the USA. Cochrane Methods, 2015,
Suppl 1:33-35.

32. Tsafnat G et al. Systematic review automation technologies. Systematic Reviews, 2014, 3:74.

33. Tsertsvadze A et al. How to conduct systematic reviews more expeditiously? Systematic Reviews,
2015, 4:160.

34. Center for Evidence Synthesis in Health. Providence, (RI), US, Brown University, (https://www.
brown.edu/academics/public-health/research/evidence-synthesis-in-health/research-initiatives/
software-0, accessed 13 June 2017).

35. Covidence. Melbourne, Australia, The Alfred Hospital, (https://www.covidence.org/, accessed

13 June 2017).

36. DistillerSR. Ottawa, (ON), Canada, Evidence Partners, (https://www.evidencepartners.com/

products/distillersr-systematic-review-software/, accessed 13 June 2017).

37. EPPI-Reviewer. London, UK, Evidence for Policy and Practice Information and Co-ordinating
Centre (EPPI-Centre), (http://eppi.ioe.ac.uk/cms/Default.aspx?alias=eppi.ioe.ac.uk/cms/er4,
accessed 13 June 2017).

38. GRADEpro GDT. McMaster University, 2015 (https://gradepro.org/, accessed 13 June 2017).

39. Rayyan. Doha, Qatar, Qatar Computing Research Institute (Data Analytics), (https://rayyan.qcri.
org/, accessed 13 June 2017).

40. Review Manager 5 (RevMan 5). Copenhagen, Denmark, The Nordic Cochrane Centre, The Cochrane
Collaboration, 2014 (http://community.cochrane.org/tools/review-production-tools/revman-5,
accessed 13 June 2017).

41. JBI System for the Unified Management, Assessment and Review of Information (SUMARI).
Adelaide, Australia, The Joanna Briggs Institute, (http://joannabriggs.org/sumari.html, accessed
13 June 2017).

42. Systematic Review Toolbox. York, UK, York Health Economics Consortium. University of York,
(http://systematicreviewtools.com/, accessed 13 June 2017).

43. Marshall CG, J. Software tools to support systematic reviews. Cochrane Methods, 2016:34-35.

37 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

44. Jabbour S et al. The Lancet-American University of Beirut Commission on Syria: a new role for
global health in conflict and a call for papers. Lancet, 2016, 388:2967-2968.

45. Fouad FM et al. Health workers and the weaponisation of health care in Syria: a preliminary
inquiry for The Lancet-American University of Beirut Commission on Syria. Lancet, 2017.

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 38

3
IMPROVING QUALITY AND
EFFICIENCY IN SELECTING,
ABSTRACTING, AND
APPRAISING STUDIES
FOR RAPID REVIEWS
Ba’ Pham, Reid C. Robson, Sonia M. Thomas,
Jeremiah Hwee, Matthew J. Page, Andrea C. Tricco
KEY POINTS

• A consensus-based approach to rapid review conduct is highlighted,

including streamlined methods for literature search (i.e. search more than
one database for published studies only, use date and language search
limits where appropriate), study selection (i.e. conducted by one reviewer),
data abstraction (i.e. one reviewer abstracts, another verifies), and quality
assessment (i.e. one reviewer assesses, another verifies). The evidence-
base supporting streamlined methods is limited and evolving, and we need
further evidence to define robust approaches.

• Rapid review teams should consider including content experts (e.g. in

health policy and systems research) and experienced reviewers (e.g. in
study selection, data abstraction, and quality assessment) to increase
review rigour and expedite the review process.

• Eligibility criteria should be well-defined; stated using clear, unambiguous

language; and applied consistently.

• Screening, abstracting, and assessing forms, including explanation and

elaboration documents that define concepts and terms, ideally with
examples, should be used to support reviewers in study selection, data
abstraction, and quality assessment.

• Procedures and material should be pilot-tested by the team prior to conducting

study selection, data abstraction, and quality assessment.

• Training should be provided to all reviewers at the beginning of the review

and during the review to deal with issues that need to be reiterated for
consistency purposes.

• Authors of the studies included in the rapid review should be consulted to

gather further information on methods conduct, if time allows.
3.1 INTRODUCTION comprehensiveness, and risk of bias (3). As can
be seen, typical approaches involve abbreviation
Deciding which methods to streamline when
conducting a review rapidly is challenging for several
reasons. First, typical systematic reviews can
provide valid and reliable results when conducted
We suggest careful consideration
according to standard guidance (1), but deviations
from the standard may leave the review open to which steps are streamlined,
to bias and errors. Box 3.1 illustrates standard
factors affecting streamlined
methods and best practices that contribute to
the validity and reliability of systematic review decisions, and potential
findings, notwithstanding considerations for
consequences in terms of
time, resources, and costs. Second, alterations
of or deviations from standard systematic review validity of the review results and
methods can be made at multiple points in the
efficiency of the review process.
review process, leading to numerous rapid review
approaches. However, few of these deviations are
used consistently in the literature, and the methods
for many rapid reviews are poorly reported (2);
thus, there is currently no single best approach. of any or all methods for literature search, study
selection, data abstraction, and quality assessment.
Table 3.1 displays rapid review approaches identified Only limited data are available to inform trade-offs
by an international survey of diverse stakeholders, in streamlined methods and the downstream
ordered according to preferences in trade-off consequences on review findings.
considerations between feasibility, timeliness,

BOX 3.1. Generally accepted standards for study selection, data abstraction, and quality
assessment for systematic reviews

• Use two or more reviewers, working independently, to screen and select studies (4-6). Define
in advance the process for resolving discrepancies (5).

• Train screeners using written documentation (4). Test and retest screeners to improve
accuracy and consistency in study selection (4).

• Use two or more reviewers, working independently, to extract quantitative and other critical
data from each study (4-6). For other types of data, one reviewer could extract the data,
and the second reviewer could then independently check for accuracy and completeness
(4-6). Define in advance the process for resolving discrepancies (4, 5).

• Use two or more people, working independently, to apply the risk-of-bias or quality assessment
tool to each included study. Define in advance the process for resolving disagreements (5-7).
Pilot the risk-of-bias or quality assessment tool (6). If resources are limited, priority should
be given to assessment of the key sources of bias (6).

41 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

TABLE 3.1. Consensus-rankinga of rapid review approaches relative to systematic review
approach

Rapid review Comprehen- Quality

approach Feasibility Timeliness siveness Assessment

APPROACH 1:
• Literature search: >1
database, published only
• Search limit: both date and
language
• Study selection:
one reviewer
• Data abstraction: one person
abstracts, other verifies
• Risk-of-bias assessment:
one person assesses, other
verifies

APPROACH 2:
• Literature search: updating
the literature search of a
previous review, published
only
• Search limit: none
• Study selection:
one reviewer
• Data abstraction:
one reviewer
• Risk-of-bias assessment:
not performed

APPROACH 3:
• Literature search: >1
database, grey literature
• Search limit: both date and
language
• Study selection:
one reviewer
• Data abstraction:
one reviewer
• Risk-of-bias assessment:
not performed

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 42

 Rapid review Comprehen- Quality
approach Feasibility Timeliness siveness Assessment

APPROACH 4:
Literature search: >1 database,
grey literature
Search limit: either date or
language
Study selection: one reviewer
Data abstraction: one reviewer
Risk-of-bias assessment:
not performed

APPROACH 5:
Literature search: >1 database,
grey literature
Search limit: date
Study selection: one reviewer
Data abstraction: one reviewer
Risk-of-bias assessment:
one reviewer

APPROACH 6:
Literature search: >1 database,
grey literature
Search limit: both date and
language
Study selection:
two independent reviewers
Data abstraction: one reviewer
Risk-of-bias assessment:
not performed

a
More smiley faces indicate stronger preference (higher ranking) by the Delphi panel of international stakeholders. Colours are used to
emphasize the number of smiley faces but otherwise, have no differential values.

Source: This study included an international survey of rapid review producers and modified Delphi to solicit experiences with and perceptions
of rapid reviews from stakeholders, including researchers, policy-makers, industry, journal editors, and health-care providers. Results were
derived from 40 rapid review producers responding to the survey and 113 stakeholders participated in the Delphi panel. Adapted with
permission from Tricco et al., 2016 (3)

In this chapter, we present methods for selecting, results and conclusions, as well as the related
abstracting, and assessing studies for rapid workload (A. Tricco, unpublished data, 2016).
reviews of health policy and systems interventions Note that methods for limiting the literature search
(Box .2). We also present factors affecting validity are not discussed here as they often depend on
and reliability (Box 3.2). Finally, we describe the review topics.
impact of using streamlined methods on the

43 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

BOX 3.2. Evidence supporting decisions regarding streamlined methods for rapid reviews

Study selection
• Single-reviewer screening of titles/abstracts missed on average 8%–20% of eligible studies
(8, 9) but substantially reduced screening time (by about 60%), relative to screening by two
reviewers (10).

• Decisions regarding study selection should not be based on study results, and eligibility
criteria should be well defined; stated with clear, unambiguous language; and applied
consistently (11, 12).

• Errors and inconsistencies in application of study eligibility criteria are common (13).

• The review team should include members with different levels of content expertise and should
include members with study screening experience to expedite screening of titles/abstracts
(8, 13, 14). A lack of relevant information in titles/abstracts contributes to discrepancies
between screeners (15, 16).

Data abstraction
• Compared with dual data abstraction, single abstraction with verification resulted in more
errors (a relative increase of 22%) but saved time (a relative saving of 36%) (17). However,
the errors did not cause major changes in the effect estimates.

• Variation in the reporting of study results (especially variation due to selective reporting)
commonly contributes to discrepancies in data abstraction (18, 19).

• Use of experienced abstractors can expedite the process (20, 21).

• In general, continuous outcome data involving specific summary measures such as means
and standard deviations are prone to errors during data abstraction, with the potential to
significantly alter the overall impression of the effectiveness of clinical, health policy and
systems interventions (22, 23).

Study quality assessment

• Quality assessment can be influenced by characteristics of the included studies (24, 25).
Training and piloting of assessment forms, explanation/elaboration documents, and related
material specific to each review is important.

• The “unclear” option in quality assessment is often applied because of inadequate reporting
of methodological details in study reports. Reviewers may be able to reduce this frequency
by supplementing published reports with information collected directly from study authors
(26, 27).

• Discrepancies in quality assessment are mainly due to differing interpretation of assessment

tools, rather than to differing information found in the study reports (24).

• Discrepancies in quality assessment are common, regardless of the level of experience of

the quality appraisers (28).

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 44

3.2 STUDY SELECTION challenging than reviews of other topics, such as
biomedical interventions.
> 3.2.1 Methods for study selection
The first five rapid review approaches listed in
Once articles have been identified from a literature Table 3.1 involve the use of one reviewer in study
search, reviewers must decide which of these selection, instead of two.This substantially reduces
studies meet the eligibility criteria. Such decisions screening time and resources, but increases the
may involve judgement, prior opinion, subjectivity, chance of missing eligible studies. For example,
inconsistency, and random errors (11). Currently, single screening of titles/abstracts missed on
the most effective means of reducing discrepancies average 8%–20% more eligible studies relative to
in study selection is to have two or more reviewers, dual screening, but the impact of the missed eligible
working independently, to select studies for studies on the review results and conclusions
inclusion (Box 3.1). Reviewer disagreement is were unclear (8, 9). Even systematic reviews that
common in title/abstract screening (10%–20%) were conducted using rigorous methods (e.g.
(14) and the use of two or more reviewers using two reviewers) ended up missing some
here, and during full-text screening, provides an eligible studies, although the validity of the review
opportunity to discuss the reasons for disagreement findings was not compromised, suggesting that it
and clarify the discrepancy. This is especially is essential to capture the key studies, with some
relevant to reviews on health policy and systems tolerance for a low frequency of missed eligible
research, which are generally more complex and studies, as outlined in Box 3.3.

BOX 3.3. Reproducibility of systematic reviews

With one exception, no prospective studies have evaluated the reproducibility of systematic
reviews. One study assessed the reproducibility of systematic reviews evaluating the association
between endometrial cancer and food, nutrition, and physical activity (1). The reviews were
conducted independently by two centres, on two continents, with the same instructions
(e.g. peer-reviewed protocol, using two reviewers) and the same resources. The two reviews
identified a total of 310 relevant studies. Due to a combination of reasons (related to eligibility
criteria, literature searches, and study selection), the first review missed 12% of relevant studies,
and the second review missed 34%. However, despite differences in study inclusion, the overall
conclusions were comparable. The results suggest that rapid reviews might be completed
with evidence from key studies rather than with exhaustive evidence from all studies, which
consumes substantially more time and resources.

> 3.2.2 Factors affecting the accuracy health policy and systems research is generally
and reliability of study selection challenging because the criteria often involves
the specification of real-world populations of
Study eligibility criteria should be well-defined; stated patients with multiple comorbidity, complex
using clear, unambiguous language; and applied interventions with multiple components, contextual
consistently (11, 12). Mistakes in assessing study factors affecting the intervention implementation,
eligibility can lead to inclusion of ineligible studies and multiple outcomes. An explanation and
and exclusion of eligible studies (11). This issue is elaboration document can help support reviewers
more common than expected; indeed, one estimate with study selection, especially with respect to
suggests that 10% of systematic reviews contain defining concepts and terms used in questions
discrepancies relative to the truly eligible studies related to the eligibility criteria, and to illustrate
(12). Defining the eligibility criteria for reviews in them with examples. Training should be provided

45 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

to all reviewers at the beginning of the review conclusions), or that are missing information
and during the review to deal with issues that required for eligibility assessment (16). These
need to be re-iterated for consistency purposes. challenges suggest that if screening is conducted
A decision log is useful to document key decisions by one reviewer, the review team should pilot
made during study selection (and other steps of screening forms and related explanation and
the review process) so that the same criteria (or elaboration documents. This may help identify
rules and nuances in their interpretation) are applied potential discrepancies related to lack of information
consistently. Finally, decisions regarding study in titles/abstracts and inform how to handle these
selection should not be based on study results; decisions consistently during screening.
in particular, studies should not be excluded solely
because they have negative findings (11, 12). Various tools exist to facilitate screening, record
the flow of records during the review, and allow
The composition of the review team can affect the multiple reviewers to simultaneously screen
accuracy and reliability of study selection. Rapid the same set of studies. For example, a web-
review teams with content experts (e.g. in health based platform facilitated workflow and improved
policy and systems research) and experienced screening accuracy relative to a paper-based
reviewers (e.g. in screening, data abstraction, tool and a tool based on reference management
and methodological quality appraisal) can increase software (29). However, the average screening
review rigour and expedite the review process (8, times were not considerably different between the
13, 14). Graduate students may screen titles and three modalities (29). Alternatively, the development
abstracts more consistently than content experts of text-mining methods for study selection is
(14). However, content experts may be better able active and promising. Text-mining tools that
to discern truly irrelevant abstracts quickly (13). prioritize the order in which potentially relevant
Although the least experienced screener may records are screened are considered safe for
produce more consistent selection than more use in “live” reviews (30). This, for example,
experienced screeners, the average screening involves using an initial set of screened records
time is substantially longer for inexperienced to develop a regression model (e.g. based upon
screeners (8). For reviews of health policy and a bag of keywords) to predict the probability of
systems research, including content experts can a record being the eligible one, and order the
expedite screening tasks if they are experienced records according to their eligibility potential for
reviewers. Conversely, including policy-makers as priority screening. In addition, progress has been
members on the rapid review team may increase made in the use of text-mining tools as a “second
the amount of time required for screening if they screener” (30). Interested readers are referred to
are inexperienced reviewers, but may help to excellent sources available elsewhere (10, 30).
identify truly irrelevant abstracts more quickly.
Importantly, policy-makers and content experts 3.3 DATA ABSTRACTION
contribute multiple perspectives to the review
team as well as to the review. > 3.3.1 Methods for data abstraction

Lack of relevant information in titles/abstracts In Table 3.1, the consensus-based Approach 1 for
can also affect accuracy and reliability of study rapid reviews uses one reviewer for data abstraction
selection (15, 16). For example, screening only and another to verify the data abstracted (i.e. single
article titles resulted in a slightly higher number abstraction with verification). Compared with dual
of discrepancies compared with screening both abstraction (i.e. conducted by two reviewers,
titles and abstracts (although this did not lead working independently), single abstraction with
to greater exclusion of eligible studies) (15). It is verification saved time (a relative saving of 36%),
also challenging to screen citations with titles but resulted in more errors (a relative increase of
only, or with abstracts that are not structured 22%) (17). However, the errors did not substantially
(e.g. sections for background, methods, results, affect the conclusion of the review (17). For rapid

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 46

reviews of health policy and systems research more time should be allocated for data abstraction
that do not consider meta-analysis to synthesize for these reviews.
outcome data, the use of a single abstractor could
be reasonable, with a verifier for the data that are Data abstraction errors (i.e. incorrect abstraction)
important to decision-makers (4). Dual abstraction may be more common than expected; one estimate
remains essential for abstracting data that are suggested that errors were present in 20 of
used in quantitative or mixed methods syntheses, 34 reviews (although the errors did not affect
such as data related to the effectiveness of the conclusions) (21). Discrepancies can occur
interventions (31). Computer-assisted single regardless of the level of experience of the data
abstraction with verification has been proposed abstractors. Experienced abstractors generally
and is undergoing evaluation in a randomized take less time to complete data abstraction;
controlled trial against dual abstraction and single in particular, abstractors experienced in the content
abstraction with verification (32). area of the review (e.g. researchers or policy
analysts working for the commissioning institution)
> 3.3.2 Factors affecting the accuracy can expedite data abstraction. The review team
and reliability of data abstraction might also consider including members with less
experience, such as graduate students, to improve
Reviewers should be aware that selective reporting the capacity to meet tight deadlines. However,
of results in one or more of the included studies abstracted data from experienced abstractors
may affect data abstraction (18). For example, are likely more consistent than respective data
researchers might have measured various abstracted by graduate students (23).
outcomes in the study, but reported only those
that were statistically significant. Selective reporting Continuous data, which are summarized in the
could also occur when there is the possibility of form of means and standard deviations, are more
selecting from multiple effectiveness measures, prone to abstraction errors than dichotomous
multiple time points when the outcomes were data, which are summarized as categories (e.g.
measured, and multiple analyses using different use or non-use of health services). Such errors can
statistical methods, among others. In the absence substantially affect assessment of the intervention
of clear guidance, the abstractor could selectively effects, especially when the study data are
or arbitrarily include outcome measures when there synthesized quantitatively (23, 34). Care should
are multiple options. Reviewers should pre-specify be exercised in handling this type of data, even if
the required data items in advance, and engage the data are simply displayed in summary tables,
stakeholders, including clinicians, decision-makers, for example, to avoid simple but serious errors
content experts, patients, and caregivers to help such as mistakenly displaying the means and
prioritize which outcomes are most important standard deviations of the intervention group for the
and relevant (33). Reviewers should also use data control group and vice versa (34). The review team
abstraction forms and supporting explanation and should consider dual abstraction for continuous
elaboration documents to ensure consistency in data, with involvement of or consultation with
handling variation in the reporting and selection experienced review methodologists. This approach
of study results. is particularly suitable for rapid reviews of health
policy and systems research, which can include
The team should also consider training of important continuous outcomes, such as patient
reviewers and piloting of data abstraction forms and policy-maker preferences, functional measures,
and supporting documents. It should be anticipated health-related quality-of-life measures, performance
that reviews in health policy and systems research measures, quality indicators, resource utilization,
potentially involve large variation in the reporting costs, and cost-effectiveness measures.
of included studies, as they are generally more
complex than reviews of clinical topics. As such, Various tools are available to facilitate data
abstraction, especially tools facilitating source

47 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

verification (comparing the abstracted data against for comparative effectiveness evaluation rather
the original study reports). For example, the use than studies with other designs; and 3) selective
of dual displays from the same desktop or laptop reporting bias, with effectiveness evaluation
computer has been shown to reduce the average studies rather than studies evaluating other
data abstraction time by 24 minutes per study, hypotheses such as testing causal associations
relative to single display (35). This difference is (24). In rapid reviews of health policy and systems
possibly due to the reduction in time required for research, a multitude of objective and subjective
switching between or scrolling within computer outcomes are common, and reviewers should
displays during data abstraction and verification. therefore expect more discrepancies in quality
Readers interested in data abstraction tools assessments among the assessors, compared
are referred to a survey of available tools (36), to other types of reviews, such as those with a
a systematic review of automation tools (37), clinical focus. Training and piloting of assessment
and a step-by-step tutorial on data abstraction (38). forms, explanation and elaboration documents,
and related material specific to each review will
3.4 ASSESSMENT OF THE be important.
METHODOLOGICAL QUALITY OF
INCLUDED STUDIES Often the “unclear” response is applied in quality
assessment because of inadequate reporting
> 3.4.1 Methods of methodological details in study reports (24).
The reliability of quality assessment may increase
Quality assessment refers to the assessment of when published reports are supplemented with
the risk of bias in included studies (e.g. the risk additional information collected from study authors
that the study may overestimate or underestimate (26). Also, studies that are assessed as low quality
the true intervention effect), as well as the critical can sometimes be re-assessed as higher quality
appraisal of included studies (e.g. an investigation when methodological clarification is obtained from
of the extent to which study authors conducted the study investigators (27). As such, there may
their research to the highest possible standards) be value in contacting authors of studies with
(5). Approach 1 for rapid reviews recommends that missing methods information to clarify details
one reviewer assesses study quality and another pertaining to quality assessment, provided this
verifies the assessment (Table 3.1). Among the is feasible with the time and resources available.
six approaches considered for rapid reviews in
Table 3.1, only one other approach includes quality Discrepancies in quality assessment are mainly
assessment, while the remaining four approaches due to differing interpretation of the items on the
omit quality assessment altogether. There is quality assessment tools, rather than being due
a need for further research to understand the to different information identified in the study
reliability of single assessment with verification reports (24). This problem can be addressed
relative to dual assessment. through training for reviewers and additional
guidance from the tool developers. Even among
> 3.4.2 Factors affecting the reviewers experienced with such tools, between-
reliability of methodological rater agreement generally ranges from poor to
quality assessment fair in studies evaluating the reliability of quality
assessment (24, 27).
Quality assessment can be influenced by a
study’s characteristics (24, 25). For example, Discrepancies in quality assessment are common,
more consistent quality assessments have been regardless of the level of experience of the
observed in the assessment of 1) performance assessors (28). For example, between-rater
bias, with studies having objective outcomes agreement was good for quality assessment of
rather than subjective outcomes; 2) selection epidemiological studies by graduate students who
bias and performance bias, with studies designed were trained to follow an assessment manual (39),

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 48

yet the agreement between graduate students (by up to 60%), relative to using two reviewers
was poor with the Jadad quality assessment scale (10). Data abstraction by one reviewer (with
for randomized trials, and ranged from poor to independent verification) substantially reduces
fair with the Newcastle–Ottawa scale for cohort data abstraction time (a relative reduction of 36%),
and case–control studies (28). compared with data abstraction by two reviewers
(17). It would take 30 minutes per study for quality
3.5 ALLOCATING RESOURCES assessment, on average, and about 10 minutes
FOR SELECTING, ABSTRACTING, per study to discuss and resolve discrepancies
AND ASSESSING STUDIES (25). On average, it would take between 1000
and 2000 hours to complete a well-conducted
Only limited data are available to inform how review including meta-analysis, depending on the
much resources are required for the planning and number of titles/abstracts retrieved (40). Box 3.4
conduct of systematic reviews and rapid reviews. illustrates the implementation of a rapid review
Using one reviewer to screen titles/abstracts where much less time was available, and a quick
substantially reduces the average screening time turnaround was required.

BOX 3.4. Example of methodological decisions to allow a review to be conducted rapidly

• The ABC Research Unit is currently at full capacity, managing nearly a dozen systematic reviews
on a variety of topics, with an average 6–12 month timeline and 1000–2000 person-hours of
staff time per review. ABC has just received a request from the World Health Organization
(WHO) South African office for a review to evaluate the effectiveness of medical malpractice
policies in reducing obstetrics litigation.

• The WHO needs the review within 6 weeks to formulate initial recommendations on policy
reform, which will affect multiple levels of the South African health system. The organization
has provided a limited budget. Processes will have to be streamlined if ABC is to meet the
deadline with limited capacity and budget, but how?

• ABC’s first move is to form a review team with two experienced reviewers and four graduate
students. ABC knows that while the students will help maximize the budget, quality results
can still be achieved, provided experienced reviewers are on the team. They also enlist three
content experts (an obstetrician, an obstetrics care nurse, and a lawyer with experience in
obstetrics-related litigation), who will be able to quickly identify ineligible studies and speed
up the process of data abstraction.

• ABC will use Approach 1 to conduct the rapid review. The team is ready to go.

3.6 OTHER CONSIDERATIONS of the chapter may not be relevant to qualitative

reviews, where the initial research questions
We have discussed methods for selecting, may be modified with emerging evidence as
abstracting, and assessing studies for rapid the review progresses (see Chapter 4). Also,
reviews in health policy and systems research. an efficient way to meet tight timelines is to
In suggesting streamlined methods to support engage decision-makers throughout the review
timely decision-making, we have assumed that process, especially in scoping the review objectives
the review’s objectives and related research and research questions to ensure that the end
questions are well-defined in advance. However, results are useful and fit the purposes of the
some of the key points suggested at the beginning decision-makers (see Chapter 5).

49 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

3.7 CONCLUSION assessment. Rapid review teams should include
both content experts and other reviewers to
Rapid reviews of health policy and systems provide multiple perspectives to the review
research may present more challenges than process, and if feasible, include experienced
systematic reviews of clinical topics. Careful screeners, abstractors, and assessors to expedite
consideration should be given to which steps the process. Two independent abstractors should
are streamlined, factors affecting streamlined be considered to collect continuous data that
decisions, and potential consequences in terms are pivotal to decision-making. Training should
of validity of the review results and efficiency of be provided to all reviewers at the beginning of
the review process. It will be particularly important the review and during the review to deal with
to ensure that reviewers accurately and reliably issues that need to be reiterated for consistency
interpret study eligibility criteria, have clear rules purposes. Consensus-based approaches for rapid
for study selection, and are supported with pilot reviews are provided in Table 3.1, with rankings
forms and explanation and elaboration documents obtained from a study that engaged international
for study selection, data abstraction, and quality stakeholders in rapid reviews.

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 50

REFERENCES

1. Thompson RL et al. Reproducibility of systematic literature reviews on food, nutrition, physical

activity and endometrial cancer. Public Health Nutrition, 2008, 11:1006-1014.

2. Tricco AC et al. A scoping review of rapid review methods. BMC Medicine, 2015, 13:224.

3. Tricco AC et al. An international survey and modified Delphi approach revealed numerous rapid
review methods. Journal of Clinical Epidemiology, 2016, 70:61-67.

4. Eden J et al., eds. Finding What Works in Health Care: Standards for Systematic Reviews. Washington,
(DC), US, National Academies Press, 2011.

5. Higgins J, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0
[updated March 2011], The Cochrane Collaboration, 2011.

6. Systematic Reviews: CRD’s guidance for undertaking reviews in health care. York, UK, Centre
for Reviews and Dissemination, University of York, 2009 (https://www.york.ac.uk/media/crd/
Systematic_Reviews.pdf, accessed 24 May 2017).

7. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Rockville, (MD), US,
Agency for Healthcare Research and Quality, 2014 (https://www.ncbi.nlm.nih.gov/books/NBK47095/,
accessed 24 May 2017).

8. Edwards P et al. Identification of randomized controlled trials in systematic reviews: accuracy and
reliability of screening records. Statistics in Medicine, 2002, 21:1635-1640.

9. Glasziou P et al., eds. Abstract screening - the value of two reviewers. Proceedings of the 4th
Symposium on Systematic Reviews: Pushing the Boundaries; 2002; Oxford, UK.

10. Shemilt I et al. Use of cost-effectiveness analysis to compare the efficiency of study identification
methods in systematic reviews. Systematic Reviews, 2016, 5:140.

11. Ford AC et al. Errors in the conduct of systematic reviews of pharmacological interventions for
irritable bowel syndrome. American Journal of Gastroenterology, 2010, 105:280-288.

12. Olsen O et al. Quality of Cochrane reviews: Assessment of sample from 1998. British Medical
Journal, 2001, 323:829-832.

13. Cooper H, Ribble RG. Influences on the outcome of literature searches for integrative research
reviews. Knowledge-Creation Diffusion Utilization, 1989, 10:179-201.

14. Cooper M, Ungar W, Zlotkin S. An assessment of inter-rater agreement of the literature filtering process
in the development of evidence-based dietary guidelines. Public Health Nutrition, 2006, 9:494-500.

15. Mateen F et al., eds. Title-Abstract versus Title-Only Citation Screening Strategies for Systematic
Reviews and Meta-Analyses. Cochrane Colloquium; 19-22 October 2011; Barcelona, Spain.

51 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

16. Doust JA et al. Identifying studies for systematic reviews of diagnostic tests was difficult due to
the poor sensitivity and precision of methodologic filters and the lack of information in the abstract.
Journal of Clinical Epidemiology, 2005, 58:444-449.

17. Buscemi N et al. Single data extraction generated more errors than double data extraction in
systematic reviews. Journal of Clinical Epidemiology, 2006, 59:697-703.

18. Page MJ, McKenzie JE, Forbes A. Many scenarios exist for selective inclusion and reporting of results
in randomized trials and systematic reviews. Journal of Clinical Epidemiology, 2013, 66:524-537.

19. Carroll C, Scope A, Kaltenthaler E. A case study of binary outcome data extraction across three
systematic reviews of hip arthroplasty: errors and differences of selection. BMC Research Notes,
2013, 6:539.

20. Horton J et al. Systematic review data extraction: cross-sectional study showed that experience
did not increase accuracy. Journal of Clinical Epidemiology, 2010, 63:289-298.

21. Jones AP et al. High prevalence but low impact of data extraction and reporting errors were found
in Cochrane systematic reviews. Journal of Clinical Epidemiology, 2005, 58:741-742.

22. Gøtzsche PC et al. Data extraction errors in meta-analyses that use standardized mean differences.
Journal of the American Medical Association, 2007, 298:430-437.

23. Tendal B et al. Disagreements in meta-analyses using outcomes measured on continuous or rating
scales: observer agreement study. British Medical Journal, 2009, 339:b3128.

24. Hartling L et al. Testing the risk of bias tool showed low reliability between individual reviewers and
across consensus assessments of reviewer pairs. Journal of Clinical Epidemiology, 2013, 66:973-981.

25. Robertson C et al. Practicalities of using a modified version of the Cochrane Collaboration risk of bias
tool for randomised and non-randomised study designs applied in a health technology assessment
setting. Research Synthesis Methods, 2014, 5:200-211.

26. Vale CL, Tierney JF, Burdett S. Can trial quality be reliably assessed from published reports of
cancer trials: evaluation of risk of bias assessments in systematic reviews. British Medical Journal,
2013, 346:f1798.

27. Littlewood C et al. The quality of reporting might not reflect the quality of the study: implications
for undertaking and appraising a systematic review. Journal of Manual & Manipulative Therapy,
2012, 20:130-134.

28. Oremus M et al. Inter-rater and test-retest reliability of quality assessments by novice student
raters using the Jadad and Newcastle-Ottawa Scales. BMJ Open, 2012, 2.

29. Ng L et al. Title and abstract screening and evaluation in systematic reviews (TASER): A pilot randomised
controlled trial of title and abstract screening by medical students. Systematic Reviews, 2014, 3.

30. O’Mara-Eves A et al. Using text mining for study identification in systematic reviews: A systematic
review of current approaches. Systematic Reviews, 2015, 4.

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 52

31. Tonkin-Crine S et al. Discrepancies between qualitative and quantitative evaluation of randomised
controlled trial results: achieving clarity through mixed methods triangulation. Implementation
Science, 2016, 11:66.

32. Saldanha IJ et al. Evaluating Data Abstraction Assistant, a novel software application for data
abstraction during systematic reviews: protocol for a randomized controlled trial. Systematic
Reviews, 2016, 5:196.

33. Cottrell E et al. Defining the Benefits of Stakeholder Engagement in Systematic Reviews [research
white paper]. Rockville, (MD), US, Agency for Healthcare Research and Quality (US), 2014 (14-EHC006-
EF).

34. Gotzsche PC et al. Data extraction errors in meta-analyses that use standardized mean differences.
Journal of the American Medical Association, 2007, 298:430-437.

35. Wang Z et al. Dual computer monitors to increase efficiency of conducting systematic reviews.
Journal of Clinical Epidemiology, 2014, 67:1353-1357.

36. Elamin MB et al. Choice of data extraction tools for systematic reviews depends on resources and
review complexity. Journal of Clinical Epidemiology, 2009, 62:506-510.

37. Jonnalagadda SR, Goyal P, Huffman MD. Automating data extraction in systematic reviews: a
systematic review. Systematic Reviews, 2015, 4:78.

38. Li T et al. Innovations in Data Collection, Management, and Archiving for Systematic Reviews.
Annals of Internal Medicine, 2015, 162:287-294.

39. Sands ML, Murphy JR. Use of kappa statistic in determining validity of quality filtering for meta-
analysis: A case study of the health effects of electromagnetic radiation. Journal of Clinical
Epidemiology, 1996, 49:1045-1051.

40. Allen IE, Olkin I. Estimating time to conduct a meta-analysis from number of citations retrieved.
Journal of the American Medical Association, 1999, 282:634-635.

53 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 54
4
SELECTING RAPID REVIEW
METHODS FOR COMPLEX
QUESTIONS RELATED
TO HEALTH POLICY AND
SYSTEM IMPROVEMENTS
Sandy Oliver, Michael Wilson, G. J. Melendez-Torres,
Mukdarut Bangpan, Kelly Dickson, Carol Vigurs
KEY POINTS

• A two-stage process of first scoping the literature, then selecting a focus,

is an effective approach for conducting health policy and systems reviews
under time pressure.

• The complexity of health policy and systems research requires transdisciplinary

collaboration, which can, if managed well, speed and enhance a review.

• Initializing a rapid review requires a framework from which to organize the

concept under study, based on a set of focused questions or an existing
framework (either borrowed or customized) which either remains unchanged
– static – or is allowed to evolve as knowledge accumulates from the search.

• Using a static framework may speed a review, but this benefit must be
balanced against the risk of missing the significance of a theme that
emerges from the literature.

• In areas already covered extensively by existing systematic reviews, a search

identifying existing reviews may allow reviewers to simply summarize and
integrate the review findings, resynthesize the primary studies, or update
the search and reanalyse one or more of the systematic reviews.
4.1 INTRODUCTION 4.2 STRATEGIC DECISIONS FOR
RAPID REVIEW
A growing literature addresses methods to
accelerate or streamline the operational steps The choice of methods for delivering rapid
of systematic reviewing (searching, screening, reviews is intertwined with decisions about
data extraction, and appraisal). These methods
how to manage projects, the amount of work to
(addressed in Chapters 2 and 3) are typically
be done, and the knowledge already available.
applied in discussion with stakeholders to
maximize relevance to their needs, and usually
adopt a narrow focus and/or PICO framework > 4.2.1 Project management choices
(1-3). However, a PICO framework has limited
utility for exploring the complexities that arise from Reviewing can be accelerated by increasing the
‘variations within populations or interventions, size of the team without expanding the scope
of the work. However, large teams require more
effort for coordination and a shared understanding
of the key concepts. Table 4.1 illustrates these
Some methods are better suited
trade-offs by comparing two rapid reviews of
to reviewing rapidly, needing similar scales and topics: medical malpractice
to balance the urgency of rapid policies (6) and no-fault compensation schemes
(7). The first of these was completed in 12 weeks:
reviews with the precision of full
4 weeks of clarifying the focus and expectations
systematic reviews. of the policy-makers was followed by an 8-week
period during which the bulk of the work was
completed by a coordinated team of nine reviewers
or about the mechanisms of action or causal working with standardized procedures to briefly
pathways thought to mediate outcomes, other describe policies, models, and frameworks from
contextual factors that might similarly moderate different countries. The second of these rapid
outcomes, or how and when these mechanisms
reviews, despite its narrower focus, took nearly
and elements interact’ (4). Developing and testing
twice as long. In this case, a period of 5 months
theories to explain these complexities requires
configuring diverse qualitative, quantitative, allowed a team of less than half the size to
and mixed methods studies, not only aggregating complete a more configurative, theory-building
findings from similar quantitative designs (5). review, investigating the potential mechanisms
and contexts that would influence the policy
In this chapter, we consider the options for outcomes. The more interpretive nature of this
synthesizing knowledge about broad issues second review was readily achieved by a smaller
quickly by building on prior work and employing team, but necessarily took longer. The scoping
frameworks to span complex areas. We offer
review of malpractice, which presented findings
several examples to illustrate the types of
as thematic summaries, required fewer staff days
strategic methodological choices that are required
(and the relationship between these decisions over fewer months than the more complex but
and organizational context) for conducting rapid narrower review leading to theory generation.
reviews to inform complex questions related to Tables 4.1 and 4.2 consider in more detail such
health policy and system improvements. decisions about what work is to be done, and how.

57 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

TABLE 4.1. Scoping of medical malpractice policies in obstetrics

Tasksa Team Months

members

1 2 3
Negotiating scope 5b

Developing protocol 4b

Developing conceptual 3
framework

Searching for studies 2

Screening outputs: 3,004 9

Coding/extracting data 9

Synthesizing findings: 3
43 studies
c
Writing report 3
a
Bold indicates tasks performed by senior staff. / b Including commissioner and topic expert. / c Draft report submitted.

TABLE 4.2 Theory generation for no-fault compensation schemes

Tasksa Team Months

members
1 2 3 4 5
Negotiating scope 6b

Developing protocol 8b

Developing conceptual 3
framework

Searching for studies 2

Screening outputs: 3
2,170

Coding/extracting 2
data

Synthesizing findings: 2
44 papers
c
Writing report. 3
a
Bold indicates tasks performed by senior staff. / b Including three policy staff members and a knowledge broker. / c Draft report submitted.
Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 58
> 4.2.2 Choosing the scale and of analytic methods appropriate for the literature
focus of a rapid review: the available (5). Such two-stage rapid reviews are
two‑stage process increasingly common and particularly well-suited
to broad policy questions (8). These reviews
The scale of a rapid review – the amount of work provide an explicit point in the process for an
involved – is not always obvious in advance, evidence-informed decision about the final focus
particularly for complex questions related to of the review. The first stage identifies studies
health policy and system improvements. It will relevant to an often-broad review question and
depend on the amount of literature available maps the studies according to their substantive
addressing different aspects of the policy or focus. This mapping then forms the basis for one
system, how easy it is to find, its quality, and the or more reviews in the second stage, where
depth of analysis required. This uncertainty argues the studies considered most likely to produce
for starting a review with informal scoping or a useful evidence are appraised and synthesized
descriptive map, followed by decisions about (see Figure 4.1).
the review’s substantive focus and the choice

FIGURE 4.1. Two-stage review conducted in discussion with stakeholders

Stage one Stage two

Map of
Synthesis
studies
Develop
Map of Refine/ narrow
review
studies review question
question
Exclude
studies
Discussion Discussion Discussion
with with with
stakeholders stakeholders stakeholders
to initiate to tailor to interpret
review review findings

For example, in a rapid review undertaken to and efficiencies. Rather than necessarily providing
support decision-making on community-based a polished final text mapping the literature,
provision of diagnostic testing, reviewers first the aim of the first stage is to provide an empirical
surveyed and mapped the relevant evidence as basis for targeting the areas most likely to yield
identified from a search of the MEDLINE database informative results under time pressure. By first
(9). This map then informed the choice of three conducting a targeted search of one database,
priority areas for in-depth analysis (logistics of this approach provides a “snapshot” of evidence
provision, ways of providing ultrasound services, that can be subsequently supplemented with
diagnostic pathways for breathlessness), each of a refined search in additional databases. Lastly,
which was followed by a more extensive database it provides an auditable basis for “follow-up”
search. Throughout this process, the review of different areas that may not be immediate
focused on studies considered to be of highest priorities, but that could be of interest either to
relevance to health systems. researchers or to policy-makers at a later time.
Table 4.3 describes options for rapidly conducting
From the perspective of managing a review, the first stage.
a two‑stage process affords several opportunities

59 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

TABLE 4.3. Accelerating the early work in a two-stage reviewa

Time Method Interim product

available

2 hours Automated clustering of terms List of frequently occurring terms

1 day Automated clustering of terms, List of frequently occurring concepts

then manual organization

2 days Coding of random sample of titles Description of the focus and methods
and abstracts of existing research

2 weeks Coding of titles and abstracts, Approximate map of the focus and
with adjustment of depth of methods of existing research
detailed coding according to
number of studies

2 months Coding of full reports, with Accurate map of the focus and
adjustment of depth of detailed methods of existing research
coding according to number of
studies
a
Recent advances in information technology can support automated clustering of similar studies (10). Clustering of terms can be used to
speed early description (i.e. mapping) of the literature before a decision is made on where to focus most of the reviewing effort. The times
outlined here offer rules of thumb for various methods.

Two-stage reviewing can also inform the depth (or validity), and report the findings. Focusing on the
detail) of synthesis. This is illustrated by a review studies with richer findings resulted in a narrow
conducted to suit the timetable of Guidance but more in-depth synthesis to better suit the
Committees of the UK’s National Institute for ethos of qualitative research (12).
Health and Care Excellence (NICE) (elsewhere,
commonly known as Guideline Development Finally, the two-stage reviewing process also
Groups). As the views of those using social- allows informed discussions with stakeholders
care services were considered important for to reveal their priority issues and commonly
this review, the initial map produced in the held definitions, policy options and constraints,
scoping exercise took into account key study and other contextual factors (see Chapter 5 for
characteristics, such as population and setting, more on stakeholder involvement).
and highlighted those studies with titles and
abstracts that indicated the likelihood of “rich” > 4.2.3 Transdisciplinary working
data or findings (i.e. titles with terms indicating
qualitative data collection such as “face-to-face Urgent, real-world problems requiring rapid
interview” or terms indicating qualitative analysis reviews rarely align neatly within single academic
methods such as “grounded theory” rather disciplines or policy sectors. Addressing their
than “survey”). The Guideline Development complexities frequently requires not only efficiency
Group was thus able to select the substantive in applying review methods but also skills in
focus and methodological approach of greatest working across boundaries to draw on knowledge
interest before the review team, work within the from different stakeholder networks or bodies
accepted guidance (11), select and appraise the of literature, to see connections between their
relevant studies (in terms of internal and external different ways of thinking, and to access and

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 60

make use of their different knowledge resources. review process is also valuable to complement or
These are hallmarks of current approaches to interpret the findings. For instance, review findings
transdisciplinary research (13). are presented to NICE Guidance Committees
alongside testimony from expert witnesses
Multiple stakeholders are essential both to who may come from government and policy,
membership of Guideline Development Groups research, practice, individual patients, people
and to extensive, well-defined consultation using services, carers, or the community and
processes during the development of review voluntary sector (18).
protocols (14). Making good use of their input
is easier with the help of individuals who “wear However, hearing from the various stakeholders is
multiple hats” or knowledge brokers skilled in only the first step. Understanding their evidence
navigating the policy–research interface (15). needs and working with them to define the
For instance, a clinician with policy experience, concepts and frameworks that will underpin the
a social scientist with knowledge brokering reviews, in terms that can be readily understood
experience, and two laboratory scientists were and agreed upon, are important boundary spanning
able to prepare, within 2 weeks, an evidence- skills. Figure 4.2 illustrates how consulting
based policy brief to inform the implementation stakeholders informs methodological options
of screening new-born babies in Pakistan for and consequently the choice of rapid product to
congenital hypothyroidism (16). Such rapid be delivered. This is a key element of framework
“evidence checks” are also routinely prepared synthesis, a method that is particularly suitable for
by the Sax Institute with the help of a knowledge rapid reviews spanning disciplines or sectors (19).
broker (17). Tacit knowledge for informing the

FIGURE 4.2. Diagram showing how stakeholder input influences methodological choices

Inception with stakeholders

• Setting answerable question(s)
• Initial conceptual framework Faster reviews involve
• Agreeing the format of report, • Fewer stakeholders,
timescale & further meetings
• Fewer discussions,
Initial search strategy • Less iteration, and
Gauge the literature • Greater use of past
Discussion with stakeholders accumulative work
• Most relevant type of evidence
• Check conceptual framework
Final search strategy
Cluster, tabulate &
Limit or expand initial search summarise studies Annotated
Final search bibliography

Discussion with stakeholders

Static conceptual framework Tabulate studies to
• Most relevant aspects of studies inform narrative Summary
• Check conceptual framework Code, appraise & analyse studies synthesis of themes

Evolving conceptual framework Tabulate studies to

inform narrative Framework
Code, appraise & analyse studies synthesis synthesis

Rapid review of evidence

• Report to stakeholders

61 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

Figure 4.2 illustrates the stakeholder touch by inspecting titles and abstracts, and discussing
points that inform methodological decisions past research efforts (but not findings) with
and the final rapid review product (see boxes stakeholders, then choosing the substantive
at left side of figure): at inception, to agree on and methodological focus, as well as level of
key concepts and initial assumptions about how detail, appropriate to the literature and time
concepts are related, and principles about the available. For example, in the two-stage review
evidence required to test these assumptions on community-based provision of diagnostic
(the conceptual framework); after the initial testing mentioned above, the initial MEDLINE
search, to gauge the scale and depth of the search helped to establish the evaluability of
literature; and, optionally, after the final search priority areas (9).
and description of the literature in terms of the
initial conceptual framework. As shown here, 4.2.4.2 Selecting the review methodology
a discussion of evidence needs (at inception)
and evaluation of the scale and depth of the Some synthesis methods that are both detailed
literature available (after the initial search) are and iterative appear inherently unsuitable for
sufficient to prepare an annotated bibliography working rapidly. For example, line-by-line thematic
or a summary of themes; however, a framework coding, where coding refers to the systematic
synthesis requires follow-up discussion in light application of markers, words or short phrases
of characteristics of the studies identified. which represent and summarise key features
of studies included in a review, is the basis
> 4.2.4 Methodological choices of thematic synthesis (5) and collaborative
interpretation of concepts, which is used for
4.2.4.1 Evaluability assessment meta-ethnography (21) and critical interpretive
synthesis (5), render these methods meticulous
Evidence-informed policy development requires but time-consuming and hence likely unsuitable
clarifying the problem to be addressed, for rapid reviews.
establishing possible policy and programmatic
options to address the problem, and deriving Reviewing is fastest if consensus about key
testable assumptions about whether and how concepts achieved at the stage of shaping
the proposed options will work. Together, these the question can be maintained unchanged
judgements constitute an evaluability assessment throughout the synthesis. This is typical of
(i.e. determination of readiness for programme effectiveness reviews addressing clinical practice
evaluation), which is a long-standing criterion where the population, intervention, comparison,
for judging what type of evaluation should be and outcomes as ‘PICO’ framework are all
undertaken (e.g. acceptability or feasibility of an pre-defined. This type of structured synthesis,
intervention, a pilot of measures, and research for time-sensitive policy questions, typically
procedures or controlled trial) (20). satisfies decision-makers who are facing pressing
health system issues (as opposed to a thematic
The same principles apply to judging the suitability analysis, which could take much more time).
of the available literature for policy evaluation.
Reviews of effectiveness are typically preceded However, static frameworks limit what can be
by scoping the scale of literature available learned from the literature, especially where
to ascertain whether sufficient (randomized) themes or questions are not clear from the
controlled trials exist to make a review worth the beginning, or the search identifies other important
effort. In contrast, an evaluability assessment themes or definitions of key concepts. “Framework
involves gauging the maturity of the literature synthesis” allows for new themes to emerge
to determine what review focus is likely to from the literature and for these to influence the
be most fruitful. Quick judgements about the framing of the review. Examples of static and
literature’s suitability for review have been made evolving frameworks are described in Box 4.1.

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 62

BOX 4.1. Approaches to framing a rapid review

A framework is to a literature review as a wooden frame is to a new house. The frame of a

house provides the living blueprint from which workers will construct the building, in terms
of the rooms, their sizes, and eventual overall look. Similarly, the framework for a rapid
review provides a structure for reviewers to follow: the “rooms” are the topics relevant to
the stakeholder’s request, and their size indicates the amount of literature or the topic’s
importance. Just as similar houses can have different layouts, reviews can have different
frameworks to address questions in different ways.

Three approaches for framing rapid reviews are described here, starting with the fastest and
least sophisticated synthesis:

Focused questions and sub-questions: A focused question and sub-questions are developed
in discussion with the stakeholders, to guide a targeted, rapid search of the most relevant
evidence. A structured synthesis of findings from the included studies is then prepared, using
tables to map and summarize the literature according to the themes or domains of interest
that have been identified in advance with the original stakeholders.

Static thematic frameworks: Formal but rapid synthesis can be achieved by applying existing
frameworks reflecting acknowledged theory, policy, or practice. For example, the Tanahashi
framework for evaluation of health systems, which focuses on availability, accessibility,
acceptability, and effectiveness, was applied in a review of reviews (22). Alternatively, reviewing
health systems often lends itself to clustering evidence according to its country of origin,
another thematic framework. This approach was feasible for collating the international legal
literature about medical malpractice within 10 weeks (6), and for addressing the integration
of oral-health services in health systems within 6 weeks (23). Similarly, in another review,
established legislative and policy frameworks have proved useful to meet the timetables
of the National Institute for Health and Care Excellence (NICE) Guidance Committee (24).
A widely-recognized framework for social determinants of health underpinned a rapid equity
analysis (25). Classifications for complex organizations were used to frame some of the
evidence about accountable care organizations (26).

Evolving frameworks: An initial framework may be borrowed from existing theories

considered a “best fit” (27) as a starting point for synthesis, or constructed in discussions
between the review team and review stakeholders to align with their prior knowledge and
values (19). Typically, a “best fit” approach is taken when a review addresses a question that
matches a well-developed literature within an academic discipline, where existing theories
are more readily available. In contrast, frameworks are typically constructed when review
questions are transdisciplinary so that existing frameworks spanning the whole literature
are unlikely to exist. The framework can then evolve during the course of the review, in light
of concepts emerging from the literature and further discussion with stakeholders. This is
framework synthesis. It is an adaptation of framework analysis (28), used to analyse primary
data, and the method has the advantage of transparency, which supports discussion with
stakeholders for maintaining relevance and teamwork for speed of the review. Nevertheless,
even a few iterative steps to enhance the framework could significantly extend the time
required for completing the review.

63 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

4.2.4.3 Undertaking a systematic review targets the existing reviews can prove a helpful
of reviews starting point. After relevant systematic reviews
have been identified, reviewers can take one of
In areas that have been extensively covered several paths (see Box 4.2).
by previous systematic reviews, a search that

BOX 4.2. Sources of prior systematic evidence and their application in a rapid review

The time for reviewing activity can be reduced by drawing on prior systematic analysis in
the following ways:

Review-level synthesis: The least time-consuming approach restricts synthesis to the review
level, meaning the results of the reviews themselves are of interest, but their component
studies are not examined. The review-level synthesis, sometimes called an overview, describes
and integrates the review findings regarding quality and strength of the evidence for different
intervention strategies, frequently without further statistical analysis. For example, reviewers
interested in surveying the evidence on how inequity occurs in public health interventions
focused only on systematic reviews that discussed differential health effects by socioeconomic
status (29).

Reanalysis of primary studies from systematic reviews: Questions related to complex

interventions can be informed by a set of reviews where the individual reviews address
different intervention components, for instance a series of rapid overviews revealed core
components of effective support for patient self-management and a parallel rapid review
of implementation studies revealed the requirement for a whole systems approach to
implementation at the level of individual patients, practitioners, and organizations (30).
Systematic reviews of systematic reviews also offer a shortcut to a coherent set of studies
identified systematically by prior reviewers, and then available for analyses not previously
reported. Similarly, an equity analysis of the impacts of population-based physical activity
interventions was feasible in a compressed time frame because it used existing systematic
reviews (and their included studies) that were found in a specialist register of publications
related to “active living” (29). The reviewers then applied their own inclusion criteria to the
primary studies from each of the relevant reviews, and used an equity framework to reanalyse
the findings from the relevant studies.

Updates of systematic reviews: Existing systematic reviews can be supplemented by

updating the literature searches. For example, one group undertook a rapid review to inform
ongoing UK Department of Health decision-making on how to increase rates of generic
prescribing. They located 10 reviews, gleaned their individual studies and studies from
“top‑up” searches to cover intervention types that were not adequately addressed in the
original reviews before the final analysis (31).

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 64

BOX 4.2. Sources of prior systematic evidence and their application in a rapid review
(continued)

Systematic evidence as source for primary studies: Existing systematic reviews addressing
broad questions can provide a shortcut to finding relevant primary studies for a rapid review
with a narrower or overlapping question. However, because of how the authors of the existing
systematic reviews framed their work, a little lateral thinking may be required to identify
relevant reviews. For instance, a systematic review about education and peace-building,
which focused on envisaged solutions (32), included primary studies relevant to policy
interest elsewhere focused on pressing problems of ‘problematic masculinities’ expressed
as violence, aggression, and discriminatory gender norms. Similarly, systematic reviews
prepared for World Health Organization (WHO) guidance on shifting responsibilities for tasks
between different cadres in existing programmes (33) included evidence that later proved
useful for developing and implementing a new programme (16).

Multilevel uses of existing reviews: In reviews seeking to answer multiple related questions,
a multilevel synthesis strategy combining the above methods can be developed to provide
reliable knowledge (34). For example, in a rapid evidence synthesis to support delivery of
emergency mental health treatment, reviewers worked with stakeholders to develop a
pathway of mental health crisis interventions. They then systematically sought evidence for
key interventions at each point in the pathway, according to a hierarchy of evidence sources:
first, relevant, empirically supported guidance; second, overviews of reviews; third, systematic
reviews; and fourth (where no relevant evidence synthesis existed and gaps were identified
by the service user group), primary studies identified through database searches (35).

Because health systems cross academic > 4.2.5 Rapid reviews as a social and
disciplines, the methodological approaches methodological enterprise
and standards of available reviews may vary.
As such, applying the same review-selection Rapid reviews, which are typically produced in
standards across disciplines may exclude valuable response to a specific need, are first shaped by
learning. This drawback became apparent during strategic decisions about the amount of work
the conduct of a rapid review about committee to be done (team size and timescale), the scale
structures and processes for making collective and focus of the work. The team’s research
decisions about technical issues, such as clinical, knowledge and skills need to suit the available
legal or financial recommendations (36). Although literature and appropriate synthesis methods,
the review was commissioned to inform how and be supplemented by interpersonal skills
decisions would be made within the health sector, for knowledge brokering. Thus, management
relevant evidence was available from social and methodological decisions are interlinked
psychology and business administration, as well throughout the review with analytical and
as health services research. If methodological interpersonal tasks.
conventions from health services research had
been applied, the review would have excluded Typical management and methodological options
most business administration evidence because for reviewing systematically within time limits
the search strategies were reported in less detail, are described in Table 4.4, along with a range
and offered little new learning to the health sector. of possible products. Key distinctions between

65 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

the products are listing or clustering the findings to synthesize evidence from reviews and/or
reported in systematic reviews that address primary studies (when time is less restricted).
specific questions (when time is more restricted), The pathways for preparing these products are
and applying static or evolving frameworks illustrated in Table 4.4.

TABLE 4.4. Outline of what a rapid review can achieve, according to three different time
frames (days, weeks, months)

Options for rapid review and what can be done in the available timea

Purpose Project management Building on prior work Synthesis methods

Informing One or two reviewers Search for and within Cluster and tabulate
internal policy sharing the task existing syntheses systematic reviews
discussions, (reviews, evidence- identified to create
Quick iterations
management gap maps, evidence- an annotated
between review
decisions, informed guidance) bibliography, a stand-
team and policy team
within days alone product for rapid
to compare evidence Search databases
responses, or an interim
needed with presenting evidence-
text to focus discussion
evidence identified quality standards
with stakeholders for
or judgements (e.g.
rapid reviews before
DoPHERb, Health
analysing across the set
Systems Evidence,
of studies to generate
Cochrane, Campbell,
new knowledge
3iec)
Prepare summary
tables outlining key
findings from systematic
reviews, quality
appraisal of systematic
reviews (e.g. for those
indexed on Health
Systems Evidence
database, which
provides appraisals
for all reviews that it
contains), countries
where studies were
conducted (e.g. for
systematic reviews
found on Health
Systems Evidence
database)(35)

Informing Small core team As above, and search Apply a static framework
public debates, to allow collective within specialist topic to analyse across a
within weeks interpretation before sources set of studies with
and after review, with summary tables and a
Reanalyse existing
policy customer and summary of themes
systematic reviews to
within team (e.g. a rapid review of
address new questions
malpractice frameworks
Large review team
and models)(6)
to apply standardized
procedures

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 66

 Options for rapid review and what can be done in the available timea

Informing Small review team As above, and perform Apply an evolving

urgent policy to allow collective Boolean searches of framework to synthesize
decisions, interpretation and bibliographic databases findings to suit evidence
within months iteration, with policy (balancing sensitivity needs and the extent
customer and within and specificity to suit and maturity of the
team the time available) literature
(a framework
synthesis)

a
For each option, the tasks listed are those that can be performed in the time available, and the products achievable are highlighted in bold text. /
b
Database of promoting health effectiveness reviews. / c International Initiative for Impact Evaluation.

4.3 CONCLUSION reviews, as well as making methodological

decisions. Lastly, we have considered specific
In this chapter, we have argued why some methods organizational contexts that can support the
are better suited than others to reviewing rapidly, production of rapid reviews. Generally, reviews
acknowledging a tension between speedy work for that are done quickly involve fewer stakeholders,
urgent decisions and the methodological diversity less discussion, less iteration, and greater use
and precision that full systematic reviews offer. of prior accumulative work. When time allows,
The concept of “methods” has been interpreted increased discussion with stakeholders and
broadly here to include project management greater iteration draw out appreciable learning
and evidence-informed deliberation for shaping from the literature.

67 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

REFERENCES

1. Crawford C et al. Rapid Evidence Assessment of the Literature (REAL((c))): streamlining the
systematic review process and creating utility for evidence-based health care. BMC Research
Notes, 2015, 8:631.

2. Featherstone RM et al. Advancing knowledge of rapid reviews: an analysis of results, conclusions

and recommendations from published review articles examining rapid reviews. Systematic
Reviews, 2015, 4:50.

3. Ganann R, Ciliska D, Thomas H. Expediting systematic reviews: methods and implications of

rapid reviews. Implementation Science, 2010, 5:56.

4. Anderson LM et al. Investigating complexity in systematic reviews of interventions by using a

spectrum of methods. Journal of Clinical Epidemiology, 2013, 66:1223-1229.

5. Gough D, Thomas J, Oliver S. Clarifying differences between review designs and methods.
Systematic Reviews, 2012, 1:28.

6. Cardoso R et al. Rapid scoping review of medical malpractice policies in obstetrics. Toronto,
(ON), Canada, BreaKThrough, Knowledge Translation Program, Li Ka Shing Knowledge Institute,
St. Michael’s Hospital, 2015 (http://www.afro.who.int/en/south-africa/south-africa-publications.
html, accessed 22 May 2017).

7. Dickson K et al. No-fault compensation schemes: A rapid realist review. London, UK, EPPI-Centre,
Social Science Research Unit, UCL Institute of Education, University College London, 2016 (https://
eppi.ioe.ac.uk/cms/Default.aspx?tabid=3687, accessed 24 May 2017).

8. Thomas J, Newman M, Oliver S. Rapid evidence assessments of research to inform social

policy: taking stock and moving forward. Evidence & Policy: A Journal of Research, Debate and
Practice, 2013, 9:5-27.

9. Evidence for models of diagnostic service provision in the community: literature mapping exercise
and focused rapid reviews. In: Chambers D, Booth A, Baxter SK, Johnson M, Dickinson KC, Goyder
EC, eds. Health Services and Delivery Research. Southampton, UK, NIHR Journals Library, 2016.

10. Stansfield C, Thomas J, Kavanagh J. ‘Clustering’ documents automatically to support scoping

reviews of research: a case study. Research synthesis methods, 2013, 4:230-241.

11. Developing NICE Guidelines: The Manual. London, UK, National Institute for Health and Care
Excellence (NICE), 2015 (https://www.ncbi.nlm.nih.gov/books/NBK310375/, accessed 24 May 2017).

12. Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured
methodological review. Systematic Reviews, 2016, 5:74.

13. Bernstein J. Transdisciplinarity: A review of its origins, development, and current issues. Journal
of Research Practice, 2015, 11:Article R1.

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 68

14. Brockis E et al. A Review of NICE Methods Across Health Technology Assessment Programmes:
Differences, Justifications and Implications. London, UK, The Office of Health Economics,
2016 (https://www.ohe.org/publications/review-nice-methods-across-health-technology-assessment-
programmes-differences, accessed 20 May 2017).

15. Oliver S, Bangpan M, Dickson A. Producing policy relevant systemtic reviews: navigating the
policy-research interface. Evidence & Policy: A Journal of Research, Debate and Practice, in press.

16. Shaukat M et al., eds. Screening newborns for congenital hypothyroidism (CHT). London, UK,
Social Science Research Unit, UCL Institute of Education, University College London, 2015.

17. Moore G et al. Does knowledge brokering improve the quality of rapid review proposals? A before
and after study. Systematic Reviews, 2017, 6:23.

18. NICE’s approach to public involvement in guidance and standards: a practical guide. London,
UK, National Institute for Health and Care Excellence, (https://www.nice.org.uk/media/default/
About/NICE-Communities/Public-involvement/Public-involvement-programme/PIP-process-guide-
apr-2015.pdf, accessed 22 May 2017).

19. Oliver SR et al. A multidimensional conceptual framework for analysing public involvement in
health services research. Health Expectations, 2008, 11:72-84.

20. Wholey J. Evaluability assessment: Developing program theory. New Directions for Evaluation,
1987, 1987:77–92.

21. Toye F et al. Meta-ethnography 25 years on: challenges and insights for synthesising a large
number of qualitative studies. BMC Medical Research Methodology, 2014, 14:80.

22. Chopra M et al. Strategies to improve health coverage and narrow the equity gap in child survival,
health, and nutrition. Lancet, 2012, 380:1331-1340.

23. Bhuiya A, Wilson MG. Rapid Synthesis: Identifying the Effects of and Approaches to Integrating
Oral-Health Services in Health Systems. Hamilton, (ON), Canada, McMaster University, 2016 (https://
macsphere.mcmaster.ca/handle/11375/21193, accessed 24 May 2017).

24. Transition between inpatient hospital settings and community or care home settings for adults with
social care needs. London, UK, National Institute for Health and Care Excellence, 2015 (https://
www.nice.org.uk/guidance/ng27, accessed 23 May 2017).

25. Lorenc T et al. What types of interventions generate inequalities? Evidence from systematic
reviews. Journal of Epidemiology and Community Health, 2013, 67:190-193.

26. Guta A, Wilson MG, Lavis JN. Rapid Synthesis: Examining the Impacts of Accountable Care
Organizations on Patient Experience, Population Health and Costs. Hamilton, (ON), Canada,
McMaster University, 2016 (https://macsphere.mcmaster.ca/handle/11375/19411, accessed
23 May 2017).

27. Booth A, Carroll C. How to build up the actionable knowledge base: the role of ‘best fit’ framework
synthesis for studies of improvement in healthcare. BMJ quality and safety, 2015, 24:700-708.

69 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

28. Ritchie J, Spencer L. Qualitative data analysis for applied policy research. In: Bryman A, Burgess
R, eds. Analyzing Qualitative Data. London, UK, Routledge, 1994.

29. Humphreys DK, Ogilvie D. Synthesising evidence for equity impacts of population-based physical
activity interventions: a pilot study. The International Journal of Behavioral Nutrition and Physical
Activity, 2013, 10:76.

30. A rapid synthesis of the evidence on interventions supporting self-management for people with
long-term conditions: PRISMS – Practical systematic Review of Self-Management Support for
long-term conditions. In: Taylor S, Pinnock H, Epiphaniou E, Pearce G, Parke H, Schwappach A,
et al., eds. Health Services and Delivery Research. Southampton, UK, NIHR Journals Library, 2014.

31. Moe-Byrne T et al. Behaviour change interventions to promote prescribing of generic drugs: a
rapid evidence synthesis and systematic review. BMJ Open, 2014, 4:e004623.

32. Lopes Cardozo M et al. Literature Review: Youth Agency, Peacebuilding and Education. Amsterdam,
Netherlands, Research Consortium Education and Peacebuilding, University of Amsterdam,
2015 (https://educationanddevelopment.files.wordpress.com/2014/11/youth-literature-review.
pdf, accessed 22 May 2017).

33. WHO Recommendations: Optimizing health worker roles to improve access to key maternal
and newborn health interventions through task shifting. Geneva, Switzerland, World Health
Organization (WHO), 2012 (http://www.who.int/reproductivehealth/publications/maternal_perinatal_
health/978924504843/en/, accessed 24 May 2017).

34. Paton F et al. Improving outcomes for people in mental health crisis: a rapid synthesis of the
evidence for available models of care. Health Technology Assessment, 2016, 20:1-162.

35. Rapid Response Program: Summary of service timelines. Hamilton, (ON), Canada, McMaster
University, (https://www.mcmasterhealthforum.org/docs/default-source/program-handouts/rapid-
response-service-summary.pdf?sfvrsn=2, accessed 22 May 2017).

36. Oliver S, Hollingworth K, Briner R. Effectiveness and efficiency of committee work: a rapid
systematic review for NICE by its Research Support Unit. London, UK, EPPI-Centre, Social Science
Research Unit, UCL Institute of Education, University College London, 2015 (http://eppi.ioe.
ac.uk/cms/Portals/0/PDF%20reviews%20and%20summaries/Effectiveness%20committee%20
work%202015%20Oliver.pdf?ver=2016-01-11-152731-973, accessed 24 May 2017).

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 70

5
ENGAGING POLICY-MAKERS
AND HEALTH SYSTEMS
MANAGERS IN THE CONDUCT
OF RAPID REVIEWS

Andrea C. Tricco, Wasifa Zarin, Vera Nincic, Patricia Rios,

Paul A. Khan, Marco Ghassemi, Sanober S. Motiwala,
Ba’ Pham, Sandy Oliver, Sharon E. Straus,
Etienne V. Langlois
KEY POINTS

• Engaging policy-makers and health systems managers in rapid reviews

increases the relevance and applicability of the reviews to decision-making
processes, yet it is time- and resource-intensive.

• There are many ways in which the producers of rapid reviews can engage
policy-makers or health systems managers ranging from ad hoc engagement
to involvement throughout the entire review process.

• Engagement with policy-makers or health systems managers throughout

the review is encouraged, yet such extensive involvement necessitates
additional time and resources.

• The level of engagement should be meaningful, yet tailored to available

resources, and will depend on the objectives of engagement, the points at
which engagement occurs in the review process, and the methods used
for engagement.

• Conceptual frameworks are available to provide a structure and mechanism

to facilitate engagement.
5.1 INTRODUCTION policy or systems level because their decisions
may influence a large proportion of the population.
Health-care researchers traditionally have had However, researchers often do not know how
little engagement with the decision-makers to engage with decision-makers.
who could implement their research findings.
The questions that researchers posed were purely Decision-maker engagement can be defined
academic (i.e. curiosity-driven), and only rarely as “an iterative process of actively soliciting
were decision-makers involved in developing the knowledge, experience, judgement,
those questions. This culture has led to significant and values of individuals selected to represent
research waste (1) and a broad range of direct
slow implementation interests in a particular
of research findings (2). issue, for the dual
The field of health policy The level of engagement should purposes of: creating
and systems research (3) a shared understanding
has advocated a shift in be meaningful, yet tailored to [and] making
this culture to promote available resources. relevant, transparent,
more efficient uptake and effective decisions”
of research results (4). We consider
by decision-makers. decision-maker
engagement to include
In situations where urgent or timely decisions are opportunities for decision-makers, specifically
required, rapid reviews are often commissioned policy-makers and health systems managers (5)
by: governments, health-system stakeholders, for the purpose of this chapter, to interact in a
international organizations, and civil society. meaningful way with the process for, or results
In these contexts, close collaboration between of, a rapid review. This chapter also covers the
the decision-maker and the producer of the rapid objectives of engagement (i.e. desired outcomes
review is essential to ensure that results are of engagement), points and processes, supportive
relevant and workable, which should enhance structures and mechanisms, as well as benefits
evidence uptake. Such collaboration is particularly and challenges.
important for decision-makers who act at the

5.2 OBJECTIVES OF ENGAGEMENT

Among the various objectives for engaging policy-makers and health systems managers in rapid
reviews are the following:

to establish a research agenda (6, 7);

to prioritize indicators (8, 9);

to establish learning materials to be included in a curriculum (10, 11);

to develop a framework (12, 13);

to establish clinical, policy, or system recommendations (14, 15);

to develop a tool kit to support evidence use (16);

to finalize knowledge translation and uptake strategies (17, 18);

to aid decision-makers in their decision-making processes (19, 20).

The objectives for engagement help to determine the points in the rapid review process when
engagement will occur.

73 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

5.3 POINTS OF ENGAGEMENT they can participate in numerous steps during
the rapid review process (21), at the initiation
When policy-makers and health systems managers or planning phase, during the review conduct,
are involved throughout the review process, and at the end of the review, as outlined below.

Engagement opportunities

AT THE INITIATION AND DURING CONDUCT OF THE AT THE END OF THE

PLANNING OF THE REVIEW REVIEW REVIEW

• selecting review topics; • selecting studies, whether • developing key messages

• defining the research by screening studies for and other knowledge
question; inclusion or providing input translation activities.
• developing or reviewing the about whether specific
protocol; studies meet eligibility
• providing input about key criteria;
terms to include in the • abstracting data, whether
literature search; and/or identifying data elements
• setting or providing input for abstraction or
into the eligibility criteria. participating in the data
abstraction itself;
• providing input into data
analysis or synthesis of
results;
• interpreting results; and/or
• drafting or reviewing the
resulting report.

A practical example of an integrated approach covering all phases of the review process is presented in
Figure 5.1 (22).

FIGURE 5.1. A practical example of an integrated approach to engage policy-makers and

health systems managers throughout the review process

Stakeholder engagement opportunities

Stakeholder Stakeholder as Stakeholder Stakeholder

Stakeholder
Topic Review Team Reaction Involvement
Input Meeting
Consultation Member Meeting in Dissemination

Data Extraction
Initial Assessment Draft of
Literature and Report Written
Research of Study Findings
Search Evidence and Published
Question Relevance Produced
Synthesis

Systematic Review Steps

This figure shows how stakeholders can be involved at all steps of the systematic review process.
Source: Keown et al., 2008 (22)

Reproduced with permission from Wolters Kluwer Health, Inc. (2017)

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 74

Alternatively, policy-makers or health systems The process of engagement, as well as its
managers may be involved in a single step frequency and intensity, will depend on
of the process. In one review to assess the the objective of engagement, the points at
effectiveness of self-management for chronic which engagement occurs, the engagement
conditions, experts were consulted using a methods, and the time and resources available
modified Delphi (or agreement-building) method (Figure 5.2). Engagement methods can include
to identify keywords for “self-management” and workshops, policy dialogues, team work, surveys,
“chronic condition”, before beginning the search interviews, and modified Delphi approaches.
(23). Saan and colleagues (23) provide a tool, These engagement activities can take place
TRASI (Tool for Recording and Accounting for involving various modes, such as teleconferences,
Stakeholder Involvement in Systematic Reviews) videoconferences, in-person meetings, and e-mail
(23) for recording stakeholder involvement in exchanges between researchers and decision-
development of the literature search for this makers to obtain feedback at one or more
review. specific points of the review process. Some of
these engagement methods are highlighted in
5.4 PROCESSES OF ENGAGEMENT the examples provided in Table 5.1 and Table 5.2.

A range of engagement methods can be used

to engage decision-makers in rapid reviews.

FIGURE 5.2. Frequency and Intensity of Engagement

+
CONSULTATION

1 +1
MORE THAN 1 AT EVERY STEP
ONE-TIME CONSULTATION
(initial research question,
CONSULTATION
(e.g. initial research literature search, assessment of
(e.g. initial research question and study relevance, data extraction
question) report written and evidence synthesis, draft
and published) of findings produced, report
written and published)

75 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

TABLE 5.1. Case examples of decision-maker engagement in rapid review programmes

Aspect of Pineault et al. Hayden et al. (19) Khangura et al. (20)

engagement (24)

Type of Decision-makers Policy-makers Health-care managers

decision-
Academics Clinicians
makers
involved Clinicians Health-care managers

Type of Workshops Workshops Feedback on the question

engagement and proposal
Videoconferences
Feedback on the report
Feedback on the
report Knowledge transfer
activities
Delphi survey

Challenges to Difficulties Not reported Not reported

engagement incorporating
expert opinions
and decision-
makers’
viewpoints

Benefits of Not reported Opportunities to interact Established relationships

engagement with decision-makers, between researchers and
leading to established decision-makers
relationships
Led to subsequent
Opportunities for collaborations
capacity-building of
decision-makers
Led to subsequent
collaborations

Outcomes of Not reported Survey on decision- Not reported

engagement maker satisfaction

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 76

TABLE 5.2. Case examples of various levels of engagement for systematic reviews of
health policy and systems research

Aspect of Odendaal (25) Akl et al. Pantoja (26)

engagement (EA. Akl,
unpublished
data, 2016)

Type of decision- Policy-makers Policy-makers Policy-makers

makers involved
Academics Academics Academics
Clinicians

Type of Workshops Modules to Interviews

engagement prioritize review
Electronic surveys to Electronic surveys
topics
prioritize review topics
Priority-setting
Workshops
Local and international reviews based on
presentations Modified Delphi topics most pertinent
survey to policy-makers
Embedding a policy-maker
into the review team Priority-setting tool
for health policy
and systems
research reviews

Challenges to Not reported Not reported Difficulties balancing

engagement stakeholder opinions
at different levels
(e.g. local versus
national)
Difficulties converting
a policy question to a
research question

5.5 ONGOING ENGAGEMENT can be improved. A combination of engagement

OF DECISION-MAKERS methods such as teleconferences or in-person
meetings, surveys or interviews, and workshops
Where sufficient time and money are available or webinars can facilitate ongoing engagement
and the objective of engagement is to aid policy- and improve the level of engagement with
makers or health systems managers in their policy-makers or health systems managers for
decision-making, researchers can select from future rapid reviews. Over time, it is hoped that
a range of engagement methods and modes to positive relationships built on trust and mutual
maintain ongoing partnership and communication respect will develop (19, 20). Engagement
during the review process. To help facilitate future can also be fostered through other supportive
engagement, at the end of the rapid review, structures and mechanisms, as described in the
policy-makers or health systems managers following section.
can be asked, through surveys and interviews,
if they were satisfied with the product and if the
results were helpful and how future products

77 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

5.6 SUPPORTIVE STRUCTURES the relevance of research, decreases research
AND MECHANISMS FOR waste, and enhances both mutual learning and
ENGAGEMENT IN RAPID REVIEWS the transparency of research (28). In the context
of reviews, perceived benefits include more
A conceptual framework or model, characterizing comprehensive literature searches, more rigorous
the engagement process, can be a useful guide review findings, and greater clarity of the review
for researchers seeking to involve policy-makers results (22). Another benefit is establishing
and health systems managers in rapid reviews. positive relationships between researchers and
One example is the framework for effective policy-makers or health systems managers, which
engagement in comparative effectiveness could lead to subsequent collaborations (19,
research (4). This framework covers the following 20). Policy-makers or health systems managers
aspects of engagement: inputs (e.g. professional who are engaged in reviews report increases in
or patient experience and values), quantitative their own feelings of appreciation, the relevance
methods for combining inputs (e.g. Delphi and utility of the review results, their interest in
survey), qualitative methods (e.g. facilitated sharing the review findings with colleagues, and a
workshops), outputs (e.g. decisions made by the greater understanding of the review findings
decision-maker group, such as study design) and (22). Engaging policy-makers or health systems
outcomes regarding the engagement process managers may improve the relevance of the
(e.g. decision-maker trust), and the comparative review questions, transparency of the review
effectiveness research itself (e.g. useful evidence procedures, and usefulness of the results. As well,
for a decision). In addition, this framework can be it is anticipated that engagement of policy-makers
employed to tease out what would be expected or health systems managers increases uptake
from the engagement, and what methods will be of the review’s results, with the greatest uptake
useful for enabling engagement for a particular occurring with engagement of multiple decision-
rapid review. making organizations (29). More information on
how to improve the uptake of rapid reviews is
Researchers may also wish to consider using provided in Chapter 8.
the conceptual framework of Oliver and Dickson
(27), which provides models and mechanisms for Engaging policy-makers and health systems
engaging policy-makers in systematic reviews managers in reviews can also lead to several
of health policy and systems research, to guide challenges. For example, decision-maker
the level of engagement for rapid reviews. engagement is time- and resource-intensive
This framework includes components, such as and may lead to less rigorous review findings
obtaining policy-maker input, building relationships, (22). Difficulty in finding a policy-maker or health
and increasing policy-maker awareness and skills. systems manager who is willing to participate
Mechanisms to facilitate engagement described as a review team member has occurred (22).
by these authors include obtaining stable funding Moreover, even if a policy-maker or health
so that researchers can address policy-makers’ systems manager agrees to participate, findings
queries, providing training and support to foster of the review could contradict what they believe,
constructive iterative engagement, and having a creating conflict (22). Another challenge lies in
team including members with previous experience creating demand for reviews to inform decision-
in a decision-making role or who have long-term making and managing expectations of the policy-
experience working with decision-makers. maker or health systems manager about what
questions the review is capable of answering
5.7 BENEFITS AND CHALLENGES (29). Matching the research question posed by
OF ENGAGEMENT the decision-maker with the way in which the
literature is structured has sometimes proven
It has been observed that policy-maker and difficult (29). Furthermore, it can be challenging
health systems manager engagement increases to incorporate policy-makers’ or health systems

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 78

managers’ perspectives when interpreting the financial resources are required to implement
review findings (24). As well, there may be meaningful engagement at multiple points in
a perception that the engagement of policy- the review process. Ideally, researchers should
makers or health systems managers is purely seek stable funding and establish a relationship
tokenistic. It can also be challenging to balance with the commissioner of the rapid review(s).
input from multiple policy-makers or health Face-to-face contact is often not possible,
systems managers and conflicts of competing but there are a variety of effective processes
interests must be managed carefully, particularly that can enable communication and build trust,
in countries where governance and accountability including webinars, teleconferences, and email
remain important challenges. communication. The application of several of
these processes and structures can be seen
5.8 CONCLUSION in the example in Box 5.1, where researchers
conducted a rapid review for a national regulatory
Although engagement with policy-makers or health agency. Conceptual frameworks are available,
systems managers can be challenging, experience and provide a structure and mechanism to facilitate
in health policy and systems research shows this process. Researchers should ensure that
that early, active, and continuing engagement is the level of engagement is fit for purpose and
imperative for evidence-informed health policy- tailored to existing resources, while ensuring
making (30). Adequate availability of time and that the engagement itself is meaningful.

BOX 5.1. Conducting a rapid review with maximal engagement in mind

As a producer of rapid reviews, you have been contacted by a national regulatory agency
(Health Canada) to conduct a rapid review with a 5-month timeline. The agency is thinking about
creating a social media platform to detect adverse drug reactions, and needs information on
which platforms are currently available and the reliability of data obtained from social media
sources (21). The new platform will be used to monitor adverse drug reactions for the entire
Canadian population and to make important policy decisions. You believe that engaging the
policy-makers from Health Canada who commissioned the review will help to ensure that
the review is relevant to their needs. However, you are unsure how to proceed.

You use the conceptual framework of Oliver and Dickson (27) to guide engagement. Stable
funding was secured from the Canadian government and training and support is provided to
the policy-makers from Health Canada. You hold teleconferences at the beginning of the study
to scope the review question and finalize the protocol. You obtain approval for your protocol,
and enter into an agreement with Health Canada that the review will be completed within a
5-month time frame and that monthly updates on progress will be provided. You invite Health
Canada to participate in pilot tests for screening and data abstraction. At month 4, you host a
webinar to share preliminary results and get the agency’s input on key messages. At month
5, you submit the review report, and organize another webinar to gain the agency’s input on
interpretation of results and knowledge uptake strategies. With substantial guidance from
policy-makers from Health Canada, the report is highly relevant and provides them with
recommendations that can be implemented.

79 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

REFERENCES

1. Moher D et al. Increasing value and reducing waste in biomedical research: who’s listening?
Lancet, 2016, 387:1573-1586.

2. Sketris IS et al. Community-Engaged Scholarship to Catalyze Innovation: A Case Study of the

Uptake of Metered-Dose Inhalers with Spacers to Deliver Respiratory Medication in a Pediatric
Emergency Department in Nova Scotia. Healthcare Papers, 2016, 16:42-54.

3. Strategy on health policy and systems research: changing mindsets. Geneva, Switzerland, World
Health Organization, 2012 (http://www.who.int/alliance-hpsr/alliancehpsr_changingmindsets_
strategyhpsr.pdf, accessed 13 Jun 2017).

4. Deverka PA et al. Stakeholder participation in comparative effectiveness research: defining a

framework for effective engagement. Journal of Comparative Effectiveness Research, 2012, 1:181-
194.

5. Tricco AC et al. Barriers, facilitators, strategies and outcomes to engaging policymakers, healthcare
managers and policy analysts in knowledge synthesis: a scoping review protocol. BMJ Open,
2016, 6:e013929.

6. Nielsen CP, Funch TM, Kristensen FB. Health technology assessment: research trends and future
priorities in Europe. Journal of Health Services Research and Policy, 2011, 16:6-15.

7. Adam-Poupart A et al. Climate change and occupational health and safety in a temperate climate:
potential impacts and research priorities in Quebec, Canada. Industrial Health, 2013, 51:68-78.

8. Brennan Ramirez LK et al. Indicators of activity-friendly communities: an evidence-based consensus

process. American Journal of Preventive Medicine, 2006, 31:515-524.

9. Barber CE et al. Development of system-level performance measures for evaluation of models

of care for inflammatory arthritis in Canada. Journal of Rheumatology, 2016, 43:530-540.

10. Pears S et al. Development and feasibility study of very brief interventions for physical activity
in primary care. BMC Public Health, 2015, 15:333.

11. Sidibe S et al. Communicating about microbicides with women in mind: tailoring messages for
specific audiences. Journal of the International AIDS Society, 2014, 17(3 Suppl 2):19151.

12. Pakes BN. Ethical Analysis in Public Health Practice [thesis]. Toronto, (ON), Canada, Institute of
Medical Sciences, University of Toronto, 2014.

13. Reeve C, Humphreys J, Wakerman J. A comprehensive health service evaluation and monitoring
framework. Evaluation and Program Planning, 2015, 53:91-98.

14. Ardalan A et al. 2012-2025 Roadmap of IR Iran’s Disaster Health Management. PLoS Currents,
2012, 4:e4f93005fbcb34.

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 80

15. DiCenso A et al. Advanced practice nursing in Canada: overview of a decision support synthesis.
Nursing Leadership (Toronto, Ontario), 2010, 23 Special Number 2010:15-34.

16. Davidson EM et al. Behavior change interventions to improve the health of racial and ethnic
minority populations: a tool kit of adaptation approaches. Milbank Quarterly, 2013, 91:811-851.

17. Clark R et al. Evidence and obesity prevention: developing evidence summaries to support
decision making. Evidence & Policy: A Journal of Research, Debate and Practice, 2013, 9:547-556.

18. Guise J-M et al. A practice-based tool for engaging stakeholders in future research: a synthesis
of current practices. Journal of Clinical Epidemiology, 2013, 66:666-674.

19. Hayden JA et al. Methods of a multi-faceted rapid knowledge synthesis project to inform the
implementation of a new health service model: Collaborative Emergency Centres. Systematic
Reviews, 2015, 14:1.

20. Khangura S et al. Evidence summaries: the evolution of a rapid review approach. Systematic
Reviews, 2012, 1:10.

21. Tricco AC et al. Utility of social media and crowd-sourced data for pharmacovigilance: a scoping
review protocol. BMJ Open, 2017, 7:e013474.

22. Keown K, Van Eerd D, Irvin E. Stakeholder engagement opportunities in systematic reviews:
knowledge transfer for policy and practice. Journal of Continuing Education in the Health
Professions, 2008, 28:67-72.

23. Saan MC et al. Recording and accounting for stakeholder involvement in systematic reviews.
Health Information and Libraries Journal, 2015, 32:95-106.

24. Pineault R et al. Conceptual and methodological challenges in producing research syntheses for
decision-and policy-making: an Illustrative case in primary healthcare. Evaluation, 2010, 16:137-152.

25. Odendaal A. South African initiative for systematic reviews on health policies and systems (SAI).
South Africa, South African Medical Research Council, 2016 (http://www.mrc.ac.za/healthsystems/
sai.htm, accessed 18 May 2017).

26. Pantoja T, ed. Approaches for prioritizing questions for systematic reviews in health policy and
systems research. Third Global Symposium on Health Systems Research; 2014; Cape Town,
South Africa.

27. Oliver S, Dickson K. Policy-relevant systematic reviews to strengthen health systems: models
and mechanisms to support their production. Evidence & Policy: A Journal of Research, Debate
and Practice, 2016, 12:235-259.

28. Concannon TW et al. A systematic review of stakeholder engagement in comparative effectiveness

and patient-centered outcomes research. Journal of General Internal Medicine, 2014, 29:1692-1701.

81 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

29. Atkins D, Fink K, Slutsky J. Better information for better health care: the Evidence-based Practice
Center program and the Agency for Healthcare Research and Quality. Annals of Internal Medicine,
2005, 142:1035-1041.

30. Langlois EV et al. Enhancing evidence informed policymaking in complex health systems: lessons
from multi-site collaborative approaches. Health Research Policy and Systems, 2016, 14:20.

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 82

6
FOSTERING THE CONDUCT
AND USE OF RAPID
REVIEWS OF HEALTH
POLICY AND SYSTEMS
RESEARCH IN LOW- AND
MIDDLE-INCOME COUNTRIES
Rhona Mijumbi-Deve, Fadi El-Jardali
KEY POINTS

• Although there is some momentum in the use of rapid reviews for decision-
making processes in low- and middle-income countries (LMICs), experience
with this form of evidence summary remains limited in these settings.

• Several challenges impede optimal production and use of rapid reviews,

including wide variation in their definition, methods, and applicability;
inadequacy of resources; and poor acceptability among academics who
may not believe their results.

• To ensure that the full potential of rapid reviews is achieved in LMICs,

there is a need to mobilize and sustain adequate resources. Furthermore,
review producers need to address the methodological concerns associated
with these reviews.

• Rapid review producers and knowledge users alike need to set up structures
and systems supportive of rapid reviews and also need to improve the
sharing of knowledge that arises from producing and using these reviews.
6.1 INTRODUCTION LMICs, this section highlights their potential
use in these settings.
The conduct and use of rapid reviews are gaining
momentum in low- and middle-income countries Rapid reviews are often considered in light of
(LMICs). There has been a slow start because decision-making by policy-makers, among other
many LMICs lack supportive political, economic, types of decision-makers. Much as their longer
and scientific institutions and procedures. These counterparts – systematic reviews – and other
gaps make it challenging for researchers to types of research are viewed mainly as academic
conduct, and decision-makers to use, rapid or scholarly products, rapid reviews and other rapid
reviews to inform health policy-making and knowledge syntheses are viewed as products
health system strengthening in these settings. meant to support policy- and decision-making
However, the challenges have not deterred processes that result in evidence-informed policy
those who recognize the potential benefits of and practice approaches (1-5).

Academics and decision-makers have noted

a variety of potential uses for rapid products,
Supportive systems, structures including their use interimly to inform further
investigation of a given topical issue (or to
and resources need to be define the need for such investigation), guideline
developed to foster conducting development on very focused topics, and policy
decisions needed on quick turnaround (2). In these
rapid reviews in low- and and other circumstances, it is thought that rapid
middle-income countries. review products may be more relevant within
some specific health systems than in others;
that is, they are specific to both the context and
the organization (6).
rapid reviews from engaging with rapid reviews
and seeking ways to improve the use of such However, researchers are cautioned that there
reviews to inform their decisions. It is therefore may also be situations where rapid reviews
important for research producers, knowledge are inappropriate, even if the circumstances
brokers, and entrepreneurs to look at ways of outlined above are in place. For example, where
making rapid reviews more available, useful, the evidence will feed into development of
and usable in LMIC systems. To accomplish broad (e.g. international and some national)
these goals, stakeholders and knowledge users guidelines, a rapid review may not be appropriate
need to consider the lessons that have been (2). Therefore, although they have unique value,
learned so far about rapid reviews in their own especially for decision-making, rapid reviews
and other health systems, identifying both complement, but do not replace, other sources
good practices and the challenges that must be of evidence for decisions.
addressed to optimize the process. This chapter
harnesses lessons learned from conducting rapid Aside from providing timely and relevant evidence
reviews relevant to health system settings in for decisions, rapid reviews may improve the
LMICs and identifies strategies for overcoming clarity and accessibility of research evidence
challenges and fostering the conduct and use for decision-makers (4). Furthermore, for many
of policy-relevant rapid reviews of health policy policy- and decision-making institutions, rapid
and systems research. reviews have increased the uptake of evidence
to inform time-sensitive, system-level decision-
6.2 THE POTENTIAL FOR RAPID making (7). In several cases, decision-makers
REVIEWS IN LMICS have valued the responsiveness of the rapid
review process and have perceived it as being
To emphasize the rationale for improving the a credible source of unbiased, evidence-based
production and application of rapid reviews in information supporting advice for policy-making

85 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

bodies (8). In addition, rapid reviews have been quality of and findings from pertinent evidence,
valuable to and influential on policy decision- that is conducted over a short period, using
making, informing high-impact health system various methods to accelerate the knowledge-
decisions, providing guidance, and resulting synthesis process. This rapid approach involves
in implementation of recommendations, all of abbreviating the process by tailoring conventional
which can, in turn, save substantial resources (3). systematic review methods towards the most
rigorous methods that the delivery time frame
A retrospective survey, assessing among other allows, although (as noted in Chapters 2 and 3)
things users’ perspectives on how and when there is no agreement on which aspects should
they used rapid reviews, found that rapid reviews be abbreviated (12, 13). Overall, the focus is
have influenced and inspired partnerships and on two important aspects that are reflected
plans to modernize current practices (7). Insights in the term “rapid review”: the time frame for
from the rapid response service at the Center for completion and the extent of synthesis of the
Systematic Reviews on Health Policy and Systems evidence therein (12). However, academics
Research (SPARK) in Lebanon allude to the same and policy managers in LMICs have very little
conclusion, whereby partnerships have been experience with rapid reviews and have noted the
formed with government authorities and other variations in methods and the poor transparency
key stakeholders to address pertinent problems, in their reporting).
including the region’s refugee crisis. Thus, as can
be seen through their widespread application in Variation is seen not only in the definition of
high-income countries and through demonstrated and methods used to conduct rapid reviews,
success in developing countries like Lebanon and but also in their application (14).
Syria, rapid reviews have enormous potential
in LMICs. However, LMIC settings present > 6.3.2 Poor acceptance
considerable challenges for the establishment by stakeholders
of rapid reviews.
With no standardized definition, methods,
6.3 CHALLENGES OF or application, and their general variation away
CONDUCTING AND USING RAPID from conventional systematic reviews (which are
REVIEWS IN LMICS considered to have high quality because of their
rigorous methods), rapid reviews have not been
As noted earlier in this Guide (Chapter 1), readily accepted by some stakeholders, especially
the profile of rapid reviews has been rising partly those in academia, despite their acknowledged
because of a push for more evidence-informed benefits. Evidence suggests that rapid reviews are
decision-making to reduce waste, increase equity, generally viewed as “quick and dirty”, and there
and strengthen health systems. The conduct of are often major concerns about the reliability of
rapid reviews is now emerging as a strategy to their results (15). Although existing comparative
overcome various barriers that decision-makers evidence has shown that similar conclusions
experience in accessing and using high-quality, are derived from rapid reviews and systematic
relevant evidence when they need it (9, 10). reviews, the lack of evidence comparing potential
bias in these two approaches to knowledge
> 6.3.1 Variation in methods for and synthesis is still a cause for concern for many
application of rapid reviews in the field (6).

The methods of rapid review production and use > 6.3.3 Low capacity to conduct and
vary greatly. In addition, there is a significant use rapid reviews in LMICs
lack of transparency and inadequate reporting
of the processes used for rapid reviews (11). The call for and need to conduct and use rapid
Although there may be variants in the definition reviews may be growing steadily, but the capacity
of a rapid review, this type of evidence summary to do so is still very limited internationally,
is typically understood to be a synthesis of the especially in LMICs. Very few centres and

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 86

institutions in LMICs are conducting rapid reviews, identification of lessons and challenges. Table
and even among those that are performing this 6.1 lists several centres in LMICs that are known
type of knowledge synthesis, only a few have to be doing rapid reviews.
documented their work sufficiently to allow for

TABLE 6.1. Institutions in low- and middle-income countries that are involved in prepa-
ration of rapid reviews

Country Institution

Argentina Institute for Clinical Effectiveness and Health Policy (IECS)

Ethiopia Ethiopia Evidence-Based Health Care Centre, Jimma University

Lebanon Center for Systematic Reviews on Health Policy and Systems Research
(SPARK), American University of Beirut (AUB)

South Africa Cochrane South Africa

South African Medical Research Council, Health Systems Unit

Uganda School of Medicine and School of Public Health, Makerere University College
of Health Sciences (MakCHS)

Even within these centres in LMICs, there is a 6.4 STRATEGIES TO IMPROVE THE
paucity of capacity to conduct and use knowledge CONDUCT AND USE OF RAPID
syntheses, let alone specialized knowledge REVIEWS IN LMICS
syntheses like rapid reviews. The number of
institutions with the necessary structures and > 6.4.1 Raise the profile of rapid
skilled personnel to carry out these specialized reviews in LMICs
syntheses is small but growing, in a bid to meet
the potential demand from policy- and decision- Rapid reviews are increasingly seen as being of
makers. Yet very few users of rapid reviews are importance within a variety of settings that face
conversant with their production and how to use constraints of not only time but also resources
them for decision-making. (16). The profile of rapid reviews needs to be
raised in LMICs to create demand and motivation,
> 6.3.4 Absence of appropriate and to ensure the necessary resources are
structures and systems made available for their production and use.
A higher profile and closer engagement with
Even in places where rapid review products have decision-makers can also ensure that the reviews
been attempted or availed, few decision-making produced meet stakeholders’ needs, which
processes offer structures or an environment will in turn mean a greater likelihood that these
suitable for absorbing evidence of any kind, products will be used.
let alone rapid reviews. As such, LMIC centres
need more support and development to maximize The profile of rapid reviews might be enhanced
the potential benefits of research products from through increased and broader communication
rapid review (5). This capacity development would about what a rapid review is (and is not) and
enhance the motivation to conduct or demand through sharing of knowledge about successes
such products in LMIC settings. and potential benefits. There needs to be deliberate
advocacy for this type of review, through

87 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

champions and opinion leaders. Such advocacy Conversely, some users of rapid reviews
efforts should be aimed at both producers and appreciate the uniqueness of the various methods
users of rapid reviews, encouraging the former to of producing rapid reviews. Indeed, as noted in
develop the skills for and interest in conducting section 6.2, rapid review approaches may be
reviews, and encouraging the latter to make use described as context- and/or organization-specific
of them and even to get involved in the review (6). Furthermore, these methods are seen as
process to the extent possible. “flexible and pragmatic”, aiming to balance the
objectivity and rigour required of rapid reviews
In addition, funders have a critical role in raising within the limited time frames in which they must
the profile of and capacity for rapid reviews. be produced (19). This flexibility is what leads
Aside from supporting the development of to great variation across products with regard
skills, funders may become involved in building to duration and depth of analysis. This variation
and supporting networks and collaborations, allows rapid reviews to be specifically tailored
in experience-sharing, and in contributing to to address targeted policy questions, which has
the understanding of different contexts (e.g. led, in turn, to emphasis on methods involving
economic and political). These efforts at various strategies to improve transparency, instead of
organizational and administrative levels will attempts to harmonize variant methods. Indeed,
increase not only the conduct and use of rapid some evidence producers encourage a diversity
reviews, but also the use of evidence more of methods (5). They acknowledge that although
generally and therefore the making of evidence- one consistent methodological approach may
informed health decisions. Further discussion not be optimal or appropriate, it is important
of how to increase the uptake of rapid reviews to provide detailed descriptions of the chosen
can be found in Chapter 8. methods and to discuss their implications in
terms of potential bias (20).
> 6.4.2 Address
methodological concerns Where methods are harmonized, or there is an
attempt to get some basic elements common
When this Guide was in preparation, we found across producers, researchers in LMICs could
no evidence to suggest that rapid reviews should establish some core principles of evidence-based
not be conducted or that they are misleading synthesis. These principles should apply to rapid
because of their methods or for other reasons reviews as well, in order to minimize bias to the
(4). However, some institutions and researchers extent possible (21) but also to give guidance
have become aware of a lack of confidence that as new capacity is built for and around different
absence of a standardized procedure inspires rapid review methods. (One should note that
in their academic peers and in many potential these core principles would change and evolve
knowledge users. Several academics have as stakeholders gain more experience with
examined the different methods used in rapid these rapid reviews). Users of rapid reviews
reviews; in addition to identifying vairous levels have provided indications of such basic tenets,
of quality of these methods, they have noted expressing a strong preference for the following
the lack of methodological reporting (4). These review methods and characteristics: strength of
deficiencies make it difficult to aggregate the evidence assessments, quality rating of studies,
disparate rapid review methods into a common use of evidence tables, and use of summary
one. Hence, there is a pressing call, from some tables of results and conclusions (2). Furthermore,
academic peers and users of the evidence, reviewers could take the additional steps of having
for more transparency from producers of rapid subject and methodological expertise on their
reviews in laying out methods and operating review teams, highlighting the limitations of the
procedures (5, 14, 17). Ongoing development approaches taken, and communicating regularly
of reporting standards aims at addressing this with knowledge users, other team members,
challenge, to facilitate the equity and credibility and experts (22). Users and producers alike have
accorded to them (14, 18). indicated that the type of analysis/synthesis and

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 88

the quality/strength of evidence are significantly review methodology, experience in conducting
important for decision-making and that the reviews, and experience in knowledge use
most acceptable trade-offs to increase reviewer (23). In addition, there is a need for sustained
efficiencies were in limiting the literature search investment of financial and other resources
and performing single screening of abstracts and to support the conduct of rapid reviews and
full texts for relevance (2). Whatever approach to continue building capacity. Centres that
producers opt for, especially when cutting short are already established to do other types of
conventional processes used for systematic research syntheses, such as systematic reviews,
reviews, they should consider shortcuts that may build on what is already available, because it
are unlikely to affect the quality or risk of bias of is sometimes feasible for systematic processes
the review (4). These could include limiting the to be expedited if additional resources are made
scope of the review, limiting data extraction to available (13). In other words, with more resources,
key characteristics and results, and restricting longer processes like those of conventional
the study types included in the review. systematic reviews can be shortened.

In their quest to understand rapid reviews and, Those with experience in producing rapid reviews
in turn, to make them understandable, producers for decision-makers (24) have acknowledged
could also consider and highlight that a rapid that because there may not be much lead time
review may be a “living” and evolving document, before the knowledge user needs the rapid
one that can be updated or augmented in the review findings, maintaining a highly skilled
future. As such, the protocol and resulting report staff (or being able to mobilize staff members
can be updated as more evidence becomes quickly) is critical to organizational readiness to
available and as more input is obtained from produce rapid reviews. In addition, having few
stakeholders, especially given that rapid reviews and/or narrow and focused questions (e.g. related
must be considered in light of other pieces of to emerging technologies, single interventions,
evidence that are available not only to producers specific populations, single systems pillars like
but also to users. health services delivery, operational efficiency,
or quality improvement) was deemed necessary,
Acknowledging the limited experience that many as was restricting the scope of practice for the
LMIC settings have with rapid reviews, further rapid review programme itself, considering the
research from these countries is encouraged, implications for financial and human resources (25).
to understand how rapid reviews fit within existing
methods of knowledge synthesis and to explore Many of the centres currently conducting rapid
conduct and reporting guidelines specific to reviews lack continuing funding, and currently
rapid reviews. In addition, research to document conduct projects that are specifically funded by
and understand variations in methodology will donors or through seed funding. However, these
be important in addressing concerns about the establishments subsequently face sustainability
methods used in rapid reviews. challenges. In addition to encouraging health
systems authorities to invest in dedicated
> 6.4.3 Increase human, financial, rapid review centres, existing centres may
and other resources consider a “user-pays” model in which the
review commisioner pays the costs incurred in
Rapid reviews in LMICs are hindered by the severe producing the rapid review (26). Such a model
limitations on resources available to produce would require formal evaluation of the work and
and use them. There is a need to improve the its impact, so as to demonstrate its benefits to
number of rapid review experts and to increase the system and to provide a basis for advocating
their skill to do this work. Some essential skills for funds and other resources in the future.
have been identified, including content expertise,
information specialization, expertise in systematic

89 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

> 6.4.4 Provide supportive systems some academics (15). This viewpoint may hinder
and structures the search for ways to improve methods for and
application of the practice and must therefore
The provision of certain supportive systems and be addressed. The main concern about rapid
structures would facilitate the conduct and use reviews that academics have expressed is the
of rapid reviews in LMICs. Such systems and relation between time and quality. The question
structures provide a conducive environment continually recurs as to whether the limited time
for researchers, academics, and evidence available allows for appropriate methods and
entrepreneurs to conduct and promote rapid therefore results in a rapid review of sufficient
reviews as a beneficial form of evidence synthesis. quality (15). Furthermore, the frequent absence
In addition, these rapid review promoters would of information about methods represents a
be able to build capacity for their conduct and missing link in judging the quality of the work
use. Furthermore, such structures and systems (11). The transparency of methods and the
may support users of the rapid review products provision of sufficient information are important,
in their quest to find, understand, and use the not only to knowledge users but also to academic
rapid reviews. A few of these structures and peers. Researchers desire to make their own
systems are discussed below. judgements on the potential value and quality
of rapid review products, and information that
6.4.4.1 Decision-making systems and aids in these assessments will go a long way
structures towards dispelling the scepticism with which
rapid reviews are viewed.
In many LMICs, it is not obligatory to use evidence
in making decisions, and doing so may even be In their bid to understand rapid reviews, systematic
looked upon as an extra and burdensome step reviewers have often labelled or categorized
for the decision-maker. Indeed, the barriers these products as being “quick and dirty” relative
that decision-makers face while attempting to to other categories of research or knowledge
use evidence for decision-making are widely generation (15). The term “quick and dirty”
documented (9, 10). With no structures for carries the connotation that any review done
accountability to the public about the use of quickly must have been done sloppily. However,
evidence for decisions, the default position of this is not the case. Furthermore, rapid reviews
some decision-makers may be to shun evidence should ideally be measured against what they are
and the process of incorporating it into their intended to do and how they are intended to be
decisions. Rapid reviews aim to support decision- done, rather than being assessed in relation to
making; therefore, there is need for systems and other research methods (see also descriptions of
structures enabling their conduct, with strategies timelines and steps for rapid reviews in Chapter
incorporated to facilitate uptake of the resulting 2). There is a need to clearly dissociate rapid
reviews (26). For example, when policy- and reviews from this comparative category and
decision-makers put into place policies related label them rightfully for what they are: quick,
to the use of research and other evidence for timely, and relevant.
decisions, they may enable the use of such
evidence and may ensure that rapid reviews In addition to improving transparency in terms
are valued, sought, commissioned, considered, of methods, research producers and knowledge
and used in decision-making. brokers need to improve knowledge and sensitivity
among their peers concerning what rapid reviews
6.4.4.2 Peer acceptance in the academic are and what benefits they have over other
community sources of knowledge and evidence and other
types of research. Producers of rapid reviews
One of the greatest barriers to the advancement may take for granted that others have a clear
of rapid reviews has been strong scepticism from understanding of these products, but such is often

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 90

not the case. There is also a need to emphasize to have input on the production of new reviews,
that rapid reviews are complementary to (not and to enable them to make use of rapid review
replacements for) other evidence sources and products. Such systems and structures constitute
to highlight when they may be more appropriate knowledge translation strategies, and may include
than other methods. stakeholder meetings or workshops; preparation
of summaries, executive summaries, or evidence
6.4.4.3 Scholarly recognition of rapid summaries; and use of social media and webinars
reviews (25). Chapter 5 of this Guide provides further
guidance on engaging policy-makers and other
Another role of academia and its peers is that of end-users.
recognizing and attaching value to rapid reviews
as contributing to academic achievement. A lot of In several LMICs, rapid response services for
the work that goes into policy-maker engagement knowledge translation have been shown to
and achievement of high-impact decisions, be feasible (3) and are now being scaled up.
policies, and practice goes unrecognized. There These services have been set up in academic
is a need for academia to develop a system settings, in ministries of health, and as semi-
whereby work, like that done on rapid reviews, autonomous government entities. The variable
is valued as a scholarly endeavor and accorded implications of these different settings relate
status in faculty review processes. For example, to access to the policy process, availability
beyond the need for academic institutions to of personnel to support different parts of the
promote, not castigate, rapid review methods, production process, and political influence (27,
the institutions would ensure that rapid review 28). Preliminary findings from lessons compiled
production could lead to faster advancement by rapid response services in three African
or promotion on the academic research track, countries reveal the strengths and weaknesses
thus providing further motivation for those who of various types of host institutions (27, 28).
conduct rapid reviews. In addition, academics For example, having an academic institution as
involved in producing rapid reviews should the host of a rapid response service provides
be encouraged to publish them, even those easy access to research through institutional
developed in response to a policy- or decision- subscriptions, as well as easy access to support
making process. Currently, few rapid reviews are from other researchers; a neutral or unbiased
published, yet publication is a recognized (and view of the policy questions is assumed of this
often rewarded) step of academic achievement. setting. The downsides to this approach are that
academia is often looked upon with “suspicion”
6.4.4.4 Knowledge translation for by knowledge users (on the assumption that
evidence uptake academic researchers will be promoting the
institution’s own research) and that it has limited
Aside from systems for the actual production access to current decisions in the policy-making
and use of rapid reviews, arrangements are world. Yet these services are ideal, in that they
needed to ensure appropriate interaction between provide an enabling environment to improve the
producers and users. Several academics have use of rapid reviews, linking the policy world to
pointed out that the production of rapid reviews rapid review producers, increasing the demand
alone is not enough to ensure they will be for rapid reviews, and in turn increasing the
taken up by users (26). For example, a system motivation of producers. Their presence may
is needed to ensure that the topics addressed relieve the burden of knowledge translation
are relevant to current policy decision-making from the users or the producers, who are often
processes. Furthermore, processes are needed not as well-equipped or as skilled as knowledge
to inform policy-makers about the existence of brokers or entrepreneurs.
rapid reviews already produced, to allow them

91 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

> 6.4.5 Ensure documentation 6.5 CONCLUSION
and knowledge-sharing
Rapid reviews represent a growing support
Documentation of the practices, experiences, for policy- and decision-making processes in
and lessons of emerging groups doing rapid LMICs, yet more systematic fostering of their
reviews in LMICs is important. These records will conduct and use is needed. Although the limited
allow for growth through constructive critique. experience of LMICs with rapid reviews is now
They will also provide examples that others growing, challenges to the conduct and use
can replicate, adapt, and build upon. Indeed, of such reviews have been noted, including
some academics have called for more rapid methodological challenges; a paucity of human,
reviews to be published in the peer-reviewed financial, and other resources to produce and
literature (20), an ideal form of documentation use these reviews; and a lack of structures to
that allows wide dissemination to academic enable their optimal uptake. To improve the
audiences. Other forms of dissemination to reach production and use of rapid reviews, researchers
a wide range of stakeholders are also important; and users will need to not only build capacity and
for example, targeted webinars and meetings. mobilize resources, but also develop supportive
arrangements in both academia and policy
settings, raising the profile of these reviews and
sharing the knowledge they generate.

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 92

REFERENCES

1. Khangura S et al. Evidence summaries: the evolution of a rapid review approach. Systematic
Reviews, 2012, 1:10.

2. Hartling L et al. EPC Methods: AHRQ End-User Perspectives of Rapid Reviews [research white
paper]. Rockville, (MD), US, Agency for Healthcare Research and Quality (US), 2016 (16-EHC014-EF).

3. Mijumbi RM et al. Feasibility of a rapid response mechanism to meet policymakers’ urgent needs
for research evidence about health systems in a low income country: a case study. Implementation
Science, 2014, 9:114.

4. Haby MM et al. What are the best methodologies for rapid reviews of the research evidence for
evidence-informed decision making in health policy and practice: a rapid review. Health Research
Policy and Systems, 2016, 14:83.

5. Clifford TJ, Kelly SE. Expediting evidence synthesis for healthcare decision-making: exploring
attitudes and perceptions towards rapid reviews using Q methodology. International Journal of
Technology Assessment in Health Care, 2016, 4:e2522.

6. Abou-Setta AM et al. Methods for Developing Evidence Reviews in Short Periods of Time: A
Scoping Review. PloS One, 2016, 11:e0165903.

7. Peterson K et al. User survey finds rapid evidence reviews increased uptake of evidence by
Veterans Health Administration leadership to inform fast-paced health-system decision-making.
Systematic Reviews, 2016, 5:132.

8. McIntosh HM et al. The Healthcare Improvement Scotland evidence note rapid review process:
providing timely, reliable evidence to inform imperative decisions on healthcare. International
journal of evidence-based healthcare, 2016, 14:95-101.

9. Oliver K et al. A systematic review of barriers to and facilitators of the use of evidence by
policymakers. BMC Health Services Research, 2014, 14:2.

10. Lavis J et al. Towards systematic reviews that inform health care management and policy-making.
Journal of Health Services Research and Policy, 2005, Suppl 1:35-48.

11. Moher D et al. Quality of conduct and reporting in rapid reviews: an exploration of compliance
with PRISMA and AMSTAR guidelines. PeerJ, 2016, 5:79.

12. Hartling L et al. A taxonomy of rapid reviews links report types and methods to specific decision-
making contexts. Journal of Clinical Epidemiology, 2015, 68:1451-1462.e1453.

13. Kelly SE, Moher D, Clifford TJ. Defining Rapid Reviews: A modified delphi consesus approach.
International Journal of Technology Assessment in Health Care, 2016, 32:265-275.

14. Featherstone RM et al. Advancing knowledge of rapid reviews: an analysis of results, conclusions
and recommendations from published review articles examining rapid reviews. Systematic
Reviews, 2015, 4:50.

93 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

15. Kelly S. Deconstructing rapid reviews: An exploration of knowledge, traits and attitudes [thesis].
Ottawa, (ON), Canada, School of Epidemiology, Public Health and Preventive Medicine, Faculty
of Medicine, University of Ottawa, 2015.

16. Kaltenthaler E et al. The use of rapid review methods in health technology assessments: 3 case
studies. BMC Medical Research Methodology, 2016, 16:108.

17. De Buck E et al. Use of evidence-based practice in an aid organisation: a proposal to deal with
the variety in terminology and methodology. International journal of evidence-based healthcare,
2014, 12:39-49.

18. Harker J, Kleijnen J. What is a rapid review? A methodological exploration of rapid reviews in Health
Technology Assessments. International journal of evidence-based healthcare, 2012, 10:397-410.

19. Lambert R et al. Practical applications of rapid review methods in the development of Australian
health policy. Australian Health Review, 2016.

20. Ganann R, Ciliska D, Thomas H. Expediting systematic reviews: methods and implications of
rapid reviews. Implementation Science, 2010, 5:56.

21. Garritty C et al. Cochrane Rapid Reviews Methods Group to play a leading role in guiding the
production of informed high-quality, timely research evidence syntheses. Systematic Reviews,
2016, 5:184.

22. O’Leary DF et al. Using rapid reviews: an example from a study conducted to inform policy-
making. Journal of Advanced Nursing, 2016.

23. Tricco AC et al. A scoping review of rapid review methods. BMC Medicine, 2015, 13:224.

24. Hartling L et al. EPC Methods: An Exploration of Methods and Context for the Production of
Rapid Reviews [research white paper]. Rockville, (MD), US, Agency for Healthcare Research and
Quality (US), 2015 (15-EHC008-EF).

25. Polisena J et al. Rapid review programs to support health care and policy decision making: a
descriptive analysis of processes and methods. Systematic Reviews, 2015, 4:26.

26. Haby MM et al. Designing a rapid response program to support evidence-informed decision-
making in the Americas region: using the best available evidence and case studies. Implementation
Science, 2016, 11:117.

27. Mijumbi-Deve RM. Rapid Response Services in four LMICs: Lessons learned. Kampala, Uganda,
Supporting Use of Research Evidence (SURE) for Policy in African Health Systems, 2014.

28. El-Jardali F et al. Capturing lessons learned from evidence-to-policy initiatives through structured
reflection. Health Research Policy and Systems, 2014, 12:2.

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 94

7
REPORTING AND
DISSEMINATING RAPID
REVIEW FINDINGS

Shannon E. Kelly, Sharon E. Straus,

Jessie McGowan, Kim Barnhardt
KEY POINTS

• Knowledge users should be identified and engaged early and throughout

the rapid review process.

• Approaches to reporting and dissemination should be discussed with the

primary knowledge user as early as the protocol stage.

• Rapid reviews should prioritize the practical needs of the primary knowledge
user over traditional or academic approaches to dissemination, with tailoring
of the message and approach to the needs of knowledge users.

• Relevant reporting guidelines should be used in the development of rapid

review reports, to ensure comprehensive and transparent documentation
of the rapid review process.
 goals, target audiences, and anticipated impact
7.1 INTRODUCTION or influence of the rapid review (Section 7.3).
This chapter outlines how to report findings
> 7.1.1 Goals of research reporting from a rapid review of health policy and systems
and dissemination research, and discusses options for dissemination
to the appropriate knowledge users.
Once the data collection, analysis, and interpretation
stages of a review are finished, there is still much 7.2 GUIDANCE AND METHODS FOR
work to be accomplished before the review can REPORTING RAPID REVIEWS
be considered complete.
> 7.2.1 Core principles of reporting
For research to be valuable, it must be reported knowledge syntheses
clearly and transparently. Clear reporting of
evidence syntheses, including rapid reviews, Rapid reviews are an important and useful tool for
enables uptake and appropriate use of research
knowledge users; however, insufficient reporting
findings across a variety of knowledge users,
can potentially reduce the utility of a knowledge
including policy-makers and health systems
synthesis product if the knowledge users do not
managers (1). Although approaches to rapid
have enough information to evaluate the strengths
reviews for health policy and systems research
and weaknesses of the synthesis process and/or
may vary, the considerations for reporting and
the results (3). Regardless of the aim of methods
disseminating findings apply to all. Given the
methodological tailoring of rapid reviews, which used for a rapid review, maintaining research
helps to expedite the review timeline, it is important integrity depends upon a few core principles to
that reporting reflect protocol-driven decisions, guide the processes of conducting the review and
processes, and findings (see section 7.2). preparing its report (4). In particular, knowledge
users are interested in both the findings of the
Dissemination involves communicating research review and its methods. Similar to other knowledge
results for a specific audience, with the goal synthesis approaches, authors must take care to
of maximizing both uptake and impact (2). limit reporting bias, by having the protocol (and
Dissemination activities and tools should be any amendments) on hand as the report is written
customized for each review through consideration (5, 6). In general, authors of a rapid review should
of the significance of the findings, dissemination follow these core principles:

FIGURE 7.1 Core principles of rapid reviews

WORK FROM A PROTOCOL and use it to guide the conduct and reporting of the review;

ACCURATELY AND TRANSPARENTLY DOCUMENT all steps and judgements in the review
process (such as: “Did the rapid review team make any methodological concessions to answer the
research question[s] within available resources?”) (7, 8);

USE CLEAR LANGUAGE that will be understandable to knowledge users. Write at a level that
someone without a university degree can understand, and avoid the use of jargon or technical terms,
except where such terms are essential. Be mindful of technical terminology or terms that may have a
slightly different definition in the review setting than in everyday usage (e.g. blinding, control, practice)
(9, 10);

PROVIDE ENOUGH DETAIL about the methods that a knowledgeable reader could reproduce
the review;

SUMMARIZE THE METHODOLOGICAL STRENGTHS AND WEAKNESSES using language

designed to help non-experts interpret and judge the value of the review (11).

97 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

If these basic principles are not followed, the knowledge user may lack adequate information to
determine the reliability or validity of the review as a guide to decision-making.

7.2.1.1 Special considerations for rapid reporting requirements of the knowledge user
reviews of health policy and systems is essential, and one size does not fit all when it
research comes to rapid reviews. We suggest discussing
how findings will be reported early in the review
Health policy and systems research often involves process, so that the needs of knowledge users
the assessment of complex interventions. Rapid may be considered. For example, knowledge users
reviews in this area may describe multifaceted may require that their institutional or organizational
or context-specific interventions that may be reporting template be used, may want authors to
investigated through a variety of study designs apply an existing report format (e.g. the 1:3:25
(e.g. controlled before-and-after, interrupted time format (10)), or may have additional requirements
series, qualitative, or nonrandomized studies). beyond the traditional research findings report (e.g.
This complexity, and any difficulties encountered a slide deck or policy brief). It could be helpful
during the review process as a result, should be to provide knowledge users with a template
carefully described in the research report, keeping commonly used to report rapid reviews, and ask if
in mind that a wide variety of stakeholders may any information should be added to the template.
be interested in the results. Time spent discussing the report in advance of
its completion will help to limit the time required
As with any knowledge synthesis, reporting for for subsequent revisions.
rapid reviews of health policy and systems research
should be as comprehensive as possible within It may also be useful to send a summary of
the time frame for review completion. In particular, preliminary findings to the knowledge users and
it is important that any methodological tailoring suggest a meeting or telephone call to discuss.
during the conduct of the review be noted in the Their input may then be used to inform the final
methods section. In addition, it may help to describe report. Above all, the report should be tailored to
the differences between a rapid review and the the needs of the knowledge users, while balancing
content of a more comprehensive review, to frame timelines and available resources. Reporting
the limitations and to emphasize caution around should balance comprehensive accounting of
interpretation (8). We suggest that this material be the research process and findings with what
provided in the discussion/interpretation section is sufficient to meet the requirements of the
of the rapid review report, which should include knowledge users (and/or other stakeholders if
a description of the review limitations. Authors important) (12).
of rapid reviews should also provide a disclaimer
section in the executive summary, as part of > 7.2.2 Reporting guidelines
the discussion, or as a note on the cover page, and checklists
to highlight these limitations and any perceived
impact on the findings of the review. Reporting guidelines exist to ensure that research
reports contain enough information about the work
7.2.1.2 Consideration of knowledge users’ to make it usable, appraisable, and replicable.
needs In short, the guidelines aim to fix (or prevent)
deficiencies common to research reporting by
Rapid reviews are frequently commissioned by setting a minimum standard or template that
a knowledge user to inform a specific decision. should be applied when reporting a review.
These individuals are likely to be an integral part The Enhancing the QUAlity and Transparency Of
of the research process, from defining the scope health Research (EQUATOR) Network provides a
and setting the research question to finalizing the comprehensive searchable database of research
results. As such, they should also be included reporting guidelines and links to other appropriate
in the reporting process. Understanding the resources (13).

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 98

Research has shown that reporting of the rapid to facilitate the reporting of review protocols,
review approach and tailoring of the methodology is which also may be useful to rapid review authors
often inadequate (1, 14, 15). A detailed assessment when developing their protocol (16). Other similar
of the reporting quality of published rapid reviews, organization-specific guidance is available (e.g.
using the Preferred Reporting Items for Systematic manual of the Joanna Briggs Institute (17)).
Reviews and Meta-Analyses (PRISMA) Statement, In addition, individual groups or organizations may
also found the reporting to be of poor quality across have internal reporting guidelines or standards.
the included rapid reviews (1). These assessments It may be helpful for review authors to check
found that key decisions in the review process the websites of rapid review producers to see
and conduct are often presented with insufficient examples of templates and key features (15, 18).
detail or omitted completely.
An extension to the PRISMA reporting
Guidance documents and checklists (such as guideline specific to rapid reviews is currently
the PRISMA Statement) are available through under development ((13); A. Stevens, personal
the EQUATOR Network, to inform the reporting communication, 2017). As there is not yet a
of various knowledge synthesis approaches, published protocol or research plan, it remains
but to date there is only one tool that provides unclear how the PRISMA extension for rapid
any guidance specific to rapid reviews: a checklist reviews will address the variety of approaches
developed by Abrami and colleagues (8). Their used or whether it will provide guidance specific
checklist reminds authors to provide explanations to health policy and systems reviews.
in key decision areas, and recommends reporting
of the research question, inclusion criteria, search Notably, there have been gaps in the reporting of
strategies, inter-rater agreement (if applicable during some essential items. For example, many rapid
study identification, calculation of effects, and/or reviews fail to mention the use of a protocol
coding of study features), outcome extraction, study (14), which conflicts with a report that over
features, analysis, interpretation and implications, 90% of organizations producing rapid reviews
cautions and limitations, and conclusions. However, use a protocol (15). Reporting is often brief or
the checklist omits several key areas that are truncated, and methods may be reported in
worth noting: use of a protocol, inclusion of a documentation separate from the rapid review
structured abstract, explicit identification of the report itself. Other items noted in the literature
report as a rapid review, internal or external peer as being poorly reported are the study screening
review of the review, and critical appraisal of the and data collection processes, definitions of study
information included in the review and the types eligibility, methods of assessing risk of bias in or
of information sought (e.g. reviews, quantitative across studies, processes used for syntheses,
or qualitative studies, or other types of research). and limitations in the review process (1).

To ensure that reporting is complete and The PRISMA checklist provides a starting point
transparent, future exploration of reporting (and for items to be included in a rapid review report
conduct) guidelines specific to rapid reviews is (with certain adjustments specific to the context,
warranted. Certain other guidelines and checklists such as having the title identify the study as a rapid
are relevant to rapid reviews, although they review, rather than a systematic review). However,
focus on the reporting of systematic reviews, it may be more helpful to use the reporting items
such as PRISMA (4, 5). The PRISMA Statement listed in Table 7.1, which encompass some of
is specifically aimed at systematic reviews and the PRISMA items but are tailored specifically
meta-analyses of health care interventions, to rapid reviews. These items may be more or
yet many of its checklist items are relevant to less applicable, depending on the rapid review
the rapid review approach. An extension to the approach used (19).
PRISMA statement (called PRISMA-P) endeavours

99 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide

TABLE 7.1. Suggested minimum reporting items for rapid reviews of health policy and
systems research

Category Items to consider

Protocol Was a protocol used?

If so, was the protocol made public, published in a journal, and/or
registered (if so, provide reference and/or registration number, or link to
protocol)?

Overall scope Was the scope limited in any way?

Were there a limited number of research or policy questions?
Were the research questions of limited type (e.g. effectiveness only,
specific populations)?
Was the number of included studies limited?

Comprehensiveness Was the search strategy limited in any way (e.g. number of databases,
grey literature, date, setting, language)?
Were there limits on the types of study designs included (e.g. existing
systematic reviews, randomized controlled trials)?
Was textual analysis limited (e.g. no full-text review and/or limits on the
number of items extracted)?

Rigour and quality Was the process of dual study selection or dual data extraction modified or
control omitted?
Was the internal or external review of the final research report limited or
omitted?

Synthesis Was the assessment of risk of bias or quality of evidence limited or

omitted?
Was qualitative or quantitative analysis limited or omitted?

Other When making statements about the findings of the rapid review, were the
conclusions simplified or omitted?
Is it appropriate to provide a disclaimer and/or limitations section in context
with your findings?

7.3 DISSEMINATION OF tailoring of the knowledge to make it usable to the

REVIEW FINDINGS intended stakeholders (2). Typical dissemination
and communication activities undertaken by
Dissemination involves the communication and researchers are described below.
distribution of rapid review findings to specific
target audiences, across or within settings, and the

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 100
Before starting the dissemination process, consider is the significance of the research findings and
the following basic questions: how generalizable or remarkable they may be to
intended knowledge users.
• What is the review authors’ goal?
> 7.3.1 Overview of available research
-- Is it dissemination only? For example, dissemination frameworks
is the goal simply to share review results
with other researchers? Funders? Policy- The aim of a dissemination framework is to
makers? Members of the public? assist with conceptualizing or organizing research
dissemination and implementation activities.
-- Is it uptake (i.e. implementation)? An in-depth discussion of frameworks for research
For example, is the goal for the review dissemination and implementation is outside the
findings to inform or influence decision- scope of this chapter. However, it is worth noting
making? that many different frameworks are available.
Some are broad, whereas others are more specific
If the goal is dissemination only, it is important to their particular purpose; all have benefits and
to identify the targets of the research. These limitations. Some of the more common frameworks
could include researchers, the general public, that could be used to develop a dissemination
practitioners or policy-makers. Next, authors must and implementation process for a rapid review
decide upon the dissemination strategy. This may are the Knowledge-to-Action Cycle, the Ottawa
include presentations at meetings, publications in Model of Research Use, and the COM-B (capability,
peer reviewed journals, or creation of policy briefs opportunity, motivation, and behaviour) model (2).
or media releases. Review authors also need to It is also important to evaluate dissemination and
consider how to engage with policy-makers and implementation activities. For detailed descriptions
other types of decision-makers to share their of applying dissemination frameworks and
research results. evaluating dissemination and implementation
activities, we refer readers to other resources,
If the goal is to influence decision-making, compiled in the book Knowledge Translation in
information needs or requests of the primary Health Care (2).
knowledge user will guide dissemination
and implementation activities. Although the > 7.3.2 Engagement meetings
dissemination strategy should focus on meeting and dialogue
these needs in the context requested by the
primary knowledge user, review authors may also Clear dialogue and continued engagement are
consider that if one knowledge user has asked essential to ensure that the needs of knowledge
a question, it is likely of concern to others in the users are considered in the rapid review. Early
same or similar circumstances. As such, review engagement meetings between the review
authors could focus dissemination on the needs producer and the policy- and decision-makers
of just the primary knowledge user or they could who will use the findings of the rapid review
also contextualize findings for a broader audience are essential. This dialogue between the review
of interested knowledge users. Discussion of producer and the primary knowledge users begins
implementation efforts is beyond the scope of this with agreement on the review topic, scope,
chapter and we refer readers to other resources and research question(s), but may also cover what
for further information (2). methodological limitations or trade-offs they are
willing to accept to expedite the synthesis process
The answers to these simple questions will (20). It is important that this dialogue continues
form the basis of the research dissemination throughout the review process, and that the
strategy and will help frame the scope of the review authors and knowledge users maintain a
plan (Box 7.1) (2). Another key factor to consider close working relationship. The practical needs of

101 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide
the knowledge users should also be prioritized 7.3.3.2 Metrics
when planning for reporting and dissemination
activities, and discussions should be initiated early. A variety of metrics can be used to measure the
The knowledge users may prefer summary-of- impact of published articles. Citation analysis is used
findings tables over lengthy narratives, or may be to measure how often a work is cited. One example
content with one approach for immediate results of a citation metric is the journal impact factor,
(e.g. a meeting or presentation), followed by a published in the Web of Science’s Journal Citation
more comprehensive written record at a later date. Reports, which measures the impact of a journal
Without a clear dialogue, these important details through its citation by subsequent authors (24).
and opportunities for engagement may be lost. Altmetrics are also important. These non-traditional
metrics include citations and downloads to web-
> 7.3.3 Dissemination activities and based scholarly articles, discussions on research
tools blogs, media coverage, citations to public policy
documents, and mentions on social networks
Rapid reviews aim to inform fast-moving policy such as Twitter or Facebook. The more hits from
processes; as such, practical use of the findings by these sources, the higher the Altmetric score (25).
the knowledge users will likely take priority over
academic publication or other broad dissemination 7.3.3.3 Scientific meetings and symposia
approaches. Rapid review producers may also
choose to disseminate research findings through Disseminating research findings at scientific or
publication in peer-reviewed journals, stakeholder professional meetings, conferences, and symposia
meetings or workshops, online summaries and is a way to reach large groups of knowledge users
databases, social media posts, video summaries, who may be interested in the research findings.
or e-mail distribution (15). These activities may There are a variety of ways to participate and
complement or be in addition to the specific needs present at meetings and symposia, including
of the policy- or decision-makers who requested posters, oral presentations, and participation
the review, but their impact on the uptake of in panel discussions. If review authors plan to
information can be limited (21, 22). disseminate their research this way, it is important
to prepare key messages that will be relevant
7.3.3.1 Publication of rapid reviews to the target audience. Scientific meetings and
symposia often have themes, or an organization
Publishing articles is a traditional approach to may have a certain focus based on its particular
knowledge translation. The “gold standard” of goals. It is important to understand the audience
journal publishing is to publish in a peer-reviewed and ensure that the meeting is the right place
journal, namely, a journal that asks individuals in to target the knowledge users. Finally, review
the research community (“peers”) to evaluate the authors must consider who the most credible
article as a way of validating the research before messenger is, and whether the authors are the
publication. Authors can determine whether a right persons to deliver the message. For example,
journal is peer-reviewed by checking the journal’s a policy-maker (who can place the evidence in
instructions to authors and its editorial statement, context with their policy expertise) may be a
typically found on the journal’s website. Authors better messenger than a researcher for a health
should evaluate any web-based journal publication systems audience.
carefully to ensure that it is a valid journal and
not from a “predatory publisher” (i.e. a company
that exploits or defrauds authors and readers by
promising reputable publishing platforms and
then failing to meet its promises) (23).

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 102
BOX 7.1. Essential questions for developing a research dissemination plan

?
To help disseminate your research, answers to the following questions will form the basis of a plan:

1 Why do you want to raise awareness of your research?

• To meet the urgent requirement of a knowledge user?
• To raise general awareness?
• To connect with other researchers?
• To generate national or international attention?
• To change policy or practice?
• To satisfy funders?

2 What is interesting about your findings?

(in other words, “Why should anyone care?”)

• What is novel or different?

• Is it a large study?
• Are the results contrary to previous evidence?
• What is the relevance?
• Why now?
• Is it a hot topic?
• Is it seasonal?
• Does your review tap into popular trends?

3 How might you generate interest in your findings?

Consider the following:

• Are you publishing in a journal?

• How does the journal generate awareness of papers?

4 Who will be interested? Consider the following audiences:

• General public
• Patients
• Health-care professionals
• Researchers
• Policy-makers, government
• Funders
• Corporations.

103 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide
5 Should I tailor the message to my audience?
• How can you make your findings interesting to target audiences?
• What are your key messages?
• Do you need simpler messages for the general public?
• How do these differ from messages for policy-makers, researchers?

6 What tools can you use to communicate?

What can be shared on social media?

• News releases
• Photos
• Infographics
• Video
• Podcasts
• Blogs.

7 Who can best help to deliver your messages? B

• Different team members may be good for different platforms (e.g.
television interviews, social media, blogging)
• Presenters can often be tailored to the audience (e.g. a policy-maker for
health system audiences, a researcher for a large research meeting)
• A health system stakeholder may be able to talk about your research
(e.g. a patient representative, a member of the public or a funding
agency spokesperson).

8 How will you measure success?

• Number of reads or downloads
• Citation metrics
• Altmetrics.

Other tips
• Use plain language
• Avoid jargon, technical terms
• Develop a short summary of research findings
• Tell a story
• Provide context
• Don’t overstate findings
• Inform your institution’s communications team in case they can help
disseminate your research.

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 104
 of health policy and systems research. Research
Relevant reporting into the dissemination of rapid reviews is limited.
Two studies of rapid review producers (15, 18)
guidelines should be used identified variation in research dissemination
approaches and tools. In some cases, public
in the development of rapid dissemination activities may be extremely limited.
For example, organizations may choose to post
review reports, to ensure a summary paragraph describing the research,
without disseminating a full report (18). Most rapid
comprehensive and transparent review producers (about 70%) chose to disseminate
their reports beyond the commissioning individual
documentation of the rapid or body (15). In deciding the dissemination strategy,
influencing factors that have been cited include
review process. the need for permission from the requester, legal
implications or sensitivity of the topic, and type
of approach used for the rapid review.

7.3.3.4 Traditional media and social media Although we have described some of the traditional
methods for research dissemination here, rapid
Traditional media and social media can be used reviews of health policy and systems research
to publicize research findings to patients and the may require specific dissemination strategies
general public, as well as to researchers, policy- to reach their target audiences and maximize
makers, and other audiences (26). Traditional impact. Some alternative methods to consider are
media include newspapers, radio, television, focus groups, public meetings, and open houses.
magazines, and online-only news sites. Social If an advisory board is informing the rapid review
media encompass online and mobile tools, process, its members may be able to suggest
such as Facebook, Twitter, and Instagram, where how to present findings in a way that will reach
users directly create, post, and share content. all potential knowledge users. If it is an expert
Both traditional media and social media can be part group, the advisory board may also assist with
of a broader dissemination strategy. By allowing directly disseminating the results of the rapid
review authors to link to health system trends review to interested individuals or groups.
in the news, these resources may be especially
useful when publishing the rapid review in a 7.4 CONCLUSION
journal or presenting the results at conferences.
Although producers of rapid reviews have
> 7.3.4 Special considerations for access to the same dissemination tools and
rapid reviews of health policy and channels as systematic reviews, they will need
systems research to prioritize the practical needs of the knowledge
user over traditional or academic approaches to
Knowledge translation strategies are universally dissemination. A checklist of essential questions
translatable to all forms of research, yet some to assist researchers in the development of a
considerations may be unique to rapid reviews dissemination plan is presented in Box 7.1.

105 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide
REFERENCES

1. Kelly SE, Moher D, Clifford TJ. Quality of conduct and reporting in rapid reviews: an exploration of
compliance with PRISMA and AMSTAR guidelines. Systematic Reviews, 2016, 5:79.

2. Straus S, Tetroe J, Graham I, eds. Knowledge translation in health care: moving from evidence to
practice. Oxford, UK, John Wiley & Sons, 2013.

3. Liberati A et al. The PRISMA statement for reporting systematic reviews and meta-analyses of
studies that evaluate healthcare interventions: explanation and elaboration. British Medical Journal,
2009, 339:b2700.

4. Nicholls SG et al. Reporting transparency: making the ethical mandate explicit. BMC Medicine,
2016, 14:44.

5. Moher D et al. Preferred reporting items for systematic review and meta-analysis protocols
(PRISMA-P) 2015 statement. Systematic Reviews, 2015, 4:1.

6. Shamseer L et al. Preferred reporting items for systematic review and meta-analysis protocols
(PRISMA-P) 2015: elaboration and explanation. British Medical Journal, 2015, 349:g7647.

7. Grimshaw J. A guide to knowledge synthesis. Ottawa, (ON), Canada, Canadian Institutes for Health
Research, 2010 (http://www.cihr-irsc.gc.ca/e/41382.html, accessed 29 January 2017).

8. Abrami PC et al. Issues in conducting and disseminating brief reviews of evidence. Evidence &
Policy: A Journal of Research, Debate and Practice, 2010, 6:371-389.

9. Plain Language Summary Tool. Austin, (TX), US, Center on Knowledge Translation for Disability
and Rehabilitation Research (KTDRR), 2017 (http://ktdrr.org/resources/plst/, accessed 29 Jan 2017).

10. Reader-friendly writing - 1:3:25. Ottawa, (ON), Canada, Canadian Health Services Research Foundation,
(http://www.cfhi-fcass.ca/migrated/pdf/communicationnotes/cn-1325_e.pdf, accessed 29 Jan 2017).

11. Eden J et al. Standards for Reporting Systematic Reviews. In: Eden J, Levit L, Berg A, et al, eds.
Finding What Works in Health Care: Standards for Systematic Reviews. Washington, (DC), US,
National Academies Press (US), 2011.

12. Kelly SE, Moher D, Clifford TJ. Defining Rapid Reviews: A modified delphi consesus approach.
International Journal of Technology Assessment in Health Care, 2016, 32:265-275.

13. The EQUATOR Network. Oxford, UK, UK EQUATOR Centre, (http://www.equator-network.org/,

accessed 13 June 2017).

14. Tricco AC et al. A scoping review of rapid review methods. BMC Medicine, 2015, 13:224.

15. Polisena J et al. Rapid review programs to support health care and policy decision making: a
descriptive analysis of processes and methods. Systematic Reviews, 2015, 4:26.

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 106
16. Ganann R, Ciliska D, Thomas H. Expediting systematic reviews: methods and implications of rapid
reviews. Implementation Science, 2010, 5:56.

17. The Joanna Briggs Institute Reviewers’ Manual 2015: Methodology for JBI Scoping Reviews.
Adelaide, Australia, The Joanna Briggs Institute, 2015 (https://joannabriggs.org/assets/docs/sumari/
Reviewers-Manual_Methodology-for-JBI-Scoping-Reviews_2015_v2.pdf, accessed 27 January 2017).

18. Abou-Setta AM et al. Methods for Developing Evidence Reviews in Short Periods of Time: A Scoping
Review. PloS One, 2016, 11:e0165903.

19. Hartling L et al. A taxonomy of rapid reviews links report types and methods to specific decision-
making contexts. Journal of Clinical Epidemiology, 2015, 68:1451-1462.e1453.

20. Hartling L et al. Fit for purpose: perspectives on rapid reviews from end-user interviews. Systematic
Reviews, 2017, 6:32.

21. Brinn MP et al. Mass media interventions for preventing smoking in young people. The Cochrane
database of systematic reviews, 2010:Cd001006.

22. Giguere A et al. Printed educational materials: effects on professional practice and healthcare
outcomes. The Cochrane database of systematic reviews, 2012, 10:Cd004398.

23. Beall J. Best practices for scholarly authors in the age of predatory journals. Annals of the Royal
College of Surgeons of England, 2016, 98:77-79.

24. Journal Citation Reports. Philadelphia, (PA), US, Clarivate Analytics, (http://clarivate.com/?product=journal-
citation-reports, accessed 20 January 2017).

25. What are Altmetrics? London, UK, Altmetric, (https://www.altmetric.com/about-altmetrics/what-

are-altmetrics, accessed 20 January 2017).

26. Mathelus S, Pittman G, Yablonski-Crepeau J. Promotion of research articles to the lay press: a
summary of a three-year project. Learned Publishing, 2012, 25:207-212.

107 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide
Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 108
8
IMPROVING THE UPTAKE
OF RAPID REVIEWS

Andrea C. Tricco, Roberta Cardoso, Sonia M. Thomas,

Sanober S. Motiwala, Shannon Sullivan,
Michael (Ryan) Kealey, Brenda Hemmelgarn,
Mathieu Ouimet, Laure Perrier, Sharon E. Straus
KEY POINTS

• Although rapid reviews can be helpful for health care decision-making,

policy-makers and health systems managers do not always commission
and use rapid reviews to inform their decisions.

• Barriers to the commissioning and use of rapid reviews include the belief that
the results of rapid reviews are not useful or valid, a lack of understanding
of how to identify and access relevant rapid reviews, a lack of skills to
assess or interpret rapid reviews, and organizational resistance to applying
new evidence.

• Researchers can facilitate the uptake of rapid reviews by developing

partnerships with policy-makers or health systems managers, and by
providing education about the validity and applicability of rapid review results,
as well as how to identify rapid reviews, and assess and interpret findings.

• In terms of the content of a rapid review report, the following elements will
promote uptake: a section on policy implications; a focus on the results and
interpretation (with less emphasis on the methods); presenting a summary
of the study results using a standardized format (e.g. summary‑of-findings
tables); targeting messages to key audiences; ensuring that the results
are tailored to the knowledge user of the review; and consistent reporting
of effect sizes (for quantitative reviews, such as those that include a
meta‑analysis or statistical combination of multiple studies).

• In terms of formatting a rapid review report, the following aspects will

promote uptake: preparing a one-page plain-language summary (i.e. research
brief) that includes key messages and the publication date (to indicate how
recently the review was performed); using white space to break up dense
text; and providing simple one-page tables.
8.1 INTRODUCTION
Contextualizing review findings
Knowledge synthesis products can help policy-
makers or health systems managers make and focusing on results and
decisions, by summarizing all available evidence
interpretation from the lens of
related to a particular question. According to some
decision-makers one such product, the rapid policy-makers or health system
review, is particularly helpful (1-3) because these
managers will improve uptake
reviews provide information in a timely manner
(4). However, the use of evidence (including of rapid review findings.
evidence provided in rapid reviews) to inform
decision-making processes varies widely (5-7).
not be aware of existing reviews or how to go
8.2 BARRIERS AND FACILITATORS about commissioning a review. Several other
TO THE UPTAKE OF factors can also affect the uptake of rapid
RAPID REVIEWS reviews. These can be thought of as factors that
promote (i.e. facilitators) or hinder (i.e. barriers)
Decision-makers who work at the health policy uptake. Factors from the policy-maker or health
and systems level face many challenges in systems manager perspective can be classified
incorporating evidence into decision-making. as attitudes, knowledge, skills, and behaviours,
For example, these individuals often lack the whereas factors from the rapid review producer
time and skills to search for evidence when perspective can be classified as skills and
faced with time-sensitive situations, and may behaviours (Box 8.1) (8, 9).

BOX 8.1. Barriers and facilitators to the uptake of rapid reviews for health care decision‑making

Barriers:

• belief that results of rapid reviews are not useful or valid;

• lack of understanding about how to identify and access relevant
rapid reviews;
• inability to assess or interpret rapid reviews;
• organizational resistance to implementing new evidence;
• lack of understanding about what evidence is required and how it can be used to
influence and constitute policy.

111 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide
BOX 8.1. Barriers and facilitators to the uptake of rapid reviews for health care decision‑making
(continued)

Facilitators:

• belief in the validity and applicability of rapid review results;

• awareness of the importance of rapid reviews;
• skills in finding, appraising, and interpreting rapid reviews;
• collaboration between policy-makers and the researchers who
produce rapid reviews;
• trust in the rapid review producer;
• embedding of policy-makers into the rapid review team;
• use of rapid response services;
• involvement of policy-makers in prioritizing rapid review topics;
• conducting workshops on how to identify rapid reviews and appraise their quality;
• forecasting when a decision will potentially be made by a policy-maker or health
systems manager;
• contextualizing the review findings that are specific
to the policy-maker’s current situation.

> 8.2.1 Attitudes or health systems managers’ freedom to make

the decision they desire; the belief that reviews
Positive attitudes towards rapid reviews and the assessing the effects of a policy or programme
belief that rapid reviews are useful will increase cannot be used to determine causality; and mistrust
the uptake of results by policy-makers or health of the results or disagreement with the authors’
systems managers. All types of knowledge interpretation. As well, the belief that a systematic
syntheses, including rapid reviews and systematic review is the gold standard, and that a rapid
reviews, can be done with varying levels of review should be used only under exceptional
quality. Specifically related to rapid reviews, circumstances, will hinder the uptake of rapid
producers must answer the time-sensitive needs reviews (11).
of decision-makers, and simultaneously ensure
that the scientific imperative of methodological > 8.2.2 Knowledge
rigour is satisfied (10). Acknowledging this,
and being aware that the trustworthiness of Policy-makers’ or health systems managers’
a rapid review depends on the methods used awareness that reviews are more useful for
and how transparently methods are reported, decision-making than primary studies is a facilitator
will facilitate uptake (11). In addition, uptake to uptake. In addition, knowing the types of
will increase when policy-makers or health situations where rapid reviews are the most
systems managers trust the researchers who useful will increase uptake. Such situations include
conducted the rapid review, either because the those requiring urgent policy and health systems
researcher has worked with the policy-maker or decisions, those requiring policy decisions at
health systems manager previously or because the local level, and those involving updates of
the researcher (or the researcher’s institution previous reviews or guidelines or simply getting
or research group) has a good reputation (12). a sense of the current literature (12). Knowing
that the rapid review is just one of several types
Attitudinal barriers to the uptake of rapid reviews of information used in decision-making will
include the perception that reviews dictate also promote uptake (12). In contrast, a lack of
decisions, thereby removing the policy-makers’ awareness of how to locate rapid reviews and

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 112
a lack of knowledge regarding their importance systems managers as members of the rapid
are barriers. review team so they can participate in various
steps of the review process (13-16). For example,
> 8.2.3 Skills the Canadian Agency for Drugs and Technologies
in Health embeds Liaison Officers in each
Training policy-makers or health systems managers province to engage on an ongoing basis with
to search the literature and appraise the quality policy-makers, clinicians, and other stakeholders,
of rapid reviews (7), and training rapid review which may increase uptake of evidence from
producers on how to contextualize the evidence research, including rapid reviews (19). Chapter
to meet decision-makers’ needs will also increase 5 provides guidance on approaches to engage
uptake. policy-makers or health systems managers in
the rapid review process.
> 8.2.4 Behaviours
In contrast, behavioural barriers to the uptake of
Collaborations and strong relationships between rapid reviews by policy-makers or health systems
researchers and policy-makers or health systems managers include resistance at the organizational
managers will facilitate the uptake of rapid reviews level to applying the rapid review results. Other
(12). There are numerous examples of “rapid barriers include contradictory findings across
response services”, whereby researchers respond reviews on the same topic, difficulty locating
to queries from policy-makers or health systems key messages in a review, lack of time, and lack
managers through rapid reviews (13-16) that of availability of relevant rapid reviews.
have provided significant utility to their recipients
(4). These rapid response services increase 8.3 CONSIDERATIONS IN WRITING
the relevance of rapid reviews, and facilitate THE RAPID REVIEW REPORT
the interpretation of rapid reviews by way of
the collaborative relationships that have been The way in which researchers present the results
established (12). Providing policy-makers or of a rapid review can increase uptake (Box 8.2)
health systems managers with timely access (8, 12, 20). For example, including a section on
to relevant rapid reviews when decisions need policy implications promotes uptake. A focus
to be made in a context where trust has been on the results and interpretation, with less
established between the rapid review producer emphasis on the methods, also promotes
and the policy-maker or manager facilitates uptake. uptake. A concise summary of the study results
in standardized form (e.g. summary-of-findings
Creating demand for rapid reviews by policy- tables) and consistent reporting of effect sizes
makers or health systems managers facilitates (for quantitative reviews, such as those that
the uptake of rapid reviews (17). This can be done include a meta-analysis or statistical combination
in a variety of ways, such as educating policy- of multiple studies) will also facilitate uptake of
makers to recognize the value and use of rapid results. Furthermore, presenting the evidence in
reviews, undertaking priority-setting activities to standardized tabular format and describing the
identify rapid review topics (E. Akl, unpublished strength of the evidence (i.e. methodological
data, 2016; (18)), conducting workshops on quality) will promote uptake (12). Finally, targeting
how to identify rapid reviews and appraise messages to different knowledge user audiences
their quality, forecasting when a decision will is also effective. More information on how to
potentially be made by a policy-maker or health prepare the report is provided in Chapter 7.
systems manager (i.e. identifying an “opportunity
window”), or embedding policy-makers or health

113 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide
BOX 8.2. Methods to increase the uptake of rapid reviews

Content

• focus on reporting and interpreting the results;

• frame the evidence in terms of policy implications;
• ensure consistency in the reporting of effect sizes of interventions;
• contextualize the findings of the rapid review that are specific
to the policy-makers’ and health systems managers’ current situation;
• target key messages to each key audience.

Format

• use ample white space with bullet points and simple tables;
• include a key messages section at the beginning;
• include a section on policy implications;
• focus on the results of the review and their interpretation,
with less emphasis on the methods;
• include a one-page plain language research brief with the key messages,
publication date, and logo of the funding agency.

The format of the review can also be pertinent to the summary is intended for (21). Figure 8.1
uptake. For example, a one-page plain language provides an example of a one-page summary with
summary that lists key messages and states the effective use of white space, for a rapid review
publication date (to indicate how recently the conducted for the World Health Organization.
review was performed) is helpful to knowledge As well, it may be helpful to develop different
users (Box 8.2) (8). Avoiding dense text through review formats for different types of policy
the judicious use of white space and providing decision-makers (22). Additional information on
simple one-page tables can also improve uptake. the format for the rapid review is available in
As well, contextualizing the review findings that Chapter 7, including other knowledge products
are specific to the policy-maker’s current situation that could be provided to the decision-makers
facilitates uptake, through methods such as the that are based on the review results.
SUPPORT summaries of systematic reviews,
which provides information pertaining to who

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 114
FIGURE 8.1. Research brief: Rapid scoping review of medical malpractice policies/
models/frameworks

Research Brief

Rapid Scoping Review of Medical Malpractice Policies/

Models/Frameworks

Summary What is the current situation?

The medical community and health  Litigation costs can range from 2.4% to 10% of health care spending.
systems are facing a malpractice crisis
with increasing litigation costs. We  The clinical specialty of obstetrics is under particular scrutiny for paying amongst
conducted a rapid scoping review to the highest litigation rates.
identify medical malpractice policies
and programs found to reduce legal What is the objective?
damages. Despite the enormous costs
associated with medical malpractice To complete a rapid scoping review to map the available evidence regarding
litigations, very few papers described medical malpractice models/frameworks/policies to control damages in obstetrical
such models. Most of the literature is procedures across all countries.
from the United States, which is likely
because of the large number of
medical malpractice claims that occur How was the review conducted?
per year. None of the included papers  Five-stage rapid scoping review framework was followed: 1) identifying the
originated from low to middle income
research question, 2) identifying relevant studies, 3) selecting studies, 4)
economy countries. Most reports were
informal discussion papers without charting the data, and 5) synthesizing and reporting the results.
formal evaluation. Favourable  MEDLINE, EMBASE, LexisNexis Academic, Legal Scholarship Network, Justis,
outcomes have been reported for no- LegalTrac, QuickLaw and HeinOnline were searched for publications in English
fault compensation of severe birth- from 2004 until June of 2015.
related injuries, patient safety
programs and apology laws.  All levels of screening and data collection were done in duplicate.

Implications What did the review find?

A number of medical malpractice  Forty-three articles were included. The majority (n=31) of the reports were
models for reducing litigation costs focused on the United States.
were identified. However, many were
 A number of initiatives were reported: (1) no-fault compensation system for
reported without a systematic
defined medical injuries, (2) safety program and practice guidelines for reduction
evaluation of programs and outcomes.
and mitigation of medical risks and errors, (3) specialized courts and alternative
Only 10 formal evaluations were
claim resolution for handling medical malpractice claims using a non-judicial
identified. Further research in this area
system, (4) communication and resolution strategies to reach a mutual
is warranted.
agreement on dispute and fair compensation outside the court-room; (5) caps on
compensation and attorney fees, (6) alternative payment system and liabilities to
reduce the burden of liability pressure and financial burden of claims payment
Reference: Cardoso R, Zarin W,
(7) limitations on litigation to control the type and amount of medical malpractice
Nincic V, et al. Rapid Scoping Review
of Medical Malpractice Policies in claims entering the system, and (8) multi-component models that include to a
Obstetrics. World Health Organization; combination of the aforementioned strategies.
2015.  No-fault systems for severe birth-related injury in Florida and Viriginia were
Available from: World Health reported to reduce tort premiums, apology laws as a communication and
Organization resolution strategy were found to decrease compensation payments, and many
of the patient safety and practice guidelines reduced medical errors and
malpractice claims. Caps on compensation and attorney fees had inconclusive
For more information, please results.
contact Dr. Andrea Tricco:
[email protected]

Funded by the World Health Organization

Source: Cardoso et al., 2015 (26)

115 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide
8.4 CONCLUSION helping policy-makers and health systems
managers to tease out the differences in
There are many ways in which producers of contradictory results across rapid reviews.
rapid reviews can enhance the uptake of results,
such as: When writing the report of a rapid review, authors
can implement several strategies in terms of
fostering collaborations with policy-makers and formatting and presentation of results to promote
conducting workshops on how to identify reviews uptake. The bulk of the report should focus on the
and appraise their quality; results and interpretation, with less emphasis on
the methods. However, the methods should be
forecasting when a decision will potentially be transparently reported and the use of a methods
made (i.e. identifying an “opportunity window”) and appendix may provide interested readers with
providing the review in time for decision‑making; the methodological details required. A section
on policy implications should be included. Use of
creating demand for rapid reviews and establishing plain language, avoidance of dense text through
rapid response services whereby researchers judicious use of white space, and targeting of
respond to queries posed by policy-makers or messages for key audiences may increase uptake.
health systems managers; As well, contextualizing the review findings that
are specific to the policy-maker’s current situation
conducting priority-setting activities related will improve uptake (23), in addition to an effective
to selection of topics for rapid reviews and knowledge translation strategy. Box 8.3 gives
embedding policy-makers on the rapid review an example of a rapid review applying several
team; working with policy-makers and managers of these strategies.
during preparation of the review to ensure that
it will be relevant to their decision-making;

BOX 8.3. Conducting a rapid review with maximum uptake in mind

You lead a small research team, supporting a non-government organization responsible

for providing health services during natural disasters. Your country has just been hit by an
earthquake, and the Red Cross has asked your organization to coordinate efforts at the national
level. Your director wants you to make sure the best-known approach is used, and time is
of the essence. With much effort, your team has established evidence-informed decision-
making as the norm in the organization, but this has always been for decisions with enough
lead time to conduct a full systematic review. How can you locate and synthesize what you
need in such a short time, while maintaining a systematic approach?

Luckily you have spent time developing a strong relationship with the leader who will coordinate
the relief efforts, and you know this will speed up your review. You engage her right away,
and include two people from her team as members of the review team. Not only will they
ensure the right questions are asked, and that the results will be relevant, but they will also
work with your team at every step of the review. They provide a welcome resource, and more
importantly, these knowledge users will be able to provide input into the rapid review process,
which you know will be essential to ensure the final product is on target, and on time.

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 116
BOX 8.3. Conducting a rapid review with maximum uptake in mind (continued)

You restrict your search to just two literature databases (PubMed and Embase), and identify
three reviews relevant to your research question. Your primary interest is in models of
coordination between entities funding or delivering health services in humanitarian crises in
a low-income country. One of these reviews is particularly helpful, and describes five models
for coordination between entities, whether during a crisis or afterwards (24):

• cluster approach: uses a framework of agreed objectives between agencies to avoid

resource gaps;

• the 4Ws (“Who is Where, When, doing What”) mapping tool: focuses on mental health;
coordinates responsibilities across agencies;

• sphere project: provides guidance to humanitarian responders in all sectors (not limited
to health);

• the 5×5 model: focuses on mental health; provides five skills and implementation rules;

• model of information coordination: uses internet and ship-to-shore teleconferences to

liaise between agencies.

Since none of the models have been rigorously tested, you decide to go with the cluster
approach, which was the most commonly used model reported in the literature. You and
a member from your colleague’s team, who participated in the review, write a brief report
with a one-page summary, a judicious use of white space, and a focus more on results
and implications, than on the methods applied. Your colleague then works closely with the
Red Cross to implement a plan of action to fund and deliver health services during this
humanitarian crisis.

Some of these recommendations will be easier substantial effort to inform policy-makers or health
to apply than others. One approach that is within systems managers about the value of reviews,
researchers’ control is trying to create demand and this process must highlight that reviews are
for rapid reviews by identifying opportunities to not meant to rigidly dictate decisions. Rather,
respond to the questions posed by policy-makers a rapid review is a tool that can be used to inform
or health systems managers. As well, it should the decision-making process, with other types
be feasible to work closely with policy-makers of evidence (including experiences, preferences,
before the project begins, to ensure that the and values) also influencing the decision. In some
resulting review will be relevant. In addition, cases, a primary study that is specific to the
tailoring key messages to specific stakeholders context of the policy-maker or health system
is typically an easy task. manager may be preferred over a rapid review,
especially if the primary study was not included
Other recommendations will be more challenging in the review, because of specific contextual
to implement. For example, it takes years factors that are not relevant in other settings.
to establish trusting relationships between Programmes that allow rapid reviews to be
researchers and policy-makers or health systems conducted alongside primary studies (25) will
managers. This is especially given that there likely advance the uptake and relevance of rapid
is a high turn-over of policy-makers and health reviews for health policy and systems decision-
systems managers in their workforce. It also takes making.

117 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide
REFERENCES

1. Hailey D. A preliminary survey on the influence of rapid health technology assessments. International
Journal of Technology Assessment in Health Care, 2009, 25:415-418.

2. McGregor M, Brophy JM. End-user involvement in health technology assessment (HTA)

development: a way to increase impact. International Journal of Technology Assessment in
Health Care, 2005, 21:263-267.

3. Zechmeister I, Schumacher I. The impact of health technology assessment reports on decision

making in Austria. International Journal of Technology Assessment in Health Care, 2012, 28:77-84.

4. Jayakumar KL et al. Evidence synthesis activities of a hospital evidence-based practice center

and impact on hospital decision making. Journal of Hospital Medicine, 2016, 11:185-192.

5. Harker J, Kleijnen J. What is a rapid review? A methodological exploration of rapid reviews in Health
Technology Assessments. International journal of evidence-based healthcare, 2012, 10:397-410.

6. Polisena J et al. Rapid review programs to support health care and policy decision making: a
descriptive analysis of processes and methods. Systematic Reviews, 2015, 4:26.

7. Peterson K et al. User survey finds rapid evidence reviews increased uptake of evidence by
Veterans Health Administration leadership to inform fast-paced health-system decision-making.
Systematic Reviews, 2016, 5:132.

8. Tricco AC et al. Barriers and facilitators to uptake of systematic reviews by policy makers and
health care managers: a scoping review. Implementation Science, 2016, 11:4.

9. Oliver K, Lorenc T, Innvaer S. New directions in evidence-based policy research: a critical analysis
of the literature. Health Research Policy and Systems, 2014, 12:34.

10. Garritty C et al. Cochrane Rapid Reviews Methods Group to play a leading role in guiding the
production of informed high-quality, timely research evidence syntheses. Systematic Reviews,
2016, 5:184.

11. Kelly SE, Moher D, Clifford TJ. Expediting evidence synthesis for healthcare decision-making: exploring
attitudes and perceptions towards rapid reviews using Q methodology. PeerJ, 2016, 4:e2522.

12. Hartling L et al. EPC Methods: AHRQ End-User Perspectives of Rapid Reviews [research white
paper]. Rockville, (MD), US, Agency for Healthcare Research and Quality (US), 2016 (16-EHC014-EF).

13. Odendaal A. South African initiative for systematic reviews on health policies and systems (SAI).
South Africa, South African Medical Research Council, 2016 (http://www.mrc.ac.za/healthsystems/
sai.htm, accessed 18 May 2017).

14. Keown K, Van Eerd D, Irvin E. Stakeholder engagement opportunities in systematic reviews:
knowledge transfer for policy and practice. Journal of Continuing Education in the Health
Professions, 2008, 28:67-72.

Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide 118
15. Greenhalgh T et al. Achieving Research Impact Through Co creation in Community Based Health
Services: Literature Review and Case Study. Milbank Quarterly, 2016, 94:392-429.

16. Jagosh J et al. Uncovering the benefits of participatory research: implications of a realist review
for health research and practice. Milbank Quarterly, 2012, 90:311-346.

17. Langlois EV et al. Enhancing evidence informed policymaking in complex health systems: lessons
from multi-site collaborative approaches. Health Research Policy and Systems, 2016, 14:20.

18. Pantoja T, ed. Approaches for prioritizing questions for systematic reviews in health policy and
systems research. Third Global Symposium on Health Systems Research; 2014; Cape Town,
South Africa.

19. Polisena J et al. Rapid Review Summit: an overview and initiation of a research agenda. Systematic
Reviews, 2015, 4:111.

20. Perrier L et al. Using a systematic review in clinical decision making: a pilot parallel, randomized
controlled trial. Implementation Science, 2015, 10:118.

21. Is general practice effective? Switzerland, Swiss Tropical and Public Health Institute, 2011 (http://
www.supportsummaries.org/support-summaries/show/is-general-practice-effective, accessed
10 April 2017).

22. Pawson R, ed. Evidence-based policy: a realist perspective. London, UK, Sage Publications
Ltd., 2006.

23. Bermudez-Tamayo C et al. Priorities and needs for research on urban interventions targeting
vector-borne diseases: rapid review of scoping and systematic reviews. Infectious diseases of
poverty, 2016, 5:104.

24. Lotfi T et al. Coordinating the Provision of Health Services in Humanitarian Crises: a Systematic
Review of Suggested Models. PLoS Currents, 2016, 8.

25. Moher D, Stewart L, Shekelle P. All in the Family: systematic reviews, rapid reviews, scoping
reviews, realist reviews, and more. Systematic Reviews, 2015, 4:183.

26. Cardoso R et al. Rapid scoping review of medical malpractice policies in obstetrics. Toronto,
(ON), Canada, BreaKThrough, Knowledge Translation Program, Li Ka Shing Knowledge Institute,
St. Michael’s Hospital, 2015 (http://www.afro.who.int/en/south-africa/south-africa-publications.
html, accessed 22 May 2017).

119 Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide
RAPID REVIEWS TO STRENGTHEN
HEALTH POLICY AND SYSTEMS:
A PRACTICAL GUIDE
EDITED BY:
ANDREA C. TRICCO, ETIENNE V. LANGLOIS, SHARON E. STRAUS

Policy-makers require valid evidence to support time-sensitive decisions regarding the coverage, quality,
efficiency, and equity of health systems. Systematic reviews and other types of evidence syntheses are
increasingly employed to inform policy-making and produce guidance for health systems. However, the time
and cost to produce a systematic review is often a barrier to its use in decision-making. Rapid reviews are
a timely, and affordable approach that can provide actionable and relevant evidence to strengthen health
policy and systems. This Practical Guide explores different approaches and methods for expedited synthesis
of health policy and systems research, and provides guidance on how to plan, conduct, and promote the
use of rapid reviews, while highlighting key challenges including their application in low- and middle-income
countries. Our proposed solutions will help provide policy-makers and health systems managers with strategic
evidence to make crucial decisions about health systems’ response in emergency situations, as well as in
routine decision-making.

World Health Organization ISBN 978-92-4-151276-3

Avenue Appia 20
CH-1211 Genève 27
Switzerland
[email protected]
http://www.who.int/alliance-hpsr