Heart & Lung 43 (2014) 453e461

Contents lists available at ScienceDirect

Heart & Lung

journal homepage: www.heartandlung.org

Research Methods

Methods for knowledge synthesis: An overview

Robin Whittemore, PhD, APRN, FAAN, Ariana Chao, MSN, RN, FNP-BC,
Myoungock Jang, MSN, RN *, Karl E. Minges, MPH, Chorong Park, MSN, RN
Yale University School of Nursing, 400 West Campus Drive, Orange, CT 06477, USA

a r t i c l e i n f o a b s t r a c t

Article history: Objectives: To highlight recent advances in knowledge synthesis methods and reporting guidelines.
Received 13 March 2014 Background: Knowledge synthesis is critical to advancing practice, research, and policy, but synthesizing
Received in revised form knowledge from an often-heterogenous body of literature is challenging.
2 May 2014
Methods: A review of knowledge synthesis methods and reporting guidelines for health sciences research
Accepted 2 May 2014
was completed using Google Scholar, Medline, CINAHL, and PsycInfo. Relevant information was critiqued
Available online 8 July 2014
and summarized for applicability to health science and practice.
Results: Recent advances and guidelines pertaining to systematic reviews, meta-analysis, qualitative
Keywords:
Systematic review
synthesis, mixed studies reviews, integrative reviews, scoping reviews, RE-AIM reviews, and umbrella
Meta-analysis reviews are discussed and examples of the application of each method to cardiopulmonary research are
Metasynthesis provided. Methods of quality appraisal are also presented.
Mixed studies reviews Conclusions: Advancements in knowledge synthesis and reporting guidelines enhance the quality, scope,
Integrative review and applicability of results; thus improving health science and clinical practice, and advancing health
policy.
Ó 2014 Elsevier Inc. All rights reserved.

Introduction enhance appropriate and accurate reporting of individual studies,

to determine an appropriate review method when undertaking a
While knowledge synthesis e a summary of all pertinent studies review of the literature, to consider quality and bias of studies, and
on a specific question or topic1 e is critical to advancing practice, to interpret reviews.2 Thus, the purpose of this manuscript is to
research, and policy, synthesizing knowledge from a heterogenous highlight recent advances in knowledge synthesis methods and
body of literature in a clear and accurate manner can be chal- reporting guidelines for reviews and individual studies.
lenging. Searching the literature for all relevant studies, assessing Searches in Google Scholar, Medline, CINAHL, and PsycInfo were
quality of studies, and synthesizing heterogenous data are some of undertaken to identify publications and websites on knowledge
the methodological challenges researchers experience when synthesis methodology and reporting guidelines for reviews and
undertaking a systematic review of the literature. individual studies. The reference lists of relevant publications and
Established knowledge synthesis methods include the system- websites were also reviewed for additional information. An over-
atic review, meta-analysis, qualitative synthesis, and integrative view of each knowledge synthesis method, guidelines for knowl-
review. More recently, methods on mixed-studies reviews, scoping edge synthesis methods, and examples from cardiopulmonary
reviews, reviews that examine efficacy and generalizability research are presented. In addition, guidelines on the reporting of
(i.e., RE-AIM review), and reviews of systematic reviews (i.e., individual studies and systematic reviews are discussed, as well as
umbrella reviews) have been proposed. These methods and opportunities for further methodological development of knowl-
guidelines for reporting results from reviews, as well as individual edge synthesis techniques.
studies, continue to be revised to enhance the rigor of knowledge
synthesis methods and the applicability and transparency of find-
ings to practice and policy. Clinicians and scientists benefit from Overview of knowledge synthesis methodology
information about the different types of synthesis methods and
current resources on reporting guidelines. This knowledge helps to All knowledge synthesis methods must include a systematic and
auditable approach to assure that individual studies have trust-
worthy findings and that the synthesized findings accurately
* Corresponding author. Tel.: þ1 203 785 2389. represent the aggregation or synthesis of individual studies.3
E-mail address: [email protected] (M. Jang). Common to all knowledge synthesis methods are: a) an explicit

0147-9563/$ e see front matter Ó 2014 Elsevier Inc. All rights reserved.
http://dx.doi.org/10.1016/j.hrtlng.2014.05.014
454 R. Whittemore et al. / Heart & Lung 43 (2014) 453e461

aim; b) development of a methodological protocol; c) compre- statement,22 and it provides an explicit and systematic approach
hensive search strategies to find relevant research articles; d) a that researchers may use to conduct a review and summarize
method or tradition of evaluating quality and potential risk of bias findings.19 The PRISMA Statement comprises a 27-item checklist
in individual studies; and e) explicit data collection and synthesis and a four-phase flow diagram.21 The report emphasizes the need
procedures.3e5 Presentation of the results varies depending on the to assess risk of bias in individual studies and risk of bias across
specific method and the underlying assumptions of the method. studies that may affect the cumulative evidence.19
Gough and colleagues2 have articulated a differentiation in review An example of a systematic review in heart failure is included in
methods that either use aggregative logic or configuring logic, Table 3. The purpose of this review was to evaluate different types
which subsequently affects the presentation of results. Reviews of treatment for depression in adults with heart failure.6 Six
that include homogenous empirical data in order to determine different types of treatment were evaluated with some demon-
average effect sizes or numerical counting are based in aggregative strating moderate to strong evidence of efficacy and others with
logic (e.g., some systematic reviews, meta-analyses), and results are inconclusive or inconsistent results. This review begins to high-
typically presented as numerical data and figures. In contrast, light the types of treatment that may be beneficial to prescribe in
reviews that are more exploratory and seek variation and clinical practice as well as the types of treatment that need more
complexity of phenomenon are based on configuring logic (e.g., research.
qualitative synthesis, mixed-studies review) with results presented
as a conceptual model or narrative text. Meta-analysis

Guidelines for knowledge synthesis methods Meta-analysis is a type of systematic review that combines ev-
idence using statistical methods to determine overall effects and
Methodological guidelines for systematic reviews, meta- magnitude of effect size.23 This technique can be used to answer
analysis, qualitative synthesis, and mixed-studies reviews are questions about the convergence, divergence, and robustness of
available. Guidelines specify the type of study design required for study results, statistical significance of an effect with increased
that type of review, search strategies and data collection methods, power, development of a more precise estimate of effects, and ex-
quality appraisal tools, data extraction and analysis procedures, and amination of moderators of effects (i.e., whether they are consistent
reporting recommendations. Guidelines for systematic reviews and across subgroups and conditions).24,25 The objective of these
meta-analyses are well developed and provide detailed informa- studies is to present a balanced and impartial summary of existing
tion and checklists to conduct a rigorous review and the necessary research, and numerous strategies have been designed to increase
information to include in the publication of a review method. In the quality of findings.
contrast, guidelines for qualitative syntheses, mixed studies re- There are a variety of unique features and considerations to think
views, and umbrella reviews are emerging and provide direction to about when conducting a meta-analysis to help ensure the quality of
conduct the review and synthesize text, however there is no findings. First, it is suggested that researchers assign quality scores
consensus on the methodological approach at this time. to each individual study included in the analysis. Though numerous
One of the major challenges for all review methods is the tools have been developed,26,27 the most well-established quality
appraisal of the quality of studies included in the review. Assess- appraisal tool is by the Cochrane Collaboration, which recommends
ment of the quality of individual studies is difficult and potentially assessing the risk of bias of each individual study.24 Second, to assess
biased, and with the exception of the meta-analysis, it is unclear the impact of low quality studies and sample size, it is
how to factor in the quality of an individual study into the review recommended to include a sensitivity analysis.28,29 Techniques have
analysis. Nonetheless, quality standards for all review methods also been developed for calculating effect sizes including odds ratios
have been proposed. and risk ratios, and assessing publication bias using funnel plots29,30
Resources for reporting guidelines and review methods are and Egger’s regression tests.25,31 Researchers can use forest plots to
provided in Tables 1 and 2 and examples of different types of provide a pictorial presentation of the results and can use Cochrane’s
reviews conducted on heart failure are included in Table 3. Each of Q to test the heterogeneity of the studies and to guide analysis
the review methods will be briefly discussed. decisions (e.g., using weighted averages or a fixed or random effects
model).32,33 Metaregression can be utilized to further explore these
Systematic review heterogenous factors.34 Despite the numerous choices researchers
have when conducting meta-analyses, guidelines are essential to
A systematic review is a method used to combine evidence of help researchers and clinicians conduct a high quality meta-analysis
multiple studies by identifying relevant research, appraising study and assess the comprehensiveness and completeness of the review.
quality, and summarizing findings.17,18 It utilizes explicit, system- Guidelines have been established for meta-analyses of both
atic, and transparent methods to minimize bias and provide reliable randomized controlled trials and observational studies.
findings and evidence.5 Systematic reviews can be conducted using Researchers, reviewers, and editors often use these guidelines as
a statistical analysis (if study designs and outcomes are homoge- standards for publication. The guideline to improve the reporting of
nous) or a narrative analysis. Regardless of analytic approach, meta-analyses of randomized controlled trials is the same guide-
synthesizing the results of individual studies that includes in- line used for systematic reviews, PRISMA, as previously discussed.
terventions with multiple components or different measures poses The Meta-Analysis of Observational Studies in Epidemiology
additional challenges. Poor quality and inconsistent reporting of (MOOSE) checklist has been created for those conducting a meta-
the systematic review may diminish their usefulness to make sci- analysis of observational studies.35
entific inferences and conclusions.19,20 An example of a meta-analysis in the heart failure literature is
To enhance the standardization of reporting in systematic provided in Table 3. The purpose of this review was to evaluate the
reviews, the Preferred Reporting Items for Systematic Reviews and benefits and harms of cardiac resynchronization therapy (CRT) in
Meta-Analyses (PRISMA) has been recommended to improve the patients with advanced heart failure and those with less symp-
transparency of reporting systematic reviews and enhance tomatic disease.8 As can be seen by this example, the study design,
consistency across systematic reviews.21 The PRISMA statement the interventions, and the outcomes were very homogenous. A
evolved from the Quality of Reporting Meta-analyses (QUOROM) more focused research question allows for the determination of an
 R. Whittemore et al. / Heart & Lung 43 (2014) 453e461 455

Table 1
Guidelines for knowledge synthesis methods and reporting guidelines.

Review type Purpose Examples of reporting Content of the guideline Quality appraisal
guideline
Systematic review To synthesize the results of Preferred Reporting Items for 4-Phase flow diagram and 27-item Risk of bias in individual studies.
homogenous Systematic Reviews and checklist. Risk of bias across studies that
experimental OR non-experimental Meta-analyses (PRISMA) Sections include: Title, abstract, may affect the cumulative evidence
research using a narrative analysis introduction,
methods, results, & discussion
Meta-analysis of To synthesize the effect of Cochrane Handbook for Handbook with sections that include: Risk of bias in individual studies
RCT homogenous Systematic Background, method, results, conclusion, (sequence generation, allocation
interventions evaluated by RCT using Reviews of Interventions and discussion4-Phase flow diagram concealment, blinding participants,
established statistical procedures PRISMA and 27-item checklist. personnel and outcome assessors,
Sections include: incomplete outcome data, selective
Abstract, introduction, outcome reporting, other sources of
methods, results, & discussion bias)
Meta-analysis of To synthesize the results of Meta-analysis of observational 35-Item checklist sections include: Assessment of study quality
observational homogenous observational studies in epidemiology background, search strategy, methods, (blinding of quality assessors;
study studies using established (MOOSE) results, discussion & conclusion stratification or regression on
statistical procedures possible predictors of study results)
Qualitative To synthesize the results of Enhancing transparency 21-Item checklist that includes: No well-established criteria
synthesis qualitative in reporting the synthesis of Introduction, methods, literature although
research using a narrative analysis qualitative research (ENTREQ) search, appraisal, and synthesis) numerous guidelines are available
Mixed studies To synthesize the results of None N/A No well-established criteria
review experimental although
and non-experimental research using scoring system for mixed studies
a statistical approach and/or a reviews proposed (qualitative,
narrative quantitative experimental,
analysis with integration of results quantitative observational, and
mixed methods criteria)
Integrative review To synthesize the results of research None N/A No well-established criteria
or theory using a narrative analysis
Scoping review To map the key concepts and None N/A No well-established criteria
evidence
of a particular phenomenon
RE-AIM review To evaluate and synthesize the reach, RE-AIM framework 21-Item validated extraction tool. No well-established criteria
efficacy, adoption, implementation, Sections
and maintenance of interventions include: Reach, efficacy, adoption,
implementation, and maintenance
Umbrella review To summarize results from None N/A No well-established criteria
systematic
reviews on a topic

effect size of a specific intervention on a select outcome when more popular methods of meta-ethnography, metasynthesis, and
applied to a specific patient population. Guidelines for the use of meta-grounded theory will be briefly discussed.
CRT in patients with different grades of heart failure could be The meta-ethnography has been the most widely cited quali-
specified. tative synthesis method and comprises an inductive and interpre-
tive approach to qualitative data analysis.12,44 It involves theoretical
sampling and reciprocal translations of findings from each primary
Qualitative synthesis study into the overall synthesis, which can then be presented
textually and graphically via summary tables, models, or dia-
The purposes of conducting a synthesis of primary qualitative grams.45 The meta-ethnography often results in new insights to a
studies include to: develop a deeper understanding of the phe- phenomenon, including the generation of new research questions
nomenon under investigation and uncover previously hidden pat- and theories, or indicates a point where saturation has been
terns, generate theory, explore the barriers, facilitators, reached.
perceptions, and attitudes from the participants’ perspective, The metasynthesis is another frequently used method to syn-
identify emerging areas of practice and research, and reveal reasons thesize qualitative studies. While there is a lack of consensus about
for variation in complex interventions and clinical trials.36e38 The the technique’s methods,46 and its somewhat ambiguous and
number of published syntheses of qualitative studies continues to interchangeable use,38,47 the umbrella term we are referring to here
increase,39,40 and there is diversity and disagreement as to the is the qualitative metasynthesis method as provided by Sande-
number of approaches one may take to conduct a qualitative syn- lowski and Barroso.48 Qualitative metasynthesis is a systematic
thesis, ranging from four to nine synthesis methods that span both approach that integrates results from a number of different, but
interpretative and integrative paradigms.41e43 Ultimately, the inter-related qualitative studies to synthesize research findings and
choice of which method to employ varies on several factors, contribute to a new higher conceptual and contextual under-
including the question and purpose of the synthesis, number of standing of the phenomenon of interest.46,48
published studies, and knowledge and expertise of the team con- Meta-grounded theory methodology, also known as “grounded
ducting the synthesis,42 as well as the researcher’s epistemological formal theory,” involves an inductive approach that uses constant
stance, methodology of primary qualitative studies, and whether comparative and theoretical sampling to analyze primary grounded
one seeks to describe or synthesize results.41 While no method to theory studies. It often informs higher-level insight into the processes
synthesizing qualitative research has become well-established, the of a phenomenon and the development of new theories.47,49,50
456 R. Whittemore et al. / Heart & Lung 43 (2014) 453e461

Table 2 heart failure.9 By combining the results of several qualitative studies,

Resources for synthesis methodology. an understanding of the physical and psychosocial challenges
Review type Resources for method unique to the older adult are identified which has important clinical
Systematic review Handbook.cochrane.org84 and research implications. Improved clinical assessment and/or
PRISMA21 (http://www.prisma-statement.org) intervention development could result from this type of review.
Meta-analysis of RCT handbook.cochrane.org84
Meta-analysis of handbook.cochrane.org84
Mixed studies review
observational study
Qualitative synthesis Barnett-Page and Thomas,41 2009
Centre for Reviews and Dissemination,95 2009 Mixed studies review is a literature review that concurrently
Dixon-Woods et al,44 2007 examines qualitative, quantitative and/or mixed methods primary
Noblit and Hare,12 1988 studies.67 The term mixed studies review is often used inter-
Sandelowski and Barroso,48 2006
Kearney,50 1998
changeably with integrative review, mixed methods review, mixed
Mixed studies review Heyvaert et al,96 2013 methods (research) synthesis, or synthesis of results of quantitative
Pluye et al,67 2009 and qualitative studies.67 Pluye and colleagues67 have suggested
Sandelowski et al,97 2006 the term ‘mixed studies review’ be used for reviews that include
Integrative review Whittemore and Knafl,15 2005
quantitative, qualitative, and/or mixed methods study designs.
Scoping review Arksey and O’Malley,74 2005
Armstrong et al,98 2011 A mixed studies review aims to address the same overlapping or
Levac et al,72 2010 complementary review questions3 by combining the strengths of
http://hlwiki.slais.ubc.ca/index.php both qualitative and quantitative studies. Thus, a mixed studies
/Scoping_reviews99 review can provide robust insight to a complex phenomenon67 as
RE-AIM review http://www.re-aim.org/index.html100
http://www.re-aim.org/resources_and_tools
well as development of an intervention by answering “what is it
/measures/literaturecoding.pdf101 about this kind of intervention that works, for whom, in what cir-
Umbrella review http://joannabriggs.org/assets/docs/sumari/ cumstances, in what respects, and why.”68
ReviewersManual-Methodology-JBI_Umbrella However, there is a lack of consensus for guidelines to plan,
%20Reviews-2014.pdf69
design, and assess mixed studies reviews. Depending on the review
handbook.cochrane.org84
question(s), a mixed studies review can be conducted with a
diverse combination of different types of primary studies (quali-
tative, quantitative-observational, quantitative-experimental, and
Although searching for primary qualitative studies can be a mixed method studies) and with varied ways of integrating those
difficult task, several search strategies exist to locate and retrieve findings. Due to the nature of mixed studies review, it is challenging
qualitative studies. These approaches use database filters, and to develop a well-specified guideline or criteria for appraising the
maximizing sensitivity and specificity algorithms to develop methodological quality of a mixed studies review. The Joanna
optimal search strategies for key databases.51e54 Briggs Institute (JBI), a non-profit organization focused on evidence
Identification of inclusion and exclusion criteria is contingent based practice, has developed a manual that contains guidelines
upon the qualitative synthesis method that is selected. Exclusion on about methods to conduct mixed methods reviews.69 The manual
the basis of quality is also a consideration with some qualitative also contains guidelines of essential components to include when
techniques. Indeed, the decision to apply quality appraisal guide- reporting results from a mixed studies review. To our knowledge,
lines to primary qualitative studies is one of contention.40,55e57 only one guideline for evaluating quality of primary research,
While many guidelines and criteria exist for assessing qual- which is unique to mixed studies reviews has been developed thus
ity,10,44,58e60 none have reached consensus as the preferred far.67 The scoring system is straightforward and simple (having a
approach. Further, it has not yet been determined how to incor- total of 15 quality criteria within four types of primary studies),
porate quality scores into reporting of qualitative synthesis however this quality appraisal tool is limited because it mainly
techniques. focuses on randomized controlled studies and its criteria were not
While the debate to include quality appraisal guidelines applied derived from usual quality appraisal standards for systematic
to primary qualitative studies continues, some strategies may be reviews such as QUOROM and MOOSE. More work is required in
suggested to enhance the quality of qualitative syntheses. Much in developing a systematic guideline and quality appraisal criteria for
the same way that researchers conducting primary qualitative mixed studies reviews.
studies achieve methodological rigor through trustworthiness An example of a mixed studies review in the heart failure
(credibility, transferability, dependability, and confirmability),61 literature is provided in Table 3. The purpose of this study was to
researchers conducting a qualitative synthesis should report describe caregivers’ activities contributing to self-care in adults
transparency in the methods used to synthesize primary studies. with heart failure.11 The quantitative data were used to highlight
One example of a reporting tool that incorporates transparency is the operationalization of the caregiver activities and the relation-
entitled “Enhancing Transparency in Reporting the Synthesis of ship between the caregiver activities and patient outcomes. The
Qualitative Research (ENTREQ).”62 This guideline uses a 21-item qualitative data highlighted the caregiver activities and varying
checklist that entails the appraisal of key aspects of the synthesis, responses to the role. The combination of quantitative and quali-
including introduction, methods, and literature search. While tative data provides complementary and in-depth perceptions of
several other guidelines exist for the reporting of qualitative syn- caregivers of adults with heart failure. Because this mixed studies
theses reviews,62e65 key considerations encompassed in most review focused on descriptive data, results have the potential to
quality appraisal guides include an audit trail, member checking, inform clinical assessment and/or intervention development.
and transparency of methodology. Other comprehensive guides to
manage, appraise, analyze, and synthesize qualitative research are Integrative review
available.41,63,66
An example of a metasynthesis in the heart failure literature is Integrative reviews are a category of knowledge synthesis
provided in Table 3. The purpose of this study was to synthesize methods that include a broad approach and sampling frame that
results of qualitative studies on how older people live with chronic can include empirical or theoretical literature, or both, depending
Table 3
Application of knowledge synthesis methods to heart failure (HF).

Method Study aim Method/reporting guidelinesNumber Main findings Quality appraisal Clinical implications
Author, year of publications inclusion Criteria
Systematic review To evaluate the effects of PRISMA Six types of interventions: (1) selective Qualitative assessmentTool Evidence supports that pharmacology,
Woltz et al,6 2012 interventions on depression N ¼ 23 serotonin reuptake inhibitors; (2) an for Quantitative CAM, and exercise may improve
in adults with HF. Experimental or erythropoiesis-stimulating agent; (3) Studies depression.
quasiexperimental designs exercise; (4) disease management programs; (QATQS)7 Insufficient/inconsistent
with HF patients (NYHA class (5) complementary and alternative medicine evidence to support the development
II and III HF) and outcome (CAM); and (6) a multimodal intervention of of clinical guidelines for the treatment
of depression cognitive behavioral therapy and exercise. of depression in persons with HF.

Meta-analysis To assess the benefits PRISMA (not explicitly stated in In patients with milder symptoms of heart Cochrane Risk of Bias Tool7 CRT is beneficial for patients with
Al-Majed et al,8 2011 and harms of cardiac manuscript but required by failure (class I and II), CRT improved LVEF reduced LVEF, symptoms of HF,
resynchronization therapy journal) and reduced all-cause mortality and HF and prolonged QRS, regardless of
(CRT) in patients with N ¼ 25 hospitalization without improving functional NYHA class
advanced HF and those with RCTs with >25 outcomes or quality of life; in NYHA class III or IV, Evidence to support
less symptomatic disease. HF patients and left ventricular CRT improved functional outcomes and reduced expansion of indications for CRT
ejection fraction (LVEF) .40; both all-cause mortality and HF hospitalizations to less symptomatic patients with
compared CRT with inactive HF who have LVEF <.35 and QRS
pacing, right ventricular duration >120 ms and are in sinus

R. Whittemore et al. / Heart & Lung 43 (2014) 453e461

pacing alone, left ventricular rhythm but results for NYHA class I
pacing alone, implantable symptoms and asymptomatic left
cardioverter-defibrillator alone, ventricular dysfunction is inconclusive
or usual care; reported
all-cause mortality, heart failure
rehospitalization, change
in LVEF, or functional outcomes.

Qualitative synthesis To conduct a systematic Method not reported. Participants conceptualized HF as a debilitating Standards for assessing Patients need to make sense of the
Yu et al,9 2008 review of qualitative Metasynthesis inferred. and distressing condition that had many quality (Popay, Rogers, experience, an essential step to
studies of how older people N ¼ 14 detrimental physical and psychosocial sequelae. and Williams,10 1998) initiate successful adjustment.
live with chronic HF. Studies were included Most viewed effective self-care as the key factor Interventions to address feelings
if they focused on older patients’ that ensured successful management of HF. Yet, of powerlessness and hopelessness
experiences of living with chronic HF. some older patients who were unable to relate to may be important to promote disease
the acute and chronic symptoms of HF described acceptance and adjustment.
having distressing symptoms, compromised Consider gender differences in
physical functioning, feelings of powerlessness the way older patients with HF
and hopelessness, and social and role conceive the illness experience and
dysfunction behaviors. design interventions accordingly.
Males interpreted the disease as a devitalizing
condition whereas female patients focused
more on psychological adjustment.

Mixed studies review To describe the caregivers’ PRISMA and meta-ethnography Caregivers’ activities included: 1) Critical Appraisal Skills HF patients and caregivers should
Buck et al,11 2014 activities contributing to (Noblit and Hare,12 1988) measurement/quantitative skills (weighing Program checklist (CASP)13 be educated and supported by
self-care in patients with HF N ¼ 40 (qualitative the patient or blood pressure monitoring), clinicians and researchers
N ¼ 17, quantitative 2) system navigators (scheduling appointments, as a unit of care.
N ¼ 23) Quantitative or equipment, services), and 3) interpersonal skills
qualitative studies of informal (encouraging/facilitating exercise).
caregivers of adults with HF Few studies (N ¼ 2) linked caregivers’
contribution with patient outcomes.
Caregivers’ experience was described
as burdensome in 7 studies and
positive in 5 studies.
(continued on next page)

457
 458 R. Whittemore et al. / Heart & Lung 43 (2014) 453e461

on the purpose of the review.15 Integrative reviews can focus on

be based on individual characteristics,

methodology (e.g., evaluating the conceptual and operational def-
initions of a concept), theory (e.g., evaluating different behavior

homecare follow-up within 72 h

length of follow-up may need to

change theories), and/or research.70 Due to the methodological
programs have the potential to
improve hospital readmission

programs with inpatient and

development of knowledge synthesis methods, the integrative
Nurse-led transitional care

rates and quality of life in

review method is best suited for synthesizing knowledge on pri-

were most successful. mary research combined with methodological and/or theoretical
Clinical implications

Multidisciplinary

manuscripts. If all of the articles in a review are primary research,

adults with HF.

such as acuity.
then a systematic review, meta-analysis, or mixed studies review
method should be used.
Dose and

Methodological guidelines for the integrative review have been

proposed15; however quality appraisal is challenging, particularly
with the inclusion of theoretical and empirical sources. Evaluating
Smith and Stullenbarger,16

the quality of primary sources needs to be considered and one

possible method would be to use the approach of historical
research, examining authenticity, methodological quality, infor-
Quality appraisal

mational value, and representativeness of primary sources.71

An example of an integrative review in the heart failure litera-
ture is provided in Table 3. The purpose of this study was to
examine the effects of nurse-led transitional care interventions for
1991

adults with heart failure on various outcomes.14 Some

interventions improved hospital readmission rates and patient
quality of life; however, all interventions were multi-faceted and
complex and it was difficult for the authors to ascertain the optimal
Strategies included: Education, counseling,
medication titration, progressive activities,

combination of strategies to improve outcomes. As can be seen, this

and cost savings (n ¼ 3) were reported.
Improvements in hospital readmission
Interventions were multi-faceted and

integrative review could also be classified as a systematic review as

rates (n ¼ 7), quality of life (n ¼ 5),

all manuscripts included in the review were quantitative research

diverse with respect to strategies,

and dietary, physical therapy or

evaluating the efficacy of an intervention. As greater attention is

dose, and length of follow-up.

occupational therapy consult.

paid to the developing methodology of different types of reviews,

the distinguishing features of each method and the use of
terminology should become more consistent.
Main findings

Scoping review

Scoping reviews have recently been proposed as a knowledge

synthesis method to summarize a range of evidence in order to
understand broadly what is known about a phenomenon.72
Scoping reviews “aim to rapidly identify the key concepts under-
Method/reporting guidelinesNumber

Integrative review (Whittemore and

intervention and a control group of

pinning a research area and the main sources and types of evidence
of publications inclusion Criteria

available, and can be undertaken as stand alone projects in their

own right, especially where an area is complex or has not been
reviewed comprehensively before.”73 Scoping reviews are differ-
entiated from systematic reviews in their broad approach to a topic,
purposive sampling frame, and identification of gaps in the litera-
Transitional care

ture.74 Methodological guidelines have been proposed72 and qual-

Knafl,15 2005)

ity is generally not appraised due to the variety of primary sources

usual care

that can be used in the scoping review. A scoping review about

N ¼ 20

heart failure was unable to be located.

RE-AIM review
the effects of nurse-led

hospital readmissions,

In response to concerns regarding the need to assess the public

To critically examine

adults with HF on

cost-effectiveness
quality of life and

health impact of interventions and translation of research to

interventions for
transitional care

practice, Glasgow and colleagues75e77 have undertaken extensive

work to establish guidelines that promote and evaluate the reach,
Study aim

efficacy or effectiveness, adoption, implementation, and mainte-

nance (RE-AIM Framework) of interventions. This framework has
recently been extended to knowledge synthesis methods78e80
through the use of a 23-item data extraction tool.81 This tool can
Stamp et al,14 2014
Integrative review
Table 3 (continued )

be used to code and synthesize data from individual studies across

the five RE-AIM dimensions.81 The result of this type of review
Author, year

provides evidence not only on the efficacy of an intervention, but

Method

also the generalizability of the intervention. An RE-AIM review

about heart failure was unable to be located; however the method
has been applied to the evaluation of physical activity and health
 R. Whittemore et al. / Heart & Lung 43 (2014) 453e461 459

literacy interventions.78,80 These reviews report not only on the peer reviewers, and reporting guideline developers. In addition, the
efficacy of the interventions but also the processes that could network also organizes training courses, workshops, and meetings
influence implementation. (http://www.equator-network.org/home/).92

Umbrella review
Discussion
Driven by the rapid increase in the number of systematic
reviews and need for timely translation of evidence to practice and As evidence continues to burgeon and accumulate, knowledge
policy, the need for and utility of umbrella reviews has recently synthesis has become an invaluable component of research, prac-
become apparent. Umbrella reviews are also known as meta- tice, and policy. Similar to the research process that scientists use
reviews and overviews of reviews, and are reviews of existing when conducting and reporting results from an individual study, it
systemic reviews. Umbrella reviews focus on breadth with the aim is necessary to apply the same scientific integrity and principles
of summarizing available evidence. They can be used to assess when conducting a knowledge synthesis: the trustworthiness, ac-
similarities and differences in published reviews to summarize curacy, and validity of conclusions rely on this. The importance of
what is known about a topic69,82,83 and typically involve a number knowledge synthesis and standards is reflected in the numerous
of the different types of syntheses. Guidelines for conducting um- methods and guidelines for knowledge synthesis and reporting
brella reviews and reporting the results are available from JBI69 and that have been created.
Cochrane (referred to as overviews of reviews).84 Within these Guidelines for methods and reporting continue to develop to
types of reviews, researchers often use the assessment of multiple keep pace with increasingly sophisticated and diverse primary
systematic reviews (AMSTAR), a validated 11-item measurement study designs. While guidelines for systematic reviews and meta-
tool, to assess the methodological quality of the systematic analyses are well-established, more recent developments have
reviews85; however, there are no clearly established guidelines. An occurred for qualitative, mixed methods and scoping reviews.
umbrella review about heart failure was unable to be located; There are also numerous methods for qualitative synthesis and the
however the method has been applied to summarize the results similarities and differences of these approaches have not been well
from 22 studies on a diverse array of factors related to overweight specified. Mixed methods have recently been established and there
and obesity.86 is less consensus on standards for conducting and reporting these
types of reviews.
Guidelines for reporting of individual studies In this regard, this paper has identified several areas for future
methodological development of knowledge synthesis techniques.
Accurate reporting of individual studies is essential to improve While researchers should first consider the underlying assump-
the quality of any knowledge synthesis method. A lack of stan- tions of each type of review, opportunities exist to address meth-
dardization on reporting of data in individual studies can make odological approaches and reporting guidelines, as well as quality
quality appraisal difficult when conducting a review and has the appraisal for qualitative synthesis, mixed studies review, integra-
potential to contribute to missing data. Thus, guidelines for tive review, and scoping reviews. As several of these methods are
reporting of individual studies continue to be developed and new (e.g., mixed studies review, scoping review, umbrella reviews),
revised. Many journals require the adherence to specific reporting the advanced development of each type of review will ultimately
guidelines in order for a research manuscript to be considered for improve the quality, scope, and applicability of the method to
publication. synthesis future health care research.
A guideline for reporting results of a randomized control trial Of note, many of these standards and guidelines are primarily
(RCT) was one of the first to be developed with the Consolidated focused on internal validity and only recently has there been dis-
Standards of Reporting Trials (CONSORT). The CONSORT guidelines cussion of external validity in regards to knowledge synthesis with
specify the minimum criteria for reporting randomized trials in a the development of the RE-AIM guidelines for systematic
25-item checklist, which includes recommendations to diagram reviews.81 Given the research to practice gap,93 explicit attention is
participant flow through the trial.87 The CONSORT standards have needed to assess and synthesize the external validity of
gained extensive support by editorial groups and are required by interventions. Without such work, the ability for researchers, cli-
most journals for publication of a research report on an RCT. nicians, and policymakers to transfer and generalize results from
The improvement in standardization of reporting trails that intervention studies is diminished.
resulted from the use of the CONSORT guidelines has resulted in Ultimately, the validity and quality of knowledge synthesis
additional initiatives to improve the reporting of specific types of studies relies on the quality of the individual studies included: good
trials. For example, there are guidelines for the reporting of cluster reporting of individual studies and knowledge synthesis methods
randomized trials and eHealth interventions.88,89 Guidelines have are inextricably linked. There has been an implosion of guidelines
also been developed for different types of research. For example, for reporting the results of individual studies (e.g., CONSORT
guidelines for reporting of observational studies (STROBE),90 Statement, RE-AIM). While these reporting guidelines are often
reporting for non-randomized behavioral interventions onerous and present challenges to authors and editors managing
(TREND),91 and reporting qualitative research have been developed journal page constraints, they have increased the standardization of
(COREQ).65 reporting study results and often help to ensure that crucial
Since the development of the CONSORT guidelines in 1993, over information is available for knowledge synthesis reviews, as well as
200 different reporting guidelines according to the type of critiquing and interpreting the primary study results.
study/research design have been proposed. As a result of this bur- To increase the transparency and quality of different types of
geoning information that is critical to scientists worldwide, the reviews, as well as avoid the duplication of effort, an international
Catalogue of Reporting Guidelines for Health Research e EQUATOR initiative has been recently undertaken to promote the registration
Network e was established. This repository was specifically of systematic reviews. The International Prospective Register of
established to improve the use of guidelines in the reporting of Systematic Reviews was developed in February 2011, and as of
health research. The network provides easily accessible resources February 2014 a total of 1704 systematic review protocols were
related to health research reporting for authors, journal editors, registered.94 This organization uses the terminology of ‘systematic
460 R. Whittemore et al. / Heart & Lung 43 (2014) 453e461

review’ to mean any type of review, with the exclusion of inte- the metaquality study cross sectional study. BMJ. 2005;330(7499):
1053e1055.
grative and scoping reviews.
21. Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic
Discussion is also occurring in funding agencies about the need reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):
for some type of knowledge synthesis review to be completed e1000097.
before any new research is conducted. For example, some research 22. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the
quality of reports of meta-analyses of randomised controlled trials: the
funding agencies in the United Kingdom have made this a require- QUOROM statement. Lancet. 1999;354(9193):1896e1900.
ment for funding.3 Thus, knowledge synthesis reports will not only 23. Glass GV. Primary, secondary, and meta-analysis of research. Educ Res.
be important to inform practice and policy, but also new research. 1976;5(10):3e8.
24. The Cochrane Collaboration. Module 9: Assessing the Quality of Studies.
Collecting data from relevant studies Web site. http://www.cochrane-net.org/
openlearning/html/mod7-6.htm; Updated 2012. Accessed 03.03.14.
Conclusion 25. Egger M, Smith GD. Meta-analysis. Potentials and promise. BMJ.
1997;315(7119):1371e1374.
Advancements in methodology of knowledge synthesis and 26. Chalmers TC, Smith Jr H, Blackburn B, et al. A method for assessing the quality
of a randomized control trial. Control Clin Trials. 1981;2(1):31e49.
reporting guidelines have the potential to improve science and the 27. Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports of ran-
translation of evidence into practice and policy. This manuscript domized clinical trials: is blinding necessary? Control Clin Trials. 1996;
has provided an overview of these methods. Knowledge synthesis 17(1):1e12.
28. Kjaergard LL, Villumsen J, Gluud C. Correction: reported methodologic quality
methods and guidelines are evolving and becoming more specific and discrepancies between large and small randomized trials in meta-
both in content and design; however, their usefulness for scientists analyses (annals of internal medicine (2001) 135 (982-989)). Ann Intern
performing, assessing, and interpreting results from a knowledge Med. 2008;149(3):219.
29. Sterne JAC, Egger M, Smith GD. Systematic reviews in health care: investi-
synthesis cannot be understated. Rigorous knowledge synthesis gating and dealing with publication and other biases in meta-analysis. BMJ.
methods enhance the quality, scope, and applicability of results; 2001;323(7304):101e105.
thus improving clinical care and advancing health research and 30. Sterne JAC, Egger M. Funnel plots for detecting bias in meta-analysis:
guidelines on choice of axis. J Clin Epidemiol. 2001;54(10):1046e1055.
policy.
31. Thornton A, Lee P. Publication bias in meta-analysis: Its causes and conse-
quences. J Clin Epidemiol. 2000;53(2):207e216.
32. Sutton AJ, Higgins JPT. Recent developments in meta-analysis. Stat Med.
References 2008;27(5):625e650.
33. Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in
1. Kastner M, Tricco AC, Soobiah C, et al. What is the most appropriate knowl- meta-analyses. BMJ. 2003;327(7414):557e560.
edge synthesis method to conduct a review? Protocol for a scoping review. 34. Deeks JJ, Altman DG, Bradburn MJ. Statistical methods for examining het-
BMC Med Res Methodol. 2012;12:1e10. erogeneity and combining results from several studies in meta-analysis. In:
2. Gough D, Thomas J, Oliver S. Clarifying differences between review designs Egger M, Davey Smith G, Altman DG, eds. Systematic Reviews in Healthcare:
and methods. Syst Rev. 2012;1(1):1e9. Meta-analysis in Context. 2nd ed. London, England: BMJ Books; 2001:
3. Harden A, Thomas J. Methodological issues in combining diverse study types 285e312.
in systematic reviews. Int J Soc Res Methodol. 2005;8(3):257e271. 35. Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies
4. Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic in epidemiology: a proposal for reporting. J Am Med Assoc. 2000;283(15):
review. J R Soc Med. 2003;96(3):118e121. 2008e2012.
5. Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evi- 36. Candy B, King M, Jones L, Oliver S. Using qualitative synthesis to explore
dence for clinical decisions. Ann Intern Med. 1997;126(5):376e380. heterogeneity of complex interventions. BMC Med Res Methodol. 2011;11.
6. Woltz PC, Chapa DW, Friedmann E, Son H, Akintade B, Thomas SA. Effects of 37. Popay J. Moving Beyond Effectiveness in Evidence Synthesis: Methodological Is-
interventions on depression in heart failure: a systematic review. Heart Lung J sues in the Synthesis of Diverse Sources of Evidence. London, UK: National
Acute Crit Care. 2012;41(5):469e483. Institute for Health and Clinical Excellence; 2006.
7. Effective Public Health Practice Project (EPHPP). Quality Assessment 38. Ring N, Ritchie K, Mandava L, Jepson R. A guide to synthesising qualitative
Tool for Quantitative Studies. http://www.ephpp.ca/tools.html; Updated research for researchers undertaking health technology assessments and
2009. Accessed 20.04.14. systematic reviews. http://www.nhshealthquality.org/nhsqis/8837.html;
8. Al-Majed NS, McAlister FA, Bakal JA, Ezekowitz JA. Meta-analysis: cardiac Updated 2010. Accessed 07.02.14.
resynchronization therapy for patients with less symptomatic heart failure. 39. Jones ML. Application of systematic review methods to qualitative research:
Ann Intern Med. 2011;154(6):401e412. practical issues. J Adv Nurs. 2004;48(3):271e278.
9. Yu DSF, Lee DTF, Kwong ANT, Thompson DR, Woo J. Living with chronic heart 40. Mays N, Pope C. Qualitative research in health care: assessing quality in
failure: a review of qualitative studies of older people. J Adv Nurs. 2008;61(5): qualitative research. BMJ. 2000;320(7226):50e52.
474e483. 41. Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a
10. Popay J, Rogers A, Williams G. Rationale and standards for the systematic critical review. BMC Med Res Methodol. 2009;9(1):1e11.
review of qualitative literature in health services research. Qual Health Res. 42. Centre for Reviews and Dissemination. Systematic Reviews: CRD’s Guidance for
1998;8(3):341e351. Undertaking Reviews in Health Care. York, UK: University of York; 2009.
11. Buck HG, Harkness K, Wion R, et al. Caregivers’ contributions to heart failure 43. Finlayson KW, Dixon A. Qualitative meta-synthesis: a guide for the novice.
self-care: a systematic review. Eur J Cardiovasc Nurs; 2014:1e11. Nurse Res. 2008;15(2):59e71.
12. Noblit GW, Hare RD. Meta-ethnography: Synthesizing Qualitative Studies, vol. 44. Dixon-Woods M, Booth A, Sutton AJ. Synthesizing qualitative research: a re-
11. SAGE; 1988. view of published reports. Qual Res. 2007;7(3):375e421.
13. Critical Appraisal Skills Programme (CASP). Making Sense of Evidence. http:// 45. Campbell R, Pound P, Morgan M, et al. Evaluating meta-ethnography: sys-
www.casp-uk.net/find-appraise-act/appraising-the-evidence; Updated 2013. tematic analysis and synthesis of qualitative research. Health Technol Assess.
Accessed 20.04.14. 2011;15(43):1e164.
14. Stamp KD, Machado MA, Allen NA. Transitional care programs improve out- 46. Walsh D, Downe S. Meta-synthesis method for qualitative research: a liter-
comes for heart failure patients: an integrative review. J Cardiovasc Nurs. ature review. J Adv Nurs. 2005;50(2):204e211.
2014;29(2):140e154. 47. Paterson BL, Canam C. Meta-study of Qualitative Health Research: A Practical
15. Whittemore R, Knafl K. The integrative review: updated methodology. J Adv Guide to Meta-analysis and Meta-synthesis, vol. 3. SAGE; 2001.
Nurs. 2005;52(5):546e553. 48. Sandelowski M, Barroso J. Handbook for Synthesizing Qualitative Research.
16. Smith MC, Stullenbarger E. A prototype for integrative review and meta- Springer Publishing Company; 2006.
analysis of nursing research. J Adv Nurs. 1991;16(11):1272e1283. 49. Finfgeld DL. Metasynthesis: the state of the art e so far. Qual Health Res.
17. Crowther M, Lim W, Crowther MA. Systematic review and meta-analysis 2003;13(7):893e904.
methodology. Blood. 2010;116(17):3140e3146. 50. Kearney MH. Focus on qualitative methods: ready-to-wear: discovering
18. Smith V, Devane D, Begley CM, Clarke M. Methodology in conducting a sys- grounded formal theory. Res Nurs Health. 1998;21(2):179e186.
tematic review of systematic reviews of healthcare interventions. BMC Med 51. McKibbon KA, Wilczynski NL, Haynes RB. Developing optimal search strate-
Res Methodol. 2011;11:1e6. gies for retrieving qualitative studies in PsycINFO. Eval Health Prof.
19. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting 2006;29(4):440e454.
systematic reviews and meta-analyses of studies that evaluate health care 52. Walters LA, Wilczynski NL, Haynes RB. Developing optimal search strategies
interventions: explanation and elaboration. Ann Intern Med. 2009;151(4):W- for retrieving clinically relevant qualitative studies in EMBASE. Qual Health
65eW-94. Res. 2006;16(1):162e168.
20. Moja LP, Telaro E, D’Amico R, Moschetti I, Coe L, Liberati A. Assessment of 53. Wilczynski NL, Marks S, Haynes RB. Search strategies for identifying quali-
methodological quality of primary studies by systematic reviews: results of tative studies in CINAHL. Qual Health Res. 2007;17(5):705e710.
 R. Whittemore et al. / Heart & Lung 43 (2014) 453e461 461

54. Wong SS, Wilczynski NL, Haynes RB. Developing optimal search strategies for 79. Glasgow RE, Bull SS, Gillette C, Klesges LM, Dzewaltowski DA. Behavior
detecting clinically relevant qualitative studies in MEDLINE. Stud Health change intervention research in healthcare settings: a review of recent re-
Technol Inform. 2004;107:311e314. ports with emphasis on external validity. Am J Prev Med. 2002;23(1):62e69.
55. Atkins S, Lewin S, Smith H, Engel M, Fretheim A, Volmink J. Conducting a 80. Blackman KCA, Zoellner J, Berrey LM, et al. Assessing the internal and external
meta-ethnography of qualitative literature: lessons learnt. BMC Med Res validity of mobile health physical activity promotion interventions: a sys-
Methodol. 2008;8. tematic literature review using the RE-AIM framework. J Med Internet Res.
56. Barbour RS. Checklists for improving rigour in qualitative research: a case of 2013;15(10).
the tail wagging the dog? BMJ. 2001;322(7294):1115e1117. 81. Virginia Tech College of Agriculture and Life Science. Resources and tools.
57. Sandelowski M. Rigor or rigor mortis: the problem of rigor in qualitative Coding sheet for publications reporting on RE-AIM elements Web site. http://
research revisited. ANS Adv Nurs Sci. 1993;16(2):1e8. www.re-aim.org/resources_and_tools/measures/literaturecoding.pdf; Upda-
58. Cochrane Qualitative Research Methods Group. Tools to assist qualitative ted 2014. Accessed 03.03.14.
reviewers. http://www.joannabriggs.edu.au/cqrmg/tools_3.html; Updated 82. Thomson D, Russell K, Becker L, Klassen T, Hartling L. The evolution of a new
2014. Accessed 07.02.14. publication type: steps and challenges of producing overviews of reviews. Res
59. Noyes J, Popay J, Pearson A, Hannes K. 20 qualitative research and cochrane re- Synth Methods. 2011;1(3e4):198e211.
views. In: Cochrane Handbook for Systematic Reviews of Interventions; 2008:571. 83. Ioannidis JPA. Integration of evidence from multiple meta-analyses: a primer
60. Whittemore R, Chase SK, Mandle CL. Validity in qualitative research. Qual on umbrella reviews, treatment networks and multiple treatments meta-
Health Res. 2001;11(4):522e537. analyses. CMAJ. 2009;181(8):488e493.
61. Lincoln YS, Guba EG. Naturalistic Inquiry, vol. 75. SAGE Publications, Incor- 84. Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of In-
porated; 1985. terventions Version 5.1.0, http://www.cochrane-handbook.org; Updated 2011.
62. Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in Accessed 05.03.14.
reporting the synthesis of qualitative research: ENTREQ. BMC Med Res 85. Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a mea-
Methodol. 2012;12. surement tool to assess the methodological quality of systematic reviews.
63. Hannes K, Lockwood C, Pearson A. A comparative analysis of three online BMC Med Res Methodol. 2007;7.
appraisal instruments’ ability to assess validity in qualitative research. Qual 86. Monasta L, Batty GD, Cattaneo A, et al. Early-life determinants of overweight
Health Res. 2010;20(12):1736e1743. and obesity: a review of systematic reviews. Obes Rev. 2010;11(10):695e708.
64. Higgins JP, Green S. Cochrane Handbook for Systematic Reviews of Interventions: 87. Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guide-
Cochrane Book Series; 2008:1e649. lines for reporting parallel group randomised trials. BMJ. 2010;340(7748):
65. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative 698e702.
research (COREQ): a 32-item checklist for interviews and focus groups. Int J 88. Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and
Qual Health Care. 2007;19(6):349e357. standardizing evaluation reports of web-based and mobile health in-
66. Joanna Briggs Institute. SUMARI: The Joanna Briggs Institute System for the terventions. J Med Internet Res. 2011;13(4).
Unified Management, Assessment and Review of Information. http://www 89. Campbell MK, Elbourne DR, Altman DG. CONSORT statement: extension to
.joannabriggs.edu.au/services/sumari.php; Updated 2014. Accessed 06.02.14. cluster randomised trials. BMJ. 2004;328(7441):702e708.
67. Pluye P, Gagnon M, Griffiths F, Johnson-Lafleur J. A scoring system for 90. Von Elm E, Egger M, Altman DG, Pocock SJ, Gøtzsche PC, Vandenbroucke JP.
appraising mixed methods research, and concomitantly appraising qualita- Strengthening the reporting of observational studies in epidemiology
tive, quantitative and mixed methods primary studies in mixed studies (STROBE) statement: guidelines for reporting observational studies. BMJ.
reviews. Int J Nurs Stud. 2009;46(4):529e546. 2007;335(7624):806e808.
68. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review e a new method 91. Des Jarlais DC, Lyles C, Crepaz N. Improving the reporting quality of non-
of systematic review designed for complex policy interventions. J Health Serv randomized evaluations of behavioral and public health interventions: the
Res Policy. 2005;10(suppl 1):21e34. TREND statement. Am J Public Health. 2004;94(3):361e366.
69. The Joanna Briggs Institute. Joanna Briggs Institute Reviewers’ Manual: 2014 92. Enhancing the Quality and Transparency of Health Research
Edition/Supplement. Australia: The Joanna Briggs Institute; 2014. (EQUATER network). Resource centre. http://www.equator-network.org/
70. Whittemore R. Combining evidence in nursing research: methods and im- resource-centre/; Updated 2012. Accessed 27.03.13.
plications. Nurs Res. 2005;54(1):56e62. 93. McGinnis JM, Foege W. Guide to community preventive services: harnessing
71. Kirkevold M. Integrative nursing research e an important strategy to further the science. Am J Prev Med. 2000;18(1 suppl 1):1e2.
the development of nursing science and nursing practice. J Adv Nurs. 94. University of York Centre for Reviews and Dissemination. International pro-
1997;25(5):977e984. spective register of systematic reviews. Welcome to PROSPERO: International
72. Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the method- Prospective Register of Systematic Reviews Web site. http://www.crd.york.ac.
ology. Implement Sci. 2010;5(1). uk/PROSPERO/; Updated 2013. Accessed 03.03.14.
73. Mays N, Roberts E, Popay J. Synthesising research evidence. In: Fulop N, 95. Centre for Reviews and Dissemination. Chapter 6: Incorporating qualitative
Allen P, Clarke A, Black N, eds. Studying the Organisation and Delivery of Health evidence in or alongside effectiveness reviews. In: Systematic Reviews: CRD’s
Services: Research Methods. London: Routledge; 2001:188e220. Guidance for Undertaking Reviews in Health Care. York, UK: University of York;
74. Arksey H, O’Malley L. Scoping studies: towards a methodological framework. 2009:219e237.
Int J Soc Res Methodol. 2005;8(1):19e32. 96. Heyvaert M, Maes B, Onghena P. Mixed methods research synthesis: defini-
75. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health tion, framework, and potential. Qual Quant. 2013;47(2):659e676.
promotion interventions: the RE-AIM framework. Am J Public Health. 97. Sandelowski M, Voils CI, Barroso J. Defining and designing mixed research
1999;89(9):1322e1327. synthesis studies. Res Sch. 2006;13(1):29.
76. Glasgow RE, McKay HG, Piette JD, Reynolds KD. The RE-AIM framework for 98. Armstrong R, Hall BJ, Doyle J, Waters E. ‘Scoping the scope’ of a cochrane
evaluating interventions: what can it tell us about approaches to chronic review. J Public Health (Oxf). 2011;33(1):147e150.
illness management? Patient Educ Couns. 2001;44(2):119e127. 99. HLWIKI International. Scoping reviews. http://hlwiki.slais.ubc.ca/index.php/
77. Glasgow RE, Lichtenstein E, Marcus AC. Why don’t we see more translation of Scoping_reviews; Updated 2014. Accessed 28.02.14.
health promotion research to practice? Rethinking the efficacy-to- 100. Virginia Tech College of Agriculture and Life Science. Reach Effectiveness
effectiveness transition. Am J Public Health. 2003;93(8):1261e1267. Adoption Implementation Maintenance (RE-AIM). http://www.re-aim.org/
78. Allen K, Zoellner J, Motley M, Estabrooks PA. Understanding the internal and index.html; Updated 2014. Accessed 20.02.14.
external validity of health literacy interventions: a systematic literature re- 101. Virginia Tech College of Agriculture and Life Science. Measures and checklists.
view using the RE-AIM framework. J Health Commun. 2011;16(suppl 3): Resources and tools Web site. http://www.re-aim.org/resources_and_tools/
55e72. measures/index.html; Updated 2014. Accessed 21.04.14.