CASP Checklist:

For systematic reviews with meta-analysis of randomised controlled

trials (RCTs)

Reviewer Name: ó

Paper Title:

Author: É ú

Web Link:

Appraisal Date:

During critical appraisal, never make assumptions about what the researchers have done. If it is not
possible to tell, use the “Can’t tell” response box. If you can’t tell, at best it means the researchers
have not been explicit or transparent, but at worst it could mean the researchers have not
undertaken a particular task or process. Once you’ve finished the critical appraisal, if there are a large
number of “Can’t tell” responses, consider whether the findings of the randomised controlled trial
are trustworthy and interpret the results with caution.
Section A: Is the basic study design valid for a systematic review?

I
1. Did the systematic review address a clearly ☐Yes ☐ No ☐ Can’t Tell
formulated research question?

CONSIDER:
For a systematic review of RCTs, a research question can be ‘formulated’ in terms of the:
● Population
● Intervention
● Comparator
● Outcome/s
● Time, e.g., study timeframe, or follow-up intervals
2. Did the researchers search for appropriate ☐Yes ☐ No ☐ Can’t Tell
study design(s) to answer the research
question?

CONSIDER:
If the research question is concerned with the efficacy of an intervention, the RCT is the appropriate
study design for a systematic review. The most common type of RCT is the parallel RCT in which
individuals are randomised to study groups; other methods of randomisation, however, could be
relevant depending on the research question:
• Crossover RCTs are designed to investigate interventions that have short-term effects in people
with long-term conditions; two or more interventions are given to all participants, but the order in
which each participant receives the interventions is randomised, and participants act as their own
controls. This study design should not be used to assess interventions with long-term effects.
• Cluster RCTs randomise groups, such as families, clinics, schools, or communities, and are usually
used to investigate interventions designed to be administered at a “cluster” level, such as those
relating to service provision/delivery or policy.
Notes to support interpretation of Section A, Questions 1 and 2:
If you answered “No” to both these questions:
● It is likely that the researchers did not clearly formulate the fundamental aspects of the research
question, and the most appropriate way of answering it. If this is the case, it is likely other
problems will arise during the conduct of the systematic review
● Consider whether it would be useful to continue with the critical appraisal process

Section B: Is the systematic review methodologically sound?

3. Were all the important, relevant primary ☐Yes ☐ No ☐ Can’t Tell

research studies likely to have been
included in the systematic review?
a) Searching for primary research studies

CONSIDER:
● Was the search strategy comprehensive and clearly reported?
● Did the search include 1 or more of the major bibliographic databases, e.g., MEDLINE/PubMed,
Embase?
● Were relevant subject-specific bibliographic databases searched?
● Did the search include non-English language studies?
● Did the search include hand-searching of reference lists from primary research studies included in
the systematic review?
● Did the search include unpublished studies? For instance, did the search include registers of
ongoing trials or preprint repositories (e.g., arXiv) to find unpublished studies?
● Did the researchers consult experts in the field about potential primary research studies or
ongoing trials that could be included?
b) Screening primary research studies from the ☐Yes X ☐ No ☐ Can’t Tell
search

CONSIDER:
● Did the researchers define appropriate screening criteria?
● Did the researchers design and implement a robust process to screen the primary research
studies? For instance, two researchers working independently to screen the titles and abstracts of
the primary research studies, with a third researcher to settle any disagreements.

c) Selecting primary research studies to X

☐Yes ☐ No ☐ Can’t Tell
include in the systematic review

CONSIDER:
● Did the researchers define appropriate eligibility (inclusion/exclusion) criteria?
● Did the researchers design and implement a robust process to apply the eligibility criteria to the
primary research studies? For instance, two researchers working independently to select primary
research studies based on the full papers, with a third researcher to settle disagreements.
● Was the level of agreement between the researchers responsible for selecting the primary research
studies for inclusion in the systematic review assessed?

d) Summarising the search and its outputs X

☐Yes ☐ No ☐ Can’t Tell

CONSIDER:
Did the researchers present a PRISMA-type flowchart, including the numbers of primary research
studies that were:
● Duplicates?
● Screened out?
● Excluded, with the reasons for exclusion?
● Included in the systematic review?
● Included in the meta-analysis (data may not have been complete in some of the primary research
studies)?
4. Did the researchers assess the validity or X
☐Yes ☐ No ☐ Can’t Tell
methodological rigour of the primary
research studies included in the systematic
review?

CONSIDER:
Lack of methodological rigour in the individual primary research studies can affect the validity and
interpretation of the findings of the systematic review with meta-analysis.
• Did the researchers use a validated tool to assess the methodological rigour of the primary research
studies included in the systematic review?
• Was the tool appropriate to assess the type(s) of study design(s) included in the systematic review?
For example, the Cochrane Risk of Bias tool specifically for RCTs or the McMaster EPHPP tool for any
quantitative study design, including RCTs.
• Did the researchers present the findings from their quality assessment, and interpret them
accurately?
5. Did the researchers extract, and present X ☐ No ☐ Can’t Tell
☐Yes
information from the individual primary
research studies appropriately and
transparently?

(a) Extraction of data

CONSIDER:
● Did the researchers design and implement a robust process for the extraction of data from the
individual primary research studies?
● Did the researchers use a standardised form or software programme to record the data to ensure
completeness and accuracy of recording?
(b) Presentation of data X
☐Yes ☐ No ☐ Can’t Tell

CONSIDER:
• Did the researchers present the key characteristics of the individual primary research studies, e.g., in
a table? For instance, the number of participants, the profile of participants (age, sex), the
intervention, the comparator, the outcome/s evaluated, and the study timeframe.
• Did the researchers present the results of the individual primary research studies in a Forest plot or
combination of table and Forest plot? For instance, the effect size/s, the confidence intervals, and
the P value/s. NB: The Forest plot should also show the overall result from the systematic review.

Notes to support the interpretation of Section B, Questions 3-5:

If you answered “No” to these questions, it is likely that there is a lack of methodological rigour in the
conduct of the systematic review, which means it is best to interpret the results with caution, and to
assess how those aspects of poor methodology will have an impact on the results of the systematic
review.
● For Question 3, a “No” response indicates that this systematic review may have missed primary
research studies that could have contributed to answering the research question; in a systematic
review with meta-analysis, the results of any missing primary research studies could have altered
the effect estimate for the systematic review.
● For Question 4, a “No” response indicates that the researchers did not identify any systematic bias
or confounding factors in the primary research studies that could have affected the results of the
systematic review; in the absence of this information, it is not possible for you to assess in what
ways the results of the systematic review could have been affected, and it is best to be cautious
when interpreting the results.
● For Question 5, a “No” response indicates that the researchers did not organise the data from the
primary research studies in a coherent way such that it could analysed appropriately, and thereby
reliable conclusions drawn from it.
If you answered “No” to all three questions in Section B, consider whether it would be useful to
continue with the critical appraisal process.

Section C: Are the results of the systematic review trustworthy?

6. Did the researchers analyse the pooled X

☐Yes ☐ No ☐ Can’t Tell
results of the individual primary research
studies appropriately?

CONSIDER:
• Did the researchers undertake a power calculation during the design and planning of the
systematic review, and did the number of participants whose outcomes were entered into the
analysis meet the power calculation, i.e., was the systematic review sufficiently powered to detect
any effect on the outcomes of interest?
• Did the researchers assess the level of statistical heterogeneity (variability) among the primary
research studies? For example, using the I2 statistic.
• Did the researchers use an appropriate model of meta-analysis for the level of heterogeneity
among the primary research studies (a random-effects model if there was heterogeneity or a
fixed-effects model if the primary research studies were all investigating the same underlying
effect)?
• Did the researchers provide confidence intervals for the effect estimates in the systematic review?
• Did the researchers provide p values for the effect estimates in the systematic review?
• Did the researchers assess the potential for publication bias in the systematic review (e.g., using a
funnel plot)?
7. Did the researchers report any limitations of ☐YesX ☐ No ☐ Can’t Tell
the systematic review and, if so, do the
limitations discussed cover all the issues you
have identified during critical appraisal?

CONSIDER:
• Was the systematic review sufficiently powered to detect an effect on the outcomes of interest?
• Did the researchers consider whether important relevant primary research studies could have been
missed?
• Based on the quality assessment, did the researchers identify methodological issues or potential
sources of bias and/or confounding in the primary research studies, and discuss the implications
for the results of the systematic review?
• Did the researchers identify reasons for any potential heterogeneity across the primary research
studies and discuss the implications for the results of the systematic review?
• Did the researchers reflect on the precision of the results of the systematic review, i.e., the range
of the confidence intervals (a smaller range, the narrower the confidence intervals, means the
result is more precise, and closer to the true effect)?
• If relevant, did the researchers note whether the confidence-interval range included the “line of no
effect” (0 for a difference, 1 for a ratio, where the null hypothesis holds true), or whether the lower
limit of the confidence-interval range was close to the “line of no effect” and discuss the
implications for the results of the systematic review?
• If the results were statistically significant (i.e., they were less likely to be due to chance), did the
researchers discuss whether the results would be important or meaningful for both the responsible
professionals and for the individuals and/or populations receiving the intervention?
• Did the researchers discuss the implications of any publication bias on the results of the systematic
review?

Notes to support interpretation of Section C, Questions 6 and 7: If you answered “No” to these
questions, it is likely that the researchers did not analyse the information from the primary research
studies appropriately, nor did they discuss the limitations of the systematic review as fully as possible;
as such, it is not possible for you to assess the trustworthiness (validity and credibility) of the results of
the systematic review.

Section D: Are the results of the systematic review relevant locally?

8. Can the results of the systematic review be X

☐Yes ☐ No ☐ Can’t Tell
applied to your local population/in your
local setting or context?

CONSIDER:
• Are you clear about what the results of the systematic review show?
• Are the participants from the primary research studies in the systematic review similar to or
different from your local population?
• Are the local settings or contexts from the primary research studies in the systematic review similar
to or different from your local setting or context?
• Are there any outcomes the researchers could have studied that would have been useful to you in
deciding whether to act upon the results of the systematic review?

Notes to support interpretation of Section D, Question 8:

● If you answered “No” to this question, it is not necessary to answer Questions 9 and 10 because,
irrespective of a systematic review’s methodological rigour, the results are not applicable to the
individuals or populations for whom you are responsible.
● If you answered “Yes” to Question 8, answer Questions 9 and 10

Section E: Will the implementation of the results represent greater value for your service users or
population?
9. If the results of the systematic review can be X
☐Yes ☐ No ☐ Can’t Tell
applied to your local population/in your
setting, would the benefits of acting upon the
results outweigh any potential disadvantages,
harms and/or additional demand for
resources associated with implementation?

CONSIDER:
● Did the researchers identify any potential disadvantages or harms associated with the
intervention?
● Did the researchers assess any disadvantages or harms against the benefits of the intervention,
and discuss the balance between the two?
● Did the researchers report any information on the potential demand for resources (e.g., cost,
workforce, time, skills/skill mix, IT) that might be associated with acting upon the results of the
systematic review?
If the researchers did not address the balance of benefits to potential disadvantages, harms, and/or
demand for resources, what do you think – see Question 10?
10. If actioned, would the findings from the ☐Yes
X ☐ No ☐ Can’t Tell
systematic review represent greater or
additional value for the individuals or
populations for whom you are responsible?

CONSIDER:
Value equals the Outcome/s (Benefit minus Harm) divided by the Resources required for
implementation.
● What resources would be needed to implement the findings of the systematic review? Take
account of various types of resource, not only expenditure, but also time, skills mix, skills
development or training needs, IT requirements, and other material resources.
● Are you able to disinvest resources elsewhere to be able to re-invest in the implementation of the
findings from systematic review?

Notes to support interpretation of Section E, Questions 9 and 10:

● If you answered “No” to these questions, it is likely that the findings of the systematic review will
not confer greater or additional benefit or value on the individuals and/or populations for whom
you are responsible, despite the systematic review’s applicability to your local setting.
● If you answered “Yes” to one or both questions, it is likely that the findings of the systematic
review will confer greater or additional benefit or value on the individuals and/or populations for
whom you are responsible, and you need to discuss with colleagues whether it would be
appropriate to implement the findings in your local setting.

What is your conclusion about the systematic ☐Yes X XCan’t Tell

☐ No ☐
review – can it be used to support
apresento eficácio mas precisa de
decision-making?
mais estudos a
congo prozo
.
 CONSIDER:
● Would you use it to change your practice or to recommend changes to care/interventions used in
your organisation?
● Could you judiciously implement this intervention without delay?

CASP General SR Checklist: Collation of critical appraisal responses

Yes Checklist question Can’ No
t tell
A. Is the basic study design valid for a systematic review?

1. Did the systematic review address a clearly formulated research

X question?

2. Did the researchers search for appropriate study designs to answer the
X research question?
B. Is the systematic review methodologically sound?

3. Were all the important, relevant primary research studies likely to have
been included in the systematic review?
X
4. Did the researchers assess the validity or methodological rigour of the

É primary research studies included in the systematic review?

5. Did the researchers extract, and present information on the individual
primary research studies appropriately and transparently?
C. Are the results of the systematic review trustworthy?

6. Did the researchers analyse the results of the individual primary research
X studies appropriately?
7. Did the researchers report any limitations of the systematic review and,
X if so, do the limitations discussed cover all the issues in your critical
appraisal?
D. Are the results of the systematic review relevant locally? If so how,
and to what extent?
8. Can the results of the systematic review be applied to your local
X population/in your local setting or context?
E. Will the implementation of the results represent greater value for your service
users or population?
9. If the results of the systematic review can be applied to your local
X population/in your setting or context, would the benefits of acting upon
the results outweigh any potential disadvantages, harms and/or
demand for additional resources associated with implementation?
10.If actioned, would the findings from the systematic review represent
X greater or additional value for the individuals or populations for whom
you are responsible?

APPRAISAL SUMMARY: List key points from your critical appraisal that need to be considered
when assessing the validity of the results and their usefulness in decision-making.
Positive Negative
Positive Negative