Retrospective/Prospective

1. Is the study design prospective, retrospective, or mixed? [Abstractor: Prospective design

requires that the outcome has not occurred at the time the study is initiated and information is
collected over time to assess relationships with the outcome (and includes nested case-control
studies). Mixed design includes case-control or cohort studies in which one group is studied
prospectively and the other retrospectively. A retrospective design analyzes data from past records.
The question is not applicable to cross-sectional studies.]
Prospective .......................................................... Explanation for rating:

Mixed ..................................................................

Retrospective .......................................................

Cannot determine/not applicable .........................

Inclusion/Exclusion Criteria
2. Are critical inclusion/exclusion criteria clearly stated (does not require the reader to infer)?
[Principal Investigator (PI): Provide direction to abstractors by listing individual criteria of a
priori significance and minimal requirements for criteria to be considered “clearly stated.” Include
this question to identify specific inclusion/exclusion criteria that should be consistently recorded
across studies] [Abstractor: Use “Partially” if only some criteria are stated or if some criteria are
not clearly stated (corresponding to directions provided by the PI). Note that studies may describe
inclusion criteria alone (i.e., include x), exclusion criteria (i.e., do not include x), or a combination
of inclusion and exclusion criteria.]
PI:

Yes ....................................................................... Explanation for rating:

Partially: some, but not all, criteria stated or

some criteria not clearly stated ............................

No ........................................................................
3. Are the inclusion/exclusion criteria measured using valid and reliable measures? [PI:
Separately specify each criterion that abstractors should consider based on its relevance to study
bias. It is unlikely that all criteria will need to be evaluated in relation to this question. Provide
direction to abstractors on valid and reliable measurement of each criterion that is to be
considered. For example, prior exposure or disease status is a frequent inclusion/exclusion
criterion, particularly in inception cohorts. Subjective measures based on self-report tend to have
lower reliability and validity than objective measures such as clinical reports and lab findings.
Replicate question to evaluate each individual inclusion/exclusion criterion.]
PI:

Yes ....................................................................... Explanation for rating:

No.........................................................................

Cannot determine; measurement approach not

reported ................................................................
4. Did the study apply inclusion/exclusion criteria uniformly to all comparison groups/arms of the
study? [PI: Drop question if not relevant to entire body of evidence (e.g., all case-series, single-
arm studies).]
PI:

Yes ....................................................................... Explanation for rating:

Partially: some, but not all criteria, applied to

all arms or not clearly stated if some criteria
are applied to all arms ..........................................

No .......................................................................

Cannot determine: article does not specify ..........

Not applicable: study has only one arm and so

does not include comparison groups ...................
5. Was the strategy for recruiting participants into the study the same across study groups/arms
of the study? [PIs: This question is likely to be more relevant for prospective or mixed designs
than retrospective designs. Drop question if not relevant to entire body of evidence (e.g., all studies
generally have only one arm).]
PI:

Yes...................................................................... Explanation for rating:

No .......................................................................

Cannot determine ...............................................

Not applicable: one study group/arm .................

6. Was the sample size sufficiently large to detect a clinically significant difference of 5% or more
between groups in at least one primary outcome measure? [PI: Specify a different percent, if
clinically relevant for each outcome of interest. Question relates to precision; reviewers whose
evaluation of quality is limited to considerations of systematic error or risk of bias (not random
error/precision) need not include this question. Reviewers who include both precision and
systematic error in their evaluation of quality but rely on meta-analysis for pooled estimates need
not include this question. PIs who choose to include considerations of precision in their assessment
may include the question, but should be aware of the need for collaboration between clinical and
statistical expertise in determining the threshold for a clinically adequate sample size.]

PI:

Yes ..................................................................... Explanation for rating:

No ......................................................................

Interventions/Exposure
Clear Specification
7. What is the level of detail in describing the intervention or exposure? [PI: Specify which details
need to be stated (e.g., intensity, duration, frequency, route, setting, and timing of
intervention/exposure). For case-control studies, consider whether the condition, timing, frequency,
and setting of symptoms are provided in the case definition. PI needs to establish criteria for high,
medium, or low response.]

PI:

High: very clear, all PI-required details provided

Explanation for rating:

Medium: somewhat clear, majority of PI-

required details provided .....................................

Low: unclear, many PI-required details

missing .................................................................

Outcomes
Clear Specifications
8. Are the important outcomes prespecified by the researchers? Do not consider harms in
answering this question unless they should have been pre-specified. [PI: This question can be
asked for all outcomes together or replicated for each event. Each adverse event of interest should
be specified for abstractors. Relevant source information includes all study data, including what
 may have been established in relation to an initial randomized controlled trial. Drop question if not
relevant (e.g., primary outcome for case-control studies).]

PI:

Yes ....................................................................... Explanation for rating:

Partially ................................................................

No ........................................................................

Not applicable ......................................................

Creation of Treatment Groups

Allocation
9. Is the selection of the comparison group appropriate, after taking into account feasibility and
ethical considerations. [PI: Provide instruction to the abstractor based on the type of study.
Interventions with community components are likely to have contamination if all groups are drawn
from the same community. Interventions without community components should select groups from
the same source (e.g., community or hospital) to reduce baseline differences across groups. For
case-control studies, controls should represent the population from which cases arose; that is,
controls should have met the case definition if they had the outcome.]

PI:

Yes ....................................................................... Explanation for rating:

No ........................................................................

Cannot determine or no description of the

derivation of the comparison group .....................

Not applicable: study does not include a

comparison group (case series, one study arm) ...
Any Attempt To Balance
10. Any attempt to balance the allocation between the groups (e.g., through stratification,
matching, propensity scores). [PI: This is most likely to be used in case-control study designs.
Drop if not relevant to the body of evidence.]

PI:

Yes or study accounts for imbalance between

groups through a post hoc approach such as
multivariate analysis ............................................ Explanation for rating:

No or cannot determine........................................

Not applicable: study does not include a

comparison group (case series or one study arm)

Contamination
11. Did researchers isolate the impact from a concurrent intervention or an unintended exposure
that might bias results, e.g., through multivariate analysis, stratification, or subgroup
analysis? [PI: specify interventions or exposures for abstractors.]

PI:

Yes ....................................................................... Explanation for rating:

Partially ................................................................

No or do not know: concurrent intervention or

unintended exposure is not described) .................

Not applicable: no concurrent interventions or

unintended exposures likely.................................
12. Did execution of the study vary from the intervention protocol proposed by the investigators
and therefore compromise the conclusions of the study? [PI: Consider intensity, duration,
frequency, route, setting, and timing of intervention/exposures. Drop if not relevant for body of
literature.]

PI:
 Yes ....................................................................... Explanation for rating:

Partially ...............................................................

No ........................................................................

Cannot determine .................................................

Not applicable: not an intervention study ............

Blinding
Blind Outcomes Assessment
13. Were the outcome assessors blinded to the intervention or exposure status of participants?
[PI: There may be circumstances where clinical evaluators cannot be blinded to exposure status.
Drop if not relevant to the body of literature.]

PI:

Yes ....................................................................... Explanation for rating:

No ........................................................................

Not applicable: assessor cannot be blinded .........

Soundness of Information
Source of Information Re Interventions/Exposure
14. Are interventions/exposures assessed using valid and reliable measures, implemented
consistently across all study participants? [PI: Important measures may be listed separately. PI
may need to establish a threshold for what would constitute acceptable measures based on study
topic. When subjective or objective measures could be collected, subjective measures based on self-
report may be considered as being less reliable and valid than objective measures such as clinical
reports and lab findings. Replicate question when needed.]

PI:

Yes ....................................................................... Explanation for rating:

No ........................................................................

Cannot determine or measurement approach not

reported ................................................................
Source of Information for Outcomes
15. Are outcomes assessed using valid and reliable measures, implemented consistently across all
study participants? [PI: Primary outcomes should be identified for abstractors and if there is
more than one, they may be listed separately. Also, identify any relevant secondary outcomes and
harms. Subjective measures based on self-report tend to have lower reliability and validity than
objective measures such as clinical reports and lab findings. Note for case-control studies: consider
whether the ascertainment of cases was independent of exposure.]

PI:

Yes ....................................................................... Explanation for rating:

No ........................................................................

Cannot determine or measurement approach not

reported ................................................................

Follow-Up
Equality of Length of Follow-Up for Participants
16. Is the length of follow-up the same for all groups? [For case-control studies, are cases and
controls matched on length of followup? Abstractor: When follow-up was the same for all study
participants, the answer is yes. If different lengths of follow-up were adjusted by statistical
techniques, (e.g., survival analysis), the answer is yes. Studies in which differences in follow-up
were ignored should be answered no.]

Yes ....................................................................... Explanation for rating:

No or cannot determine........................................

Not applicable: cross-sectional or only one

group followed over time .....................................
Length of Followup Adequate
17. Is the length of time following the intervention/exposure sufficient to support the evaluation of
primary outcomes and harms? [PI: Primary outcomes (including harms) should be identified for
abstractors. Important measures may be listed separately. Abstractors should be provided with
specific criteria for sufficient length of follow-up based on prior research or theory. Drop if entire
body of evidence is cross-sectional or if minimal length of follow-up period is specified through
inclusion criteria.]

PI:

Yes ....................................................................... Explanation for rating:

Partially: some primary outcomes are followed

for a sufficient length of time ..............................

No ........................................................................

Cannot determine .................................................

Not applicable: cross-sectional ............................

Completeness of Follow-Up
18. Did attrition from any group exceed [x] percent? [PI: Attrition is measured in relation to the
time between baseline (allocation in some instances) and outcome measurement for both
retrospective and prospective studies and could include data loss from crossover. Attrition rates
may vary by outcome and time of measurement. Specify the criterion to meet relevant standards for
the topic. Specify measurement period of interest, if repeated measures. Cochrane standard for
attrition is 20 percent for shorter term (<1 year) and 30 percent for longer term ( ≥ 1 year). Drop if
entire body of evidence is cross-sectional]

PI:

Yes ....................................................................... Explanation for rating:

No ........................................................................

Cannot determine: includes retrospective

designs not stating number eligible at baseline....

Not applicable: cross-sectional ............................

19. Did attrition differ between groups by more than 20 percent? [PI: If appropriate, modify
difference criterion to meet relevant standards for the topic. Attrition rates may vary by outcome
and time of measurement. Drop if entire body of evidence is cross-sectional or case series.]

PI:

Yes ....................................................................... Explanation for rating:

No ........................................................................

Cannot determine: includes retrospective

designs not stating number eligible at baseline....

Not applicable: cross-sectional or only one

group followed—case series, one-arm study .......

Analysis Comparability
Assessment of Baseline Comparability
20. Does the analysis control for baseline differences between groups? [PI: Drop if entire body of
evidence is case series or case control. Define adequate control. List critical baseline differences
that need to be controlled.]

Yes ....................................................................... Explanation for rating:

No ........................................................................

Insufficient reporting to be able to determine ......

Not applicable: only one group, no comparison

group (case series), or case-control study, no
difference in measured baseline characteristics ...
Identification of Prognostic Factors (Effect Modifiers and Confounders)
21. Are confounding and/or effect modifying variables assessed using valid and reliable measures
across all study participants? [PI: Some characteristics may require that sources for establishing
their validity and/or reliability be described or referenced. If so, provide instruction to abstractors.]

PI:

Yes ....................................................................... Explanation for rating:

No ........................................................................

Cannot determine or source for measures not

reported ................................................................

Not applicable: no confounders or effect

modifiers included in the study ............................

Case-Mix Adjustment
22. Were the important confounding and effect modifying variables taken into account in the
design and/or analysis (e.g., through matching, stratification, interaction terms, multivariate
analysis, or other statistical adjustment)? [PI: Provide instruction to abstractors on adequate
adjustment for confounding and testing for effect modification.]

PI:

Yes ....................................................................... Explanation for rating:

Partially: some variables taken into account or

adjustment achieved to some extent ....................

No: not accounted for or not identified ................

Cannot determine .................................................

Analysis Outcome
Intention-to-Treat Analysis
23. In cases of high loss to follow-up (or differential loss to follow-up), is the impact assessed (e.g.,
through sensitivity analysis or other adjustment method)?
Yes ....................................................................... Explanation for rating:

No ........................................................................

Cannot determine .................................................

Not applicable: no loss to follow-up or loss to

follow-up was not considered to be high, cross-
sectional study, or case-control study selected
on outcome...........................................................

Appropriate Analytic Methods

24. Are any important primary outcomes missing from the results? [PI: Identify all primary
outcomes, including timing of measurement, that one would expect to be reported in the study.]

PI:

Yes ...................................................................... Explanation for rating:

No ........................................................................

Cannot determine .................................................

25. Are the statistical methods used to assess the primary benefit outcomes appropriate to the
data? [Abstractor: Question relates to precision and may not be relevant for systematic reviews
that are able to pool data. The statistical techniques used must be appropriate to the data and take
into account issues such as controlling for dose-response, small sample size, clustering, rare
outcomes, and multiple comparisons. In normally distributed data the standard error, standard
deviation, or confidence intervals should be reported. In non-normally distributed data, inter-
quartile range should be reported. For cohort studies, if the outcome has a greater than 10 percent
prevalence, consider if the risk ratio and relative risk need to be calculated]

Yes ....................................................................... Explanation for rating:

Partially ................................................................

No ........................................................................

Cannot determine .................................................

26. Are any important harms or adverse events that may be a consequence of the
intervention/exposure missing from the results? [PI: Identify all important harms, including
timing of measurement, that one would expect be reported in the study. Drop if not relevant to body
of literature.]

PI:

Yes ....................................................................... Explanation for rating:

Partially ................................................................

No ........................................................................

Assessment of harms not applicable to this

study.....................................................................
27. Are the statistical methods used to assess the main harm or adverse event outcomes appropriate
to the data? [Abstractor: Question relates to precision and may not be relevant for systematic
reviews that are able to pool data. The statistical techniques used must be appropriate to the data
and take into account issues such as controlling for dose-response, small sample size, clustering,
rare outcomes, and multiple comparisons. In normally distributed data, the standard error,
standard deviation, or confidence intervals should be reported. In non-normally distributed data,
inter-quartile range should be reported.]

Yes ....................................................................... Explanation for rating:

Partially ................................................................

No ........................................................................

Not applicable: harms not reported ......................

Interpretation
Appropriately Based on Results
28. Are results believable taking study limitations into consideration? [Abstractor:This question is
intended to capture the overall quality of the study. Consider issues that may limit your ability to
interpret the results of the study. Review responses to earlier questions for specific criteria.]

Yes ....................................................................... Explanation for rating:

Partially ................................................................

No ........................................................................
Presentation and Reporting
Completeness, Clarity, and Structure
29. Is the source of funding identified? [PI: The relevance of this question will depend upon the
topic. This question may be modified to identify particular sources of funding (e.g., industry,
government, university, or foundation funding).]
PI:

Yes ....................................................................... Explanation for rating:

No ........................................................................