META-ANALYSIS

1
HISTORY
 The first meta-analysis was performed by Karl Pearson in 1904, in an
attempt to overcome the problem of reduced statistical power in studies
with small sample sizes; analyzing the results from a group of studies can
allow more accurate data analysis (O'Rourke, 2007)
 The term "meta-analysis" was coined by Gene V. Glass, who was the first
modern statistician to formalize the use of meta-analysis, and is widely
recognized as the modern founder of the method (Glass, 1976).
  Meta-analysis is a systematic methodology to synthesize the findings
from existing empirical studies in order to shed light on the future
development of the field (Glass, 1976; Johnson & Christensen,2008).
 Meta-analysis combines quantitative results of different investigations on
a related topic (Glass, 1976)
 Meta analysis refer to the analysis of analysis. Also refer to the statistical
analysis of a large collection of analysis results from individual studies for
the purpose of integrating the findings (Glass,1976)
 Meta-analysis combines quantitative results of different investigations on
a related topic (Glass, 1976).
2
WHAT IS
META-ANALYSIS
 Meta-analysis is a statistical technique for combining the findings
from independent studies.
 Meta-analysis is most often used to assess the clinical effectiveness
of healthcare interventions; it does this by combining data from two
or more randomised control trials.
 Meta-analysis of trials provides a precise estimate of treatment
effect, giving due weight to the size of the different studies included.
 The validity of the meta-analysis depends on the quality of the
systematic review on which it is based.
 Good meta-analyses aim for complete coverage of all relevant
studies, look for the presence of heterogeneity, and explore the
robustness of the main findings using sensitivity analysis.

(Davies & Crombie, 2001)

3
WHAT IS META ANALYSIS?
Quantitative approach for systematically combining results of
previous research to arrive at conclusions about the body of
research (Arindam Basu,2005)
.

WHAT DOES IT MEAN?

 Quantitative : numbers
 Systematic : methodical
 combining: putting together
 previous research: what's already done
 conclusions: new knowledge
 THE POPULARITY OF META ANALYSES publications

3000

2500

2000
Number of Publications

1500

1000

500

0
93-94 94-95 95-96 96-97 97-98 98-99 99-00 2000-1 2001-2 2002-3 2003-4
Year of Publications

Number of Meta Analysis publications are steadily increasing since

1993. We graphed the counts of journal articles included “meta
analysis” as “publication type” from Pubmed, from years 1993
through 2004
BENEFITS OF META-ANALYSES
Offers a rational and helpful way of dealing with a number of
practical difficulties that beset anyone trying to make sense of
effectiveness research.

1. A clearer picture
-Small studies tend to be inconclusive- they may show no statistical
difference between the treated and control groups, but on the other hand
they may be unable to exclude the possibility of there being a sizeable
effect (that is, they have low power).
-Aggregating studies in a systematic and unbiased way may allow a clearer
picture to emerge. The question we are asking is whether, on average, a
particular treatment confers significant benefits when used for specific
patient groups.
-Meta-analysis allows this aggregate picture to emerge.

(Davies & Crombie, 2001)

6
BENEFITS OF META-ANALYSES
2. Overcoming bias

-The danger of unsystematic (or narrative) reviews is that there is plenty

of scope for bias. Certain (perhaps favourable) reports may be preferred
over those that show no benefit.
-Informal synthesis may be tainted by the prior beliefs of the
reviewer.
-Meta analysis carried out on a rigorous systematic review can
overcome these dangers -offering an unbiased synthesis of the
empirical data.

(Davies & Crombie, 2001)

7
BENEFITS OF META-ANALYSES
3. Precision
-The precision with which the size of any effect can be estimated
depends on (among other things) the number of patients studied.
-Meta-analyses that draw on patients studied in many trials thus
have more power to detect small but clinically significant effects,
and can give more precise estimates of the size of any effects
uncovered.
-This may be especially important when an investigator is looking
for beneficial (or deleterious) effects in specific subgroups.
-Individual studies may contain too few patients in the subgroup
of interest to show anything, but a clearer picture may be
presented by the systematic aggregation of data from many
individual studies
(Davies & Crombie, 2001)
8
BENEFITS OF META-ANALYSES
4. Transparency

-It is not simply the case that meta-analyses can always exclude bias
better than other forms of review.
-Their advantage also lies in the openness with which good meta-
analyses reveal all the decisions that have been taken throughout the
process of achieving the final aggregate effect sizes.
-Thus, good meta analyses should allow readers to determine for
themselves the reasonableness of the decisions taken and their likely
impact on the final estimate of effect size.

(Davies & Crombie, 2001)

9
 REQUIREMENTS FOR META ANALYSIS

 The main requirement for a worthwhile meta-analysis is first and

foremost a well-executed systematic review.

-if the original review was partial, flawed or otherwise

unsystematic, then the meta-analysis may provide a
precise quantitative estimate that is simply wrong.
-The main requirements of systematic review are easier to state than to
execute: a complete, unbiased collection of original, high-quality
studies that examine the same therapeutic question.
-Guidance on avoiding the pitfalls of systematic reviews is readily
available.

10
INTERPRETING THE MAIN
FINDINGS
 The usual way of displaying data from a meta-analysis is by a pictorial
representation (sometimes known as a blobbogram), and a summary
measure of effect size known as an odds ratio.

1. Blobbograms
The inelegantly named ‘blobbogram’ displays the findings from each
individual study as a blob or square (the measured effect), with a
horizontal line (usually the 95% confidence interval) around the main
finding. The size of the blob or square (sometimes just a small vertical
line) may vary to reflect the amount of information in that individual
study;
The length of the horizontal line represents the uncertainty of the
estimate of the treatment effect for that study. The aggregate effect size
for certain sub groupings and the overall effect size are also usually
displayed in the same figure. An example is shown in Figure 1.5

11
INTERPRETING THE MAIN
FINDINGS

2. Odds ratios
-The main measure of effect usually used in meta-analyses is the odds
ratio (OR). This somewhat confusing measure is used because it offers
some technical advantages when combining data from different studies.
-For most practical purposes, the odds ratio can be interpreted as though
it were a relative risk.6 That is, an odds ratio of 2 implies that the
defined outcome happens about twice as often in the intervention
group as in the control group; an odds ratio of 0.5 implies around a 50%
reduction in the defined event in the treated group Compared with the
controls.
-Interpreting an odds ratio in this way (as an increase or decrease in risk)
will always tend to overestimate the effect of treatment.
-However, the level of this overstatement is almost always quite small,
especially when the event rate in the intervention and control groups is
less than 30%.

12
FLAWS IN META-ANALYSES
1. Location and selection of studies
 The key difficulty lies in deciding which sets of studies are
‘combinable’.
 A meta-analysis is only as good as the set of studies on which it is
based.
 failure to find all relevant studies may produce misleading meta-
analyses.
 Once all relevant studies have been identified, decisions must be
taken about which studies are sufficiently well conducted to be
worth including.
 This process may again introduce bias, so good meta-analyses will
use explicit and objective criteria for inclusion or rejection of
studies on quality grounds.

13
FLAWS IN META-ANALYSES
2. Heterogeneity
 A major concern about meta-analyses is the extent to which
they mix studies different in kind (heterogeneity).
 Clearly, to get a precise answer to a specific question it
makes sense to collate only studies exactly matching that
question. Unfortunately, all studies differ on a number of
different dimensions:
-The patient groups studied will often differ.
-The interventions applied may differ between individual studies.
-The primary outcome examined may differ between studies, yet a
meta-analysis seeks to aggregate data to some common end-point.
-All studies are carried out in distinct settings, which may differ
markedly between the studies
14
FLAWS IN META-ANALYSES
3. Loss of information on important outcomes

 To have some ‘common currency’ of outcome on which to

combine data from different studies, meta-analyses may have to
discard some data.
 Typically, meta-analyses are concerned with dichotomous
outcomes (eg, alive/dead; cured/still ill; recurrence/non
recurrence of disease).
 To fit this framework, data from individual studies may have to
be discarded, resulting in a potential loss of information

15
FLAWS IN META-ANALYSES
4. Inappropriate subgroup analyses
 Exploring subgroup findings is a common feature of meta-
analyses, sometimes as a way of explaining a failure to find any
overall effect.
 A subset of studies may be examined in isolation (perhaps those
with high methodological quality) or, if individual patient data
have been used in conducting the meta-analysis, subgroups of
patients may be analysed separately.
 Despite the underlying studies being randomised, this
randomisation and the subsequent likely balance between
treated and control groups does not extend to subgroups defined
after the fact.
 There is great potential for confounding and misleading findings

16
FLAWS IN META-ANALYSES
5. Inadequate sensitivity analyses

-Because of the many ways in which decisions taken about

selection, inclusion and aggregation of data may affect the main
findings, it is usual for meta-analysts to carry out some sensitivity
analysis.

6. Conflict with new experimental data

 Meta-analyses seek new knowledge from existing data. One test of the validity of
this new knowledge is to compare the results from meta-analyses with subsequent
findings from large-scale, well-conducted, randomised control trials (so-called
‘mega trials’).
 The results of such comparisons have, so far, been mixed - good agreement in the
majority of cases but some discrepancies in others.
 With the benefit of hindsight, the flaws in meta-analyses that have been
subsequently contradicted by data from mega trials can often be uncovered.

17
ISSUES IN META ANALYSIS
 Choosing a model
 Fixed effects model or random effects?
 Bias in meta analysis
 poor quality of trials
 publication bias
 Quality control in meta analysis
 QUOROM guidelines
 Statistical Software for meta analysis
FIXED EFFECTS OR RANDOM EFFECTS
MODEL?
Fixed Effects Model Random Effects Model
• conduct if it is • Conduct if test of
reasonable to assume
heterogeneity is
underlying Rx effect is
SAME for all studies significant (shows
• Pooling: Mantel heterogeneity)
Haenszel OR • Assume that TRUE
• Test: test of log odds ratio comes
heterogeneity from a normal
• If significant, go for distribution
random effects model • Method: DerSimonian
• short 95% CI for Lair’s method (DSL)
summary of calculating Odds’
• smaller summary Ratio
estimate
• OR=0.78 [0.69,0.88]
• OR=0.77 [0.72,0.83]
 BIAS IN META ANALYSIS

 Poor Quality of Trials

 To avoid them, learn more at CONSORT statement
[http://www.consort-statement.org]
 Publication Bias
 study showing beneficial effects of new treatment more
likely to be published than one showing no effect
 negative trials assumed to contribute less; never show
up in the literature base
 use several approaches to avoid this
 Use Funnel Plots to examine the influence of
publication bias
QUALITY CONTROL IN MA:QUOROM TABLE

• Detailed Guidelines
• A Good Checklist
• Use it for reporting
• Meta Analysis
• Systematic
reviews
 Technical details on conducting meta-analyses can be found in the
literature (Glass, et al., 1981; Hedges, et al., 1985; Cooper, 1989),
though the process usually includes these steps:
 Independent variables (e.g., cooperative learning as a teaching and
learning method) and outcome variables (e.g., academic
achievement) of interest are clarified.
 Quantitative research that addresses the independent and outcome
variables of interest is identified.
 Quantitative information from each study that indicates the effect
the independent variable has on the outcome variable is determined.
 The data are "normalized" by determining the effect size for the data
reported in the study. The effect size is the difference between the
means of the outcome scores of the experimental and control groups
divided by the standard deviation of the scores. A positive effect of
the instructional strategy on the outcome variable is indicated by a
mean effect size across the studies that is greater than 0.

22
HOW TO DO META ANALYSIS
FOUR STEPS OF META ANALYSIS
 Identify your studies
 Determine eligibility of studies
 Inclusion: which ones to keep
 Exclusion: which ones to throw out
 Abstract Data from the studies
 Analyze data in the studies statistically
STEPS TO PERFORM A META-
ANALYSIS
 Define the meta-analytic research question
 Locate the relevant literature
 Calculate effect sizes and code moderating
variables
 Analyze the meta-analytic database
 Report and interpret the results
DEFINE THE META-ANALYTIC
RESEARCH QUESTION
 Purpose of this step
 Determine what hypotheses your meta-analysis
will test
 Estimate the strength of an effect
 Determine moderators of an effect
 Determine what types of studies you will include
in your analysis
DEFINE THE META-ANALYTIC
RESEARCH QUESTION
 Goals for this step
 Hypotheses should have theoretical value
 Should have specific inclusion criteria to make
locating studies easier
 Included studies should be appropriate for the
hypotheses being tested
LOCATE THE RELEVANT LITERATURE

 Purpose of this step

 Obtain the population of studies related to your
research hypotheses
 Modify the hypotheses and inclusion criteria of
your analysis to better fit the literature
 Goals for this step
 Should find every study that has investigated the
effect of interest
 Make your hypotheses better address the
questions that researchers have investigated in
primary research
CALCULATE EFFECT SIZES AND
CODE MODERATING VARIABLES
 Purpose of this step
 Determine what effects you will examine in each study
 Compute a specific estimate of the size of each effect
 Determine the value of your moderating variables for
each effect
 Goals for this step
 Accurately determine effect size estimates and
moderator codes
 Should try to have estimates for every effect
 Typically have two different people calculate effect
sizes and code moderators so you can estimate
reliability
ANALYZE THE META-ANALYTIC
DATABASE
 Purpose of this step
 Perform descriptive analyses to determine the
overall strength and consistency of the effect
 Perform moderator analyses to determine if
study characteristics influence the effect size
 Goal for this step
 Analyses should be valid
 Analyses should directly answer the research
questions
REPORT AND INTERPRET THE
RESULTS
 Purpose of this step
 Summarize the results of your analyses
 Relate your analyses to the research questions
 Draw conclusions based on your analyses

 Goals for this step

 Verbally describe the implications of your analyses
 Report any limitations you see regarding your analysis
 Violation of assumptions

 Power

 Representativeness

 Suggest areas of future research

CONCLUSION
 Meta-analyses offer a more systematic and quantitative approach
to reviewing important therapeutic questions.
 Nonetheless, pitfalls abound in the execution of meta-analyses
and they are fundamentally limited by the quality of the
underlying studies (the so called GIGO principle of ‘garbage in,
garbage out’).

32
RUJUKAN
1. O'Rourke, Keith (2007). "An historical perspective on meta-analysis: dealing
quantitatively with varying study results". J R Soc Med 100 (12): 579-582.
2. Glass, G. V (1976). "Primary, secondary, and meta-analysis of
research". Educational Researcher, 5, 3-8.
3. Johnson, B., & Christensen, L. (2008). Educational research: Quantitative,
qualitative, and mixed approaches.Thousand Oaks, CA: Sage.
4. Higgerson, H. (2005). Evaluation of violence prevention programs for
adolescents: A meta analysis. Unpublished doctoral dissertation. The
University of Alabama at Birmingham.
5. Davies, H.T. O & Crombie I. K. (2001). What is meta-analysis? Hayward
Group plc.(1):8, 1-8
6. Glass, G. V., McGaw, B., & Smith, M. L. Meta-analysis in Social
Research. Sage: Beverly Hills, CA, 1981.
7. Hedges, L. V., and Olkin, I. Statistical Methods for Meta-Analysis.Academic
Press: Orlando, FL, 1985.
8. Cooper, H. M. Integrating Research: A Guide for Literature Reviews. (2nd
ed.). Sage: Newbury Park, CA, 1989.
9. Arindam Basu (2005). www.pitt.edu/~super7/19011-20001/19431.ppt. assess
on 1 Mac 2012
33