Dr Simone Calabrich

PRP1001/ JHX 1003

Class Notes

Week 11

Replication Crisis

The replication crisis, also known as the reproducibility crisis, is a

methodological issue that has been plaguing various scientific fields, including
psychology, linguistics, and sports science in recent years. It refers to the growing
concern that many scientific studies, even those published in reputable journals, are
difficult or impossible to reproduce or replicate, which undermines confidence in their
findings.

 Replication involves conducting a study again, following the identical methods

and procedures used in the original study, to verify its results.
 Replication is a cornerstone of scientific research. It increases confidence in
the validity of the findings, refines theoretical frameworks, and dispels biases.
 Journals often favour novel, groundbreaking research, which may incentivise
researchers to publish innovative but less reproducible studies.
 Inadequate sharing of methodological details can hinder successful
replication.
 Promoting transparency and openness in research, such as pre-registering
studies and sharing data, can help.

Meta Analysis

Meta-analysis is a statistical technique used in research to combine and

analyse the results of multiple studies on a similar topic, thereby generating a single
estimate of the main effect. It provides a more objective way to summarise research
evidence, making it a powerful tool in evidence-based practice.

The primary goal of a meta-analysis is to increase statistical power by

increasing sample size, thereby providing more accurate effect size estimates. It

1
 Dr Simone Calabrich

helps to reconcile inconsistent findings across studies and to uncover trends that
might not be apparent in individual studies.

Effect size measures the magnitude of a relationship (e.g., between variables)

or the size of an effect (e.g., treatment or intervention) in a population. It quantifies
the size of the difference, which can help in understanding the practical significance
of research results.

Effect sizes are crucial for meta-analyses as they provide a common metric
that allows for the comparison and aggregation of results from different studies. They
give context to statistical significance, helping to discern whether a result, while
statistically significant, is practically meaningful.

Publication Bias

Publication bias occurs when the outcome of a study influences its likelihood
of being published. Studies with positive or significant findings are more likely to be
published than those with non-significant or negative results.

Publication bias can significantly distort the results of a meta-analysis. If the

studies included in a meta-analysis are not representative of all studies conducted
on the topic (because negative or non-significant results remain unpublished), the
meta-analysis may overestimate the true effect size. This bias can lead to an overly
optimistic conclusion about the effectiveness of an intervention or the strength of a
relationship.

The push for open science practices, such as the publication of all research
regardless of findings, can help mitigate this bias in the future.

Effect sizes

Effect sizes are statistical measures that quantify the magnitude of an

observed effect or relationship in a study. They provide meaningful information about
the strength and direction of the effect, allowing for a better understanding and
interpretation of research findings. Including effect sizes along with p-values in
publications is crucial because it enhances the transparency and comprehensibility
of the results.

2
 Dr Simone Calabrich

Effect sizes are particularly important because p-values alone do not provide
information about the practical significance of an effect. A p-value only indicates the
probability of obtaining the observed results under the null hypothesis, assuming the
effect does not exist. By contrast, effect sizes provide a standardised measure of the
effect, making it easier to compare and interpret results across different studies or
contexts.

There are various types of effect sizes, each appropriate for different research
designs and statistical analyses. Some common effect sizes include:

 Pearson's correlation coefficient (r): Measures the strength and direction of

the linear relationship between two continuous variables. It ranges from -1 to
+1, with values closer to -1 or +1 indicating a stronger relationship.
 Cohen's d: Frequently used in studies comparing means between two
groups. It quantifies the difference between the group means in terms of
standard deviations. A larger Cohen's d indicates a larger effect size.
 Eta squared (η²): Used in analysis of variance (ANOVA) to measure the
proportion of variance explained by the independent variable(s). It ranges
from 0 to 1, with higher values indicating a larger effect size.

Confidence Intervals

Confidence intervals (CIs) are a statistical concept that provides a range of

values within which we can reasonably expect the true population parameter to lie.
They help quantify the uncertainty associated with estimating an unknown parameter
based on a sample from the population.

A confidence interval consists of two numbers: an upper limit and a lower

limit. These limits define a range of values within which we can be confident, at a
certain level of confidence, that the true parameter lies. Commonly used levels of
confidence are 95% or 99%.

The interpretation of a confidence interval is that if we were to repeat the

study numerous times and calculate a confidence interval for each sample,
approximately 95% (or the chosen confidence level) of those intervals would contain
the true population parameter.

3
 Dr Simone Calabrich

It is important to note that the calculated interval provides an estimate of the

population parameter, but it is not guaranteed to contain the true value in any given
study.

When a confidence interval contains zero, it implies that the observed effect or
difference is not statistically significant at the chosen level of confidence. In other
words, the interval includes the possibility of the true parameter being zero or having
no effect.

 For example, if a confidence interval for the difference in means between two
groups contains zero, it suggests that there is insufficient evidence to
conclude that there is a significant difference between the groups. Similarly, if
a confidence interval for the correlation coefficient includes zero, it indicates
that there is no significant linear relationship between the variables.
 It's important to note that a confidence interval containing zero does not
definitively prove the absence of an effect or relationship. It simply suggests
that the observed data do not provide strong evidence to reject the null
hypothesis or support the presence of a statistically significant effect.

A small confidence interval indicates a high level of precision and a narrower

range of values within which the true parameter is likely to lie. This suggests that the
sample data provide strong evidence for estimating the parameter with greater
accuracy. A small confidence interval indicates a relatively low amount of variability
or uncertainty in the estimate.

On the other hand, a large confidence interval indicates a lower level of precision
and a wider range of plausible values for the true parameter. This suggests that the
sample data provide less conclusive evidence, and the estimate is associated with a
higher degree of uncertainty or variability.

A larger sample size generally results in smaller confidence intervals as it

reduces the variability and provides more precise estimates.

Interpreting the width of a confidence interval should consider the practical

significance of the estimated effect, the specific research question, and the context
of the study. A small confidence interval indicates a more precise estimate and
provides greater confidence in the findings, while a large confidence interval
suggests more uncertainty and warrants caution in drawing conclusions.

4
 Dr Simone Calabrich

Error bars

Error bars can be used to visually represent confidence intervals in graphs or

plots. They provide a graphical representation of the uncertainty or variability
associated with the estimated values.

In a typical setting, error bars are displayed as vertical lines extending above and
below a point estimate on a graph. The length of the error bars corresponds to the
width of the confidence interval. If the confidence interval is narrow, the error bars
will be short, indicating a more precise estimate. Conversely, if the confidence
interval is wide, the error bars will be long, indicating a larger degree of uncertainty in
the estimate.

Interpreting error bars depends on the context and the specific information being
conveyed. Here are a few common interpretations of error bars:

Overlapping error bars: When error bars from different groups or conditions
overlap, it suggests that there is no statistically significant difference between the
groups at the chosen level of confidence. This interpretation is based on the idea
that if the confidence intervals overlap, the estimated values are not significantly
different from each other.

Non-overlapping error bars: When error bars do not overlap, it suggests that
there may be a statistically significant difference between the groups. However, it is
important to note that non-overlapping error bars do not guarantee statistical
significance. The calculation of p-values or conducting hypothesis tests is necessary
to determine the significance.

It's crucial to note that error bars represent uncertainty and variability, and they
do not provide definitive proof of statistical significance. They are a visual tool to aid
in understanding the range of plausible values for the estimated parameter. The
interpretation of error bars should be done in conjunction with appropriate statistical
analysis and considering the context and research question.

5
 Dr Simone Calabrich

Statistical Power

Statistical power refers to the probability of correctly rejecting the null hypothesis
when it is false. In other words, it measures the ability of a statistical test to detect a
true effect or relationship if it exists in the population.

Power is influenced by several factors, including the significance level (α), effect
size, sample size, and variability in the data. A higher power indicates a greater
likelihood of correctly detecting a true effect, while a lower power suggests an
increased chance of failing to detect a true effect.

Sample size plays a crucial role in determining statistical power. Increasing the
sample size generally leads to higher power, as it provides more information
and reduces variability in the data. With a larger sample size, the estimate of the
effect becomes more precise, making it easier to distinguish it from random
variability.

When planning a study or experiment, it is important to consider the desired

power level. Researchers typically aim for a minimum power of 80% or higher, which
means they want to have an 80% chance of correctly detecting a true effect. To
achieve the desired power level, sample size calculations can be performed, taking
into account the expected effect size, variability, and significance level.