RESEARCH METHODS IN

PSYCHOLOGY (PSYC4045)
Q U A N T I T A T I V E A N A LY S I S C O M P O N E N T
C O M PA R I N G M E A N S : A N OVA
WHAT HAPPENS IF YOU HAVE MORE
THAN TWO GROUPS TO COMPARE?

• A Student’s t-test is only set up to

compare the means of two groups
•So, what happens if you have three or
more groups? Lot’s of t-tests?
•First, that means lots of t-tests (for three
groups that’s three t-tests: A-B, A-C, and B-
C)
•Second each time you do a t-test you have
Type 1 error; each time you do a t-test you
add onto that error
ANALYSIS OF VARIANCE (ANOVA)

• General name for a set of statistical

procedures (also called General Linear
Model)
• Allow one to analyse the effect of one or
more independent variables on a
dependent variable
• Each of the independent variables may
have 2 or more levels/groups
DECISION TREE (LAHER, 2016)
 ANOVA SINGLE FACTOR DESIGNS

(1) Simple 1-way ANOVA

(single factor between-subjects
design)

(2) 1-way ANOVA with repeated measures

(within-subjects design)
(1) Simple 1-way ANOVA

A1 A2 A3
S1 S6 S11
S2 S7 S12
S3 S8 S13
S4 S9 S14
S5 S10 S15
(2) 1-way ANOVA with repeated measures

A1 A2 A3
S1 S1 S1
S2 S2 S2
S3 S3 S3
S4 S4 S4
S5 S5 S5
 SIMPLE 1-WAY ANOVA
•Extension of t-test for independent
means
• 1 factor (independent variable)
• 2 or more levels of the IV
• Data measured on the interval or ratio
scale
• Research Question: Is there a
difference between any of the means?
 DECISION TREE QUESTIONS
• How many • How many IVs?
groups/conditions? • 1
• 3 or more • 2 or more
• Related or • Nature (esp. scale
independent of measure) of DV?
groups? • Numerical
• Independent • Ordinal (otherwise a
groups/observations Kruskal-Wallis test)
• Related groups (esp. • DV at least interval
pairs)
• Related observations • Parametric
(repeated measures) assumptions?

9
 STATISTICAL HYPOTHESES

H0: All the means are equal

μ1 = μ2 = …. = μk

H1: Not all the means are equal

 VARIANCE IN INFERENCE

• If scores vary a lot within groups (suggesting they

vary a lot in the population), then there is a good
chance any differences between sample group means
are there by chance (because of sampling variation)

• If there is less variance within groups, and/or enough

variance (big enough difference/s) between group
means, then it is much less likely that the observed
differences between sample group means are there
by chance (because of sampling variation) – we take
this as evidence of underlying differences in means in
the population
GRAPHICALLY, WHAT ARE YOU
TESTING?
OR, TO PUT IT ANOTHER WAY …
 SAME LOGIC, SO WHAT’S THE BIG
DEAL?

t-Test F-test
Distributio Normal Positively skewed
n
Tell us which Does not tell us
means are which means are
Alternative
different different
hypothesis
Can be directional Cannot be
directional
Family- Becomes high Controls for this
wise error running many with a single test
(Type 1) tests
Conclusion: Always use F-test if there are more
than two groups
 SOME NEW TERMS TO CALCULATE
ANOVA

• Sums of squares: SS = (X -

X)2

• Mean square: MS = SS / df

• Degrees of freedom: df = (n-1)

 ANOVA SUMMARY TABLE

Simple 1-way ANOVA

Source SS df MS F p
value

A (between SSA dfA MSA FA

subjects)

Error SSError dfError MSError

Total SSTotal dfTotal

ANOVA CALCULATION
 ASSUMPTIONS OF THE ANOVA

• DV is measured on an interval or
ratio scale
• DV is normally distributed
• The variances of the DV across the
different groups are similar
(Homogeneity of variances)
 VIOLATION OF ASSUMPTIONS

Normality
In general, F test is robust for
violations, especially of normality.
However, can transform the data
to make it more normal
 HOMOGENEITY OF VARIANCES

Particularly a problem when

unequal group sizes.
Options:
- Ignore
- Recalculate adjusted F-statistic
(Welch statistic)
- Transform the data
 ONE-WAY ANOVA: EFFECT SIZE

Calculated as a ratio of how much of the total

variation in the DV is (statistically) explained by
variation in the IV (called an eta-squared or η²):
Variance (sum of
squares)
between groups

Total variance
(sum of squares)

Smal Moderat Large

l e
0.0 0.06 0.14
 ONE-WAY ANOVA: POST-HOC

• A significant F-test tells us at least one pair of

means is different ()
• Post-hoc analyses tell us which pairs are different
• Essentially the same logic as an independent
samples t-test, for each pair of means

Bonferroni/
Tukey Fisher’s LSD
Groups More than 3 Less than 5
Family-wise
error Conservative Liberal
ANOVA CLASS EXAMPLE
 APPROACHING RESEARCH QUESTIONS
Candidat
Nature Unit Candidate Possibiliti
e Relations Candidate
of Ontolo Epistemol Methodolo of data es and
sampling hip to data
questio gy ogy gy analy collection limits of
techniqu time analyses
n sis techniques inference
es
What is What is What are What What Where How can How Given Given O&E
the the the kinds & combinatio is the and in we many paradigm, and M &
purpose nature limits of n of level what form optimally data unit of analytic
(aim) of of the knowing methods at can we ‘extract’, points analysis, technique
asking world and that are which access source and collection what
the this knowledge aligned to we these and/or across instrument, inferences
questio questio that this the O&E of are targets at produce what sequence & can or
n? n question this analyz the information period of aim what more
assume assumes question ing specified that time do analytic importantl
s? (prescribes can we our level? reflects the we need techniques y cannot
)? use to data? units and to collect are possible be drawn?
answer it? target of data? for
analysis? answering
the
question?

Compar Wellbein Objective Quantitativ Individ Big sample Non- Cross- Inferences
e groups g+ reality e, mono- uals (N=1853), probability sectional about means
(e.g. office method, employed (3+ groups),
office types adults ANOVA or
types) Kruskall-
Wallis
 ANOVA CLASS EXAMPLE

• IV = Office types
• IV levels
1. Private space
2. Shared space
3. Open-plan space
4. Agile space
5. Reservable space
• DVs = perceived health, [physical discomfort],perceived
psychological discomfort, perceived psychological wellbeing. We’ll
look at psychological discomfort as the DV
• Research question: Do different office types have different
psychological wellbeing means?
• Statistical hypothesis:
H0 is 1 = 2 = 3 = 4 = 5
H1 is at least one of: 1  2, 1  3, 1  4, 1  4, 1  5, 2  3, etc.
 ANOVA CLASS EXAMPLE

• Let α =5%=0.05
• Assumptions
• Normality
• Random independent sampling (a person can be in only
one group)
• Scale of measurement is additive (i.e. at least of an
interval scale)
• Homogeneity of variance
 ANOVA CLASS EXAMPLE

FIRST: TEST EQUALITY OF VARIANCE

• HO: 21 = 22 = 23 = 24 = 25
• H1: At least one variance is different
• Let α =5%=0.05
• No assumption tests were reported (no tests of
normality, no equality of variance test, no
skewness or kurtosis reported)
 ASSUMPTION TESTS

• Normality:
Skewness = -0.205 Kurtosis = -0.488
Kilmogorov-Smirnov – 0.066, p=0.000
Shapiro-Wilks – 0.986, p=0.000

• Homogeneity of variance:
Levene’s test – 1.898, p=0.108
 ANOVA CLASS EXAMPLE

MEAN COMPARISONS:
• Test statistic: ANOVA
• See ANOVA on p. 835 [F = 24.984, p=0.000]
• p< α
• Therefore reject Ho
• Thus accept H1
• At least one pair of means is different
• Only tells us that two or more of the means are different, but doesn’t tell
us how means are different
• Thus: post-hoc tests: a series of tests where we compare each group’s
mean against the means of all the other groups
• Use for more than 5 groups: Bonferroni or Tukey’s HSD
 ANOVA CLASS EXAMPLE
• List means in descending order (p. 835-836):

Mean
Reservable 3.14
Agile 3.05
Shared 3.05
Open-plan 2.99
Private 2.12
 ANOVA CLASS EXAMPLE
• Conservative: 5 groups or more
• 5 Groups: Tukey’s HSD or Bonferroni
• Exemplar:
• Ho: µ1= µ2 (µ1= Private office and µ2= Shared office)
• H1: µ1 µ2
• Let α =5%=0.05
• p = 0.01
• p<α
• Therefore reject Ho
• Thus sufficient evidence to indicate difference between people working in a
Private office and people working in a Shared office on perceived
psychological discomfort
 ANOVA CLASS EXAMPLE

Post-hoc test (Bonferroni)

Write out mean differences for each group (THIS IS JUST AN EXAMPLE AS IT ISN’T IN
THE PAPER AND I HAVE ONLY TAKEN A SMALL PART OF THE TABLE)
Groups Mean Direction Significance
difference

1 and 2 0.93 negative S

1 and 3 0.87 negative S

1 and 4 0.93 negative S

1 and 5 1.03 negative S

 CLASS EXAMPLE
• Write out a summary in English (focus on significant differences and the
smallest and largest differences)
• E.g. Groups 2, 3, AND 4 are not significantly different from each other
but all have significantly higher discomfort than group 1.
• Thus, people in Private offices have less psychological discomfort than
people in Shared offices, Open-plan offices, Agile offices, and Reserved
offices
• Overall effect size
• Eta-squared
• η2=
• η2= = 0.05 = Weak [this is calculated from the actual dataset as we
aren’t given this information in the paper]
• 0.01 = weak
• 0.06 = moderate
• 0.14 = strong