Chapter 10

Analysis of Variance

STAT 190
Introduction to Statistical Methodology

Dr. Aruni Jayathilaka

Fall 2024

1
Analysis of Variance
• In chapter 8 , you learned methods for testing
𝐻0 : 𝜇1 − 𝜇2 = 0 (or equivalently 𝐻0 : 𝜇1 = 𝜇2 )
where 𝜇1 and 𝜇2 are the means of two different populations.

• Suppose we are interested in the average weight of Americans, but we are looking at three different
age groups.
• 18-30 years old.
• 31-50 years old.
• Over 50 years old.

• These three groups have means 𝜇1 , 𝜇2 and 𝜇3 respectively.

• We want to test the null hypothesis that the population means are identical.
𝐻0 : 𝜇1 = 𝜇2 = 𝜇3 vs
𝐻1 : 𝑎𝑡 𝑙𝑒𝑎𝑠𝑡 𝑜𝑛𝑒 𝑜𝑓 𝑡ℎ𝑒 𝑝𝑜𝑝𝑢𝑙𝑎𝑡𝑖𝑜𝑛 𝑚𝑒𝑎𝑛𝑠 𝑑𝑖𝑓𝑓𝑒𝑟𝑠 𝑓𝑟𝑜𝑚 𝑜𝑛𝑒 𝑜𝑓 𝑡ℎ𝑒 𝑜𝑡ℎ𝑒𝑟𝑠.
2
Analysis of Variance

3
One-way ANOVA

• One-way ANOVA is dependent on estimates of spread/dispersion.

• “One-way” indicates that there is a single factor that distinguishes the populations from each other.
• In our previous example, age is the distinguishing factor.

Assumptions:
• Assume the k populations are independent.
• Assume the k populations normally distributed.
• Each sample (group) needs ≥ 5 observations.

4
ANOVA

Note:
• The number of observations in each sample does not need to be the same.
• Draw a sample of size 𝑛𝑖 from group 𝑖 that has population mean 𝝁𝒊 and population variance 𝝈𝟐𝒊 .
ഥ𝒊 and group variance 𝒔𝟐𝒊 .
• For each sample (group), we get group mean 𝒙

Notations:
• The total number of observations in the data set: 𝑛 = 𝑛1 + 𝑛2 + ⋯ + 𝑛𝑘
• Grand total: 𝑇 = 𝑛1 𝑥ҧ1 + 𝑛2 𝑥ҧ2 + ⋯ + 𝑛𝑘 𝑥ҧ𝑘
𝑻
• Grand (overall) mean: mean of all observations in all groups 𝒙
ഥ=
𝒏

5
Analysis of Variance

6
 ANOVA

• When more than two populations or treatments are being compared, the characteristic that
distinguishes the populations or treatments from one another is called the factor under investigation.
• This factor can be qualitative or quantitative.

• Analysis of variance (ANOVA) problem involves a comparison of 𝑘 population or treatment means,

µ1 , µ2 , … µ𝑘 .
• The objective is to test

𝐻0 : 1 = 2 = . . . = k vs 𝐻1 : At least one of 's are different

7
Sources of Variation

• There are two sources of variation in this ANOVA setup:

• Within group variability - how different each participant (individual values) is from their own
group mean.
• Between group variability- variation of the population means around the grand (overall) mean.

8
Sources of Variation

• The variation of the individual values around their population means.

• Other words, how different each participant is from their own group mean.
• Within group variability(within group variance or residual variance): 𝒔𝟐𝒘 or 𝑴𝑺𝒘

2
𝑛1 − 1 𝑠12 + 𝑛2 − 1 𝑠22 + ⋯ + (𝑛𝑘 − 1)𝑠𝑘2
𝑠𝑤 =
𝑛1 + 𝑛2 + ⋯ + 𝑛𝑘 − 𝑘

2 2 2
2
𝑛1 − 1 𝑠1 + 𝑛 2 − 1 𝑠2 + ⋯ + (𝑛 𝑘 − 1)𝑠𝑘
𝑠𝑤 =
𝑛−𝑘

2
𝑆𝑆𝑤
𝑠𝑤 = = 𝑀𝑆𝑤
𝑑𝑓𝑤

9
Sources of Variation

• The variation of the population means around the grand (overall) mean.
• Other words, the total variation between each group mean and the overall mean.
• Between group variability: 𝐬𝒃𝟐 or 𝐌𝐒𝐛

2
𝑛1 𝑥ҧ1 − 𝑥ҧ + 𝑛2 𝑥ҧ2 − 𝑥ҧ 2 + ⋯ + 𝑛𝑘 𝑥ҧ𝑘 − 𝑥ҧ 2
𝑠𝑏2 =
𝑘−1

𝑆𝑆𝑏
𝑠𝑏2 = = 𝑀𝑆𝑏
𝑑𝑓𝑏

10
Sources of Variation

• We assume that there is a common variance σ2 for the populations.

• When the variability within the k populations is small relative to the variability among their
respective means, this suggests that the population means are in fact different.

11
ANOVA Testing Procedures

• Verify assumptions of independence and normality for the validity of the test.

Step 1:
State the hypotheses:
𝐻0 : 1 = 2 = . . . = k vs 𝐻1 : At least one population mean is different

12
ANOVA Testing Procedures

Step 2:
• Estimate measures of variability:
• 𝑠𝑤 2 , measures the within-group variability.

• 𝑠𝑏2 , measures the between-group variability.

• The question now becomes:

• Do the sample means vary around the grand mean more than the individual observations around
the sample means?
• i.e. Is 𝒔𝟐𝒃 > 𝒔𝟐𝒘 ?

• If they do, this provides evidence that the population means are different.

13
ANOVA Testing Procedures

Step 2:
• To test the null hypothesis with 𝛼 = 0.05, we use the following test statistic:

2
∗
𝑠𝑏 𝑀𝑆b
𝐹 = 2 =
𝑠𝑤 MSw

2
• Under 𝐻0 , both 𝑠𝑤 and 𝑠𝑏2 estimate σ2 .
• Thus, we expect 𝐹 ∗ to be close to 1.

• If a difference exists between population means, 𝑠𝑏2 > 𝑠𝑤

2 and 𝐹 ∗ > 1.

14
ANOVA Testing Procedures

• The ratio of variances follows an F distribution with two sets of degrees of freedom as parameters
𝑑𝑓1 and 𝑑𝑓2 .
• 𝑑𝑓1 = k − 1 in the numerator.
• 𝑑𝑓2 =n − k in the denominator.

• Notation: 𝑋~𝐹𝑘−1,𝑛−𝑘

• If k = 2, 𝐹1,𝑛−𝑘 ≈ 𝑡𝑛1+𝑛1−2 , i.e this F-test reduces to a two-sample two-tailed t-test.

15
F distribution
Recall:
• The F-distribution is a skewed distribution of
probabilities similar to a chi-squared distribution

• Its support be the set of positive (non-negative) real

numbers.

• As the shape of an F-distribution depends on two sets

of degrees of freedom as parameters 𝑑𝑓1 and 𝑑𝑓2 .
• Increase in the degrees of freedom 𝑑𝑓1 and 𝑑𝑓2 , its
skewness decreases.

16
ANOVA Testing Procedures

Step 3:
• P-values & critical values use an F distribution with 𝑑𝑓1 = k − 1 and 𝑑𝑓2 = n − k.
• No 2-tailed testing here! Only upper-tail!

17
ANOVA Testing Procedures

Step 4:
• Rejection Decision:
• Reject 𝐻0 , if p−value ≤ 𝛼
• OR
• Reject 𝐻0 , if 𝐹 ∗ > 𝐹𝟏−𝛼,𝑘−1,𝑛−𝑘

• State conclusion in the context of the problem.

18
ANOVA
Example:
Consider comparing weights (pounds) over the three age groups.
▪ 18-30 years old
▪ 31-50 years old
▪ Over 50 years old
These three independent populations have means μ1, μ2, and μ3, respectively. Test if there is a
difference in weight for at least one age group at the 𝛼 = 0.05 significance level.
Sample summary,

19
ANOVA
Step 1:
State the hypotheses:
• vs

Step 2:
• Estimate measures of variability:
2 =
𝑠𝑤

20
ANOVA

𝑇
𝑥ҧ =
𝑛

𝑠𝑏2 =

21
ANOVA

• To test the null hypothesis with 𝛼 = 0.05, we use the following test statistic:

𝐹∗ =

• For F distribution,
• 𝑑𝑓1 = 3 − 1 = 2
• 𝑑𝑓2 = 26 + 31 + 44 − 3 = 98

22
ANOVA

Step 3:
• p-value =
• C.V. =

Step 4:

Or

• Conclude that at least one of the age groups differs from one of the others in terms of weight.

23
ANOVA Table

24
ANOVA Table

Source SS df MS F p-value

Between 7536.772 2 3768.386 36.588 1.353 × 10−12

(Treatment)

Within 10093.51 98 102.995 - -

(Error)

Total 17630.28 100 - - -

25
ANOVA
• Given actual data, we can do this all by hand or use R.

R Code:
• anova(aov(y~x))
• First, create an ANOVA object, using aov() function.
• Summarize using the anova() function.
• Let Y be the continuous variable (quantitative variable) and X be the grouping variable
(categorical variable ).

• There are no arguments for confidence level or alternative because:

• Use p-value, compare to whatever 𝛼 you want!
• Always a 1-sided test!

26
Multiple Comparisons

• Our null hypothesis is 𝐻0 : 1 = 2 = . . . = k for all groups

• When 𝐻0 is rejected, we can conclude at least one mean is differ.
• But, we do not know exactly which means differ.

• We would need to conduct additional tests to find where the differences are.

27
Multiple Comparisons

• In this case, we are performing multiple comparisons.

• Multiple comparisons conducts an analysis of all possible pairwise means.

• For example, with three brands of toothpastes, A, B, and C, if the ANOVA test was significant, then
multiple comparison methods would compare the three possible pairwise comparisons: 32 = 3
• Brand A to Brand B
• Brand A to Brand C
• Brand B to Brand C

28
Multiple Comparisons
Issue:
Recall: whenever you perform a hypothesis test, there is always a chance of committing a type I error.
• 𝛼 = 𝑃(𝑜𝑛𝑒 𝑡𝑒𝑠𝑡 𝑟𝑒𝑗𝑒𝑐𝑡𝑠 |𝐻0 𝑖𝑠 𝑡𝑟𝑢𝑒) = 𝑃(𝑖𝑛𝑐𝑜𝑟𝑟𝑒𝑐𝑡 |𝑟𝑒𝑗𝑒𝑐𝑡𝑖𝑜𝑛)

• Performing multiple hypothesis, we increase our overall probability of committing a type I error.
𝑘
• For k groups, there are g = 2
possible two-sample t-tests
• If each test uses 𝛼, the set of g tests have familywise error rate.

𝐹𝑎𝑚𝑖𝑙𝑦𝑤𝑖𝑠𝑒 𝑒𝑟𝑟𝑜𝑟 𝑟𝑎𝑡𝑒 = 1 − (1 − 𝛼)𝑔

where:
𝛼 : The significance level for a single hypothesis test. g : The total number of tests.

29
Multiple Comparisons
Example:
If we conduct just one hypothesis test using 𝛼 = 0.05,
𝛼𝑓𝑤 =
If we conduct three hypothesis tests at once using 𝛼 = 0.05,
𝛼𝑓𝑤 =

It’s easy to see that as we increase the number of statistical tests, increase our overall probability of
committing a type I error.
• One way to deal with this is by using a Bonferroni Correction.
• Goal: Keep overall “familywise error rate ” at desired 𝛼 level, 𝛼𝑓𝑤 = 𝛼
• How? Be stricter with each t-test – harder to reject.

30
Bonferroni Correction
• To perform the Bonferroni correction, simply divide the original significance level 𝛼 by the number of
tests being performed.
• It gives wide intervals.
• Let, Bonferroni correction as 𝛼 ∗ , which is the new significance level for an individual comparison.

𝛼 𝛼
𝛼∗ = = 𝑘
𝑔
2
𝑘
where, 𝑔 = 2

𝜶
• Compare the “original” t-test pvalue to 𝜶∗ =
𝒈

Or
• Compare the “adjusted” t-test pvalue to the 𝛼𝑓𝑤 = 𝛼 → adjusted p−value = original pvalue × g

31
Multiple Comparisons
• Consider the null hypothesis 𝐻0 : i = j that compares populations 𝑖 and 𝑗.
𝛼
• Suppose we want to test this hypothesis with a significance level of 𝛼 ∗ = 𝑘
2
𝑥ҧ𝑖 − 𝑥𝑗ҧ
• Calculate 𝑡𝑖𝑗 =
2 1 1
𝑠𝑤 +
𝑛𝑖 𝑛𝑗

• This is the test statistics for a two-sample t-test.

• But, we estimate variance (pooled variance - 𝒔𝟐𝒘 ) based on all populations, not just populations
𝑖 and 𝑗.

• 𝑡𝑖𝑗 has a t distribution with 𝑑𝑓 = 𝑛 − 𝑘, under 𝐻0 .

• Calculate the p-value, p based on a t distribution with 𝑛 − 𝑘 degrees of freedom.
• Reject 𝐻0 , if p−𝐯𝐚𝐥𝐮𝐞 ≤ 𝜶∗ or adj. p-value ≤ 𝜶.
32
Multiple Comparisons
Example:
Return to the weight by age group example. We found that the population means were not all
identical. Compare each pair of age groups to see where the difference lies. There will be three
possible pairwise comparisons: 32 = 3.

𝑥ҧ1 − 𝑥ҧ2
𝑡12 = =
2 1 1
𝑠𝑤 +
𝑛1 𝑛2

Since p−value ≤ 𝛼 ∗ = 0.0167, reject 𝐻0 and conclude there is a difference in the mean weights of 18-
30 year olds and 31-50 year olds.
33
Multiple Comparisons
Example:

𝑥ҧ1 − 𝑥ҧ3
𝑡13 = =
2 1 1
𝑠𝑤 +
𝑛1 𝑛3

Since p−value ≤ 𝛼 ∗ = 0.0167, reject 𝐻0 and conclude there is a difference in the mean weights of 18-
30 year olds and over 50 year olds.
34
Multiple Comparisons
Example:

𝑥ҧ2 − 𝑥ҧ3
𝑡23 = =
2 1 1
𝑠𝑤 +
𝑛2 𝑛3

Since p−value ≤ 𝛼 ∗ = 0.0167, reject 𝐻0 and conclude there is a difference in the mean weights of 31-
50 year olds and over 50 year olds.
35
Multiple Comparisons

Example:
• In this case, all three comparisons were found to be significant.
• This does not always have to be the case.
• Some population means may be the same whereas others are different.

36
Bonferroni Correction in Confidence Intervals
• Take groups i and j and build a two-sample confidence interval with TWO changes
2 again and penalize the alpha in the critical value.
• Use 𝑠𝑤

2
1 1
(𝑥ҧ𝑖 − 𝑥𝑗ҧ ) ± 𝑡1−𝛼
,𝑑𝑓=𝑛−𝑘
𝑠𝑤 +
2×𝑔 𝑛𝑖 𝑛𝑗

[18-30] – [31-50];

I am 95% confident that the interval (-19.572, -6.428) captures the true difference in average weight
between 18-30 year olds and 31-50 year olds. 37
Multiple Comparisons

• Multiple testing procedures other than the Bonferroni procedure exist.

• Bonferroni is traditionally one of the most conservative measures and can suffer from lack of power.
• Different comparison procedures can be useful for different situations.

38