Linear Contrasts

and Multiple Comparisons

(Chapter 9)
• One-way classified design AOV example.
• Develop the concept of multiple comparisons and linear
contrasts.
• Multiple comparisons methods needed due to potentially large
number of comparisons that may be made if Ho rejected in the
one-way AOV test.
Terms: MCPs:
Linear Contrasts Fisher’s Protected LSD
Multiple comparisons Tukey’s W (HSD)
Data dredging Studentized range distribution
Mutually orthogonal contrasts Student-Newman-Keuls procedure
Experimentwise error rate Scheffe’s Method
Comparisonwise error rate Dunnett’s procedure
STA 6166 - MCP 1
 One-Way Layout Example

A study was performed to examine the effect of a new sleep inducing drug on a
population of insomniacs. Three (3) treatments were used:
Standard Drug
New Drug
Placebo (as a control) What is the role of the placebo in this study?
What is a control in an experimental study?

18 individuals were drawn (at random) from a list of known insomniacs

maintained by local physicians. Each individual was randomly assigned to
one of three groups. Each group was assigned a treatment. Neither the
patient nor the physician knew, until the end of the study, which treatment they
were on (double-blinded).
Why double-blind?
A proper experiment should be:
randomized, controlled, and double-blinded. STA 6166 - MCP 2
 Response: Average number of hours of sleep per night.

Placebo: 5.6, 5.7, 5.1, 3.8, 4.6, 5.1

Standard Drug: 8.4, 8.2, 8.8, 7.1, 7.2, 8.0
New Drug: 10.6, 6.6, 8.0, 8.0, 6.8, 6.6

yij = response for the j-th individual on the i-th treatment.

Degrees
Standard Sums of of Mean
Placebo Drug New Drug Source Squares Freedom Square F statistic P-value
5.60 8.40 10.60 Between Groups 33.16 2 16.582 15.04 0.00026
5.70 8.20 6.60 Within Groups 16.54 15 1.102
5.10 8.80 8.00 Total 49.70 17
3.80 7.10 8.00
4.60 7.20 6.80 TSS  SSW  SSB
5.10 8.00 6.60
sum
mean
29.900
4.983
47.700
7.950
46.600
7.767
(y
i, j
ij  y  ) 2   ( yij  yi ) 2
i, j
  ni ( yi  y ) 2
i
variance 0.494 0.455 2.359
pooled variance 1.102 SSB
2
MSB  sb 
SSW 16.537 t 1
variance of the means 2.764 2 SSW
Between mean SSQ (SSB) 16.582 MSW  sw 
nT  t
MSB
F ~ Fdfb ,dfw
Hartley’s test for equal variances: MSW

Fmax = 4.77 < Fmax_critical = 10.8 STA 6166 - MCP 3

 Excell Analysis Tool Output

Anova: Single Factor

SUMMARY
Groups Count Sum Average Variance
Placebo 6 29.9 4.983333 0.493667
Standard Drug 6 47.7 7.95 0.455
New Drug 6 46.6 7.766667 2.358667

ANOVA
Source of Variation SS df MS F P-value F crit
Between Groups 33.163333 2 16.58167 15.04082 0.00026 3.682317
Within Groups 16.536667 15 1.102444

Total 49.7 17

What do we conclude here?

STA 6166 - MCP 4

 Linear Contrasts and Multiple Comparisons

If we reject H0 of no differences in treatment means in favor of

HA, we conclude that at least one of the t population means
differs from the other t-1.

Which means differ from each other?

Multiple comparison procedures have been developed to help

determine which means are significantly different from each other.

Many different approaches - not all produce the same result.

Data dredging and data snooping - analyzing only those
comparisons which look interesting after looking at the data
– affects the error rate!

Problems with the confidence assumed for the comparisons:

1-a for a particular pre-specified comparison?
1-a for all unplanned comparisons as a group?
STA 6166 - MCP 5
 Linear Comparisons

Any linear comparison among t population means, m1, m2, ...., mt can
be written as:
l  a1m1  a2m2 at mt
t
Where the ai are constants satisfying the constraint:  ai  0
i 1
Example: To compare m1 to m2 we use the equation:

a1  1
l  m1  m2 with
coefficients a2  1 Note
constraint
a3  a4    at  0 is met!

m2  m3
l  m1   (1)m1  (  21 )m 2  (  21 )m 3
2
STA 6166 - MCP 6
 Linear Contrast

A linear comparison
estimated by using group
l  m 1  m 2  y1  y 2
means is called a linear

l  m  m 2  m 3  (1)y  (  1 )y  (  1 )y
contrast.  
1 1 2 2 2 3
2
Variance of a linear contrast:
2  a1
2 2 2

t 2
sw2  MSE

a2 at 2 ai
V (l)  sw  n   n   sw
 1 n2 t
i 1
ni
( MSW )
2
 t  MSl
  ai yi  F ~ F1,nT t
Test of
  i 1 
significance MSl
 t ai2 
MSE
  
H : l = 0 vs. H : l  0
o a  i 1 ni 
STA 6166 - MCP 7
 Orthogonal Contrasts

lˆ1  a1 y1  a2 y2    at yt
lˆ  b y  b y    b y
2 1 1 2 2 t t

These two contrasts are said to be orthogonal if:

t
l1  l2  a1b1  a2b2    at bt   ai bi  0
i 1

in which case l1 conveys no information about l2 and vice-

versa.

A set of three or more contrasts are said to

be mutually orthogonal if all pairs of linear
contrasts are orthogonal.
STA 6166 - MCP 8
 ˆl  y   y2  y3  Compare average of drugs (2,3) to placebo (1).
Contrast drugs (2,3).
1 1
 2 
lˆ  y  y
2 2 3

a1  1 b1  0 Orthogonal
1
a2   b2  1
2
a3  
1
b3  1 lˆ3  y1  y2
2
lˆ4  y1  y3

Non-orthogonal a1  1 b1  1
a2  1 b2  0
Contrast Standard drug (2) to placebo (1). a3  0 b3  1
Contrast New drug (3) to placebo (1).
STA 6166 - MCP 9
 Drug Comparisons
Degrees
Standard Sums of of Mean
Placebo Drug New Drug Source Squares Freedom Square F statistic P-value
5.60 8.40 10.60 Between Groups 33.16 2 16.582 15.04 0.00026
5.70 8.20 6.60 Within Groups 16.54 15 1.102
5.10 8.80 8.00 Total 49.70 17
3.80 7.10 8.00
4.60 7.20 6.80
5.10 8.00 6.60
sum 29.900 47.700 46.600
y y   7.95  7.77 
mean 4.983 7.950 7.767 lˆ1  y1   2 3   4.98     2.88
variance 0.494 0.455 2.359  2   2 
pooled variance 1.102
SSW 16.537 lˆ  y  y  7.95  7.77  0.18
2 2 3
variance of the means 2.764
Between mean SSQ (SSB) 16.582

 2  a1 a22 a32   (1) 2 (.5) 2 (.5) 2 

2
ˆ
V l1  sW      1.102     1.102(0.25)
MSl1  1
n n n 3   6 6 6 
F
2
~ F1,nT t
MSE
 2  b1 b22 b32   (0) 2 (1) 2 (1) 2 
2
ˆ
V l2  sW      1.102     1.102(0.33)
 1
n n n 3   6 6 6 
MSlˆ1 33.10
2

F1    30.04 lˆ1
2
 2.88
2
MSE 1.102 SSlˆ1  MSlˆ1    33.10
ai2 0.25
MSlˆ2 0.10 i n
F2    0.09 i
MSE 1.102
ˆ ˆ
SSl2  MSl2 
lˆ2
2

0.18
2
 0.10
bi2 0.33
F1,15,.05  4.54 i n STA 6166 - MCP 10
i
 Importance of Mutual Orthogonality

Assume t treatment groups, each group having n individuals (units).

• t-1 mutually orthogonal contrasts can be formed from the t

means. (Remember t-1 degrees of freedom.)
• Treatment sums of squares (SSB) can be computed as the sum
of the sums of squares associated with the t-1 orthogonal
contrasts. (i.e. the treatment sums of squares can be partitioned
into t-1 parts associated with t-1 mutually orthogonal contrasts).

contrasts orthogonal  SSl1    SSlt 1  SSB

t-1 independent pieces of information about the
variability in the treatment means.
STA 6166 - MCP 11
 Example of Linear Contrasts
Objective: Test the wear quality of a new paint.
Treatments: Weather and wood combinations.

Treatment Code Combination

A m1 hardwood, dry climate
B m2 hardwood, wet climate
C m3 softwood, dry climate
D m4 softwood, wet climate
(Obvious) Questions:
Q1: Is the average life on hardwood the same as average life
on softwood?
Q2: Is the average life in dry climate the same as average life
in wet climate?
Q3: Does the difference in paint life between wet and dry
climates depend upon whether the wood is hard STAor soft?
6166 - MCP 12
 Treatment
Mean
A B C D
Q1
(in years) 13 14 20 21
ni 3 3 3 3 MSE= 5
Population t= 4
parameter m1 m2 m3 m4 nt -t= 8

Q1: Is the average life on hardwood the same as average life on softwood?
m1  m 2 m3  m 4  m1  m 2   m3  m 4 
1
H :
0  OR  2  2   0
2 2    
Comparison: l1  ( 12 )m1  ( 12 )m 2  (  12 )m3  (  12 )m 4
l̂1  ( 12 )y1  ( 12 )y 2  (  12 )y3  (  12 )y 4 Estimated Contrast

Test H0: l1 = 0 versus HA: l1  0

MSl1 What is MSl1 ?
Test Statistic: F 
MSE
Rejection Region: Reject H0 if F  F1,nT  t,a
STA 6166 - MCP 13
 2
 t 
 i i 
a y
  1
 y   1
 y    1
 y    1
 y 
2

MSl1   i1t 2   
2 1 2 2 2 3 2 4

 ai    1  2  1 2   1 2   1 2 
   2
 2
 2
 2

 i 1 ni   n1 n2 n3 n4 

2
 t 
  ai yi    2   2   2   2   49
1
13  1
14   1
20   1
21 
2

MSl1   i1t 2      147

 1   2   12    12  
 ai  2 2 2 2
1
 
1
 2
     3
 i1 i 
n  3 3 3 3   


MSl1 147
F= = = 29.4
MSE 5
F1,8,0.05 = 5.32

Conclusion: Since F=29.4 > 5.32 we reject H0 and conclude that

there is a significant difference in average life on hard versus
soft woods.
STA 6166 - MCP 14
 Treatment
Mean
A B C D
Q2
(in years) 13 14 20 21
ni 3 3 3 3 MSE= 5
Population t= 4
parameter m1 m2 m3 m4 nt -t= 8

Q2: Is the average life in dry climate the same as average life in wet climate?
m1  m 3 m2  m4  m1  m 3   m 2  m 4 
H 02 :  OR   0
2 2  2   2 
Comparison: l2  ( 12 )m1  (  12 )m2  ( 12 )m3  (  12 )m 4
l̂2  ( 12 )y1  (  12 )y 2  ( 12 )y3  (  12 )y 4 Estimated Contrast

Test H0: l2 = 0 versus HA: l2  0

MSl2
Test Statistic: F =
MSE
Rejection Region: Reject H0 if F > F1,nT - t,a
STA 6166 - MCP 15
 2
 t 
 i i 
a y
  1
 y    1
 y   1
 y    1
 y 
2

MSl2   i1t 2   
2 1 2 2 2 3 2 4

 ai    1 2   1 2  1 2   1 2 
   2
 2
 2
 2

 i 1 n i   n1 n2 n3 n4 

2
 t 
  ai yi   2  2 2
1
13   1
14  1
20   2   12
 1
21 
2

MSl2   i1     3
 1    12   2   12  
 ai 
t 2 2 2 2 2
1
 
1
 2
     3
 i1 i 
n  3 3 3 3   


MSl2 3
F= = = 0.6
MSE 5
F1,8,0.05 = 5.32

Conclusion: Since F=0.6 < 5.32 we do not reject H0 and

conclude that there is not a significant difference in average life
in wet versus dry climates.
STA 6166 - MCP 16
 Treatment
Mean
A B C D
Q3
(in years) 13 14 20 21
ni 3 3 3 3 MSE= 5
Population t= 4
parameter m1 m2 m3 m4 nt -t= 8

Q3: Does the difference in paint life between wet and dry climates depend
upon whether the wood is hard or soft?
H30 : m1  m 2  m3  m 4 OR (m1  m 2 )  (m3  m 4 )  0

Comparison: l3  (1)m1  ( 1)m2  ( 1)m3  (1)m 4

l̂3  (1)y1  ( 1)y 2  ( 1)y3  (1)y 4 Estimated Contrast

Test H0: l3 = 0 versus HA: l3  0

MSl3
Test Statistic: F 
MSE
Rejection Region: Reject H0 if F  F1,nT  t,a
STA 6166 - MCP 17
 2
 t 
 i i a y
 1  y   1  y    1 y  1 y 
2

MSl3   i1t 2   
1 2 3 4

 ai   12  12  12 12 

      
 i 1 n i   n1 n2 n3 n4 

2
 t 
  ai yi       
1 13  1 14  1 20     02
1 21 
2

MSl3   i1     0
 ai 
t 2
 1 2
 1
2
 1
2
1 
2
 4
        3
 i1 i 
n  3 3 3 3   


MSl3 0
F= = = 0
MSE 5
F1,8,0.05 = 5.32

Conclusion: Since F=0 < 5.32 we do not reject H0 and conclude

that the difference between average paint life between wet and
dry climates does not depend on wood type. Likewise, the
difference between average paint life for the wood types does
STA 6166 - MCP
not depend on climate type (i.e. there is no interaction). 18
 Mutual Orthogonality
Contrast a1 a2 a3 a4 l1  l2  14  14  14  14  0
l1 1 1
 12  12
2 2
l1  l3  12  12  12  12  0
l2 1
2
 1
2
1
2
 12
l3 1 1 1 1 l2  l3  12  12  12  12  0

The three are mutually orthogonal.

SSl1 = MSl1 = 147
The three mutually orthogonal contrasts
SSl2 = MSl2 = 3
add up to the Treatment Sums of
SSl3 = MSl3 = 0
Squares.
Treatment SS = 150

Total Error SS = dferror x MSE = 8 x 5 = 40

STA 6166 - MCP 19

(Type I)
Error
Rate

STA 6166 - MCP 20

 If Ho is true, and
α=0.05, we can
expect to make a
Type I error 5% of
the time…

1 out of every 20
will yield
p-value<0.05,
even though
there is no
effect!

STA 6166 - MCP 21

 Types of Error Rates
Compairsonwise Error Rate - the probability of making a Type I error
in a single test that involves the comparison of two means. (Our
usual definition of Type I error thus far…)

Question: How should we define Type I error in an experiment (test)

that involves doing several tests? What is the “overall” Type I
error?

The following definition seems sensible:

Experimentwise Error Rate - the probability of observing an

experiment in which one or more of the pairwise comparisons are
incorrectly declared significantly different. This is the probability of
making at least one Type I error.

STA 6166 - MCP 22

 Error Rates: Problems
Suppose we make c mutually orthogonal
Number of Type I Experimentwise
(independent) comparisons, each with Type I comparsons Error Rate Error Rate
comparisonwise error rate of a. The 1
2
0.05
0.05
0.050
0.098
experimentwise error rate, e, is then: 3 0.05 0.143
4 0.05 0.185
5 0.05 0.226

e  1  (1  a ) c 6
7
8
0.05
0.05
0.05
0.265
0.302
0.337
9 0.05 0.370
10 0.05 0.401
11 0.05 0.431
If the comparisons are not orthogonal, then the 12 0.05 0.460
13 0.05 0.487
experimentwise error rate is smaller. 14 0.05 0.512
15 0.05 0.537
16 0.05 0.560
Thus in most situations we actually have: 17 0.05 0.582
18 0.05 0.603

e  1  (1  a ) c 19 0.05 0.623
20 0.05 0.642

STA 6166 - MCP 23

 The Bonferroni Solution
Solution: set e=0.05 and solve for a:
But there’s a problem… a  1  (1  e) 1/ c

E.g. if c=8, we get a=0.0064!

Very conservative…, thus type II error is large.

Bonferroni’s inequality provides an approximate

e  1  (1  a )
solution to this that guarantees: c

We set:
a  e/c
E.g. if c=8, we get a=0.05/8=0.0063.
Still conservative! STA 6166 - MCP 24
 Multiple Comparison Procedures (MCPs):
Overview

Terms:
• If the MCP requires a significant overall F test, then the procedure is
called a protected method.

• Not all procedures produce the same results. (An optimal procedure
could be devised if the degree of dependence, and other factors,
among the comparisons were known…)

• The major differences among all of the different MCPs is in the

calculation of the yardstick used to determine if two means are
significantly different. The yardstick can generically be referred to as
the least significant difference. Any two means greater than this
difference are declared significantly different.

y i  y j " yardstick" " TabledValue"" SEof difference"

STA 6166 - MCP 25
 Multiple Comparison Procedures: Overview

y i  y j " yardstick" " TabledValue"" SEof difference"

• Yardsticks are composed of a standard error term and a critical

value from some tabulated statistic.

• Some procedures have “fixed” yardsticks, some have “variable”

yardsticks. The variable yardsticks will depend on how far apart
two observed means are in a rank ordered list of the mean values.

• Some procedures control Comparisonwise Error, other

Experimentwise Error, and some attempt to control both. Some are
even more specialized, e.g. Dunnett’s applies only to comparisons
of treatments to a control.
STA 6166 - MCP 26
 Fisher’s Least Significant Difference - Protected

Mean of group i (mi) is significantly different from the mean of group

j (mj) if
y i  y j  LSD

LSDij  ta 2 ,dferro r MSE 1

ni  n1j 
 ta 2 ,dferro r MSE  n2 
if all groups have
same size n.

Type I (comparisonwise) error rate = a

Controls Comparisonwise Error. Experimentwise error control
comes from requiring a significant overall F test prior to performing
any comparisons, and from applying the method only to pre-planned
comparisons.
STA 6166 - MCP 27
 Tukey’s W (Honestly Significant
Difference) Procedure
Primarily suited for all pairwise comparisons among t means.
Means are different if:
MSE
yi  y j  W W  qa (t , df error )
n
{Table 10 - critical values of the studentized range.}

Experimentwise error rate = a

This MCP controls experimentwise error rate! Comparisonwise

error rates is thus very low.

STA 6166 - MCP 28

 Student Newman Keul Procedure

A modified Tukey’s MCP. Rank the t sample means from smallest

to largest. For two means that are r “steps” apart in the ranked
list, we declare the population means different if:

y i  y j  Wr

Wr  qa (r , df error ) MSE
n

{Table 10 - critical values of the studentized range. Depends on

which mean pair is being considered!}
y [1]  min y [2] y [ 3] y [4] y [ 5] y [ 6]  max

varying
r=5 r=6 yardstick
r=2 r=3 r=4 STA 6166 - MCP 29
 Duncan’s New Multiple Range Test (Passe)

Neither an experimentwise or comparisonwise error rate control alone.

Based on a ranking of the observed means.
Introduces the concept of a “protection level” (1-a)r-1
Number of steps Protection Level Probability of
Apart, r (0.95)r-1 Falsely Rejecting H0
2 .950 .050
3 .903 .097
4 .857 .143
5 .815 .185
6 .774 .226
7 .735 .265

y i  y j  Wr Wr  qa (r , df error ) MSE

n

{Table A -11 (later) in these notes}

STA 6166 - MCP 30
 Dunnett’s Procedure

A MCP that is used for comparing treatments to a control. It

aims to control the experimentwise error rate.

Compares each treatment mean (i) to the mean for the control
group (c).

yi  yc  D D  da (k , v) MSE  n2 

dα(k,v) is obtained from Table A-11 (in the book) and is based on:
• α = the desired experimentwise error rate
• k = t-1, number of noncontrol treatments
• v = error degrees of freedom.

STA 6166 - MCP 31

 Scheffé’s S Method

For any linear contrast: l  a1m1  a2m2 at mt

Estimated by: l̂  a1y1  a2 y2    at y t
With estimated variance:
t
V (l )  MSE  ni
ˆ ˆ ai2

i 1
To test H0: l = 0 versus Ha: l 0
For a specified value of a, reject H0 if: lˆ  S
where:
S  Vˆ (lˆ) (t  1) Ft 1,dferror ,a
STA 6166 - MCP 32
Adjustment for unequal sample sizes:
The Harmonic Mean

If the sample sizes are not equal in all t groups, the

value of n in the equations for Tukey and SNK can be
replaced with the harmonic mean of the sample sizes:
t
nt  (1 / n )
i 1
i

E.g. Tukey’s W becomes: W  qa (t , df err ) MSE / n

Or can also use Tukey-Cramer
method:
MSE  1 1 
W  qa (t , df err )
*

2  ni n j 

STA 6166 - MCP 33
MCP Confidence Intervals

In some MCPs we can also

form simultaneous confidence
intervals (CI’s) for any pair of
means, μi - μj.

• Fisher’s LSD: ( yi  y j )  LSDij

• Tukey’s W: ( yi  y j )  W

• Scheffe’s for a contrast I: Iˆ  S

STA 6166 - MCP 34

A Nonparametric MCP (§9.9)

• The (parametric) MCPs just discussed all assume the

data are random samples from normal distributions with
equal variances.
• In many situations this assumption is not plausible,
e.g. incomes, proportions, survival times.
• Let τi be the shift parameter (e.g. median) for
population i, i=1,…,t. Want to determine if populations
differ with respect to their shift parameters.
• Combine all samples into one, rank obs from smallest
to largest. Denote mean of ranks for group i by:
Ri
STA 6166 - MCP 35
Nonparametric Kruskal-Wallis MCP

This MCP controls experimentwise error rate.

• Perform the Kruskall-Wallis test of equality of shift
parameters (null hypothesis).
• If this test yields an insignificant p-value, declare no
differences in the shift parameters and stop.
• If not, declare populations i and j to be different if

Ri  R j  KWij

qa (t , ) nT (1  nT )  1 1 
KWij  
12 n n 
2  i j 
STA 6166 - MCP 36
Comparisonwise error
rates for different MCP

STA 6166 - MCP 37

Experimentwise error rates
for different MCP

STA 6166 - MCP 38