Douglas M.

Bates

lme4: Mixed-effects modeling

with R
June 25, 2010

Springer

Page: 1 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

Page: 2 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10
 To Phyl and Dave, with thanks for your kind hospitality. You
see, I really was “working” when I was sitting on your deck
staring at the lake.

Page: v job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

Page: vi job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10
Preface

R is a freely available implementation of John Chambers’ award-winning S

language for computing with data. It is “Open Source” software for which
the user can, if she wishes, obtain the original source code and determine
exactly how the computations are being performed. Most users of R use the
precompiled versions that are available for recent versions of the Microsoft
Windows operating system, for Mac OS X, and for several versions of the
Linux operating system. Directions for obtaining and installing R are given
in Appendix˜??.
Because it is freely available, R is accessible to anyone who cares to learn
to use it, and thousands have done so. Recently many prominent social scien-
tists, including John Fox of McMaster University and Gary King of Harvard
University, have become enthusiastic R users and developers.
Another recent development in the analysis of social sciences data is the
recognition and modeling of multiple levels of variation in longitudinal and or-
ganizational data. The class of techniques for modeling such levels of variation
has become known as multilevel modeling or hierarchical linear modeling.
In this book I describe the use of R for examining, managing, visualizing,
and modeling multilevel data.

Madison, WI, USA, Douglas Bates

October, 2005

vii
Page: viii job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10
Contents

1 A Simple, Linear, Mixed-effects Model . . . . . . . . . . . . . . . . . . . 1

1.1 Mixed-effects Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 The Dyestuff and Dyestuff2 Data . . . . . . . . . . . . . . . . . . . . . . 3
1.2.1 The Dyestuff Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.2 The Dyestuff2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3 Fitting Linear Mixed Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3.1 A Model For the Dyestuff Data . . . . . . . . . . . . . . . . . . . 7
1.3.2 A Model For the Dyestuff2 Data . . . . . . . . . . . . . . . . . . 10
1.3.3 Further Assessment of the Fitted Models . . . . . . . . . . . . 11
1.4 The Linear Mixed-effects Probability Model . . . . . . . . . . . . . . . 12
1.4.1 Definitions and Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.4.2 Matrices and Vectors in the Fitted Model Object . . . . . 14
1.5 Assessing the Variability of the Parameter Estimates . . . . . . . . 16
1.5.1 Confidence Intervals on the Parameters . . . . . . . . . . . . . 16
1.5.2 Interpreting the Profile Zeta Plot . . . . . . . . . . . . . . . . . . . 18
1.5.3 Profile Pairs Plots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.6 Assessing the Random Effects . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
1.7 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

2 Models With Multiple Random-effects Terms . . . . . . . . . . . . . 29

2.1 A Model With Crossed Random Effects . . . . . . . . . . . . . . . . . . . 29
2.1.1 The Penicillin Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.1.2 A Model For the Penicillin Data . . . . . . . . . . . . . . . . . 32
2.2 A Model With Nested Random Effects . . . . . . . . . . . . . . . . . . . . 38
2.2.1 The Pastes Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.2.2 Fitting a Model With Nested Random Effects . . . . . . . . 42
2.2.3 Parameter Estimates for Model fm04 . . . . . . . . . . . . . . . . 43
2.2.4 Testing H0 : σ2 = 0 Versus Ha : σ2 > 0 . . . . . . . . . . . . . . . 45
2.2.5 Assessing the Reduced Model, fm04a . . . . . . . . . . . . . . . 46

ix
x Contents

2.3 A Model With Partially Crossed Random Effects . . . . . . . . . . . 47

2.3.1 The InstEval Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.3.2 Structure of L for model fm05 . . . . . . . . . . . . . . . . . . . . . 52
2.4 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

3 Models for Longitudinal Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

3.1 The sleepstudy Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.1.1 Characteristics of the sleepstudy Data Plot . . . . . . . . . 59
3.2 Mixed-effects Models For the sleepstudy Data . . . . . . . . . . . . 60
3.2.1 A Model With Correlated Random Effects . . . . . . . . . . 60
3.2.2 A Model With Uncorrelated Random Effects . . . . . . . . . 63
3.2.3 Generating Z and Λ From Random-effects Terms . . . . . 65
3.2.4 Comparing Models fm07 and fm06 . . . . . . . . . . . . . . . . . . 67
3.3 Assessing the Precision of the Parameter Estimates . . . . . . . . . 67
3.4 Examining the Random Effects and Predictions . . . . . . . . . . . . 70
3.5 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

4 Building Linear Mixed Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

4.1 Incorporating categorical covariates with fixed levels . . . . . . . . 77
4.1.1 The Machines Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
4.1.2 Comparing models with and without interactions . . . . . 79
4.2 Models for the ergoStool data . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.2.1 Random-effects for both Subject and Type . . . . . . . . . . 81
4.2.2 Fixed Effects for Type, Random for Subject . . . . . . . . . 84
4.2.3 Fixed Effects for Type and Subject . . . . . . . . . . . . . . . . . 90
4.2.4 Fixed Effects Versus Random Effects for Subject . . . . 92
4.3 Covariates Affecting Mathematics Score Gain . . . . . . . . . . . . . . 93
4.4 Rat Brain example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

5 Computational Methods for Mixed Models . . . . . . . . . . . . . . . 99

5.1 Definitions and Basic Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
5.2 The Conditional Distribution (U |Y = y) . . . . . . . . . . . . . . . . . . 101
5.3 Integrating h(u) in the Linear Mixed Model . . . . . . . . . . . . . . . . 102
5.4 Determining the PLS Solutions, ũ and βb θ . . . . . . . . . . . . . . . . . 104
5.4.1 The Fill-reducing Permutation, P . . . . . . . . . . . . . . . . . . . 105
5.4.2 The Value of the Deviance and Profiled Deviance . . . . . 106
5.4.3 Determining rθ2 and βb θ . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
5.5 The REML Criterion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
5.6 Step-by-step Evaluation of the Profiled Deviance . . . . . . . . . . . 111
5.7 Generalizing to Other Forms of Mixed Models . . . . . . . . . . . . . 113
5.7.1 Descriptions of the Model Forms . . . . . . . . . . . . . . . . . . . 113
5.7.2 Determining the Conditional Mode, ũ . . . . . . . . . . . . . . . 114
5.8 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

Page: x job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

Contents xi

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

6 Generalized Linear Mixed Models for Binary Responses . . 119

6.1 Artificial contraception use in regions of Bangladesh . . . . . . . . 119
6.1.1 Plotting the binary response . . . . . . . . . . . . . . . . . . . . . . . 120
6.1.2 Initial GLMM fit to the contraception data . . . . . . . . . . 121
6.2 Link functions and interpreting coefficients . . . . . . . . . . . . . . . . 125
6.2.1 The logit link function for binary responses . . . . . . . . . . 126

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

Page: xi job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

Page: xii job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10
List of Figures

1.1 Yield of dyestuff from 6 batches of an intermediate . . . . . . . . . . 5

1.2 Simulated data similar in structure to the Dyestuff data . . . . . . 6
1.3 Image of the Λ for model fm01ML . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.4 Image of the random-effects model matrix, ZT , for fm01 . . . . . . 15
1.5 Profile zeta plots of the parameters in model fm01ML . . . . . . . . . . 17
1.6 Absolute value profile zeta plots of the parameters in model
fm01ML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.7 Profile zeta plots comparing log(σ ), σ and σ 2 in model fm01ML 18
1.8 Profile zeta plots comparing log(σ1 ), σ1 and σ12 in model fm01ML 19
1.9 Profile pairs plot for the parameters in model fm01 . . . . . . . . . . . 20
1.10 95% prediction intervals on the random effects in fm01ML,
shown as a dotplot. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
1.11 95% prediction intervals on the random effects in fm01ML
versus quantiles of the standard normal distribution. . . . . . . . . . 24
1.12 Travel time for an ultrasonic wave test on 6 rails . . . . . . . . . . . . 27

2.1 Diameter of growth inhibition zone for 6 samples of penicillin . 31

2.2 Random effects prediction intervals for model fm03 . . . . . . . . . . . 33
2.3 Image of the random-effects model matrix for fm03 . . . . . . . . . . 34
2.4 Images of Λ , ZT Z and L for model fm03 . . . . . . . . . . . . . . . . . . . 34
2.5 Profile zeta plot of the parameters in model fm03. . . . . . . . . . . . . 35
2.6 Profile pairs plot of the parameters in model fm03. . . . . . . . . . . . 36
2.7 Profile pairs plot for model fm03 (log scale) . . . . . . . . . . . . . . . . . 37
2.8 Cross-tabulation image of the batch and sample factors . . . . . . . 39
2.9 Strength of paste preparations by batch and sample . . . . . . . . . 40
2.10 Images of Λ , ZT Z and L for model fm04 . . . . . . . . . . . . . . . . . . . 43
2.11 Random effects prediction intervals for model fm04 . . . . . . . . . . . 44
2.12 Profile zeta plots for the parameters in model fm04. . . . . . . . . . . 44
2.13 Profile zeta plots for the parameters in model fm04a. . . . . . . . . . 47
2.14 Profile pairs plot of the parameters in model fm04a. . . . . . . . . . . 48
2.15 Random effects prediction intervals for model fm05 . . . . . . . . . . . 50

xiii
xiv List of Figures

2.16 Image of the sparse Cholesky factor, L, from model fm05 . . . . . 51

3.1 Lattice plot of the sleepstudy data . . . . . . . . . . . . . . . . . . . . . . . . . 58

3.2 Images of Λ , Σ and L for model fm06 . . . . . . . . . . . . . . . . . . . . . . 62
3.3 Images of Λ , Σ and L for model fm07 . . . . . . . . . . . . . . . . . . . . . . 65
3.4 Images of ZT for models fm06 and fm07 . . . . . . . . . . . . . . . . . . . . . 65
3.5 Profile zeta plots for the parameters in model fm07 . . . . . . . . . . . 68
3.6 Profile pairs plot for the parameters in model fm07 . . . . . . . . . . . 69
3.7 Plot of the conditional modes of the random effects for
model fm07 (left panel) and the corresponding subject-specific
coefficients (right panel) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.8 Comparison of within-subject estimates and conditional
modes for fm07 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.9 Comparison of predictions from separate fits and fm07 . . . . . . . . 73
3.10 Prediction intervals on the random effects for model fm07. . . . . 75

4.1 Quality score by operator and machine . . . . . . . . . . . . . . . . . . . . . 78

4.2 Effort to arise by subject and stool type . . . . . . . . . . . . . . . . . . . . 80
4.3 Profile zeta plot for the parameters in model fm06 . . . . . . . . . . . 82
4.4 Profile pairs plot for the parameters in model fm06 . . . . . . . . . . . 83
4.5 Prediction intervals on the random effects for stool type . . . . . . 84
4.6 Profile zeta plot for the parameters in model fm06 . . . . . . . . . . . 88
4.7 Comparative dotplots of gain in the mathematics scores in
classrooms within schools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
4.8 Profile plot of the parameters in model fm4. . . . . . . . . . . . . . . . . . 96
4.9 Activation of brain regions in rats . . . . . . . . . . . . . . . . . . . . . . . . . 97

6.1 Contraception use versus centered age . . . . . . . . . . . . . . . . . . . . . 121

6.2 Contraception use versus centered age (children/no children) . . 123
6.3 Inverse of the logit link function . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

Page: xiv job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

Acronyms

AIC Akaike’s Information Criterion

AMD Approximate Minimal Degree
BIC Bayesian Information Criterion (also called “Schwatz’s Bayesian
Information Criterion”)
BLUP Best Linear Unbiased Predictor
GLMM Generalized Linear Mixed Model
LMM Linear Mixed Model
LRT Likelihood Ratio Test
ML Maximum Likelihood
MLE Maximum Likelihood Estimate or Maximum Likelihood Estimator
NLMM NonLinear Mixed Model
REML The REstricted (or “REsidual”) Maximum Likelihood estimation
criterion
PRSS Penalized, Residual Sum-of-Squares
PWRSS Penalized, Weighted, Residual Sum-of-Squares

xv
Page: xvi job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10
Chapter 1
A Simple, Linear, Mixed-effects Model

In this book we describe the theory behind a type of statistical model called
mixed-effects models and the practice of fitting and analyzing such models
using the lme4 package for R. These models are used in many different dis-
ciplines. Because the descriptions of the models can vary markedly between
disciplines, we begin by describing what mixed-effects models are and by ex-
ploring a very simple example of one type of mixed model, the linear mixed
model.
This simple example allows us to illustrate the use of the lmer function in
the lme4 package for fitting such models and for analyzing the fitted model.
We describe methods of assessing the precision of the parameter estimates
and of visualizing the conditional distribution of the random effects, given
the observed data.

1.1 Mixed-effects Models

Mixed-effects models, like many other types of statistical models, describe

a relationship between a response variable and some of the covariates that
have been measured or observed along with the response. In mixed-effects
models at least one of the covariates is a categorical covariate representing
experimental or observational “units” in the data set. In the example from
the chemical industry that is given in this chapter, the observational unit is
the batch of an intermediate product used in production of a dye. In medical
and social sciences the observational units are often the human or animal
subjects in the study. In agriculture the experimental units may be the plots
of land or the specific plants being studied.
In all of these cases the categorical covariate or covariates are observed at
a set of discrete levels. We may use numbers, such as subject identifiers, to
designate the particular levels that we observed but these numbers are simply
labels. The important characteristic of a categorical covariate is that, at each

1
2 1 A Simple, Linear, Mixed-effects Model

observed value of the response, the covariate takes on the value of one of a
set of distinct levels.
Parameters associated with the particular levels of a covariate are some-
times called the “effects” of the levels. If the set of possible levels of the
covariate is fixed and reproducible we model the covariate using fixed-effects
parameters. If the levels that we observed represent a random sample from
the set of all possible levels we incorporate random effects in the model.
There are two things to notice about this distinction between fixed-effects
parameters and random effects. First, the names are misleading because the
distinction between fixed and random is more a property of the levels of the
categorical covariate than a property of the effects associated with them. Sec-
ondly, we distinguish between “fixed-effects parameters”, which are indeed pa-
rameters in the statistical model, and “random effects”, which, strictly speak-
ing, are not parameters. As we will see shortly, random effects are unobserved
random variables.
To make the distinction more concrete, suppose that we wish to model the
annual reading test scores for students in a school district and that the co-
variates recorded with the score include a student identifier and the student’s
gender. Both of these are categorical covariates. The levels of the gender co-
variate, male and female, are fixed. If we consider data from another school
district or we incorporate scores from earlier tests, we will not change those
levels. On the other hand, the students whose scores we observed would gen-
erally be regarded as a sample from the set of all possible students whom
we could have observed. Adding more data, either from more school districts
or from results on previous or subsequent tests, will increase the number of
distinct levels of the student identifier.
Mixed-effects models or, more simply, mixed models are statistical models
that incorporate both fixed-effects parameters and random effects. Because
of the way that we will define random effects, a model with random effects
always includes at least one fixed-effects parameter. Thus, any model with
random effects is a mixed model.
We characterize the statistical model in terms of two random variables: a
q-dimensional vector of random effects represented by the random variable
B and an n-dimensional response vector represented by the random variable
Y . (We use upper-case “script” characters to denote random variables. The
corresponding lower-case upright letter denotes a particular value of the ran-
dom variable.) We observe the value, y, of Y . We do not observe the value,
b, of B.
When formulating the model we describe the unconditional distribution
of B and the conditional distribution, (Y |B = b). The descriptions of the
distributions involve the form of the distribution and the values of certain
parameters. We use the observed values of the response and the covariates to
estimate these parameters and to make inferences about them.
That’s the big picture. Now let’s make this more concrete by describing a
particular, versatile class of mixed models called linear mixed models and by

Page: 2 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

1.2 The Dyestuff and Dyestuff2 Data 3

studying a simple example of such a model. First we will describe the data
in the example.

1.2 The Dyestuff and Dyestuff2 Data

Models with random effects have been in use for a long time. The first edition
of the classic book, Statistical Methods in Research and Production, edited by
O.L. Davies, was published in 1947 and contained examples of the use of ran-
dom effects to characterize batch-to-batch variability in chemical processes.
The data from one of these examples are available as the Dyestuff data in the
lme4 package. In this section we describe and plot these data and introduce
a second example, the Dyestuff2 data, described in Box and Tiao [1973].

1.2.1 The Dyestuff Data

The Dyestuff data are described in Davies and Goldsmith [1972, Table˜6.3,
p.˜131], the fourth edition of the book mentioned above, as coming from
an investigation to find out how much the variation from batch to batch in the
quality of an intermediate product (H-acid) contributes to the variation in the
yield of the dyestuff (Naphthalene Black 12B) made from it. In the experiment
six samples of the intermediate, representing different batches of works manu-
facture, were obtained, and five preparations of the dyestuff were made in the
laboratory from each sample. The equivalent yield of each preparation as grams
of standard colour was determined by dye-trial.

To access these data within R we must first attach the lme4 package to our
session using
> library(lme4)
Note that the ">" symbol in the line shown is the prompt in R and not part
of what the user types. The lme4 package must be attached before any of the
data sets or functions in the package can be used. If typing this line results in
an error report stating that there is no package by this name then you must
first install the package.
In what follows, we will assume that the lme4 package has been installed
and that it has been attached to the R session before any of the code shown
has been run.
The str function in R provides a concise description of the structure of the
data
> str(Dyestuff)
'data.frame': 30 obs. of 2 variables:
$ Batch: Factor w/ 6 levels "A","B","C","D",..: 1 1 1 1 1 2 2 2 2 2 ...
$ Yield: num 1545 1440 1440 1520 1580 ...

Page: 3 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

4 1 A Simple, Linear, Mixed-effects Model

from which we see that it consists of 30 observations of the Yield, the response
variable, and of the covariate, Batch, which is a categorical variable stored as
a factor object. If the labels for the factor levels are arbitrary, as they are
here, we will use letters instead of numbers for the labels. That is, we label
the batches as "A" through "F" rather than "1" through "6". When the labels
are letters it is clear that the variable is categorical. When the labels are
numbers a categorical covariate can be mistaken for a numeric covariate,
with unintended consequences.
It is a good practice to apply str to any data frame the first time you
work with it and to check carefully that any categorical variables are indeed
represented as factors.
The data in a data frame are viewed as a table with columns corresponding
to variables and rows to observations. The functions head and tail print the
first or last few rows (the default value of “few” happens to be 6 but we can
specify another value if we so choose)
> head(Dyestuff)

Batch Yield
1 A 1545
2 A 1440
3 A 1440
4 A 1520
5 A 1580
6 B 1540

or we could ask for a summary of the data

> summary(Dyestuff)

Batch Yield
A:5 Min. :1440
B:5 1st Qu.:1469
C:5 Median :1530
D:5 Mean :1528
E:5 3rd Qu.:1575
F:5 Max. :1635

Although the summary does show us an important property of the data,

namely that there are exactly 5 observations on each batch — a property
that we will describe by saying that the data are balanced with respect to
Batch — we usually learn much more about the structure of such data from
plots like Fig.˜1.1 than we can from numerical summaries.
In Fig.˜1.1 we can see that there is considerable variability in yield, even
for preparations from the same batch, but there is also noticeable batch-to-
batch variability. For example, four of the five preparations from batch F
provided lower yields than did any of the preparations from batches C and
E.
This plot, and essentially all the other plots in this book, were created
using Deepayan Sarkar’s lattice package for R. In Sarkar [2008] he describes

Page: 4 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

1.2 The Dyestuff and Dyestuff2 Data 5

E ● ● ● ● ●

C ● ● ● ●
●

B ● ● ● ●
Batch

A ●
● ● ● ●

D ● ● ● ● ●

F ● ● ●
● ●

1450 1500 1550 1600

Yield of dyestuff (grams of standard color)

Fig. 1.1 Yield of dyestuff (Napthalene Black 12B) for 5 preparations from each of 6
batches of an intermediate product (H-acid). The line joins the mean yields from the
batches, which have been ordered by increasing mean yield. The vertical positions
are “jittered” slightly to avoid over-plotting. Notice that the lowest yield for batch A
was observed for two distinct preparations from that batch.

how one would create such a plot. Because this book was created using Sweave
[Leisch, 2002], the exact code used to create the plot, as well as the code for
all the other figures and calculations in the book, is available on the web site
for the book. In Sect.˜?? we review some of the principles of lattice graphics,
such as reordering the levels of the Batch factor by increasing mean response,
that enhance the informativeness of the plot. At this point we will concentrate
on the information conveyed by the plot and not on how the plot is created.
In Sect.˜1.3.1 we will use mixed models to quantify the variability in yield
between batches. For the time being let us just note that the particular
batches used in this experiment are a selection or sample from the set of
all batches that we wish to consider. Furthermore, the extent to which one
particular batch tends to increase or decrease the mean yield of the process
— in other words, the “effect” of that particular batch on the yield — is not
as interesting to us as is the extent of the variability between batches. For
the purposes of designing, monitoring and controlling a process we want to
predict the yield from future batches, taking into account the batch-to-batch
variability and the within-batch variability. Being able to estimate the extent
to which a particular batch in the past increased or decreased the yield is not
usually an important goal for us. We will model the effects of the batches as
random effects rather than as fixed-effects parameters.

Page: 5 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

6 1 A Simple, Linear, Mixed-effects Model

C ● ● ● ●
●

● ● ● ●
A ●

● ●
E
Batch

● ● ●

D ● ● ●
● ●

B ● ● ●
● ●

F ● ●
● ● ●

0 5 10

Simulated response (dimensionless)

Fig. 1.2 Simulated data presented in Box and Tiao [1973] with a structure similar
to that of the Dyestuff data. These data represent a case where the batch-to-batch
variability is small relative to the within-batch variability.

1.2.2 The Dyestuff2 Data

The Dyestuff2 data are simulated data presented in Box and Tiao [1973,
Table 5.1.4, p. 247] where the authors state
These data had to be constructed for although examples of this sort undoubt-
edly occur in practice they seem to be rarely published.

The structure and summary

> str(Dyestuff2)

'data.frame': 30 obs. of 2 variables:

$ Batch: Factor w/ 6 levels "A","B","C","D",..: 1 1 1 1 1 2 2 2 2 2 ...
$ Yield: num 7.3 3.85 2.43 9.57 7.99 ...

> summary(Dyestuff2)

Batch Yield
A:5 Min. :-0.892
B:5 1st Qu.: 2.765
C:5 Median : 5.365
D:5 Mean : 5.666
E:5 3rd Qu.: 8.151
F:5 Max. :13.434

are intentionally similar to those of the Dyestuff data. As can be seen in

Fig.˜1.2 the batch-to-batch variability in these data is small compared to the
within-batch variability. In some approaches to mixed models it can be dif-
ficult to fit models to such data. Paradoxically, small “variance components”
can be more difficult to estimate than large variance components.
The methods we will present are not compromised when estimating small
variance components.

Page: 6 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

1.3 Fitting Linear Mixed Models 7

1.3 Fitting Linear Mixed Models

Before we formally define a linear mixed model, let’s go ahead and fit models
to these data sets using lmer. Like most model-fitting functions in R, lmer
takes, as its first two arguments, a formula specifying the model and the data
with which to evaluate the formula. This second argument, data, is optional
but recommended. It is usually the name of a data frame, such as those we
examined in the last section. Throughout this book all model specifications
will be given in this formula/data format.
We will explain the structure of the formula after we have considered an
example.

1.3.1 A Model For the Dyestuff Data

We fit a model to the Dyestuff data allowing for an overall level of the Yield
and for an additive random effect for each level of Batch
> fm01 <- lmer(Yield ~ 1 + (1|Batch), Dyestuff)
> print(fm01)

Linear mixed model fit by REML ['merMod']

Formula: Yield ~ 1 + (1 | Batch)
Data: Dyestuff
REML criterion at convergence: 319.6543
Random effects:
Groups Name Variance Std.Dev.
Batch (Intercept) 1764 42.00
Residual 2451 49.51
Number of obs: 30, groups: Batch, 6

Fixed effects:
Estimate Std. Error t value
(Intercept) 1527.50 19.38 78.8

In the first line we call the lmer function to fit a model with formula
Yield ~ 1 + (1 | Batch)

applied to the Dyestuff data and assign the result to the name fm01. (The
name is arbitrary. I happen to use names that start with fm, indicating “fitted
model”.)
As is customary in R, there is no output shown after this assignment. We
have simply saved the fitted model as an object named fm01. In the second
line we display some information about the fitted model by applying print
to fm01. In later examples we will condense these two steps into one but here
it helps to emphasize that we save the result of fitting a model then apply
various extractor functions to the fitted model to get a brief summary of the
model fit or to obtain the values of some of the estimated quantities.

Page: 7 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

8 1 A Simple, Linear, Mixed-effects Model

1.3.1.1 Details of the Printed Display

The printed display of a model fit with lmer has four major sections: a de-
scription of the model that was fit, some statistics characterizing the model
fit, a summary of properties of the random effects and a summary of the
fixed-effects parameter estimates. We consider each of these sections in turn.
The description section states that this is a linear mixed model in which the
parameters have been estimated as those that minimize the REML criterion
(explained in Sect.˜5.5). The formula and data arguments are displayed for
later reference. If other, optional arguments affecting the fit, such as a subset
specification, were used, they too will be displayed here.
For models fit by the REML criterion the only statistic describing the
model fit is the value of the REML criterion itself. An alternative set of pa-
rameter estimates, the maximum likelihood estimates, are obtained by spec-
ifying the optional argument REML=FALSE.
> (fm01ML <- lmer(Yield ~ 1 + (1|Batch), Dyestuff, REML=FALSE))

Linear mixed model fit by maximum likelihood ['merMod']

Formula: Yield ~ 1 + (1 | Batch)
Data: Dyestuff
AIC BIC logLik deviance
333.3271 337.5307 -163.6635 327.3271
Random effects:
Groups Name Variance Std.Dev.
Batch (Intercept) 1388 37.26
Residual 2451 49.51
Number of obs: 30, groups: Batch, 6

Fixed effects:
Estimate Std. Error t value
(Intercept) 1527.50 17.69 86.33

(Notice that this code fragment also illustrates a way to condense the assign-
ment and the display of the fitted model into a single step. The redundant set
of parentheses surrounding the assignment causes the result of the assignment
to be displayed. We will use this device often in what follows.)
The display of a model fit by maximum likelihood provides several other
model-fit statistics such as Akaike’s Information Criterion (AIC)˜[Sakamoto
et˜al., 1986], Schwarz’s Bayesian Information Criterion (BIC)˜[Schwarz, 1978],
the log-likelihood (logLik) at the parameter estimates, and the deviance (neg-
ative twice the log-likelihood) at the parameter estimates. These are all statis-
tics related to the model fit and are used to compare different models fit to
the same data.
At this point the important thing to note is that the default estimation
criterion is the REML criterion. Generally the REML estimates of variance
components are preferred to the ML estimates. However, when comparing
models it is safest to refit all the models using the maximum likelihood cri-
terion. We will discuss comparisons of model fits in Sect.˜2.2.4.

Page: 8 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

1.3 Fitting Linear Mixed Models 9

The third section is the table of estimates of parameters associated with

the random effects. There are two sources of variability in the model we have
fit, a batch-to-batch variability in the level of the response and the residual
or per-observation variability — also called the within-batch variability. The
name “residual” is used in statistical modeling to denote the part of the
variability that cannot be explained or modeled with the other terms. It is
the variation in the observed data that is “left over” after we have determined
the estimates of the parameters in the other parts of the model.
Some of the variability in the response is associated with the fixed-effects
terms. In this model there is only one such term, labeled as the (Intercept).
The name “intercept”, which is better suited to models based on straight
lines written in a slope/intercept form, should be understood to represent
an overall “typical” or mean level of the response in this case. (In case you
are wondering about the parentheses around the name, they are included so
that you can’t accidentally create a variable with a name that conflicts with
this name.) The line labeled Batch in the random effects table shows that
the random effects added to the (Intercept) term, one for each level of the
Batch factor, are modeled as random variables whose unconditional variance
is estimated as 1764.05 g2 in the REML fit and as 1388.33 g2 in the ML
fit. The corresponding standard deviations are 42.00 g for the REML fit and
37.26 g for the ML fit.
Note that the last column in the random effects summary table is the
estimate of the variability expressed as a standard deviation rather than as a
variance. These are provided because it is usually easier to visualize standard
deviations, which are on the scale of the response, than it is to visualize
the magnitude of a variance. The values in this column are a simple re-
expression (the square root) of the estimated variances. Do not confuse them
with the standard errors of the variance estimators, which are not given here.
In Sect.˜1.5 we explain why we do not provide standard errors of variance
estimates.
The line labeled Residual in this table gives the estimate of the variance of
the residuals (also in g2 ) and its corresponding standard deviation. For the
REML fit the estimated standard deviation of the residuals is 49.51 g and
for the ML fit it is also 49.51 g. (Generally these estimates do not need to
be equal. They happen to be equal in this case because of the simple model
form and the balanced data set.)
The last line in the random effects table states the number of observations
to which the model was fit and the number of levels of any “grouping factors”
for the random effects. In this case we have a single random effects term,
(1|Batch), in the model formula and the grouping factor for that term is
Batch. There will be a total of six random effects, one for each level of Batch.
The final part of the printed display gives the estimates and standard errors
of any fixed-effects parameters in the model. The only fixed-effects term in
the model formula is the 1, denoting a constant which, as explained above, is
labeled as (Intercept). For both the REML and the ML estimation criterion

Page: 9 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

10 1 A Simple, Linear, Mixed-effects Model

the estimate of this parameter is 1527.5 g (equality is again a consequence of

the simple model and balanced data set). The standard error of the intercept
estimate is 19.38 g for the REML fit and 17.69 g for the ML fit.

1.3.2 A Model For the Dyestuff2 Data

Fitting a similar model to the Dyestuff2 data produces an estimate σ

b1 = 0 in
both the REML
> (fm02 <- lmer(Yield ~ 1 + (1|Batch), Dyestuff2))

Linear mixed model fit by REML ['merMod']

Formula: Yield ~ 1 + (1 | Batch)
Data: Dyestuff2
REML criterion at convergence: 161.8283
Random effects:
Groups Name Variance Std.Dev.
Batch (Intercept) 0.00 0.000
Residual 13.81 3.716
Number of obs: 30, groups: Batch, 6

Fixed effects:
Estimate Std. Error t value
(Intercept) 5.6656 0.6784 8.352

and the ML fits.

> (fm02ML <- update(fm02, REML=FALSE))

Linear mixed model fit by maximum likelihood ['merMod']

Formula: Yield ~ 1 + (1 | Batch)
Data: Dyestuff2
AIC BIC logLik deviance
168.8730 173.0766 -81.4365 162.8730
Random effects:
Groups Name Variance Std.Dev.
Batch (Intercept) 0.00 0.000
Residual 13.35 3.653
Number of obs: 30, groups: Batch, 6

Fixed effects:
Estimate Std. Error t value
(Intercept) 5.666 0.667 8.494

(Note the use of the update function to re-fit a model changing some of the
arguments. In a case like this, where the call to fit the original model is not
very complicated, the use of update is not that much simpler than repeating
the original call to lmer with extra arguments. For complicated model fits it
can be.)
An estimate of 0 for σ1 does not mean that there is no variation between the
groups. Indeed Fig.˜1.2 shows that there is some small amount of variability

Page: 10 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

1.3 Fitting Linear Mixed Models 11

between the groups. The estimate, σ b1 = 0, simply indicates that the level of
“between-group” variability is not sufficient to warrant incorporating random
effects in the model.
The important point to take away from this example is that we must
allow for the estimates of variance components to be zero. We describe such
a model as being degenerate, in the sense that it corresponds to a linear
model in which we have removed the random effects associated with Batch.
Degenerate models can and do occur in practice. Even when the final fitted
model is not degenerate, we must allow for such models when determining
the parameter estimates through numerical optimization.
To reiterate, the model fm02 corresponds to the linear model
> summary(fm02a <- lm(Yield ~ 1, Dyestuff2))

Call:
lm(formula = Yield ~ 1, data = Dyestuff2)
Residuals:
Min 1Q Median 3Q Max
-6.5576 -2.9006 -0.3006 2.4854 7.7684

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 5.6656 0.6784 8.352 3.32e-09

Residual standard error: 3.716 on 29 degrees of freedom

because the random effects are inert, in the sense that they have a variance
of zero, and hence can be removed.
Notice that the estimate of σ from the linear model (called the Residual
standard error in the output) corresponds to the estimate in the REML fit
(fm02) but not that from the ML fit (fm02ML). The fact that the REML es-
timates of variance components in mixed models generalize the estimate of
the variance used in linear models, in the sense that these estimates coincide
in the degenerate case, is part of the motivation for the use of the REML
criterion for fitting mixed-effects models.

1.3.3 Further Assessment of the Fitted Models

The parameter estimates in a statistical model represent our “best guess” at

the unknown values of the model parameters and, as such, are important
results in statistical modeling. However, they are not the whole story. Statis-
tical models characterize the variability in the data and we must assess the
effect of this variability on the parameter estimates and on the precision of
predictions made from the model.
In Sect.˜1.5 we introduce a method of assessing variability in parameter
estimates using the “profiled deviance” and in Sect.˜1.6 we show methods of

Page: 11 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

12 1 A Simple, Linear, Mixed-effects Model

characterizing the conditional distribution of the random effects given the

data. Before we get to these sections, however, we should state in some detail
the probability model for linear mixed-effects and establish some definitions
and notation. In particular, before we can discuss profiling the deviance, we
should define the deviance. We do that in the next section.

1.4 The Linear Mixed-effects Probability Model

In explaining some of parameter estimates related to the random effects we

have used terms such as “unconditional distribution” from the theory of prob-
ability. Before proceeding further we should clarify the linear mixed-effects
probability model and define several terms and concepts that will be used
throughout the book.

1.4.1 Definitions and Results

In this section we provide some definitions and formulas without derivation

and with minimal explanation, so that we can use these terms in what fol-
lows. In Chap.˜5 we revisit these definitions providing derivations and more
explanation.
As mentioned in Sect.˜1.1, a mixed model incorporates two random
variables: B, the q-dimensional vector of random effects, and Y , the n-
dimensional response vector. In a linear mixed model the unconditional dis-
tribution of B and the conditional distribution, (Y |B = b), are both multi-
variate Gaussian (or “normal”) distributions,

(Y |B = b) ∼ N (Xβ + Zb, σ 2 I)
(1.1)
B ∼ N (0, Σθ ).

The conditional mean of Y , given B = b, is the linear predictor, Xβ + Zb,

which depends on the p-dimensional fixed-effects parameter, β , and on b.
The model matrices, X and Z, of dimension n × p and n × q, respectively,
are determined from the formula for the model and the values of covariates.
Although the matrix Z can be large (i.e. both n and q can be large), it is
sparse (i.e. most of the elements in the matrix are zero).
The relative covariance factor, Λθ , is a q × q matrix, depending on the
variance-component parameter, θ , and generating the symmetric q×q variance-
covariance matrix, Σθ , according to

Σθ = σ 2Λθ ΛθT . (1.2)

Page: 12 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

1.4 The Linear Mixed-effects Probability Model 13

The spherical random effects, U ∼ N (0, σ 2 Iq ), determine B according to

B = Λθ U .

The penalized residual sum of squares (PRSS),

r2 (θ , β , u) = ky − Xβ − ZΛθ uk2 + kuk2 , (1.3)

is the sum of the residual sum of squares, measuring fidelity of the model to
the data, and a penalty on the size of u, measuring the complexity of the
model. Minimizing r2 with respect to u,

rβ2 ,θ = min ky − Xβ − ZΛθ uk2 + kuk2 (1.4)
u

is a direct (i.e. non-iterative) computation for which we calculate the sparse

Cholesky factor, Lθ , which is a lower triangular q × q matrix satisfying

θ = Λθ Z ZΛθ + Iq .
Lθ LT T T
(1.5)

where Iq is the q × q identity matrix.

The deviance (negative twice the log-likelihood) of the parameters, given
the data, y, is

rβ2 ,θ
d(θ , β , σ |y) = n log(2πσ 2 ) + log(|Lθ |2 ) + . (1.6)
σ2
where |Lθ | denotes the determinant of Lθ . Because Lθ is triangular, its de-
terminant is the product of its diagonal elements.
Because the conditional mean, µ, is a linear function of β and u, mini-
mization of the PRSS with respect to both β and u to produce

rθ2 = min ky − Xβ − ZΛθ uk2 + kuk2 (1.7)
β ,u

is also a direct calculation. The values of u and β that provide this minimum
are called, respectively, the conditional mode, ũθ , of the spherical random
effects and the conditional estimate, βb θ , of the fixed effects. At the conditional
estimate of the fixed effects the deviance is
r2
d(θ , βbθ , σ |y) = n log(2πσ 2 ) + log(|Lθ |2 ) + θ2 . (1.8)
σ
Minimizing this expression with respect to σ 2 produces the conditional esti-
mate
2
c2 = rθ
σ (1.9)
θ
n
which provides the profiled deviance,

Page: 13 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

14 1 A Simple, Linear, Mixed-effects Model
  
˜ b 2 2πrθ2
d(θ |y) = d(θ , βθ , σ
bθ |y) = log(|Lθ | ) + n 1 + log , (1.10)
n

a function of θ alone.
The maximum likelihood estimate (MLE) of θ , written θb , is the value that
minimizes the profiled deviance˜(1.10). We determine this value by numerical
˜ θb |y) we determine βb = βb b , ũ b
optimization. In the process of evaluating d(
q θ θ
and r2b , from which we can evaluate σ
b = r2b /n.
θ θ
The elements of the conditional mode of B, evaluated at the parameter
estimates,
b̃θb = Λθb ũθb (1.11)
are sometimes called the best linear unbiased predictors or BLUPs of the
random effects. Although it has an appealing acronym, I don’t find the term
particularly instructive (what is a “linear unbiased predictor” and in what
sense are these the “best”?) and prefer the term “conditional mode”, which is
explained in Sect.˜1.6.

1.4.2 Matrices and Vectors in the Fitted Model Object

The optional argument, verbose=1, in a call to lmer produces output showing

˜ |y).
the progress of the iterative optimization of d(θ
> fm01ML <- lmer(Yield ~ 1|Batch, Dyestuff, REML=FALSE, verbose=1)

npt = 3 , n = 1
rhobeg = 0.2 , rhoend = 2e-07
0.020: 4: 327.347;0.800000
0.0020: 6: 327.328;0.764659
0.00020: 9: 327.327;0.752808
2.0e-05: 10: 327.327;0.752583
2.0e-06: 12: 327.327;0.752583
2.0e-07: 14: 327.327;0.752581
At return
17: 327.32706: 0.752581

The algorithm converges after 17 function evaluations to a profiled deviance

of 327.32706 at θ =0.752581.
The actual values of many of the matrices and vectors defined above are
available as slots of the fitted model object. In general the object returned
by a model-fitting function from the lme4 package has three upper-level slots:
re, consisting of information related to the random effects, fe, consisting of
information related to the fixed effects, and resp, consisting of information
related to the response.
For example, Λθb is stored as
> fm01ML@re@Lambda

Page: 14 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

1.4 The Linear Mixed-effects Probability Model 15

Fig. 1.3 Image of the

relative covariance fac-
tor, Λθb for model fm01ML. 1
The non-zero elements 2
are shown as darkened 3
squares. The zero ele- 4
ments are blank. 5
6

1 2 3 4 5 6

1
2
Row

3
4
5
6

5 10 15 20 25

Column

Fig. 1.4 Image of the transpose of the random-effects model matrix, Z, for model
fm01. The non-zero elements, which are all unity, are shown as darkened squares. The
zero elements are blank.

6 x 6 sparse Matrix of class "dgCMatrix"

[1,] 0.7525807 . . . . .
[2,] . 0.7525807 . . . .
[3,] . . 0.7525807 . . .
[4,] . . . 0.7525807 . .
[5,] . . . . 0.7525807 .
[6,] . . . . . 0.7525807

Often we will show the structure of sparse matrices as an image (Fig.˜1.3).

Especially for large sparse matrices, the image conveys the structure more
compactly than does the printed representation.
In this simple model Λ = θ I6 is a multiple of the identity matrix and
the 30 × 6 model matrix Z, whose transpose is shown in Fig.˜1.4, consists of
the indicator columns for Batch. Because the data are balanced with respect
to Batch, the Cholesky factor, L is also a multiple of the identity (use im-
age(fm01ML@re@L) to check if you wish). The vector u is available in fm01ML@re.
The vector β and the model matrix X are in fm01ML@fe.

Page: 15 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

16 1 A Simple, Linear, Mixed-effects Model

1.5 Assessing the Variability of the Parameter Estimates

In this section we show how to create a profile deviance object from a fitted
linear mixed model and how to use this object to evaluate confidence intervals
on the parameters. We also discuss the construction and interpretation of
profile zeta plots for the parameters and profile pairs plots for parameter
pairs.

1.5.1 Confidence Intervals on the Parameters

The mixed-effects model fit as fm01 or fm01ML has three parameters for which
we obtained estimates. These parameters are σ1 , the standard deviation of the
random effects, σ , the standard deviation of the residual or “per-observation”
noise term and β0 , the fixed-effects parameter that is labeled as (Intercept).
The profile function systematically varies the parameters in a model, as-
sessing the best possible fit that can be obtained with one parameter fixed
at a specific value and comparing this fit to the globally optimal fit, which is
the original model fit that allowed all the parameters to vary. The models are
compared according to the change in the deviance, which is the likelihood ra-
tio test (LRT) statistic. We apply a signed square root transformation to this
statistic and plot the resulting function, called ζ , versus the parameter value.
A ζ value can be compared to the quantiles of the standard normal distribu-
tion, Z ∼ N (0, 1). For example, a 95% profile deviance confidence interval
on the parameter consists of the values for which −1.960 < ζ < 1.960.
Because the process of profiling a fitted model, which involves re-fitting
the model many times, can be computationally intensive, one should exercise
caution with complex models fit to very large data sets. Because the statistic
of interest is a likelihood ratio, the model is re-fit according to the maximum
likelihood criterion, even if the original fit is a REML fit. Thus, there is a
slight advantage in starting with an ML fit.
> pr01 <- profile(fm01ML)

Plots of ζ versus the parameter being profiled (Fig.˜1.5) are obtained with
> xyplot(pr01, aspect = 1.3)

We will refer to such plots as profile zeta plots. I usually adjust the aspect
ratio of the panels in profile zeta plots to, say, aspect = 1.3 and frequently
set the layout so the panels form a single row (layout = c(3,1), in this case).
The vertical lines in the panels delimit the 50%, 80%, 90%, 95% and 99%
confidence intervals, when these intervals can be calculated. Numerical values
of the endpoints are returned by the confint extractor.
> confint(pr01)

Page: 16 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

1.5 Assessing the Variability of the Parameter Estimates 17

.sig01 .lsig (Intercept)

ζ 0

−1

−2

0 20 40 60 80 100 3.6 3.8 4.0 4.2 1500 1550

Fig. 1.5 Signed square root, ζ , of the likelihood ratio test statistic for each of the
parameters in model fm01ML. The vertical lines are the endpoints of 50%, 80%, 90%,
95% and 99% confidence intervals derived from this test statistic.

2.5
2.0

(Intercept)
.sig01

1.5
.lsig
|ζ|

1.0
0.5
0.0

0 20 40 60 80 100 3.6 3.8 4.0 4.2 1500 1550

Fig. 1.6 Profiled deviance, on the scale |ζ |, the square root of the change in the
deviance, for each of the parameters in model fm01ML. The intervals shown are 50%,
80%, 90%, 95% and 99% confidence intervals based on the profile likelihood.

2.5 % 97.5 %
.sig01 12.197461 84.063361
.lsig 3.643624 4.214461
(Intercept) 1486.451506 1568.548494

By default the 95% confidence interval is returned. The optional argument,

level, is used to obtain other confidence levels.
> confint(pr01, level = 0.99)

0.5 % 99.5 %
.sig01 NA 113.690280
.lsig 3.571290 4.326337
(Intercept) 1465.872875 1589.127125

Notice that the lower bound on the 99% confidence interval for σ1 is not
defined. Also notice that we profile log(σ ) instead of σ , the residual standard
deviation.

Page: 17 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

18 1 A Simple, Linear, Mixed-effects Model

log(σ) σ σ2

ζ 0

−1

−2

3.6 3.8 4.0 4.2 40 50 60 70 2000 3000 4000 5000

Fig. 1.7 Signed square root, ζ , of the likelihood ratio test statistic as a function of
log(σ ), of σ and of σ 2 . The vertical lines are the endpoints of 50%, 80%, 90%, 95%
and 99% confidence intervals.

A plot of |ζ |, the absolute value of ζ , versus the parameter (Fig.˜1.6),

obtained by adding the optional argument absVal = TRUE to the call to xyplot,
can be more effective for visualizing the confidence intervals.

1.5.2 Interpreting the Profile Zeta Plot

A profile zeta plot, such as Fig.˜1.5, shows us the sensitivity of the model fit
to changes in the value of particular parameters. Although this is not quite
the same as describing the distribution of an estimator, it is a similar idea
and we will use some of the terminology from distributions when describing
these plots. Essentially we view the patterns in the plots as we would those
in a normal probability plot of data values or residuals from a model.
Ideally the profile zeta plot will be close to a straight line over the region
of interest, in which case we can perform reliable statistical inference based
on the parameter’s estimate, its standard error and quantiles of the stan-
dard normal distribution. We will describe such a situation as providing a
good normal approximation for inference. The common practice of quoting
a parameter estimate and its standard error assumes that this is always the
case.
In Fig.˜1.5 the profile zeta plot for log(σ ) is reasonably straight so log(σ )
has a good normal approximation. But this does not mean that there is a
good normal approximation for σ 2 or even for σ . As shown in Fig.˜1.7 the
profile zeta plot for log(σ ) is slightly skewed, that for σ is moderately skewed
and the profile zeta plot for σ 2 is highly skewed. Deviance-based confidence
intervals on σ 2 are quite asymmetric, of the form “estimate minus a little,
plus a lot”.

Page: 18 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

1.5 Assessing the Variability of the Parameter Estimates 19

log(σ1) σ1 σ21

ζ 0

−1

−2

2 3 4 0 20 40 60 80 100 0 5000 10000

Fig. 1.8 Signed square root, ζ , of the likelihood ratio test statistic as a function of
log(σ1 ), of σ1 and of σ12 . The vertical lines are the endpoints of 50%, 80%, 90%, 95%
and 99% confidence intervals.

This should not come as a surprise to anyone who learned in an intro-

ductory statistics course that, given a random sample of data assumed to
come from a Gaussian distribution, we use a χ 2 distribution, which can be
quite skewed, to form a confidence interval on σ 2 . Yet somehow there is a
widespread belief that the distribution of variance estimators in much more
complex situations should be well approximated by a normal distribution.
It is nonsensical to believe that. In most cases summarizing the precision of
a variance component estimate by giving an approximate standard error is
woefully inadequate.
The pattern in the profile plot for β0 is sigmoidal (i.e. an elongated “S”-
shape). The pattern is symmetric about the estimate but curved in such a
way that the profile-based confidence intervals are wider than those based
on a normal approximation. We characterize this pattern as symmetric but
over-dispersed (relative to a normal distribution). Again, this pattern is not
unexpected. Estimators of the coefficients in a linear model without random
effects have a distribution which is a scaled Student’s T distribution. That
is, they follow a symmetric distribution that is over-dispersed relative to the
normal.
The pattern in the profile zeta plot for σ1 is more complex. Fig.˜1.8 shows
the profile zeta plot on the scale of log(σ1 ), σ1 and σ12 . Notice that the profile
zeta plot for log(σ1 ) is very close to linear to the right of the estimate but
flattens out on the left. That is, σ1 behaves like σ in that its profile zeta
plot is more-or-less a straight line on the logarithmic scale, except when σ1
is close to zero. The model loses sensitivity to values of σ1 that are close to
zero. If, as in this case, zero is within the “region of interest” then we should
expect that the profile zeta plot will flatten out on the left hand side.

Page: 19 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

20 1 A Simple, Linear, Mixed-effects Model

1600

1550

(Intercept)

1500

1450 0 1 2 3

4.4 4.0 4.2 4.4

4.2

4.0
.lsig

3.8

3.6
0 1 2 3

0 50 100 150

.sig01 0

−1

−2

−3
Scatter Plot Matrix

Fig. 1.9 Profile pairs plot for the parameters in model fm01. The contour lines
correspond to two-dimensional 50%, 80%, 90%, 95% and 99% marginal confidence
regions based on the likelihood ratio. Panels below the diagonal represent the (ζi , ζ j )
parameters; those above the diagonal represent the original parameters.

1.5.3 Profile Pairs Plots

A profiled deviance object, such as pr01, not only provides information on

the sensitivity of the model fit to changes in parameters, it also tells us how
the parameters influence each other. When we re-fit the model subject to a
constraint such as, say, σ1 = 60, we obtain the conditional estimates for the
other parameters — σ and β0 in this case. The conditional estimate of, say,
σ as a function of σ1 is called the profile trace of σ on σ1 . Plotting such
traces provides valuable information on how the parameters in the model are
influenced by each other.
The profile pairs plot, obtained as

Page: 20 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

1.5 Assessing the Variability of the Parameter Estimates 21

> splom(pr01)

and shown in Fig.˜1.9 shows the profile traces along with interpolated con-
tours of the two-dimensional profiled deviance function. The contours are
chosen to correspond to the two-dimensional marginal confidence regions at
particular confidence levels.
Because this plot may be rather confusing at first we will explain what is
shown in each panel. To make it easier to refer to panels we assign them (x, y)
coordinates, as in a Cartesian coordinate system. The columns are numbered
1 to 3 from left to right and the rows are numbered 1 to 3 from bottom to
top. Note that the rows are numbered from the bottom to the top, like the
y-axis of a graph, not from top to bottom, like a matrix.
The diagonal panels show the ordering of the parameters: σ1 first, then
log(σ ) then β0 . Panels above the diagonal are in the original scale of the
parameters. That is, the top-left panel, which is the (1, 3) position, has σ1 on
the horizontal axis and β0 on the vertical axis.
In addition to the contour lines in this panel, there are two other lines,
which are the profile traces of σ1 on β0 and of β0 on σ1 . The profile trace of β0
on σ1 is a straight horizontal line, indicating that the conditional estimate of
β0 , given a value of σ1 , is constant. Again, this is a consequence of the simple
model form and the balanced data set. The other line in this panel, which is
the profile trace of σ1 on β0 , is curved. That is, the conditional estimate of
σ1 given β0 depends on β0 . As β0 moves away from the estimate, βb0 , in either
direction, the conditional estimate of σ1 increases.
We will refer to the two traces on a panel as the “horizontal trace” and
“vertical trace”. They are not always perfectly horizontal and vertical lines
but the meaning should be clear from the panel because one trace will always
be more horizontal and the other will be more vertical. The one that is more
horizontal is the trace of the parameter on the y axis as a function of the
parameter on the horizontal axis, and vice versa.
The contours shown on the panel are interpolated from the profile zeta
function and the profile traces, in the manner described in Bates and Watts
[1988, Chapter 6]. One characteristic of a profile trace, which we can verify
visually in this panel, is that the tangent to a contour must be vertical where
it intersects the horizontal trace and horizontal where it intersects the vertical
trace.
The (2, 3) panel shows β0 versus log(σ ). In this case the traces actually
are horizontal and vertical straight lines. That is, the conditional estimate of
β0 doesn’t depend on log(σ ) and the conditional estimate of log(σ ) doesn’t
depend on β0 . Even in this case, however, the contour lines are not concentric
ellipses, because the deviance is not perfectly quadratic in these parameters.
That is, the zeta functions, ζ (β0 ) and ζ (log(σ )), are not linear.
The (1, 2) panel, showing log(σ ) versus σ1 shows distortion along both
axes and nonlinear patterns in both traces. When σ1 is close to zero the
conditional estimate of log(σ ) is larger than when σ1 is large. In other words

Page: 21 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

22 1 A Simple, Linear, Mixed-effects Model

small values of σ1 inflate the estimate of log(σ ) because the variability that
would be explained by the random effects gets incorporated into the residual
noise term.
Panels below the diagonal are on the ζ scale, which is why the axes on
each of these panels span the same range, approximately −3 to +3, and the
profile traces always cross at the origin. Thus the (3, 1) panel shows ζ (σ1 )
on the vertical axis versus ζ (β0 ) on the horizontal. These panels allow us
to see distortions from an elliptical shape due to nonlinearity of the traces,
separately from the one-dimensional distortions caused by a poor choice of
scale for the parameter. The ζ scales provide, in some sense, the best possible
set of single-parameter transformations for assessing the contours. On the ζ
scales the extent of a contour on the horizontal axis is exactly the same as
the extent on the vertical axis and both are centered about zero.
Another way to think of this is that, if we would have profiled σ12 instead
of σ1 , we would change all the panels in the first column but the panels on
the first row would remain the same.

1.6 Assessing the Random Effects

In Sect.˜1.4.1 we mentioned that what are sometimes called the BLUPs (or
best linear unbiased predictors) of the random effects, B, are the conditional
modes evaluated at the parameter estimates, calculated as b̃θb = Λθb ũθb .
These values are often considered as some sort of “estimates” of the ran-
dom effects. It can be helpful to think of them this way but it can also be
misleading. As we have stated, the random effects are not, strictly speak-
ing, parameters—they are unobserved random variables. We don’t estimate
the random effects in the same sense that we estimate parameters. In-
stead, we consider the conditional distribution of B given the observed data,
(B|Y = y).
Because the unconditional distribution, B ∼ N (0, Σθ ) is continuous, the
conditional distribution, (B|Y = y) will also be continuous. In general, the
mode of a probability density is the point of maximum density, so the phrase
“conditional mode” refers to the point at which this conditional density is
maximized. Because this definition relates to the probability model, the values
of the parameters are assumed to be known. In practice, of course, we don’t
know the values of the parameters (if we did there would be no purpose
in forming the parameter estimates), so we use the estimated values of the
parameters to evaluate the conditional modes.
Those who are familiar with the multivariate Gaussian distribution may
recognize that, because both B and (Y |B = b) are multivariate Gaussian,
(B|Y = y) will also be multivariate Gaussian and the conditional mode will
also be the conditional mean of B, given Y = y. This is the case for a linear
mixed model but it does not carry over to other forms of mixed models. In the

Page: 22 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

1.6 Assessing the Random Effects 23

general case all we can say about ũ or b̃ is that they maximize a conditional
density, which is why we use the term “conditional mode” to describe these
values. We will only use the term “conditional mean” and the symbol, µ, in
reference to E(Y |B = b), which is the conditional mean of Y given B, and
an important part of the formulation of all types of mixed-effects models.
The ranef extractor returns the conditional modes.
> ranef(fm01ML)

$Batch
(Intercept)
A -16.628222
B 0.369516
C 26.974671
D -21.801446
E 53.579825
F -42.494344
attr(,"class")
[1] "ranef.mer"

Applying str to the result of ranef

> str(ranef(fm01ML))

List of 1
$ Batch:'data.frame': 6 obs. of 1 variable:
..$ (Intercept): num [1:6] -16.628 0.37 26.975 -21.801 53.58 ...
- attr(*, "class")= chr "ranef.mer"

shows that the value is a list of data frames. In this case the list is of length 1
because there is only one random-effects term, (1|Batch), in the model and,
hence, only one grouping factor, Batch, for the random effects. There is only
one column in this data frame because the random-effects term, (1|Batch), is
a simple, scalar term.
To make this more explicit, random-effects terms in the model formula are
those that contain the vertical bar ("|") character. The Batch variable is the
grouping factor for the random effects generated by this term. An expression
for the grouping factor, usually just the name of a variable, occurs to the right
of the vertical bar. If the expression on the left of the vertical bar is 1, as it
is here, we describe the term as a simple, scalar, random-effects term. The
designation “scalar” means there will be exactly one random effect generated
for each level of the grouping factor. A simple, scalar term generates a block
of indicator columns — the indicators for the grouping factor — in Z. Because
there is only one random-effects term in this model and because that term
is a simple, scalar term, the model matrix Z for this model is the indicator
matrix for the levels of Batch.
In the next chapter we fit models with multiple simple, scalar terms and, in
subsequent chapters, we extend random-effects terms beyond simple, scalar
terms. When we have only simple, scalar terms in the model, each term has
a unique grouping factor and the elements of the list returned by ranef can

Page: 23 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

24 1 A Simple, Linear, Mixed-effects Model

E ●
C ●
B ●
A ●
D ●
F ●

−50 0 50

Fig. 1.10 95% prediction intervals on the random effects in fm01ML, shown as a
dotplot.

Fig. 1.11 95% prediction

intervals on the random 1.5 ●

Standard normal quantiles

effects in fm01ML versus 1.0

quantiles of the standard 0.5

●

normal distribution. 0.0

●

−0.5
●

−1.0
●
−1.5
−50 0 50

be considered as associated with terms or with grouping factors. In more

complex models a particular grouping factor may occur in more than one
term, in which case the elements of the list are associated with the grouping
factors, not the terms.
Given the data, y, and the parameter estimates, we can evaluate a measure
of the dispersion of (B|Y = y). In the case of a linear mixed model, this is
the conditional standard deviation, from which we can obtain a prediction
interval. The ranef extractor takes an optional argument, postVar = TRUE,
which adds these dispersion measures as an attribute of the result. (The
name stands for “posterior variance”, which is a misnomer that had become
established as an argument name before I realized that it wasn’t the correct
term.)
We can plot these prediction intervals using
> dotplot(ranef(fm01ML, postVar = TRUE))

(Fig.˜1.10), which provides linear spacing of the levels on the y axis, or using
> qqmath(ranef(fm01ML, postVar=TRUE))

(Fig.˜1.11), where the intervals are plotted versus quantiles of the standard
normal.
The dotplot is preferred when there are only a few levels of the grouping
factor, as in this case. When there are hundreds or thousands of random
effects the qqmath form is preferred because it focuses attention on the “im-
portant few” at the extremes and de-emphasizes the “trivial many” that are
close to zero.

Page: 24 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

1.7 Chapter Summary 25

1.7 Chapter Summary

A considerable amount of material has been presented in this chapter, espe-

cially considering the word “simple” in its title (it’s the model that is simple,
not the material). A summary may be in order.
A mixed-effects model incorporates fixed-effects parameters and random
effects, which are unobserved random variables, B. In a linear mixed model,
both the unconditional distribution of B and the conditional distribution,
(Y |B = b), are multivariate Gaussian distributions. Furthermore, this con-
ditional distribution is a spherical Gaussian with mean, µ, determined by the
linear predictor, Zb + Xβ . That is,

(Y |B = b) ∼ N (Zb + Xβ , σ 2 In ).

The unconditional distribution of B has mean 0 and a parameterized q × q

variance-covariance matrix, Σθ .
In the models we considered in this chapter, Σθ , is a simple multiple of the
identity matrix, I6 . This matrix is always a multiple of the identity in models
with just one random-effects term that is a simple, scalar term. The reason
for introducing all the machinery that we did is to allow for more general
model specifications.
The maximum likelihood estimates of the parameters are obtained by min-
imizing the deviance. For linear mixed models we can minimize the profiled
deviance, which is a function of θ only, thereby considerably simplifying the
optimization problem.
To assess the precision of the parameter estimates, we profile the deviance
function with respect to each parameter and apply a signed square root trans-
formation to the likelihood ratio test statistic, producing a profile zeta func-
tion for each parameter. These functions provide likelihood-based confidence
intervals for the parameters. Profile zeta plots allow us to visually assess the
precision of individual parameters. Profile pairs plots allow us to visualize the
pairwise dependence of parameter estimates and two-dimensional marginal
confidence regions.
Prediction intervals from the conditional distribution of the random effects,
given the observed data, allow us to assess the precision of the random effects.

Notation

Random Variables

Y The responses (n-dimensional Gaussian)

B The random effects on the original scale (q-dimensional Gaussian with
mean 0)

Page: 25 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

26 1 A Simple, Linear, Mixed-effects Model

U The orthogonal random effects (q-dimensional spherical Gaussian)

Values of these random variables are denoted by the corresponding bold-
face, lower-case letters: y, b and u. We observe y. We do not observe b or
u.

Parameters in the Probability Model

β The p-dimension fixed-effects parameter vector.

θ The variance-component parameter vector. Its (unnamed) dimension is
typically very small. Dimensions of 1, 2 or 3 are common in practice.
σ The (scalar) common scale parameter, σ > 0. It is called the common
scale parameter because it is incorporated in the variance-covariance ma-
trices of both Y and U .

The parameter θ determines the q × q lower triangular matrix Λθ , called

the relative covariance factor, which, in turn, determines the q × q sparse,
symmetric semidefinite variance-covariance matrix Σθ = σ 2Λθ ΛθT that defines
the distribution of B.

Model Matrices

X Fixed-effects model matrix of size n × p.

Z Random-effects model matrix of size n × q.

Derived Matrices

Lθ The sparse, lower triangular Cholesky factor of ΛθT ZT ZΛθ + Iq

In Chap.˜5 this definition will be modified to allow for a fill-reducing per-
mutation of the rows and columns of ΛθT ZT ZΛθ .

Vectors

In addition to the parameter vectors already mentioned, we define

y the n-dimensional observed response vector
γ the n-dimension linear predictor,

γ = Xβ + Zb = ZΛθ u + Xβ

µ the n-dimensional conditional mean of Y given B = b (or, equivalently,

given U = u)

Page: 26 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

1.7 Chapter Summary 27

D ● ● ●

C ● ● ●

F ● ● ●
Rail

A ●●●

E ●●●

B ● ● ●

40 60 80 100

Travel time for an ultrasonic wave (ms.)

Fig. 1.12 Travel time for an ultrasonic wave test on 6 rails

µ = E[Y |B = b] = E[Y |U = u]
ũθ the q-dimensional conditional mode (the value at which the conditional
density is maximized) of U given Y = y.

Exercises

These exercises and several others in this book use data sets from the MEMSS
package for R. You will need to ensure that this package is installed before
you can access the data sets.
To load a particular data set, either attach the package
> library(MEMSS)

or load just the one data set

> data(Rail, package = "MEMSS")

1.1. Check the documentation, the structure (str) and a summary of the Rail
data (Fig.˜1.12) from the MEMSS package. Note that if you used data to access
this data set (i.e. you did not attach the whole MEMSS package) then you must
use
> help(Rail, package = "MEMSS")

to display the documentation for it.

1.2. Fit a model with travel as the response and a simple, scalar random-
effects term for the variable Rail. Use the REML criterion, which is the de-
fault. Create a dotplot of the conditional modes of the random effects.
1.3. Refit the model using maximum likelihood. Check the parameter esti-
mates and, in the case of the fixed-effects parameter, its standard error. In
what ways have the parameter estimates changed? Which parameter esti-
mates have not changed?

Page: 27 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

28 1 A Simple, Linear, Mixed-effects Model

1.4. Profile the fitted model and construct 95% profile-based confidence in-
tervals on the parameters. Is the confidence interval on σ1 close to being
symmetric about the estimate? Is the corresponding interval on log(σ1 ) close
to being symmetric about its estimate?

1.5. Create the profile zeta plot for this model. For which parameters are
there good normal approximations?

1.6. Create a profile pairs plot for this model. Does the shape of the deviance
contours in this model mirror those in Fig.˜1.9?

1.7. Plot the prediction intervals on the random effects from this model. Do
any of these prediction intervals contain zero? Consider the relative magni-
tudes of σ
c1 and σ b in this model compared to those in model fm01 for the
Dyestuff data. Should these ratios of σ1 /σ lead you to expect a different
pattern of prediction intervals in this plot than those in Fig.˜1.10?

Page: 28 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

Chapter 2
Models With Multiple Random-effects
Terms

The mixed models considered in the previous chapter had only one random-
effects term, which was a simple, scalar random-effects term, and a single
fixed-effects coefficient. Although such models can be useful, it is with the
facility to use multiple random-effects terms and to use random-effects terms
beyond a simple, scalar term that we can begin to realize the flexibility and
versatility of mixed models.
In this chapter we consider models with multiple simple, scalar random-
effects terms, showing examples where the grouping factors for these terms
are in completely crossed or nested or partially crossed configurations. For
ease of description we will refer to the random effects as being crossed or
nested although, strictly speaking, the distinction between nested and non-
nested refers to the grouping factors, not the random effects.

2.1 A Model With Crossed Random Effects

One of the areas in which the methods in the lme4 package for R are particu-
larly effective is in fitting models to cross-classified data where several factors
have random effects associated with them. For example, in many experiments
in psychology the reaction of each of a group of subjects to each of a group
of stimuli or items is measured. If the subjects are considered to be a sample
from a population of subjects and the items are a sample from a population
of items, then it would make sense to associate random effects with both
these factors.
In the past it was difficult to fit mixed models with multiple, crossed
grouping factors to large, possibly unbalanced, data sets. The methods in
the lme4 package are able to do this. To introduce the methods let us first
consider a small, balanced data set with crossed grouping factors.

29
30 2 Models With Multiple Random-effects Terms

2.1.1 The Penicillin Data

The Penicillin data are derived from Table˜6.6, p.˜144 of Davies and Gold-
smith [1972] where they are described as coming from an investigation to
assess the variability between samples of penicillin by the B.˜subtilis method.
In this test method a bulk-innoculated nutrient agar medium is poured into
a Petri dish of approximately 90 mm. diameter, known as a plate. When the
medium has set, six small hollow cylinders or pots (about 4 mm. in diameter)
are cemented onto the surface at equally spaced intervals. A few drops of the
penicillin solutions to be compared are placed in the respective cylinders, and
the whole plate is placed in an incubator for a given time. Penicillin diffuses
from the pots into the agar, and this produces a clear circular zone of inhibition
of growth of the organisms, which can be readily measured. The diameter of
the zone is related in a known way to the concentration of penicillin in the
solution.

As with the Dyestuff data, we examine the structure

> str(Penicillin)

'data.frame': 144 obs. of 3 variables:

$ diameter: num 27 23 26 23 23 21 27 23 26 23 ...
$ plate : Factor w/ 24 levels "a","b","c","d",..: 1 1 1 1 1 1 2 2 2 2..
$ sample : Factor w/ 6 levels "A","B","C","D",..: 1 2 3 4 5 6 1 2 3 4 ..

and a summary
> summary(Penicillin)

diameter plate sample

Min. :18.00 a : 6 A:24
1st Qu.:22.00 b : 6 B:24
Median :23.00 c : 6 C:24
Mean :22.97 d : 6 D:24
3rd Qu.:24.00 e : 6 E:24
Max. :27.00 f : 6 F:24
(Other):108

of the Penicillin data, then plot it (Fig.˜2.1).

The variation in the diameter is associated with the plates and with the
samples. Because each plate is used only for the six samples shown here we
are not interested in the contributions of specific plates as much as we are
interested in the variation due to plates and in assessing the potency of the
samples after accounting for this variation. Thus, we will use random effects
for the plate factor. We will also use random effects for the sample factor
because, as in the dyestuff example, we are more interested in the sample-to-
sample variability in the penicillin samples than in the potency of a particular
sample.
In this experiment each sample is used on each plate. We say that the
sample and plate factors are crossed, as opposed to nested factors, which
we will describe in the next section. By itself, the designation “crossed” just

Page: 30 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

2.1 A Model With Crossed Random Effects 31

A ● C E
B D F ●

m ● ●

t ● ●

o ● ●

k ● ●

h ● ●

b ● ●

a ● ●

n ● ●

l ● ●

d ● ●

c ● ●
Plate

p ● ●

e ● ●

v ● ●

r ● ●

q ● ●

f ● ●

w ● ●

j ● ●

i ● ●

u ● ●

x ● ●

s ● ●

g ● ●

18 20 22 24 26

Diameter of growth inhibition zone (mm)

Fig. 2.1 Diameter of the growth inhibition zone (mm) in the B. subtilis method of
assessing the concentration of penicillin. Each of 6 samples was applied to each of the
24 agar plates. The lines join observations on the same sample.

means that the factors are not nested. If we wish to be more specific, we
could describe these factors as being completely crossed, which means that
we have at least one observation for each combination of a level of sample and
a level of plate. We can see this in Fig.˜2.1 and, because there are moderate
numbers of levels in these factors, we can check it in a cross-tabulation
> xtabs(~ sample + plate, Penicillin)

plate
sample a b c d e f g h i j k l m n o p q r s t u v w x
A 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
B 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
C 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
D 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
E 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
F 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Like the Dyestuff data, the factors in the Penicillin data are balanced.
That is, there are exactly the same number of observations on each plate and

Page: 31 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

32 2 Models With Multiple Random-effects Terms

for each sample and, furthermore, there is the same number of observations
on each combination of levels. In this case there is exactly one observation for
each combination of sample and plate. We would describe the configuration
of these two factors as an unreplicated, completely balanced, crossed design.
In general, balance is a desirable but precarious property of a data set.
We may be able to impose balance in a designed experiment but we typically
cannot expect that data from an observation study will be balanced. Also,
as anyone who analyzes real data soon finds out, expecting that balance in
the design of an experiment will produce a balanced data set is contrary to
“Murphy’s Law”. That’s why statisticians allow for missing data. Even when
we apply each of the six samples to each of the 24 plates, something could
go wrong for one of the samples on one of the plates, leaving us without a
measurement for that combination of levels and thus an unbalanced data set.

2.1.2 A Model For the Penicillin Data

A model incorporating random effects for both the plate and the sample is
straightforward to specify — we include simple, scalar random effects terms
for both these factors.
> (fm03 <- lmer(diameter ~ 1 + (1|plate) + (1|sample), Penicillin))

Linear mixed model fit by REML ['merMod']

Formula: diameter ~ 1 + (1 | plate) + (1 | sample)
Data: Penicillin
REML criterion at convergence: 330.8606
Random effects:
Groups Name Variance Std.Dev.
plate (Intercept) 0.7169 0.8467
sample (Intercept) 3.7309 1.9316
Residual 0.3024 0.5499
Number of obs: 144, groups: plate, 24; sample, 6

Fixed effects:
Estimate Std. Error t value
(Intercept) 22.9722 0.8086 28.41

This model display indicates that the sample-to-sample variability has the
greatest contribution, then plate-to-plate variability and finally the “resid-
ual” variability that cannot be attributed to either the sample or the plate.
These conclusions are consistent with what we see in the Penicillin data plot
(Fig.˜2.1).
The prediction intervals on the random effects (Fig.˜2.2) confirm that
the conditional distribution of the random effects for plate has much less
variability than does the conditional distribution of the random effects for
plate, in the sense that the dots in the bottom panel have less variability than
those in the top panel. (Note the different horizontal axes for the two panels.)

Page: 32 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

2.1 A Model With Crossed Random Effects 33

A ●
C ●
E ●
D ●
B ●
F ●

−3 −2 −1 0 1 2

m ●
t ●
o ●
k ●
h ●
b ●
a ●
l ●
n ●
d ●
c ●
p ●
e ●
v ●
r ●
q ●
f ●
w ●
j ●
i ●
u ●
x ●
s ●
g ●

−2 −1 0 1 2

Fig. 2.2 95% prediction intervals on the random effects for model fm03 fit to the
Penicillin data.

However, the conditional distribution of the random effect for a particular

sample, say sample F, has less variability than the conditional distribution of
the random effect for a particular plate, say plate m. That is, the lines in the
bottom panel are wider than the lines in the top panel, even after taking the
different axis scales into account. This is because the conditional distribution
of the random effect for a particular sample depends on 24 responses while the
conditional distribution of the random effect for a particular plate depends
on only 6 responses.
In chapter˜1 we saw that a model with a single, simple, scalar random-
effects term generated a random-effects model matrix, Z, that is the matrix
of indicators of the levels of the grouping factor. When we have multiple,
simple, scalar random-effects terms, as in model fm03, each term generates a
matrix of indicator columns and these sets of indicators are concatenated to
form the model matrix Z. The transpose of this matrix, shown in Fig.˜2.3,
contains rows of indicators for each factor.
The relative covariance factor, Λθ , (Fig.˜2.4, left panel) is no longer a
multiple of the identity. It is now block diagonal, with two blocks, one of size
24 and one of size 6, each of which is a multiple of the identity. The diagonal
elements of the two blocks are θ1 and θ2 , respectively. The numeric values of
these parameters can be obtained as
> fm03@re@theta

Page: 33 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

34 2 Models With Multiple Random-effects Terms

5
10
15
20
25

50 100

Fig. 2.3 Image of the transpose of the random-effects model matrix, Z, for model
fm03. The non-zero elements, which are all unity, are shown as darkened squares. The
zero elements are blank.

5 5 5
10 10 10
15 15 15
20 20 20
25 25 25

5 10 15 20 25 5 10 15 20 25 5 10 15 20 25

Λ Z'Z L

Fig. 2.4 Images of the relative covariance factor, Λ , the cross-product of the random-
effects model matrix, ZT Z, and the sparse Cholesky factor, L, for model fm03.

[1] 1.539678 3.512415

The first parameter is the relative standard deviation of the random effects
for plate, which has the value 0.84671/0.54992 = 1.53968 at convergence, and
the second is the relative standard deviation of the random effects for sample
(1.93157/0.54992 = 3.512443).
Because Λθ is diagonal, the pattern of non-zeros in ΛθT ZT ZΛθ + I will be
the same as that in ZT Z, shown in the middle panel of Fig.˜2.4. The sparse
Cholesky factor, L, shown in the right panel, is lower triangular and has
non-zero elements in the lower right hand corner in positions where ZT Z has
systematic zeros. We say that “fill-in” has occurred when forming the sparse
Cholesky decomposition. In this case there is a relatively minor amount of fill
but in other cases there can be a substantial amount of fill and we shall take
precautions so as to reduce this, because fill-in adds to the computational
effort in determining the MLEs or the REML estimates.
A profile zeta plot (Fig.˜2.5) for the parameters in model fm03 leads to con-
clusions similar to those from Fig.˜?? for model fm1ML in the previous chapter.
The fixed-effect parameter, β0 , for the (Intercept) term has symmetric inter-
vals and is over-dispersed relative to the normal distribution. The logarithm

Page: 34 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

2.1 A Model With Crossed Random Effects 35

.sig01 .sig02 .lsig (Intercept)

0
ζ

−1

−2

0.6 0.8 1.0 1.2 1 2 3 4 −0.7 −0.6 −0.5 21 22 23 24 25

Fig. 2.5 Profile zeta plot of the parameters in model fm03.

of σ has a good normal approximation but the standard deviations of the

random effects, σ1 and σ2 , are skewed. The skewness for σ2 is worse than
that for σ1 , because the estimate of σ2 is less precise than that of σ1 , in
both absolute and relative senses. For an absolute comparison we compare
the widths of the confidence intervals for these parameters.
> confint(pr03)

2.5 % 97.5 %
.sig01 0.6335658 1.1821040
.sig02 1.0957822 3.5563194
.lsig -0.7218645 -0.4629033
(Intercept) 21.2666274 24.6778176

In a relative comparison we examine the ratio of the endpoints of the interval

divided by the estimate.
> confint(pr03)[1:2,]/c(0.8455722, 1.770648)

2.5 % 97.5 %
.sig01 0.7492746 1.397993
.sig02 0.6188594 2.008485

The lack of precision in the estimate of σ2 is a consequence of only having

6 distinct levels of the sample factor. The plate factor, on the other hand,
has 24 distinct levels. In general it is more difficult to estimate a measure
of spread, such as the standard deviation, than to estimate a measure of
location, such as a mean, especially when the number of levels of the factor
is small. Six levels are about the minimum number required for obtaining
sensible estimates of standard deviations for simple, scalar random effects
terms.
The profile pairs plot (Fig.˜2.6) shows patterns similar to those in Fig.˜??
for pairs of parameters in model fm1 fit to the Dyestuff data. On the ζ scale

Page: 35 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

36 2 Models With Multiple Random-effects Terms

26

24
(Intercept)
22

20 0 1 2 3

−0.4 −0.6 −0.5 −0.4

−0.5

−0.6 .lsig

−0.7
0 1 2 3
−0.8
6
4 5 6
5

4
.sig02
3

1 0 1 2 3

0.6 0.8 1.0 1.2 1.4

.sig01 0

−1

−2

−3
Scatter Plot Matrix

Fig. 2.6 Profile pairs plot for the parameters in model fm03 fit to the Penicillin
data.

(panels below the diagonal) the profile traces are nearly straight and orthog-
onal with the exception of the trace of ζ (σ2 ) on ζ (β0 ) (the horizontal trace
for the panel in the (4, 2) position). The pattern of this trace is similar to
the pattern of the trace of ζ (σ1 ) on ζ (β0 ) in Fig.˜??. Moving β0 from its
estimate, βb0 , in either direction will increase the residual sum of squares. The
increase in the residual variability is reflected in an increase of one or more
of the dispersion parameters. The balanced experimental design results in a
fixed estimate of σ and the extra apparent variability must be incorporated
into σ1 or σ2 .
Contours in panels of parameter pairs on the original scales (i.e. panels
above the diagonal) can show considerable distortion from the ideal elliptical
shape. For example, contours in the σ2 versus σ1 panel (the (1, 2) position)
and the log(σ ) versus σ2 panel (in the (2, 3) position) are dramatically non-

Page: 36 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

2.1 A Model With Crossed Random Effects 37

26

24
(Intercept)
22

20 0 1 2 3

−0.4 −0.6 −0.5 −0.4

−0.5

−0.6 .lsig

−0.7
0 1 2 3
−0.8
1.0 1.5
1.5

1.0
.lsig02
0.5

0.0 0 1 2 3

−0.6 −0.4 −0.2 0.0 0.2 0.4

.lsig01 0

−1

−2

−3
Scatter Plot Matrix

Fig. 2.7 Profile pairs plot for the parameters in model fm03 fit to the Penicillin
data. In this plot the parameters σ1 and σ2 are on the scale of the natural logarithm,
as is the parameter σ in this and other profile pairs plots.

elliptical. However, the distortion of the contours is not due to these param-
eter estimates depending strongly on each other. It is almost entirely due to
the choice of scale for σ1 and σ2 . When we plot the contours on the scale of
log(σ1 ) and log(σ2 ) instead (Fig.˜2.7) they are much closer to the elliptical
pattern.
Conversely, if we tried to plot contours on the scale of σ12 and σ22 (not
shown), they would be hideously distorted.

Page: 37 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

38 2 Models With Multiple Random-effects Terms

2.2 A Model With Nested Random Effects

In this section we again consider a simple example, this time fitting a model
with nested grouping factors for the random effects.

2.2.1 The Pastes Data

The third example from Davies and Goldsmith [1972, Table˜6.5, p.˜138] is
described as coming from
deliveries of a chemical paste product contained in casks where, in addition to
sampling and testing errors, there are variations in quality between deliveries
. . . As a routine, three casks selected at random from each delivery were sampled
and the samples were kept for reference. . . . Ten of the delivery batches were
sampled at random and two analytical tests carried out on each of the 30
samples.

The structure and summary of the Pastes data object are

> str(Pastes)

'data.frame': 60 obs. of 4 variables:

$ strength: num 62.8 62.6 60.1 62.3 62.7 63.1 60 61.4 57.5 56.9 ...
$ batch : Factor w/ 10 levels "A","B","C","D",..: 1 1 1 1 1 1 2 2 2 2..
$ cask : Factor w/ 3 levels "a","b","c": 1 1 2 2 3 3 1 1 2 2 ...
$ sample : Factor w/ 30 levels "A:a","A:b","A:c",..: 1 1 2 2 3 3 4 4 5..

> summary(Pastes)

strength batch cask sample

Min. :54.20 A : 6 a:20 A:a : 2
1st Qu.:57.50 B : 6 b:20 A:b : 2
Median :59.30 C : 6 c:20 A:c : 2
Mean :60.05 D : 6 B:a : 2
3rd Qu.:62.88 E : 6 B:b : 2
Max. :66.00 F : 6 B:c : 2
(Other):24 (Other):48

As stated in the description in Davies and Goldsmith [1972], there are

30 samples, three from each of the 10 delivery batches. We have labelled
the levels of the sample factor with the label of the batch factor followed
by ‘a’, ‘b’ or ‘c’ to distinguish the three samples taken from that batch. The
cross-tabulation produced by the xtabs function, using the optional argument
sparse = TRUE, provides a concise display of the relationship.
> xtabs(~ batch + sample, Pastes, drop = TRUE, sparse = TRUE)

10 x 30 sparse Matrix of class "dgCMatrix"

A 2 2 2 . . . . . . . . . . . . . . . . . . . . . . . . . . .
B . . . 2 2 2 . . . . . . . . . . . . . . . . . . . . . . . .
C . . . . . . 2 2 2 . . . . . . . . . . . . . . . . . . . . .

Page: 38 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

2.2 A Model With Nested Random Effects 39

batch
6

10

5 10 15 20 25

sample

Fig. 2.8 Image of the cross-tabulation of the batch and sample factors in the Pastes
data.

D . . . . . . . . . 2 2 2 . . . . . . . . . . . . . . . . . .
E . . . . . . . . . . . . 2 2 2 . . . . . . . . . . . . . . .
F . . . . . . . . . . . . . . . 2 2 2 . . . . . . . . . . . .
G . . . . . . . . . . . . . . . . . . 2 2 2 . . . . . . . . .
H . . . . . . . . . . . . . . . . . . . . . 2 2 2 . . . . . .
I . . . . . . . . . . . . . . . . . . . . . . . . 2 2 2 . . .
J . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2 2

Alternatively, we can use an image (Fig.˜2.8) of this cross-tabulation to vi-

sualize the structure.
When plotting the strength versus batch and sample in the Pastes data we
should remember that we have two strength measurements on each of the
30 samples. It is tempting to use the cask designation (‘a’, ‘b’ and ‘c’) to
determine, say, the plotting symbol within a batch. It would be fine to do this
within a batch but the plot would be misleading if we used the same symbol
for cask ‘a’ in different batches. There is no relationship between cask ‘a’ in
batch ‘A’ and cask ‘a’ in batch ‘B’. The labels ‘a’, ‘b’ and ‘c’ are used only
to distinguish the three samples within a batch; they do not have a meaning
across batches.
In Fig.˜2.9 we plot the two strength measurements on each of the samples
within each of the batches and join up the average strength for each sample.
The perceptive reader will have noticed that the levels of the factors on the
vertical axis in this figure, and in Fig.˜1.1 and 2.1, have been reordered ac-
cording to increasing average response. In all these cases there is no inherent
ordering of the levels of the covariate such as batch or plate. Rather than con-
fuse our interpretation of the plot by determining the vertical displacement
of points according to a random ordering, we impose an ordering according
to increasing mean response. This allows us to more easily check for structure
in the data, including undesirable characteristics like increasing variability of
the response with increasing mean level of the response.
In Fig.˜2.9 we order the samples within each batch separately then order
the batches according to increasing mean strength.

Page: 39 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

40 2 Models With Multiple Random-effects Terms

H:b ● ●

H
H:c ● ●

H:a ●●

A:c ● ●

A
A:a ●●

A:b ● ●

C:c ● ●
C

C:b ● ●

C:a ● ●

F:a ● ●
F

F:c ● ●

F:b
Sample within batch

● ●

G:a ●●
G

G:b ● ●

G:c ● ●

D:c ●●
D

D:b ● ●

D:a ● ●

B:a ● ●
B

B:c ● ●

B:b ● ●

I:b ●

I:c ● ●
I

I:a ●

J:b ●

J:a ●
J

J:c ● ●

E:c ● ●
E

E:a ●

E:b ● ●

54 56 58 60 62 64 66

Paste strength

Fig. 2.9 Strength of paste preparations according to the batch and the sample within
the batch. There were two strength measurements on each of the 30 samples; three
samples each from 10 batches.

Figure˜2.9 shows considerable variability in strength between samples rel-

ative to the variability within samples. There is some indication of variability
between batches, in addition to the variability induced by the samples, but
not a strong indication of a batch effect. For example, batches I and D, with
low mean strength relative to the other batches, each contained one sam-
ple (I:b and D:c, respectively) that had high mean strength relative to the
other samples. Also, batches H and C, with comparatively high mean batch
strength, contain samples H:a and C:a with comparatively low mean sample
strength. In Sect.˜2.2.4 we will examine the need for incorporating batch-to-
batch variability, in addition to sample-to-sample variability, in the statistical
model.

Page: 40 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

2.2 A Model With Nested Random Effects 41

2.2.1.1 Nested Factors

Because each level of sample occurs with one and only one level of batch we
say that sample is nested within batch. Some presentations of mixed-effects
models, especially those related to multilevel modeling˜[Rasbash et˜al., 2000]
or hierarchical linear models˜[Raudenbush and Bryk, 2002], leave the im-
pression that one can only define random effects with respect to factors that
are nested. This is the origin of the terms “multilevel”, referring to multiple,
nested levels of variability, and “hierarchical”, also invoking the concept of
a hierarchy of levels. To be fair, both those references do describe the use
of models with random effects associated with non-nested factors, but such
models tend to be treated as a special case.
The blurring of mixed-effects models with the concept of multiple, hier-
archical levels of variation results in an unwarranted emphasis on “levels”
when defining a model and leads to considerable confusion. It is perfectly le-
gitimate to define models having random effects associated with non-nested
factors. The reasons for the emphasis on defining random effects with respect
to nested factors only are that such cases do occur frequently in practice and
that some of the computational methods for estimating the parameters in
the models can only be easily applied to nested factors.
This is not the case for the methods used in the lme4 package. Indeed there
is nothing special done for models with random effects for nested factors.
When random effects are associated with multiple factors exactly the same
computational methods are used whether the factors form a nested sequence
or are partially crossed or are completely crossed.
There is, however, one aspect of nested grouping factors that we should
emphasize, which is the possibility of a factor that is implicitly nested within
another factor. Suppose, for example, that the sample factor was defined as
having three levels instead of 30 with the implicit assumption that sample
is nested within batch. It may seem silly to try to distinguish 30 different
batches with only three levels of a factor but, unfortunately, data are fre-
quently organized and presented like this, especially in text books. The cask
factor in the Pastes data is exactly such an implicitly nested factor. If we
cross-tabulate batch and cask
> xtabs(~ cask + batch, Pastes)

batch
cask A B C D E F G H I J
a 2 2 2 2 2 2 2 2 2 2
b 2 2 2 2 2 2 2 2 2 2
c 2 2 2 2 2 2 2 2 2 2

we get the impression that the cask and batch factors are crossed, not nested.
If we know that the cask should be considered as nested within the batch then
we should create a new categorical variable giving the batch-cask combina-
tion, which is exactly what the sample factor is. A simple way to create such a

Page: 41 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

42 2 Models With Multiple Random-effects Terms

factor is to use the interaction operator, ‘:’, on the factors. It is advisable, but
not necessary, to apply factor to the result thereby dropping unused levels of
the interaction from the set of all possible levels of the factor. (An “unused
level” is a combination that does not occur in the data.) A convenient code
idiom is
> Pastes$sample <- with(Pastes, factor(batch:cask))

or
> Pastes <- within(Pastes, sample <- factor(batch:cask))

In a small data set like Pastes we can quickly detect a factor being implic-
itly nested within another factor and take appropriate action. In a large data
set, perhaps hundreds of thousands of test scores for students in thousands
of schools from hundreds of school districts, it is not always obvious if school
identifiers are unique across the entire data set or just within a district. If you
are not sure, the safest thing to do is to create the interaction factor, as shown
above, so you can be confident that levels of the district:school interaction
do indeed correspond to unique schools.

2.2.2 Fitting a Model With Nested Random Effects

Fitting a model with simple, scalar random effects for nested factors is done
in exactly the same way as fitting a model with random effects for crossed
grouping factors. We include random-effects terms for each factor, as in
> (fm04 <- lmer(strength ~ 1 + (1|sample) + (1|batch), Pastes, REML=FALSE))

Linear mixed model fit by maximum likelihood ['merMod']

Formula: strength ~ 1 + (1 | sample) + (1 | batch)
Data: Pastes
AIC BIC logLik deviance
255.9945 264.3718 -123.9972 247.9945
Random effects:
Groups Name Variance Std.Dev.
sample (Intercept) 8.434 2.9041
batch (Intercept) 1.199 1.0951
Residual 0.678 0.8234
Number of obs: 60, groups: sample, 30; batch, 10

Fixed effects:
Estimate Std. Error t value
(Intercept) 60.0533 0.6421 93.52

Not only is the model specification similar for nested and crossed factors,
the internal calculations are performed according to the methods described in
Sect.˜1.4.1 for each model type. Comparing the patterns in the matrices Λ ,
ZT Z and L for this model (Fig.˜2.10) to those in Fig.˜2.4 shows that models
with nested factors produce simple repeated structures along the diagonal of

Page: 42 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

2.2 A Model With Nested Random Effects 43

10 10 10

20 20 20

30 30 30

10 20 30 10 20 30 10 20 30

Λ Z'Z L

Fig. 2.10 Images of the relative covariance factor, Λ , the cross-product of the
random-effects model matrix, ZT Z, and the sparse Cholesky factor, L, for model
fm04.

the sparse Cholesky factor, L, after reordering the random effects (we discuss
this reordering later in Sect.˜5.4.1). This type of structure has the desirable
property that there is no “fill-in” during calculation of the Cholesky factor.
In other words, the number of non-zeros in L is the same as the number of
non-zeros in the lower triangle of the matrix being factored, Λ T ZT ZΛ + I
(which, because Λ is diagonal, has the same structure as ZT Z).
Fill-in of the Cholesky factor is not an important issue when we have a few
dozen random effects, as we do here. It is an important issue when we have
millions of random effects in complex configurations, as has been the case in
some of the models that have been fit using lmer.

2.2.3 Assessing Parameter Estimates in Model fm04

The parameter estimates are: σ c1 =2.904, the standard deviation of the ran-
dom effects for sample; σ
c2 =1.095, the standard deviation of the random effects
for batch; σ =0.823, the standard deviation of the residual noise term; and
b
βb0 =60.053, the overall mean response, which is labeled (Intercept) in these
models.
The estimated standard deviation for sample is nearly three times as large
as that for batch, which confirms what we saw in Fig.˜2.9. Indeed our con-
clusion from Fig.˜2.9 was that there may not be a significant batch-to-batch
variability in addition to the sample-to-sample variability.
Plots of the prediction intervals of the random effects (Fig.˜2.11) confirm
this impression in that all the prediction intervals for the random effects for
batch contain zero. Furthermore, the profile zeta plot (Fig.˜2.12) shows that
the even the 50% profile-based confidence interval on σ2 extends to zero.

Page: 43 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

44 2 Models With Multiple Random-effects Terms

H ●
A ●
C ●
F ●
G ●
D ●
B ●
I ●
J ●
E ●

−3 −2 −1 0 1 2

H:b ●
D:c ●
I:b ●
C:c ●
F:a ●
H:c ●
C:b ●
G:a ●
A:c ●
A:a ●
B:a ●
A:b ●
B:c ●
F:c ●
G:b ●
J:b ●
E:c ●
J:a ●
F:b ●
H:a ●
J:c ●
D:b ●
I:c ●
C:a ●
B:b ●
G:c ●
D:a ●
E:a ●
E:b ●
I:a ●

−5 0 5

Fig. 2.11 95% prediction intervals on the random effects for model fm04 fit to the
Pastes data.

.sig01 .sig02 .lsig (Intercept)

0
ζ

−1

−2

2.0 2.5 3.0 3.5 4.0 4.5 0 1 2 3 −0.4 −0.2 0.0 0.258 59 60 61 62

Fig. 2.12 Profile zeta plots for the parameters in model fm04.

Page: 44 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

2.2 A Model With Nested Random Effects 45

Because there are several indications that σ2 could reasonably be zero,

resulting in a simpler model incorporating random effects for batch only, we
perform a statistical test of this hypothesis.

2.2.4 Testing H0 : σ2 = 0 Versus Ha : σ2 > 0

One of the many famous statements attributed to Albert Einstein is “Every-

thing should be made as simple as possible, but not simpler.” In statistical
modeling this principal of parsimony is embodied in hypothesis tests compar-
ing two models, one of which contains the other as a special case. Typically,
one or more of the parameters in the more general model, which we call the
alternative hypothesis, is constrained in some way, resulting in the restricted
model, which we call the null hypothesis. Although we phrase the hypothesis
test in terms of the parameter restriction, it is important to realize that we
are comparing the quality of fits obtained with two nested models. That is,
we are not assessing parameter values per se; we are comparing the model
fit obtainable with some constraints on parameter values to that without the
constraints.
Because the more general model, Ha , must provide a fit that is at least as
good as the restricted model, H0 , our purpose is to determine whether the
change in the quality of the fit is sufficient to justify the greater complexity
of model Ha . This comparison is often reduced to a p-value, which is the
probability of seeing a difference in the model fits as large as we did, or even
larger, when, in fact, H0 is adequate. Like all probabilities, a p-value must
be between 0 and 1. When the p-value for a test is small (close to zero) we
prefer the more complex model, saying that we “reject H0 in favor of Ha ”. On
the other hand, when the p-value is not small we “fail to reject H0 ”, arguing
that there is a non-negligible probability that the observed difference in the
model fits could reasonably be the result of random chance, not the inherent
superiority of the model Ha . Under these circumstances we prefer the simpler
model, H0 , according to the principal of parsimony.
These are the general principles of statistical hypothesis tests. To perform a
test in practice we must specify the criterion for comparing the model fits, the
method for calculating the p-value from an observed value of the criterion, and
the standard by which we will determine if the p-value is “small” or not. The
criterion is called the test statistic, the p-value is calculated from a reference
distribution for the test statistic, and the standard for small p-values is called
the level of the test.
In Sect.˜1.5 we referred to likelihood ratio tests (LRTs) for which the test
statistic is the difference in the deviance. That is, the LRT statistic is d0 − da
where da is the deviance in the more general (Ha ) model fit and d0 is the de-
viance in the constrained (H0 ) model. An approximate reference distribution
for an LRT statistic is the χν2 distribution where ν, the degrees of freedom,

Page: 45 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

46 2 Models With Multiple Random-effects Terms

is determined by the number of constraints imposed on the parameters of Ha

to produce H0 .
The restricted model fit
> (fm04a <- lmer(strength ~ 1 + (1|sample), Pastes, REML=FALSE))

Linear mixed model fit by maximum likelihood ['merMod']

Formula: strength ~ 1 + (1 | sample)
Data: Pastes
AIC BIC logLik deviance
254.4017 260.6847 -124.2008 248.4017
Random effects:
Groups Name Variance Std.Dev.
sample (Intercept) 9.633 3.1037
Residual 0.678 0.8234
Number of obs: 60, groups: sample, 30

Fixed effects:
Estimate Std. Error t value
(Intercept) 60.0533 0.5765 104.2

is compared to model fm04 with the anova function

> anova(fm04a, fm04)

Data: Pastes
Models:
fm04a: strength ~ 1 + (1 | sample)
fm04: strength ~ 1 + (1 | sample) + (1 | batch)
Df AIC BIC logLik Chisq Chi Df Pr(>Chisq)
fm04a 3 254.40 260.69 -124.20
fm04 4 255.99 264.37 -124.00 0.4072 1 0.5234

which provides a p-value of 0.5234. Because typical standards for “small” p-

values are 5% or 1%, a p-value over 50% would not be considered significant
at any reasonable level.
We do need to be cautious in quoting this p-value, however, because the
parameter value being tested, σ2 = 0, is on the boundary of set of possible
values, σ2 ≥ 0, for this parameter. The argument for using a χ12 distribution
to calculate a p-value for the change in the deviance does not apply when the
parameter value being tested is on the boundary. As shown in Pinheiro and
Bates [2000, Sect.˜2.5], the p-value from the χ12 distribution will be “conser-
vative” in the sense that it is larger than a simulation-based p-value would
be. In the worst-case scenario the χ 2 -based p-value will be twice as large as
it should be but, even if that were true, an effective p-value of 26% would
not cause us to reject H0 in favor of Ha .

2.2.5 Assessing the Reduced Model, fm04a

The profile zeta plots for the remaining parameters in model fm04a (Fig.˜2.13)

Page: 46 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

2.3 A Model With Partially Crossed Random Effects 47

.sig01 .lsig (Intercept)

ζ 0

−1

−2

2.5 3.0 3.5 4.0 4.5 −0.4 −0.2 0.0 0.2 59 60 61

Fig. 2.13 Profile zeta plots for the parameters in model fm04a.

are similar to the corresponding panels in Fig.˜2.12, as confirmed by the

numerical values of the confidence intervals.
> confint(pr04)

2.5 % 97.5 %
.sig01 2.1579337 4.05358894
.sig02 NA 2.94658934
.lsig -0.4276761 0.08199287
(Intercept) 58.6636504 61.44301637

> confint(pr04a)

2.5 % 97.5 %
.sig01 2.4306377 4.12201052
.lsig -0.4276772 0.08199277
(Intercept) 58.8861831 61.22048353

The confidence intervals on log(σ ) and β0 are similar for the two models.
The confidence interval on σ1 is slightly wider in model fm04a than in fm04,
because the variability that is attributed to batch in fm04 is incorporated into
the variability due to sample in fm04a.
The patterns in the profile pairs plot (Fig.˜2.14) for the reduced model
fm04a are similar to those in Fig.˜??, the profile pairs plot for model fm1.

2.3 A Model With Partially Crossed Random Effects

Especially in observational studies with multiple grouping factors, the con-

figuration of the factors frequently ends up neither nested nor completely
crossed. We describe such situations as having partially crossed grouping fac-
tors for the random effects.

Page: 47 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

48 2 Models With Multiple Random-effects Terms

62

61

60 (Intercept)

59

0 1 2 3
58

0.2 0.0 0.2

0.0

.lsig
−0.2

−0.4
0 1 2 3

2.5 3.0 3.5 4.0 4.5 5.0

.sig01 0

−1

−2

−3
Scatter Plot Matrix

Fig. 2.14 Profile pairs plot for the parameters in model fm04a fit to the Pastes data.

Studies in education, in which test scores for students over time are also
associated with teachers and schools, usually result in partially crossed group-
ing factors. If students with scores in multiple years have different teachers
for the different years, the student factor cannot be nested within the teacher
factor. Conversely, student and teacher factors are not expected to be com-
pletely crossed. To have complete crossing of the student and teacher factors
it would be necessary for each student to be observed with each teacher,
which would be unusual. A longitudinal study of thousands of students with
hundreds of different teachers inevitably ends up partially crossed.
In this section we consider an example with thousands of students and
instructors where the response is the student’s evaluation of the instructor’s
effectiveness. These data, like those from most large observational studies,
are quite unbalanced.

Page: 48 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

2.3 A Model With Partially Crossed Random Effects 49

2.3.1 The InstEval Data

The InstEval data are from a special evaluation of lecturers by students at the
Swiss Federal Institute for Technology–Zürich (ETH–Zürich), to determine
who should receive the “best-liked professor” award. These data have been
slightly simplified and identifying labels have been removed, so as to preserve
anonymity.
The variables
> str(InstEval)

'data.frame': 73421 obs. of 7 variables:

$ s : Factor w/ 2972 levels "1","2","3","4",..: 1 1 1 1 2 2 3 3 3 ..
$ d : Factor w/ 1128 levels "1","6","7","8",..: 525 560 832 1068 6..
$ studage: Ord.factor w/ 4 levels "2"<"4"<"6"<"8": 1 1 1 1 1 1 1 1 1 1 ..
$ lectage: Ord.factor w/ 6 levels "1"<"2"<"3"<"4"<..: 2 1 2 2 1 1 1 1 1..
$ service: Factor w/ 2 levels "0","1": 1 2 1 2 1 1 2 1 1 1 ...
$ dept : Factor w/ 14 levels "15","5","10",..: 14 5 14 12 2 2 13 3 3 ..
$ y : int 5 2 5 3 2 4 4 5 5 4 ...

have somewhat cryptic names. Factor s designates the student and d the
instructor. The dept factor is the department for the course and service indi-
cates whether the course was a service course taught to students from other
departments.
Although the response, y, is on a scale of 1 to 5,
> xtabs(~ y, InstEval)

y
1 2 3 4 5
10186 12951 17609 16921 15754

it is sufficiently diffuse to warrant treating it as if it were a continuous re-

sponse.
At this point we will fit models that have random effects for student,
instructor, and department (or the dept:service combination) to these data.
In the next chapter we will fit models incorporating fixed-effects for instructor
and department to these data.
> (fm05 <- lmer(y ~ 1 + (1|s) + (1|d)+(1|dept:service), InstEval, REML=FALSE))

Linear mixed model fit by maximum likelihood ['merMod']

Formula: y ~ 1 + (1 | s) + (1 | d) + (1 | dept:service)
Data: InstEval
AIC BIC logLik deviance
237663.3 237709.3 -118826.6 237653.3
Random effects:
Groups Name Variance Std.Dev.
s (Intercept) 0.10541 0.3247
d (Intercept) 0.26256 0.5124
dept:service (Intercept) 0.01213 0.1101
Residual 1.38495 1.1768

Page: 49 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

50 2 Models With Multiple Random-effects Terms

15:1 ●
8:1 ●
4:0 ●
7:0 ●
5:1 ●
11:0 ●
4:1 ●
10:1 ●
14:0 ●
9:0 ●
5:0 ●
6:0 ●
3:1 ●
8:0 ●
12:0 ●
1:1 ●
2:0 ●
3:0 ●
1:0 ●
12:1 ●
7:1 ●
15:0 ●
2:1 ●
14:1 ●
9:1 ●
10:0 ●
11:1 ●
6:1 ●

−0.5 0.0 0.5

Fig. 2.15 95% prediction intervals on the random effects for the dept:service factor
in model fm05 fit to the InstEval data.

Number of obs: 73421, groups: s, 2972; d, 1128; dept:service, 28

Fixed effects:
Estimate Std. Error t value
(Intercept) 3.25521 0.02824 115.3

(Fitting this complex model to a moderately large data set takes less than
two minutes on a modest laptop computer purchased in 2006. Although this
is more time than required for earlier model fits, it is a remarkably short time
for fitting a model of this size and complexity. In some ways it is remarkable
that such a model can be fit at all on such a computer.)
All three estimated standard deviations of the random effects are less than
σ
b , with σ b3 , the estimated standard deviation of the random effects for the
dept:service interaction, less than one-tenth the estimated residual standard
deviation.
It is not surprising that zero is within all of the prediction intervals on the
random effects for this factor (Fig.˜2.15). In fact, zero is close to the middle of
all these prediction intervals. However, the p-value for the LRT of H0 : σ3 = 0
versus Ha : σ3 > 0
> fm05a <- lmer(y ~ 1 + (1|s) + (1|d), InstEval, REML=0)
> anova(fm05a,fm05)

Data: InstEval
Models:
fm05a: y ~ 1 + (1 | s) + (1 | d)
fm05: y ~ 1 + (1 | s) + (1 | d) + (1 | dept:service)

Page: 50 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

2.3 A Model With Partially Crossed Random Effects 51

Fig. 2.16 Image of the sparse Cholesky factor, L, from model fm05

Df AIC BIC logLik Chisq Chi Df Pr(>Chisq)

fm05a 4 237786 237823 -118889
fm05 5 237663 237709 -118827 124.43 1 < 2.2e-16

is highly significant. That is, we have very strong evidence that we should
reject H0 in favor of Ha .
The seeming inconsistency of these conclusions is due to the large sample
size (n = 73421). When a model is fit to a very large sample even the most
subtle of differences can be highly “statistically significant”. The researcher
or data analyst must then decide if these terms have practical significance,
beyond the apparent statistical significance.
The large sample size also helps to assure that the parameters have good
normal approximations. We could profile this model fit but doing so would
take a very long time and, in this particular case, the analysts are more
interested in a model that uses fixed-effects parameters for the instructors,
which we will describe in the next chapter.
We could pursue other mixed-effects models here, such as using the dept
factor and not the dept:service interaction to define random effects, but we
will revisit these data in the next chapter and follow up on some of these
variations there.

Page: 51 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

52 2 Models With Multiple Random-effects Terms

2.3.2 Structure of L for model fm05

Before leaving this model we examine the sparse Cholesky factor, L, (Fig.˜2.16),
which is of size 4128 × 4128. Even as a sparse matrix this factor requires a
considerable amount of memory,
> object.size(fm05@re@L)

6904640 bytes

> unclass(round(object.size(fm05@re@L)/2^20, 3)) # size in megabytes

[1] 6.585

but as a triangular dense matrix it would require nearly 10 times as much.

There are (4128 × 4129)/2 elements on and below the diagonal, each of which
would require 8 bytes of storage. A packed lower triangular array would re-
quire
> (8 * (4128 * 4129)/2)/2^20 # size in megabytes

[1] 65.01965

megabytes. The more commonly used full rectangular storage requires

> (8 * 4128^2)/2^20 # size in megabytes

[1] 130.0078

megabytes of storage.
The number of nonzero elements in this matrix that must be updated for
each evaluation of the deviance is
> nnzero(as(fm05@re@L, "sparseMatrix"))

[1] 566960

Comparing this to 8522256, the number of elements that must be updated in

a dense Cholesky factor, we can see why the sparse Cholesky factor provides
a much more efficient evaluation of the profiled deviance function.

2.4 Chapter Summary

A simple, scalar random effects term in an lmer model formula is of the form
(1|fac), where fac is an expression whose value is the grouping factor of the
set of random effects generated by this term. Typically, fac is simply the name
of a factor, such as in the terms (1|sample) or (1|plate) in the examples in
this chapter. However, the grouping factor can be the value of an expression,
such as (1|dept:service) in the last example.

Page: 52 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

2.4 Chapter Summary 53

Because simple, scalar random-effects terms can differ only in the descrip-
tion of the grouping factor we refer to configurations such as crossed or nested
as applying to the terms or to the random effects, although it is more accurate
to refer to the configuration as applying to the grouping factors.
A model formula can include several such random effects terms. Because
configurations such as nested or crossed or partially crossed grouping factors
are a property of the data, the specification in the model formula does not
depend on the configuration. We simply include multiple random effects terms
in the formula specifying the model.
One apparent exception to this rule occurs with implicitly nested factors,
in which the levels of one factor are only meaningful within a particular level
of the other factor. In the Pastes data, levels of the cask factor are only
meaningful within a particular level of the batch factor. A model formula of
strength ~ 1 + (1 | cask) + (1 | batch)

would result in a fitted model that did not appropriately reflect the sources
of variability in the data. Following the simple rule that the factor should
be defined so that distinct experimental or observational units correspond to
distinct levels of the factor will avoid such ambiguity.
For convenience, a model with multiple, nested random-effects terms can
be specified as
strength ~ 1 + (1 | batch/cask)

which internally is re-expressed as

strength ~ 1 + (1 | batch) + (1 | batch:cask)

We will avoid terms of the form (1|batch/cask), preferring instead an ex-

plicit specification with simple, scalar terms based on unambiguous grouping
factors.
The InstEval data, described in Sec.˜2.3.1, illustrate some of the charac-
teristics of the real data to which mixed-effects models are now fit. There is a
large number of observations associated with several grouping factors; two of
which, student and instructor, have a large number of levels and are partially
crossed. Such data are common in sociological and educational studies but
until now it has been very difficult to fit models that appropriately reflect
such a structure. Much of the literature on mixed-effects models leaves the
impression that multiple random effects terms can only be associated with
nested grouping factors. The resulting emphasis on hierarchical or multilevel
configurations is an artifact of the computational methods used to fit the
models, not the models themselves.
The parameters of the models fit to small data sets have properties similar
to those for the models in the previous chapter. That is, profile-based confi-
dence intervals on the fixed-effects parameter, β0 , are symmetric about the
estimate but overdispersed relative to those that would be calculated from
a normal distribution and the logarithm of the residual standard deviation,
log(σ ), has a good normal approximation. Profile-based confidence intervals

Page: 53 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

54 2 Models With Multiple Random-effects Terms

for the standard deviations of random effects (σ1 , σ2 , etc.) are symmetric on
a logarithmic scale except for those that could be zero.
Another observation from the last example is that, for data sets with a
very large numbers of observations, a term in a model may be “statistically
significant” even when its practical significance is questionable.

Exercises

These exercises use data sets from the MEMSS package for R. Recall that to
access a particular data set, you must either attach the package
> library(MEMSS)

or load just the one data set

> data(ergoStool, package = "MEMSS")

We begin with exercises using the ergoStool data from the MEMSS package.
The analysis and graphics in these exercises is performed in Chap.˜4. The
purpose of these exercises is to see if you can use the material from this
chapter to anticipate the results quoted in the next chapter.

2.1. Check the documentation, the structure (str) and a summary of the
ergoStool data from the MEMSS package. (If you are familiar with the Star
Trek television series and movies, you may want to speculate about what,
exactly, the “Borg scale” is.) Use
> xtabs(~ Type + Subject, ergoStool)

to determine if these factors are nested, partially crossed or completely

crossed. Is this a replicated or an unreplicated design?

2.2. Create a plot, similar to Fig.˜2.1, showing the effort by subject with lines
connecting points corresponding to the same stool types. Order the levels of
the Subject factor by increasing average effort.

2.3. The experimenters are interested in comparing these specific stool types.
In the next chapter we will fit a model with fixed-effects for the Type factor
and random effects for Subject, allowing us to perform comparisons of these
specific types. At this point fit a model with random effects for both Type
and Subject. What are the relative sizes of the estimates of the standard
deviations, σ
b1 (for Subject), σ
b2 (for Type) and σ
b (for the residual variability)?

2.4. Refit the model using maximum likelihood. Check the parameter esti-
mates and, in the case of the fixed-effects parameter, β0 , its standard error.
In what ways have the parameter estimates changed? Which parameter esti-
mates have not changed?

Page: 54 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

2.4 Chapter Summary 55

2.5. Profile the fitted model and construct 95% profile-based confidence in-
tervals on the parameters. (Note that you will get the same profile object
whether you start with the REML fit or the ML fit. There is a slight advan-
tage in starting with the ML fit.) Is the confidence interval on σ1 close to
being symmetric about its estimate? Is the confidence interval on σ2 close to
being symmetric about its estimate? Is the corresponding interval on log(σ1 )
close to being symmetric about its estimate?

2.6. Create the profile zeta plot for this model. For which parameters are
there good normal approximations?

2.7. Create a profile pairs plot for this model. Comment on the shapes of the
profile traces in the transformed (ζ ) scale and the shapes of the contours in
the original scales of the parameters.

2.8. Create a plot of the 95% prediction intervals on the random effects for
Type using
> dotplot(ranef(fm, which = "Type", postVar = TRUE), aspect = 0.2,
+ strip = FALSE)

(Substitute the name of your fitted model for fm in the call to ranef.) Is there
a clear winner among the stool types? (Assume that lower numbers on the
Borg scale correspond to less effort).

2.9. Create a plot of the 95% prediction intervals on the random effects for
Subject.

2.10. Check the documentation, the structure (str) and a summary of the
Meat data from the MEMSS package. Use a cross-tabulation to discover whether
Pair and Block are nested, partially crossed or completely crossed.

2.11. Use a cross-tabulation

> xtabs(~ Pair + Storage, Meat)

to determine whether Pair and Storage are nested, partially crossed or com-
pletely crossed.

2.12. Fit a model of the score in the Meat data with random effects for Pair,
Storage and Block.

2.13. Plot the prediction intervals for each of the three sets of random effects.

2.14. Profile the parameters in this model. Create a profile zeta plot. Does
including the random effect for Block appear to be warranted. Does your con-
clusion from the profile zeta plot agree with your conclusion from examining
the prediction intervals for the random effects for Block?

Page: 55 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

56 2 Models With Multiple Random-effects Terms

2.15. Refit the model without random effects for Block. Perform a likelihood
ratio test of H0 : σ3 = 0 versus Ha : σ3 > 0. Would you reject H0 in favor of
Ha or fail to reject H0 ? Would you reach the same conclusion if you adjusted
the p-value for the test by halving it, to take into account the fact that 0 is
on the boundary of the parameter region?

2.16. Profile the reduced model (i.e. the one without random effects for Block)
and create profile zeta and profile pairs plots. Can you explain the apparent
interaction between log(σ ) and σ1 ? (This is a difficult question.)

Page: 56 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

Chapter 3
Models for Longitudinal Data

Longitudinal data consist of repeated measurements on the same subject

(or some other “experimental unit”) taken over time. Generally we wish to
characterize the time trends within subjects and between subjects. The data
will always include the response, the time covariate and the indicator of the
subject on which the measurement has been made. If other covariates are
recorded, say whether the subject is in the treatment group or the control
group, we may wish to relate the within- and between-subject trends to such
covariates.
In this chapter we introduce graphical and statistical techniques for the
analysis of longitudinal data by applying them to a simple example.

3.1 The sleepstudy Data

Belenky et˜al. [2003] report on a study of the effects of sleep deprivation on

reaction time for a number of subjects chosen from a population of long-
distance truck drivers. These subjects were divided into groups that were
allowed only a limited amount of sleep each night. We consider here the
group of 18 subjects who were restricted to three hours of sleep per night
for the first ten days of the trial. Each subject’s reaction time was measured
several times on each day of the trial.
> str(sleepstudy)
'data.frame': 180 obs. of 3 variables:
$ Reaction: num 250 259 251 321 357 ...
$ Days : num 0 1 2 3 4 5 6 7 8 9 ...
$ Subject : Factor w/ 18 levels "308","309","310",..: 1 1 1 1 1 1 1 1 1..

In this data frame, the response variable Reaction, is the average of the
reaction time measurements on a given subject for a given day. The two
covariates are Days, the number of days of sleep deprivation, and Subject, the
identifier of the subject on which the observation was made.

57
58 3 Models for Longitudinal Data

0 2 4 6 8 0 2 4 6 8 0 2 4 6 8

372 333 352 331 330 337

●●
450
●
400 ●
● ●
●● ● ●
● ● ●
350 ● ● ●●●● ● ●
●● ●
●● ● ● ●
●● ● ●●
● ●
300 ● ●● ● ● ●● ● ● ●
● ●●● ●● ●●● ●
●
●●
250
●
200
308 371 369 351 335 332
Average reaction time (ms)

●
● 450
●
●
400
●
● ●
● ● ●● ● ● 350
●
● ●
●
●●
● ●●●
● ● ●
●
300
●●● ●
●●● ●● ● ● ● ● ● ● ●
●● ● ● ● ●●● ●
● ●
● ●●● ●●
250
200
310 309 370 349 350 334
450
400 ●●
● ●
●● ●
350 ● ● ●
● ● ●●
● ●
300
● ● ● ●●
● ● ● ●
●●● ●● ● ●● ●
250 ●●● ● ● ●●● ●●● ● ●
● ● ●● ●
●●● ●●●
200 ●●

0 2 4 6 8 0 2 4 6 8 0 2 4 6 8

Days of sleep deprivation

Fig. 3.1 A lattice plot of the average reaction time versus number of days of sleep
deprivation by subject for the sleepstudy data. Each subject’s data are shown in a
separate panel, along with a simple linear regression line fit to the data in that panel.
The panels are ordered, from left to right along rows starting at the bottom row, by
increasing intercept of these per-subject linear regression lines. The subject number
is given in the strip above the panel.

As recommended for any statistical analysis, we begin by plotting the data.

The most important relationship to plot for longitudinal data on multiple sub-
jects is the trend of the response over time by subject, as shown in Fig.˜3.1.
This plot, in which the data for different subjects are shown in separate pan-
els with the axes held constant for all the panels, allows for examination of
the time-trends within subjects and for comparison of these patterns between
subjects. Through the use of small panels in a repeating pattern Fig.˜3.1 con-
veys a great deal of information, the individual time trends for 18 subjects
over 10 days — a total of 180 points — without being overly cluttered.

Page: 58 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

3.1 The sleepstudy Data 59

3.1.1 Characteristics of the sleepstudy Data Plot

The principles of “Trellis graphics”, developed by Bill Cleveland and his

coworkers at Bell Labs and implemented in the lattice package for R by
Deepayan Sarkar, have been incorporated in this plot. As stated above, all
the panels have the same vertical and horizontal scales, allowing us to eval-
uate the pattern over time for each subject and also to compare patterns
between subjects. The line drawn in each panel is a simple least squares line
fit to the data in that panel only. It is provided to enhance our ability to
discern patterns in both the slope (the typical change in reaction time per
day of sleep deprivation for that particular subject) and the intercept (the
average response time for the subject when on their usual sleep pattern).
The aspect ratio of the panels (ratio of the height to the width) has been
chosen, according to an algorithm described in Cleveland [1993], to facilitate
comparison of slopes. The effect of choosing the aspect ratio in this way is
to have the slopes of the lines on the page distributed around ±45◦ , thereby
making it easier to detect systematic changes in slopes.
The panels have been ordered (from left to right starting at the bottom
row) by increasing intercept. Because the subject identifiers, shown in the
strip above each panel, are unrelated to the response it would not be helpful to
use the default ordering of the panels, which is by increasing subject number.
If we did so our perception of patterns in the data would be confused by the,
essentially random, ordering of the panels. Instead we use a characteristic
of the data to determine the ordering of the panels, thereby enhancing our
ability to compare across panels. For example, a question of interest to the
experimenters is whether a subject’s rate of change in reaction time is related
to the subject’s initial reaction time. If this were the case we would expect
that the slopes would show an increasing trend (or, less likely, a decreasing
trend) in the left to right, bottom to top ordering.
There is little evidence in Fig.˜3.1 of such a systematic relationship be-
tween the subject’s initial reaction time and their rate of change in reaction
time per day of sleep deprivation. We do see that for each subject, except
335, reaction time increases, more-or-less linearly, with days of sleep depri-
vation. However, there is considerable variation both in the initial reaction
time and in the daily rate of increase in reaction time. We can also see that
these data are balanced, both with respect to the number of observations on
each subject, and with respect to the times at which these observations were
taken. This can be confirmed by cross-tabulating Subject and Days.
> xtabs(~ Subject + Days, sleepstudy)

Days
Subject 0 1 2 3 4 5 6 7 8 9
308 1 1 1 1 1 1 1 1 1 1
309 1 1 1 1 1 1 1 1 1 1
310 1 1 1 1 1 1 1 1 1 1
330 1 1 1 1 1 1 1 1 1 1

Page: 59 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

60 3 Models for Longitudinal Data

331 1 1 1 1 1 1 1 1 1 1
332 1 1 1 1 1 1 1 1 1 1
333 1 1 1 1 1 1 1 1 1 1
334 1 1 1 1 1 1 1 1 1 1
335 1 1 1 1 1 1 1 1 1 1
337 1 1 1 1 1 1 1 1 1 1
349 1 1 1 1 1 1 1 1 1 1
350 1 1 1 1 1 1 1 1 1 1
351 1 1 1 1 1 1 1 1 1 1
352 1 1 1 1 1 1 1 1 1 1
369 1 1 1 1 1 1 1 1 1 1
370 1 1 1 1 1 1 1 1 1 1
371 1 1 1 1 1 1 1 1 1 1
372 1 1 1 1 1 1 1 1 1 1

In cases like this where there are several observations (10) per subject
and a relatively simple within-subject pattern (more-or-less linear) we may
want to examine coefficients from within-subject fixed-effects fits. However,
because the subjects constitute a sample from the population of interest and
we wish to drawn conclusions about typical patterns in the population and
the subject-to-subject variability of these patterns, we will eventually want
to fit mixed models and we begin by doing so. In section˜3.4 we will com-
pare estimates from a mixed-effects model with those from the within-subject
fixed-effects fits.

3.2 Mixed-effects Models For the sleepstudy Data

Based on our preliminary graphical exploration of these data, we fit a mixed-

effects model with two fixed-effects parameters, the intercept and slope of the
linear time trend for the population, and two random effects for each subject.
The random effects for a particular subject are the deviations in intercept and
slope of that subject’s time trend from the population values.
We will fit two linear mixed models to these data. One model, fm06, allows
for correlation (in the unconditional distribution) of the random effects for
the same subject. That is, we allow for the possibility that, for example,
subjects with higher initial reaction times may, on average, be more strongly
affected by sleep deprivation. The second model provides independent (again,
in the unconditional distribution) random effects for intercept and slope for
each subject.

3.2.1 A Model With Correlated Random Effects

The first model is fit as

Page: 60 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

3.2 Mixed-effects Models For the sleepstudy Data 61

> (fm06 <- lmer(Reaction ~ 1 + Days + (1 + Days|Subject), sleepstudy,

+ REML=FALSE))

Linear mixed model fit by maximum likelihood ['merMod']

Formula: Reaction ~ 1 + Days + (1 + Days | Subject)
Data: sleepstudy
AIC BIC logLik deviance
1763.9393 1783.0971 -875.9697 1751.9393
Random effects:
Groups Name Variance Std.Dev. Corr
Subject (Intercept) 565.52 23.781
Days 32.68 5.717 0.081
Residual 654.94 25.592
Number of obs: 180, groups: Subject, 18

Fixed effects:
Estimate Std. Error t value
(Intercept) 251.405 6.632 37.91
Days 10.467 1.502 6.97

Correlation of Fixed Effects:

(Intr)
Days -0.138

From the display we see that this model incorporates both an intercept and
a slope (with respect to Days) in the fixed effects and in the random effects.
Extracting the conditional modes of the random effects
> head(ranef(fm06)[["Subject"]])

(Intercept) Days
308 2.815683 9.0755341
309 -40.048489 -8.6440673
310 -38.433155 -5.5133788
330 22.832297 -4.6587506
331 21.549991 -2.9445202
332 8.815587 -0.2352092

confirms that these are vector-valued random effects. There are a total of
q = 36 random effects, two for each of the 18 subjects.
The random effects section of the model display,
Groups Name Variance Std.Dev. Corr
Subject (Intercept) 565.52 23.781
Days 32.68 5.717 0.081
Residual 654.94 25.592

indicates that there will be a random effect for the intercept and a random
effect for the slope with respect to Days at each level of Subject and, further-
more, the unconditional distribution of these random effects, B ∼ N (0, Σ ),
allows for correlation of the random effects for the same subject.
We can confirm the potential for correlation of random effects within sub-
ject in the images of Λ , Σ and L for this model (Fig.˜3.2). The matrix Λ has

Page: 61 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

62 3 Models for Longitudinal Data

10 10 10

20 20 20

30 30 30

10 20 30 10 20 30 10 20 30

Λ Σ L

Fig. 3.2 Images of Λ , Σ and L for model fm06

18 triangular blocks of size 2 along the diagonal, generating 18 square, sym-

metric blocks of size 2 along the diagonal of Σ . The 18 symmetric blocks on
the diagonal of Σ are identical. Overall we estimate two standard deviations
and a correlation for a vector-valued random effect of size 2, as shown in the
model summary.
Often the variances and the covariance of random effects are quoted, rather
than the standard deviations and the correlation shown here. We have already
seen that the variance of a random effect is a poor scale on which to quote the
estimate because confidence intervals on the variance are so badly skewed. It
is more sensible to assess the estimates of the standard deviations of random
effects or, possibly, the logarithms of the standard deviations if we can be
confident that 0 is outside the region of interest. We do display the estimates
of the variances of the random effects but mostly so that the user can compare
these estimates to those from other software or for cases where an estimate of
a variance is expected (sometimes even required) to be given when reporting
a mixed model fit.
We do not quote estimates of covariances of vector-valued random effects
because the covariance is a difficult scale to interpret, whereas a correlation
has a fixed scale. A correlation must be between −1 and 1, allowing us to
conclude that a correlation estimate close to those extremes indicates that Σ
is close to singular and the model is not well formulated.
The estimates of the fixed effects parameters are βb = (251.41, 10.467)T .
These represent a typical initial reaction time (i.e. without sleep deprivation)
in the population of about 250 milliseconds, or 1/4 sec., and a typical in-
crease in reaction time of a little more than 10 milliseconds per day of sleep
deprivation.
The estimated subject-to-subject variation in the intercept corresponds
to a standard deviation of about 25 ms. A 95% prediction interval on this
random variable would be approximately ±50 ms. Combining this range with
a population estimated intercept of 250 ms. indicates that we should not be

Page: 62 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

3.2 Mixed-effects Models For the sleepstudy Data 63

surprised by intercepts as low as 200 ms. or as high as 300 ms. This range is
consistent with the reference lines shown in Figure˜3.1.
Similarly, the estimated subject-to-subject variation in the slope corre-
sponds to a standard deviation of about 5.7 ms./day so we would not be
surprised by slopes as low as 10.5 − 2 · 5.7 = −0.9 ms./day or as high as
10.5+2·5.7 = 21.9 ms./day. Again, the conclusions from these rough, “back of
the envelope” calculations are consistent with our observations from Fig.˜3.1.
The estimated residual standard deviation is about 25 ms. leading us to
expect a scatter around the fitted lines for each subject of up to ±50 ms.
From Figure˜3.1 we can see that some subjects (309, 372 and 337) appear
to have less variation than ±50 ms. about their within-subject fit but others
(308, 332 and 331) may have more.
Finally, we see the estimated within-subject correlation of the random ef-
fect for the intercept and the random effect for the slope is very low, 0.081,
confirming our impression that there is little evidence of a systematic rela-
tionship between these quantities. In other words, observing a subject’s initial
reaction time does not give us much information for predicting whether their
reaction time will be strongly affected by each day of sleep deprivation or
not. It seems reasonable that we could get nearly as good a fit from a model
that does not allow for correlation, which we describe next.

3.2.2 A Model With Uncorrelated Random Effects

In a model with uncorrelated random effects we have B ∼ N (0, Σ ) where

Σ is diagonal. We have seen models like this in previous chapters but those
models had simple scalar random effects for all the grouping factors. Here we
want to have a simple scalar random effect for Subject and a random effect
for the slope with respect to Days, also indexed by Subject. We accomplish
this by specifying two random-effects terms. The first, (1|Subject), is a simple
scalar term. The second has Days on the left hand side of the vertical bar.
It may seem that the model formula we want should be
Reaction ~ 1 + Days + (1 | Subject) + (Days | Subject)

but it is not. Because the intercept is implicit in linear models, the second ran-
dom effects term is equivalent to (1+Days|Subject) and will, by itself, produce
correlated, vector-valued random effects.
We must suppress the implicit intercept in the second random-effects term,
which we do by writing it as (0+Days|Subject), read as “no intercept and
Days by Subject”. An alternative expression for Days without an intercept by
Subject is (Days - 1 | Subject). Using the first form we have
> (fm07 <- lmer(Reaction ~ 1 + Days + (1|Subject) + (0+Days|Subject),
+ sleepstudy, REML=FALSE))

Page: 63 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

64 3 Models for Longitudinal Data

Linear mixed model fit by maximum likelihood ['merMod']

Formula: Reaction ~ 1 + Days + (1 | Subject) + (0 + Days | Subject)
Data: sleepstudy
AIC BIC logLik deviance
1762.0033 1777.9680 -876.0016 1752.0033
Random effects:
Groups Name Variance Std.Dev.
Subject (Intercept) 584.25 24.171
Subject Days 33.63 5.799
Residual 653.12 25.556
Number of obs: 180, groups: Subject, 18

Fixed effects:
Estimate Std. Error t value
(Intercept) 251.405 6.708 37.48
Days 10.467 1.519 6.89

Correlation of Fixed Effects:

(Intr)
Days -0.194

As in model fm06, there are two random effects for each subject
> head(ranef(fm07)[["Subject"]])
(Intercept) Days
308 1.854656 9.2364346
309 -40.022307 -8.6174730
310 -38.723163 -5.4343801
330 23.903319 -4.8581939
331 22.396321 -3.1048404
332 9.052001 -0.2821598

but no correlation has been estimated

Groups Name Variance Std.Dev.
Subject (Intercept) 584.25 24.171
Subject Days 33.63 5.799
Residual 653.12 25.556
The Subject factor is repeated in the “Groups” column because there were
two distinct terms generating these random effects and these two terms had
the same grouping factor.
Images of the matrices Λ , Σ and L (Fig.˜3.3) show that Σ is indeed diago-
nal. The order of the random effects in Σ and Λ for model fm07 is different from
the order in model fm06. In model fm06 the two random effects for a particular
subject were adjacent. In model fm07 all the intercept random effects occur
first then all the Days random effects. The sparse Cholesky decomposition,
L, has the same form in both models because the fill-reducing permutation
(described in Sect.˜5.4.1) calculated for model fm07 provides a post-ordering
to group random effects with similar structure in Z.
Images of ZT for these two models (Fig.˜3.4) shows that the columns of Z
(rows of ZT ) from one model are the same those from the other model but
in a different order.

Page: 64 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

3.2 Mixed-effects Models For the sleepstudy Data 65

10 10 10

20 20 20

30 30 30

10 20 30 10 20 30 10 20 30

Λ Σ L

Fig. 3.3 Images of Λ , the relative covariance factor, Σ , the variance-covariance ma-
trix of the random effects, and L, the sparse Cholesky factor, in model fm07

10

20

30

10

20

30

Fig. 3.4 Images of ZT for models fm06 (upper panel) and fm07 (lower panel)

3.2.3 Generating Z and Λ From Random-effects Terms

Let us consider these columns in more detail, starting with the columns of Z
for model fm07. The first 18 columns (rows in the bottom panel of Fig.˜3.4)
are the indicator columns for the Subject factor, as we would expect from the
simple, scalar random-effects term (1|Subject). The pattern of zeros and non-
zeros in the second group of 18 columns is determined by the indicators of the
grouping factor, Subject, and the values of the non-zeros are determined by
the Days factor. In other words, these columns are formed by the interaction
of the numeric covariate, Days, and the categorical covariate, Subject.
The non-zero values in the model matrix Z for model fm06 are the same
as those for model fm06 but the columns are in a different order. Pairs of
columns associated with the same level of the grouping factor are adjacent.
One way to think of the process of generating these columns is to extend the

Page: 65 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

66 3 Models for Longitudinal Data

idea of an interaction between a single covariate and the grouping factor to

generating an “interaction” of a model matrix and the levels of the grouping
factor. In other words, we begin with the two columns of the model matrix
for the expression 1 + Days and the 18 columns of indicators for the Subject
factor. The result will have 36 columns that we regard as 18 adjacent pairs.
The values within each of these pairs of columns are the values of the 1 +
Days columns, when the indicator is 1, otherwise they are zero.
We can now describe the general process of creating the model matrix,
Z, and the relative covariance factor, Λ from the random-effects terms in
the model formula. Each random-effects term is of the form (expr|fac). The
expression expr is evaluated as a linear model formula, producing a model
matrix with s columns. The expression fac is evaluated as a factor. Let k
be the number of levels in this factor, after eliminating unused levels, if any.
The ith term generates si ki columns in the model matrix, Z, and a diagonal
block of size si ki in the relative covariance factor, Λ . The si ki columns in Z
have the pattern of the interaction of the si columns from the ith expr with
the ki indicator columns for the ith grouping factor fac. The diagonal block
in Λ is itself block diagonal, consisting of ki blocks, each a lower triangular
matrix of size si . In fact, these inner blocks are repetitions of the same lower
triangular si × si matrix. The i term contributes si (si + 1)/2 elements to the
variance-component parameter, θ , and these are the elements in the lower
triangle of this si × si template matrix.
Note that when counting the columns in a model matrix we must take into
account the implicit intercept term. For example, we could write the formula
for model fm06 as
Reaction ~ Days + (Days | Subject)

realizing that the linear model expression, Days, actually generates two
columns because of the implicit intercept.
Whether or not to include an explicit intercept term in a model formula
is a matter of personal taste. Many people prefer to write the intercept ex-
plicitly so as to emphasize the relationship between terms in the formula and
coefficients or random effects in the model. Others omit these implicit terms
so as to economize on the amount of typing required. Either approach can
be used. The important point to remember is that the intercept must be
explicitly suppressed when you don’t want it in a term.
Also, the intercept term must be explicit when it is the only term in the
expression. That is, a simple, scalar random-effects term must be written as
(1|fac) because a term like (|fac) is not syntactically correct. However, we
can omit the intercept from the fixed-effects part of the model formula if we
have any random-effects terms. That is, we could write the formula for model
fm1 in Chap.˜1 as
Yield ~ (1 | Batch)

or even

Page: 66 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

3.3 Assessing the Precision of the Parameter Estimates 67

Yield ~ 1 | Batch
although omitting the parentheses around a random-effects term is risky.
Because of operator precedence, the vertical bar operator, |, takes essentially
everything in the expression to the left of it as its first operand. It is advisable
always to enclose such terms in parentheses so the scope of the operands to
the | operator is clearly defined.

3.2.4 Comparing Models fm07 and fm06

Returning to models fm06 and fm07 for the sleepstudy data, it is easy to see
that these are nested models because fm06 is reduced to fm07 by constraining
the within-group correlation of random effects to be zero (which is equivalent
to constraining the element below the diagonal in the 2 × 2 lower triangular
blocks of Λ in Fig.˜3.2 to be zero).
We can use a likelihood ratio test to compare these fitted models.
> anova(fm07, fm06)
Data: sleepstudy
Models:
fm07: Reaction ~ 1 + Days + (1 | Subject) + (0 + Days | Subject)
fm06: Reaction ~ 1 + Days + (1 + Days | Subject)
Df AIC BIC logLik Chisq Chi Df Pr(>Chisq)
fm07 5 1762.0 1778.0 -876.00
fm06 6 1763.9 1783.1 -875.97 0.0639 1 0.8004

The value of the χ 2 statistic, 0.0639, is very small, corresponding to a p-value

of 0.80 and indicating that the extra parameter in model fm06 relative to fm07
does not produce a significantly better fit. By the principal of parsimony we
prefer the reduced model, fm07.
This conclusion is consistent with the visual impression provided by
Fig.˜3.1. There does not appear to be a strong relationship between a sub-
ject’s initial reaction time and the extent to which his or her reaction time is
affected by sleep deprivation.
In this likelihood ratio test the value of the parameter being tested, a
correlation of zero, is not on the boundary of the parameter space. We can
be confident that the p-value from the LRT adequately reflects the underlying
situation.
(Note: It is possible to extend profiling to the correlation parameters and
we will do so but that has not been done yet.)

3.3 Assessing the Precision of the Parameter Estimates

Plots of the profile ζ for the parameters in model fm07 (Fig.˜3.5) show that

Page: 67 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

68 3 Models for Longitudinal Data

.sig01 .lsig (Intercept) TypeT2 TypeT3

0
ζ

−1

−2

0.5 1.0 1.5 2.0 2.5 −0.2 0.0 0.2 0.4 7 8 9 10 2.5 3.0 3.5 4.0 4.5 5.0 1.0 1.5 2.0 2.5 3.0 3.5

Fig. 3.5 Profile zeta plot for each of the parameters in model fm07. The vertical lines
are the endpoints of 50%, 80%, 90%, 95% and 99% profile-based confidence intervals
for each parameter.

confidence intervals on σ1 and σ2 will be slightly skewed; those for log(σ ) will
be symmetric and well-approximated by methods based on quantiles of the
standard normal distribution and those for the fixed-effects parameters, β1
and β2 will be symmetric and slightly over-dispersed relative to the standard
normal. For example, the 95% profile-based confidence intervals are
> confint(pr07)

2.5 % 97.5 %
.sig01 0.7342443 2.2872576
.lsig -0.2082812 0.3293787
(Intercept) 7.4238425 9.6872687
TypeT2 2.8953043 4.8824734
TypeT3 1.2286377 3.2158068
TypeT4 -0.3269179 1.6602512

The profile pairs plot (Fig.˜3.6) shows, for the most part, the usual pat-
terns. First, consider the panels below the diagonal, which are on the (ζi , ζ j )
scales. The ζ pairs for log(σ ) and β0 , in the (4, 3) panel, and for log(σ ) and
β1 , in the (5, 3) panel, show the ideal pattern. The profile traces are straight
and orthogonal, producing interpolated contours on the ζ scale that are con-
centric circles centered at the origin. When mapped back to the scales of
log(σ ) and β0 or β1 , in panels (3, 4) and (3, 5), these circles become slightly
distorted, but this is only due to the moderate nonlinearity in the profile ζ
plots for these parameters.
Examining the profile traces on the ζ scale for log(σ ) versus σ1 , the (3, 1)
panel, or versus σ2 , the (3, 2) panel, and for σ1 versus σ2 , the (2, 1) panel,
we see that close to the estimate the traces are orthogonal but as one vari-
ance component becomes small there is usually an increase in the others.
In some sense the total variability in the response will be partitioned across
the contribution of the fixed effects and the variance components. In each of

Page: 68 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

3.3 Assessing the Precision of the Parameter Estimates 69

1
TypeT4
0
0 1 2 3
−1
2 3
3

2 TypeT3

1 0 1 2 3

5 4 5

4 TypeT2
3
0 1 2 3

10 9 10

9
(Intercept)
8
7 0 1 2 3

0.4 0.2 0.4

0.2
.lsig
0.0
−0.2 0 1 2 3

0.5 1.5 2.5 3.5

.sig01 0
−1
−2
−3
Scatter Plot Matrix

Fig. 3.6 Profile pairs plot for the parameters in model fm07. The contour lines
correspond to marginal 50%, 80%, 90%, 95% and 99% confidence regions based on
the likelihood ratio. Panels below the diagonal represent the (ζi , ζ j ) parameters; those
above the diagonal represent the original parameters.

these panels the fixed-effects parameters are at their optimal values, condi-
tional on the values of the variance components, and the variance components
must compensate for each other. If one is made smaller then the others must
become larger to compensate.
The patterns in the (4, 1) panel (σ1 versus β0 , on the ζ scale) and the (5, 2)
panel (σ2 versus β1 , on the ζ scale) are what we have come to expect. As the
fixed-effects parameter is moved from its estimate, the standard deviation
of the corresponding random effect increases to compensate. The (5, 1) and
(4, 2) panels show that changing the value of a fixed effect doesn’t change the
estimate of the standard deviation of the random effects corresponding to

Page: 69 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

70 3 Models for Longitudinal Data

the other fixed effect, which makes sense although the perfect orthogonality
shown here will probably not be exhibited in models fit to unbalanced data.
In some ways the most interesting panels are those for the pair of fixed-
effects parameters: (5, 4) on the ζ scale and (4, 5) on the original scale. The
traces are not orthogonal. In fact the slopes of the traces at the origin of
the (5, 4) (ζ scale) panel are the correlation of the fixed-effects estimators
(−0.194 for this model) and its inverse. However, as we move away from
the origin on one of the traces in the (5, 4) panel it curves back toward the
horizontal axis (for the horizontal trace) or the vertical axis (for the vertical
trace). In the ζ scale the individual contours are still concentric ellipses but
their eccentricity changes from contour to contour. The innermost contours
have greater eccentricity than the outermost contours. That is, the outermost
contours are more like circles than are the innermost contours.
In a fixed-effects model the shapes of projections of deviance contours onto
pairs of fixed-effects parameters are consistent. In a fixed-effects model the
profile traces in the original scale will always be straight lines. For mixed
models these traces can fail to be linear, as we see here, contradicting the
widely-held belief that inferences for the fixed-effects parameters in linear
mixed models, based on T or F distributions with suitably adjusted degrees
of freedom, will be completely accurate. The actual patterns of deviance
contours are more complex than that.

3.4 Examining the Random Effects and Predictions

The result of applying ranef to fitted linear mixed model is a list of data
frames. The components of the list correspond to the grouping factors in the
random-effects terms, not to the terms themselves. Model fm07 is the first
model we have fit with more than one term for the same grouping factor
where we can see the combination of random effects from more than one
term.
> str(rr1 <- ranef(fm07))

List of 1
$ Subject:'data.frame': 18 obs. of 2 variables:
..$ (Intercept): num [1:18] 1.85 -40.02 -38.72 23.9 22.4 ...
..$ Days : num [1:18] 9.24 -8.62 -5.43 -4.86 -3.1 ...
- attr(*, "class")= chr "ranef.mer"

The plot method for "ranef.mer" objects produces one plot for each grouping
factor. For scalar random effects the plot is a normal probability plot. For
two-dimensional random effects, including the case of two scalar terms for the
same grouping factor, as in this model, the plot is a scatterplot. For three or
more random effects per level of the grouping factor, the plot is a scatterplot
matrix. The left hand panel in Fig.˜3.7 was created with plot(ranef(fm07)).

Page: 70 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

3.4 Examining the Random Effects and Predictions 71

● ●

280
● ●
● ●
20 ● ●
● ●
● ●
● 260 ●
(Intercept)

(Intercept)
● ●
● ● ● ●
0 ● ● ● ●

● ●

● 240 ●

−20
● ● ● ●

220
● ●
−40 ● ●

−10 −5 0 5 10 0 5 10 15 20

Days Days

Fig. 3.7 Plot of the conditional modes of the random effects for model fm07 (left
panel) and the corresponding subject-specific coefficients (right panel)

The coef method for a fitted lmer model combines the fixed-effects esti-
mates and the conditional modes of the random effects, whenever the column
names of the random effects correspond to the names of coefficients. For model
fm07 the fixed-effects coefficients are (Intercept) and Days and the columns
of the random effects match these names. Thus we can calculate some kind
of per-subject “estimates” of the slope and intercept and plot them, as in the
right hand panel of Fig.˜3.7. By comparing the two panels in Fig.˜3.7 we can
see that the result of the coef method is simply the conditional modes of the
random effects shifted by the coefficient estimates.
It is not entirely clear how we should interpret these values. They are a
combination of parameter estimates with the modal values of random vari-
ables and, as such, are in a type of “no man’s land” in the probability model.
(In the Bayesian approach˜[Box and Tiao, 1973] to inference, however, both
the parameters and the random effects are random variables and the inter-
pretation of these values is straightforward.) Despite the difficulties of inter-
pretation in the probability model, these values are of interest because they
determine the fitted response for each subject.
Because responses for each individual are recorded on each of ten days
we can determine the within-subject estimates of the slope and intercept
(that is, the slope and intercept of each of the lines in Fig.˜3.1). In Fig.˜3.8
we compare the within-subject least squares estimates to the per-subject
slope and intercept calculated from model fm07. We see that, in general, the
per-subject slopes and intercepts from the mixed-effects model are closer to
the population estimates than are the within-subject least squares estimates.
This pattern is sometimes described as a shrinkage of coefficients toward the
population values.

Page: 71 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

72 3 Models for Longitudinal Data

Mixed model ● Within−group Population

330 337
331
●

280
333 352
●
●
●

● 372

335 332
351 ●

260 ●

● 371 369
●
●
(Intercept)

● ●

240
308
●
334

● ●
350
220

●
● 349
370
309
200 310

0 5 10 15 20

Days

Fig. 3.8 Comparison of the within-subject estimates of the intercept and slope for
each subject and the conditional modes of the per-subject intercept and slope. Each
pair of points joined by an arrow are the within-subject estimates and conditional
modes of the random for a particular subject. The arrow points from the within-
subject estimate to the conditional mode for the mixed-effects model. The subject
identifier number is at the head of each arrow.

The term “shrinkage” may have negative connotations. John Tukey chose
to characterize this process in terms of the estimates for individual subjects
“borrowing strength” from each other. This is a fundamental difference in the
models underlying mixed-effects models versus strictly fixed-effects models.
In a mixed-effects model we assume that the levels of a grouping factor are a
selection from a population and, as a result, can be expected to share charac-
teristics to some degree. Consequently, the predictions from a mixed-effects
model are attenuated relative to those from strictly fixed-effects models.
The predictions from model fm07 and from the within-subject least squares
fits for each subject are shown in Fig.˜3.9. In may seem that the shrinkage

Page: 72 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

3.4 Examining the Random Effects and Predictions 73

Within−subject Mixed model Population

0 2 4 6 8 0 2 4 6 8 0 2 4 6 8

372 333 352 331 330 337

●●
450
●
400 ●
● ●
●● ● ●
● ●
350 ● ●● ●●● ● ●
● ●● ●
●● ● ● ●
●● ● ●●
● ● ●
300 ●
● ●●●
●● ●
●● ●● ● ●
●●●
● ●
●
●●
250
●
200
Average reaction time (ms)

308 371 369 351 335 332

●
● 450
●
●
400
●
● ●
● ● ●● ● ● 350
● ●
●
●
●●
● ●●●
● ● ●
●
300
●●● ●●
●●● ●● ● ● ● ● ● ●
●● ● ● ● ●●● ●
● ●
● ●●● ●●
250
200
310 309 370 349 350 334
450
400 ●●
● ●
●● ●
350 ● ● ●
● ● ●●
● ●
300
● ● ● ●●
● ● ● ●
●●● ●● ● ●● ●
250 ●●● ● ● ●●● ●●● ● ●
●
● ● ●●●●
200 ●● ●●●●

0 2 4 6 8 0 2 4 6 8 0 2 4 6 8

Days of sleep deprivation

Fig. 3.9 Comparison of the predictions from the within-subject fits with those from
the conditional modes of the subject-specific parameters in the mixed-effects model.

from the per-subject estimates toward the population estimates depends only
on how far the per-subject estimates (solid lines) are from the population es-
timates (dot-dashed lines). However, careful examination of this figure shows
that there is more at work here than a simple shrinkage toward the popula-
tion estimates proportional to the distance of the per-subject estimates from
the population estimates.
It is true that the mixed model estimates for a particular subject are
“between” the within-subject estimates and the population estimates, in the
sense that the arrows in Fig.˜3.8 all point somewhat in the direction of the
population estimate. However, the extent of the attenuation of the within-
subject estimates toward the population estimates is not simply related to the
distance between those two sets of estimates. Consider the two panels, labeled
330 and 337, at the top right of Fig.˜3.9. The within-subject estimates for 337

Page: 73 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

74 3 Models for Longitudinal Data

are quite unlike the population estimates but the mixed-model estimates are
very close to these within-subject estimates. That is, the solid line and the
dashed line in that panel are nearly coincident and both are a considerable
distance from the dot-dashed line. For subject 330, however, the dashed line is
more-or-less an average of the solid line and the dot-dashed line, even though
the solid and dot-dashed lines are not nearly as far apart as they are for
subject 337.
The difference between these two cases is that the within-subject estimates
for 337 are very well determined. Even though this subject had an unusually
large intercept and slope, the overall pattern of the responses is very close to
a straight line. In contrast, the overall pattern for 330 is not close to a straight
line so the within-subject coefficients are not well determined. The multiple
R2 for the solid line in the 337 panel is 93.3% but in the 330 panel it is only
15.8%. The mixed model can pull the predictions in the 330 panel, where
the data are quite noisy, closer to the population line without increasing the
residual sum of squares substantially. When the within-subject coefficients
are precisely estimated, as in the 337 panel, very little shrinkage takes place.
We see from Fig.˜3.9 that the mixed-effects model smooths out the
between-subject differences in the predictions by bringing them closer to a
common set of predictions, but not at the expense of dramatically increasing
the sum of squared residuals. That is, the predictions are determined so as to
balance fidelity to the data, measured by the residual sum of squares, with
simplicity of the model. The simplest model would use the same prediction
in each panel (the dot-dashed line) and the most complex model, based on
linear relationships in each panel, would correspond to the solid lines. The
dashed lines are between these two extremes. We will return to this view of
the predictions from mixed models balancing complexity versus fidelity in
Sect.˜5.3, where we make the mathematical nature of this balance explicit.
We should also examine the prediction intervals on the random effects
(Fig.˜3.10) where we see that many prediction intervals overlap zero but
there are several that do not. In this plot the subjects are ordered from
bottom to top according to increasing conditional mode of the random effect
for (Intercept). The resulting pattern in the conditional modes of the random
effect for Days reinforces our conclusion that the model fm07, which does not
allow for correlation of the random effects for (Intercept) and Days, is suitable.

3.5 Chapter Summary

Problems

3.1. Check the structure of documentation, structure and a summary of the

Orthodont data set from the MEMSS package.

Page: 74 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

3.5 Chapter Summary 75

(Intercept) Days
337 ● ●
330 ● ●
331 ● ●
352 ● ●
333 ● ●
372 ● ●
332 ● ●
351 ● ●
369 ● ●
308 ● ●
371 ● ●
335 ● ●
334 ● ●
350 ● ●
349 ● ●
370 ● ●
310 ● ●
309 ● ●

−60 −40 −20 0 20 40 60 −15 −10 −5 0 5 10 15

Fig. 3.10 Prediction intervals on the random effects for model fm07.

(a) Create an xyplot of the distance versus age by Subject for the female sub-
jects only. You can use the optional argument subset = Sex == "Female"
in the call to xyplot to achieve this. Use the optional argument type =
c("g","p","r") to add reference lines to each panel.
(b) Enhance the plot by choosing an aspect ratio for which the typical slope of
the reference line is around 45o . You can set it manually (something like
aspect = 4) or with an automatic specification (aspect = "xy"). Change
the layout so the panels form one row (layout = c(11,1)).
(c) Order the panels according to increasing response at age 8. This is
achieved with the optional argument index.cond which is a function of
arguments x and y. In this case you could use index.cond = function(x,y)
y[x == 8]. Add meaningful axis labels. Your final plot should be like

Page: 75 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

76 3 Models for Longitudinal Data

Distance from pituitary to pterygomaxillary fissure (mm)

89 11 13 89 11 13 89 11 13 89 11 13 89 11 13

F10 F06 F09 F03 F01 F02 F05 F07 F08 F04 F11
28 ● ●

●
26 ●
●
● ● ●
● ● ●
24 ● ● ●
● ● ●
● ● ● ● ●
● ● ●
22 ●
● ● ● ● ●
● ● ● ● ●
●
20 ● ● ●
●
● ●

18
●
16
89 11 13 89 11 13 89 11 13 89 11 13 89 11 13 89 11 13

Age (yr)

(d) Fit a linear mixed model to the data for the females only with random
effects for the intercept and for the slope by subject, allowing for corre-
lation of these random effects within subject. Relate the fixed effects and
the random effects’ variances and covariances to the variability shown in
the figure.
(e) Produce a “caterpillar plot” of the random effects for intercept and slope.
Does the plot indicate correlated random effects?
(f) Consider what the Intercept coefficient and random effects represents.
What will happen if you center the ages by subtracting 8 (the baseline
year) or 11 (the middle of the age range)?
(g) Repeat for the data from the male subjects.

3.2.
Fit a model to both the female and the male subjects in the Orthodont data
set, allowing for differences by sex in the fixed-effects for intercept (probably
with respect to the centered age range) and slope.

Page: 76 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

Chapter 4
Building Linear Mixed Models

In the previous chapter we incorporated the time covariate of longitudinal

data into both the fixed-effects terms and the random-effects terms of linear
mixed models. In this chapter we will extend the use of covariates and discuss
general approaches to building and assessing linear mixed models.
Statistical model building is still somewhat more of an art than a science
and there are many practical issues that we should bear in mind while engaged
in it. We discuss how some of the optional arguments to the lmer function
can be used to check for possible problems.
We begin with a discussion of categorical covariates and how they are
incorporated in the fixed-effects terms.

4.1 Incorporating categorical covariates with fixed levels

As we have seen in the earlier chapters, each random-effects terms in a linear

mixed model is defined with respect to a grouping factor, which is a categorical
covariate whose levels are regarded as a sample from a population of possible
levels. In some cases we may have other categorical covariates with a fixed
set of levels (such as gender, with fixed levels male and female) that we wish
to incorporate into the model. Although incorporating such covariates into
the model formula is straightforward, the interpretation of coefficients and
the model’s structure can be subtle.

4.1.1 The Machines Data

Milliken and Johnson [2009, Table 23.1] discuss data from an experiment
to measure productivity on a manufacturing task according to the type of
machine used and the operator. No further details on the experiment are

77
78 4 Building Linear Mixed Models

A ● B C

● ●●
3

5 ● ●●
Worker

●
1 ● ●

●
4 ● ●

●
2 ● ●

6 ● ● ●

45 50 55 60 65 70

Quality and productivity score

Fig. 4.1 A quality and productivity score for each of six operators (the Worker factor)
on each of three machine types.

given and it is possible that these data, which are available as Machines in the
MEMSS package, were constructed for illustration and are not observed data
from an actual experiment.
> str(Machines)

'data.frame': 54 obs. of 3 variables:

$ Worker : Factor w/ 6 levels "1","2","3","4",..: 1 1 1 2 2 2 3 3 3 4 ...
$ Machine: Factor w/ 3 levels "A","B","C": 1 1 1 1 1 1 1 1 1 1 ...
$ score : num 52 52.8 53.1 51.8 52.8 53.1 60 60.2 58.4 51.1 ...

> xtabs(~ Machine + Worker, Machines)

Worker
Machine 1 2 3 4 5 6
A 3 3 3 3 3 3
B 3 3 3 3 3 3
C 3 3 3 3 3 3

The cross-tablulation shows that each of the six operators used each of
the three machines on three occasions producing replicate observations of the
“subject-stimulus” combinations. Although the operators represent a sample
from the population of potential operators, the three machines are the specific
machines of interest. That is, we regard the levels of Machine as fixed levels
and the levels of Worker as a random sample from a population. In other
studies we may regard the levels of the stimulus as a random sample from a
population of stimuli.
A plot of these data (Fig.˜4.1) shows high reproducibility of measurements
on the same operator-machine combination. On each line the scores on a
particular machine are tightly clustered. There are considerable, apparently
systematic, differences between machines and somewhat smaller differences
between operators except for one unusual combination, operator 6 on machine
B. The pattern for operator 6 is very different from the pattern for the other

Page: 78 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

4.1 Incorporating categorical covariates with fixed levels 79

five operators. We will expect a significant interaction between the Worker

and Machine factors because of this unusual pattern. As we will see, we can
incorporate such an interaction into a linear mixed model in different ways.

4.1.2 Comparing models with and without interactions

We fit and compare three models for these data: fm10 without interactions,
fm11 with vector-valued random effects to allow for interactions, and fm12 with
interactions incorporated into a second simple scalar random-effects term.
> fm10 <- lmer(score ~ Machine + (1|Worker), Machines, REML=FALSE)
> fm11 <- lmer(score ~ Machine + (Machine|Worker), Machines, REML=FALSE)
> fm12 <- lmer(score ~ Machine + (1|Worker) + (1|Machine:Worker), Machines, REML=FALSE)
> anova(fm10, fm11, fm12)

Data: Machines
Models:
fm10: score ~ Machine + (1 | Worker)
fm12: score ~ Machine + (1 | Worker) + (1 | Machine:Worker)
fm11: score ~ Machine + (Machine | Worker)
Df AIC BIC logLik Chisq Chi Df Pr(>Chisq)
fm10 5 303.70 313.65 -146.85
fm12 6 237.27 249.20 -112.64 68.4338 1 < 2e-16
fm11 10 236.42 256.31 -108.21 8.8516 4 0.06492

Notice that in the anova summary the order of the models has been re-
arranged according to the complexity of the models, as measured by the
number of parameters to be estimated. The simplest model, fm10, incorpo-
rates three fixed-effects parameters and two variance components. Model fm12,
with scalar random effects for Worker and for the Machine:Worker combination
incorporates one additional variance component for a total of six parameters,
while model fm11 adds five variance-component parameters to those in fm10,
for a total of 10 parameters.
In the comparison of models fm10 and fm12 (i.e. the line labeled fm12 in the
anova table) we see that the additional parameter is highly significant. The
change in the deviance of 68.4338 (in the column labeled Chisq) at a cost of
one additional parameter is huge; hence we prefer the more complex model
fm12. In the next line, which is the comparison of the more complex model
fm11 to the simpler model fm12, the change in the deviance is 8.8516 at a
cost of 4 additional parameters with a p-value of 6.5%. In formal hypothesis
tests we establish a boundary, often chosen to be 5%, below which we regard
the p-value as providing “significant” evidence to prefer the more complex
model and above which the results are regarded as representing an “insignif-
icant” improvement. Such boundaries, while arbitrary, help us to assess the
numerical results and here we prefer model fm12, of intermediate complexity.

Page: 79 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

80 4 Building Linear Mixed Models

T1 ● T2 T3 T4

B ●

A ●

G ●

C ●

F ●

I ●

D ●

E ●

H ●

8 10 12 14

Effort to arise (Borg scale)

Fig. 4.2 Subjective evaluation of the effort required to arise (on the Borg scale) by
9 subjects, each of whom tried each of four types of stool.

fit models having different configurations of simple, scalar random effects

but always with the same fixed-effects specification of an intercept, or overall
mean response, only.
It is common in practice to have several fixed-effects terms involving one
or more covariates in the specification of a linear mixed model. Indeed, the
purpose of fitting a linear mixed model is often to draw inferences about the
effects of the covariates while appropriately accounting for different sources
of variability in the responses.
In this chapter we demonstrate how fixed-effects terms are incorporated in
a linear mixed model and how inferences about the effects of the covariates
are drawn from a fitted linear mixed model.

4.2 Models for the ergoStool data

Problems˜2.1 and 2.2 in Chap.˜2 involve examining the structure of the er-
goStool data from the MEMSS package
> str(ergoStool)

'data.frame': 36 obs. of 3 variables:

$ effort : num 12 15 12 10 10 14 13 12 7 14 ...
$ Type : Factor w/ 4 levels "T1","T2","T3",..: 1 2 3 4 1 2 3 4 1 2 ...
$ Subject: Factor w/ 9 levels "A","B","C","D",..: 1 1 1 1 2 2 2 2 3 3 ...

and plotting these data, as in Fig.˜4.2. These data are from an ergometrics
experiment where nine subjects evaluated the difficulty to arise from each of
four types of stools. The measurements are on the scale of perceived exertion
developed by the Swedish physician and researcher Gunnar Borg. Measure-

Page: 80 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

4.2 Models for the ergoStool data 81

ments on this scale are in the range 6-20 with lower values indicating less
exertion.
From Fig.˜4.2 we can see that all nine subjects rated type T1 or type T4 as
requiring the least exertion and rated type T2 as requiring the most exertion.
Type T3 was perceived as requiring comparatively little exertion by some
subjects (H and E) and comparatively greater exertion by others (F, C and
G).
Problem˜2.3 involves fitting and evaluating a model in which the effects
of both the Subject and the Type factors are incorporated as random effects.
Such a model may not be appropriate for these data where we wish to make
inferences about these particular four stool types. According to the distinction
between fixed- and random-effects described in Sect.˜1.1, if the levels of the
Type factor are fixed and reproducible we generally incorporate the factor in
the fixed-effects part of the model.
Before doing so, let’s review the results of fitting a linear mixed model
with random effects for both Subject and Type.

4.2.1 Random-effects for both Subject and Type

A model with random effects for both Subject and Type is fit in the same way
that we fit such in Chap.˜2,
> (fm06 <- lmer(effort ~ 1 + (1|Subject) + (1|Type), ergoStool, REML=FALSE))

Linear mixed model fit by maximum likelihood ['merMod']

Formula: effort ~ 1 + (1 | Subject) + (1 | Type)
Data: ergoStool
AIC BIC logLik deviance
144.0224 150.3564 -68.0112 136.0224
Random effects:
Groups Name Variance Std.Dev.
Subject (Intercept) 1.704 1.305
Type (Intercept) 2.265 1.505
Residual 1.213 1.101
Number of obs: 36, groups: Subject, 9; Type, 4

Fixed effects:
Estimate Std. Error t value
(Intercept) 10.2500 0.8883 11.54

from which we determine that the mean effort to arise, across stool types and
across subjects, is 10.250 on this scale, with standard deviations of 1.305 for
the random-effects for the Subject factor and 1.505 for the Type factor.
One question we would want to address is whether there are “significant”
differences between stool types, taking into account the differences between
subjects. We could approach this question by fitting a reduced model, with-

Page: 81 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

82 4 Building Linear Mixed Models

.sig01 .sig02 .lsig (Intercept)

0
ζ

−1

−2

0.5 1.0 1.5 2.0 2.5 3.0 1 2 3 4 5 −0.2 0.0 0.2 0.4 8 10 12

Fig. 4.3 Profile zeta plot for the parameters in model fm06 fit to the ergoStool data

out random effects for Type, and comparing this fit to model fm06 using a
likelihood-ratio test.
> fm06a <- lmer(effort ~ 1 + (1|Subject), ergoStool, REML=FALSE)
> anova(fm06a, fm06)
Data: ergoStool
Models:
fm06a: effort ~ 1 + (1 | Subject)
fm06: effort ~ 1 + (1 | Subject) + (1 | Type)
Df AIC BIC logLik Chisq Chi Df Pr(>Chisq)
fm06a 3 164.15 168.90 -79.075
fm06 4 144.02 150.36 -68.011 22.128 1 2.551e-06

The p-value in this test is very small, indicating that the more complex model,
fm06, which allows for differences in the effort to arise for the different stool
types, provides a significantly better fit to the observed data.
In Sect.˜2.2.4 we indicated that, because the constraint on the reduced
model, σ2 = 0, is on the boundary of the parameter space, the p-value for
this likelihood ratio test statistic calculated using a χ12 reference distribution
will be conservative. That is, the p-value one would obtain by, say, simulation
from the null distribution, would be even smaller than the p-value, 0.0000026,
reported by this test, which is already very small.
Thus the evidence against the null hypothesis (H0 : σ2 = 0) and in favor of
the alternative, richer model (Ha : σ2 > 0) is very strong.
Another way of addressing the question of whether it is reasonable for σ2
to be zero is to profile fm06 and examine profile zeta plots (Fig.˜4.3) and the
corresponding profile pairs plot (Fig.˜4.4).
We can see from the profile zeta plot (Fig.˜4.3) that both σ1 , the standard
deviation of the Subject random effects, and, σ2 , the standard deviation of
the Type random effects, are safely non-zero. We also see that σ2 is very
poorly determined. That is, a 95% profile-based confidence interval on this
parameter, obtained as

Page: 82 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

4.2 Models for the ergoStool data 83

14

12

10 (Intercept)

6 0 1 2 3

0.6
0.2 0.4 0.6

0.4

0.2
.lsig
0.0

−0.2 0 1 2 3

8
6 8

4 .sig02

2
0 1 2 3

1 2 3

.sig01 0

−1

−2

−3
Scatter Plot Matrix

Fig. 4.4 Profile pairs plot for the parameters in model fm06 fit to the ergoStool
data

> confint(pr06)[".sig02",]
2.5 % 97.5 %
0.7925434 3.7958505

is very wide. The upper end point of this 95% confidence interval, 3.796, is
more than twice as large as the estimate, σ c2 = 1.505.
A plot of the prediction intervals on the random effects for Type (Fig.˜4.5)
confirms the impression from Fig.˜4.2 regarding the stool types. Type T2
requires the greatest effort and type T1 requires the least effort. There is con-
siderable overlap of the prediction intervals for types T1 and T4 and somewhat
less overlap between types T4 and T3 and between types T3 and T2.
In an analysis like this we begin by asking if there are any significant dif-
ferences between the stool types, which we answered for this model by testing
the hypothesis H0 : σ2 = 0 versus Ha : σ2 > 0. If we reject H0 in favor of Ha

Page: 83 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

84 4 Building Linear Mixed Models

T2 ●

T3 ●

T4 ●

T1 ●

−2 −1 0 1 2 3

Fig. 4.5 95% prediction intervals on the random effects for Type from model fm06
fit to the ergoStool data

— that is, if we conclude that the more complex model including random
effects for Type provides a significantly better fit than the simpler model —
then usually we want to follow up with the question, “Which stool types are
significantly different from each other?”. It is possible, though not easy, to
formulate an answer to that question from a model fit such as fm06 in which
the stool types are modeled with random effects, but it is more straightfor-
ward to address that question when we model the stool types as fixed-effects
parameters, which we do next.

4.2.2 Fixed Effects for Type, Random for Subject

To incorporate the Type factor in the fixed-effects parameters, instead of as

a grouping factor for random effects, we remove the random-effects term,
(1|Type), and add Type to the fixed-effects specification.
> (fm07 <- lmer(effort ~ 1 + Type + (1|Subject), ergoStool, REML = 0))

Linear mixed model fit by maximum likelihood ['merMod']

Formula: effort ~ 1 + Type + (1 | Subject)
Data: ergoStool
AIC BIC logLik deviance
134.1444 143.6456 -61.0722 122.1444
Random effects:
Groups Name Variance Std.Dev.
Subject (Intercept) 1.578 1.256
Residual 1.076 1.037
Number of obs: 36, groups: Subject, 9

Fixed effects:
Estimate Std. Error t value
(Intercept) 8.5556 0.5431 15.754
TypeT2 3.8889 0.4890 7.952
TypeT3 2.2222 0.4890 4.544
TypeT4 0.6667 0.4890 1.363

Correlation of Fixed Effects:

Page: 84 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

4.2 Models for the ergoStool data 85

(Intr) TypeT2 TypeT3

TypeT2 -0.450
TypeT3 -0.450 0.500
TypeT4 -0.450 0.500 0.500

It appears that the last three levels of the Type factor are now modeled
as fixed-effects parameters, in addition to the (Intercept) parameter, whose
estimate has decreased markedly from that in model fm06. Furthermore, the
estimates of the fixed-effects parameters labeled TypeT2, TypeT3 and TypeT4,
while positive, are very much smaller than would be indicated by the average
responses for these types.
It turns out, of course, that the fixed-effects parameters generated by a
factor covariate do not correspond to the overall mean and the effect for each
level of the covariate. Although a model for an experiment such as this is
sometimes written in a form like

yi j = µ + αi + b j + εi j , i = 1, . . . , 4, j = 1, . . . 9 (4.1)

where i indexes the stool type and j indexes the subject, the parameters
{µ, α1 , α2 , α3 , α4 }, representing the overall mean and the effects of each of the
stool types, are redundant. Given a set of estimates for these parameters we
would not change the predictions from the model if, for example, we added
one to µ and subtracted one from all the α’s. In statistical terminology we
say that this set of parameters is not estimable unless we impose some other
conditions on them. The estimability condition ∑4i=1 αi = 0 is often used in
introductory texts.
The approach taken in R is not based on redundant parameters that are
subject to estimability conditions. While this approach may initially seem
reasonable, in complex models it quickly becomes unnecessarily complex to
need to use constrained optimization for parameter estimation. Instead we
incorporate the constraints into the parameters that we estimate. That is,
we reduce the redundant set of parameters to an estimable set of contrasts
between the levels of the factors.

4.2.2.1 The default contrasts generated for a factor

Although the particular set of contrasts used for a categorical factor can be
controlled by the user, either as a global option for a session (see ?options) or
by the optional contrasts argument available in most model-fitting functions,
most users do not modify the contrasts, preferring to leave them at the default
setting, which is the “treatment” contrasts (contr.treatment) for an unordered
factor and orthogonal polynomial contrasts (contr.poly) for an ordered factor.
You can check the current global setting with
> getOption("contrasts")

Page: 85 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

86 4 Building Linear Mixed Models

unordered ordered
"contr.treatment" "contr.poly"

Because these were the contrasts in effect when model fm07 was fit, the
particular contrasts used for the Type factor, which has four levels, correspond
to
> contr.treatment(4)

2 3 4
1 0 0 0
2 1 0 0
3 0 1 0
4 0 0 1

In this display the rows correspond to the levels of the Type factor and the
columns correspond to the parameters labeled TypeT2, TypeT3 and TypeT4.
The values of Type in the data frame, whose first few rows are
> head(ergoStool)

effort Type Subject

1 12 T1 A
2 15 T2 A
3 12 T3 A
4 10 T4 A
5 10 T1 B
6 14 T2 B

combined with the contrasts produce the model matrix X, whose first few
rows are
> head(model.matrix(fm07))

6 x 4 Matrix of class "dgeMatrix"

(Intercept) TypeT2 TypeT3 TypeT4
[1,] 1 0 0 0
[2,] 1 1 0 0
[3,] 1 0 1 0
[4,] 1 0 0 1
[5,] 1 0 0 0
[6,] 1 1 0 0

We see that the rows of X for observations on stool type T1 have zeros in the
last three columns; the rows for observations on stool type T2 have a 1 in the
second column and zeros in the last two columns, and so on. As before, the
(Intercept) column is a column of 1’s.
When we evaluate Xβ in the linear predictor expression, Xβ + Zb, we take
the p elements of the fixed-effects parameter vector, β , whose estimate is
> fixef(fm07)

(Intercept) TypeT2 TypeT3 TypeT4

8.5555556 3.8888889 2.2222222 0.6666667

Page: 86 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

4.2 Models for the ergoStool data 87

and the p elements of a row of the matrix X, multiply corresponding compo-

nents and sum the resulting products. For example, the fixed-effects predictor
for the first observation (stool type T1) will be

8.5556 × 1 + 3.8889 × 0 + 2.2222 × 0 + 0.6667 × 0 = 8.5556

and the fixed-effects predictor for the second observation (stool type T2) will
be
8.5556 × 1 + 3.8889 × 1 + 2.2222 × 0 + 0.6667 × 0 = 12.4444
We see that the parameter labeled (Intercept) is actually the fixed-effects
prediction for the first level of Type (i.e. level T1) and the second parameter,
labeled TypeT2, is the difference between the fixed-effects prediction for the
second level (T2) and the first level (T1) of the Type factor.
Similarly, the fixed-effects predictions for the T3 and T4 levels of Type are
8.5556 + 2.2222 = 10.7778 and 8.5556 + 0.6667 = 9.2222, respectively, as can
be verified from
> head(as.vector(model.matrix(fm07) %*% fixef(fm07)))

[1] 8.555556 12.444444 10.777778 9.222222 8.555556 12.444444

The fact that the parameter labeled TypeT2 is the difference between the
fixed-effects prediction for levels T2 and T1 of the Type factor is why we refer
to the parameters as being generated by contrasts. They are formed by con-
trasting the fixed-effects predictions for some combination of the levels of the
factor. In this case the contrast is between levels T2 and T1.
In general, the parameters generated by the “treatment” contrasts (the
default for unordered factors) represent differences between the first level of
the factor, which is incorporated into the (Intercept) parameter, and the
subsequent levels. We say that the first level of the factor is the reference
level and the others are characterized by their shift relative to this reference
level.

4.2.2.2 Profiling the contrasts

Because some of the contrasts that are of interest to us correspond to fixed-

effects parameter values, we can profile the fitted model (Fig.˜4.6) and check,
say, the confidence intervals on these parameters.
> confint(pr07, c("TypeT2", "TypeT3", "TypeT4"))

2.5 % 97.5 %
TypeT2 2.8953043 4.882473
TypeT3 1.2286377 3.215807
TypeT4 -0.3269179 1.660251

According to these intervals, and from what we see from Fig.˜4.6, types T2
and T3 are significantly different from type T1 (the intervals do not contain

Page: 87 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

88 4 Building Linear Mixed Models

.sig01 .lsig (Intercept) TypeT2 TypeT3 TypeT4

2
1
0
ζ

−1
−2

0.5 1.0 1.5 2.0 2.5 −0.2 0.0 0.2 0.4 7 8 9 102.5 3.5 4.5 1.0 2.0 3.0 −0.5 0.5 1.5

Fig. 4.6 Profile zeta plot for the parameters in model fm06 fit to the ergoStool data

zero) but type T4 is not (the confidence interval on this contrast contains
zero).
However, this process must be modified in two ways to provide a suitable
answer. The most important modification is to take into account the fact
that we are performing multiple comparisons simultaneously. We describe
what this means and how to accomodate for it in the next subsection. The
other problem is that this process only allows us to evaluate contrasts of the
reference level, T1, with the other levels and the reference level is essentially
arbitrary. For completeness we should evaluate all six possible contrasts of
pairs of levels.
We can do this by refitting the model with a difference reference level for
the Type factor and profiling the modified model fit. The relevel function
allows us to change the reference level of a factor.
> pr07a <- profile(lmer(effort ~ 1 + Type + (1|Subject),
+ within(ergoStool, Type <- relevel(Type, "T2")),
+ REML=FALSE))
> pr07b <- profile(lmer(effort ~ 1 + Type + (1|Subject),
+ within(ergoStool, Type <- relevel(Type, "T3")),
+ REML=FALSE))

The other constrasts of interest are

> confint(pr07a, c("TypeT3", "TypeT4"))

2.5 % 97.5 %
TypeT3 -2.660251 -0.6730821
TypeT4 -4.215807 -2.2286377

> confint(pr07b, "TypeT4")

2.5 % 97.5 %
TypeT4 -2.54914 -0.561971

Page: 88 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

4.2 Models for the ergoStool data 89

from which would conclude that type T2 requires significantly greater effort
than any of the other types at the 5% level (because none of the 95% confi-
dence intervals on contrasts with T2 contain zero) and that types T3 and T4
are significantly different at the 5% level.
However, we must take into account that we are performing multiple, sim-
ulataneous comparisons of levels.

4.2.2.3 Multiple comparisons

In the technical definition of a confidence interval we regard the end points as

being random (because they are calculated from the random variable which is
the observed data) and the value of the parameter or the contrast of interest
as being fixed (because it is determined from the fixed, but unknown, values of
the parameters). Thus we speak of the probability that an interval covers the
true parameter value rather than the probability that the parameter falls in
the interval. The distinction may seem, and probably is, somewhat pedantic.
We introduce it here simply to clarify the term “coverage probability” used
throughout this section.
We have evaluated six possible pairwise comparisons of the four levels
of the Type factor. A 95% confidence interval on a particular contrast has,
in theory, a 5% probability of failing to cover the true difference. That is,
if the difference between two levels was in fact zero, there would still be a
5% probability that a 95% confidence interval on that contrast would not
include zero. When we consider the coverage of the six intervals contrasting
all possible pairs of stool types we usually have in mind that there should be a
95% probability of all six intervals covering the true, but unknown, differences
in effort for the stool types. That is, we think of the coverage probability as
applying to the simultaneous coverage of the family of intervals, not to the
coverage of one specific interval.
But the intervals calculated in the previous section were based on 95%
coverage for each specific interval. In the worst case scenario the family-wise
coverage could be as low as 1 − 0.05 ∗ 6 = 0.70 or 70%. For factors with more
than four levels there are even more possible pairwise comparisons (for k levels
there are k(k − 1)/2 possible pairs) and this worst-case coverage probability
is even further from the nominal level of 95%.
Several methods have been developed to compensate for multiple com-
parisons in the analysis of linear models with fixed effects only. One of the
simplest, although somewhat coarse, compensations is the Bonferroni cor-
rection where the individual intervals are chosen to have a greater coverage
probability in such a way that the “worst-case” probability is the desired level.
With six comparisons to get a family-wise coverage probability of 95% the
individual intervals are chosen to have coverage of
> (covrge <- 1 - 0.05/6)
[1] 0.9916667

Page: 89 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

90 4 Building Linear Mixed Models

or a little more than 99%. We can specify this coverage level for the individual
intervals to ensure a family-wise coverage of at least 95%.
> rbind(confint(pr07, c("TypeT2","TypeT3","TypeT4"), covrge),
+ confint(pr07a, c("TypeT3","TypeT4"), covrge),
+ confint(pr07b, "TypeT4", covrge))

0.417 % 99.583 %
TypeT2 2.5109497 5.2668280
TypeT3 0.8442830 3.6001613
TypeT4 -0.7112726 2.0446058
TypeT3 -3.0446059 -0.2887275
TypeT4 -4.6001614 -1.8442831
TypeT4 -2.9334948 -0.1776164

We again reach the conclusion that the only pair of stool types for which
zero is within the confidence interval on the difference in effects is the (T1,T4)
pair but, for these intervals, the family-wise coverage of all six intervals is at
least 95%.
There are other, perhaps more effective, techniques for adjusting intervals
to take into account multiple comparisons. The purpose of this section is to
show that the profile-based confidence intervals can be extended to at least
the Bonferroni correction.
The easiest way to apply other multiple comparison adjustment methods
is to model both the Type and the Subject factors with fixed effects, which we
do next.

4.2.3 Fixed Effects for Type and Subject

Even though the subjects in this study are chosen as representatives of a

population, many statisticians would regard Subject as a blocking factor in
the experiment and fit a model with fixed-effects for both Type and Subject.
A blocking factor is a known source of variability in the response. We are
not interested in the effects of the levels of the blocking factor—we only wish
to accomodate for this source of variability when comparing levels of the
experimental factor, which is the Type factor in this example.
We will discuss the advantages and disadvantages of the fixed- versus
random-effects choice for the Subject factor at the end of this section. For
the moment we proceed to fit the fixed-effects model, for which we could use
the lm function or the aov function. These two functions produce exactly the
same model fit but the aov function returns an object of class "aov" which
extends the class "lm", providing more options for examining the fitted model.
> summary(fm08 <- aov(effort ~ Subject + Type, ergoStool))

Df Sum Sq Mean Sq F value Pr(>F)

Subject 8 66.500 8.3125 6.8662 0.0001061

Page: 90 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

4.2 Models for the ergoStool data 91

Type 3 81.194 27.0648 22.3556 3.935e-07

Residuals 24 29.056 1.2106

As seen above, the summary method for objects of class "aov" provides an
analysis of variance table. The order in which the terms are listed in the model
formula can affect the results in this table, if the data are unbalanced, and we
should be cautious to list the terms in the model in the appropriate order, even
for a balanced data set like the ergoStool. The rule is that blocking factors
should precede experimental factors because the contributions of the terms
are assessed sequentially. Thus we read the rows in this table as measuring
the variability due to the Subject factor and due to the Type factor after
taking into account the Subject. We want to assess the experimental factor
after having removed the variability due to the blocking factor.
If desired we can assess individual coefficients by applying the summary
method for "lm" objects, called summary.lm to this fitted model. For example,
the coefficients table is available as
> coef(summary.lm(fm08))

Estimate Std. Error t value Pr(>|t|)

(Intercept) 1.055556e+01 0.6352554 1.661624e+01 1.146367e-14
SubjectB 3.804158e-15 0.7780258 4.889502e-15 1.000000e+00
SubjectC -1.500000e+00 0.7780258 -1.927957e+00 6.577394e-02
SubjectD -3.000000e+00 0.7780258 -3.855913e+00 7.577323e-04
SubjectE -3.750000e+00 0.7780258 -4.819892e+00 6.565314e-05
SubjectF -2.000000e+00 0.7780258 -2.570609e+00 1.678081e-02
SubjectG -1.500000e+00 0.7780258 -1.927957e+00 6.577394e-02
SubjectH -4.000000e+00 0.7780258 -5.141218e+00 2.907651e-05
SubjectI -2.250000e+00 0.7780258 -2.891935e+00 8.010522e-03
TypeT2 3.888889e+00 0.5186838 7.497610e+00 9.753420e-08
TypeT3 2.222222e+00 0.5186838 4.284348e+00 2.562927e-04
TypeT4 6.666667e-01 0.5186838 1.285304e+00 2.109512e-01

but often the individual coefficients are of less interest than the net effect
of the variability due to the levels of the factor, as shown in the analysis of
variance table. For example, in the summary of the coefficients shown above
the (Intercept) coefficient is the predicted response for the reference subject
(subject A) on the reference stool type (type T1). Other coefficients generated
by the Subject term are the differences from the reference subject to other
subjects. It is not clear why we would want to compare all the other subjects
to subject A.
One of the multiple comparison methods that we can apply to fm08 is
Tukey’s Honest Significant Difference (HSD) method
> TukeyHSD(fm08, which = "Type")

Tukey multiple comparisons of means

95% family-wise confidence level
Fit: aov(formula = effort ~ Subject + Type, data = ergoStool)

$Type

Page: 91 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

92 4 Building Linear Mixed Models

diff lwr upr p adj

T2-T1 3.8888889 2.4580431 5.3197347 0.0000006
T3-T1 2.2222222 0.7913764 3.6530680 0.0013709
T4-T1 0.6666667 -0.7641791 2.0975125 0.5807508
T3-T2 -1.6666667 -3.0975125 -0.2358209 0.0182037
T4-T2 -3.2222222 -4.6530680 -1.7913764 0.0000115
T4-T3 -1.5555556 -2.9864013 -0.1247098 0.0295822

from which we reach essentially the same conclusions as before, although

perhaps less arduously.

4.2.4 Fixed Effects Versus Random Effects for Subject

These three analyses provoke the question of whether to use fixed-effects

parameters or random effects for the Subject factor. That is, should Subject
be treated as a blocking factor or as a sample of levels from a population.
If the sole purpose of the experiment is to rate the stool types relative to
each other then it may be more convenient to consider Subject as a blocking
factor in the experiment and incorporate it as the first term in a fixed-effects
model for this completely balanced data set from a designed experiment.
Strictly speaking, the inferences about the stool types that we would draw
apply to these particular nine subjects only, not to a general population,
but in practice it is not too much of a stretch to think of them as applying
to a population unless we want to predict the score that a general member
of the population would give to a particular stool type. We can formulate
the prediction but assessing the variability in the prediction is difficult when
using fixed-effects models.
Random effects are preferred, perhaps even necessary, if we wish to make
inferences that apply to the population of potential users. Also, when we
have unbalanced data or large samples, the flexibility of mixed models be-
comes important in stabilizing the estimation of parameters. The estimation
of parameters in fixed-effects models by least squares is somewhat rigid. In
theory it requires that the columns of the model matrix, X, are linearly in-
dependent. In practice, it is very difficult to determine if the columns of a
large model matrix are indeed linear independent or, equivalently, if the rank
of X is exactly p. The best we can do is determine if the condition number,
κ, of X (the ratio of the largest to smallest singular values — see ?kappa) is
sufficiently small to trust the numerical linear algebra results. One way of
thinking of κ is as a “magnification factor” for numerical perturbations. In
the worst-case scenario, perturbations of relative size ε in the elements of y
results in perturbations of relative size κε in the coefficients βb.
When the number of columns, p, of X is small the condition number tends
to be small.
> kappa(fm08, exact = TRUE)

Page: 92 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

4.3 Covariates Affecting Mathematics Score Gain 93

[1] 10.76924

However, when p is large the condition number of X tends to become very

large and evaluation of fixed-effects parameter estimates and their standard
errors is an ill-conditioned problem.
Calculations involving the random effects model matrix, Z, are not as
sensitive to ill-conditioning. The numerical accuracy is determined by the
condition number of the sparse Cholesky factor, Lθ , defined in (1.5) which is
less than the condition number of ΛθT ZT ZΛθ , even when Λθ is singular.
> kappa.tri(as(as(fm06@re@L, "sparseMatrix"), "matrix"), exact = TRUE)

[1] 4.83993

The evaluation of Lθ is an example of regularization methods for solving

ill-posed problems or to prevent overfitting.

4.3 Covariates Affecting Mathematics Score Gain

West et˜al. [2007] provides comparisons of several different software systems

for fitting linear mixed models by fitting sample models to different data sets
using each of these software systems. The lmer function from the lme4 package
is not included in these comparisons because it was still being developed when
that book was written.
In this section we will use lmer to fit models described in Chap.˜4 of West
et˜al. [2007] to data on the gain in mathematics scores for students in a
selection of classrooms in several schools.
> str(classroom)

'data.frame': 1190 obs. of 11 variables:

$ sex : Factor w/ 2 levels "M","F": 2 1 2 1 1 2 1 1 2 1 ...
$ minority: Factor w/ 2 levels "N","Y": 2 2 2 2 2 2 2 2 2 2 ...
$ mathkind: int 448 460 511 449 425 450 452 443 422 480 ...
$ mathgain: int 32 109 56 83 53 65 51 66 88 -7 ...
$ ses : num 0.46 -0.27 -0.03 -0.38 -0.03 0.76 -0.03 0.2 0.64 0.13 ...
$ yearstea: num 1 1 1 2 2 2 2 2 2 2 ...
$ mathknow: num NA NA NA -0.11 -0.11 -0.11 -0.11 -0.11 -0.11 -0.11 ...
$ housepov: num 0.082 0.082 0.082 0.082 0.082 0.082 0.082 0.082 0.082 0.082 ..
$ mathprep: num 2 2 2 3.25 3.25 3.25 3.25 3.25 3.25 3.25 ...
$ classid : Factor w/ 312 levels "1","2","3","4",..: 160 160 160 217 217 217 ..
$ schoolid: Factor w/ 107 levels "1","2","3","4",..: 1 1 1 1 1 1 1 1 1 1 ...

The response modeled in Chap.˜4 of West et˜al. [2007] is mathgain, the

difference between a student’s mathematics score at the end of grade one and
at the end of kindergarten. To allow comparisons with the results in West
et˜al. [2007] we will use that response in this section.
There is one observation per student. In the terminology of the multilevel
modeling literature the levels of variability correspond to student (level 1),

Page: 93 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

94 4 Building Linear Mixed Models

146 ●
● ●
83 ● ● ● ● ●
● ●

70
152 ● ● ● ●● ●
261 ●
311 ● ●
244 ●
298 ● ● ● ● ●

68
271 ● ●●
263 ● ●●
57 ●● ● ●
10 ● ●
177 ● ● ●
17
216 ● ●
87 ●
7 ●
5 ●
92 ●
15

85 ● ● ● ● ● ● ●
119 ●
260 ● ● ●
40 ● ● ●
36 ●
57
Classroom within school

171 ●
250 ● ●
272 ●
229 ●●
● ● ●
26 ● ●● ● ● ●● ●●
●
33

100 ●
32 ● ● ● ●
88 ● ●
19 ● ●
71 ● ●
46

133 ●● ●
240 ● ● ●
69 ● ● ●
196 ● ● ●
84 ●
85

157 ● ● ● ●
66 ● ●● ● ● ●
186 ● ● ● ● ● ●
209 ● ●
120 ●● ●● ●● ● ●
71

80 ● ● ● ● ● ●
304 ● ● ● ●
264 ●
● ● ● ● ● ●
81 ● ● ●● ● ● ●●
251 ● ● ●
12

49 ● ● ● ● ● ●
20 ● ● ●
302 ● ●● ●● ● ●
236 ● ●
226 ● ●
99

205 ● ● ●● ● ● ● ●
266 ● ● ● ●●
214 ●
28 ● ● ● ● ●
287 ●● ●● ●
76

136 ● ● ●● ● ●
277 ● ● ● ●● ● ●
8 ● ●● ●

0 50 100 150 200

Gain in Mathematics score from kindergarten to first grade

Fig. 4.7 Comparative dotplots of gain in the mathematics scores in classrooms

within schools.

classroom (level 2) and school (level 3) with the assumption that classroom
is nested within school. The concept of “levels” can only be applied to models
and data sets in which the grouping factors of the random effects form a
nested sequence. In crossed or partially crossed configurations there are no
clearly defined levels.
At this point we should check if there is implicit nesting. That is, are the
levels of the classid factor nested within schoolid factor. We could simply
create the interaction factor to avoid the possibility of implicit nesting but it
saves a bit of trouble if we check before doing so
> with(classroom, isNested(classid, schoolid))

[1] TRUE

Page: 94 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

4.3 Covariates Affecting Mathematics Score Gain 95

A model with simple, scalar random effects and without any fixed-effects
terms (other than the implicit intercept) is called the “unconditional model”
in the multilevel modeling literature. We fit it as
> (fm09 <- lmer(mathgain ~ (1|classid) + (1|schoolid), classroom))

Linear mixed model fit by REML ['merMod']

Formula: mathgain ~ (1 | classid) + (1 | schoolid)
Data: classroom
REML criterion at convergence: 11768.76
Random effects:
Groups Name Variance Std.Dev.
classid (Intercept) 99.23 9.961
schoolid (Intercept) 77.49 8.803
Residual 1028.23 32.066
Number of obs: 1190, groups: classid, 312; schoolid, 107

Fixed effects:
Estimate Std. Error t value
(Intercept) 57.427 1.443 39.79

The results from this model fit using the REML criterion can be compared
to Table 4.6 (page 156) of West et˜al. [2007].
It seems that the housepov value is a property of the school. We can
check this by considering the number of unique combinations of housepov
and schoolid and comparing that to the number of levels of schoolid. For
safety we check the number of levels of factor(schoolid) in case there are
unused levels in schoolid.
> with(classroom, length(levels(factor(schoolid))))

[1] 107

> nrow(unique(subset(classroom, select = c(schoolid, housepov))))

[1] 107

In some formulations of multilevel models or hierarchical linear models it

is important to associate covariates with different levels in the hierarchy.
> (fm7 <- lmer(mathgain ~ 1 + I(mathkind-450) + sex + minority + ses
+ + housepov + (1|classid) + (1|schoolid), classroom))

Linear mixed model fit by REML ['merMod']

Formula: mathgain ~ 1 + I(mathkind - 450) + sex + minority + ses + housepov + (1 | class
Data: classroom
REML criterion at convergence: 11378.06
Random effects:
Groups Name Variance Std.Dev.
classid (Intercept) 81.56 9.031
schoolid (Intercept) 77.76 8.818
Residual 734.42 27.100
Number of obs: 1190, groups: classid, 312; schoolid, 107

Fixed effects:

Page: 95 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

96 4 Building Linear Mixed Models

sexF minorityY ses housepov

−1

−2

−4 −2 0 2 −14 −10 −6 −4 −2 2 4 6 8 −30 −10 0 10

ζ

.sig01 .sig02 .lsig (Intercept) I(mathkind − 450)

−1

−2

4 6 8 10 12 6 8 10 12 3.24 3.28 3.32 3.36 70 75 80 −0.52 −0.48 −0.44

Fig. 4.8 Profile plot of the parameters in model fm4.

Estimate Std. Error t value

(Intercept) 73.17077 2.80273 26.107
I(mathkind - 450) -0.47086 0.02228 -21.133
sexF -1.23459 1.65743 -0.745
minorityY -7.75587 2.38499 -3.252
ses 5.23971 1.24497 4.209
housepov -11.43920 9.93736 -1.151

Correlation of Fixed Effects:

(Intr) I(-450 sexF mnrtyY ses
I(mthk-450) -0.233
sexF -0.279 -0.032
minorityY -0.492 0.153 -0.015
ses -0.105 -0.165 0.019 0.144
housepov -0.555 0.035 -0.009 -0.184 0.078

A profile plot of the parameters in model fm7 is shown in Fig.˜4.8

4.4 Rat Brain example

> ftable(xtabs(activate ~ animal + treatment + region, ratbrain))

region BST LS VDB

animal treatment
R100797 Basal 458.16 245.04 237.42
Carbachol 664.72 587.10 726.96
R100997 Basal 479.81 261.19 195.51

Page: 96 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

4.4 Rat Brain example 97

VDB ●

Carbachol
LS ●

BST ● R100797 ●

R100997
R110597
R111097
VDB ●
R111397
Basal

LS ●

BST ●

200 300 400 500 600 700

activate

Fig. 4.9 Activation of brain regions in rats

Carbachol 515.29 437.56 604.29

R110597 Basal 462.79 278.33 262.05
Carbachol 589.25 493.93 621.07
R111097 Basal 366.19 199.31 187.11
Carbachol 371.71 302.02 449.70
R111397 Basal 375.58 204.85 179.38
Carbachol 492.58 355.74 459.58

Description of the Rat Brain data should go here.

Page: 97 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

Page: 98 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10
Chapter 5
Computational Methods for Mixed
Models

In this chapter we describe some of the details of the computational methods

for fitting linear mixed models, as implemented in the lme4 package, and the
theoretical development behind these methods. We also provide the basis for
later generalizations to models for non-Gaussian responses and to models
in which the relationship between the conditional mean, µ, and the linear
predictor, γ = Xβ + Zb = ZΛθ u + Xβ , is a nonlinear relationship.
This material is directed at those readers who wish to follow the theory
and methodology of linear mixed models and how both can be extended to
other forms of mixed models. Readers who are less interested in the “how”
and the “why” of fitting mixed models than in the results themselves should
not feel obligated to master these details.
We begin by reviewing the definition of linear mixed-effects models and
some of the basics of the computational methods, as given in Sect.˜1.1.

5.1 Definitions and Basic Results

As described in Sect.˜1.1, a linear mixed-effects model is based on two vector-

valued random variables: the q-dimensional vector of random effects, B, and
the n-dimensional response vector, Y . Equation (1.1) defines the uncondi-
tional distribution of B and the conditional distribution of Y , given B = b,
as multivariate Gaussian distributions of the form

(Y |B = b) ∼ N (Xβ + Zb, σ 2 I)
B ∼ N (0, Σθ ).

The q × q, symmetric, variance-covariance matrix, Var(B) = Σθ , depends

on the variance-component parameter vector, θ , and is positive semidefinite,
which means that
bT Σθ b ≥ 0, ∀ b 6= 0. (5.1)

99
100 5 Computational Methods for Mixed Models

(The symbol ∀ denotes “for all”.) The fact that Σθ is positive semidefinite
does not guarantee that Σθ−1 exists. We would need a stronger property,
bT Σθ b > 0, ∀ b 6= 0, called positive definiteness, to ensure that Σθ−1 exists.
Many computational formulas for linear mixed models are written in terms
of Σθ−1 . Such formulas will become unstable as Σθ approaches singularity.
And it can do so. It is a fact that singular (i.e. non-invertible) Σθ can and
do occur in practice, as we have seen in some of the examples in earlier
chapters. Moreover, during the course of the numerical optimization by which
the parameter estimates are determined, it is frequently the case that the
deviance or the REML criterion will need to be evaluated at values of θ that
produce a singular Σθ . Because of this we will take care to use computational
methods that can be applied even when Σθ is singular and are stable as Σθ
approaches singularity.
As defined in (1.2) a relative covariance factor, Λθ , is any matrix that
satisfies
Σθ = σ 2Λθ ΛθT .
According to this definition, Σ depends on both σ and θ and we should
write it as Σσ ,θ . However, we will blur that distinction and continue to write
Var(B) = Σθ . Another technicality is that the common scale parameter, σ ,
can, in theory, be zero. We will show that in practice the only way for its
estimate, σb , to be zero is for the fitted values from the fixed-effects only, Xβb ,
to be exactly equal to the observed data. This occurs only with data that
have been (incorrectly) simulated without error. In practice we can safely
assume that σ > 0. However, Λθ , like Σθ , can be singular.
Our computational methods are based on Λθ and do not require evaluation
of Σθ . In fact, Σθ is explicitly evaluated only at the converged parameter
estimates.
The spherical random effects, U ∼ N (0, σ 2 Iq ), determine B as

B = Λθ U . (5.2)

Although it may seem more intuitive to write U as a linear transformation

of B, we cannot do that when Λθ is singular, which is why (5.2) is in the
form shown.
We can easily verify that (5.2) provides the desired distribution for B. As
a linear transformation of a multivariate Gaussian random variable, B will
also be multivariate Gaussian. Its mean and variance-covariance matrix are
straightforward to evaluate,

E[B] = Λθ E[U ] = Λθ 0 = 0 (5.3)

and

Page: 100 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

5.2 The Conditional Distribution (U |Y = y) 101
h i h i
T T
Var(B) = E (B − E[B])(B − E[B]) = E BB (5.4)
h i
= E Λθ U U Λθ = Λθ E[U U T ]ΛθT = Λθ Var(U )ΛθT
T T

= Λθ σ 2 Iq ΛθT = σ 2Λθ ΛθT = Σθ

and have the desired form.

Just as we concentrate on how θ determines Λθ , not Σθ , we will concentrate
on properties of U rather than B. In particular, we now define the model
according to the distributions

(Y |U = u) ∼ N (ZΛθ u + Xβ , σ 2 In )
(5.5)
U ∼ N (0, σ 2 Iq ).

To allow for extensions to other types of mixed models we distinguish

between the linear predictor

γ = ZΛθ u + Xβ (5.6)

and the conditional mean of Y , given U = u, which is

µ = E [Y |U = u] . (5.7)

For a linear mixed model µ = γ. In other forms of mixed models the condi-
tional mean, µ, can be a nonlinear function of the linear predictor, γ. For
some models the dimension of γ is a multiple of n, the dimension of µ and y,
but for a linear mixed model the dimension of γ must be n. Hence, the model
matrix Z must be n × q and X must be n × p.

5.2 The Conditional Distribution (U |Y = y)

In this chapter it will help to be able to distinguish between the observed

response vector and an arbitrary value of Y . For this chapter only we will
write the observed data vector as yobs , with the understanding that y without
the subscript will refer to an arbitrary value of the random variable Y .
The likelihood of the parameters, θ , β , and σ , given the observed data,
yobs , is the probability density of Y , evaluated at yobs . Although the nu-
merical values of the probability density and the likelihood are identical, the
interpretations of these functions are different. In the density we consider the
parameters to be fixed and the value of y as varying. In the likelihood we
consider y to be fixed at yobs and the parameters, θ , β and σ , as varying.
The natural approach for evaluating the likelihood is to determine the
marginal distribution of Y , which in this case amounts to determining the
marginal density of Y , and evaluate that density at yobs . To follow this course

Page: 101 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

102 5 Computational Methods for Mixed Models

we would first determine the joint density of U and Y , written fU ,Y (u, y),
then integrate this density with respect u to create the marginal density,
fY (y), and finally evaluate this marginal density at yobs .
To allow for later generalizations we will change the order of these steps
slightly. We evaluate the joint density function, fU ,Y (u, y), at yobs , producing
the unnormalized conditional density, h(u). We say that h is “unnormalized”
because the conditional density is a multiple of h

h(u)
fU |Y (u|yobs ) = R . (5.8)
Rq h(u) du

In some theoretical developments the normalizing constant, which is the inte-

gral in the denominator of an expression like (5.8), is not of interest. Here it
is of interest because the normalizing constant is exactly the likelihood that
we wish to evaluate,
Z
L(θ , β , σ |yobs ) = h(u) du. (5.9)
Rq

For a linear mixed model, where all the distributions of interest are mul-
tivariate Gaussian and the conditional mean, µ, is a linear function of both
u and β , the distinction between evaluating the joint density at yobs to pro-
duce h(u) then integrating with respect to u, as opposed to first integrating
the joint density then evaluating at yobs , is not terribly important. For other
mixed models this distinction can be important. In particular, generalized lin-
ear mixed models, described in Sect.˜??, are often used to model a discrete
response, such as a binary response or a count, leading to a joint distribution
for Y and U that is discrete with respect to one variable, y, and contin-
uous with respect to the other, u. In such cases there isn’t a joint density
for Y and U . The necessary distribution theory for general y and u is well-
defined but somewhat awkward to describe. It is much easier to realize that
we are only interested in the observed response vector, yobs , not some arbi-
trary value of y, so we can concentrate on the conditional distribution of U
given Y = yobs . For all the mixed models we will consider, the conditional
distribution, (U |Y = yobs ), is continuous and both the conditional density,
fU |Y (u|yobs ), and its unnormalized form, h(u), are well-defined.

5.3 Integrating h(u) in the Linear Mixed Model

The integral defining the likelihood in (5.9) has a closed form in the case of
a linear mixed model but not for some of the more general forms of mixed
models. To motivate methods for approximating the likelihood in more gen-
eral situations, we describe in some detail how the integral can be evaluated
using the sparse Cholesky factor, Lθ , and the conditional mode,

Page: 102 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

5.3 Integrating h(u) in the Linear Mixed Model 103

ũ = arg max fU |Y (u|yobs ) = arg max h(u) = arg max fY |U (yobs |u) fU (u) (5.10)
u u u

The notation arg maxu means that ũ is the value of u that maximizes the
expression that follows.
In general, the mode of a continuous distribution is the value of the ran-
dom variable that maximizes the density. The value ũ is called the conditional
mode of u, given Y = yobs , because ũ maximizes the conditional density of
U given Y = yobs . The location of the maximum can be determined by max-
imizing the unnormalized conditional density because h(u) is just a constant
multiple of fU |Y (u|yobs ). The last part of (5.10) is simply a re-expression of
h(u) as the product of fY |U (yobs |u) and fU (u). For a linear mixed model
these densities are
!
2
1 ky − Xβ − ZΛθ uk
fY |U (y|u) = exp − (5.11)
(2πσ 2 )n/2 2σ 2
 
1 kuk2
fU (u) = exp − (5.12)
(2πσ 2 )q/2 2σ 2

with product
!
1 kyobs − Xβ − ZΛθ uk2 + kuk2
h(u) = exp − . (5.13)
(2πσ 2 )(n+q)/2 2σ 2

On the deviance scale we have

kyobs − Xβ − ZΛθ uk2 + kuk2

− 2 log (h(u)) = (n + q) log(2πσ 2 ) + . (5.14)
σ2
Because (5.14) describes the negative log density, ũ will be the value of u that
minimizes the expression on the right of (5.14).
The only part of the right hand side of (5.14) that depends on u is the
numerator of the second term. Thus

ũ = arg min kyobs − Xβ − ZΛθ uk2 + kuk2 . (5.15)

u

The expression to be minimized, called the objective function, is described as

a penalized residual sum of squares (PRSS) and the minimizer, ũ, is called the
penalized least squares (PLS) solution. They are given these names because
the first term in the objective, kyobs − Xβ − ZΛθ uk2 , is a sum of squared
residuals, and the second term, kuk2 , is a penalty on the length, kuk, of u.
Larger values of u (in the sense of greater lengths as vectors) incur a higher
penalty.
The PRSS criterion determining the conditional mode balances fidelity to
the observed data (i.e. producing a small residual sum of squares) against
simplicity of the model (small kuk). We refer to this type of criterion as

Page: 103 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

104 5 Computational Methods for Mixed Models

a smoothing objective, in the sense that it seeks to smooth out the fitted
response by reducing model complexity while still retaining reasonable fidelity
to the observed data.
For the purpose of evaluating the likelihood we will regard the PRSS crite-
rion as a function of the parameters, given the data, and write its minimum
value as
rθ2 ,β = min kyobs − Xβ − ZΛθ uk2 + kuk2 . (5.16)
u

Notice that β only enters the right hand side of (5.16) through the linear
predictor expression. We will see that ũ can be determined by a direct (i.e.
non-iterative) calculation and, in fact, we can minimize the PRSS criterion
with respect to u and β simultaneously without iterating. We write this
minimum value as

rθ2 = min kyobs − Xβ − ZΛθ uk2 + kuk2 . (5.17)

u,β

The value of β at the minimum is called the conditional estimate of β given

θ , written βb θ .

5.4 Determining the PLS Solutions, ũ and βb θ

One way of expressing a penalized least squares problem like (5.16) is by incor-
porating the penalty as “pseudo-data” in an ordinary least squares problem.
We extend the “response vector”, which is yobs − Xβ when we minimize with
respect to u only, with q responses that are 0 and we extend the predictor
expression, ZΛθ u with Iq u. Writing this as a least squares problem produces
    2
yobs − Xβ ZΛ θ

ũ = arg min − u (5.18)
u 0 Iq

with a solution that satisfies

 
ΛθT ZT ZΛθ + Iq ũ = ΛθT ZT (yobs − Xβ ) (5.19)

To evaluate ũ we form the sparse Cholesky factor, Lθ , which is a lower

triangular q × q matrix that satisfies

θ = Λθ Z ZΛθ + Iq .
Lθ LT T T
(5.20)

The actual evaluation of sparse Cholesky factor, Lθ , often incorporates a

fill-reducing permutation, which we describe next.

Page: 104 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

5.4 Determining the PLS Solutions, ũ and βb θ 105

5.4.1 The Fill-reducing Permutation, P

In earlier chapters we have seen that often the random effects vector is re-
ordered before Lθ is created. The re-ordering or permutation of the elements
of u and, correspondingly, the columns of the model matrix, ZΛθ , does not
affect the theory of linear mixed models but can have a profound effect on
the time and storage required to evaluate Lθ in large problems. We write the
effect of the permutation as multiplication by a q × q permutation matrix, P,
although in practice we apply the permutation without ever constructing P.
That is, the matrix P is only a notational convenience only.
The matrix P consists of permuted columns of the identity matrix, Iq ,
and it is easy to establish that the inverse permutation corresponds to mul-
tiplication by PT . Because multiplication by P or by PT simply re-orders the
components of a vector, the length of the vector is unchanged. Thus,

kPuk2 = kuk2 = kPT uk2 (5.21)

and we can express the penalty in (5.17) in any of these three forms. The
properties of P that it preserves lengths of vectors and that its transpose is
its inverse are summarized by stating that P is an orthogonal matrix.
The permutation represented by P is determined from the structure of
Λθ ZT ZΛθ + Iq for some initial value of θ . The particular value of θ does not
T

affect the result because the permutation depends only the positions of the
non-zeros, not the numerical values at these positions.
Taking into account the permutation, the sparse Cholesky factor, Lθ , is
defined to be the sparse, lower triangular, q × q matrix with positive diagonal
elements satisfying
 
Lθ LTθ = P Λθ
T T
Z ZΛ θ + I T
q P . (5.22)

Note that we now require that the diagonal elements of Λθ be positive. Prob-
lems˜5.1 and 5.2 indicate why we can require this. Because the diagonal ele-
ments of Λθ are positive, its determinant, |Λθ |, which, for a triangular matrix
such as Λθ , is simply the product of its diagonal elements, is also positive.
Many sparse matrix methods, including the sparse Cholesky decomposi-
tion, are performed in two stages: the symbolic phase in which the locations
of the non-zeros in the result are determined and the numeric phase in which
the numeric values at these positions are evaluated. The symbolic phase for
the decomposition (5.22), which includes determining the permutation, P,
need only be done once. Evaluation of Lθ for subsequent values of θ requires
only the numeric phase, which typically is much faster than the symbolic
phase.
The permutation, P, serves two purposes. The first and most important
purpose is to reduce the number of non-zeros in the factor, Lθ . The factor
is potentially non-zero at every non-zero location in the lower triangle of the

Page: 105 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

106 5 Computational Methods for Mixed Models

matrix being decomposed. However, as we saw in Fig.˜?? of Sect.˜2.1.2, there

may be positions in the factor that get filled-in even though they are known
to be zero in the matrix being decomposed. The fill-reducing permutation is
chosen according to certain heuristics to reduce the amount of fill-in. We use
the approximate minimal degree (AMD) method described in Davis [1996].
After the fill-reducing permutation is determined, a “post-ordering” is applied.
This has the effect of concentrating the non-zeros near the diagonal of the
factor. See Davis [2006] for details.
The pseudo-data representation of the PLS problem, (5.18), becomes
    2
yobs − Xβ ZΛ θ P T
ũ = arg min − Pu (5.23)
u 0 PT

and the system of linear equations satisfied by ũ is

 
Lθ LT
θ Pũ = P Λθ
T T
Z ZΛθ + I q PT Pũ = PΛθT ZT (yobs − Xβ ) . (5.24)

Obtaining the Cholesky factor, Lθ , may not seem to be great progress

toward determining ũ because we still must solve (5.24) for ũ. However, it
is the key to the computational methods in the lme4 package. The ability
to evaluate Lθ rapidly for many different values of θ is what makes the
computational methods in lme4 feasible, even when applied to very large
data sets with complex structure. Once we evaluate Lθ it is straightforward
to solve (5.24) for ũ because Lθ is triangular.
In Sect.˜5.6 we will describe the steps in determining this solution. First,
though, we should show that the solution, ũ, and the value of the objective
at the solution, rθ2 ,β , do allow us to evaluate the deviance.

5.4.2 The Value of the Deviance and Profiled Deviance

After evaluating Lθ and using that to solve for ũ, which also produces rβ2 ,θ ,
we can write the PRSS for a general u as

kyobs − Xβ − ZΛθ uk2 + kuk2 = rθ2 ,β + kLT

θ (u − ũ)k
2
(5.25)

which finally allows us to evaluate the likelihood. We plug the right hand side
of (5.25) into the definition of h(u) and apply the change of variable

LT
θ (u − ũ)
z= . (5.26)
σ
The determinant of the Jacobian of this transformation,

Page: 106 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

5.4 Determining the PLS Solutions, ũ and βb θ 107
   T 
 dz   Lθ  |Lθ |
 = = (5.27)
 du   σ  σq

is required for the change of variable in the integral. We use the letter z for the
transformed value because we will rearrange the integral to have the form of
the integral of the density of the standard multivariate normal distribution.
That is, we will use the result
Z 2
e−kzk /2
dz = 1. (5.28)
Rq (2π)q/2

Putting all these pieces together gives

Z
L(θ , β , σ ) = h(u) du
Rq
!
Z
1 rθ2 ,β + kLTθ (u − ũ)k2
= exp − du
R (2πσ )
q 2 (n+q)/2 2σ 2
 2 
rθ ,β
exp − 2σ Z
!
2
kLT ũ)k2
1 θ (u − |Lθ | du
= exp −
(2πσ 2 )n/2 Rq (2π)q/2 2σ 2 |Lθ | σ q (5.29)
 2 
rθ ,β
exp − 2σ 2 Z 2
e−kzk /2
= dz
(2πσ 2 )n/2 |Lθ | Rq (2π)q/2
 2 
rθ ,β
exp − 2σ 2
= .
(2πσ 2 )n/2 |Lθ |

The deviance can now be expressed as

2
rβ2 ,θ
d(θ , β , σ |yobs ) = −2 log (L(θ , β , σ |yobs )) = n log(2πσ ) + 2 log |Lθ | + ,
σ2
as stated in (1.6). The maximum likelihood estimates of the parameters are
those that minimize this deviance.
Equation (1.6) is a remarkably compact expression, considering that the
class of models to which it applies is very large indeed. However, we can
do better than this if we notice that β affects (1.6) only through rβ2 ,θ , and,
for any value of θ , minimizing this expression with respect to β is just an
extension of the penalized least squares problem. Let βb θ be the value of β
that minimizes the PRSS simultaneously with respect to β and u and let rθ2
be the PRSS at these minimizing values. If, in addition, we set σc2 = r2 /n,
θ θ
which is the value of σ 2 that minimizes the deviance for a given value of rθ2 ,
then the profiled deviance, which is a function of θ only, becomes

Page: 107 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

108 5 Computational Methods for Mixed Models
  
˜ 2πrθ2
d(θ |yobs ) = 2 log |Lθ | + n 1 + log . (5.30)
n
˜ |yobs ) with respect to θ de-
Numerical optimization (minimization) of d(θ
termines the MLE, θb . The MLEs for the other parameters, βb and σb , are the
corresponding conditional estimates evaluated at θb .

5.4.3 Determining rθ2 and βb θ

To determine ũ and βb θ simultaneously we rearrange the terms in (5.23) as

        2
ũ yobs
ZΛθ PT X Pu .
= arg min − (5.31)
βb θ u,β 0 PT 0 β

The PLS values, ũ and βb θ , are the solutions to


   T T 
P ΛθT ZT ZΛθ + Iq PT PΛθT ZT X Pũ PΛθ Z yobs
= (5.32)
XT ZΛθ PT XT X βb θ XT yobs .

To evaluate these solutions we decompose the system matrix as


   T 
P ΛθT ZT ZΛθ + Iq PT PΛθT ZT X Lθ 0 Lθ RZX
= T (5.33)
XT ZΛθ PT XT X RZX RTX 0 RX

where, as before, Lθ , the sparse Cholesky factor, is the sparse lower triangular
q × q matrix satisfying (5.22). The other two matrices in (5.33): RZX , which
is a general q × p matrix, and RX , which is an upper triangular p × p matrix,
satisfy
Lθ RZX = PΛθT ZT X (5.34)
and
RT T T
X RX = X X − RZX RZX (5.35)
Those familiar with standard ways of writing a Cholesky decomposition
as either LLT or RT R (L is the factor as it appears on the left and R is as
it appears on the right) will notice a notational inconsistency in (5.33). One
Cholesky factor is defined as the lower triangular fractor on the left and the
other is defined as the upper triangular factor on the right. It happens that
in R the Cholesky factor of a dense positive-definite matrix is returned as the
right factor, whereas the sparse Cholesky factor is returned as the left factor.
One other technical point that should be addressed is whether XT X −
RTZX RZX is positive definite. In theory, if X has full column rank, so that X X
T

is positive definite, then the downdated matrix, XT X − RT ZX RZX , must also

be positive definite (see Prob.˜5.4). In practice, the downdated matrix can

Page: 108 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

5.5 The REML Criterion 109

become computationally singular in ill-conditioned problems, in which case

an error is reported.
The extended decomposition (5.33) not only provides for the evaluation of
˜ ), (5.30) but also allows us to define and
the profiled deviance function, d(θ
evaluate the profiled REML criterion.

5.5 The REML Criterion

The so-called REML estimates of variance components are often preferred

to the maximum likelihood estimates. (“REML” can be considered to be an
acronym for “restricted” or “residual” maximum likelihood, although neither
term is completely accurate because these estimates do not maximize a like-
lihood.) We can motivate the use of the REML criterion by considering a
linear regression model,
Y ∼ N (Xβ , σ 2 In ), (5.36)
in which we typically estimate σ 2 as

b 2
c2 = kyobs − Xβ k
σ (5.37)
R
n− p

even though the maximum likelihood estimate of σ 2 is

b 2
c2 = kyobs − Xβ k .
σ (5.38)
L
n

The argument for preferring σ c2 to σc2 as an estimate of σ 2 is that the nu-

R L
merator in both estimates is the sum of squared residuals at βb and, although
the residual vector, yobs − Xβb , is an n-dimensional vector, the residual at θb
satisfies p linearly independent constraints, XT (yobs − Xβb ) = 0. That is, the
residual at θb is the projection of the observed response vector, yobs , into an
(n − p)-dimensional linear subspace of the n-dimensional response space. The
estimate σ c2 takes into account the fact that σ 2 is estimated from residuals
R
that have only n − p degrees of freedom.
Another argument often put forward for REML estimation is that σ c2 is an
R
unbiased estimate of σ 2 , in the sense that the expected value of the estimator
is equal to the value of the parameter. However, determining the expected
value of an estimator involves integrating with respect to the density of the
estimator and we have seen that densities of estimators of variances will be
skewed, often highly skewed. It is not clear why we should be interested in
the expected value of a highly skewed estimator. If we were to transform to a
more symmetric scale, such as the estimator of the standard deviation or the
estimator of the logarithm of the standard deviation, the REML estimator

Page: 109 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

110 5 Computational Methods for Mixed Models

would no longer be unbiased. Furthermore, this property of unbiasedness of

variance estimators does not generalize from the linear regression model to
linear mixed models. This is all to say that the distinction between REML
and ML estimates of variances and variance components is probably less
important that many people believe.
Nevertheless it is worthwhile seeing how the computational techniques
described in this chapter apply to the REML criterion because the REML
parameter estimates θb R and σc2 for a linear mixed model have the property
R
c
that they would specialize to σ 2 from (5.37) for a linear regression model, as
R
seen in Sect.˜1.3.2.
Although not usually derived in this way, the REML criterion (on the
deviance scale) can be expressed as
Z
dR (θ , σ |yobs ) = −2 log L(θ , β , σ |yobs ) dβ . (5.39)
Rp

The REML estimates θb R and σ c2 minimize d (θ , σ |y ).

R R obs
To evaluate this integral we form an expansion, similar to (5.25), of rθ2 ,β
about βb θ
rθ2 ,β = rθ2 + kRX (β − βb θ )k2 . (5.40)
In the same way that (5.25) was used to simplify the integral in (5.29), we
can derive  2   
rθ ,β rθ2
Z exp − 2σ 2 exp − 2σ 2
dβ = (5.41)
R p (2πσ 2 )n/2 |Lθ | (2πσ 2 )(n−p)/2 |Lθ ||RX |
corresponding to a REML criterion on the deviance scale of

rθ2
dR (θ , σ |yobs ) = (n − p) log(2πσ 2 ) + 2 log (|Lθ ||RX |) + . (5.42)
σ2

Plugging in the conditional REML estimate, σ c2 = r2 /(n − p), provides the

R θ
profiled REML criterion
  2 
2πr
d˜R (θ |yobs ) = 2 log (|Lθ ||RX |) + (n − p) 1 + log θ
(5.43)
n− p

The REML estimate of θ is

θb R = arg min d˜R (θ |yobs ), (5.44)

θ

and the REML estimate of σ 2 is the conditional REML estimate of σ 2 at θb R ,

c2 = r2 /(n − p).
σ (5.45)
R θb R

Page: 110 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

5.6 Step-by-step Evaluation of the Profiled Deviance 111

It is not entirely clear how one would define a “REML estimate” of β because
the REML criterion, dR (θ , σ |y), defined in (5.42), does not depend on β .
However, it is customary (and not unreasonable) to use βb R = βb θb as the
R
REML estimate of β .

5.6 Step-by-step Evaluation of the Profiled Deviance

As we have seen, an object returned by lmer contains an environment, ac-

cessed with the env extractor. This environment contains several matrices and
vectors that are used in the evaluation of the profiled deviance. In this section
we use these matrices and vectors from one of our examples to explicitly trace
the steps in evaluating the profiled deviance. This level of detail is provided
for those whose style of learning is more of a “hands on” style and for those
who may want to program modifications of this approach.
Consider our model fm08, fit as
> fm08 <- lmer(Reaction ~ 1 + Days + (1 + Days|Subject), sleepstudy,
+ REML=FALSE, verbose=1)

npt = 7 , n = 3
rhobeg = 0.2 , rhoend = 2e-07
0.020: 11: 1752.65; 1.00913 -0.0128508 0.200209
0.0020: 19: 1752.07;0.973276 0.0253858 0.209423
...

The environment of the model contains the converged parameter vector,

θ (theta), the relative covariance factor, Λθ (Lambda), the sparse Cholesky
factor, Lθ (L), the matrices RZX (RZX) and RX (RX), the conditional mode, ũ
(u), and the conditional estimate, βb θ (fixef). The permutation represented
by P is contained in the sparse Cholesky representation, L.
Although the model matrices, X (X) and ZT (Zt), and the response vector,
yobs (y), are available in the environment, many of the products that involve
only these fixed values are precomputed and stored separately under the
names XtX (XT X), Xty, ZtX and Zty.
To provide easy access to the objects in the environment of fm08 we at-
tach it to the search path. Please note that this is done here for illustration
only. The practice of attaching a list or a data frame or, less commonly, an
environment in an R session is overused, somewhat dangerous (because of
the potential of forgetting to detach it later) and discouraged. The preferred
practice is to use the with function to gain access by name to components
of such composite objects. For this section of code, however, using with or
within would quickly become very tedious and we use attach instead.
To update the matrix Λθ to a new value of θ we need to know which of
the non-zeros in Λ are updated from which elements of θ . Recall that the
dimension of θ is small (3, in this case) but Λ is potentially large (18 × 18

Page: 111 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

112 5 Computational Methods for Mixed Models

with 54 non-zeros). The environment contains an integer vector Lind that

maps the elements of theta to the non-zeros in Lambda.
Suppose we wish to recreate the evaluation of the profiled deviance at the
initial value of θ = (1, 0, 1). We begin by updating Λθ and forming the product
UT = ΛθT ZT
> str(fm08@re@Lambda)

Formal class 'dgCMatrix' [package "Matrix"] with 6 slots

..@ i : int [1:54] 0 1 1 2 3 3 4 5 5 6 ...
..@ p : int [1:37] 0 2 3 5 6 8 9 11 12 14 ...
..@ Dim : int [1:2] 36 36
..@ Dimnames:List of 2
.. ..$ : NULL
.. ..$ : NULL
..@ x : num [1:54] 0.9292 0.0182 0.2226 0.9292 0.0182 ...
..@ factors : list()

> str(fm08@re@Lind)

int [1:54] 1 2 3 1 2 3 1 2 3 1 ...

> fm08@re@Lambda@x[] <- c(1,0,1)[fm08@re@Lind]

> str(fm08@re@Lambda@x)

num [1:54] 1 0 1 1 0 1 1 0 1 1 ...

> Ut <- crossprod(fm08@re@Lambda, fm08@re@Zt)

The Cholesky factor object, L, can be updated from Ut without forming UT U+

I explicitly. The optional argument mult to the update method specifies a
multiple of the identity to be added to UT U
> L <- update(fm08@re@L, Ut, mult = 1)

Then we evaluate RZX and RX according to (5.34) and (5.35)

> RZX <- solve(L, solve(L, crossprod(fm08@re@Lambda, fm08@re@Zt %*% fm08@fe@X), sys = "P"), s

Solving (5.32) for ũ and βb θ is done in stages. Writing cu and cβ for the
intermediate results that satisfy
    T T 
Lθ 0 cu PΛθ Z yobs
= (5.46)
RTZX RX
T cβ XT yobs .

we evaluate
The next set of equations to solve is
 T    
Lθ RZX Pũ c
b = U (5.47)
0 RX βθ cβ

We can now create the conditional mean, mu, the penalized residual sum of
squares, prss, the logarithm of the square of the determinant of L, ldL2, and
the profiled deviance, which, fortuitously, equals the value shown earlier.

Page: 112 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

5.7 Generalizing to Other Forms of Mixed Models 113

The last step is detach the environment of fm08 from the search list to
avoid later name clashes.
In terms of the calculations performed, these steps describe exactly the
evaluation of the profiled deviance in lmer. The actual function for evaluating
the deviance, accessible as fm08@setPars, is a slightly modified version of what
is shown above. However, the modifications are only to avoid creating copies
of potentially large objects and to allow for cases where the model matrix, X,
is sparse. In practice, unless the optional argument compDev = FALSE is given,
the profiled deviance is evaluated in compiled code, providing a speed boost,
but the R code can be used if desired. This allows for checking the results
from the compiled code and can also be used as a template for extending the
computational methods to other types of models.

5.7 Generalizing to Other Forms of Mixed Models

In later chapters we cover the theory and practice of generalized linear mixed
models (GLMMs), nonlinear mixed models (NLMMs) and generalized non-
linear mixed models (GNLMMs). Because quite a bit of the theoretical and
computational methodology covered in this chapter extends to those models
we will cover the common aspects here.

5.7.1 Descriptions of the Model Forms

We apply the name “generalized” to models in which the conditional distri-

bution, (Y |U = u), is not required to be Gaussian but does preserve some
of the properties of the spherical Gaussian conditional distribution

(Y |U = u) ∼ N (ZΛθ u + Xβ , σ 2 In )

from the linear mixed model. In particular, the components of Y are con-
ditionally independent, given U = u. Furthermore, u affects the distribu-
tion only through the conditional mean, which we will continue to write
as µ, and it affects the conditional mean only through the linear predictor,
γ = ZΛθ u + Xβ .
Typically we do not have µ = γ, however. The elements of the linear pre-
dictor, γ, can be positive or negative or zero. Theoretically they can take
on any value between −∞ and ∞. But many distributional forms used in
GLMMs put constraints on the value of the mean. For example, the mean
of a Bernoulli random variable, modeling a binary response, must be in the
range 0 < µ < 1 and the mean of a Poisson random variable, modeling a count,
must be positive. To achieve these constraints we write the conditional mean,

Page: 113 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

114 5 Computational Methods for Mixed Models

µ as a transformation of the unbounded predictor, written η. For historical,

and some theoretical, reasons the inverse of this transformation is called the
link function, written
η = g(µ), (5.48)
and the transformation we want is called the inverse link, writted g−1 .
Both g and g−1 are determined by scalar functions, g and g−1 , respectively,
applied to the individual components of the vector argument. That is, η must
be n-dimensional and the vector-valued function µ = g−1 (η) is defined by the
component functions µi = g−1 (ηi ), i = 1, . . . , n. Among other things, this means
dµ
that the Jacobian matrix of the inverse link, dη , will be diagonal.
Because the link function, g, and the inverse link, g−1 , are nonlinear func-
tions (there would be no purpose in using a linear link function) many people
use the terms “generalized linear mixed model” and “nonlinear mixed model”
interchangably. We reserve the term “nonlinear mixed model” for the type
of models used, for example, in pharmacokinetics and pharmacodynamics,
where the conditional distribution is a spherical multivariate Gaussian

(Y |U = u) ∼ N (µ, σ 2 In ) (5.49)

but µ depends nonlinearly on γ. For NLMMs the length of the linear predic-
tor, γ, is a multiple, ns, of n, the length of µ.
Like the map from η to µ, the map from γ to µ has a “diagonal” property,
which we now describe. If we use γ to fill the columns of an n × s matrix,
Γ , then µi depends only on the ith row of Γ . In fact, µi is determined by a
nonlinear model function, f , applied to the i row of Γ . Writing µ = f(γ) based
on the component function f , we see that the Jacobian of f, dµ dγ , will be the
vertical concatenation of s diagonal n × n matrices.
Because we will allow for generalized nonlinear mixed models (GNLMMs),
in which the mapping from γ to µ has the form

γ → η → µ, (5.50)

we will use (5.50) in our definitions.

5.7.2 Determining the Conditional Mode, ũ

For all these types of mixed models, the conditional distribution, (U |Y =

yobs ), is a continuous distribution for which we can determine the unscaled
conditional density, h(u). As for linear mixed models, we define the condi-
tional mode, ũ as the value that maximizes the unscaled condtional density.
Determining the conditional mode, ũ, in a nonlinear mixed model is a
penalized nonlinear least squares (PNLS) problem

Page: 114 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

5.8 Chapter Summary 115

ũ = arg min kyobs − µk2 + kuk2 (5.51)

u

which we solve by adapting the iterative techniques, such as the Gauss-

Newton method˜[Bates and Watts, 1988, Sect.˜2.2.1], used for nonlinear least
squares. Starting at an initial value, u(0) , (the bracketed superscript denotes
the iteration number) with conditional mean, µ (0) , we determine an increment
δ (1) by solving the penalized linear least squares problem,
   (0)  2
yobs − µ (0) U
δ (1)
= arg min − δ (5.52)
δ 0−u Iq
(0)

where 
(0) dµ 
U = . (5.53)
du u(0)
(0)
Naturally, we use the sparse Cholesky decomposition, Lθ , satisfying
   T 
(0) (0)
Lθ Lθ = P U (0)
U + Iq PT
(0)
(5.54)

to determine this increment. The next iteration begins at

u(1) = u(0) + kδ (1) (5.55)

where k is the step factor chosen, perhaps by step-halving˜[Bates and Watts,

1988, Sect.˜2.2.1], to ensure that the penalized residual sum of squares de-
creases at each iteration. Convergence is declared when the orthogonality
convergence criterion˜[Bates and Watts, 1988, Sect.˜2.2.3] is below some pre-
specified tolerance.
The Laplace approximation to the deviance is

2
rθ2 ,β
d(θ , β , σ |yobs ) ≈ n log(2πσ ) + 2 log |Lθ ,β | + , (5.56)
σ2
where the Cholesky factor, Lθ ,β , and the penalized residual sum of squares,
rθ2 ,β , are both evaluated at the conditional mode, ũ. The Cholesky factor
depends on θ , β and u for these models but typically the dependence on β
and u is weak.

5.8 Chapter Summary

The definitions and the computational results for maximum likelihood estima-
tion of the parameters in linear mixed models were summarized in Sect.˜1.4.1.
A key computation is evaluation of the sparse Cholesky factor, Λθ , satisfying

Page: 115 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

116 5 Computational Methods for Mixed Models

eqn.˜5.22,  
Lθ LT
θ = P Λθ Z ZΛθ + Iq PT .
T T

where P represents the fill-reducing permutation determined during the sym-

bolic phase of the sparse Cholesky decomposition.
An extended decomposition (eqn.˜5.33) provides the q × p matrix RZX
and the p × p upper triangular RX that are used to determine the conditional
mode ũθ , the conditional estimate βb θ and the minimum penalized residual
sum of squares, rθ2 from which the profiled deviance
  2 
˜ |yobs ) = 2 log |Lθ | + n 1 + log 2πr θ
d(θ .
n

or the profile REML criterion

  
˜ 2πrθ2
dR (θ |yobs ) = 2 log (|Lθ ||RX |) + (n − p) 1 + log
n− p

can be evaluated and optimized (minimized) with respect to θ .

Exercises

Unlike the exercises in other chapters, these exercises establish theoretical

results, which do not always apply exactly to the computational results.

5.1. Show that the matrix Aθ = PΛθT ZT ZΛθ PT + Iq is positive definite. That
is, bT Ab > 0, ∀b 6= 0.

5.2. (a) Show that Λθ can be defined to have non-negative diagonal elements.
(Hint: Show that the product Λθ D where D is a diagonal matrix with
diagonal elements of ±1 is also a Cholesky factor. Thus the signs of the
diagonal elements can be chosen however we want.)
(b) Use the result of Prob.˜5.1 to show that the diagonal elements of Λθ must
be non-zero. (Hint: Suppose that the first zero on the diagonal of Λθ is in
the ith position. Show that there is a solution x to ΛθT x = 0 with xi = 1
and x j = 0, j = i + 1, . . . , q and that this x contradicts the positive definite
condition.)

5.3. Show that if X has full column rank, which means that there does not
exist a β 6= 0 for which Xβ = 0, then XT X is positive definite.

5.4. Show that if X has full column rank then

 
ZΛθ PT X
PT 0

Page: 116 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

5.8 Chapter Summary 117

  full column rank. (Hint: First show that u must be zero in

also must have
u
any vector satisfying
β
  
ZΛθ PT X u
= 0.
PT 0 β

Use this result and (5.33) to show that

  T 
Lθ 0 Lθ RZX
RTZX RX
T 0 RX

is positive definite and, hence, RX is non-singular.)

Page: 117 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

Page: 118 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10
Chapter 6
Generalized Linear Mixed Models for
Binary Responses

In this chapter we consider mixed-effects models for data sets in which the
response is binary, in the sense that it represents yes/no or true/false or
correct/incorrect responses.
Because the response can only take on one of two values we adapt our mod-
els to predict the probability of the positive response. We retain the concept
of the linear predictor, Xβ + Zb, depending on the fixed-effects parameters,
β , and the random effects, b, with the corresponding model matrices, X and
Z, determining the conditional mean, µ, of the response, given the random
effects, but the relationship is more general than that for the linear mixed
model. The linear predictor, which we shall write as η, determines µ accord-
ing to a link function, g. For historical reasons it is the function taking an
element of µ to the corresponding element of η that is called the link. The
transformation in the opposite direction, from η to µ, is called the inverse
link.
As described in earlier chapters, models based on a Gaussian distribution
for the response vector with its mean determined by the linear predictor
are called linear models. Models incorporating coefficients in a linear predic-
tor but allowing for more general forms of the distribution of the response
are called generalized linear models. When the linear predictor incorporates
random effects in addition to the fixed-effects parameters we call them gen-
eralized linear mixed models (GLMMs) and fit such models with the glmer
function. As in previous chapters, we will begin with an example to help
illustrate these ideas.

6.1 Artificial contraception use in regions of Bangladesh

One of the test data sets from the Center for Multilevel Modelling, University
of Bristol is derived from the 1989 Bangladesh Fertility Survey, [Huq and
Cleland, 1990]. The data are a subsample of responses from 1934 women

119
120 6 Generalized Linear Mixed Models for Binary Responses

grouped in 60 districts and are available as the Contraception data set in the
mlmRev package.

> str(Contraception)

'data.frame': 1934 obs. of 6 variables:

$ woman : Factor w/ 1934 levels "1","2","3","4",..: 1 2 3 4 5 6 7 8 9..
$ district: Factor w/ 60 levels "1","2","3","4",..: 1 1 1 1 1 1 1 1 1 1..
$ use : Factor w/ 2 levels "N","Y": 1 1 1 1 1 1 1 1 1 1 ...
$ livch : Factor w/ 4 levels "0","1","2","3+": 4 1 3 4 1 1 4 4 2 4 ...
$ age : num 18.44 -5.56 1.44 8.44 -13.56 ...
$ urban : Factor w/ 2 levels "N","Y": 2 2 2 2 2 2 2 2 2 2 ...

The response of interest is use — whether the woman chooses to use artificial
contraception. The covariates include the district in which the woman resides,
the number of live children she currently has, her age and whether she is in
a rural or an urban setting.
Note that the age variable is centered about a particular age so some values
are negative. Regretably, the information on what the centering age was does
not seem to be available.

6.1.1 Plotting the binary response

Producing informative graphical displays of a binary response as it relates

to covariates is somewhat more challenging that the corresponding plots for
responses on a continuous scale. If we were to plot the 1934 responses as 0/1
values versus, for example, the woman’s centered age, we would end up with
a rather uninformative plot because all the points would fall on one of two
horizontal lines.
One approach to illustrating the structure of the data more effectively is
to add scatterplot smoother lines (Fig.˜6.1) to show the trend in the response
with respect to the covariate. Once we have the smoother lines in such a plot
we can omit the data points themselves, as we did here, because they add
very little information.
The first thing to notice about the plot is that the proportion of women
using contraception is not linear in age, which, on reflection, makes sense. A
woman in the middle of this age range (probably corresponding to an age
around 25) is more likely to use artificial contraception than is a girl in her
early teens or a woman in her forties. We also see that women in an urban
setting are more likely to use contraception than those in a rural setting and
that women with no live children are less likely than women who have live
children. There do not seem to be strong differences between women who
have 1, 2 or 3 or more children compared to the differences between women
with children and those without children.
Interestingly, the quadratic pattern with respect to age does not seem
to have been noticed. Comparisons of model fits through different software

Page: 120 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

6.1 Artificial contraception use in regions of Bangladesh 121

0 1 2 3+
−10 0 10 20

N Y
1.0

0.8
Proportion

0.6

0.4

0.2

0.0

−10 0 10 20

Centered age

Fig. 6.1 Contraception use versus centered age for women in the Bangladesh Fertility
Survey 1989. Panels are determined by whether the woman is in an urban setting or
not. Lines within the panels are scatterplot smoother lines for women with 0, 1, 2
and 3 or more live children.

systems, as provided by the Center for Multilevel Modelling, incorporate only

a linear term in age, even though the pattern is clearly nonlinear. The lesson
here is similar to what we have seen in other examples; careful plotting of the
data should, whenever possible, precede attempts to fit models to the data.

6.1.2 Initial GLMM fit to the contraception data

As for the lmer function, the first two arguments to the glmer function for
fitting generalized linear mixed models are the model formula and the name
of the data frame. The third argument to glmer, named family, describes the
type of conditional distribution of the response given the random effects. Ac-
tually, as the name family implies, it contains more information than just the
distribution type in that each distribution and link are described by several
functions. Certain distributions, including the binomial, have canonical link
functions associated with them and if we specify just the distribution type
we get the family with the canonical link. Thus our initial fit is generated as
> fm10 <- glmer(use ~ 1+age+I(age^2)+urban+livch+(1|district),
+ Contraception, binomial)

When displaying a fitted model like this that has several fixed-effects co-
efficients it is helpful to specify the optional argument corr=FALSE to suppress
printing of the rather large correlation matrix of the fixed effects estimators.
> print(fm10, corr=FALSE)

Page: 121 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

122 6 Generalized Linear Mixed Models for Binary Responses

Generalized linear mixed model fit by maximum likelihood ['merMod']

Family: binomial
Formula: use ~ 1 + age + I(age^2) + urban + livch + (1 | district)
Data: Contraception
AIC BIC logLik deviance
2388.728 2433.267 -1186.364 2372.728
Random effects:
Groups Name Variance Std.Dev.
district (Intercept) 0.2261 0.4755
Number of obs: 1934, groups: district, 60

Fixed effects:
Estimate Std. Error z value
(Intercept) -1.0353439 0.1743809 -5.937
age 0.0035350 0.0092314 0.383
I(age^2) -0.0045624 0.0007252 -6.291
urbanY 0.6972788 0.1198849 5.816
livch1 0.8150294 0.1621961 5.025
livch2 0.9164494 0.1851060 4.951
livch3+ 0.9150402 0.1857760 4.926

The interpretation of the coefficients in this model is somewhat different

from the linear mixed models coefficients that we examined previously but
many of the model-building steps are similar. A rough assessment of the
utility of a particular term in the fixed-effects part of the model can be
obtained from examining the estimates of the coefficients associated with it
and their standard errors. To test whether a particular term is useful we omit
it from the model, refit and compare the reduced model fit to the original
according to the change in deviance.
We will examine the terms in the model first and discuss the interpretation
of the coefficients in Sec.˜6.2.
Recall from Chap.˜4 that the default set of contrasts for a factor such as
livch is offsets relative to the reference level, in this case women who do not
have any live children. The coefficients labeled livch1, livch2 and livch3+ are
all large relative to their standard errors but they are also close similar to
each other. This confirms our earlier impression that the main distinction is
between women with children and those without and, for those who do have
children, the number of children is not an important distinction.
After incorporating a new variable ch — an indicator of whether the woman
has any children — in the data
> Contraception <- within(Contraception,
+ ch <- factor(livch != 0, labels = c("N", "Y")))

we fit a reduced model, fm11, with summary

Generalized linear mixed model fit by maximum likelihood ['merMod']
Family: binomial
Formula: use ~ age + I(age^2) + urban + ch + (1 | district)
Data: Contraception
AIC BIC logLik deviance
2385.200 2418.604 -1186.600 2373.200

Page: 122 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

6.1 Artificial contraception use in regions of Bangladesh 123

N Y
−10 0 10 20

N Y
1.0

0.8
Proportion

0.6

0.4

0.2

0.0

−10 0 10 20

Centered age

Fig. 6.2 Contraception use versus centered age for women in the Bangladesh Fertility
Survey 1989. Panels are determined by whether the woman is in an urban setting or
not. Lines within the panels are scatterplot smoother lines for women without children
and women with one or more live children.

Random effects:
Groups Name Variance Std.Dev.
district (Intercept) 0.2251 0.4744
Number of obs: 1934, groups: district, 60

Fixed effects:
Estimate Std. Error z value
(Intercept) -0.9876044 0.1677069 -5.889
age 0.0067102 0.0078362 0.856
I(age^2) -0.0046674 0.0007161 -6.518
urbanY 0.6834724 0.1196144 5.714
chY 0.8479485 0.1471716 5.762

Comparing this model to the previous model

> anova(fm11,fm10)

Data: Contraception
Models:
fm11: use ~ age + I(age^2) + urban + ch + (1 | district)
fm10: use ~ 1 + age + I(age^2) + urban + livch + (1 | district)
Df AIC BIC logLik Chisq Chi Df Pr(>Chisq)
fm11 6 2385.2 2418.6 -1186.6
fm10 8 2388.7 2433.3 -1186.4 0.4722 2 0.7897

indicates that the reduced model is adequate.

A plot of the smoothed observed proportions versus centered age according
to ch and urban (Fig.˜6.2) indicates that all four groups have a quadratic trend
with respect to age but the location of the peak proportion is shifted for those
without children relative to those with children. Incorporating an interaction
of age and ch allows for such a shift.

Page: 123 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

124 6 Generalized Linear Mixed Models for Binary Responses

Generalized linear mixed model fit by maximum likelihood ['merMod']

Family: binomial
Formula: use ~ age * ch + I(age^2) + urban + (1 | district)
Data: Contraception
AIC BIC logLik deviance
2379.202 2418.173 -1182.601 2365.202
Random effects:
Groups Name Variance Std.Dev.
district (Intercept) 0.2231 0.4723
Number of obs: 1934, groups: district, 60

Fixed effects:
Estimate Std. Error z value
(Intercept) -1.2998180 0.2134408 -6.090
age -0.0459409 0.0217227 -2.115
chY 1.1891294 0.2058998 5.775
I(age^2) -0.0057677 0.0008351 -6.907
urbanY 0.7011683 0.1201614 5.835
age:chY 0.0672373 0.0253180 2.656

Comparing this fitted model to the previous one

> anova(fm11, fm12)

Data: Contraception
Models:
fm11: use ~ age + I(age^2) + urban + ch + (1 | district)
fm12: use ~ age * ch + I(age^2) + urban + (1 | district)
Df AIC BIC logLik Chisq Chi Df Pr(>Chisq)
fm11 6 2385.2 2418.6 -1186.6
fm12 7 2379.2 2418.2 -1182.6 7.9983 1 0.004682

confirms the usefulness of this term.

Continuing with the model-building we turn our attention to the random
effects specification to see whether urban/rural differences vary significantly
between districts and whether the distinction between childless women and
women with children varies between districts. We fit a succession of models,
described in the exercises for this chapter, before settling on
Generalized linear mixed model fit by maximum likelihood ['merMod']
Family: binomial
Formula: use ~ age * ch + I(age^2) + urban + (1 | urban:district)
Data: Contraception
AIC BIC logLik deviance
2368.504 2407.475 -1177.252 2354.504
Random effects:
Groups Name Variance Std.Dev.
urban:district (Intercept) 0.3238 0.569
Number of obs: 1934, groups: urban:district, 102

Fixed effects:
Estimate Std. Error z value
(Intercept) -1.3077641 0.2199094 -5.947
age -0.0442332 0.0218690 -2.023

Page: 124 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

6.2 Link functions and interpreting coefficients 125

chY 1.1878907 0.2075693 5.723

I(age^2) -0.0056381 0.0008428 -6.689
urbanY 0.7646328 0.1706424 4.481
age:chY 0.0647256 0.0254839 2.540

Correlation of Fixed Effects:

(Intr) age chY I(g^2) urbanY
age 0.702
chY -0.864 -0.790
I(age^2) -0.096 0.297 -0.093
urbanY -0.334 -0.056 0.082 -0.015
age:chY -0.579 -0.929 0.674 -0.495 0.049

Data: Contraception
Models:
fm12: use ~ age * ch + I(age^2) + urban + (1 | district)
fm13: use ~ age * ch + I(age^2) + urban + (1 | urban:district)
Df AIC BIC logLik Chisq Chi Df Pr(>Chisq)
fm12 7 2379.2 2418.2 -1182.6
fm13 7 2368.5 2407.5 -1177.2 10.698 0 < 2.2e-16

Notice that although there are 60 distinct districts there are only 102
distinct combinations of urban:district represented in the data. In 15 of the
60 districts there are no rural women in the sample and in 3 districts there
are no urban women in the sample, as shown in
> xtabs(~ urban + district, Contraception)

district
urban 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
N 54 20 0 19 37 58 18 35 20 13 21 23 16 17 14 18 24
Y 63 0 2 11 2 7 0 2 3 0 0 6 8 101 8 2 0
district
urban 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
N 33 22 15 10 20 15 14 49 13 39 45 25 45 27 24 7 26
Y 14 4 0 8 0 0 0 18 0 5 4 7 16 6 0 7 9
district
urban 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51
N 28 14 13 7 24 12 23 6 28 27 34 74 9 26 4 15 20
Y 20 3 0 7 2 29 3 5 17 0 5 12 6 16 0 4 17
district
urban 52 53 55 56 57 58 59 60 61
N 42 0 0 24 23 20 10 22 31
Y 19 19 6 21 4 13 0 10 11

6.2 Link functions and interpreting coefficients

To this point the only difference we have encountered between glmer and lmer
as model-fitting functions is the need to specify the distribution family in a
call to glmer. The formula specification is identical and the assessment of the
significance of terms using likelihood ratio tests is similar. This is intentional.

Page: 125 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

126 6 Generalized Linear Mixed Models for Binary Responses

We have emphasized the use of likelihood ratio tests on terms, whether fixed-
effects or random-effects terms, exactly so the approach will be general.
However, the interpretation of the coefficient estimates in the different
types of models is different. In a linear mixed model the linear predictor is
the conditional mean (or “expected value”) of the response given the random
effects. That is, if we assume that we know the values of the fixed-effects
parameters and the random effects then the expected response for a particular
combination of covariate values is the linear predictor. Individual coefficients
can be interpreted as slopes of the fitted response with respect to a numeric
covariate or shifts between levels of a categorical covariate.
To interpret the estimates of coefficients in a GLMM we must define and
examine the link function that we mentioned earlier.

6.2.1 The logit link function for binary responses

The probability model for a binary response is the Bernoulli distribution,

which is about the simplest probability distribution we can concoct. There
are only two possible values: 0 and 1. If the probability of the response 1
is p then the probability of 0 must be 1 − p. It is easy to establish that
the expected value is also p. For consistency across distribution families we
write this expected response as µ instead of p. We should, however, keep in
mind that µ corresponds to a probability for this distribution and hence must
satisfy 0 ≤ µ ≤ 1.
In general we don’t want to have restrictions on the values of the linear
predictor so we equate the linear predictor to a function of µ that has an
unrestricted range. In the case of the Bernoulli distribution with the canonical
link function we equate the linear predictor to the log odds or logit of the
positive response. That is
 
µ
η = logit(µ) = log . (6.1)
1−µ

To understand why this is called the “log odds” recall that µ corresponds
µ
to a probability in [0, 1]. The corresponding the odds ratio, 1−µ , is in [0, ∞)
and the logarithm of the odds ratio, logit(µ), is in (−∞, ∞).
The inverse of the logit link function,
1
µ= (6.2)
1 + exp(−η)

shown in Fig.˜6.3, takes a value on the unrestricted range, (−∞, ∞), and maps
it to the probability range, [0, 1]. It happens this function is also the cumu-
lative distribution function for the standard logistic distribution, available in
R as the function plogit. In some presentations the relationship between the

Page: 126 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

6.2 Link functions and interpreting coefficients 127

1.0

1 + exp(− η)
0.8

0.6
1

0.4
µ=

0.2

0.0

−5 0 5
η

Fig. 6.3 Inverse of the logit link function. The linear predictor value, η, which is on
an unrestricted scale, is mapped to µ on the probability scale, [0, 1].

logit link and the logistic distribution is emphasized but that often leads to
questions of why we should focus on the logistic distribution. Also, it is not
clear how this approach would generalize to other distributions such as the
Poisson or the Gamma distributions.
A way of deriving the logit link that does generalize to a class of common
distributions in what is called the exponential family is to consider the loga-
rithm of the probability function (for discrete distributions) or the probability
density function (for continuous distributions). The probability function for
the Bernoulli distribution is µ for y = 1 and 1 − µ for y = 0. If we write this in
a somewhat peculiar way as µ y + (1 − µ)1−y for y ∈ {0, 1} then the logarithm
of the probability function becomes
 
y 1−y

µ
log µ + (1 − µ) = log(1 − µ) + y log . (6.3)
1−µ

Notice that the logit link function is the multiple of y in the last term.
For members of the exponential family the logarithm of the probability
or probability density function can be expressed as a sum of up to three
terms: one that involves y only, one that involves the parameters only and the
product of y and a function of the parameters. This function is the canonical
link. −µ y
In the case of the Poisson distribution the probability function is e y!µ for
y ∈ {0, 1, 2, . . . } so the log probability function is

− log(y!) − µ + y log(µ). (6.4)

and the canonical link function is log(µ).

Returning to the interpretation of the estimated coefficients in model fm13
we apply exactly the same interpretation as for a linear mixed model but
taking into account that slopes or differences in levels are with respect to
the logit or log-odds function. If we wish to express results in the probability

Page: 127 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

128 6 Generalized Linear Mixed Models for Binary Responses

scale then we should apply the plogit function to whatever combination of

coefficients is of interest to us.
For example, we see from Fig.˜6.2 that the observed proportion of childless
women with a centered age of 0 living in a rural setting who use artificial
contraception is about 20%. The fitted value of the log-odds for a typical
district (i.e. with a random effect of zero) is -1.30776 corresponding to a
fitted probability of
> plogis(fixef(fm13)[[1]])

[1] 0.2128612

or 21.3%.
Similarly the predicted log-odds of a childless woman with a centered age
of 0 in an urban setting of a typical district using artificial contraception is
> sum(fixef(fm13)[c("(Intercept)","urbanY")])

[1] -0.5431313

corresponding to a probability of
> plogis(sum(fixef(fm13)[c("(Intercept)","urbanY")]))

[1] 0.3674595

The predicted log-odds and predicted probability for a woman with children
and at the same age and location are
> logodds <- sum(fixef(fm13)[c("(Intercept)","chY","urbanY")])
> c("log-odds"=logodds, "probability"=plogis(logodds))

log-odds probability
0.6447594 0.6558285

We should also be aware that the random effects are defined on the linear
predictor scale and not on the probability scale.

Page: 128 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

References

Douglas˜M. Bates and Donald˜G. Watts. Nonlinear Regression Analysis and Its
Applications. Wiley, Hoboken, NJ, 1988. ISBN 0-471-81643-4.
Gregory Belenky, Nancy˜J. Wesensten, David˜R. Thorne, Maria˜L. Thomas, He-
len˜C. Sing, Daniel˜P. Redmond, Michael˜B. Russo, and Thomas˜J. Balkin. Pat-
terns of performance degradation and restoration during sleep restriction and sub-
sequent recovery: a sleep dose-response study. Journal of Sleep Research, 12:1–12,
2003.
G.E.P. Box and G.C. Tiao. Bayesian Inference in Statistical Analysis. Addison-
Wesley, Reading, MA, 1973.
Bill Cleveland. Visualizing Data. Hobart Press, Summit, NJ, 1993.
Owen˜L. Davies and Peter˜L. Goldsmith, editors. Statistical Methods in Research
and Production. Hafner, 4th edition, 1972.
Tim Davis. An approximate minimal degree ordering algorithm. SIAM J. Matrix
Analysis and Applications, 17(4):886–905, 1996.
Tim Davis. Direct Methods for Sparse Linear Systems. SIAM, Philadelphia, PA,
2006.
N.˜M. Huq and J.˜Cleland. Bangladesh fertility survey 1989 (main report). Tech-
nical report, National Institute of Population Research and Training, Dhaka,
Bangladesh, 1990.
Friedrich Leisch. Sweave: Dynamic generation of statistical reports using literate data
analysis. In Wolfgang Härdle and Bernd Rönz, editors, Compstat 2002 — Proceed-
ings in Computational Statistics, pages 575–580. Physica Verlag, Heidelberg, 2002.
URL http://www.stat.uni-muenchen.de/~leisch/Sweave. ISBN 3-7908-1517-9.
George˜A. Milliken and Dallas˜E. Johnson. Analysis of Messy Data: Volume 1,
Designed Experiments. CRC Press, 2nd edition, 2009.
José˜C. Pinheiro and Douglas˜M. Bates. Mixed-effects Models in S and S-PLUS.
Springer, 2000.
J.˜Rasbash, W.˜Browne, H.˜Goldstein, M.˜Yang, and I.˜Plewis. A User’s Guide to
MLwiN. Multilevel Models Project, Institute of Education, University of London,
London, 2000.
Stephen˜W. Raudenbush and Anthony˜S. Bryk. Hierarchical Linear Models: Appli-
cations and Data Analysis Methods. Sage, 2nd edition, 2002. ISBN 0-7619-1904-X.
Y.˜Sakamoto, M.˜Ishiguro, and G.˜Kitagawa. Akaike Information Criterion Statis-
tics. Reidel, Dordrecht, Holland, 1986.
Deepayan Sarkar. Lattice: Multivariate Data Visualization with R. Springer, 2008.
G.˜Schwarz. Estimating the dimension of a model. Annals of Statistics, 6:461–464,
1978.

129
130 References

Brady˜T. West, Kathleen˜B. Welch, and Andrzej˜T. Galecki. Linear Mixed Models:
A Practical Guide Using Statistical Software. Chapman and Hall/CRC, Boca
Raton, FL, 2007. ISBN 1-58488-480-0.

Page: 130 job: lMMwR macro: svmono.cls date/time: 25-Jun-2010/17:10

Index

binary response, 119 fm13, 125

Center for Multilevel Modelling, 119 generalized linear model, 119

Cleveland,William, 59 glm, 119
Contraception data, 120 glmer, 119
GLMM, 119
data set
Contraception, 120
response
fitted models binary, 119
fm10, 121
fm11, 123 Sarkar,Deepayan, 59
fm12, 124 scatterplot smoother, 120

131