Download the full version of the textbook now at textbookfull.

com

Advanced linear modeling statistical learning

and dependent data 3rd Edition Christensen R

https://textbookfull.com/product/advanced-linear-
modeling-statistical-learning-and-dependent-
data-3rd-edition-christensen-r/

Explore and download more textbook at https://textbookfull.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

Advanced R Statistical Programming and Data Models:

Analysis, Machine Learning, and Visualization 1st Edition
Matt Wiley
https://textbookfull.com/product/advanced-r-statistical-programming-
and-data-models-analysis-machine-learning-and-visualization-1st-
edition-matt-wiley/
textbookfull.com

Functional Data Structures in R: Advanced Statistical

Programming in R Mailund

https://textbookfull.com/product/functional-data-structures-in-r-
advanced-statistical-programming-in-r-mailund/

textbookfull.com

Functional Data Structures in R: Advanced Statistical

Programming in R Thomas Mailund

https://textbookfull.com/product/functional-data-structures-in-r-
advanced-statistical-programming-in-r-thomas-mailund/

textbookfull.com

Dressed in dreams a black girl s love letter to the power

of fashion First Edition Ford

https://textbookfull.com/product/dressed-in-dreams-a-black-girl-s-
love-letter-to-the-power-of-fashion-first-edition-ford/

textbookfull.com
You Can’t Know It All: Leading in the Age of Deep
Expertise Wanda T. Wallace

https://textbookfull.com/product/you-cant-know-it-all-leading-in-the-
age-of-deep-expertise-wanda-t-wallace/

textbookfull.com

The Uyghur Community: Diaspora, Identity and Geopolitics

1st Edition Güljanat Kurmangaliyeva Ercilasun

https://textbookfull.com/product/the-uyghur-community-diaspora-
identity-and-geopolitics-1st-edition-guljanat-kurmangaliyeva-
ercilasun/
textbookfull.com

Coulson and Richardson’s Chemical Engineering, Fourth

Edition: Volume 3A: Chemical and Biochemical Reactors and
Reaction Engineering R. Ravi
https://textbookfull.com/product/coulson-and-richardsons-chemical-
engineering-fourth-edition-volume-3a-chemical-and-biochemical-
reactors-and-reaction-engineering-r-ravi/
textbookfull.com

Pharmacotherapy A Pathophysiologic Approach Eleventh

Edition Joseph Dipiro Gary Yee L Michael Posey Stuart T
Haines Thomas D Nolin Vicki Ellingrod
https://textbookfull.com/product/pharmacotherapy-a-pathophysiologic-
approach-eleventh-edition-joseph-dipiro-gary-yee-l-michael-posey-
stuart-t-haines-thomas-d-nolin-vicki-ellingrod/
textbookfull.com

Image Analysis and Recognition 17th International

Conference ICIAR 2020 Póvoa de Varzim Portugal June 24 26
2020 Proceedings Part I Aurélio Campilho
https://textbookfull.com/product/image-analysis-and-recognition-17th-
international-conference-iciar-2020-povoa-de-varzim-portugal-
june-24-26-2020-proceedings-part-i-aurelio-campilho/
textbookfull.com
Incremental Software Architecture A Method for Saving
Failing IT Implementations 1st Edition Michael Bell

https://textbookfull.com/product/incremental-software-architecture-a-
method-for-saving-failing-it-implementations-1st-edition-michael-bell/

textbookfull.com
Springer Texts in Statistics

Ronald Christensen

Advanced
Linear
Modeling
Statistical Learning and Dependent Data
Third Edition
Springer Texts in Statistics

Series Editors
G. Allen, Department of Statistics, Houston, TX, USA
R. De Veaux, Department of Mathematics and Statistics, Williams College,
Williamstown, MA, USA
R. Nugent, Department of Statistics, Carnegie Mellon University, Pittsburgh,
PA, USA
Springer Texts in Statistics (STS) includes advanced textbooks from 3rd- to 4th-year
undergraduate courses to 1st- to 2nd-year graduate courses. Exercise sets should be
included. The series editors are currently Genevera I. Allen, Richard D. De Veaux,
and Rebecca Nugent. Stephen Fienberg, George Casella, and Ingram Olkin were
editors of the series for many years.

More information about this series at http://www.springer.com/series/417

Ronald Christensen

Advanced Linear Modeling

Statistical Learning and Dependent Data

Third Edition
Ronald Christensen
Department of Mathematics and Statistics
University of New Mexico
Albuquerque, NM, USA

ISSN 1431-875X ISSN 2197-4136 (electronic)

Springer Texts in Statistics
ISBN 978-3-030-29163-1 ISBN 978-3-030-29164-8 (eBook)
https://doi.org/10.1007/978-3-030-29164-8

© Springer Nature Switzerland AG 1991, 2001, 2019

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology
now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG.
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
To Pete, Russ, and Scot,
my pals from high school;

also

Wes, Ed, and everyone from graduate school.

Preface to the Third Edition

This is the third edition of Advanced Linear Modeling (ALM). It is roughly 50%
longer than the previous edition. It discusses the extension of linear models into
areas beyond those usually addressed in regression and analysis of variance. As in
previous editions, its primary emphasis is on models in which the data display some
form of dependence and many of the changes from the previous edition were made
to systematize this emphasis on dependent data. Nonetheless, it begins with topics
in modern regression analysis related to nonparametric regression and penalized
estimation (regularization). R code for the analyses in the book is available at http://
www.stat.unm.edu/∼fletcher/R-ALMIII.pdf.
Mathematical background is contained in Appendix A on differentiation and Kro-
necker products. Also some notation used throughout the book is set in Sect. 1.1.
This edition has been written in conjunction with the fifth edition of Christensen
(2011), often hereafter referred to as PA. Some discussions that previously appeared
in PA have been moved here. Obviously, you cannot do advanced linear modeling
without previously learning about linear modeling. I have tried to make this book
readable to people who have studied linear model theory from sources other than
PA, but I need to cite some source for basic results on linear models, so obviously I
cite PA. In cases where I need to cite results for which the new version of PA is dif-
ferent from the previous edition(s), the citations are given as PA-V. I have rearranged
the topics from the previous edition of ALM so that the material related to indepen-
dent data comes first followed by the material on dependent data. The chapter on
response surfaces has been dropped but is available in a new volume downloadable
from my website: http://www.stat.unm.edu/∼fletcher/TopicsInDesign. Some famil-
iarity with inner products is assumed, especially in Chaps. 1 and 3. The required
familiarity can be acquired from PA.
Chapter 1 expands the previous introduction to nonparametric regression. The
discussion follows what is commonly known as the basis function approach, despite
the fact that many of the techniques do not actually involve the use of basis functions
per se. In fact, when dealing with spaces of functions the very idea of a basis is
subject to competing definitions. Tim Hanson pointed out to me the obvious fact
that if a group of functions are linearly independent, they always form a basis for
vii
viii Preface to the Third Edition

the space that they span, but I think that in nonparametric regression the idea is to
approximate wider collections of functions than just these spanning sets. Chapter 1
now also includes a short introduction to models involving an entire function of
predictor variables.
Chapter 2 is an expanded version of the discussion of penalized regression from
Christensen (2011). A new Chap. 3 extends this by introducing reproducing kernel
Hilbert spaces.
Chapter 4 is new except for the last section. It gives results on an extremely gen-
eral linear model for dependent or heteroscedastic data. It owes an obvious debt to
Christensen (2011, Chapter 12). It contains several particularly useful exercises. In
a standard course on linear model theory, the theory of estimation and testing for
dependent data is typically introduced but not developed, see for example Chris-
tensen (2011, Sections 2.7 and 3.8). Section 4.1 of this book reviews, but does not
re-prove, those results. This book then applies those fundamental results to develop
theory for a wide variety of practical models.
I finally figured out how, without overwhelming the ideas in abstruse notation, to
present MINQUE as linear modeling, so I have done that in Chap. 4. In a technical
subsection, I give in to the abstruse notation so as to derive the MINQUE equations.
Previously, I just referred the reader to Rao for the derivation.
Chapter 5 on mixed models originally appeared in PA. It has been shortened in
places due of overlap with Chap. 4 but includes several new examples and exer-
cises. It contains a new emphasis on linear covariance structures that leads not only
to variance component models but the new Sect. 5.6 that examines a quite general
longitudinal data model. The details of the recovery of interblock information for a
balanced incomplete block design from PA no longer seem relevant, so they were
relegated, along with the response surface material, to the volume on my website.
Chapters 6 and 7 introduce time series: first the frequency domain which uses
models from Chap. 1 but with random effects as in Chap. 5 and then the time domain
approach which can be viewed as applications of ideas from the frequency domain.
Chapter 8 on spatial data is little changed from the previous edition. Mostly, the
references have been updated.
The former chapter on multivariate models has been split into three: Chap. 9 on
general theory with a new section relating multivariate models to spatial and time se-
ries models and a new discussion of multiple comparisons, Chap. 10 on applications
to specific models, and Chap. 11 with an expanded discussion of generalized mul-
tivariate linear models (also known as generalized multivariate analysis of variance
(GMANOVA) and growth curve models).
Chapters 12 and 14 are updated versions of the previous chapters on discriminant
analysis and principal components. Chapter 13 is a new chapter on binary regres-
sion and discrimination. Its raison d’être is that it devotes considerable attention to
support vector machines. Chapter 14 contains a new section on classical multidi-
mensional scaling.
From time to time, I mention the virtues of Bayesian approaches to problems
discussed in the book. One place to look for more information is BIDA, i.e., Chris-
tensen, Johnson, Branscum, and Hanson (2010).
Preface to the Third Edition ix

Thanks to my son Fletcher who is always the first person I ask when I have
doubts. Joe Cavanaugh and Mohammad Hattab have been particularly helpful as
have Tim Hanson, Wes Johnson, and Ed Bedrick. Finally, my thanks to Al Nosedal-
Sanchez, Curt Storlie, and Thomas Lee for letting me modify our joint paper into
Chap. 3.
As I have mentioned elsewhere, the large number of references to my other works
is as much about sloth as it is ego. In some sense, with the exception of BIDA, all of
my books are variations on a theme.

Albuquerque, NM, USA Ronald Christensen

February 2019
Preface to the Second Edition

This is the second edition of Linear Models for Multivariate, Time Series and Spatial
Data. It has a new title to indicate that it contains much new material. The primary
changes are the addition of two new chapters: one on nonparametric regression and
the other on response surface maximization. As before, the presentations focus on
the linear model aspects of the subject. For example, in the nonparametric regression
chapter there is very little about kernel regression estimation but quite a bit about
series approximations, splines, and regression trees, all of which can be viewed as
linear modeling.
The new edition also includes various smaller changes. Of particular note are a
subsection in Chap. 1 on modeling longitudinal (repeated measures) data and a sec-
tion in Chap. 6 on covariance structures for spatial lattice data. I would like to thank
Dale Zimmerman for the suggestion of incorporating material on spatial lattices.
Another change is that the subject index is now entirely alphabetical.

Albuquerque, NM, USA Ronald Christensen

May 9, 2000

xi
Visit https://textbookfull.com
now to explore a rich
collection of eBooks, textbook
and enjoy exciting offers!
Preface to the First Edition

This is a companion volume to Plane Answers to Complex Questions: The Theory of

Linear Models. It consists of six additional chapters written in the same spirit as the
last six chapters of the earlier book. Brief introductions are given to topics related
to linear model theory. No attempt is made to give a comprehensive treatment of the
topics. Such an effort would be futile. Each chapter is on a topic so broad that an
in-depth discussion would require a book-length treatment.
People need to impose structure on the world in order to understand it. There
is a limit to the number of unrelated facts that anyone can remember. If ideas can
be put within a broad, sophisticatedly simple structure, not only are they easier to
remember but often new insights become available. In fact, sophisticatedly simple
models of the world may be the only ones that work. I have often heard Arnold
Zellner say that, to the best of his knowledge, this is true in econometrics. The
process of modeling is fundamental to understanding the world.
In Statistics, the most widely used models revolve around linear structures. Often
the linear structure is exploited in ways that are peculiar to the subject matter. Cer-
tainly, this is true of frequency domain time series and geostatistics. The purpose
of this volume is to take three fundamental ideas from standard linear model theory
and exploit their properties in examining multivariate, time series and spatial data.
In decreasing order of importance to the presentation, the three ideas are: best linear
prediction, projections, and Mahalanobis distance. (Actually, Mahalanobis distance
is a fundamentally multivariate idea that has been appropriated for use in linear
models.) Numerous references to results in Plane Answers are made. Nevertheless,
I have tried to make this book as independent as possible. Typically, when a result
from Plane Answers is needed not only is the reference given but also the result it-
self. Of course, for proofs of these results the reader will have to refer to the original
source.
I want to reemphasize that this is a book about linear models. It is not traditional
multivariate analysis, time series, or geostatistics. Multivariate linear models are
viewed as linear models with a nondiagonal covariance matrix. Discriminant anal-
ysis is related to the Mahalanobis distance and multivariate analysis of variance.
Principal components are best linear predictors. Frequency domain time series in-
xiii
xiv Preface to the First Edition

volves linear models with a peculiar design matrix. Time domain analysis involves
models that are linear in the parameters but have random design matrices. Best lin-
ear predictors are used for forecasting time series; they are also fundamental to the
estimation techniques used in time domain analysis. Spatial data analysis involves
linear models in which the covariance matrix is modeled from the data; a primary
objective in analyzing spatial data is making best linear unbiased predictions of
future observables. While other approaches to these problems may yield different
insights, there is value in having a unified approach to looking at these problems.
Developing such a unified approach is the purpose of this book.
There are two well-known models with linear structure that are conspicuous by
their absence in my two volumes on linear models. One is Cox’s (1972) proportional
hazards model. The other is the generalized linear model of Nelder and Wedderburn
(1972). The proportional hazards methodology is a fundamentally nonparametric
technique for dealing with censored data having linear structure. The emphasis on
nonparametrics and censored data would make its inclusion here awkward. The in-
terested reader can see Kalbfleisch and Prentice (1980). Generalized linear models
allow the extension of linear model ideas to many situations that involve indepen-
dent non-normally distributed observations. Beyond the presentation of basic linear
model theory, these volumes focus on methods for analyzing correlated observa-
tions. While it is true that generalized linear models can be used for some types
of correlated data, such applications do not flow from the essential theory. McCul-
lagh and Nelder (1989) give a detailed exposition of generalized linear models, and
Christensen (1997) contains a short introduction.

Acknowledgments

I would like to thank MINITAB1 for providing me with a copy of release 6.1.1,
BMDP with copies of their programs 4M, 1T, 2T, and 4V, and Dick Lund for pro-
viding me with a copy of MSUSTAT. Nearly all of the computations were performed
with one of these programs. Many were performed with more than one.
I would not have tackled this project but for Larry Blackwood and Bob Shumway.
Together Larry and I reconfirmed, in my mind anyway, that multivariate analysis is
just the same old stuff. Bob’s book put an end to a specter that has long haunted me:
a career full of half-hearted attempts at figuring out basic time series analysis.
At my request, Ed Bedrick, Bert Koopmans, Wes Johnson, Bob Shumway, and
Dale Zimmerman tried to turn me from the errors of my ways. I sincerely thank
them for their valuable efforts. The reader must judge how successful they were
with a recalcitrant subject. As always, I must thank my editors, Steve Fienberg and
Ingram Olkin, for their suggestions. Jackie Damrau did an exceptional job in typing
the first draft of the manuscript.

1MINITAB is a registered trademark of Minitab, Inc., 3081 Enterprise Drive, State College, PA
16801, telephone: (814) 238–3280.
Preface to the First Edition xv

Finally, I have to recognize the contribution of Magic Johnson. I was so upset

when the 1987–88 Lakers won a second consecutive NBA title that I began writing
this book in order to block the mental anguish. I am reminded of Woody Allen’s
dilemma: is the importance of life more accurately reflected in watching The Sorrow
and the Pity or in watching the Knicks? (In my case, the Jazz and the Celtics.) It’s a
tough call. Perhaps life is about actually making movies and doing statistics.

Albuquerque, NM, USA Ronald Christensen

April 19, 1990

References

Christensen, Ronald (1997). Log-Linear Models and Logistic Regression, Second

Edition. Springer-Verlag, New York.
Christensen, R. (2011). Plane answers to complex questions: The theory of linear
models (4th ed.). New York: Springer.
Christensen, R., Johnson, W., Branscum, A., & Hanson, T. E. (2010). Bayesian ideas
and data analysis: An introduction for scientists and statisticians. Boca Raton,
FL: Chapman and Hall/CRC Press.
Cox, D. R. (1972). Regression models and life tables (with discussion). Journal of
the Royal Statistical Society, Series B, 34, 187–220.
Kalbfleisch, J. D., & Prentice, R. L. (1980). The statistical analysis of failure time
data. New York: Wiley.
McCullagh, P., & Nelder, J. A. (1989). Generalized linear models (2nd Ed.). Lon-
don: Chapman and Hall.
Nelder, J. A., & Wedderburn, R. W. M. (1972). Generalized linear models. Journal
of the Royal Statistical Society, Series A, 135, 370–384.
Contents

Prefaces
............................................................. vii
1 Nonparametric Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Basic Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Linear Approximations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Simple Nonparametric Regression . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.4 Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.4.1 Polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.4.2 Cosines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.4.3 Haar Wavelets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.4.4 Cubic Splines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.4.5 Orthonormal Series Estimation . . . . . . . . . . . . . . . . . . . . . . 16
1.5 Variable Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.6 Heteroscedastic Simple Nonparametric Regression . . . . . . . . . . . . 24
1.7 Approximating-Functions with Small Support . . . . . . . . . . . . . . . . . 25
1.7.1 Polynomial Splines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
1.7.2 Fitting Local Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
1.7.3 Local Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
1.8 Nonparametric Multiple Regression . . . . . . . . . . . . . . . . . . . . . . . . . 38
1.8.1 Redefining φ and the Curse of Dimensionality . . . . . . . . . 39
1.8.2 Reproducing Kernel Hilbert Space Regression . . . . . . . . . 40
1.9 Testing Lack of Fit in Linear Models . . . . . . . . . . . . . . . . . . . . . . . . 43
1.10 Regression Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
1.11 Regression on Functional Predictors . . . . . . . . . . . . . . . . . . . . . . . . . 53
1.12 Density Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
1.13 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

xvii
xviii Contents

2 Penalized Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
2.1.1 Reparameterization and RKHS Regression:
It’s All About the Penalty . . . . . . . . . . . . . . . . . . . . . . . . . . 62
2.1.2 Nonparametric Regression . . . . . . . . . . . . . . . . . . . . . . . . . . 63
2.2 Ridge Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
2.2.1 Generalized Ridge Regression . . . . . . . . . . . . . . . . . . . . . . . 64
2.2.2 Picking k . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
2.2.3 Nonparametric Regression . . . . . . . . . . . . . . . . . . . . . . . . . . 66
2.3 Lasso Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
2.4 Bayesian Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
2.5 Another Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
2.5.1 Geometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
2.5.2 Equivalence of Approaches . . . . . . . . . . . . . . . . . . . . . . . . . 81
2.6 Two Other Penalty Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
3 Reproducing Kernel Hilbert Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
3.1.1 Interpolating Splines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
3.2 Banach and Hilbert Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
3.2.1 Banach Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
3.2.2 Hilbert Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
3.3 Reproducing Kernel Hilbert Spaces . . . . . . . . . . . . . . . . . . . . . . . . . 98
3.3.1 The Projection Principle for an RKHS . . . . . . . . . . . . . . . . 103
3.4 Two Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
3.4.1 Testing Lack of Fit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
3.5 Penalized Regression with RKHSs . . . . . . . . . . . . . . . . . . . . . . . . . . 109
3.5.1 Ridge and Lasso Regression . . . . . . . . . . . . . . . . . . . . . . . . 109
3.5.2 Smoothing Splines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
3.5.3 Solving the General Penalized Regression Problem . . . . . 110
3.5.4 General Solution Applied to Ridge Regression . . . . . . . . . 117
3.5.5 General Solution Applied to Cubic Smoothing Splines . . . 119
3.6 Choosing the Degree of Smoothness . . . . . . . . . . . . . . . . . . . . . . . . . 120
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
4 Covariance Parameter Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
4.1 Introduction and Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
4.1.1 Estimation of β . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
4.1.2 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
4.1.3 Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
4.1.4 Quadratic Estimation of θ . . . . . . . . . . . . . . . . . . . . . . . . . . 132
4.2 Maximum Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
4.2.1 Generalized Likelihood Ratio Tests . . . . . . . . . . . . . . . . . . 136
4.3 Restricted Maximum Likelihood Estimation . . . . . . . . . . . . . . . . . . 137
4.4 Linear Covariance Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
Contents xix

4.5 MINQUE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

4.5.1 Deriving the MINQUE Equations . . . . . . . . . . . . . . . . . . . . 145
4.6 MIVQUE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
4.7 The Effect of Estimated Covariances . . . . . . . . . . . . . . . . . . . . . . . . 148
4.7.1 Mathematical Results* . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
5 Mixed Models and Variance Components . . . . . . . . . . . . . . . . . . . . . . . . 161
5.1 Mixed Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
5.2 Mixed Model Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
5.3 Equivalence of Random Effects and Ridge Regression . . . . . . . . . . 167
5.4 Partitioning and Linear Covariance Structures . . . . . . . . . . . . . . . . . 168
5.5 Variance Component Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
5.5.1 Variance Component Estimation . . . . . . . . . . . . . . . . . . . . . 172
5.6 A Longitudinal Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
5.7 Henderson’s Method 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
5.7.1 Additional Estimates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
5.8 Exact F Tests for Variance Components . . . . . . . . . . . . . . . . . . . . . . 185
5.8.1 Wald’s Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
5.8.2 Öfversten’s Second Method . . . . . . . . . . . . . . . . . . . . . . . . . 187
5.8.3 Comparison of Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
6 Frequency Analysis of Time Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
6.1 Stationary Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
6.2 Basic Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
6.3 Spectral Approximation of Stationary Time Series . . . . . . . . . . . . . 206
6.4 The Random Effects Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
6.5 The White Noise Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
6.5.1 The Reduced Model: Estimation . . . . . . . . . . . . . . . . . . . . . 221
6.5.2 The Reduced Model: Prediction . . . . . . . . . . . . . . . . . . . . . 225
6.5.3 Summary of Sects. 6.2, 6.3, 6.4, and 6.5 . . . . . . . . . . . . . . 228
6.6 Linear Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
6.6.1 Recursive Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
6.6.2 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
6.7 The Coherence of Two Time Series . . . . . . . . . . . . . . . . . . . . . . . . . . 234
6.8 Fourier Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
6.9 Additional Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
7 Time Domain Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
7.1 Correlations and Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
7.1.1 Partial Correlation and Best Linear Prediction . . . . . . . . . 250
7.1.2 The Durbin-Levinson Algorithm . . . . . . . . . . . . . . . . . . . . . 251
7.1.3 Innovations Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
xx Contents

7.2 Time Domain Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252

7.2.1 Autoregressive Models: AR(p)s . . . . . . . . . . . . . . . . . . . . . 253
7.2.2 Moving Average Models: MA(q)s . . . . . . . . . . . . . . . . . . . 257
7.2.3 Autoregressive Moving Average Models:
ARMA(p, q)s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
7.2.4 Autoregressive Integrated Moving Average Models:
ARIMA(p, d, q)s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
7.3 Time Domain Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
7.3.1 Conditioning on Y∞ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
7.4 Nonlinear Least Squares . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
7.4.1 The Gauss–Newton Algorithm . . . . . . . . . . . . . . . . . . . . . . 271
7.4.2 Nonlinear Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
7.5 Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
7.5.1 Correlations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
7.5.2 Conditional Estimation for AR(p) Models . . . . . . . . . . . . . 276
7.5.3 Conditional Least Squares for ARMA(p, q)s . . . . . . . . . . . 278
7.5.4 Conditional MLEs for ARMA(p, q)s . . . . . . . . . . . . . . . . . . 280
7.5.5 Unconditional Estimation for ARMA(p, q) Models . . . . . . 282
7.5.6 Estimation for ARIMA(p, d, q) Models . . . . . . . . . . . . . . . . 286
7.6 Model Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
7.6.1 Box–Jenkins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
7.6.2 Model Selection Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
7.6.3 An Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
7.7 Seasonal Adjustment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
7.8 The Multivariate State-Space Model and the Kalman Filter . . . . . . 302
7.8.1 The Kalman Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
7.8.2 Parameter Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
7.8.3 Missing Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
7.9 Additional Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
8 Linear Models for Spatial Data: Kriging . . . . . . . . . . . . . . . . . . . . . . . . . 321
8.1 Modeling Spatial Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
8.1.1 Stationarity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324
8.2 Best Linear Unbiased Prediction of Spatial Data: Kriging . . . . . . . 328
8.2.1 Block Kriging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
8.2.2 Gaussian Process Regression . . . . . . . . . . . . . . . . . . . . . . . . 331
8.3 Prediction Based on the Semivariogram: Geostatistical
Kriging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
8.4 Measurement Error and the Nugget Effect . . . . . . . . . . . . . . . . . . . . 334
8.5 The Effect of Estimated Covariances on Prediction . . . . . . . . . . . . . 337
8.6 Models for Covariance Functions and Semivariograms . . . . . . . . . 338
8.6.1 The Linear Covariance Model . . . . . . . . . . . . . . . . . . . . . . . 339
8.6.2 Nonlinear Isotropic Covariance Models . . . . . . . . . . . . . . . 340
8.6.3 Modeling Anisotropic Covariance Functions . . . . . . . . . . . 341
8.6.4 Nonlinear Semivariograms . . . . . . . . . . . . . . . . . . . . . . . . . 342
Contents xxi

8.7 Models for Spatial Lattice Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343

8.7.1 Spatial Covariance Selection Models . . . . . . . . . . . . . . . . . 343
8.7.2 Spatial Autoregression Models . . . . . . . . . . . . . . . . . . . . . . 344
8.7.3 Spatial Autoregressive Moving Average Models . . . . . . . . 345
8.8 Estimation of Covariance Functions and Semivariograms . . . . . . . 346
8.8.1 Nonlinear Covariance Functions . . . . . . . . . . . . . . . . . . . . . 347
8.8.2 Linear Covariance Functions . . . . . . . . . . . . . . . . . . . . . . . . 348
8.8.3 Traditional Geostatistical Estimation . . . . . . . . . . . . . . . . . 349
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352
9 Multivariate Linear Models: General . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
9.1 The Univariate Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
9.2 BLUEs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360
9.3 Unbiased Estimation of Σ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362
9.4 Maximum Likelihood Estimates . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364
9.5 Hypotheses and Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366
9.6 Test Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
9.6.1 Equivalence of Test Statistics . . . . . . . . . . . . . . . . . . . . . . . 377
9.7 Prediction and Confidence Regions . . . . . . . . . . . . . . . . . . . . . . . . . . 378
9.8 Multiple Testing Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379
9.9 Multivariate Time Series and Spatial Models . . . . . . . . . . . . . . . . . . 384
9.9.1 A Tensor Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384
9.10 Additional Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387
10 Multivariate Linear Models: Applications . . . . . . . . . . . . . . . . . . . . . . . . 389
10.1 One-Sample Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389
10.2 Two-Sample Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393
10.3 One-Way Analysis of Variance and Profile Analysis . . . . . . . . . . . . 397
10.3.1 Profile Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
10.3.2 Comparison with Split Plot Analysis . . . . . . . . . . . . . . . . . 408
10.3.3 Computations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414
10.3.4 Covariance Matrix Modeling . . . . . . . . . . . . . . . . . . . . . . . . 414
10.4 Growth Curves for One-Way MANOVA . . . . . . . . . . . . . . . . . . . . . 415
10.5 Testing for Additional Information . . . . . . . . . . . . . . . . . . . . . . . . . . 416
10.6 Additional Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421
11 Generalized Multivariate Linear Models and Longitudinal Data . . . . 423
11.1 Generalized Multivariate Linear Models . . . . . . . . . . . . . . . . . . . . . 423
11.2 Generalized Least Squares Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 426
11.3 MACOVA Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430
11.4 Rao’s Simple Covariance Structure . . . . . . . . . . . . . . . . . . . . . . . . . . 435
11.4.1 Reduced Models in Z . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437
11.4.2 Unbiased Covariance Parameter Estimation . . . . . . . . . . . 437
11.4.3 Testing the SCS Assumption . . . . . . . . . . . . . . . . . . . . . . . . 439
xxii Contents

11.5 Longitudinal Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440

11.5.1 Full Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440
11.5.2 Incomplete Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443
11.6 Functional Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446
11.7 Generalized Split Plot Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 451
11.7.1 GMLMs Are GSP Models . . . . . . . . . . . . . . . . . . . . . . . . . . 452
11.8 Additional Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454
12 Discrimination and Allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457
12.1 The General Allocation Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461
12.1.1 Mahalanobis Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461
12.1.2 Maximum Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461
12.1.3 Bayesian Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462
12.2 Estimated Allocation and QDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465
12.3 Linear Discrimination Analysis: LDA . . . . . . . . . . . . . . . . . . . . . . . 468
12.4 Cross-Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 471
12.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473
12.6 Stepwise LDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475
12.7 Linear Discrimination Coordinates . . . . . . . . . . . . . . . . . . . . . . . . . . 481
12.7.1 Finding Linear Discrimination Coordinates . . . . . . . . . . . . 486
12.7.2 Using Linear Discrimination Coordinates . . . . . . . . . . . . . 490
12.7.3 Relationship to Mahalanobis Distance Allocation . . . . . . 492
12.7.4 Alternate Choice of Linear Discrimination
Coordinates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494
12.8 Linear Discrimination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495
12.9 Additional Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500
13 Binary Discrimination and Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . 503
13.1 Binomial Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504
13.1.1 Data Augmentation Ridge Regression . . . . . . . . . . . . . . . . 507
13.2 Binary Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 508
13.3 Binary Generalized Linear Model Estimation . . . . . . . . . . . . . . . . . 510
13.4 Linear Prediction Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
13.4.1 Loss Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514
13.4.2 Least Squares Binary Prediction . . . . . . . . . . . . . . . . . . . . . 515
13.5 Support Vector Machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515
13.5.1 Probability Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517
13.5.2 Parameter Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517
13.5.3 Advantages of SVMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
13.5.4 Separating Hyper-Hogwash . . . . . . . . . . . . . . . . . . . . . . . . . 521
13.6 Best Prediction and Probability Estimation . . . . . . . . . . . . . . . . . . . 523
13.7 Binary Discrimination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 530
Contents xxiii

14 Principal Components, Classical Multidimensional Scaling,

and Factor Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533
14.1 The Theory of Principal Components . . . . . . . . . . . . . . . . . . . . . . . . 534
14.1.1 Sequential Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 534
14.1.2 Joint Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 536
14.1.3 Other Derivations of Principal Components . . . . . . . . . . . 538
14.1.4 Principal Components Based on the Correlation
Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 542
14.2 Sample Principal Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543
14.2.1 The Sample Prediction Error . . . . . . . . . . . . . . . . . . . . . . . . 544
14.2.2 Using Principal Components . . . . . . . . . . . . . . . . . . . . . . . . 546
14.3 Classical Multidimensional Scaling . . . . . . . . . . . . . . . . . . . . . . . . . 551
14.4 Factor Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555
14.4.1 Additional Terminology and Applications . . . . . . . . . . . . . 557
14.4.2 Maximum Likelihood Theory . . . . . . . . . . . . . . . . . . . . . . . 560
14.4.3 Principal Factor Estimation . . . . . . . . . . . . . . . . . . . . . . . . . 564
14.4.4 Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566
14.4.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566
14.5 Additional Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 570
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 573
A Mathematical Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575
A.1 Differentiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575
A.2 Vec Operators and Kronecker Products . . . . . . . . . . . . . . . . . . . . . . . 578
A.3 Quadratic Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 579
A.3.1 Application to Support Vector Machines . . . . . . . . . . . . . . 582
B Best Linear Predictors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 587
B.1 Properties of Best Linear Predictors . . . . . . . . . . . . . . . . . . . . . . . . . 587
B.2 Irrelevance of Units in Best Multivariate Prediction . . . . . . . . . . . . 593
C Residual Maximum Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595
C.1 Maximum Likelihood Estimation for Singular Normal
Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595
C.2 Residual Maximum Likelihood Estimation . . . . . . . . . . . . . . . . . . . 596
Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 597

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 599

Subject Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603
Visit https://textbookfull.com
now to explore a rich
collection of eBooks, textbook
and enjoy exciting offers!
Chapter 1
Nonparametric Regression

Abstract This chapter introduces nonparametric regression for a single predictor

variable, discusses the curse of dimensionality that plagues nonparametric regres-
sion with multiple predictor variables, and discusses the kernel trick and related
ideas as methods for overcoming the curse of dimensionality.

In the late 1990s, the orthogonal series approach to nonparametric regression be-
came increasingly popular; see Hart (1997), Ogden (1997), and Efromovich (1999).
In this approach, orthogonal series of functions are used to approximate the regres-
sion function. Later, the orthogonality was de-emphasized so that now a series of
more general basis functions is often used to approximate the regression function.
Basis functions, and other linear-approximation methods for which many of the se-
ries elements have small support, methods such as splines and wavelets, seem par-
ticularly useful. We discuss these approaches to nonparametric regression as fitting
linear models.
Suppose we have a dependent variable y and a vector of predictor variables
x. Regression is about estimating E(y|x). In linear regression, we assume that
E(y|x) = x β for some unknown parameter vector β . Recall that this includes fit-
ting indicator variables and polynomials as special cases. In nonlinear regression
we assume that E(y|x) = f (x; β ), where the function f is known but the vector β
is unknown; see Sect. 7.4 and Christensen (1996, Chapter 18; 2015, Chapter 23).
A special case of nonlinear regression involves linearizable models, including gen-
eralized linear models, that assume E(y|x) = f (x β ) for f known, cf. Christensen
(1997, Chapter 9). The key idea in nonlinear regression is using calculus to linearize
the model. In nonparametric regression, we assume that E(y|x) = f (x), where the
function f is unknown. Note the absence of a vector of parameters β , hence the
name nonparametric. Often, f is assumed to be continuous or to have some speci-
fied number of derivatives. In reality, nonparametric regression is exactly the oppo-
site of what its name suggests. Nonparametric regression involves fitting far more
parameters than either standard linear or nonlinear regression.

© Springer Nature Switzerland AG 2019 1

R. Christensen, Advanced Linear Modeling, Springer Texts in Statistics,
https://doi.org/10.1007/978-3-030-29164-8_1
2 1 Nonparametric Regression

EXAMPLE 1.0.1. Table 1.1 presents data from Montgomery and Peck (1982) and
Eubank (1988) on voltage drops y over time t displayed by an electrical battery used
in a guided missile. The 41 times go from 0 to 20. The variable x results from divid-
ing t by 20, thus standardizing the times into the [0, 1] interval. The data comprise
a time series as discussed in Chaps. 6 and 7, but the idea here is that the behavior
over time is not a stationary stochastic process but rather a complicated regression
function. An unusual feature of these data is that the ti values are equally spaced
(i.e., the ti s are ordered and ti+1 − ti is a constant). This typically occurs only when
the data collection process is very well-controlled. However, when equal spacing
does occur, it considerably simplifies data analysis. 2

Table 1.1 Battery voltage drops versus time

Case y t x Case y t x
1 8.33 0.0 0.000 22 14.92 10.5 0.525
2 8.23 0.5 0.025 23 14.37 11.0 0.550
3 7.17 1.0 0.050 24 14.63 11.5 0.575
4 7.14 1.5 0.075 25 15.18 12.0 0.600
5 7.31 2.0 0.100 26 14.51 12.5 0.625
6 7.60 2.5 0.125 27 14.34 13.0 0.650
7 7.94 3.0 0.150 28 13.81 13.5 0.675
8 8.30 3.5 0.175 29 13.79 14.0 0.700
9 8.76 4.0 0.200 30 13.05 14.5 0.725
10 8.71 4.5 0.225 31 13.04 15.0 0.750
11 9.71 5.0 0.250 32 12.06 15.5 0.775
12 10.26 5.5 0.275 33 12.05 16.0 0.800
13 10.91 6.0 0.300 34 11.15 16.5 0.825
14 11.67 6.5 0.325 35 11.15 17.0 0.850
15 11.76 7.0 0.350 36 10.14 17.5 0.875
16 12.81 7.5 0.375 37 10.08 18.0 0.900
17 13.30 8.0 0.400 38 9.78 18.5 0.925
18 13.88 8.5 0.425 39 9.80 19.0 0.950
19 14.59 9.0 0.450 40 9.95 19.5 0.975
20 14.05 9.5 0.475 41 9.51 20.0 1.000
21 14.48 10.0 0.500

In Sect. 1.2, we discuss the basics of the linear-approximation approach. In

Sect. 1.3, we examine its relationship to linear models. In Sect. 1.4, we discuss and
illustrate least squares estimation and we discuss some of the estimation approaches
specifically proposed for orthogonal series. In Sect. 1.5, we discuss variable selec-
tion and relate an orthogonal series proposal to the Cp statistic. Section 1.6 exam-
ines problems involving heteroscedastic variances. In Sect. 1.7, we discuss details
of splines and introduce kernel estimation and other local polynomial regression
techniques. Section 1.8 introduces nonparametric multiple regression. Section 1.9
examines testing lack of fit. Section 1.10 looks at regression trees. Finally, Sect. 1.11
makes a few comments on density estimation and Sect. 1.12 includes exercises.
1.1 Basic Notation 3

1.1 Basic Notation

Before proceeding we set some notation that is used throughout the book, unless
defined otherwise for particular purposes. A linear model has Y = X β + e where Y
is an n × 1 vector of observable random variables, X is an n × p matrix of known
values, β is a p × 1 vector of fixed but unknown coefficients, and e is an n × 1 vector
of unobservable random errors. For this to be a linear model we need E(e) = 0 so
that E(Y ) = X β . A standard linear model assumes that an individual observation
or error has variance σ 2 and that Cov(Y ) = Cov(e) = σ 2 I. The assumption that the
observations have a multivariate normal distribution can be written Y ∼ N(X β , σ 2 I).
A partitioned linear model is written Y = X β + Z γ + e where Z is also a matrix
of known values and γ is also a vector of fixed, unknown coefficients. If Z has s
columns, write
⎡ ⎤ ⎡ ⎤
x1 z1
⎢ .. ⎥ ⎢ .. ⎥
X = [X1 , . . . , Xp ] = ⎣ . ⎦ ; Z = [Z1 , . . . , Zs ] = ⎣ . ⎦ .
xn zn

For any vector v, v2 ≡ v v is the squared (Euclidean) length of v. The Euclidean
inner product between two vectors u and v is u v. They are perpendicular (orthog-
onal), written v ⊥ u, if v u = 0. A− denotes the generalized inverse of the matrix
A, r(A) denotes its rank, and tr(A) denotes its trace. M denotes the perpendicular
projection operator (ppo) onto the column space of X. The column space of X is
denoted C(X). (With tongue slightly in cheek) the Fundamental Theorem of Least
Squares Estimation is that in a linear model, β̂ is a least squares estimate if and
only if
X β̂ = MY.
More generally, MA denotes the ppo onto C(A). C(A)⊥ denotes the orthogonal com-
plement of C(A), i.e. all the vectors that are orthogonal to C(A). If C(X) ⊂ C(A),
⊥
C(X)C(A) denotes the orthogonal complement of C(X) with respect to C(A), i.e. all
vectors in C(A) that are orthogonal to C(X). An r × c matrix of 1s is denoted Jrc with
Jn ≡ Jn1 and J ≡ Jn .
This is all common notation and, except for the use of M and J, it is pretty much
standard notation. (Some authors prefer P and 1.) It is important to understand the
theory associated with this notation. For example, I expect the reader to know (or at
⊥
least believe when I write) that the ppo onto C(X)C(A) is MA − M. Such background
can be found in many places including PA.
4 1 Nonparametric Regression

1.2 Linear Approximations

The key idea behind linear approximations is that a finite linear combination of some
known functions can approximate a wide variety of functions on a closed bounded
set, cf. the famous Stone-Weierstrass theorem. For convenience, we initially assume
that f is defined on the interval [0, 1] and is continuous. There are many ways to
approximate f including polynomials, sines and cosines, step functions, and also by
things similar to step functions called wavelets. Most often we assume that for some
predictor variable x
∞
f (x) = ∑ β j φ j (x),
j=0

where the φ j s are known functions that can be defined in many ways. Later we will
use this characterization with x being a p vector instead of a scalar. In particular,
with p = 1 and functions defined on the unit interval, we can take for j = 0, 1, 2, . . .

φ j (x) = x j , (1.2.1)

or
φ j (x) = cos(π jx), (1.2.2)
or
φ2 j (x) = cos(π jx) φ2 j+1 (x) = sin(π jx). (1.2.3)
When using (1.2.2), it should be noted that the derivative of every cos(π jx) function
is 0 at x = 0, so the derivative of f (x) should be 0 at x = 0.
In practice we approximate f with a finite number of terms which determines a
linear model in which only the β j s are unknown. We need to determine an appro-
priate finite approximation and estimate the corresponding β j s
With a single predictor, another obvious approximation uses step functions but
some care must be used. Let IA be the indicator function for the set A, namely

1 if x ∈ A
IA (x) =
0 otherwise.
Obviously, if we define

φ j (x) = I( j−1 , j ] (x), j = 0, 1, . . . , m,

m m

we can approximate any continuous function f , and as m → ∞ we can approximate

f arbitrarily well. Note that φ0 (x) is essentially I{0} (x). Technically, rather than the
infinite sum characterization, we are defining a triangular array of functions φ jm ,
j = 1, . . . , m; m = 1, 2, 3, . . . and assuming that
m
f (x) = lim
m→∞
∑ β jm φ jm (x). (1.2.4)
j=0
1.2 Linear Approximations 5

More generally, we could define the indicator functions using intervals between
knots, x̃−1,m < 0 = x̃0,m < x̃1,m < x̃2,m < · · · < x̃m,m = 1 with the property that
maxi {x̃i+1,m − x̃i,m } goes to zero as m goes to infinity.
Splines are more complicated than indicator functions. Choosing m − 1 knots in
the interior of [0, 1] is fundamental to the use of splines. Rather than indicators,
we can fit some low dimensional polynomial between the knots. In this context,
indicator functions are 0 degree polynomials. For polynomials of degree greater
than 0, traditional splines force the polynomials above and below each knot in (0,1)
to take the same value at the knot, thus forcing the splines to give a continuous
function on [0,1]. B-splines use functions φ jm that are nonzero only on small but
overlapping subintervals with locations determined by (often centered around) a
collection of knots. As with indicator functions, to get good approximations to an
arbitrary regression function, the distances between consecutive knots must all get
(asymptotically) small. As a practical matter, one tries to find one appropriate set
of knots for the problem at hand. Technically, methods based on knots are not basis
function methods because they do not provide a countable set of functions that are
linearly independent and span the space of continuous functions. (B-spline is short
for “basis spline” but that is something of a misnomer.)
As with basis function approaches based on an infinite sum, any triangular array
satisfying Eq. (1.2.4) allows us to approximate f with a finite linear model in which
only β1m , . . . , βmm are unknown. Triangular array approximations can also be used
with vector inputs.
Rather than defining a triangular array of indicator functions, we can use the
following device to define a single infinite series :

φ0 (x) = 1, φ1 (x) = I(0,.5] (x), φ2 (x) = I(.5,1] (x),

φ3 (x) = I(0,.25] (x), φ4 (x) = I(.25,.5] (x),

φ5 (x) = I(.5,.75] (x), φ6 (x) = I(.75,1] (x),
φ7 (x) = I(0,2−3 ] (x), . . . , φ14 (x) = I({23 −1}2−3 ,1] (x),
φ15 (x) = I(0,2−4 ] (x), . . . .
Technically, these φ j s constitute a spanning set of functions but are not basis func-
tions. Except for approximating the point f (0), including the function φ0 (x) is irrele-
vant once we include φ1 (x) and φ2 (x). Similarly, φ1 (x) and φ2 (x) are made irrelevant
by φ3 (x), . . . , φ6 (x).
A sequence of basis functions, one that is equivalent to this spanning set of step
functions, is the Haar wavelet collection

φ0 (x) = 1, φ1 (x) = I(0,.5] (x) − I(.5,1] (x),

φ2 (x) = I(0,.25] (x) − I(.25,.5] (x), φ3 (x) = I(.5,.75] (x) − I(.75,1] (x),
φ4 (x) = I(0,1/8] (x) − I(1/8,2/8] (x), . . . , φ7 (x) = I(6/8,7/8] (x) − I(7/8,1] (x),
φ8 (x) = I(0,1/16] (x) − I(1/16,2/16] (x), . . . .
Random documents with unrelated
content Scribd suggests to you:
worked harder, probably, than any three other members of the
Kehillah; but in his hands the fairest projects seemed to receive a
blight, and the most promising business ventures turned to
wormwood and ashes, to apples of Sodom and grapes of bitterness.
But the Schlemihl, perfectly useless though he was to himself and his
family, had one very evident purpose in the scheme of life, namely, to
open the hearts of his brethren to impulses of kindness and
benevolence. They certainly acted toward him in the most
sympathetic and brotherly manner, and permitted neither him nor
his family to suffer. At the time of my arrival in Nordheim, Isaac had
just managed, through one of his usual transactions, to lose all he
had, and to have his house, which he had received as part of the
dowry of his wife, seized in satisfaction of his debts. But the
Nordheim Kehillah, assisted by some benevolent friends from other
places, paid off his debts, redeemed the house, and furnished him
with a certain amount of capital with which to begin life anew. For
safety’s sake the Kehillah retained the title in the house; for, as Uncle
Koppel said to me in confidence, “We might otherwise have to buy
the house every year.”
A peculiarly interesting character was David the horse-dealer, a
jovial, hale fellow, handsome too, and tall and strong as a lion, a very
“mighty man in Israel.” He was a stanch friend and reliable, and
could be depended upon to go through thick and thin for one who
had once gained his friendship. But David had one weakness, not
unnatural, perhaps, in those of his vocation. He knew no scruples of
conscience in regard to transactions in horseflesh; and some of his
achievements in that line had been, if report spoke truly, to say the
least, extremely venturesome. Thus he was credited with having once
sold a Prussian major who prided himself on his expert knowledge of
the equine species, a horse with only three hoofs. The manner in
which David was said to have done the trick was as follows: The deal
took place in midwinter, when the ground was covered with snow to
the depth of a foot or more. The horse was a fine animal, coal black
and of handsome form, except that the left front hoof was lacking.
David led the horse out of the stable; and as it stood in the deep snow
before the Prussian major, who was critically examining it through
his eyeglasses, the absence of the hoof was not noticeable. He then
put it through its paces, cracking his whip furiously, so that the horse
leaped and dashed in a most fiery manner, and the absence of the
hoof was again not noticeable. The major was charmed with the fire
and grace of the animal, bought and paid for it at once, and ordered
it to be sent to his quarters. It is said that the major was furious later,
not so much on account of the money loss, but because he, the
expert, had been so neatly duped, and because he had no legal
remedy against David. Had David put a false hoof in place of the
lacking member, he would have been liable to a heavy penalty for
fraud; but he had not done so, and had made no false representation.
And therefore the major not only had no case against him, but could
not even demand the cancellation of the sale. Thus the story for
whose veracity I will not guarantee. But, however weak David’s
conscience may have been in matters of horsetrading, his conduct
otherwise merited no reproach and he was well liked.
Many were the estimable and lovable characters in Nordheim’s
Kehillah, and I cannot attempt to describe or even mention them all.
Of Uncle Koppel and Aunt Caroline I have already spoken. Uncle
Koppel was a typical Jewish Baal-Ha-Bayith, or householder, a
business man of probity, whose word was as good as his bond, a
faithful worshipper at the altar of Israel’s God, and a worthy
upholder, by character, if not by learning, of the reputation of Reb
Shemayah, his father. Aunt Caroline was a true mother in Israel,
loyal, conscientious, and devout. Their able sons and charming dark-
eyed daughters were imbued with their spirit, and together they
formed an ideal household. Nor must I forget Aunt Gella, the only
other child of Reb Shemayah who had remained in the native village,
a woman of noble parts, who, had her lot been cast somewhere else
in the great world, might have played an important part in history.
Her noble brow, which emerged so modestly from the recesses of her
Scheitel and her mild and clear blue eyes, showed her the possessor
of a strong and well-developed intellect; and her wise and well-
considered conversation showed that the reality corresponded to the
indications. Her heart was as warm and good and her spirit as firm
and courageous as her mind was keen and clear; and she was, so to
speak, the combined oracle and Lady Bountiful of the village. Was
any female or, for that matter, any male villager in trouble, in want of
counsel or help, she or he would direct her or his steps to the neat
cottage in the Long Street where dwelt Aunt Gella, and there would
find counsel or comfort, or whatever help was required. A plague of
dysentery came once upon the village, and then it was that Aunt
Gella showed herself the veritable angel of help. While it continued
she hardly ate or drank or slept or changed her clothes. She worked
with tireless energy at her mission of mercy, going from house to
house among the afflicted ones, bringing the right medicine to one,
the right food to the other, and money to the third. Dear Aunt Gella:
methinks I see her sweet, mild face now, and hear the words of
blessing with which peasant and Jew mentioned her name. And
besides those whom I have mentioned, there were dozens of
householders in which piety, probity, and loving kindness were the
constantly practised rule of life.
Yes, Nordheim, I loved thee well, and I love thy memory. I loved
thee for thy simplicity, for thy natural goodness, for the true and
unpretentious way in which thou didst lay stress upon that which is
pure and noble, and didst reject that which is base and vile in human
life; for the picture which thou didst show me of the beautifying and
sanctifying effect of a simple, sincere, and honest Judaism, simply
and sincerely lived. Thou wast one of the forces which did lead me to
love and uphold the Torah, and to cleave to the faith which my and
thy ancestors received at Sinai from Sinai’s God.
Oh, that this tale of thee might work likewise upon the hearts of
others like me, children of an unbelieving and irreverent age, and stir
them to love for Israel’s God and devotion to Israel’s sacred heritage!
 THE LITTLE HORSERADISH WOMAN.

How many of my readers know the little horseradish woman?

Many, I have no doubt, are more or less acquainted with her; and
those who are not can make her acquaintance without any difficulty.
Almost any afternoon and late into the evening, except on Sabbaths
or Jewish holidays, she may be found at her post in one of the blocks
of upper Third Avenue, New York, standing behind her improvised
little table, industriously rubbing away at her acrid merchandise,
with only occasional pauses to wipe away with the corner of her
snow-white apron the tears which her lachrymose occupation forces
from her eyes, or to give customers extraordinarily liberal portions of
her finished product. The size of the portions she sells is quite
astonishing to the customer; but the little horseradish woman is
scrupulously honest in matters of weight and measure, of mine and
thine, and would not think of giving less.

THE LITTLE
HORSERADISH
 WOMAN

Page 84

Her tears, too, are quite remarkable. Indeed, I believe that

horseradish tears have not been appreciated as they should be, for
they are a species entirely sui generis, and not to be confused with
any other tears that are shed on earth. Ordinary, every-day tears
indicate sorrow and produce weakness; crocodile tears indicate
hypocrisy and produce disgust; but horseradish tears are born of
industry, and their offspring are energy and good-humor. Such, at
least, is the case with our little horseradish woman; for, no sooner
has she wiped away one of her periodical outbursts of tears, than she
begins to rub away again with the utmost energy and the best humor
in the world. My observation of the tears the horseradish woman
sheds has made me their confirmed admirer. I have no liking for the
lachrymose ebullitions of love-lorn maidens, of snivelling swains, or
of wheezing or wheedling Pecksniffs. Give me horseradish tears; they
are the honestest, cheerfullest—I had almost said—manliest tears in
the world.
Our horseradish woman is known by various names. Some call her
“the old Rebecca”; others, desiring to speak more formally or
respectfully, refer to her as “old Mrs. Levy”; but the appellation by
which she is most widely and popularly known is das Meerrettich
Weible—the little horseradish woman. It makes no difference,
however, by what designation she is known, she is popular under
them all; for the little horseradish woman is liked. Some like her for
her courage in toiling so constantly and industriously, and
supporting herself at her advanced age; others like her because of her
unfailing cheeriness and good-humor; others, again, because of her
simple, trustful faith and earnest piety, for the little horseradish
woman is more than usually religious, and is to be found in the
synagogue, not only on Sabbaths and holidays, but also at the early
morning and evening services on week-days, and is one of the most
attentive listeners to the rabbi when he expounds the Sedrah on
Sabbath mornings, or “learns Shiur” on Sabbath afternoons or week-
day evenings.
 It is a truly pleasing picture which the little horseradish woman
presents when she stands at her post ready for business. Her regular
and refined features, of the familiar Jewish type, are, it is true, worn
and wrinkled, and the hair which peeps out from under the cloth
band and the old-fashioned bonnet which surmount her head is
whitened by the seventy or more winters which have passed over her;
but the light of intelligence, of benevolence, and of pure and refined
sentiments shines in her countenance and makes it singularly
attractive. Her clothing is of the plainest. She wears a dress of some
simple, dark material and over it a long, white apron; but no patch,
tear, nor stain is visible anywhere, and we feel instinctively that we
have before us a person who, though in humble, even lowly
circumstances, is naturally and intrinsically refined.
But as yet we do not know the little horseradish woman. It is only
upon entering into conversation with her that we really find out what
she is, and a great surprise awaits us then. For this poor, little, old
woman who stands upon the street in all weather and seasons, and
toils so hard to earn a few cents by the sale of her commodity, comes
of excellent family, has had, for her time, an exceptionally good
training, and is, in some respects, a remarkably well-educated
woman.
She was born as the daughter of a rabbi in a small provincial city of
Germany, and her father, besides instilling into her soul the seeds of
fervent Hebraic piety, saw to it that she received a thorough secular
and religious training. As a consequence her manners are those of
polite and well-bred circles, her German is pure and correct in
grammar and pronunciation, and what is most surprising and
pleasing to the Jewish scholar, she is acquainted with the entire Bible
in the original Hebrew. The Book of Psalms she knows by heart and
quotes with amazing fluency; and from her experience in her father’s
house she has derived a large number of technical Talmudic phrases,
which she uses in her conversation with entire correctness of
expression and application.
And the most remarkable thing of all is the entire lack of self-
consciousness on the part of the little horseradish woman. She is
entirely unaware that there is anything out of the ordinary in her life,
her characteristics, or her circumstances. She never comments upon
the different conditions that prevail to-day, never boasts nor
condemns, is simple, natural, and unaffected; a typical, humble,
pious Jewish woman. Oh, that you might come, you artificial,
affected daughters of an artificial, affected age, and learn simple
refinement and natural dignity from this lowly sister of yours! The
lesson is needed and would prove effective.
Last Saturday night, after the “going out” of the Sabbath, my wife
and I also determined to go out for a stroll on Third Avenue. We
often take these strolls, and enjoy them. My wife loves the excitement
of the lights and the crowds, which make it doubly pleasant to meet
an acquaintance or make an occasional purchase; and I am equally
fond of studying human nature where it makes its most
characteristic appearance, in the busy throngs of men. We had not
seen the little horseradish woman for some time, for she had given
up of late her habit of coming to our house with her wares, and her
stand was not on any of the blocks we usually traversed.
That evening we extended our walk a little further than usual. As
we neared —th Street, suddenly Mrs. —— exclaimed: “Look, there is
the little horseradish woman!” Sure enough it was she, and we
immediately went up to her.
While she was returning our greeting with great cordiality and
friendliness, I noticed that she did not appear to be as well as usual.
Her movements were lacking in their customary vivacity, and her
face seemed thinner and paler than its wont.
“How are you getting on, Mrs. Levy?” I said, while she was filling a
bag with our ordered portion of horseradish.
“Boruch Hashem, quite well,” she responded with a smile. “My
friends are good and patronize me steadily, but I feel that I am
growing older. I was quite ill the other day. I nearly fainted here on
the street; but the people in the delicatessen store were very kind.
They took me in and gave me cold water, and kept me there until I
recovered; and I am feeling quite well now.”
While listening to her words, I thought to myself how hard her lot
was; and I asked myself whether it really was necessary for her to
stand on the street and earn her living in such a trying manner.
“My good Mrs. Levy,” I said, “don’t you think your life is too hard
for you? Would you not rather go to some institution where you
would be cared for?”
 “Oh, no, thank you,” she responded. “I don’t wish to go to a home.
I have a husband, although he is old and feeble, and good children
who do what they can for me; and I am glad that I still can earn
something myself. You know what King David says in the Psalms,”
and she quoted glibly, “Yegia keppecho ki sochel, ashrecho ve-tov-
loch” (“If thou eatest what thy hands earn, thou art happy, and it is
well with thee”). “I eat what my hands earn, so I am happy.”
“Why don’t you come to our house any more?” broke in my wife.
“Oh,” answered the little horseradish woman, “I heard that
another woman brings you your horseradish, and I did not wish to be
massig gevool.”
Our package was now ready and we departed. But my thoughts
gave me no rest. I was thinking continually of the little horseradish
woman, and whether it was not possible to devise some means of
improving her lot.
A few blocks down the avenue we met Mr. and Mrs. Bergheim.
They are friends and neighbors of ours, and our greetings were
cordial. I soon turned the conversation to that which was uppermost
in my thoughts.
“You know the little horseradish woman, do you not?” I asked.
The Bergheims nodded assent.
“Don’t you think something could be done for her?” I continued.
“It does seem wrong that such a worthy old person should be forced
to stand on the street and toil so hard for a livelihood.”
The Bergheims smiled at each other peculiarly.
“What would you do for her?” asked Mr. Bergheim. “She is much
too proud to accept charity; besides, she really does not need to
work, as her children supply her with all she requires for herself and
husband. Her horseradish receipts are so much extra income that
she earns.”
I must confess that this reply rather staggered me. There appeared
to be a mystery about the horseradish woman which was puzzling, to
say the least.
“But why, in the name of common sense,” I demanded, “does such
an old and not overstrong woman toil on the streets, in rain and
shine, by day and by night, if she has all she requires and does not
need to work? It doesn’t seem reasonable. She isn’t touched in her
upper story, I hope?”
“Oh, no, not at all,” said Bergheim; “but you see, she has rather
unusual and exalted notions about duty. Since the requirements of
herself and husband are satisfied and she has some strength, she
thinks it her duty to labor for the poor. Every cent she earns by
selling horseradish she gives to the poor. It is quite an amount, for
she has many customers; and quite a long list of widows and orphans
and feeble old men who are regular pensioners on her charity.
“Every Rosh Chodesh there is quite a gathering in her humble flat.
All sorts of needy and afflicted persons, men, women, and children,
crowd her rooms, and she divides among them, with the most kindly
sympathy but with excellent judgment, all the money she has earned
during the month. The blessings she gets are innumerable, and she
considers herself well rewarded thereby for all her trouble.
“I found this out by accident, as she never says a word about it to
any one. When I asked her why she went to all this trouble, she
quoted a passage from the Pentateuch: ‘Verily, thou shalt not harden
thy heart nor close thy hand against thy poor brother’; and in
another from the Ethics of the Fathers, ‘The poor shall be the
children of thy house,’ and said those were her reasons.
“That, my dear ——, is why you cannot do anything for the little
horseradish woman, except to be her customer and patronize her
liberally. She wants no charity, and will take no gifts for ‘her poor,’
whom she wishes to assist with her own earnings.”
So that was the explanation of the riddle. The little horseradish
woman was emulating the work of the Master of the universe, was
toiling early and late to feed His hungry ones, to dry the tears of His
afflicted, to care for His poor. I was lost in admiration, both of the
noble soul of this humble daughter of Israel and the sublime glory of
Israel’s law, which put such thoughts into her soul.
I have made up my mind that the next time I see the little
horseradish woman I shall pronounce over her the benediction
which the rabbis ordain to be spoken at the sight of kings and
queens, for she is a real queen, an uncrowned queen of mercy and
love. “Blessed art Thou, O Lord, who hast given of Thy glory to flesh
and blood.”
 THE GENERAL.

I have distinguished company in my study this morning. No less a

personage than Gen. Sergei Pavlowitz, late commander of the —th
division of the regular Russian army, has paid your humble servant
the honor of a visit, and is now seated in the rocking-chair opposite
my desk. I must, however, ask my readers not to strain their
imaginations unduly in summoning up before their mental vision a
suitable picture of military pomp and splendor. The general is not in
full uniform heavily braided and trimmed with gold lace, nor radiant
with glittering epaulets and buttons. No plumed helmet surmounts
his head; no clanking sabre swings at his side; he is neither gloved,
booted, nor spurred. His appearance would not dazzle the onlooker,
nor overawe the most timid; in fact, no one would, at first sight,
think of connecting him in any way with marching hosts or warlike
scenes. As he sits there in my rocking-chair, gazing at me with his
mild blue eyes, upon his head a little black skull-cap, his long, snow-
white beard flowing down upon the front of his shirt and his black
broadcloth coat; in his hand a stout cane to assist the steps which age
has made somewhat uncertain, while he descants upon a matter of
purely synagogical interest, there is no suggestion about him of
martial glory, no hint of the groan and agony and heroism of battle.
He seems just a plain, every-day, elderly Russian Jew, diffident and
retiring in worldly affairs, but bright enough in matters of Jewish
concern, of Hebrew learning, and religious practice, such a man, in a
word, as may be found in any of the orthodox synagogues throughout
New York but particularly on the lower East Side, where the places of
worship and solemn assembly of his brethren and countrymen most
abound.
 THERE IS
SOMETHING
COMMANDING
, SOMETHING
INDEFINITELY
MILITARY
AND
AUTHORITATI
VE ABOUT
HIM

Page 96

But now my visitor has concluded the business which brought him
hither and rises to depart. Immediately one can notice a vast change
in the impression he makes. He does seem different now from the
ordinary so-called Ghetto type he appeared identical with a moment
ago. There is something commanding, something indefinitely
military and authoritative about him. Though feeble, he stands
perfectly erect, and his figure and bearing are thoroughly military.
Military, too, is the almost painful neatness which characterizes his
attire, from his well-brushed hat and coat down to his brightly
polished shoes, a far-off reminder, as it were, of the days when a dull
button or a frayed coat sleeve meant disgrace and the guard-house;
but most military of all is his right sleeve, for it hangs empty, with
only a short stump filling the upper part near the shoulder, a mute
reminder of bloody Sebastopol, where a British sabre cleft the arm to
which it belonged in twain, and its owner hovered for many a day
’twixt life and death.
This is the General. Perhaps, strictly speaking, he does not deserve
the title, for he long since was stricken from the Russian army list,
and might even meet with condign punishment were he to return to
his native land; but once he bore it with full right and authority, and
no military shortcoming, no lack of loyalty or courage upon the
battlefield was responsible for its forfeiture. It is, therefore, only
natural that his friends and neighbors who know his history give him
the title. So “the General” he is, and “the General” he will remain,
until death calls him to his last long bivouac. What a tremendous
change in state and fortune! Once a distinguished military
commander, whose slightest behest thousands hastened to obey
because of his heroism; beloved by his countrymen and honored by
his emperor; the husband of a renowned general’s daughter, and
with every prospect promising rapid advancement and eventually
loftiest rank; now the humble denizen of an obscure street in the
Jewish quarter of New York, his life in nowise different from that of
the other long-bearded habitués of the synagogue and the Beth
Hammidrash.
How came this Jew, son of a proscribed and pariah race, to attain
to such distinguished rank in the service of the persecutors of his
people? How came he to lose it, and to sink back again into the
lowliness from which he sprang? It is a strange tale, showing what
sombre romances, what heartrending tragedies Jewish life is still
capable of producing in the empire of the Czars. I shall tell it you.
Some seventy years ago there lived in one of the western provinces
of Russia a young couple. Israel Rabbinowitz was the husband’s
name, and Malka Feige that of the spouse. They were a pious and
worthy pair. The husband was a respected merchant, whose
scrupulous honesty and commercial rectitude were no less esteemed
than his unswerving religious fidelity, and the accuracy and extent of
the Hebrew scholarship which he displayed in the Talmudic debates
of the circle of “learners” in the Beth Hammidrash. Malka Feige was
a worthy mate of such a husband. Kindhearted, unwearyingly
industrious, and devout, she was a typical Jewish housewife.
They had but one child, a blue-eyed, fair-haired boy of eight,
whom they loved with the passionate devotion of which parental
hearts are capable when they have but one object upon which to
concentrate their affection. He was literally the apple of their eyes.
His father cared for his intellectual welfare, and provided the best
and most highly esteemed Melammedim to introduce him into the
intricacies of the Jewish education of that time; and the lad, who had
a bright and acute intellect, responded well to these efforts, and at
eight was quite a little prodigy of Biblical and Talmudical learning.
His mother, on the other hand, looked after his physical well-being,
fed him on delicate food, clothed him in a jubitza of extra fine
material, brushed and combed his little peoth until they shone, and
set her pride upon making him finer and brighter in appearance than
his comrades. Like Hannah of old, she had determined to dedicate
her offspring to the Lord. Already in imagination she saw him seated
upon the rabbi’s seat, greeted by the plaudits of admiring thousands;
and so strong was her faith in that future for her son that she rarely
called him by his given name, which was Saul Isaac, but always
referred to him as “my little rabbi.” Thus the love, the hopes, the
ambition of these parents were all wrapped up in this, their only son.
Troublous times were just beginning then for the descendants of
Jacob living on Muscovite soil. Nicholas the First sat on the throne of
the Czars; and, like so many of the Russian potentates before and
after him, could find no more pressing task to perform than to
convert his Hebrew subjects to Christianity. He had no respect for
the conscientious scruples which kept the Jews faithful to their
ancestral religion; he could not appreciate the heroism with which
they endured every conceivable suffering and martyrdom rather than
grow recreant to the allegiance plighted to their God. In his eyes they
were only a mass of obdurate, stubborn, and pestiferous heretics,
who refused to see the beauties and accept the salvation of
Christianity. He thought and thought and cudgelled his brains to
devise some scheme by which to overcome the endless resistance of
Judaism to its own dissolution, and finally evolved a plan which for
sheer deviltry and refinement of heartless brutality would have done
credit to the blackest fiend in the legions of Satan; and this, too, in
the name of the religion which claims love and tenderness as its own
special prerogative, and calmly assumes all the progress of humanity
and civilization as its doing.
 His plan, in brief, was to separate the parents and the children.
With the old Jews, he knew nothing could be done. They would go to
the stake or the dungeon, and would not recant; but if, he reasoned,
the young Jews could be removed from parental influence, could be
caught, so to speak, before their characters were formed, and be
placed in charge of priests or other Christian officials, they would be
unable to resist, but would succumb to the powerful pressure
brought to bear upon them and would become genuine Christians.
This fiendish plan he proceeded, with icy deliberation, to put into
execution. What cared he for the cruelty or violent dissolution of
natural relations, for the tears of terrified children, for the
immeasurable woes and heart-breakings of bereaved parents. His
tyrant’s view of statecraft approved the plan and other
considerations had no weight. Then were legions of brutal emissaries
sent into the provinces reserved for the habitation of the children of
Jacob. Their conduct resembled that of brigands rather than of
officers of the law. In numbers so great as to defy resistance, they
would fall upon some unsuspecting Hebrew settlement, generally at
dead of night; would burst into the houses, and with utter disregard
of all considerations of justice or frenzied appeals for mercy, would
tear the weeping and terror-stricken children from the arms of their
screaming and frantically resisting parents, would throw them into
the ready standing wagons and would carry them off, never more to
return.
It would take the pen of a Dante and the brush of their own
Verestchagin fitly to depict the awful scenes which occurred on the
occasions of these visitations, the demoniacal brutality of the
despot’s henchmen, the helpless terror of the childish victims, and
the unutterable, paralyzed agony of the wretched fathers and
mothers who saw their beloved ones dragged away to that which for
them was worse than death, and could do nothing to save them from
their fate.
The same fate befell our Saul Isaac. It was a cold midwinter night.
The Rabbinowitz family were sleeping peacefully, all unsuspecting of
evil. Suddenly the sound of powerful blows upon the door caused
them to awake in terror. Too well they knew what those sounds
meant, although there had been no report that the “chappers,” as
they were called, were coming to their province. Hastily the agonized
parents sought to find some place of concealment for their son. A
second later the door fell beneath the shower of blows rained upon it,
and several ruffianly looking men, dressed in uniform, burst into the
room. Without showing any warrant or offering a word of
explanation, they seized the shrinking lad. Roughly they thrust aside
Israel, who would have protested, and flung off Malka Feige, who
clung to them in a half-insane effort to rescue her boy. The lad
himself they tossed into the wagon, into the midst of twenty or more
other lads, who already cowered there, and drove off.
Let us draw a veil over the unutterable sorrows of that parent pair,
thus foully deprived of the beloved of their souls. Heaven alone has
power to right wrongs such as these, and to the mercy and justice of
heaven we must commend them.
Let us follow Saul Isaac on the course which he was obliged to
pursue. His experience was not at first different from that of
thousands of others. He was taken to the convent of St. Sophia in the
neighborhood of Moscow. There a thorough Russian and Christian
education was given him, and every effort was made, by means of
mingled kindness and severity, to induce him voluntarily to accept
baptism, for even the perverted and tyrannical minds of his captors
perceived that a compulsory administration of the rite could have no
binding obligation upon the conscience. To be sure, their notions of
voluntary action were rather remarkably casuistical. Severe beatings,
periodical starvation, and longer or shorter terms of imprisonment
were all considered legitimate forms of missionary effort with which
to persuade the cantonists, as the abducted Hebrew children were
called, of the superiority of Christianity to Judaism, and to induce
them voluntarily to accept it.
It is a glorious tribute to the power of Jewish teachings that most
of these helpless victims, despite their tender years and pitiful
condition, were by no means quick to yield to the maltreatment or
blandishments of their masters. Most of them resisted for years;
some never yielded.
Four years were required to bring our Saul Isaac into the frame of
mind requisite for the acceptance of Christianity. At first he wept and
wailed constantly and would touch no food except dry bread and
water; and, young as he was, he refused to listen to the instruction of
the Russian monks. But as the weeks rolled into months and the
months into years, without seeing other than Gentile faces and
without any word from his parents or any other Jews, gradually his
recollections grew dimmer and his resolution weaker. Finally he no
longer objected to the Christian instructions, and in his twelfth year
he was baptized with great pomp and parade in the chapel of the
monastery, receiving the name of Sergei Pavlowitz. From this time
on his advancement was rapid. After three years of general education
he decided to enter upon the military career, and in his fifteenth year
he entered the Imperial Cadet School at St. Petersburg.
The memory of his parents had quite faded from his mind; or if the
thought of them ever came to him, they seemed like ghostly figures
of an unreal world, entirely devoid of actuality or connection with his
present existence.
Sergei Pavlowitz was one of the most popular students at the Cadet
School. His quick intellect, which had enabled him to comprehend
the abstruse debates of the Talmud, stood him in good stead in
mastering the details of military science, while his handsome figure
in the neat Russian uniform and his polite and obliging ways were
universally pleasing. In due course of time he graduated as a
lieutenant of artillery.
His career in the army justified the expectations of his student
years. He combined the two most requisite military qualities, high
capacity and rigid fidelity to duty. He became in rapid succession a
captain and then a colonel of artillery.
While holding the latter office he attracted the attention and then
aroused the love of Olga, the beautiful daughter of General Wladimir
de Mitkiewicz. Shortly afterward the General sent for him, and in due
form and in the most flattering terms offered to make him his son-
in-law. Such a distinguished honor could not be refused. To be sure,
a momentary pang went through the heart of the young colonel; and
the shadowy faces of his father and mother seemed to rise from the
gloomy recesses of the past and gaze at him reproachfully, but these
sensations were too dim and faint to have any effect. He accepted the
offer of the venerable General, which was, indeed, a most
complimentary one, and because of which he became the object of
many congratulations and no little envy.
In the magnificent cathedral of Kurski-Kazan the nuptials of the
dashing Colonel Pavlowitz and the beautiful and accomplished Olga
de Mitkiewicz were consummated with all the gorgeous ceremonial
of the Greek Church, and amidst an unprecedented display of wealth
and luxury. The vast edifice was crowded with representatives of the
noblest and finest families of the province, while the streets
surrounding the cathedral were thronged with a vast multitude of the
baser sort; and the personal interest and gratification which all
displayed were quite extraordinary.
It cannot be denied that the striking attentions and adulations of
which Colonel Pavlowitz became the recipient did almost turn his
head. In no other country are honors so much appreciated as in
Russia; and those he had received were quite exceptional, both in
extent and in cordiality.
He was happy, very happy; happy in the possession of the radiant,
beauteous creature he could now call his own, and from whose
sparkling eyes love and devotion, ardent and sincere, shone forth; he
was happy in the evident sympathy and admiration of all his
associates, and he was happy in the consciousness that his future was
secure and that he was destined to a brilliant and distinguished
career. Very faint and dim, indeed, were now the images of the
ghostly past, and they did not affect his actions in the slightest; but
somehow or other they would not forsake him, and he often found
himself wondering with a peevish sort of dissatisfaction and
impatience, why they did not leave him to enjoy undisturbed the
pleasures and honors of his present station.
Shortly after his marriage the Crimean war broke out. Russia was
engaged in a titanic struggle with the Western Powers, and Colonel
Pavlowitz was among those summoned to defend the fatherland. The
parting from his young wife was marked by tears and sobs; but still
he heard the summons to war with stern joy, for, like a true soldier,
he longed to display in actual combat the qualities he had gained in
theoretic instruction; and then he longed for action—intense, stirring
action—to drive away the shadowy, reproachful faces which tortured
him by their constant recurrence.
He was one of the commanders in charge of the defence of
Sebastopol. He was personally engaged, and displayed the greatest
gallantry in many of the desperate conflicts of that bloody campaign.
At Balaklava he was in command of a part of the artillery, which
received the world-renowned charge of the Light Brigade; and it was
while fiercely beating off that attack that an unexpected blow of a
British sabre took off his right arm near the shoulder.
For three months our hero lay in the hospital, the object of
universal sympathy and interest, for the good-will which had been
previously entertained toward him had been greatly heightened by
the splendid bravery and skill he had displayed in the war and the
cruel wound he had received.
The Emperor himself had sent several times to inquire concerning
his condition, and the visits and inquiries of lesser personages were
innumerable.
As soon as he was able to resume his active duties, the Emperor
ordered a review of the entire army. It was a glittering spectacle, a
sea of brilliant uniforms, shining bayonets, swords and cannons,
interspersed with magnificent bands of music, an ocean of deeply
interested onlookers. Our hero rode at the head of his regiment on a
splendid black charger, his empty sleeve hanging useless at his right
side. As he passed the grand stand where stood the Emperor and his
brilliant retinue of officers and aides, His Majesty ordered the parade
to halt. Then in the presence of the army and the serried throngs of
spectators, the Emperor addressed him as follows:
“Gen. Sergei Pavlowitz, my good and faithful servitor. I have
noticed the courage and devotion with which you have served in my
army. It is always my wish fitly to reward virtue and fidelity, and I
therefore appoint you to the command of the —th division of my
regular army.”
Hardly had these words, which His Majesty pronounced in a loud
and clear voice, been spoken, than the entire army, breaking for a
moment through the restraints of discipline, and the vast throng of
spectators, burst into enthusiastic hurrahs and cheered again and
again the name of Sergei Pavlowitz. It was a glorious and inspiring
moment.
Our hero flushed with pride and gratification; but, obedient to the
rules of military etiquette, said no word, but merely saluted with
profound reverence, and a second later the stern command rang
forth and the host marched on.
Words cannot describe the exultation which now filled the soul of
General Pavlowitz. He was fairly intoxicated with joy. Every ambition
of his life seemed gratified, and with rapture he thought of the
delight with which the news of his great advancement would fill the
heart of his beloved Olga, who had visited him during his stay in the
hospital, and had now returned to their home in Kursky Kazan.

AS THE CAVALCADE
PASSED A CORNER
THE GENERAL HEARD
A CRY

Page 111

Little did he reck that a tremendous change was impending, that

an event was about to occur which would recall with irresistible force
the events of his early life and change the entire current of his
military career. But so it was, and the climax of his military ambition
was also destined to mark its sudden and complete end.
The parade had been dismissed. The spectators had dispersed, and
the various regiments were marching back to their several barracks.
Accompanied only by his staff and a small escort of cavalry,
General Pavlowitz was returning to his headquarters. Their road led
through some of the old streets of the town. As the cavalcade passed
a corner the General heard a cry. He alone of all the company noticed
it, but there was something in it that thrilled and chilled him and
filled his frame with violent agitation. It was a wailing, sobbing cry in
a woman’s voice, and its burden was made up of a few words, oft-
repeated, in the Russo-Jewish dialect: “Oh, woe is me, my little
rabbi, my Saul Isaac! oh, woe is me, my little rabbi, my little rabbi!”
General Pavlowitz heard the cry and understood the words. Though
for more than twenty years he had heard and spoken only Russian,
yet those words came to him as the far-off echoes of his own past,
intelligible, familiar, sweet, and unutterably sad. Like a flash there
rolled away the many years of Russian, Christian, and military
training, and he saw himself again in the happy days of his
childhood, a little innocent Jewish boy, proudly reciting his week’s
lesson before a circle of admiring neighbors, while father and mother
beamed with satisfaction. Then, again, the memory of the awful night
when he was snatched from them, and he quivered again with fresh
horror and indignation. Turning his head as his horse trotted on, he
saw, standing at the corner an elderly Jewish couple, gazing after
him, with tears streaming from their eyes and an expression of
intensest anguish upon their faces, the woman wailing and sobbing
as in frenzy. He knew them at once. They were his father and mother.
His resolution was instantly formed. His parents and he should
meet. Hastily summoning a subaltern, who like himself was a
baptized Jew, he bade him leave the ranks unobserved, go back to
the old couple and inform them that the General would see them that
evening at a certain quiet hotel of the town.
Faithfully the subaltern fulfilled his chief’s commission, ignorant,
of course, of the reasons thereof, but with his soul filled with an
indefinable sympathy with its object, which instinctively he felt was
noble. Quietly he dropped behind the troop, and in a few hastily
spoken words communicated to the aged couple the wish of the
General, whereupon he put spurs to his horse and speedily rejoined
his companions, none of whom had observed his action.
That evening a young man in civilian attire inquired at the office of
the Narodski Hotel whether a certain Jewish couple were not at the
hotel, and was shown to the room where his parents (it was the
General) were awaiting him. The meeting was pathetic, almost tragic,
in the intensity of the emotions it aroused. The first sentiment was
that of great, overwhelming joy. The reunited parents and child wept
and smiled alternately, and embraced each other with a fervor only
possible to those whose hunger for love had remained so long
unsatisfied. Especially did Malka Feige clasp her long-lost son to her
breast in a paroxysm of maternal affection, and very, very reluctantly
did she release him from her embrace. But finally the first mighty
ebullitions of emotion had subsided somewhat and they began to
discuss their eventful career and the difficulties of their present
position.
The parents’ story was soon told. Their presence in Sebastopol was
quite accidental, or rather, as they devoutly believed, providential.
During all these years they had been unable to learn anything of the
fate of their boy. They knew neither the place where he had been kept
during the first few years after his abduction, nor anything of his
subsequent experiences; and all of their efforts to obtain some
information had remained entirely fruitless, so that finally they had
despaired of learning anything of him any more. A few days previous
to the memorable occasion of their reunion, Israel had received a
favorable business proposition which required his presence at
Sebastopol; and as Malka Feige did not care to remain at home in
utter solitude, she had determined to accompany him. They had not
gone to the review, for they had no heart for pageantry or splendor,
and it was quite by chance that they happened to be standing at the
corner of the street when the little company of cavalrymen with the
general rode by. Gazing at the company in a casual and apathetic
way, Malka Feige’s sharp eyes had at once noticed, despite the
disparity of age and brilliant uniform, the resemblance in the
features of the leader to those of her own Saul Isaac, and her
mother’s heart told her that this was her stolen boy. Then had she, in
a sudden and irrepressible outburst of feeling, uttered the cry which
attracted the attention of the General and brought about the
meeting.
Saul Isaac then told his parents the story of his experience, which,
as it is well known to my readers, need not be repeated. After he had
concluded, the conversation turned upon their future relations, and
they all recognized that it was a most difficult and dangerous one.
“Ah, dear son,” said Malka Feige, “what shall our future be? I
cannot live without you, now that my eyes have seen you alive; but
how can we come together, since we are but a humble Jewish couple
and you a great general, and especially since you have become, alas
for my sins! a Christian? It is indeed impossible for us to live
together. The Czar would never allow it.”
“Yes,” chimed in Israel, “and think what a disgrace it would be for
us to have it known in the Kehillah that my son, the Illuy and Charif,
was a Meshummed! I could never endure the shame of it. All your
glory would be no compensation.”
It was indeed a knotty and thorny problem. But Saul Isaac had
already reflected upon the matter in all its aspects, and with
customary promptness of resolution had determined what he would
do.
“Dear parents,” said he, “be at rest. Never shall I forsake you more.
Now that God, the God of my fathers, has brought us together thus
wonderfully, we shall never be separated again. I shall stay with you
and be a Jew, a sincere, loyal Jew. I know that I must renounce my
high rank, to which the Emperor has just appointed me, and all my
hopes for the future, and leave this country; for, as a Jew, not only
would every avenue be closed to me, but as an apostate I would be
sure of severe punishment, and, perhaps, even of death. But what
care I for that! I have never been sincerely a Christian. I only became
such because my power of resistance was gone and there seemed no
other prospect in life. But now that I see you again, my resolution is
formed, and is unalterable. I love you; I love my poor, persecuted
people; I love my God. I shall return to you and to Him with all my
heart and soul.”
The parents shed tears of joy, not unmingled with grief and
apprehension, at this heroic announcement.
“But how about your wife?” asked Malka Feige. “You are married
to one who is not of our religion, but who accepted you in good faith
and intention. Lawfully you may not abide with her, but honor
forbids you to leave her. What shall you do?”
“Of that, too, I have thought,” answered Saul Isaac. “I love my Olga
dearly, but my faith and my God are more precious to me than the
love of woman. I shall go to Olga, tell her frankly of all the
circumstances which surround me and ask her to accept our faith
and become a Jewess. If she consents, we shall leave the country
together and all will be well. If she refuses, I shall tell her that it were
better that we parted, for true, God-pleasing marriage cannot exist
between persons of different faiths. But, under all circumstances, I
am determined henceforth to be a true Jew, to live and die as such.”
The parents declared themselves satisfied with this solution of the
problem, and they separated with the understanding that Israel and
Malka Feige were to go home and Saul Isaac was to keep them
informed of all his movements.
The first step of General Pavlowitz after the reunion with his
parents was to seek leave of absence from the army to visit his wife in
Kursky-Kazan. This was granted him without difficulty, in
consideration of his meritorious services and his natural desire to
share the joy of his advancement with his wife. With every external
manifestation of joy, but with a heart filled with secret misgivings, he
set out on his journey. He feared much for the result upon his wife of
the revelation that he had reverted to Judaism, and hardly dared to
hope that she would look with favor upon his proposition that she
should accept the faith of her husband.
Knowing only too well the intense aversion with which his
brethren were regarded by the Russians belonging to the official
Greek Church, and having often had occasion to notice with what
scorn and contempt the name “Zid” was uttered by the haughty
representatives of Muscovite self-conceit, he realized keenly that no
greater shock could possibly be inflicted upon his Olga than the
announcement that her husband was one of the despised and hated
Jews. But it appeared to him that no other course was consistent
with honor and rectitude, and he determined not to deviate from the
straight path of duty.
Often during the long and tedious journey he tried to imagine the
answer which Olga would give. Sometimes he thought of her as
declaring that her husband’s faith and people should be hers, and
that with him she would go to the uttermost ends of the earth; at
other times he imagined her saying that the faith of her fathers stood
higher to her than aught else, and that she would never forsake it.
But in his wildest imaginings he did not form any notion of what the
actual reception of his words would be.
He had determined to make his announcement immediately after
his arrival at home; but when he saw the radiant face of his wife and
felt her warm kiss upon his lips, his heart failed him. How could he
speak words which might bring sorrow to such a beautiful and
affectionate creature. He suffered himself to be carried to his
splendid residence, and partook of the luxurious repast which Olga
had prepared for him. He simulated gayety, and spoke with affected
animation of the war and his part in it and his advancement and
brilliant future prospects. He determined to make his announcement
on the morrow. But on the morrow his courage had not returned,
and he could not speak. He who had faced charging armies
undaunted and looked death in the eye without flinching could not
make a statement which might grieve the woman to whom he had
given his name and who loved him so ardently. But on this day he
was abstracted and dejected, and could not suppress the sighs which
from time to time forced themselves from his breast.
Olga could not help noticing his melancholy. That evening she
determined to speak to him concerning its cause.
“Sergei, my love,” said she, when the evening repast had been
served and the servants had withdrawn, and they were nestling side
by side upon the luxurious divan, “Sergei, my love, something is
troubling you. My woman’s heart tells me that some secret grief is
eating out your soul. Will you not tell your Olga what it is? Will you
not let me share your grief?”
“Olga, dearest,” said Sergei, gazing at her with troubled eyes, while
sudden pains shot through his heart, “Olga, dearest, how can I tell
you what I know will grieve you and bring great sorrow upon her
whom I love and cherish more than myself?”
“Tell me,” she pleaded; “am I not your wife? Did I not swear to be
the partner of your joys and sorrows? Tell me your burden; and no
matter what it is, I shall help you bear it.”
“Well, then,” answered he, “since you urge me, I shall tell you.
Know, then, I am a Jew. Your husband, the great General Pavlowitz,
is but one of that abhorred race, one of those wretched pariahs whom
the Emperor and the people alike despise—a ‘Zid.’ Is it not sufficient
cause for grief that the high-born Olga de Mitkiewicz should be tied
to such a one, that he should be able to call her wife?”
Olga looked at him with eyes in which a curious light shone.
“What folly you speak, Sergei,” she said. “How can you call yourself
a Jew? To be sure, I know, and when I gave you my hand I knew, that
Hebrew blood flows in your veins; but it is now many years since you
renounced the sinful heresy of Judaism and were baptized into our
holy Greek Church in the chapel of the monastery of St. Sophia. How,
then, can you call yourself a Jew, since the church and our gracious
Emperor recognize you as good a Christian as any of us? Put away
these foolish thoughts, dear Sergei, and let not the fact of your
Hebrew descent trouble you in the least; and be assured that it does
not diminish my love for you in the slightest degree.”
Sergei gazed with tear-stained eyes for a moment at his wife, and
then spoke in a voice choking with emotion:
“Dearest Olga, what you say is well put, but I cannot recognize it as
correct. I was baptized against my will; my consent was insincere and
superficial. For a time I could disguise my real sentiments; to-day I
can do so no more. I am a Jew, in faith as well as in blood. I have
seen again my parents, and the sight of them has revived all my
olden feelings, all the childish love for my faith. No longer will I wear
the mask, will I play the part of being Christian. I am determined to
be a Jew. I intend to renounce all my offices and dignities and flee to
a land where I may be at liberty to live according to the dictates of
my conscience as such. My wife, too, should be a Jewess, should
share my beliefs and hopes. Olga, can you go with me; can you accept
our Jewish faith in one God and His holy law; can you resolve to
share my lot in my unknown future home and be a true partner to
me for life and for eternity? If you can, you will fill my heart with joy;
but I do not urge you to make the sacrifice. If you choose to remain
in your faith and your native land, you will be entitled to a legal
divorce. I would leave you all my property and possessions and will
never trouble you again. Speak, Olga, and tell me your decision?”
When Sergei had concluded he gazed again into his wife’s face,
anxious to know by its expression the manner in which she had
received his words. What he saw surprised him. He had expected to
see there the expression of anger or displeasure or, at best, surprise,
uncertainty, and hesitation.
Instead, he beheld the beautiful countenance of Olga, all radiant
with a strange and inexplicable joy. She was smiling a smile of
triumph, almost of exultation; but there was withal a solemnity in
her eyes which showed that there was no levity in her joy, but that it
was based upon some profoundly earnest sentiment. While he was
gazing at her, almost stupefied at her unexpected look, Olga began to
speak.
“Sergei,” said she, “you have told me your secret. I shall tell you
mine. You belong to a proscribed race; so do I, and am now really
your sister in faith. You are a Hebrew. I descend from the Subotnikis,
those sincere seekers after God whom the renowned Zacharia of
Tambow converted to Judaism some centuries ago. As a student of
Russian history, you know that the emperors persecuted the
“Judaizing heretics,” as my people were called, with even greater
cruelty and persistency than they did yours. Imprisonment,
deprivation of civil rights, and banishment to remote sections of the
empire, and even harsher punishments were inflicted upon them.
“Under these circumstances thousands of our brethren fell away
completely; others fled to foreign countries where they openly
professed Judaism; and others nominally adhered to the Greek
Church, but in their hearts secretly cherished their faith in the one
God of Israel and endeavored to fulfil His holy law as far as in their
ignorance and their difficult circumstances they could.
“My family belonged to the last-mentioned class; but through the
high connections it has formed, it had grown quite lax and out of
touch with the brethren. But we have, nevertheless, never forgotten
our origin; and, though I feared to tell it to you, thinking you had
become a thorough Christian and would not like to be reminded of
your former state, your Hebrew descent was really one of the causes
which gained for you my affections, for we Subotnikis honor and
revere those native born in the household of Israel very much, and
esteem a marriage alliance with them a high privilege.
“Your announcement, therefore, of your intention to be a Jew,
instead of displeasing me, has afforded me the keenest joy, a joy I
never expected to feel. I shall accept your faith, dear Sergei, not
merely because I desire to please you, as my husband, but because
my heart already inclines toward it with sincere devotion. I shall
share your lot and your future, whatever they may bring of joy or
sorrow. And like Ruth of old I shall say: ‘Thy people shall be my
people and thy God my God. Whither thou goest I shall go; and
where thou diest I shall die, and there shall I be buried.’”
Words cannot describe the tremendous revulsion of feeling which
the words of Olga, so unexpected, produced in the breast of our hero,
whom we shall henceforth call only by his Hebrew cognomen of Saul
Isaac. He was transported from the depth of misery and
apprehension to the seventh heaven of joy by this so pleasing
solution of a difficulty which he had looked upon as almost insoluble.
But Olga was also filled with joy, and the radiant gladness which
shone from her beautiful eyes showed that she considered that hour,
which meant for her the beginning of exile and, perhaps, of poverty,
as the happiest of her life.
The husband and wife, now joined by a new and profound
sympathy, embraced each other with a fervor of love they had not
known before, after which they sat down to write a letter to the
parents of Saul Isaac. In this letter Saul Isaac gave expression to the
happiness which filled his heart, and Olga wrote a few kindly lines,
closing with the words, “Your loving daughter and faithful handmaid
of Abraham.”
The happy couple now made quiet preparations to leave the land.
Gradually the general disposed of his property and turned it into
cash. When this had been accomplished, after several months, the
General and his wife left the town of their residence quite openly,
under the plausible pretext of making a short foreign tour. Their first
destination was a frontier town of Roumania, whither Israel and
Malka Feige had preceded them. From this place Saul Isaac wrote to
the Minister of War, resigning his commission in the Russian army
and frankly stating his reasons for his action. Then they proceeded to
Jerusalem, where the parents of Saul Isaac had resolved to pass their
declining years in pious seclusion and the service of God. In the holy
city Olga was formally received into the community of Israel, the
name of Sarah being conferred upon her.
Here they lived for twenty years. Six children were born unto
them, all of whom received an excellent Hebrew and secular training,
and were reared to industry, virtue, and the fear of God. After the
death of the parents, which occurred in the twentieth year of their
sojourn in the holy city, Saul Isaac and Sarah thought it desirable, in
the interest of their children, to emigrate to America. Accordingly
they settled in New York some years ago. Saul Isaac and his wife
selected for their residence a portion of the city mainly inhabited by
Russian co-religionists, for in their midst they felt themselves most
at home.
 Saul Isaac finds his chief pleasure in attendance at synagogue, and
it is a question open to debate which affords him the most pleasure,
the sermons of the Maggid or the gossip and anecdotes in which the
congregation indulges in the intervals of services.
As for Sarah, she is so thoroughly Judaized, so punctual and exact
in the fulfilment of her religious duties, so particular in maintaining
the Kosher character of her household and such a fluent speaker of
the Russo-Jewish jargon, that one would never suspect in her
anything but a genuine Russian Jewess, native and to the manner
born. Their children have grown up to be handsome and talented
young men and women, good Jews and good Americans.
Saul Isaac and Sarah are happy and contented. No tinge of regret
for their former state ever enters their hearts. But often as they
worship in the synagogue there comes spontaneously to their lips the
words of Solomon: “Blessed be the Lord God, who hath given rest to
His people Israel.”
 TOO LATE, BUT ON TIME

Moses Levinsky awoke with a start upon his humble couch in the
little hall bedroom in the sixth story of the immense and crowded
tenement-house in Eldridge Street, New York City, in which he
dwelt. He very much feared that he had overslept himself and would
be late at the early morning service of the Congregation Sons of
Peace. The light which shown through the narrow window of his
room was much brighter than the pale illumination which usually
greeted his early waking eyes and seemed to show that the day was
further advanced. A glance at the cheap silver watch which lay upon
his trousers on the chair next to his bed showed him that his
apprehensions were only too well founded.
The Congregation Sons of Peace invariably began its devotions at 6
A.M. Moses Levinsky was in the habit of rising at half-past five; his
toilet and the walk to the little meeting-room in the next block
required twenty-five minutes, and he was regularly in his place five
minutes before the voice of the Chazan or precentor, chanting in
classic Hebrew, “Exalted be the living God and praised,” betokened
that the service of adoration and supplication, with which modern
Israel supplies the place of the ancient sacrificial worship, had
begun. But to-day the watch which usually indicated about a quarter
past five when he first glanced at it in the early mornings, stood at
half-past six. The congregation had already been engaged in prayer
for a full half-hour, and he could hardly hope to be with them before
the services, which usually lasted somewhat less than an hour, were
concluded. Watches and clocks are obstinate creatures. They persist
in their opinions, which can be plainly read in their faces. They care
not at all how disagreeable or unpleasant their statements may be to
those who consult them, and they can neither be reasoned with nor
stared out of countenance. And so Moses Levinsky’s watch did not
recede at all for all the hard stares which that rather confused
individual directed at it; but, on the contrary, advanced a minute or
so, while he, who had now risen upon his side and rested upon his
left arm, gazed at it with puzzled and rueful countenance.
The truth was that Moses was in doubt as to the right course to
pursue. His watch told him that he might as well make an exception
to-day from his regular practice and stay at home, for he could never
hope to be on time at the services, or even present during any
considerable portion of them. On the other hand, his conscience
smote him greatly at having overslept himself; and thus incurred the
danger of breaking his life rule, of always beginning the day in the
house of God, and in the words which the ship captain once
addressed to the prophet Jonah when he had gone to sleep in the
midst of all the turmoil of the storm, it called to him, “What aileth
thee, O sleeper? Arise, cry out unto thy God.” After a minute’s
hesitation conscience won the battle over comfort. Moses hastily
sprang from his couch, made his simple toilet as speedily as possible,
and in something less than twenty minutes was on his way to the
little synagogue (“place of prayer” was the unassuming name which
the worshippers themselves gave it) of the Congregation of the Sons
of Peace. While he is on his way thither, we will take occasion to
describe him to our readers; for many of them, no doubt, are at a loss
to understand what kind of a person he is, and particularly fail to
comprehend why he should be so dreadfully put out at the mere
possibility of being absent from prayers one morning, a thing which,
I am sure, would never disturb the majority of my worthy readers in
their mental tranquillity.