Econometric Theory, 8, 1992, 413-419. Printed in the United States of America.

APPLIED NONPARAMETRIC REGRESSION

W. Hardle
Cambridge University Press, 1930
MIGUEL A, DELGADO
Universidad Carlos III

1. OVERVIEW

Applied Nonparametric Regression by Wolfgang Hardle {Econometric Society Monographs 19, Cambridge University Press) concentrates on the statistical aspects of nonparametric regression from an applied point of view.
The literature on nonparametric functional estimation is enormous and
rapidly growing. Books on the subject are available in abundant supply. Tapia and Thompson [29], Devroye and Gyorfy [8], Silverman [27], and
Devroye [7] are monographs on nonparametric density estimation. A short
overview of the literature on nonparametric regression is given by Collomb
[5] with an update (Collomb [6]). Some chapters of Ibragimov and Hasminskii [14] deal with nonparametric regression, focusing on optimal rates of
convergence. The broad field of nonparametric curve estimation, including
nonparametric regression, is reviewed in Prakasa Rao [18]. Miiller [15] is a
monograph on nonparametric regression analysis of longitudinal data, focusing on the fixed design model. Gyorfy et al. [11] discusses nonparametric regression in time series. Hastie and Tibshirani [12] deal with estimation
of generalized additive models based on nonparametric regression techniques.
Hardle directs his book to applied statisticians who are interested in regression analysis. Much of the nonparametric literature is quite technical and
published in journals that are not frequently consulted by practitioners. However, nonparametric regression is intended to be applied in practice. Silverman
[26] points out that: "A particularly disappointing feature of the technical
nature of much of the literature on nonparametric density estimation is that
it may even have had a negative effect by scaring off potential users on the
method and by making it difficult for courses on the subject to be constructed." Hardle's book offers a balance between theoretical reasoning, intuitive explanations, discussion of assumptions and interpretation of results,
and many numerical examples with illustrative figures. The discussion is frequently illustrated with economic applications. The book, is well organized
and well written and the typographical presentation is good. It should read
well for graduate students in economics and can be used in a topics course.

414

BOOK REVIEWS

Each chapter section includes a set of exercises. Most problems are designed
for understanding the material in the text. Some of them require numerical
work. A number of exercises are proof of extensions or specializations of results in the text.
The book is divided into two parts. In the first part, alternative estimation
methods are described and compared. In the second part, important practical
issues like the choice of smoothing parameter, accuracy of nonparametric
estimates, rates of convergence, robustness, and applications to semiparametric models, are addressed. The focus of attention is on the kernel method.
The book also includes an appendix where the XploRe package is discussed.
This book covers an enormous amount of material. Each chapter contains
numerous bibliographical references. Inevitably, the 344 references do not
constitute a comprehensive bibliography and some topics have been skipped.
We have missed references to econometric literature, where periodically applications of nonparametric regression techniques can be found.
2. THE BOOK CHAPTER BY CHAPTER

Chapters 1 and 2 explain the basic model, show intuitively what a nonparametric estimate is, and motivate its use. The basic model is the regression equation Yt = m{Xi) 4- error, / = l,...,/f, where {(Yi9Xi), i =
l,...,n)
is a data set, Y is scalar and X is a p x 1 vector, and m(-) is of unknown
functional form. The value of m() evaluated at the point X = &9 m(&), is
estimated by a local average of those Y points in a neighborhood of &. That
is, w ( c ) is estimated by m(x) = S? =1 Wni($,)Yi9 where [Wni{&), i =
1,...,) is a sequence of weights which are a function of the regressors
[Xiy i = 1 , . . . , n} and a smoothing number which regulates the size of the
neighborhood around <x. The choice of the smoothing parameter is the basic problem of nonparametric regression. A large neighborhood will yield an
"oversmooth" estimate m that is biased. On the other hand, a local average
on a very small neighborhood will produce an "undersmooth59 estimate with
large variability.
Hardle gives four main motives for using nonparametric regression: exploratory data analysis, prediction, outlier detection, and treatment of missing observations. A fifth motivation, equally important in econometrics, is
the improvement of estimation of parametric models by incorporating nonparametric regression estimates; that is, semiparametric estimation.
Chapter 3 reviews various nonparametric regression techniques. Kernels,
nearest neighbors, orthogonal series, and splines are discussed in different
sections, and other smoothers like recursive techniques, the regressogram,
delta functions, median smoothing, split linear fits, and empirical regression,
are discussed. Another section compares bias and variance of kernels, nearest neighbors, and splines, and provides an interesting numerical example
comparing the three estimators under appropriate choice of the smooth-

BOOK REVIEWS

415

ing numbers. Computationally efficient algorithms for kernels and nearest

neighbors are presented for X scalar. The kernel algorithm uses the fast
Fourier transform, generalizing the algorithm of Silverman [25] for density estimation. The nearest neighbors algorithm uses a recursive formula.
The generalization of these algorithms for X muitivariate is not particularly
straightforward.
The discussion is centered on the case where X is scalar. This decision is
taken for notational simplicity and because the book proposes to analyze
multiple regression by using additive models in Chapter 10, where it suffices
to apply univariate nonparametric regression. For X muitivariate it is only
considered the product kernel. In applications, where the regressors are correlated (e.g., time series), it may be preferable to use other muitivariate kernels which use a matrix of bandwidths or some muitivariate scale factor (see,
e.g., Robinson [21]); that is, weights of the form W}(<c) =K(H~l(& -Xj))/
Tij=\ K(H~l(&-\-Xj)), where K( -) is a muitivariate kernel function and if is
a positive definite matrix of bandwidths. When H = hlp9 h scalar, and Ip the
identity matrix, &n&K(u) =IIf=i k(Ui), u = (ul9... 9up)' and k(-) is a univariate kernel function, these weights specialize to the product kernel weights.
In addition, Chapter 3 includes consistency results and asymptotic approximations for bias and variance of different nonparametric regression estimators. The moments of certain nonparametric regression estimates (e.g., kernel
estimates in the random design model) may only exist under restrictive conditions in view of the random denominator. Universal consistency results are
available, where no requirements are imposed onm(-) or the distribution of
(Y9X) but moment conditions on Y; see, for example, Stone [28] and
Devroye and Wagner [9],
Chapters 4 to 10 concentrate on the kernel method. Chapter 4 deals with
rates of convergence and accuracy of the kernel estimate, first defining different measures of accuracy and discussing lower, achievable, and optimal
rates of convergence. The moral is that rates of convergence depend on the
smoothness of m(-)9 the dimension of X, and the type of kernel that is used.
The second part of the chapter addresses topics referring to the accuracy of
the kernel estimate, including algorithms for constructing pointwise confidence intervals and variability bands, accuracy at the boundary of the observations, the behavior of the mean squared error as a function of the kernel
function, and "higher order59 kernels for reducing the bias.
Chapter 5 discusses automatic techniques for choosing the smoothing number with n fixed. The different methods optimize some quadratic error measure for the regression curve. These methods are cross-validation, penalizing
functions, and "plug-in." While the two first methods optimize some data
dependent function, "plug-in" estimates the "optimal bandwidth" which minimizes the asymptotic approximation of the mean squared error. The
"plug-in" method needs preliminary estimates of the conditional variance,
the density of the regressor, and the two first derivatives of the regression

416

BOOK REVIEWS

function. The procedure can be Iterated If desired. Hardle does not recommend it for use In practice because it does not have a better rate of convergence than its competitors, assuming smoothness of the regression function,
and introducing some noise from the prior estimates. However, many applications use the "plug-In" because it is easy to implement. Some methods are
presented for locally adapting the choice of the smoothing parameter to the
curvative of the regression curve. A method, based on canonical kernels, for
comparing bandwidths corresponding to different kernel functions Is also discussed.
Chapter 6 presents robust techniques for estimating conditional location
functions. Because a kernel estimate is based on a square loss function, robustness considerations are motivated in the same way as parametric estimation of a location parameter. Conditional R9 L, and M estimators are discussed, as well as local weighted scatter plot smoothing, a method based on
local polynomial fits. Simultaneous estimation of conditional location and
scale in order to obtain scale invariant conditional M estimates Is also considered. Scale invariant conditional M estimates can also be obtained by using
a preliminary robust conditional scale estimate as proposed by Boente and
Fraiman [2,3], who proved strong consistency and asymptotic normality under a-mlxing processes for kernels and nearest neighbors.
Chapter 7 reviews nonparametric regression techniques under mixing processes. The chapter includes an economic example where Yt = log price of
gold and Xt = Yi_l. What is found is "a quite nonlinear prediction curve
rather than a global linear prediction." The gold market is highly efficient
and linear unpredictability of asset-returns is theoretically justified. Though
economic theory cannot rule out possible nonlinearities in conditional means,
various empirical studies have found that nonparametric prediction techniques deliver no improvement on the random walk In asset-returns; see for
example, Prescott and Stengos [19] who used kernel smoothers In the Canadian gold market, and Diebold and Nason [10] who used local weighted scatter plot smoothing In the exchange rates market. In financial markets,
nonparametric estimation seems especially interesting for estimating volatility
of assets returns; see, for example, Pagan and Ullah [20]. The chapter also
discusses nonparametric regression in the nonrandom design model with correlated errors.
Chapter 8 deals with estimation under monotonicity and unimodality restrictions, and estimation of zeros and extrema. Estimation under monotonicity and unimodality restrictions is interesting for estimating Engle curves
and other economic functional. The Pool-adjacent-vlolators (PAV) algorithm consists of sorting the Ays, averaging the adjacent values of the corresponding 17s which violate the monotonicity restriction, then estimating the
regression curve at these points by this average. Estimation under unimodality Is related to estimation under monotonicity. If m(<&) is the mode of m(*)

BOOK REVIEWS

417

the regression is monotonic increasing for those values of X smaller than x

and monotonic decreasing for those values of X greater than &. Therefore,
the PAV algorithm can be applied directly. When the mode is unknown it can
be estimated from a prior estimate of m('). Alternatively, the PAV algorithm
can be applied to several possible modes and that which leads to the lowest
residual sum of squares is then chosen. The zeros of m () or its derivatives
can be estimated by the method known as Passive Stochastic Approximation
(PSA). The PSA algorithm consists of creating a sequence which converges
to the zero of m(). The algorithm can also be implemented for estimating
the location of extrema.
Chapter 9 is concerned with semiparametric inference. Only one semiparametric model is discussed: The partial linear model Yt = Xf/3 + g(Zt) + error , where Z and X are regressors, g(-) is an unknown function, and (3 a
vector of unknown parameters. Several results on partial linear models are
presented but, surprisingly, there is no mention of the asymptotic distribution of the f3 estimates. In particular, it is not mentioned that Robinson [23]
obtained an estimate based on "higher order" kernels. This estimate is asymptotically equivalent, up to the first order, to the unfeasible linear leastsquares estimate based on the regression of Yt E{ Yt \ Zt) on Xt E{Xt \ Zt)
(where the conditional expectations are perfectly known). Nonparametric regression has also been used in other semiparametric models; for example, asymptotically efficient estimation under heteroskedasticity of unknown form:
Carroll [4] and Robinson [22], asymptotic efficient estimation in simultaneous nonlinear equations models by using optimal instrumental variables:
Newey [16], or asymptotically efficient estimation in the presence of autocorrelation of unknown form: Hidalgo [13]. A survey on semiparametric estimation in econometrics is given in Robinson [24]. There is also a unified
approach for semiparametric estimation based on nonparametric regression
estimates due to Andrews [1], None of this wide-ranging subject matter is discussed in the book.
Chapter 9 also discusses shape invariant transformation of location and
scale of nonparametric regression curves and the estimation of the transformation parameters. In another section a test is presented for functional specification of regression curves based on a statistic that is a modification of
the mean squared deviation between the regression curve estimated parametricaily and nonparametrically. Also presented are bootstrap alternatives for
getting the critical values.
Chapter 10 discusses nonparametric estimation of additive models for investigating multiple regression. First Bardie explains why nonparametric multiple regression is not recommended in practice:
First, the regression function m(&) is a high dimensional surface and since it
cannot be displayed for d > 2, it does not provide a geometrical description of

418

BOOK REVIEWS

the regression relationship between X and Y. Second, the basic element of nonparametric smoothing averaging over neighborhoods will often be applied to
a relatively meager set of points since even samples of size n > 1000 are surprisingly sparse in the higher dimensional Euclidean space.

Then, it is advised to take advantage of the fact that a regression surface

may be of a simple additive form. However, assuming a specific semiparametric form of the regression function, for example, an additive model, may
yield inconsistent regression estimates. In semiparametric estimation the interest centers on the performance of the parameter estimates and the researcher wants to be sure that the nuisance function is estimated consistently.
Of course, the researcher should exploit all the available information on the
nuisance function.
Several techniques for estimating a regression surface, which is a function
of certain linear combinations of the regressors, are reviewed. These techniques include estimation of regression trees by recursive splitting, projection pursuit regression, and averaged derivative estimation. This last method
has become popular in econometrics for the estimation of index models
where m(&) = g(<iT@), and where #() is unknown and /3 is a vector of parameters. Also presented is a method, based on "alternating conditional expectations/' for finding transformation functions ^f(-) and #,-() such that
E(ty(Y)\X=x)
= Sf=i gi(T@). Algorithms are also presented for estimating generalized additive models where the dependent variable is an indicator function and G(m(&)) = S?=i i(&rj3), where G(-) is known and &()
unknown.
3. FINAL COMMENTS

Applied Nonparametric Regression is an excellent book. It will be valuable

reference for econometricians and can be used for preparing a course on the
topic. The book reads very well. The author has intuitively evoked the different topics, providing pertinent applications and exercises in the correct
place. The book makes reference to the excellent software package XploRe
(see Ng and Sickles [17] for a review) that implements many techniques discussed in the text.
REFERENCES

1. Andrews, D.W.K. "Asymptotics for semiparametric econometric models: I estimation and

testing," Preprint, 1990.
2. Boente, G. & R. Fraiman. Robust nonparametric estimation for dependent observations.
Annals of Statistics 17 (1989): 1242-1256.
3. Boente, G. & R. Fraiman. Asymptotic distribution of robust estimators for nonparametric models from mixing processes. Annals of Statistics 18 (1990): 891-906.
4. Carroll, R.J. Adapting for heteroskedasticity in linear models. Annals of Statistics 10 (1982):
1224-1233.

BOOK REVIEWS

419

5. Collomb, G. Estimation non-parametrique de la regression: revue bibliographique. International Statistical Review 49 (1981): 75-93.
6. Collomb, G. Non-parametric regression: an up-to-date bibliography. Statistics 16 (1985):
309-324.
7. Devroye, L. A Course in Density Estimation. Boston: Birkhauser, 1987.
8. Devroye, L. & L. Gyorfi. Nonparametric Density Estimation: The Lx View. New York:
Wiley, 1985.
9. Devroye, L. & T.J. Wagner. Distribution-free consistency results in nonparametric discrimination and regression function estimation. Annals of Statistics 8 (1980): 231-239.
10. Diebold, F.X. & J.A. Nason. Nonparametric exchange rate prediction? Journal of International Economics 28 (1990): 315-332.
11. Gyorfy, L., W. Handle, P. Sarda & P. Vieu. Nonparametric Curve Estimation from Time
Series. Berlin: Springer-Verlag, 1989.
12. Hastie, T.J. & R.J. Tibshirani. Generalized Additive Models. London: Chapman and Hall,
1990.
13. Hidalgo, F. J. Adaptive semiparametric estimation in the presence of autocorrelation of unknown form. Econometric Theory Journal of Time Series Analysis 13 (1992): 47-48.
14. Ibragimov, LA. & R.Z. Haminskii. Statistical Estimation. New-York: Springer-Verlag, 1981.
15. Miiller, H-G. Nonparametric Regression Analysis of Longitudinal Data. Berlin: SpringerVerlag, 1988.
16. Newey, W.K. Efficient instrumental variable estimation of nonlinear models. Econometrica
58 (1990): 809-837.
17. Ng, P.T. & R.C. Sickles. XploRe'-ing the world of nonparametric analysis. Journal of Applied Econometrics 5 (1990): 293-298.
18. Prakasa Rao, B.L.S. Nonparametric Functional Estimation. Orlando: Academic Press, 1983.
19. Prescott, D.M. & T. Stengos. "Do asset markets overlook exploitable nonlinearities? The
case of gold," Preprint, 1988.
20. Pagan, A.R. & A. Ullah. The econometric analysis of models with risk terms. Journal of
Applied Econometrics 3 (1988): 87-105.
21. Robinson, P.M. Nonparametric estimators for time series. Journal of Time Series Analysis
4(1983): 185-207.
22. Robinson, P.M. Asymptotically efficient estimation in the presence of heteroskedasticity
of unknown form. Econometrica 55 (1987): 531-548.
23. Robinson, P.M. Root-^-consistent semiparametric regression. Econometrica 56 (1988):
931-954.
24. Robinson, P.M. Semiparametric econometrics: A survey. Journal of Applied Econometrics
3(1988): 35-51.
25. Silverman, B.W. Kernel density estimation using the Fast Fourier Transform, Statistical
Algorithm 175. Applied Statistics 31 (1982): 93-97.
26. Silverman, B.W. Reviews of Nonparametric Functional Estimation by Prakasa Rao and
Nonparametric Density Estimation: The Lx View by Luc Devroye and Laszlo Gyorfy.
Annals of Statistics 13 (1985): 1630-1638.
27. Silverman, B.W. Density Estimation for Statistics and Data Analysis. London: Chapman
and Hall, 1986.
28. Stone, C.J. Consistent nonparametric regression (with discussion). Annals of Statistics 5
(1977): 595-645.
29. Tapia, R.A. & J.R. Thompson. Nonparametric Probability Density Estimation. Baltimore:
John Hopkins University Press, 1978.