An Introduction to

Partial Least Squares Regression

Randall D. Tobias, SAS Institute Inc., Cary, NC

Abstract Partial least squares (PLS) is a method for construct-

ing predictive models when the factors are many and
highly collinear. Note that the emphasis is on pre-
Partial least squares is a popular method for soft dicting the responses and not necessarily on trying
modelling in industrial applications. This paper intro- to understand the underlying relationship between the
duces the basic concepts and illustrates them with variables. For example, PLS is not usually appropriate
a chemometric example. An appendix describes the for screening out factors that have a negligible effect
experimental PLS procedure of SAS/STAT software. on the response. However, when prediction is the
goal and there is no practical need to limit the number
of measured factors, PLS can be a useful tool.
Introduction
PLS was developed in the 1960’s by Herman Wold
as an econometric technique, but some of its most
Research in science and engineering often involves avid proponents (including Wold’s son Svante) are
using controllable and/or easy-to-measure variables chemical engineers and chemometricians. In addi-
(factors) to explain, regulate, or predict the behavior of tion to spectrometric calibration as discussed above,
other variables (responses). When the factors are few PLS has been applied to monitoring and controlling
in number, are not significantly redundant (collinear), industrial processes; a large process can easily have
and have a well-understood relationship to the re- hundreds of controllable variables and dozens of out-
sponses, then multiple linear regression (MLR) can puts.
be a good way to turn data into information. However,
if any of these three conditions breaks down, MLR The next section gives a brief overview of how PLS
can be inefficient or inappropriate. In such so-called works, relating it to other multivariate techniques such
soft science applications, the researcher is faced with as principal components regression and maximum re-
many variables and ill-understood relationships, and dundancy analysis. An extended chemometric exam-
the object is merely to construct a good predictive ple is presented that demonstrates how PLS models
model. For example, spectrographs are often used are evaluated and how their components are inter-
preted. A final section discusses alternatives and
Component 1 = 0.370
Component 2 = 0.152
extensions of PLS. The appendices introduce the ex-
Component 3 = 0.337 perimental PLS procedure for performing partial least
Component 4 = 0.494
Component 5 = 0.593
squares and related modeling techniques.

How Does PLS Work?

In principle, MLR can be used with very many factors.

However, if the number of factors gets too large (for
Figure 2: Spectrograph for a mixture example, greater than the number of observations),
you are likely to get a model that fits the sampled
to estimate the amount of different compounds in a data perfectly but that will fail to predict new data well.
chemical sample. (See Figure 2.) In this case, the This phenomenon is called over-fitting. In such cases,
factors are the measurements that comprise the spec- although there are many manifest factors, there may
trum; they can number in the hundreds but are likely be only a few underlying or latent factors that account
to be highly collinear. The responses are component for most of the variation in the response. The general
amounts that the researcher wants to predict in future idea of PLS is to try to extract these latent factors,
samples. accounting for as much of the manifest factor variation

1
as possible while modeling the responses well. For successive pairs of scores is as strong as pos-
this reason, the acronym PLS has also been taken sible. In principle, this is like a robust form of
to mean ‘‘projection to latent structure.’’ It should be redundancy analysis, seeking directions in the
noted, however, that the term ‘‘latent’’ does not have factor space that are associated with high vari-
the same technical meaning in the context of PLS as ation in the responses but biasing them toward
it does for other multivariate techniques. In particular, directions that are accurately predicted.
PLS does not yield consistent estimates of what are
called ‘‘latent variables’’ in formal structural equation Another way to relate the three techniques is to note
modelling (Dykstra 1983, 1985). that PCR is based on the spectral decomposition of
X 0 X , where X is the matrix of factor values; MRA is
based on the spectral decomposition of Y^ 0 Y^ , where
Figure 3 gives a schematic outline of the method.

Y^ is the matrix of (predicted) response values; and

The overall goal (shown in the lower box) is to use

PLS is based on the singular value decomposition of

T U
X 0 Y . In SAS software, both the REG procedure and
SAS/INSIGHT software implement forms of principal
components regression; redundancy analysis can be
performed using the TRANSREG procedure.
Factors Responses If the number of extracted factors is greater than or
equal to the rank of the sample factor space, then
Sample
PLS is equivalent to MLR. An important feature of the
method is that usually a great deal fewer factors are
required. The precise number of extracted factors is
usually chosen by some heuristic technique based on
Factors Responses the amount of residual variation. Another approach
is to construct the PLS model for a given number of
factors on one set of data and then to test it on another,
Population choosing the number of extracted factors for which
the total prediction error is minimized. Alternatively,
Figure 3: Indirect modeling van der Voet (1994) suggests choosing the least
number of extracted factors whose residuals are not
the factors to predict the responses in the population. significantly greater than those of the model with
This is achieved indirectly by extracting latent vari- minimum error. If no convenient test set is available,
ables T and U from sampled factors and responses, then each observation can be used in turn as a test
respectively. The extracted factors T (also referred set; this is known as cross-validation.
to as X-scores) are used to predict the Y-scores U ,
and then the predicted Y-scores are used to construct
predictions for the responses. This procedure actu- Example: Spectrometric Calibra-
ally covers various techniques, depending on which
source of variation is considered most crucial. tion
 Principal Components Regression (PCR):
Suppose you have a chemical process whose yield
The X-scores are chosen to explain as much
has five different components. You use an instrument
of the factor variation as possible. This ap-
to predict the amounts of these components based
proach yields informative directions in the factor
on a spectrum. In order to calibrate the instrument,
space, but they may not be associated with the
you run 20 different known combinations of the five
shape of the predicted surface.
components through it and observe the spectra. The
 Maximum Redundancy Analysis (MRA) (van results are twenty spectra with their associated com-
den Wollenberg 1977): The Y-scores are cho- ponent amounts, as in Figure 2.
sen to explain as much of the predicted Y varia-
PLS can be used to construct a linear predictive
tion as possible. This approach seeks directions
model for the component amounts based on the spec-
in the factor space that are associated with the
trum. Each spectrum is comprised of measurements
most variation in the responses, but the predic-
at 1,000 different frequencies; these are the factor
tions may not be very accurate.
levels, and the responses are the five component
 Partial Least Squares: The X- and Y-scores amounts. The left-hand side of Table shows the
are chosen so that the relationship between individual and cumulative variation accounted for by

2
 Table 2: PLS analysis of spectral calibration, with cross-validation
Number of Percent Variation Accounted For Cross-validation
PLS Factors Responses Comparison
Factors Current Total Current Total PRESS P
0 1.067 0
1 39.35 39.35 28.70 28.70 0.929 0
2 29.93 69.28 25.57 54.27 0.851 0
3 7.94 77.22 21.87 76.14 0.728 0
4 6.40 83.62 6.45 82.59 0.600 0.002
5 2.07 85.69 16.95 99.54 0.312 0.261
6 1.20 86.89 0.38 99.92 0.305 0.428
7 1.15 88.04 0.04 99.96 0.305 0.478
8 1.12 89.16 0.02 99.98 0.306 0.023
9 1.06 90.22 0.01 99.99 0.304 *
10 1.02 91.24 0.01 100.00 0.306 0.091

the first ten PLS factors, for both the factors and the this case, the PLS predictions can be interpreted as
responses. Notice that the first five PLS factors ac- contrasts between broad bands of frequencies.
count for almost all of the variation in the responses,
with the fifth factor accounting for a sizable proportion.
This gives a strong indication that five PLS factors are Discussion
appropriate for modeling the five component amounts.
The cross-validation analysis confirms this: although
the model with nine PLS factors achieves the absolute As discussed in the introductory section, soft science
minimum predicted residual sum of squares (PRESS), applications involve so many variables that it is not
it is insignificantly better than the model with only five practical to seek a ‘‘hard’’ model explicitly relating
them all. Partial least squares is one solution for such
factors.
problems, but there are others, including
The PLS factors are computed as certain linear combi-
nations of the spectral amplitudes, and the responses  other factor extraction techniques, like principal
are predicted linearly based on these extracted fac- components regression and maximum redun-
tors. Thus, the final predictive function for each dancy analysis
response is also a linear combination of the spectral
amplitudes. The trace for the resulting predictor of  ridge regression, a technique that originated
the first response is plotted in Figure 4. Notice that within the field of statistics (Hoerl and Kennard
1970) as a method for handling collinearity in
regression
 neural networks, which originated with attempts
in computer science and biology to simulate the
way animal brains recognize patterns (Haykin
1994, Sarle 1994)

Ridge regression and neural nets are probably the

strongest competitors for PLS in terms of flexibility
and robustness of the predictive models, but neither
Figure 4: PLS predictor coefficients for one response of them explicitly incorporates dimension reduction---
that is, linearly extracting a relatively few latent factors
a PLS prediction is not associated with a single fre- that are most useful in modeling the response. For
quency or even just a few, as would be the case if more discussion of the pros and cons of soft modeling
we tried to choose optimal frequencies for predicting alternatives, see Frank and Friedman (1993).
each response (stepwise regression). Instead, PLS
prediction is a function of all of the input factors. In There are also modifications and extensions of partial
least squares. The SIMPLS algorithm of de Jong

3
(1993) is a closely related technique. It is exactly Haykin, S. (1994). Neural Networks, a Comprehen-
the same as PLS when there is only one response sive Foundation. New York: Macmillan.
and invariably gives very similar results, but it can
be dramatically more efficient to compute when there Helland, I. (1988), ‘‘On the structure of partial least
are many factors. Continuum regression (Stone and squares regression,’’ Communications in Statis-
Brooks 1990) adds a continuous parameter , where tics, Simulation and Computation, 17(2), 581-
0    1, allowing the modeling method to vary 607.
continuously between MLR ( = 0), PLS ( = 0:5),
and PCR ( = 1). De Jong and Kiers (1992) de- Hoerl, A. and Kennard, R. (1970), ‘‘Ridge regression:
scribe a related technique called principal covariates biased estimation for non-orthogonal problems,’’
regression. Technometrics, 12, 55-67.
In any case, PLS has become an established tool in
chemometric modeling, primarily because it is often de Jong, S. and Kiers, H. (1992), ‘‘Principal covari-
possible to interpret the extracted factors in terms ates regression,’’ Chemometrics and Intelligent
of the underlying physical system---that is, to derive Laboratory Systems, 14, 155-164.
‘‘hard’’ modeling information from the soft model. More
work is needed on applying statistical methods to the de Jong, S. (1993), ‘‘SIMPLS: An alternative approach
selection of the model. The idea of van der Voet to partial least squares regression,’’ Chemomet-
(1994) for randomization-based model comparison is rics and Intelligent Laboratory Systems, 18, 251-
a promising advance in this direction. 263.

Naes, T. and Martens, H. (1985), ‘‘Comparison of pre-

For Further Reading diction methods for multicollinear data,’’ Com-
munications in Statistics, Simulation and Com-
PLS is still evolving as a statistical modeling tech- putation, 14(3), 545-576.
nique, and thus there is no standard text yet that gives
it in-depth coverage. Geladi and Kowalski (1986) is Ranner, Lindgren, Geladi, and Wold, ‘‘A PLS kernel
a standard reference introducing PLS in chemomet- algorithm for data sets with many variables and
ric applications. For technical details, see Naes and fewer objects,’’ Journal of Chemometrics, 8, 111-
Martens (1985) and de Jong (1993), as well as the 125.
references in the latter.
Sarle, W.S. (1994), ‘‘Neural Networks and Statis-
tical Models,’’ Proceedings of the Nineteenth
References Annual SAS Users Group International Confer-
ence, Cary, NC: SAS Institute, 1538-1550.

Dijkstra, T. (1983), ‘‘Some comments on maximum Stone, M. and Brooks, R. (1990), ‘‘Continuum regres-
likelihood and partial least squares methods,’’ sion: Cross-validated sequentially constructed
Journal of Econometrics, 22, 67-90. prediction embracing ordinary least squares,
partial least squares, and principal components
Dijkstra, T. (1985). Latent variables in linear stochas- regression,’’ Journal of the Royal Statistical So-
tic models: Reflections on maximum likelihood ciety, Series B, 52(2), 237-269.
and partial least squares methods. 2nd ed. Ams-
terdam, The Netherlands: Sociometric Research van den Wollenberg, A.L. (1977), ‘‘Redundancy
Foundation. Analysis--An Alternative to Canonical Correla-
tion Analysis,’’ Psychometrika, 42, 207-219.
Geladi, P, and Kowalski, B. (1986), ‘‘Partial least-
squares regression: A tutorial,’’ Analytica Chim- van der Voet, H. (1994), ‘‘Comparing the predictive ac-
ica Acta, 185, 1-17. curacy of models using a simple randomization
test,’’ Chemometrics and Intelligent Laboratory
Frank, I. and Friedman, J. (1993), ‘‘A statistical view Systems, 25, 313-323.
of some chemometrics regression tools,’’ Tech- SAS, SAS/INSIGHT, and SAS/STAT are registered
nometrics, 35, 109-135. trademarks of SAS Institute Inc. in the USA and other
countries.  indicates USA registration.

4
Appendix 1: PROC PLS: An Exper- is equivalent to standard PLS when
there is only one response, and it
imental SAS Procedure for Partial invariably gives very similar results.
Least Squares METHOD=PCR
specifies principal components re-
An experimental SAS/STAT software procedure, gression.
PROC PLS, is available with Release 6.11 of the
You can specify the following PLS-options
SAS System for performing various factor-extraction
in parentheses after METHOD=PLS:
methods of modeling, including partial least squares.
Other methods currently supported include alternative ALGORITHM=PLS-algorithm
algorithms for PLS, such as the SIMPLS method of de gives the specific algorithm used to
Jong (1993) and the RLGW method of Rannar et al. compute PLS factors. Available algo-
(1994), as well as principal components regression. rithms are
Maximum redundancy analysis will also be included in
a future release. Factors can be specified using GLM-
ITER the usual iterative NIPALS al-
type modeling, allowing for polynomial, cross-product,
gorithm
and classification effects. The procedure offers a wide
variety of methods for performing cross-validation on SVD singular value decomposi-
the number of factors, with an optional test for the tion of X 0 Y , the most exact
appropriate number of factors. There are output data but least efficient approach
sets for cross-validation and model information as well EIG eigenvalue decomposition of
as for predicted values and estimated factor scores. Y 0 XX 0 Y
RLGW an iterative approach that
You can specify the following statements with the PLS
is efficient when there are
procedure. Items within the brackets <> are optional.
many factors

PROC PLS <options>; MAXITER=number

CLASS class-variables; gives the maximum number of itera-
MODEL responses = effects < / option >; tions for the ITER and RLGW algo-
OUTPUT OUT=SAS-data-set <options>; rithms. The default is 200.
EPSILON=number
gives the convergence criterion for
PROC PLS Statement the ITER and RLGW algorithms. The
PROC PLS <options>; default is 10,12 .
CV = cross-validation-method
You use the PROC PLS statement to invoke the PLS specifies the cross-validation method to
procedure and optionally to indicate the analysis data be used. If you do not specify a cross-
and method. The following options are available: validation method, the default action is
not to perform cross-validation. You can
specify any one of the following:
DATA = SAS-data-set
specifies the input SAS data set that con- CV = ONE
tains the factor and response values. specifies one-at-a-time cross- valida-
METHOD = factor-extraction-method tion
specifies the general factor extraction CV = SPLIT < ( n ) >
method to be used. You can specify any specifies that every nth observation
one of the following: be excluded. You may optionally
METHOD=PLS < (PLS-options) > specify n; the default is 7.
specifies partial least squares. This is CV = BLOCK < ( n ) >
the default factor extraction method.
specifies that blocks of n observa-
METHOD=SIMPLS tions be excluded. You may option-
specifies the SIMPLS method of de ally specify n; the default is 7.
Jong (1993). This is a more effi- CV = RANDOM < ( cv-random-opts ) >
cient algorithm than standard PLS; it

5
 CLASS Statement
specifies that random observations
be excluded. CLASS class-variables;
CV = TESTSET(SAS-data-set)
specifies a test set of observations to You use the CLASS statement to identify classifica-
be used for cross-validation. tion variables, which are factors that separate the
observations into groups.
You also can specify the following cv-
random-opts in parentheses after CV = Class-variables can be either numeric or character.
RANDOM: The PLS procedure uses the formatted values of
NITER = number class-variables in forming model effects. Any variable
specifies the number of random sub- in the model that is not listed in the CLASS statement
sets to exclude. is assumed to be continuous. Continuous variables
must be numeric.
NTEST = number
specifies the number of observations
in each random subset chosen for
MODEL Statement
exclusion.
SEED = number MODEL responses = effects < / INTERCEPT >;
specifies the seed value for random
number generation. You use the MODEL statement to specify the re-
CVTEST < ( cv-test-options ) > sponse variables and the independent effects used
specifies that van der Voet’s (1994) to model them. Usually you will just list the names
randomization-based model comparison of the independent variables as the model effects,
test be performed on each cross-validated but you can also use the effects notation of PROC
model. You also can specify the follow- GLM to specify polynomial effects and interactions.
ing cv-test-options in parentheses after By default the factors are centered and thus no inter-
CVTEST: cept is required in the model, but you can specify the
INTERCEPT option to override this behavior.
PVAL = number
specifies the cut-off probability for
declaring a significant difference. The
OUTPUT Statement
default is 0.10.
STAT = test-statistic OUTPUT OUT=SAS-data-set keyword = names
specifies the test statistic for the < : : : keyword = names >;
model comparison. You can specify
either T2, for Hotelling’s T 2 statistic, You use the OUTPUT statement to specify a data
or PRESS, for the predicted residual set to receive quantities that can be computed for
sum of squares. T2 is the default. every input observation, such as extracted factors
NSAMP = number and predicted values. The following keywords are
available:
specifies the number of randomiza-
tions to perform. The default is 1000.
PREDICTED predicted values for responses
LV = number
specifies the number of factors to extract. YRESIDUAL residuals for responses
The default number of factors to extract is XRESIDUAL residuals for factors
the number of input factors, in which case XSCORE extracted factors (X-scores, latent
the analysis is equivalent to a regular least vectors, T )
squares regression of the responses on
the input factors. YSCORE extracted responses (Y-scores, U )
OUTMODEL = SAS-data-set STDY standardized Y variables
specifies a name for a data set to contain STDX standardized X variables
information about the fit model. H approximate measure of influence
OUTCV = SAS-data-set
PRESS predicted residual sum of squares
specifies a name for a data set to contain
information about the cross-validation. T2 scaled sum of squares of scores

6
XQRES sum of squares of scaled residuals
for factors
YQRES sum of squares of scaled residuals
for responses

Appendix 2: Example Code

The data for the spectrometric calibration example is

in the form of a SAS data set called SPECTRA with
20 observations, one for each test combination of the
five components. The variables are

X1 : : : X1000 - the spectrum for this combination

Y1 : : : Y5 - the component amounts

There is also a test data set of 20 more observations

available for cross-validation. The following state-
ments use PROC PLS to analyze the data, using the
SIMPLS algorithm and selecting the number of factors
with cross-validation.

proc pls data = spectra

method = simpls
lv = 9
cv = testset(test5)
cvtest(stat=press);
model y1-y5 = x1-x1000;
run;

The listing has two parts (Figure 5), the first part
summarizing the cross-validation and the second part
showing how much variation is explained by each ex-
tracted factor for both the factors and the responses.
Note that the extracted factors are labeled ‘‘latent
variables’’ in the listing.

7
 The PLS Procedure
Cross Validation for the Number of Latent Variables

Test for larger

residuals than
minimum
Number of Root
Latent Mean Prob >
Variables PRESS PRESS
-----------------------------------
0 1.0670 0
1 0.9286 0
2 0.8510 0
3 0.7282 0
4 0.6001 0.00500
5 0.3123 0.6140
6 0.3051 0.6140
7 0.3047 0.3530
8 0.3055 0.4270
9 0.3045 1.0000
10 0.3061 0.0700

Minimum Root Mean PRESS = 0.304457 for 9 latent variables

Smallest model with p-value > 0.1: 5 latent variables

The PLS Procedure

Percent Variation Accounted For

Number of
Latent Model Effects Dependent Variables
Variables Current Total Current Total
----------------------------------------------------------
1 39.3526 39.3526 28.7022 28.7022
2 29.9369 69.2895 25.5759 54.2780
3 7.9333 77.2228 21.8631 76.1411
4 6.4014 83.6242 6.4502 82.5913
5 2.0679 85.6920 16.9573 99.5486

Figure 5: PROC PLS output for spectrometric calibration example