Least squares method of

estimation - I
Odoi Makuru
September 7, 2023
Introduction
• Let (x1 , y1 ), (x2 , y2 ), ... , (xn , yn ) be the n observed data points, with corresponding
errors εi , i = 1, ... , n. That is,

Yi = β0 + β1 xi + εi , i = 1, ... , n

• The three unknown parameters, the intercept parameter β0 , the slope parameter
β, and the error variance σ 2 , are estimated from the data set
• We assume that the errors εi , i = 1, ... , n are independent and identically
distributed with E(εi ) = 0, i = 1, ... , n, and Var(εi ) = σ 2 , i = 1, ... , n - N 0, σ 2


• The straight line model for the response Y for a given x is

Y = β0 + β1 x + ε

• Because we assumed that E(εi ) = 0, the expected value of Y is given by

E[Y ] = β0 + β1 x

• The estimator of the E(Y ), denoted by Ŷ , can be obtained by using the estimators
β̂0 and β̂1 of the parameters β0 and β1 , respectively
• Then, the fitted regression line we are looking for is given by

Ŷ = β0 + β1 x
Definition
For observed values (xi , yi ), we obtain the estimated value of yi as

ŷi = β̂0 + β̂1 xi

The deviation of observed yi from its predicted value ŷi , called the ith residual,
is defined by h  i
εi = yi − ŷi = yi − β̂0 + β̂1 xi

The residuals, or errors εi , are the vertical distances between observed and
predicted values of yi ‘s
Definition
The sum of squares of deviations to be minimized is
n
X n h
X  i2
2
SSE = (yi − ŷi ) = yi − β̂0 + β̂1 xi
i=1 i=1

The quantity SSE is also called the sum of squares for error
Remark
The quantities β̂0 and β̂1 that make the SSE a minimum are called the
least-squares estimates of the parameters β0 and β0 , and the corresponding
line ŷ = β̂0 + β̂1 x is called the least-squares line

Remark
SSE in some texts is referred to as Residual Sum of Squares (RSS)
Definition
The least-squares line ŷ = β̂0 + β̂1 x is one that satisfies the following property:
n
X
SSE = (yi − ŷi )2
i=1

is a minimum for any other straight-line model with

n
X
SE = (yi − ŷi ) = 0
i=1

Thus, the least-squares line is a line of the form y = β0 + β1 x for which the error
n
(yi − β0 − β1 x)2 is a minimum. The minimum is taken over
P
sum of squares
i=1
all values of β0 and β1 , and (x1 , y1 ), (x2 , y2 ), ... , (xn , yn ) are observed data pairs
Derivation of β̂0 and β̂1

If SSE attains a minimum, then the partial derivatives of SSE with respect to β0 and β1
are zeros. That is,
n 
P 2
∂ [yi − (β0 + β1 xi )]
∂ SSE i=1
=
∂β0 ∂β0
Xn
=− 2 [yi − (β0 + β1 xi )] (1)
i=1
n n
!
X X
=−2 yi − nβ0 − β1 xi =0
i=1 i=1

Least squares method of estimation - I

and
n 
P 2
∂ [yi − (β1 + β1 xi )]
∂ SSE i=1
=
∂β1 ∂β1
n
X
=− 2 [yi − (β0 + β1 xi )] xi (2)
i=1
n n n
!
X X X
=−2 xi yi − β0 xi − β 1 xi2 =0
i=1 i=1 i=1
Equations (1) and (2) are called the least squares equations for estimating the
parameters of a line
From (1) and (2) we obtain a set of linear equations called the normal equations,
n
X n
X
yi = nβ0 + β1 xi (3)
i=1 i=1

and
n
X n
X n
X
xi yi = β0 xi + β1 xi2 (4)
i=1 i=1 i=1
Solving for β0 and β1 from Equations (3) and (4), we obtain
n
P n
P
n
P n
P n
P n
P n
P xi yi
i=1 i=1
(xi − x) (yi − y) n xi y i − xi yi xi yi − n
i=1 i=1 i=1 i=1 i=1
β̂1 = n =  n 2 =  n
2 (5)
P 2 n
(xi − x)
P
xi
xi2 n
P P
n − xi
xi2 −
P i=1
i=1 i=1 i=1 n
i=1

and
β̂0 = y − β̂1 x (6)
To simplify the formula for β̂1 , set
 n
2 n n
P P P
n xi n xi yi
i=1 i=1 i=1
X X
sxx = xi2 − and sxy = xi yi −
n n
i=1 i=1

so
sxy
β̂1 =
sxx
Procedure for fitting a least-squares line

a) Form the n data points (x1 , y1 ), (x2 , y2 ), ... , (xn , yn ), and compute the following
quantities:
X n X n X n X n X n
xi , xi2 , yi , yi2 , xi yi , x and y
i=1 i=1 i=1 i=1 i=1

b) Compute sxx and sxy

c) Compute β̂0 and β̂1
d) The fitted least-squares line is ŷ = β̂0 + β̂1 x
e) For a graphical representation, in the xy-plane, plot all the data points and draw
the least-squares line obtained in step d)

Least squares method of estimation - I

Example 1
Production Usage
The manager of a car plant wishes Jan 4.51 2.48
to investigate how the plant‘s Feb 3.58 2.26
electricity usage depends upon the Mar 4.31 2.47
plant‘s production. The data set Apr 5.06 2.77
shown on the right is compiled and May 5.64 2.99
provides the plant‘s production and Jun 4.99 3.05
electrical usage for each month of Jul 5.29 3.18
the previous year. The electrical Aug 5.83 3.46
usage is in units of a million Sep 4.70 3.03
kilowatt-hours, and the production Oct 5.61 3.26
is measured as the value in Nov 4.90 2.67
million-Malawi Kwacha units of the Dec 4.20 2.53
cars produced in that month
Solution to example 1

For this example n = 12 and

12
X
xi =4.51 + · · · + 4.20 = 58.62
i=1
12
X
yi =2.48 + · · · + 2.53 = 34.15
i=1
12
X
xi2 =4.512 + · · · + 4.202 = 291.2310
i=1
X12
yi2 =2.482 + · · · + 2.532 = 98.6967
i=1

Least squares method of estimation - I

and
12
X
xi yi = (4.51 × 2.48) + · · · + (4.20 × 2.53) = 169.2532
i=1

The estimate of the slope parameter is therefore

n
 n  n 
P P P
n xi yi − xi yi
i=1 i=1 i=1
β̂1 =  n 2
n
2
P P
n xi − xi
i=1 i=1
(12 × 169.2532) − (58.62 × 34.15)
= = 0.49883
(12 × 291.2310) − 58.622

and the estimate of the intercept parameter is

 
34.15 58.62
β̂0 = y − β̂1 x = − 0.49883 × = 0.4090
12 12
The fitted regression line is thus

y = β̂0 + β̂1 x = 0.409 + 0.499x

which is shown together with the data points in the figure below. If a production level
of MWK5.5 million worth of cars is planned for next month, then the plant manager can
predict that the
ŷ |5.5 = 0.409 + (0.499 × 5.5) = 3.1535

With β̂1 = 0.499, this model predicts that the electricity usage increases by about half
a million kilowatt-hours for every additional MWK 1 million of production
The error variance

The estimate of the error variance for the given example is

n n n
yi2 − β̂0
P P P
xi yi yi − β̂1
2 SSE i=1 i=1 i=1
σ̂ = =
n−2 n−2
98.6967 − (0.4090 × 34.15) − (0.49883 × 169.2532)
=
10
=0.0299

so
√
σ̂ = 0.0299 = 0.1729

Least squares method of estimation - I

Lecture 3 ends here

Reading assignment
Read and do all examples in Section 8.2 and do Exercise 8.2 of Kandethody
M.Ramachandran & Chris P.Tsokos, (2009) Mathematical Statistics with
Applications, ISBN 13: 978-0-12-374848-5

Least squares method of estimation - I

References

A. Hayter.
Probability and Statistics for Engineers and Scientists.
Brooks/Cole, Cengage Learning, 2012.
K. M. Ramachandran and C. P. Tsokos.
Mathematical Statistics with Applications:.
Elsevier Academic Press, 2009.
D. D. Wackerly, W. Mendenhall, and R. L. . Scheaffer.
Mathematical Statistics with Applications.
Brooks/Cole, Cengage Learning, 2008.

Least squares method of estimation - I