Introduction to Regression Analysis Least squares criterion for determining regression coefficients

Statistics for Engineers (CHMT3016/CIVN2011)

Lecture series 10a - Simple Linear Regression
Dr Honest Walter Chipoyera
School of Statistics and Actuarial Science
University of the Witwatersrand,
P Bag 3, Wits 2050
Republic of South Africa
E-mail: [email protected]
Date: October 18, 2022

1 / 14
Introduction to Regression Analysis Least squares criterion for determining regression coefficients

Outline

1 Introduction to Regression Analysis

2 Least squares criterion for determining regression coefficients

The least squares estimators of the regression coefficients

2 / 14
Introduction to Regression Analysis Least squares criterion for determining regression coefficients

Introduction to Linear Regression Analysis

What this lecture series is all about

In this lecture series, you will be introduced to Basic Linear

Regression Analysis
Unit Objectives
At the end of this lecture series, you should be able to:
appreciate the vital elements of Elementary Linear Regression
State when it is appropriate to fit a linear regression model
calculate the least square estimators of the slope and intercept
of a linear model
calculate confidence intervals and test hypotheses related to
regression parameters
calculate and interpret the sample correlation coefficient
test for the significance of linear regression model
3 / 14
Introduction to Regression Analysis Least squares criterion for determining regression coefficients

Introduction to simple linear regression

Linear regression is used to determine the possibility of a

linear relationship between two or more variables.
In the simplest case, one value each is recorded on two
variables (call them X and Y ) on each subject (or
experimental unit).
The objective is then to find the best straight line through
the data.
In some situations, the slope and/or intercept have a scientific
meaning. In other cases, you use the linear regression line as a
standard curve to find new values of Y from X .

4 / 14
Introduction to Regression Analysis Least squares criterion for determining regression coefficients

In general, the goal of linear regression is to find the line that

best predicts Y from X.
Linear regression does this by finding the line that minimizes
the sum of the squares of the vertical distances of the points
from the line.
If two variables are assumed to be linearly related, you find
the slope and intercept that make a straight line best fit your
data.
Least squares criterion for determining regression coefficients

The goal of linear regression is to adjust the values of the

slope and intercept to find the line that best predicts Y using
X.
More precisely, the goal of regression is to minimize the sum
of the squares of the vertical distances of the points from the
line.
5 / 14
Introduction to Regression Analysis Least squares criterion for determining regression coefficients

6 / 14
Introduction to Regression Analysis Least squares criterion for determining regression coefficients

The linear model relating Y the (dependent variable) to X

the independent variable (on which Y is presumed to depend
on) is
Y = βo + β1 X + , (1)
where βo (called the intercept) and β1 (called the slope) are
constants.

7 / 14
Introduction to Regression Analysis Least squares criterion for determining regression coefficients

The slope quantifies the steepness of the line. It equals the

change in Y for each unit change in X. It is expressed in the
units of the Y-axis divided by the units of the X-axis.
If the slope is positive, Y increases as X increases. If the slope
is negative, Y decreases as X increases.
The Y-intercept is the Y value of the line when X equals zero.
It defines the elevation of the line.

8 / 14
Introduction to Regression Analysis Least squares criterion for determining regression coefficients

The least squares estimators of the regression coefficients

The least squares estimators of the regression coefficients

We assume we have collected n pairs of (sample) data values

on X and Y : (X1 , Y1 ), (X2 , Y2 ), · · · , (Xn , Yn ).
Together, the least squares estimators of βo and β1 , which we
denote by β̂o and β̂1 will minimize the residual sum of
squares, SSres given by:
n
X
SSres = (Yi − (βo + β1 Xi ))2 = SStotal − SSreg . (2)
i=1

9 / 14
Introduction to Regression Analysis Least squares criterion for determining regression coefficients

The least squares estimators of the regression coefficients

Lemma 1
The least squares estimators of the simple regression coefficients
are given by
SSXY
β̂1 = , and β̂o = Ȳ − β̂1 X̄ (3)
SSXX

where
n
X n
X
SSXY = (Xi − X̄ )(Yi − Ȳ ) and SSXX = (Xi − X̄ )2 . (4)
i=1 i=1

Remark 1
It can be easily verified that
n Pn Pn
X ( i=1 Xi ) ( i=1 Yi )
SSXY = Xi Yi − and (5)
n
i=1
10 / 14
Introduction to Regression Analysis Least squares criterion for determining regression coefficients

The least squares estimators of the regression coefficients

Remark 2

Together, the results in Equations (5) and (6) are the

convenient computational formulae for calculating SSXY and
SSXX .
The values of ni=1 Xi , ni=1 Yi , ni=1 Xi Yi , ni=1 Xi2 can all
P P P P
be obtained at the touch of button from a scientific calculator
once the data have been input.

11 / 14
Introduction to Regression Analysis Least squares criterion for determining regression coefficients

The least squares estimators of the regression coefficients

Example 1
The length L of a metal rod is measured at various temperatures
T.

Temperature, t(o C ) 20 30 40 50 60 70 80 85
Length, l(mm) 102 107 109 109 113 117 120 121

Draw a scatterplot for the data on graph paper. Do you think

that the two variables could be linearly related?
State an appropriate linear model relating T and L.

12 / 14
Introduction to Regression Analysis Least squares criterion for determining regression coefficients

The least squares estimators of the regression coefficients

13 / 14
Introduction to Regression Analysis Least squares criterion for determining regression coefficients

The least squares estimators of the regression coefficients

Example 2
(cont’
Comment on the suitability or otherwise of linear relationship
between L and t.Indeed, a linear relationship is suitable
P P P 2
Verify that
P ti = 435,
P 2 li = 898, ti = 27525,
ti li = 49915 and li = 101114.
Calculate SStt , SStl and SSll .
SStt = 27525 − 4352 /8 = 3871.875,
SStl = 49915 − 435 × 898/8 = 1086.25
SSll = 101114 − 8982 /8 = 313.5
Calculate the least square estimators of the regression
coefficients.
SStl 1086.25
β̂1 = = = 0.280549,
SStt 3871.875
898 435
β̂o = − 0.280549 × = 96.99515.
8 8 14 / 14