Department of M.A.C.

S
NIT-K

MACHINE LEARNING

Lecture 11 by Nandagopal S A
Assistant Lecturer
Department of M.A.C.S
NIT-K

February 14, 2025

M. FOUNDATIONS OF DATA SCIENCE

1 Machine Learning
Regression
INTRODUCTION

■ We turn now to a discussion of supervised learning, starting with regression.

■ The goal of regression is to predict the value of one or more continuous target
variables t given the value of a D-dimensional vector x of input variables.
■ We have already encountered an example of a regression problem when we
considered polynomial curve fitting. The polynomial is a specific example of a
broad class of functions called linear regression models, which share the
property of being linear functions of the adjustable parameters.

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 2 / 11

INTRODUCTION

■ We turn now to a discussion of supervised learning, starting with regression.

■ The goal of regression is to predict the value of one or more continuous target
variables t given the value of a D-dimensional vector x of input variables.
■ We have already encountered an example of a regression problem when we
considered polynomial curve fitting. The polynomial is a specific example of a
broad class of functions called linear regression models, which share the
property of being linear functions of the adjustable parameters.

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 2 / 11

INTRODUCTION

■ We turn now to a discussion of supervised learning, starting with regression.

■ The goal of regression is to predict the value of one or more continuous target
variables t given the value of a D-dimensional vector x of input variables.
■ We have already encountered an example of a regression problem when we
considered polynomial curve fitting. The polynomial is a specific example of a
broad class of functions called linear regression models, which share the
property of being linear functions of the adjustable parameters.

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 2 / 11

CONTD..

▶ The polynomial is a specific example of a broad class of functions called

linear regression models, which share the property of being linear functions of
the adjustable parameters, and which will form the focus of this chapter.
▶ The simplest form of linear regression models are also linear functions of the
input variables.
▶ However, we can obtain a much more useful class of functions by taking linear
combinations of a fixed set of nonlinear functions of the input variables, known
as basis functions.
▶ Such models are linear functions of the parameters, which gives them simple
analytical properties, and yet can be nonlinear with respect to the input
variables.

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 3 / 11

CONTD..

▶ The polynomial is a specific example of a broad class of functions called

linear regression models, which share the property of being linear functions of
the adjustable parameters, and which will form the focus of this chapter.
▶ The simplest form of linear regression models are also linear functions of the
input variables.
▶ However, we can obtain a much more useful class of functions by taking linear
combinations of a fixed set of nonlinear functions of the input variables, known
as basis functions.
▶ Such models are linear functions of the parameters, which gives them simple
analytical properties, and yet can be nonlinear with respect to the input
variables.

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 3 / 11

CONTD..

▶ The polynomial is a specific example of a broad class of functions called

linear regression models, which share the property of being linear functions of
the adjustable parameters, and which will form the focus of this chapter.
▶ The simplest form of linear regression models are also linear functions of the
input variables.
▶ However, we can obtain a much more useful class of functions by taking linear
combinations of a fixed set of nonlinear functions of the input variables, known
as basis functions.
▶ Such models are linear functions of the parameters, which gives them simple
analytical properties, and yet can be nonlinear with respect to the input
variables.

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 3 / 11

CONTD..

▶ The polynomial is a specific example of a broad class of functions called

linear regression models, which share the property of being linear functions of
the adjustable parameters, and which will form the focus of this chapter.
▶ The simplest form of linear regression models are also linear functions of the
input variables.
▶ However, we can obtain a much more useful class of functions by taking linear
combinations of a fixed set of nonlinear functions of the input variables, known
as basis functions.
▶ Such models are linear functions of the parameters, which gives them simple
analytical properties, and yet can be nonlinear with respect to the input
variables.

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 3 / 11

LINEAR BASIS

▶ The simplest linear model for regression is one that involves a linear
combination of the input variables
ypx, wq “ w0 ` w1 x1 ` . . . ` wD xD
where x “ px1 , . . . , xD qT .
▶ This is known as linear regression. The key property of this model is that it is
a linear function of the parameters w0 , . . . , wD .

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 4 / 11

LINEAR BASIS

▶ The simplest linear model for regression is one that involves a linear
combination of the input variables
ypx, wq “ w0 ` w1 x1 ` . . . ` wD xD
where x “ px1 , . . . , xD qT .
▶ This is known as linear regression. The key property of this model is that it is
a linear function of the parameters w0 , . . . , wD .

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 4 / 11

LINEAR BASIS

▶ The simplest linear model for regression is one that involves a linear
combination of the input variables
ypx, wq “ w0 ` w1 x1 ` . . . ` wD xD
where x “ px1 , . . . , xD qT .
▶ This is known as linear regression. The key property of this model is that it is
a linear function of the parameters w0 , . . . , wD .

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 4 / 11

CONTD..

▶ It is also, however, a linear function of the input variables xi , and this imposes
significant limitations on the model. We therefore extend the class of models
by considering linear combinations of fixed nonlinear functions of the input
variables, of the form

M´1
ÿ
ypx, wq “ w0 ` wj ϕj pxq
j“1

where ϕj pxq are known as basis functions. By denoting the maximum value of
the index j by M ´ 1, the total number of parameters in this model will be M.

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 5 / 11

CONTD..

▶ The parameter w0 allows for any fixed offset in the data and is sometimes
called a bias parameter (not to be confused with ’bias’ in a statistical sense).
It is often convenient to define an additional dummy ’basis function’ ϕ0 pxq “ 1
so that

M´1
ÿ
ypx, wq “ wj ϕj pxq “ wT ϕpxq
j“0
˘T
▶ where w “ pw0 , . . . , wM´1 qT and ϕ “ ϕ0 , . . . , ϕM´1 . In many practical
`

applications of pattern recognition, we will apply some form of fixed

pre-processing,or feature extraction, to the original data variables.

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 6 / 11

CONTD..

▶ The parameter w0 allows for any fixed offset in the data and is sometimes
called a bias parameter (not to be confused with ’bias’ in a statistical sense).
It is often convenient to define an additional dummy ’basis function’ ϕ0 pxq “ 1
so that

M´1
ÿ
ypx, wq “ wj ϕj pxq “ wT ϕpxq
j“0
˘T
▶ where w “ pw0 , . . . , wM´1 qT and ϕ “ ϕ0 , . . . , ϕM´1 . In many practical
`

applications of pattern recognition, we will apply some form of fixed

pre-processing,or feature extraction, to the original data variables.

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 6 / 11

TYPES OF BASIS FUNCTIONS

▶ The example of polynomial regression is a particular example of this model in

which there is a single input variable x, and the basis functions take the form
of powers of x so that ϕj pxq “ xj .
▶ One limitation of polynomial basis functions is that they are global functions of
the input variable, so that changes in one region of input space affect all other
regions.
▶ This can be resolved by dividing the input space up into regions and fit a
different polynomial in each region, leading to spline functions .

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 7 / 11

TYPES OF BASIS FUNCTIONS

▶ The example of polynomial regression is a particular example of this model in

which there is a single input variable x, and the basis functions take the form
of powers of x so that ϕj pxq “ xj .
▶ One limitation of polynomial basis functions is that they are global functions of
the input variable, so that changes in one region of input space affect all other
regions.
▶ This can be resolved by dividing the input space up into regions and fit a
different polynomial in each region, leading to spline functions .

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 7 / 11

TYPES OF BASIS FUNCTIONS

▶ The example of polynomial regression is a particular example of this model in

which there is a single input variable x, and the basis functions take the form
of powers of x so that ϕj pxq “ xj .
▶ One limitation of polynomial basis functions is that they are global functions of
the input variable, so that changes in one region of input space affect all other
regions.
▶ This can be resolved by dividing the input space up into regions and fit a
different polynomial in each region, leading to spline functions .

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 7 / 11

TYPES OF BASIS FUNCTIONS

▶ The example of polynomial regression is a particular example of this model in

which there is a single input variable x, and the basis functions take the form
of powers of x so that ϕj pxq “ xj .
▶ One limitation of polynomial basis functions is that they are global functions of
the input variable, so that changes in one region of input space affect all other
regions.
▶ This can be resolved by dividing the input space up into regions and fit a
different polynomial in each region, leading to spline functions .

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 7 / 11

CONTD..

Other possible choices for the basis functions, are:

# ` ˘2 +
x ´ µj
ϕj pxq “ exp ´
2s2

▶ where the µj govern the locations of the basis functions in input space, and
the parameter s governs their spatial scale.
▶ These are ’Gaussian’ basis functions, although it should be noted that they
are not required to have a probabilistic interpretation, and in particular the
normalization coefficient is unimportant because these basis functions will be
multiplied by adaptive parameters wj .

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 8 / 11

CONTD..

Other possible choices for the basis functions, are:

# ` ˘2 +
x ´ µj
ϕj pxq “ exp ´
2s2

▶ where the µj govern the locations of the basis functions in input space, and
the parameter s governs their spatial scale.
▶ These are ’Gaussian’ basis functions, although it should be noted that they
are not required to have a probabilistic interpretation, and in particular the
normalization coefficient is unimportant because these basis functions will be
multiplied by adaptive parameters wj .

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 8 / 11

CONTD..

▶ Another one is the sigmoidal basis function of the form

x ´ µj
ˆ ˙
ϕj pxq “ σ
s
where σpaq is the logistic sigmoid function defined by

1
σpaq “
1 ` expp´aq

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 9 / 11

CONTD..

▶ Equivalently, we can use the ’tanh’ function because this is related to the
logistic sigmoid by tanhpaq “ 2σpaq ´ 1, and so a general linear combination of
logistic sigmoid functions is equivalent to a general linear combination of
’tanh’ functions. These various choices of basis function are illustrated below.
▶ Yet another possible choice of basis function is the Fourier basis, which
leads to an expansion in sinusoidal functions.

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 10 / 11

CONTD..

▶ Equivalently, we can use the ’tanh’ function because this is related to the
logistic sigmoid by tanhpaq “ 2σpaq ´ 1, and so a general linear combination of
logistic sigmoid functions is equivalent to a general linear combination of
’tanh’ functions. These various choices of basis function are illustrated below.
▶ Yet another possible choice of basis function is the Fourier basis, which
leads to an expansion in sinusoidal functions.

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 10 / 11

NORMAL

February 14, 2025 Nandagopal S A, Department of M.A.C.S, NIT-K Surathkal 11 / 11