Programming for Data Science

Lecture 7 – Supervised Learning, Continued.

Thomas Lavastida
University of Texas at Dallas
[email protected]
Spring 2023
 Agenda

• Assignment 2 Review
• Quick review of Supervised Learning and Linear Regression
• Linear Regression in Python
• Start Regularization and Cross Validation

2
Assignment 2 Review
Supervised Learning and Regression Review
Supervised Learning

• Given – labelled data points

• – features, independent variables, predictors, columns, etc.
• – target, dependent variable, outcome, etc.
• Continuous -> then we call this regression
• Discrete/categorical -> then we call this classification

• Goal: Find a mapping/function from ’s to ’s such that

Linear Regression

• Simple class of regression models

• Let be independent variables
• Model parameters (one for each indep. variable)
• Predicted outcome computed via a linear function:

• Compute ’s by minimizing average squared error

Overfitting

• As model gets more complex it can fit data

more closely
• New data we see (and want to make
predictions about) may not be fit well (i.e.,
high error)
• This is called overfitting

• Main idea to deal with this -> split into train

and test set
• Training set – used to compute model
parameters
• Test set – used to estimate accuracy of model
on new data
PYTHON PRACTICE
Review: Overfitting

• Model with overfitting problem

• Nice performance for data in hand
• Poor predictive accuracy for new dataset

• Solution 1 – Splitting data

• Training set: train the model (get parameters)
• Test set: evaluate performance

• Solution 2 – Regularization
Regularization – Intuition

• Overfitting occurrence: Too many variables

• True relationship: 𝑦 = 𝛽0 + 𝛽1 𝑥 + 𝜀

• Fit the data w/ 10th degree polynomial

𝑦 = 𝛽0 + 𝛽1 𝑥 + 𝛽2 𝑥 2 +… + 𝛽 10 𝑥 10 + 𝜀

Fewer variables
Regularization – Intuition (Cont.)

• Overfitting occurrence: Large variance/fluctuation

• Large coefficient => large fluctuation

• Under the same scale

• Green: 4 3 2
𝑓 ( 𝑥 ) =− 𝑥 +7 𝑥 − 5 𝑥 − 31 𝑥 +30

• Blue: 1
𝑔 ( 𝑥 )=− 𝑓 ( 𝑥)
5

Smaller coefficients

https://www.datacamp.com/community/tutorials/towards-preventing-overfitting-regularization
Regularization – Intuition (Cont.)

• What we need
• Smaller coefficients (coefficient closer to 0)
• Fewer variables (coefficient = 0)

• Penalize the magnitude of coefficients

• Regularization
• Modify our original linear regression model
• Add terms to penalize the magnitude of coefficients
Regularization

• Linear regression (fit only)

• Minimize the error between actual and predicted value
𝑛
𝑓 (𝝎)=∑ ( 𝑦 𝑖 − ( 𝝎 𝑥 𝑖 +𝑏 ) )
2

𝑖=1

• Regularization (fit and overfit)

• Minimize the error between predicted and actual examples
• Penalize the coefficient magnitude of features

𝑛
𝑓 ( 𝝎 )=∑ ( 𝑦 𝑖 − ( 𝝎 𝑥 𝑖 +𝑏 ) ) + 𝑃𝑒𝑛𝑎𝑙𝑡𝑦(𝝎)
2

𝑖=1
Regularization – Two Methods

𝑛
𝑓 ( 𝝎 )=∑ ( 𝑦 𝑖 − ( 𝝎 𝑥 𝑖 +𝑏 ) ) + 𝑃𝑒𝑛𝑎𝑙𝑡𝑦(𝝎)
2

𝑖=1
Shrinkage Penalty

• Two formulation of shrinkage penalty

• L2 regularization: equivalent to the square of coefficient magnitude
=> Ridge regression

• L1 regularization: equivalent to absolute value of coefficient magnitude

=> Lasso regression
Ridge Regression

• Linear regression with L2 regularization (square of parameters)

• Minimize function:
𝑛 𝑘
𝑓 ( 𝝎 )=∑ ( 𝑦 𝑖 − ( 𝝎 𝑥 𝑖 +𝑏 ) ) + 𝜆 ∑ 𝜔
2 2 Shrinkage Penalty
𝑗
𝑖=1 𝑗=1
where

• Large magnitude increases

• the amount of penalty

Ridge Regression – Tuning Parameter

𝑛 𝑘
𝑓 ( 𝝎 )=∑ ( 𝑦 𝑖 − ( 𝝎 𝑥 𝑖 +𝑏 ) ) + 𝜆 ∑ 𝜔
2 2
𝑗
𝑖=1 𝑗=1

• the amount of penalty

• => A linear regression
• => All coefficients would be zero
• Higher , more penalty, smaller coefficients

• – hyperparameter
• NOT estimated with other parameters
• Set “manually” before model estimation
LASSO

• Linear regression with L1 regularization (absolute value of parameters)

𝑛 𝑘
𝑓 ( 𝝎 )=∑ ( 𝑦 𝑖 − ( 𝝎 𝑥 𝑖 +𝑏 ) ) + 𝜆 ∑ |𝜔 𝑗|
2

𝑖=1 𝑗=1
where .

• L1 penalty can force some coefficient estimates to be exactly zero

• Combines the shrinking advantage of ridge regression with variable selection

• LASSO: Least absolute shrinkage and selection operator

Hyperparameter Tuning and Cross
Validation
Hyperparameter Tuning

• Hyperparameters – set before running the model

• Examples
• LASSO and Ridge –
• Polynomial – degree of polynomial ()

• Intuition of tuning (polynomial case)

• Start by some potential values,
• For each , run the model
• Select the model with the best performance
Tuning Method – Grid Search

• Try all possible hyperparameters of interest

• Most commonly used method for hyperparameter tuning

• Polynomial regression case

• Define a set of potential polynomial degrees
• Estimate, evaluate, choose
Degree MSE Values

Lowest value, selected model

…

• Select the model with best performance … on which dataset?

Data Splitting – Model Training

• Model selection?
Labeled Data
• For each model, get performance
measure in test set
Training Set Test Set • Select model with best performance
in test data
Data Data
• Problem
Model Prediction and • “best model?”
Training Evaluation • “best fit for test set!”
Parameter
Estimates
• Overfitting test set

Performance measure (e.g., MSE) in test set • Solution: more splits

is unbiased (untouched new data)
Data Splitting – Model Selection

Original Training Set Test Set

Training Set Validation Set

• Validation set:
Data Data • Used for model selection (e.g.,
hyperparameter tuning)
Model Model
Training Selection • Test set:
Parameter • Untouched for training and selection
Estimates
• Used for model assessment
(generalizability)
Limitations of Single Splitting (Partition)

• Data waste: method applies to less data

• If not enough data – unreliable result

• Small training set
• Small test set

• Solution: Cross Validation

K-Fold Cross Validation

• Randomly cut dataset into segments

• Use the th segment as test set, the rest as training set
• Obtain , the mean squared error on the th segment (test set)
• After iterations, calculate mean of

𝑀𝑆 𝐸 1 𝑀𝑆 𝐸 2 𝑀𝑆 𝐸 5
1 1 1 1
2 2 2 2
3 3 3 … 3
4 4 4 4
5 5 5 5
K-Fold Cross Validation

• No data put to waste

• Small dataset
• Involves more data to train the model
• Reliable by taking the mean of multiple

• Model selection
• Using more data to evaluate performance of each model
CV for Model Selection

• Combine CV with grid search

• Example:
• Polynomial, grid search for degree, CV

• Leave a portion for test set

• Set grid for hyperparameter (let n be polynomial degree)
• Select model from CV

Degree MSE Values

Apply to test set

Lowest CV score
…
Grid Search with CV

• Manually set a grid of discrete hyperparameter values

• Set a metric for model performance

• Search exhaustively through the grid

• For each set of hyperparameters, evaluate each model’s CV score
• The optimal hyperparameters are those of the model achieving the best CV score
Tuning is expensive

• Run model repetitively

• N grid, K-fold CV => NK iterations
• Example: 20 grid, 5-fold CV

• Computationally expensive

• Sometimes very slight improvement

PYTHON PRACTICE