Machine & Deep Learning

Pr Rahhal ERRATTAHI
[email protected]
[email protected]

Lecture 03
Linear Models
 Lecture 3 Overview
• Linear models
• Linear models for regression
• Linear models for Classification
• Linear Models for multiclass classification
• Linear models overview
• Summary
 Lecture 3 Overview
• Linear models
• Linear models for regression
• Linear models for Classification
• Linear Models for multiclass classification
• Linear models overview
• Summary
ML: the basic recipe
• Abstract your problem to a standard task.
• Classification, Regression, Clustering, Density estimation, Generative
Modeling, Online learning, Reinforcement Learning,
• Choose your instances and their features.
• For supervised learning, choose a target.
• Choose your model class.
• Linear models
• Search for a good model.
• Choose a loss function, choose a search method to minimise the loss.
ML: problem setup
Problem Setting:
• Set of possible instances X
• Dataset D, given by 𝐷 = < 𝑥𝑖 , 𝑦𝑖 >, . . . , < 𝑥𝑛 , 𝑦𝑛 > ⊆ 𝑋 × 𝑌
Where:
• 𝑥𝑖 is a feature vector (ℝ𝑑 ),
• 𝑦𝑖 is a label / target variable,
• X is space of all features and
• Y is space of labels.
• Unknown target function 𝑓 ∶ 𝑋 → 𝑌
• Set of function hypotheses 𝐻 = {ℎ|ℎ ∶ 𝑋 → 𝑌 }
Linear models
• Linear models make a prediction using a linear function of the input
features 𝑋
𝑃

𝑓𝑤 𝑋 = 𝑤𝑖 ∙ 𝑥𝑖 +𝑤0
𝑖=1
• Learn 𝑤 from𝑋, given a loss function ℒ :
argmin ℒ(𝑓𝑤 𝑋 )
𝑤
Linear models
• Many algorithms with different ℒ : Least squares, Ridge, Lasso,
Logistic Regression, Linear SVMs,…
• Can be very powerful (and fast), especially for large datasets with
many features.
• Can be generalized to learn non-linear patterns: Generalized Linear
Models
• Features can be augmented with polynomials of the original features
• Features can be transformed according to a distribution (Poisson, Tweedie,
Gamma,…)
• Some linear models (e.g. SVMs) can be kernelized to learn non-linear
functions
 Lecture 3 Overview
• Linear models
• Linear models for regression
• Linear models for Classification
• Linear Models for multiclass classification
• Linear models overview
• Summary
Linear models for regression
• Prediction formula for input features x:
• 𝑤1 … 𝑤𝑝 usually called weights or coefficients ,𝑤0 the bias or intercept
• Assumes that errors are 𝑁(0, 𝜎)
𝑃

𝑦 = 𝑊𝑋 + 𝑤0 = 𝑤𝑖 ∙ 𝑥𝑖 +𝑏
𝑖=1
Ordinary Least Squares (OLS)
• Loss function is the sum of squared errors (SSE) (or residuals)
between predictions 𝑦𝑖 (red) and the true regression targets 𝑦𝑖 (blue)
on the training set.
𝑁 𝑁
2 2
ℒ𝑆𝑆𝐸 = 𝑦𝑛 − 𝑦𝑖 = 𝑦𝑛 − 𝑊 ∙ 𝑋𝑛 +𝑤0
𝑛=1 𝑛=1
Mean squared error (MSE)
• The MSE loss takes the residual for each instance in our data, squares
them, and returns the average.
𝑁 𝑁
1 2
1 2
ℒ𝑀𝑆𝐸 = 𝑦𝑛 − 𝑦𝑖 = 𝑦𝑛 − 𝑊 ∙ 𝑋𝑛 +𝑤0
𝑁 𝑁
𝑛=1 𝑛=1
Optimization
𝑝∗ = argmin ℒ(𝑓𝑝 𝑋 )
𝑝
In our example:
𝑝 = {𝑤1 … 𝑤𝑝 , 𝑤0 }
Optimization: Random Search
start with a random point p in the model space
loop:
pick a random point p’ close to p
if loss(p’) < loss(p):
p <- p’
Optimization: Random Search
• To implement the random search we need to define how to pick a
point “close to” another in model space.
• One simple option is to choose the next point by sampling uniformly
among all points with some pre-chosen distance r from the current
point.
Optimization: Random Search

Optimization algorithm may get “stuck” at

local minimum of a function

Minimum value of the function Local minimum value of the function

Optimization: Gradient Descent
• Q: Imagine you are blindfolded on a mountain, how will you go to the
bottom?

• A: Sense the slope around you, and move in the direction where
the slope points downwards
Optimization: Gradient Descent
Intuition: walking downhill using only the slope you “feel” nearby
Optimization: Gradient Descent
• Concept of gradient == “your sense of slope” for the loss function
• The slope of a linear function is simply how much it moves up if we
move one step to the right

• The gradient of a function is mathematically defined as the slope of

the tangent i.e. slope at any given point on the function
Optimization: Gradient Descent
Gradients:
Optimization: Gradient Descent
Tangent at x = 1.5
Slope = 3.0

Tangent at x = 0.8
Slope = 1.6
Optimization: Gradient Descent
Gradient Descent:
• Start with an initial, random set of weights 𝑊 0 :
• Given a differentiable loss function ℒ (e.g. ℒSSE ), compute 𝛻ℒ
𝜕ℒ𝑆𝑆𝐸 𝑁
• For least squares: 𝑊 = −2 𝑛=1 𝑦𝑛 − 𝑦𝑛 𝑥𝑛,𝑖
𝜕𝑤𝑖
• If feature 𝑋:,𝑖 is associated with big errors, the gradient wrt 𝑤𝑖 will be large
• Update all weights slightly (by step size or learning rate 𝜂) in
‘downhill’ direction.
• Basic update rule (step s):
𝑤 𝑆+1 = 𝑤 𝑆 − 𝜂𝛻ℒ(𝑤 𝑆 )
Optimization: Gradient Descent
Gradient Descent:
Optimization: Gradient Descent
• Learning rate:
Too small: slow convergence. Too large: possible divergence
Optimization: Gradient Descent
• Maximum number of iterations
• Too small: no convergence. Too large: wastes resources
• Learning rate decay with decay rate
𝜂𝑠
• E.g. exponential (𝜂 s+1 = 𝜂 0 𝑒 −𝑘𝑠 ), inverse-time (𝜂 s+1 = ),…
1+𝑘𝑠
• Many more advanced ways to control learning rate
• Adaptive techniques: depend on how much loss improved in previous step
Ridge regression
• Adds a penalty term to the least squares loss function:
𝑁 𝑃
2
ℒ𝑅𝑖𝑑𝑔𝑒 = 𝑦𝑛 − 𝑊 ∙ 𝑋𝑛 +𝑤0 +𝛼 𝑤𝑖2
𝑛=1 𝑖=1
• Model is penalized if it uses large coefficients (𝑊)
• Each feature should have as little effect on the outcome as possible
• We don’t want to penalize 𝑤0 , so we leave it out
• Regularization: explicitly restrict a model to avoid overfitting.
𝑃 2
• Called L2 regularization because it uses the L2 norm:𝛼 𝑤
𝑖=1 𝑖
• The strength of the regularization can be controlled with the
𝛼 hyperparameter
Lasso (Least Absolute Shrinkage and Selection
Operator)
• Adds a different penalty term to the least squares sum:
𝑁 𝑃
2
ℒ𝐿𝑎𝑠𝑠𝑜 = 𝑦𝑛 − 𝑊 ∙ 𝑋𝑛 +𝑤0 +𝛼 |𝑤𝑖 |
𝑛=1 𝑖=1
• Called L1 regularization because it uses the L1 norm
• Will cause many weights to be exactly 0
• Same parameter 𝛼 to control the strength of regularization.
• Will again have a ‘sweet spot’ depending on the data
Elastic-Net
• Adds both L1 and L2 regularization:
𝑁 𝑃 𝑃
2
ℒ𝐸𝑙𝑎𝑠𝑡𝑖𝑐 = 𝑦𝑛 − 𝑊 ∙ 𝑋𝑛 +𝑤0 + 𝛼𝜌 |𝑤𝑖 | + 𝛼(1 − 𝜌) 𝑤𝑖2
𝑛=1 𝑖=1 𝑖=1
• 𝜌 is the L1 ratio
• With 𝜌=1 , ℒ𝐸𝑙𝑎𝑠𝑡𝑖𝑐 = ℒ𝐿𝑎𝑠𝑠𝑜
• With 𝜌=0 , ℒ𝐸𝑙𝑎𝑠𝑡𝑖𝑐 = ℒ𝑅𝑖𝑑𝑔𝑒
• 0 < 𝜌 < 1 sets a trade-off between L1 and L2.
• Allows learning sparse models (like Lasso) while maintaining L2
regularization benefits
• E.g. if 2 features are correlated, Lasso likely picks one randomly, Elastic-Net
keeps both
Lab 4 - Linear Models for
Regression
 Lecture 3 Overview
• Linear models
• Linear models for regression
• Linear models for Classification
• Linear Models for multiclass classification
• Linear models overview
• Summary
Linear models for Classification
• Aims to find a hyperplane that separates the examples of each class.
• For binary classification (2 classes), we aim to fit the following
function:
𝑦 = 𝑤1 ∙ 𝑥1 +𝑤2 ∙ 𝑥2 + ⋯ + 𝑤𝑃 ∙ 𝑥𝑃 +𝑤0
When 𝑦 < 0, predict class -1, otherwise predict class +1
Linear models for Classification
• There are many algorithms for linear classification, differing in loss
function, regularization techniques, and optimization method
• Most common techniques:
• Convert target classes {neg,pos} to {0,1} and treat as a regression task
• Logistic regression (Log loss)
• Ridge Classification (Least Squares + L2 loss)
• Find hyperplane that maximizes the margin between classes
• Linear Support Vector Machines (Hinge loss)
• Neural networks without activation functions
• Perceptron (Perceptron loss)
• SGDClassifier: can act like any of these by choosing loss function
• Hinge, Log, Modified_huber, Squared_hinge, Perceptron
Logistic regression
• Aims to predict the probability that a point belongs to the positive
class
• Converts target values {negative (blue), positive (red)} to {0,1}
• Fits a logistic (or sigmoid or S curve) function through these points
• Maps (-Inf,Inf) to a probability [0,1]
1
𝑦 = 𝑙𝑜𝑔𝑖𝑠𝑡𝑖𝑐 𝑓𝜃 x =
1 + 𝑒 −𝑓𝜃 x
• E.g. in 1D:
1
𝑦 = 𝑙𝑜𝑔𝑖𝑠𝑡𝑖𝑐 𝑤1 ∙ 𝑥1 +𝑏 =
1 + 𝑒 −𝑤1 ∙𝑥1 −𝑤0
Logistic regression
• Models that return class probabilities can use cross-entropy loss

• Also known as log loss, logistic loss, or maximum likelihood

• Based on true probabilities p (0 or 1) and predicted
probabilities q over N instances and C classes
• Penalty (or surprise) grows exponentially as difference
between p and q increases
• Often used together with L2 (or L1) loss
Ridge Classification
• Instead of log loss, we can also use ridge loss:

• In this case, target values {negative, positive} are converted to {-1,1}

• Can be solved similarly to Ridge regression:
• Closed form solution (a.k.a. Cholesky)
• Gradient descent and variants
• E.g. Conjugate Gradient (CG) or Stochastic Average Gradient (SAG,SAGA)
• Use Cholesky for smaller datasets, Gradient descent for larger ones
Support vector machines
• Decision boundaries close to training points may generalize badly
• Very similar (nearby) test point are classified as the other class
• Choose a boundary that is as far away from training points as possible
• The support vectors are the training samples closest to the
hyperplane
• The margin is the distance between the separating hyperplane and
the support vectors
• Hence, our objective is to maximize the margin
Support vector machines
Perceptron
• Represents a single neuron (node) with inputs 𝑥𝑖 , a bias 𝑤0 , and
output 𝑦
• Each connection has a (synaptic) weight 𝑤𝑖 .
• The node outputs 𝑦 = 𝑛𝑖=1 𝑤𝑖 ∙ 𝑥𝑖 +𝑤0
• The activation function predicts 1 if xw + 𝑤0 > 0, -1 otherwise
• Weights can be learned with (stochastic) gradient descent and
Hinge(0) loss
• Updated only on misclassification, corrects output by ±1
Perceptron
 Lecture 3 Overview
• Linear models
• Linear models for regression
• Linear models for Classification
• Linear Models for multiclass classification
• Linear models overview
• Summary
Linear Models for multiclass classification
one-vs-rest (aka one-vs-all):
• Learn a binary model for each class vs. all other classes
• Create as many binary models as there are classes

Every binary classifiers makes a

prediction, the one with the highest
score (>0) wins
Linear Models for multiclass classification
one-vs-one:
• An alternative is to learn a binary model for every combination of two
classes
𝐶 𝐶−1
• For 𝐶 classes, this results in binary models
2
• Each point is classified according to a majority vote amongst all models
• Can also be a ‘soft vote’: sum up the probabilities (or decision values) for all models.
The class with the highest sum wins.
• Requires more models than one-vs-rest, but training each one is faster
• Only the examples of 2 classes are included in the training data
• Recommended for algorithms than learn well on small datasets
• Especially SVMs and Gaussian Processes
 Lecture 3 Overview
• Linear models
• Linear models for regression
• Linear models for Classification
• Linear Models for multiclass classification
• Linear models overview
• Summary
Linear models overview

•SSE: Sum of Squared Errors

•CFS: Closed-form solution
•SGD: (Stochastic) Gradient
Descent and variants
•(R)egression, (C)lassification

https://ml-course.github.io/
 Lecture 3 Overview
• Linear models
• Linear models for regression
• Linear models for Classification
• Linear Models for multiclass classification
• Linear models overview
• Summary
Summary
• Linear models
• Good for very large datasets (scalable)
• Good for very high-dimensional data (not for low-dimensional data)
• Regularization is important. Tune the regularization strength (∝)
• Ridge (L2): Good fit, sometimes sensitive to outliers
• Lasso (L1): Sparse models: fewer features, more interpretable, faster
• Elastic-Net: Trade-off between both, e.g. for correlated features
• Most can be solved by different optimizers (solvers)
• Closed form solutions or quadratic/linear solvers for smaller datasets
• Gradient descent variants (SGD,CD,SAG,CG,…) for larger ones
• Multi-class classification can be done using a one-vs-all approach
Lab 5 - Linear Models for
Classification