Scribd
Upload
792 views

Types of Regression

(1) The document describes different types of regression models including linear, lasso, ridge, elastic net, robust, and k-neighbors regression. (2) It provides details on elastic net regression, which combines lasso and ridge regression, and is useful when predictors are correlated. (3) An example using mice genetic data demonstrates that elastic net and ridge regression had lower errors than other models when the number of predictors was much larger than the number of observations.

Uploaded by

akhare16
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
792 views

Types of Regression

(1) The document describes different types of regression models including linear, lasso, ridge, elastic net, robust, and k-neighbors regression. (2) It provides details on elastic net regression, which combines lasso and ridge regression, and is useful when predictors are correlated. (3) An example using mice genetic data demonstrates that elastic net and ridge regression had lower errors than other models when the number of predictors was much larger than the number of observations.

Uploaded by

akhare16
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 21

Different types of regression: Linear, Lasso,

Ridge, Elastic net, Robust and K-neighbors


Agnieszka Prochenka
Faculty of Mathematics, Informatics and Mechanics, University of Warsaw

04.10.2009

Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

Introduction

We are given a linear problem:


y = X +
E() = 0
Var () = 2 In
where:
y is the vector of observations of length n,
X the design matrix n (p + 1), where the first column is a
ones vector,
X = (x1 , x2 , . . . , xn )T
the coefficients vector of length (p + 1),
= (0 , 1 . . . , p )T ,
the random error of dimension n.

Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

Linear Regression

If we want to find an estimator as a function of X and y which


minimizes the sum of the squared errors:
n
X
i=1

i 2 =

n
X
2
(yi xi0 )
i=1

it turns out that has to be of the form:


Estimators:
= (XT X)1 XT Y
y = X

Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

Linear Regression

R function
lm(formula, data, subset, weights,. . . )

Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

Linear Regression

R function
lm(formula, data, subset, weights,. . . )
However, in some conditions the linear regression doesnt work
well:
1
2

p>n: the matrix (XT X) is invertible


the rows of the design matrix X are highly correlated: the
coefficients are dependent on different xi

The next 4 models bring solutions to these problems.

Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

Elastic net regression


Regression elastic net solves the following problem:
"
= argbRp+1 min

1 X
(yi xiT b)2 + P (b1 , . . . , bp )
2n
n

i=1

where
P =

p 
X
1
i=1

(1

)bj2

+ |bj |

For different we can have the following types of regression:


= 1 lasso
= 0 ridge
 (0, 1) the general case of the elastic net
Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

Elastic net regression


Ridge regression is known to shrink the coefficients of
correlated predictors towards each other. Lasso is somewhat
indifferent to very correlated predictors and will tend to pick one
and ignore the rest.

Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

Elastic net regression

R function
W pakiecie glmnet:
glmnet(x, y, weights, alpha,nlambda, lambda.min , lambda,. . . )
glmnet$a0 ; glmnet$beta

Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

Elastic net regression

R function
W pakiecie glmnet:
glmnet(x, y, weights, alpha,nlambda, lambda.min , lambda,. . . )
glmnet$a0 ; glmnet$beta
literature: Friedman, Hastie, Tibshirani, Regularization Paths
for Generalized Linear Models via Coordinate Descent,
Stanford University, May 2008

Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

K-neighbors model

This method assumes calculating the y without estimating .


For each observation x , minimizing the euclidean distance, we
find k closest observations from the design matrix X with
indexes j1 , . . . , jk . y is the arithmetic mean of yj1 , . . . , yjk .
A = {1, . . . , n}
for (s in 1 : k ) {
ls = argjA min ||xj x||
A = A\{ls }
}
y = mean(yl1 , . . . , ylk )

Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

Robust regression
M estimator
M estimator minimizes a function:
n
X

(ei ) =

i=1

n
X

(yi xi0 )

i=1

where is the loss function.

Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

Robust regression
M estimator
M estimator minimizes a function:
n
X

(ei ) =

i=1

n
X

(yi xi0 )

i=1

where is the loss function.


Some examples of the loss functions:
Least-Squares
LS (e) = e2
Huber


H (e) =

for |e| 6 k ;
k |e| 21 e2 , for |e| > k .
1 2
2e ,

Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

Robust regression
Bisquare
B (e) =


3
1 1 ( ke )2
, for |e| 6 k ;

k2
6
k2
6 ,

for |e| > k .

A graph of the loss functions with k = 1.345 for Huber,


k = 4.685 for Bisquare:

Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

Robust regression

R function
W pakiecie RLMM:
rlm(x, y, weights, psi = psi.huber,. . . )

Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

Mice example p  n
In this example the X matrix has dimensions n=54, p=1000 and
comes from real genetic data of mice, y was simulated.

Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

These are the sorted values of y y for each model:

Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

Mice example p  n

Measuring theP
goodness of fit with the mean of the squared
2 , we get:
errors RSS = ni=1 (yt i xi0 )
model
lasso
elastic net
ridge
k neighbors

RSS
3.21
2.45
2.37
165.49

(The mean value of yt is 415)

Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

My masters thesis

In my masters thesis I am going to, looking at the data


structure, try to find the best model for predictions in this
situation.
1

I will numerously draw data

Choose different parameters

Try my models and look for some regularity

This is an example:

Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

I repeat N times building the real data list:


1

Drawing the matrix X from the p-dimensional normal


distribution with the correlation matrix

1 2 ...
1 ...

2 1 ...
... ... ... ...

Drawing i from a normal distribution building the vector

Choosing

Calculating the real value of y , yi = xi0 + i

For the given data list I estimate the parameters for different
models and check which fit best (for example which minimizes
the sum of the squared loss). This is an example of what can
come out:
Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

uniformly decreases from 1 to 0; = 0.1; = 0.5; n = 200;


N = 250

Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

= (1, 0, 1, 0, 1, . . .); = 0.9; = 0.5; n = 200; N = 250

Agnieszka Prochenka

Different types of regression: Linear, Lasso, Ridge, Elastic net, Ro

You might also like