Model Assessment and

Selection

1
 Goal

• Model Selection

• Model Assessment

2
A Regression Problem
• y = f(x) + noise

• Can we learn f from

this data?

• Let’s consider three

methods...

3
Linear Regression

4
Quadratic Regression

5
Joining the dots

6
 Which is best?

• Why not choose the method with the best fit to

the data?
“How well are you going to
predict future data drawn
from the same distribution?”
7
 Model Selection and Assessment

• Model Selection: Estimating performances of

different models to choose the best one
(produces the minimum of the test error)

• Model Assessment: Having chosen a model,

estimating the prediction error on new data

8
 Why Errors

• Why do we want to study errors?

• In a data-rich situation split the data:

Train Validation Test

Model Selection Model assessment

• But, that’s not usually the case

9
 Overall Motivation
• Errors
 Measurement of errors (Loss functions)
 Decomposing Test Error into Bias & Variance

• Estimating the true error

 Estimating in-sample error (analytically )
AIC, BIC, MDL, SRM with VC
 Estimating extra-sample error (efficient sample reuse)
Cross Validation & Bootstrapping

10
 Measuring Errors:
Loss Functions
• Typical regression loss functions
 Squared error:

 Absolute error:

11
 Measuring Errors:
Loss Functions
• Typical classification loss functions
 0-1 Loss:

 Log-likelihood (cross-entropy loss / deviance):

12
 The Goal: Low Test Error
• We want to minimize generalization error or test
error:
Err  E[L(Y, fˆ (X))]
• But all we really know is training error:
N
?
1
err   L(y i , fˆ (x i ))
 N i1

• And this is a bad estimate of test error

13

Bias, Variance & Complexity

Training error can always be reduced when increasing

model complexity, but risks over-fitting.
Typically 14
 Decomposing Test Error
Model: Y  f ( X )   ; E ( )  0 and Var ( )   
2

For Deviation
squared-error loss & estimate
of the average additive noise:
from the true function’s mean

Irreducible error Expected squared deviation of our

of target Y estimate around its mean 15
 Further Bias Decomposition
• For linear models
For(eg. Ridge),
standard bias
linear can
regression,
be further decomposed:
Estimation Bias = 0

Average Model Average Estimation

* is the best fitting linear approximation
  arg min E ( f ( X )   X )

Bias T 2 Bias

16
 Graphical representation of
bias & variance
Model Space
Hypothesis Space (basic linear regression)
Closest fit
(given our observation)
Realization
Mo Shrunken fit
de
lF
Truth itti
ng
Regularized Model Space
(ridge regression)
Model Bias Estimation
Estimation Variance
Closest fit
Bias
In population
(if epsilon=0) 17
 Bias & Variance
Decomposition Examples
• kNN Regression
Averaging over the training set:

1 1 p 2

N i
Err(x i )    [ f (x i )  Ef (x i )]  
2

N i
ˆ 2

• Linear Regression


Linear weights on y:

18
 Simulated Example of
Bias Variance Decomposition
Prediction error
-- + -- = -- -- + -- = -- Bias2
Regression
Variance
with squared
error loss

Bias-Variance
-- + -- <> -- -- + -- <> -- different for
Classification
with 0-1 loss 0-1 loss
than for
Estimation errors
squared error
on the right side
loss
of the boundary
don’t hurt! 19
 Optimism of The
Training Error Rate
• Typically: training error rate < true error
(same data is being used to fit the method
and assess its error)

N
1
err   L(y i , fˆ (x i )) < Err  E[L(Y, fˆ (X))]
N i1
overly optimistic

 20
 Estimating Test Error

• Can we estimate the discrepancy between

err and Err? extra-sample error
Expectation over N new
Errin --- In-sample error:responses at each xi

21
Adjustment for optimism of training error
 Optimism
N
2
Summary: Errin  E y  err  
N
 Cov  yˆ , y 
i 1
i i

for squared error, 0-1 and other lossN functions:

2
optimism: op  Errin  E y  err  op   Cov  yˆ , y 
i i
N i 1

• For linear fit with d indep inputs/basis funcs:

2
Errin  E y  err   d 2
N
 optimism linearly with # d
 Optimism as training sample size 22
 Ways to Estimate Prediction Error

• In-sample error estimates:

 AIC
 BIC
 MDL
 SRM

• Extra-sample error estimates:

 Cross-Validation
• Leave-one-out
• K-fold
 Bootstrap

23
 Estimates of In-Sample
Prediction Error
• General form of the in-sample estimate:
Êrrin  err  oˆp

• For linear fit :

2d 2
C p  err  ˆ  , so called C p statistic
N

24
 AIC & BIC

Similarly: Akaike Information Criterion (AIC)

2 d
AIC    loglik 2 
N N

Bayesian Information Criterion (BIC)


BIC  2 loglik  (log N )d

25
 AIC & BIC

AIC  LL(Data | MLE params)  (# of parameters)

log N
BIC  LL(Data | MLE params)  (# of parameters)
2

26
 MDL
(Minimum Description Length)
• Regularity ~ Compressibility
• Learning ~ Finding regularities
Input
Samples Learning model Predictions
Rn R1

Real class
R1
Real model =?

error 27
 MDL
(Minimum Description Length)
• Regularity ~ Compressibility
• Learning ~ Finding regularities
length  logPr(y | , M, X)  logPr( | M)

Length of transmitting the discrepancy Description of the model

given the model + optimal coding under optimal coding
 under the given model

MDL principle: choose the model with the minimum description length

Equivalent to maximizing the posterior: Pr(y | , M, X)  Pr( | M)

28
 SRM with VC (Vapnik-
Chernovenkis) Dimension
• Vapnik showed that with probability 1-

 4  Err 
Errtrue  Errtrain   1 1 train

2   
h log(a2 N /h)  1  log( /4)
where   a1
N
 h  VC dimension (measure of f ' s power)
As h increases

A method of selecting a class F from a family of nested classes

29
 Errin Estimation

• A trade-off between the fit to the data and

the model complexity
d
AIC  err  2   
ˆe
N
BIC  2 loglik  (log N )d

MDL length  logPr(y | , M, X)  logPr( | M)

 4  Err 
VC : Errtrue  Errtrain   2 1 1 train

  
30
 Estimation of
Extra-Sample Err

• Cross Validation

• Bootstrap

31
 Cross-Validation

test train

K-fold
……

N
1
 
CV ( )   L y i , fˆ  (i) (x i , )
N i1
32
 How many folds?

Computation increases

Variance decreases bias decreases

k fold
Leave-one-out
k increases

33
Cross-Validation: Choosing K

Popular choices for K: 5,10,N 34

 Generalized Cross-Validation

• LOOCV can be computational expensive for linear fitting with large N

• Linear fitting yˆ Sy (S is a smoother matrix)

• For linear fitting under squared-error loss:

2
1 yi  fˆ (xi) 
N 2 N


1

N i1
 ˆ i

yi  f (xi)   
N i1  1 Sii 
Sii  i'th diagonal element of S

  trace(S) N
 a computationally cheaper approximation
• GCV provides
2
1 
N ˆ
yi  f (xi) 
 GCV   
N i1  1   35
Bootstrap: Main Concept
“The bootstrap is a computer-based
method of statistical inference that can answer
many real statistical questions without formulas”
(An Introduction to the Bootstrap, Efron and Tibshirani, 1993)

Step 2: Calculate the

statistic

Step 1: Draw samples

with replacement

36
 How is it coming

Sampling distribution of sample mean x

In practice cannot afford
large number of random samples



The theory tells us

the sampling distribution

The sample stands for the population

and the distribution of x in many
resamples stands for the sampling
distribution 37
 Bootstrap:
Error Estimation with Errboot
 
B N
1 1
 S   S(Z *b )
* *
Vaˆ r[S(Z)]  S(Z *b )  S
B 1 b1 B b1
VarFˆ [S(Z)] Depends on the unknown true
distribution F
 
A straightforward application of bootstrap to error prediction

B N
1 1
Erˆrboot   
B N b1 i1
L(y i , ˆ
f *b
(x i ))

 38
 Bootstrap:
Error Estimation with Err(1)
A CV-inspired improvement on Errboot
N
1 1
Erˆr   i
(1)
 i (x i ))
L(y , ˆ
f *b

N i1 C b C  i



39
 Bootstrap:
Error Estimation with Err(.632)
An improvement on Err(1) in light-fitting cases

Erˆr(.632)  .368  err  .632  Erˆr(1)

 N # of datapoints Z  (z1,...,z n ) ?

 1 
 Probability of zi NOT being chosen when 1 point is uniformly sampled from Z : 1 - 
  N 

 1 N
 Probability of z i NOT being chosen when Z is sampled N times :1 - 
 N 

 1 N
 Probability of zi being chosen AT LEAST once when Z is sampled N times: 1 1 - 
 N 
 1 e1 Erˆr(.632)  err  .632 (Erˆr(1)  err)
40
 0.632  .368 err  .632 Erˆr(1)
 Bootstrap:
Error Estimation with Err(.632+)
An improvement on Err(.632) by adaptively
accounting for overfitting

• Depending on the amount of overfitting, the best error

estimate is as little as Err(.632) , or as much as Err(1), or
something in between

• Err(.632+) is like Err(.632) with adaptive weights, with Err(1)

weighted at least .632

• Err(.632+) adaptively mixes training error and leave-one-

out error using the relative overfitting rate (R)
41
 Bootstrap:
Error Estimation with Err(.632+)

Erˆr(.632) ranges from Erˆr(.632) if there is minimal overfitting (R 0),

to Erˆr(1) if there is maximal overfitting (R 1) 42
 Cross Validation & Bootstrap
• Why bother with cross-validation and bootstrap
when analytical estimates are known?

1) AIC, BIC, MDL, SRM all requires knowledge of d,

which is difficult to attain in most situations.

2) Bootstrap and cross validation gives similar results

to above but also applicable in more complex situation.

3) Estimating the noise variance requires a roughly

working model, cross validation and bootstrap will work
well even if the model is far from correct.
43
 Conclusion
• Test error plays crucial roles in model selection
• AIC, BIC and SRMVC have the advantage that you only need the
training error
• If VC-dimension is known, then SRM is a good method for model
selection – requires much less computation than CV and
bootstrap, but is wildly conservative
• Methods like CV, Bootstrap give tighter error bounds, but might
have more variance
• Asymptotically AIC and Leave-one-out CV should be the same
• Asymptotically BIC and a carefully chosen k-fold should be the
same
• BIC is what you want if you want the best structure instead of the
best predictor
• Bootstrap has much wider applicability than just estimating
prediction error 44