Chapter ESL 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
Model Selection
Topics
Bias variance trade-off Optimism of training error Estimates of in sample prediction error BIC VC dimension Cross-validation (chapter 3), bootstrap
1
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
Definitions
Loss functions ^ L(Y, f (X)) ^ L(G, G(X)) L(G, p(X)) ^ = = = = ^ (Y - f (X))2 squared error, ^ I(G = G(X)) 0-1 loss,
K
-2
k=1
I(G = k) log pk (X) ^ log-likelihood.
-2 log pG (X) ^
Training error (over training set T ): 1 err = N
N
^ L(yi , f (xi )).
i=1
Test Error (Generalization Error): ^ ^ ErrT = E[L(Y, f (X))|T ] (E[L(Y, G(X))|T ], E[L(Y, p(X))|T ]) ^ This is the expected loss over random realizations of new data test data.
2
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
Expected test error (expected prediction error, expected generalization error) ^ Err = E[L(Y, f (X))] = E[ErrT ] ^ ErrT is the error we can expect if we use the function f (x) trained on our particular training set T to make our predictions. Err averages in addition over all training sets. Since ErrT will depend on the particular nuances of our training set, we can make more general statements about Err than we can about ErrT .
3
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
1.2
High Bias Low Variance
Low Bias High Variance
Prediction Error
0.0 0
0.2
0.4
0.6
0.8
1.0
5
10
15
20
25
30
35
Model Complexity (df)
Behavior of test sample and training sample error as the model complexity is varied. The light blue curves show the training error err, while the light red curves show the conditional test error ErrT for 100 training sets of size 50 each, as the model complexity is increased. The solid curves show the expected test error Err and the expected training error E[err].
4
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
Bias-variance decomposition
Y = f (X) +
Err(x0 )
= = = =
^ E[(Y - f (x0 ))2 |X = x0 ] ^ ^ ^ 2 + [E f (x0 ) - f (x0 )]2 + E[f (x0 ) - E f (x0 )]2
2 ^ ^ + Bias2 (f (x0 )) + Var(f (x0 ))
Irreducible Error + Bias2 + Variance.
In the above, we need to decide whether xi in the sample are random, or assumed fixed. We assume fixed for what follows (recall homework 1).
5
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
For K-nearest neighbors:
Err(x0 )
= =
^ E[(Y - fk (x0 ))2 |X = x0 ] 1 2 + f (x0 ) - k
k
f (x( ) )
=1
2
2 + /k.
For linear regression: ^ fp (x0 ) = xT (XT X)-1 XT y = h(x0 )T y 0 Err(x0 ) = = ^ E[(Y - fp (x0 ))2 |X = x0 ] 2 2 ^ + [f (x0 ) - E fp (x0 )]2 + ||h(x0 )||2 . p 2 ^ [f (xi ) - E f (xi )]2 + , N i=1
6
N
1 N
1 2 Err(xi ) = + N i=1
N
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
Classification and 0-1 loss
Bias and variance and do not add as they do for squared error: variance tends to dominate, while bias is tolerable as long as you are on the correct side of the decision boundary. Hence biased methods often do well! Friedman (1996) "On Bias, Variance 0-1 loss..." shows ^(x0 ) - 1/2 sign(1/2 - f (x0 )) E f ^ P r(G(x0 ) = G(x0 )) ^ Var(f (x0 )) where G(x0 ) = I(f (x0 ) > 1 ) is the Bayes classifier (Exercise 7.2) 2 Hence on the wrong side of the decision boundary, increasing the variance can help
7
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
Bias-variance schematic The model space is the set of all possible predictions from the model, with the "closest fit" labeled with a black dot. The model bias from the truth is shown, along with the variance, indicated by the large yellow circle centered at the black dot labelled "closest fit in population". A shrunken or regularized fit is also shown, having additional estimation bias, but smaller prediction error due to its decreased variance.
Closest fit in population Realization Closest fit Truth
Model bias Estimation Bias
MODEL SPACE
Shrunken fit
Estimation Variance
RESTRICTED MODEL SPACE
8
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
Simulation Study (next slide)
There are 50 observations and 20 predictors, uniformly distributed in the hypercube [0, 1]20 . The situations are as follows: Left panels: Y is 1 if X1 1/2 and 0 otherwise, and we use k-nearest neighbors.
1 Right panels: Y is 1 if 10 j=1 Xj 1/2 and 0 otherwise, and we use best subset linear regression indexed by subset size p. 10
There are 100 realizations of this simulation, and the same fixed test set of size 10,000 is used for each.
9
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
k-NN - Regression
0.4 0.4
Linear Model - Regression
50
40
30
20
10
0
5
10 Subset Size p
15
20
Number of Neighbors k
k-NN - Classification
0.4 0.4
Linear Model - Classification
Prediction error, squared bias and variance for a simulated example. The top row is regression with squared error loss; the bottom row is classification with 01 loss. The models are k-nearest neighbors (left) and best subset regression of size p (right). The variance and bias curves are the same in regression and classification, but the prediction error curve is different.
0.3
0.2
0.1
0.0
0.3
0.2
0.1
0.0
50
40
30
20
10
0
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
5
10 Subset Size p
15
20
Number of Neighbors k
10
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
Optimism of the Training Error
training error 1 err = N In-sample error 1 Errin = N
N N
^ L(yi , f (xi ))
i=1
^ EY 0 [L(Yi0 , f (xi ))|T ]
i=1
where at each xi T , we observe a new Yi0 . Optimism op Errin - err. If we can estimate op, then we can estimate Errin = err + op.
11
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
For squared error, 01, and other loss functions, one can show quite generally that 2 = Ey (op) = N
N
Cov(^i , yi ), y
i=1
where Ey takes expectation wrt the yi T , but the xi T are held fixed. For linear model fitting (with d coefficients):
N 2 Cov(^i , yi ) = d y i=1
for the additive error model Y = f (X) + , and so Errin d 2 = err + 2 N
12
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
has the property that Ey Errin = Ey Errin Cp and AIC statistics: d 2 Cp = err + 2 . ^ N d 2 AIC = - loglik + 2 N N where loglik = i=1 log Pr (yi ) is the observed log-likelihood and ^ ^ is the MLE of the parameter vector .
N
13
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
Example: Cp for linear operators
Assume yi = f (xi ) + i ,
i
(0, 2 ).
Often the fitted vector ^ is linear in y: f ^ = Sy f e.g. linear regression, ridge regression, cubic smoothing splines.
N
N Ey Errin = N 2 +
i=1
[f (xi ) - {Sf }i ]2 + 2 tr(ST S)
N Ey (err) = = = E||(I - S)y||2 f T (I - S)T (I - S)f + E ( T (I - S)T (I - S) ) Bias2 + 2 tr(I - 2S + ST S)
14
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
Hence Ey Errin - Ey (err) = But this is = Ey op, and
2 2 tr(S) N
Cov(y, ^) = Cov(y, Sy) = 2 S f Based on the above, we define the effective degrees of freedom df =
N i=1
Cov(^i , yi ) y 2
For fits from linear operators like above df = tr(S).
15
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
Log-likelihood Loss
0.35 2.5 O Train Test AIC Misclassification Error O O
0-1 Loss
O
2.0
0.25
0.30
Log-likelihood
1.5
O O O
O O O O O O O O O O O O O O
O
O O 1.0 O O O O O O O O O O O 2 4 8 16 32 64 128 O
O O
O O O
0.5
0.10
0.15
0.20
O 2 4 8 16 32 64 128
Number of Basis Functions
Number of Basis Functions
Phoneme recognition example. The logistic regression coefficient function (f ) =
M m=1
hm (f )m is modeled in M spline basis functions.
Left panel: AIC statistic used to estimate Errin using log-likelihood loss. Included is an estimate of Err based on a test sample. It does well except for the overparametrized case (M = 256 parameters for N = 1000 observations). Right panel: Same is done for 01 loss. Although the AIC formula does not apply strictly here, it does a reasonable job in this case.
16
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
BIC-- Bayesian information criterion
BIC = -2 loglik + (log N ) d
2 under Guassian model is known, -2 loglik equals (up to a 2 2 ^ constant) i (yi - f (xi ))2 / , which is N err/ for squared error loss. Hence we can write N d 2 BIC = 2 err + (log N ) . N
Hence BIC is proportional to AIC, except 2 is replaced by log N . candidate models Mm , m = 1, . . . , M , with prior distribution Pr(m |Mm ) Given data Z, the posterior probability of a given model is Pr(Mm |Z) Pr(Mm ) Pr(Z|Mm ) Pr(Mm ) Pr(Z|m , Mm )Pr(m |Mm )dm ,
17
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
where Z represents the training data {xi , yi }1 . posterior odds Pr(Mm ) Pr(Z|Mm ) Pr(Mm |Z) = . Pr(M |Z) Pr(M ) Pr(Z|M ) The rightmost quantity Pr(Z|Mm ) BF(Z) = Pr(Z|M ) is called the Bayes factor
N
18
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
Typically we assume that the prior over models is uniform, so that Pr(Mm ) is constant. A Laplace approximation to the integral gives dm ^ log N + O(1). log Pr(Z|Mm ) = log Pr(Z|m , Mm ) - 2 ^ m is a maximum likelihood estimate and dm is the number of free parameters If we define our loss function to be ^ -2 log Pr(Z|m , Mm ), this is equivalent to the BIC criterion. Pr(Mm |Z) Pr(Mm )e- 2 BICm .
1
19
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
VC dimension (Vapnik-Chervonenkis)
Consider class of indicator functions F = {f (x, )} , indexed by a parameter . eg F1 = {I(0 + 1 x > 0)} , or F2 = {I(sin(x) > 0)} . VC dimension of a class F = {f (x, )} is defined to be the largest number of points (in some configuration) that can be shattered by members of F. A set of points is said to be shattered by a class of functions if, no matter how we we assign a binary label to each point, a member of the class can perfectly separate them. VC dim(F1 )=2, VC dim (F2 )=
20
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
The first three panels show that the class of lines in the plane can shatter three points. The last panel shows that this class cannot shatter four points, as no line will put the hollow points on one side and the solid points on the other. Hence the VC dimension of the class of straight lines in the plane is three. Note that a class of nonlinear curves could shatter four points, and hence has VC dimension greater than three.
21
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
sin(50 x)
-1.0 0.0
0.0
1.0
0.2
0.4
0.6
0.8
1.0
x
The solid curve is the function sin(50x) for x [0, 1]. The blue (hollow) and green (solid) points illustrate how the associated indicator function I(sin(x) > 0) can shatter (separate) an arbitrarily large number of points by choosing an appropriately high frequency .
22
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
VC bounds
Example -- binary classification, with class F = {f (x, )} with VC dimension h. With Pr > 1 - over training samples ErrT err + where 1+ = a1 1+ 4 err
2
h[log (a2 N/h) + 1] - log (/4) , N
where 0 < a1 4 and 0 < a2 2. These bounds are typically far too loose for reliable error estimation, but can nevertheless guide model selection (Structural Risk Minimization to Vapnik.)
23
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
AIC
100 % Increase Over Best 40 60 80
reg/KNN
reg/linear
class/KNN
class/linear
Boxplots show the distribution of the relative error 100 ErrT () - min ErrT () ^ max ErrT () - min ErrT ()
0
20
BIC
100 % Increase Over Best 60 80
reg/KNN
reg/linear
class/KNN
class/linear
SRM
100
% Increase Over Best
over the four scenarios. This is the error in using the chosen model relative to the best model. There are 100 training sets each of size 80 represented in each boxplot, with the errors computed on test sets of size 10,000.
0
20
40
60
80
0
20
40
reg/KNN
reg/linear
class/KNN
class/linear
24
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
Cross-validation
Simple, and best overall--see chapter 3
0.6 0.5
Misclassification Error
0.4
0.2
0.3
10 Subset Size p 15 20
0.0
0.1
5
Prediction error and ten-fold cross-validation curve estimated from a single training set, from the Linear Model - Classification scenario defined earlier.
25
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
Cross-validation: Bias due to reduced training set size
0.8 1-Err 0.0 0 0.2 0.4 0.6
50
100 150 Size of Training Set
200
Hypothetical learning curve for a classifier on a given task; a plot of 1 - Err versus the size of the training set N . With a dataset of 200 observations, fivefold cross-validation would use training sets of size 160, which would behave much like the full set. However, with a dataset of 50 observations fivefold cross-validation would use training sets of size 40, and this would result in a considerable overestimate of prediction error.
26
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
Bootstrap methods
We wish to assess the statistical accuracy of a quantity S(Z) computed from our dataset. B training sets Zb , b = 1, . . . , B each of size N are drawn with replacement from the original dataset. The quantity of interest S(Z) is computed from each bootstrap training set, and the values S(Z1 ), . . . , S(ZB ) are used to assess the statistical accuracy of S(Z).
27
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
Bootstrap replications S(Z1 ) S(Z2 ) S(ZB ) Bootstrap samples Z1 Z2 ZB
Z = (z1 , z2 , . . . , zN )
Training sample
28
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
bootstrap is useful for estimating the standard error of a statistic s(Z): we use the standard error of the bootstrap values s(Z1 ), s(Z2 ), . . . s(ZB ) E.g. s(Z) could be the prediction from a cubic spline curve at some fixed predictor value x. There is often more than one way to draw bootstrap samples--e.g. for a smoother, could draw samples from the data or draw samples from the residuals bootstrap is "non-parametric"-- i.e doesn't assume a parametric distribution for the data. If we carry the bootstrap out parametrically (i.e. draw from a normal distribution), then we get the usual textbook (Fisher information-based) formulas for standard errors as N . can get confidence intervals for an underlying population parameter from the percentiles for the bootstrap values s(Z1 ), . . . s(ZB ). Other more sophisticated confidence intervals via the bootstrap
29
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
Bootstrap estimation of prediction error
Errboot Pr{observation i bootstrap sample b} = = 1 1- 1- N 1 - e-1 0.632.
N
1 1 = BN
B
N
^ L(yi , f b (xi )).
b=1 i=1
Can be a poor estimate: Consider: 1-NN, 2 equal classes, class labels independent of features Then Errboot = 0.5(1 - 0.632) = 0.184, while Err = 0.5!
30
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
Leave-one out bootstrap: Err where C -i = {b : i bootstrap sample b}. / .632 bootstrap estimator: Err
(.632) (1)
1 = N
N
i=1
1 |C -i |
^ L(yi , f b (xi )).
bC -i
= .368 err + .632 Err
(1)
corrects for learning curve bias -- bootstrap samples represent typically 0.632 of training samples. .632+ bootstrap estimator: . . .
31
ESL Chapter 7 -- Model Selection
Trevor Hastie and Rob Tibshirani
Cross-validation
100 % Increase Over Best 0 20 40 60 80
reg/KNN
reg/linear
class/KNN
class/linear
Bootstrap
100 % Increase Over Best 0 20 40 60 80
reg/KNN
reg/linear
class/KNN
class/linear
Boxplots show the distribution of the relative error 100 [ErrT (^ ) - min ErrT ()]/[max ErrT () - min ErrT ()] over the four scenarios This is the error in using the chosen model relative to the best model. There are 20 training sets represented in each boxplot.
32