Rudyregularization-1

Rudyregularization-1 - Agenda Regularization: Ridge...

Info iconThis preview shows pages 1–9. Sign up to view the full content.

View Full Document Right Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Agenda Regularization: Ridge Regression and the LASSO Statistics 305: Autumn Quarter 2006/2007 Wednesday, November 29, 2006 Statistics 305: Autumn Quarter 2006/2007 Regularization: Ridge Regression and the LASSO Agenda Agenda 1 The Bias-Variance Tradeoff 2 Ridge Regression Solution to the 2 problem Data Augmentation Approach Bayesian Interpretation The SVD and Ridge Regression 3 Cross Validation K-Fold Cross Validation Generalized CV 4 The LASSO 5 Model Selection, Oracles, and the Dantzig Selector 6 References Statistics 305: Autumn Quarter 2006/2007 Regularization: Ridge Regression and the LASSO Part I: The Bias-Variance Tradeoff Part I The Bias-Variance Tradeoff Statistics 305: Autumn Quarter 2006/2007 Regularization: Ridge Regression and the LASSO Part I: The Bias-Variance Tradeoff Estimating As usual, we assume the model: y = f ( z ) + , (0 , 2 ) In regression analysis, our major goal is to come up with some good regression function f ( z ) = z So far, weve been dealing with ls , or the least squares solution: ls has well known properties (e.g., Gauss-Markov, ML) But can we do better? Statistics 305: Autumn Quarter 2006/2007 Regularization: Ridge Regression and the LASSO Part I: The Bias-Variance Tradeoff Choosing a good regression function Suppose we have an estimator f ( z ) = z To see if f ( z ) = z is a good candidate, we can ask ourselves two questions: 1.) Is close to the true ? 2.) Will f ( z ) fit future observations well? Statistics 305: Autumn Quarter 2006/2007 Regularization: Ridge Regression and the LASSO Part I: The Bias-Variance Tradeoff 1.) Is close to the true ? To answer this question, we might consider the mean squared error of our estimate : i.e., consider squared distance of to the true : MSE ( ) = E [ || || 2 ] = E [( ) ( )] Example: In least squares (LS), we now that: E [( ls ) ( ls )] = 2 tr[( Z Z ) 1 ] Statistics 305: Autumn Quarter 2006/2007 Regularization: Ridge Regression and the LASSO Part I: The Bias-Variance Tradeoff 2.) Will f ( z ) fit future observations well? Just because f ( z ) fits our data well, this doesnt mean that it will be a good fit to new data In fact, suppose that we take new measurements y i at the same z i s: ( z 1 , y 1 ) , ( z 2 , y 2 ) , . . . , ( z n , y n ) So if f ( ) is a good model, then f ( z i ) should also be close to the new target y i This is the notion of prediction error (PE) Statistics 305: Autumn Quarter 2006/2007 Regularization: Ridge Regression and the LASSO Part I: The Bias-Variance Tradeoff Prediction error and the bias-variance tradeoff So good estimators should, on average have, small prediction errors Lets consider the PE at a particular target point z (see the board for a derivation): PE( z ) = E Y | Z = z { ( Y f ( Z )) 2 | Z...
View Full Document

Page1 / 56

Rudyregularization-1 - Agenda Regularization: Ridge...

This preview shows document pages 1 - 9. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online