lecture9 - ISYE6414 Summer 2010 Lecture 9 Shrinkage Methods...

Info icon This preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon
ISYE6414 Summer 2010 Lecture 9 Shrinkage Methods Dr. Kobi Abayomi July 6, 2010 1 Orthogonalization The best scenario for the observed data x in multiple regression is each x j x k : the observed data are linearly independent. Remember a linear regression is a conditional expectation of the response variable Y given the observed data X = x — if the covariate predictors are completely linearly independent, they form an orthogonal basis for Y . Perfect Collinearity is the opposite of linear orthogonality: the predictors x form a degenerate (deficient rank) basis for Y . The regression coefficient estimates ˆ β are non-identifiable, in this stiuation, and their variance is inflated. The following methods are designed to mitigate the effects of collinearity (linear dependence) between the predictors by replacing them with components — linear combinations — that are generated to be linearly independent. 2 Principal Components Regression (PCR) Recall that x T x = ˆ Σ is the estimate of the covariate matrix of the predictors. Call λ = ( λ 1 , ..., λ k ) the eigenvalues of ˆ Σ, and e its eigenvectors . As such ˆ Σ = e Λ e T , with Λ = diag ( λ 1 , ..., λ k ). The jth principal component of x is z j = e T j x = e j 1 x 1 + · · · + e jk x k 1
Image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Constructing the principal components as the inner products of the eigenvectors and the predictors yields principal components z 1 , ..., z k with these properties V ar ( z j ) = λ i Cov ( z j , z k ) = 0 from the linear orthogonality of the the eigenvectors. This is the Principal Component Anal- ysis (PCA) procedure. Principal Component Regression (PCR) is replacing the predictors in the regression equation with their linearly orthogonal linear combinations: replace ˆ y = x T ˆ β with ˆ y = z T ˆ β * . The goal in the PCR program is to remove linear dependence and to express ˆ y via a low number of components. Here’s an example in R library(faraway) data(meatspec) ####data on fat content of 215 samples of meat ###with 100 channel spectrum of absorbances ###predict fat content from spectrum data ###variables 1-100 range of spectrum ###training sample is first 172 observations model1 <- lm(fat ~ ., meatspec[1:172,]) summary(model1)$r.squared ###use RMSE as a stat for gof for training and test sample rmse <- function(x,y) sqrt(mean((x-y)^2)) rmse(model1$fit,meatspec$fat[1:172])
Image of page 2
Image of page 3
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern