11 esl chapter 3 linear methods for regression trevor

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: lution satisfies normal equations: XT (y − Xβ ) = 0. If X full column rank, ˆ β= ˆ y = (X T X )− 1 X T y ˆ Xβ ˆ Also Var (β ) = (XT X)−1 σ 2 10 ESL Chapter 3 — Linear Methods for Regression Trevor Hastie and Rob Tibshirani Y • • • •• •• •• • • •• • • •• • • • • • • • • • •• •• • • • • • •• • • • •• •• • X2 • • • • • X1 The (p + 1)-dimensional geometry of least squares estimation. 11 ESL Chapter 3 — Linear Methods for Regression Trevor Hastie and Rob Tibshirani y x2 y ˆ x1 The N -dimensional geometry of least squares estimation ˆ (y − y) ⊥ xj , j = 1, . . . , p 12 ESL Chapter 3 — Linear Methods for Regression Trevor Hastie and Rob Tibshirani Properties of OLS • If X1 and X2 are mutually orthogonal matrices (XT X2 = 0), then 1 the joint regression coefficients for X = (X1 , X2 ) on y can be found from the separate regressions. ˆ ˆ ˆ Proof: XT (y − Xβ ) = XT (y − X1 β1 ) = 0. Same for β2 . 1 1 • OLS is equivariant under non-singular linear transformations of X. ˆ ˆ i.e. if β is OLS solution for X, then β ∗ = A−1 β is OLS solution for X∗ = XA for Ap×p nonsingular. Proof: OLS is defined by orthogonal projection onto column space of ˆ ˆ X. So y = Xβ = X∗ β ∗ . • Let X(p) be the submatrix of X excluding the last column xp . Let zp = xp − X(p) γ (for any γ ). Then OLS coefficient of xp is the same as OLS coefficent of zp if we replace xp by zp . Proof: previous point. 13 ESL Chapter 3 — Linear Methods for Regression Trevor Hastie and Rob Tibshirani • Let γ be the OLS coefficient of xp on X(p) . Hence zp is the residual obtained by adjusting xp for all the other variables in the model. • XTp) zp = 0 so the regression of y on (X(p) , zp ) decouples. ( • The multiple regression coefficient of xp is the same as the univariate coefficient in the regression of y on zp i.e. xp adjusted for the rest! ˆp = zp , y β ||zp ||2 • σ2 ˆ Var (βp ) = ||zp ||2 • Last statements true for all j , not just the last term p. 14 ESL Chapter 3 — Linear Methods for Regression Trevor Hastie and Rob Tibshirani The course website has some additional more technical notes (linearR.pdf) on multiple linear regression, with an emphasis on computations. 15 ESL Chapter 3 — Linear Methods for Regression Trevor Hastie and Rob Tibshirani Example: Prostate cancer • 97 observations on 9 variables (Stamey et al, 1989) • Goal to predict log(PSA) from 8 clinical measurements/ demographics on men who were about to have their prostate removed. • Next page shows a scatterplot matrix of all the date. This is created using the R expression pairs(lpsa ∼., data=lprostate). • Notice that several variables are correlated, and that svi is binary. 16 80 o 70 60 o o oo o o o oo o o o oo o o oo o o o o oo oo o oooo oo o oo oo oo o o oooo o o ooooooo o o o o o o oo o oo o o o o o ooo o o o oo o oo o oo o o o o o o o o oo o o o oo o o o oo o o o oo o o o o o o o o oo o o o oo o oo o o o oo o o o oo o o oo oooooooo oooo o o o o ooooo ooo o oo ooo oo o o o ooooo oo o 50 o oo oo oo o oo oo oo o oo ooo o o o o oo oooo oo o ooo o oooo o ooooooo o o o oo o o ooo o o o o o oo o o o o oo o oo o o o oo o o o o oo ooo ooooo oo oo o oo o 70 80 o o oo o o o oo o o oo o oooo o o ooo o oooooo o ooo o o o oo oo o oo o o o o oo o o o oo o...
View Full Document

Ask a homework question - tutors are online