# 11 esl chapter 3 linear methods for regression trevor

This preview shows page 1. Sign up to view the full content.

This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: lution satisﬁes normal equations: XT (y − Xβ ) = 0. If X full column rank, ˆ β= ˆ y = (X T X )− 1 X T y ˆ Xβ ˆ Also Var (β ) = (XT X)−1 σ 2 10 ESL Chapter 3 — Linear Methods for Regression Trevor Hastie and Rob Tibshirani Y • • • •• •• •• • • •• • • •• • • • • • • • • • •• •• • • • • • •• • • • •• •• • X2 • • • • • X1 The (p + 1)-dimensional geometry of least squares estimation. 11 ESL Chapter 3 — Linear Methods for Regression Trevor Hastie and Rob Tibshirani y x2 y ˆ x1 The N -dimensional geometry of least squares estimation ˆ (y − y) ⊥ xj , j = 1, . . . , p 12 ESL Chapter 3 — Linear Methods for Regression Trevor Hastie and Rob Tibshirani Properties of OLS • If X1 and X2 are mutually orthogonal matrices (XT X2 = 0), then 1 the joint regression coefﬁcients for X = (X1 , X2 ) on y can be found from the separate regressions. ˆ ˆ ˆ Proof: XT (y − Xβ ) = XT (y − X1 β1 ) = 0. Same for β2 . 1 1 • OLS is equivariant under non-singular linear transformations of X. ˆ ˆ i.e. if β is OLS solution for X, then β ∗ = A−1 β is OLS solution for X∗ = XA for Ap×p nonsingular. Proof: OLS is deﬁned by orthogonal projection onto column space of ˆ ˆ X. So y = Xβ = X∗ β ∗ . • Let X(p) be the submatrix of X excluding the last column xp . Let zp = xp − X(p) γ (for any γ ). Then OLS coefﬁcient of xp is the same as OLS coefﬁcent of zp if we replace xp by zp . Proof: previous point. 13 ESL Chapter 3 — Linear Methods for Regression Trevor Hastie and Rob Tibshirani • Let γ be the OLS coefﬁcient of xp on X(p) . Hence zp is the residual obtained by adjusting xp for all the other variables in the model. • XTp) zp = 0 so the regression of y on (X(p) , zp ) decouples. ( • The multiple regression coefﬁcient of xp is the same as the univariate coefﬁcient in the regression of y on zp i.e. xp adjusted for the rest! ˆp = zp , y β ||zp ||2 • σ2 ˆ Var (βp ) = ||zp ||2 • Last statements true for all j , not just the last term p. 14 ESL Chapter 3 — Linear Methods for Regression Trevor Hastie and Rob Tibshirani The course website has some additional more technical notes (linearR.pdf) on multiple linear regression, with an emphasis on computations. 15 ESL Chapter 3 — Linear Methods for Regression Trevor Hastie and Rob Tibshirani Example: Prostate cancer • 97 observations on 9 variables (Stamey et al, 1989) • Goal to predict log(PSA) from 8 clinical measurements/ demographics on men who were about to have their prostate removed. • Next page shows a scatterplot matrix of all the date. This is created using the R expression pairs(lpsa ∼., data=lprostate). • Notice that several variables are correlated, and that svi is binary. 16 80 o 70 60 o o oo o o o oo o o o oo o o oo o o o o oo oo o oooo oo o oo oo oo o o oooo o o ooooooo o o o o o o oo o oo o o o o o ooo o o o oo o oo o oo o o o o o o o o oo o o o oo o o o oo o o o oo o o o o o o o o oo o o o oo o oo o o o oo o o o oo o o oo oooooooo oooo o o o o ooooo ooo o oo ooo oo o o o ooooo oo o 50 o oo oo oo o oo oo oo o oo ooo o o o o oo oooo oo o ooo o oooo o ooooooo o o o oo o o ooo o o o o o oo o o o o oo o oo o o o oo o o o o oo ooo ooooo oo oo o oo o 70 80 o o oo o o o oo o o oo o oooo o o ooo o oooooo o ooo o o o oo oo o oo o o o o oo o o o oo o...
View Full Document

{[ snackBarMessage ]}

Ask a homework question - tutors are online