This preview shows pages 1–3. Sign up to view the full content.
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: STAT 315A Homework 3 Solutions Question 1 (a) Let V ⊥ be the p × ( N − p ) matrix that is the orthogonal complement to V in R p . Let ˆ θ solve Rθ = y . Then define β ( γ ) = V ˆ θ + V ⊥ γ for γ ∈ R N − p . Then for all γ , Xβ ( γ ) = RV T ( V ˆ θ + V ⊥ γ ) = R ˆ θ = y. Therefore, for each γ ∈ R N − p , β ( γ ) is a solution with zero residuals. Therefore, there are infinitely many solutions with zero residuals. (b) Let ˜ β be any solution with 0 residuals. First show that ˆ β has zero residuals. X ˆ β = XV D − 1 U T y = UDV T V D − 1 U T y = UDD − 1 U T y = y. In order to show that ˆ β has minimum Euclidean norm, it suffices to show that ˆ β is orthogonal to the space of all solutions with zero residuals, i.e. ˆ β T ( ˜ β − ˆ β ) = 0 for any ˜ β with X ˜ β = y. Here we have ˆ β T ( ˜ β − ˆ β ) = y T UD − 1 V T ( ˜ β − V D − 1 U T y ) = = y T UD − 2 U T UDV T ˜ β − y T UD − 2 U T y = = y T UD − 2 U T X ˜ β − y T UD − 2 U T y = y T UD − 2 U T y − y T UD − 2 U T y = 0 . Thus, bardbl ˜ β bardbl 2 = bardbl ˜ β − ˆ β + ˆ β bardbl 2 = bardbl ˜ β − ˆ β bardbl 2 + bardbl ˆ β bardbl 2 and therefore ˆ β is the unique zero residual solution with minimal norm. Question 2 (a) Let ˆ β ∈ R p be the direction of the vector onto which we project x . Then ˆ β T x/ bardbl ˆ β bardbl is the signed distance of the projected point to the origin. Therefore, any solution of Xβ = y projects onto exactly 2 points (by the construction of y ). In the previous exercise we have seen, that infinitely many solutions of this type exist. (b) As argued before, the signed distance of the projected points to the origin is ˆ β T x/ bardbl ˆ β bardbl . As ˆ β is a solution of Xβ = y with 0 residuals, by the definition of y the signed distance of the 2 points are − 1 / bardbl ˆ β bardbl and 1 / bardbl ˆ β bardbl and therefore the distance between these two points is 2 / bardbl ˆ β bardbl . (c) By using the result from (b) we see that this distance becomes largest for ˆ β with minimal Euclidean norm. In the previous exercise, this minimum norm solution was found to be ˆ β = V D − 1 U T y . (d) As we already know that the two classes are separable, we consider a linear supportvector machine without allowing for overlap. In order to find the optimal serparating hyperplane, the following convex optimization problem has to be solved: min β,β 1 2 bardbl β bardbl 2 subject to y i ( x T i β + β ) ≥ 1 , i = 1 , . . . , N The distance of the two classes is then 2 / bardbl β bardbl (this can easily be seen from the alternative formulation of the problem in equation 4.41, page 108 of ESL)....
View
Full
Document
This note was uploaded on 02/29/2012 for the course STATS 315A taught by Professor Tibshirani,r during the Winter '10 term at Stanford.
 Winter '10
 TIBSHIRANI,R
 Statistics

Click to edit the document details