This preview shows pages 1–3. Sign up to view the full content.
This preview has intentionally blurred sections. Sign up to view the full version.View Full Document
Unformatted text preview: STAT 315A Homework 3 Solutions Question 1 (a) Let V ⊥ be the p × ( N − p ) matrix that is the orthogonal complement to V in R p . Let ˆ θ solve Rθ = y . Then define β ( γ ) = V ˆ θ + V ⊥ γ for γ ∈ R N − p . Then for all γ , Xβ ( γ ) = RV T ( V ˆ θ + V ⊥ γ ) = R ˆ θ = y. Therefore, for each γ ∈ R N − p , β ( γ ) is a solution with zero residuals. Therefore, there are infinitely many solutions with zero residuals. (b) Let ˜ β be any solution with 0 residuals. First show that ˆ β has zero residuals. X ˆ β = XV D − 1 U T y = UDV T V D − 1 U T y = UDD − 1 U T y = y. In order to show that ˆ β has minimum Euclidean norm, it suffices to show that ˆ β is orthogonal to the space of all solutions with zero residuals, i.e. ˆ β T ( ˜ β − ˆ β ) = 0 for any ˜ β with X ˜ β = y. Here we have ˆ β T ( ˜ β − ˆ β ) = y T UD − 1 V T ( ˜ β − V D − 1 U T y ) = = y T UD − 2 U T UDV T ˜ β − y T UD − 2 U T y = = y T UD − 2 U T X ˜ β − y T UD − 2 U T y = y T UD − 2 U T y − y T UD − 2 U T y = 0 . Thus, bardbl ˜ β bardbl 2 = bardbl ˜ β − ˆ β + ˆ β bardbl 2 = bardbl ˜ β − ˆ β bardbl 2 + bardbl ˆ β bardbl 2 and therefore ˆ β is the unique zero residual solution with minimal norm. Question 2 (a) Let ˆ β ∈ R p be the direction of the vector onto which we project x . Then ˆ β T x/ bardbl ˆ β bardbl is the signed distance of the projected point to the origin. Therefore, any solution of Xβ = y projects onto exactly 2 points (by the construction of y ). In the previous exercise we have seen, that infinitely many solutions of this type exist. (b) As argued before, the signed distance of the projected points to the origin is ˆ β T x/ bardbl ˆ β bardbl . As ˆ β is a solution of Xβ = y with 0 residuals, by the definition of y the signed distance of the 2 points are − 1 / bardbl ˆ β bardbl and 1 / bardbl ˆ β bardbl and therefore the distance between these two points is 2 / bardbl ˆ β bardbl . (c) By using the result from (b) we see that this distance becomes largest for ˆ β with minimal Euclidean norm. In the previous exercise, this minimum norm solution was found to be ˆ β = V D − 1 U T y . (d) As we already know that the two classes are separable, we consider a linear support-vector machine without allowing for overlap. In order to find the optimal serparating hyperplane, the following convex optimization problem has to be solved: min β,β 1 2 bardbl β bardbl 2 subject to y i ( x T i β + β ) ≥ 1 , i = 1 , . . . , N The distance of the two classes is then 2 / bardbl β bardbl (this can easily be seen from the alternative formulation of the problem in equation 4.41, page 108 of ESL)....
View Full Document
This note was uploaded on 02/29/2012 for the course STATS 315A taught by Professor Tibshirani,r during the Winter '10 term at Stanford.
- Winter '10