SolHomework3

SolHomework3 - STAT 315A Homework 3 Solutions Question 1(a...

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: STAT 315A Homework 3 Solutions Question 1 (a) Let V ⊥ be the p × ( N − p ) matrix that is the orthogonal complement to V in R p . Let ˆ θ solve Rθ = y . Then define β ( γ ) = V ˆ θ + V ⊥ γ for γ ∈ R N − p . Then for all γ , Xβ ( γ ) = RV T ( V ˆ θ + V ⊥ γ ) = R ˆ θ = y. Therefore, for each γ ∈ R N − p , β ( γ ) is a solution with zero residuals. Therefore, there are infinitely many solutions with zero residuals. (b) Let ˜ β be any solution with 0 residuals. First show that ˆ β has zero residuals. X ˆ β = XV D − 1 U T y = UDV T V D − 1 U T y = UDD − 1 U T y = y. In order to show that ˆ β has minimum Euclidean norm, it suffices to show that ˆ β is orthogonal to the space of all solutions with zero residuals, i.e. ˆ β T ( ˜ β − ˆ β ) = 0 for any ˜ β with X ˜ β = y. Here we have ˆ β T ( ˜ β − ˆ β ) = y T UD − 1 V T ( ˜ β − V D − 1 U T y ) = = y T UD − 2 U T UDV T ˜ β − y T UD − 2 U T y = = y T UD − 2 U T X ˜ β − y T UD − 2 U T y = y T UD − 2 U T y − y T UD − 2 U T y = 0 . Thus, bardbl ˜ β bardbl 2 = bardbl ˜ β − ˆ β + ˆ β bardbl 2 = bardbl ˜ β − ˆ β bardbl 2 + bardbl ˆ β bardbl 2 and therefore ˆ β is the unique zero residual solution with minimal norm. Question 2 (a) Let ˆ β ∈ R p be the direction of the vector onto which we project x . Then ˆ β T x/ bardbl ˆ β bardbl is the signed distance of the projected point to the origin. Therefore, any solution of Xβ = y projects onto exactly 2 points (by the construction of y ). In the previous exercise we have seen, that infinitely many solutions of this type exist. (b) As argued before, the signed distance of the projected points to the origin is ˆ β T x/ bardbl ˆ β bardbl . As ˆ β is a solution of Xβ = y with 0 residuals, by the definition of y the signed distance of the 2 points are − 1 / bardbl ˆ β bardbl and 1 / bardbl ˆ β bardbl and therefore the distance between these two points is 2 / bardbl ˆ β bardbl . (c) By using the result from (b) we see that this distance becomes largest for ˆ β with minimal Euclidean norm. In the previous exercise, this minimum norm solution was found to be ˆ β = V D − 1 U T y . (d) As we already know that the two classes are separable, we consider a linear support-vector machine without allowing for overlap. In order to find the optimal serparating hyperplane, the following convex optimization problem has to be solved: min β,β 1 2 bardbl β bardbl 2 subject to y i ( x T i β + β ) ≥ 1 , i = 1 , . . . , N The distance of the two classes is then 2 / bardbl β bardbl (this can easily be seen from the alternative formulation of the problem in equation 4.41, page 108 of ESL)....
View Full Document

{[ snackBarMessage ]}

Page1 / 11

SolHomework3 - STAT 315A Homework 3 Solutions Question 1(a...

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online