problemset1 - CS229 Problem Set#1 1 CS 229 Public Course...

CS229 Problem Set #1 1
CS 229, Public Course
Problem Set #1: Supervised Learning
1. Newton’s method for computing least squares
In this problem, we will prove that if we use Newton’s method solve the least squares
optimization problem, then we only need one iteration to converge to θ
*
.
(a) Find the Hessian of the cost function J ( θ ) =
1
2
∑
m
i =1
( θ
T
x
( i )
− y
( i )
)
2
.
(b) Show that the first iteration of Newton’s method gives us θ
⋆
= ( X
T
X )
- 1
X
T
vector y , the
solution to our least squares problem.
2. Locally-weighted logistic regression
In this problem you will implement a locally-weighted version of logistic regression, where
we weight different training examples differently according to the query point. The locally-
weighted logistic regression problem is to maximize
ℓ ( θ ) = −
λ
2
θ
T
θ +
m summationdisplay
i =1
w
( i )
bracketleftBig
y
( i )
log h
θ
( x
( i )
) + (1 − y
( i )
) log(1 − h
θ
( x
( i )
))
bracketrightBig
.
The −
λ
2
θ
T
θ here is what is known as a regularization parameter, which will be discussed
in a future lecture, but which we include here because it is needed for Newton’s method to
perform well on this task. For the entirety of this problem you can use the value λ = 0 . 0001.
Using this definition, the gradient of ℓ ( θ ) is given by
∇
θ
ℓ ( θ ) = X
T
z − λθ
where z ∈ R m is defined by
z
i
= w
( i )
( y
( i )
− h
θ
( x
( i )
))
and the Hessian is given by
H = X
T
DX − λI
where D ∈ R
m × m
is a diagonal matrix with
D
ii
= − w
( i )
h
θ
( x
( i )
)(1 − h
θ
( x
( i )
))
For the sake of this problem you can just use the above formulas, but you should try to
derive these results for yourself as well.
Given a query point x , we choose compute the weights
w
( i )
= exp
parenleftbigg
−
|| x − x
( i )
||
2
2 τ 2
parenrightbigg
.