CS229 Problem Set #1
1
CS 229, Public Course
Problem Set #1:
Supervised Learning
1.
Newton’s method for computing least squares
In this problem, we will prove that if we use Newton’s method solve the least squares
optimization problem, then we only need one iteration to converge to
θ
*
.
(a) Find the Hessian of the cost function
J
(
θ
) =
1
2
∑
m
i
=1
(
θ
T
x
(
i
)
−
y
(
i
)
)
2
.
(b) Show that the first iteration of Newton’s method gives us
θ
⋆
= (
X
T
X
)

1
X
T
vector
y
, the
solution to our least squares problem.
2.
Locallyweighted logistic regression
In this problem you will implement a locallyweighted version of logistic regression, where
we weight different training examples differently according to the query point. The locally
weighted logistic regression problem is to maximize
ℓ
(
θ
) =
−
λ
2
θ
T
θ
+
m
summationdisplay
i
=1
w
(
i
)
bracketleftBig
y
(
i
)
log
h
θ
(
x
(
i
)
) + (1
−
y
(
i
)
) log(1
−
h
θ
(
x
(
i
)
))
bracketrightBig
.
The
−
λ
2
θ
T
θ
here is what is known as a regularization parameter, which will be discussed
in a future lecture, but which we include here because it is needed for Newton’s method to
perform well on this task. For the entirety of this problem you can use the value
λ
= 0
.
0001.
Using this definition, the gradient of
ℓ
(
θ
) is given by
∇
θ
ℓ
(
θ
) =
X
T
z
−
λθ
where
z
∈
R
m
is defined by
z
i
=
w
(
i
)
(
y
(
i
)
−
h
θ
(
x
(
i
)
))
and the Hessian is given by
H
=
X
T
DX
−
λI
where
D
∈
R
m
×
m
is a diagonal matrix with
D
ii
=
−
w
(
i
)
h
θ
(
x
(
i
)
)(1
−
h
θ
(
x
(
i
)
))
For the sake of this problem you can just use the above formulas, but you should try to
derive these results for yourself as well.
Given a query point
x
, we choose compute the weights
w
(
i
)
= exp
parenleftbigg
−

x
−
x
(
i
)

2
2
τ
2
parenrightbigg
.
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
This is the end of the preview.
Sign up
to
access the rest of the document.
 '09
 Regression Analysis, Maximum likelihood, Linear least squares

Click to edit the document details