Instructor Angela Yu
Department of Cognitive Science
UC San Diego
Due: 01/29/09
COGS 118A: ASSIGNMENT 1
Problem
1. MLE for probabilistic linear regression
Given a linear basis function model for regression, ¯
y
(
x
) =
w
0
+
∑
m
j
=1
w
j
φ
j
(
x
), and additive Gaussian
noise,
y
∼ N
(¯
y, σ
2
), show that the maximum likelihood estimate (MLE) for
w
0
is:
ˆ
w
*
0
=
1
n
n
X
i
=1
y
i

m
X
j
=1
w
j
¯
φ
j
(11)
where
¯
φ
j
=
1
n
∑
n
i
=1
φ
j
(
x
). That is,
w
0
compensates for the diﬀerence between the sample mean of
y
and the weighted sample mean of the basis functions (each evaluated at all the data points).
Problem
2. Linear basis function models of regression
(a) Minimizing least squares error in regression leads to a nice analytical solution (Bishop Eq. 3.15).
However, inverting a matrix can be computationally intense. Use
tic
and
toc
in Matlab to measure the
amount of time it takes to invert a random matrix of size k x k generated with
rand(k)
, for diﬀerent
values of
k
. Start with some small value like 500 or 1000, and then increase it up to a point when
Matlab hangs so long that you lose patience – note the value of
k
when this happens. Try at least 10
random matrices of each size, and average the computation time returned by
tic
and
toc
. Plot the
mean duration against
k
.
(b) If we use the regularized loss function:
L
λ
(
w
) = (
Φw

y
)
T
(
Φw

y
) +
λ
w
T
w
(21)
show that the optimal
w
is:
ˆ
w
*
= (
Φ
T
Φ
+
λ
I
)

1
Φ
T
y
(22)
Hint: revisit lecture notes where I derived the optimal
w
for the unregularized basic form of least
squares error.
Problem
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
This is the end of the preview.
Sign up
to
access the rest of the document.
 Fall '10
 staff
 Normal Distribution, Maximum likelihood, Bishop equation, COG 118A Cogs, 118A Cogs 118A

Click to edit the document details