This preview shows pages 1–3. Sign up to view the full content.
EECS 229A
Spring 2007
*
*
Solutions to Homework 7
1. Problem 11.6 on pg. 401 of the text.
Solution
:
Biased estimates may be better
The notation used in this problem is ﬂawed. The estimator in (11.319) based on
n
samples
is not the same as the estimator in (11.318) based on
n

1 samples. We will use the
notation
˜
S
2
n
for the estimator of (11.319). Note that
S
n
and
˜
S
n
are estimators of the
standard deviation.
(a)
E
[
¯
X
n
] =
E
[
1
n
n
X
k
=1
X
k
] =
1
n
E
[
X
k
] =
μ .
Thus
¯
X
n
is an unbiased estimator of the mean.
(b) We have
E
[
S
2
n
]
=
E
[
1
n
n
X
k
=1
(
X
k

¯
X
n
)
2
]
=
1
n
n
X
k
=1
E
[(
X
k

¯
X
n
)
2
]
(
a
)
=
E
[(
X
1

¯
X
n
)
2
]
=
E
[(
n

1
n
X
1

1
n
n
X
l
=2
X
l
)
2
]
=
(
n

1
n
)
2
σ
2
+ (
n

1)
1
n
2
σ
2
=
n

1
n
σ
2
,
where step (a) is by symmetry.
Further, since
˜
S
2
n
=
n
n

1
S
2
n
, we have
E
[
˜
S
2
n
] =
σ
2
.
Thus
S
2
n
is a biased estimator of the variance, while
˜
S
2
n
is an unbiased estimator of
the variance.
(c) We have
E
[(
S
2
n

n

1
n
σ
2
)
2
] =
E
[
S
4
n
]

(
n

1
n
)
2
σ
4
= (
n

1
n
)
2
E
[
˜
S
4
n
]

(
n

1
n
)
2
σ
4
= (
n

1
n
)
2
E
[(
˜
S
2
n

σ
2
)
2
]
.
1
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document This shows that the variance of the biased estimator of variance in (11.318) is strictly
smaller than the variance of the unbiased estimator of variance in (11.319).
2. Problem 11.7 on pg. 401 of the text.
Solution
:
Fisher information and relative entropy
We write
ln
f
(
x
;
θ
0
) = ln
f
(
x
;
θ
) + (
θ
0

θ
)
∂
∂θ
ln
f
(
x
;
θ
) +
1
2
(
θ
0

θ
)
2
∂
2
∂
2
θ
ln
f
(
x
;
θ
) +
o
x
((
θ
0

θ
)
2
)
,
where, for each ﬁxed
x
,
o
x
((
θ
0

θ
)
2
) is a function that when divided by (
θ
0

θ
)
2
goes to
0 as
θ
0
→
θ
. We next integrate both sides of this equality against
f
(
x
;
θ
)
dx
. We get
Z
f
(
x
;
θ
)ln
f
(
x
;
θ
0
)
dx
=
Z
f
(
x
;
θ
)ln
f
(
x
;
θ
)
dx
+ (
θ
0

θ
)
Z
f
(
x
;
θ
)
∂
∂θ
ln
f
(
x
;
θ
)
dx
+
1
2
(
θ
0

θ
)
2
Z
f
(
x
;
θ
)
∂
2
∂
2
θ
ln
f
(
x
;
θ
)
dx
+
o
((
θ
0

θ
)
2
)
,
(1)
if
R
f
(
x
;
θ
)
o
x
((
θ
0

θ
)
2
)
dx
=
o
((
θ
0

θ
)
2
), which we assume.
We observe that
Z
f
(
x
;
θ
)
∂
∂θ
ln
f
(
x
;
θ
)
dx
=
Z
f
(
x
;
θ
)
1
f
(
x
;
θ
)
∂
∂θ
f
(
x
;
θ
)
dx
=
Z
∂
∂θ
f
(
x
;
θ
)
dx
(
a
)
=
∂
∂θ
Z
f
(
x
;
θ
)
dx
=
0
,
where in step (a) we assume that we can interchange the derivative and the integral.
This is the end of the preview. Sign up
to
access the rest of the document.
This note was uploaded on 10/25/2010 for the course ECE 544 taught by Professor Liu during the Spring '10 term at Ill. Chicago.
 Spring '10
 LIU

Click to edit the document details