Berkeley UC Department of Statistics STAT 210A: Introduction to Mathematical Statistics Problem Set 4- Solutions Fall 2007 Issued: Thursday, September 20 Due: Thursday, September 27 Problem 4.1 1c (a) P(X x) = F (x) = 1 1(x > 1). x f (x) = cxc1 1(x > 1) = exp {(c + 1) log x + log c} 1(x > 1) : 1-dimensional exponential family with sucient statistics T = log X . (b) fT (t) = ck c e(c+1)t et 1(t > log k ) = exp {ct + log(1(t > log k )) + c log k + log c}, where log 0 = 0. Thus, T has an exponential distribution with parameters log k and 1/c. Problem 4.2 (a) Letting g (x; ) = 1 f
x
, we know that: log(g (x; )) = log() + log f x
Dierentiating with respect to : 1 xf log(g (x; )) = 1 + f So: I () = =
dy =dx x x
log(g (x; )) 1 2 1 2 xf 1+ f 1+
x x 2
2
g (x, )dx
2
1 x f dx
=
yf (y ) f (y )
f (y )dy
1
(b) For the parameter () = log(), we have from Keener 8.1.8 that: I ( ) = = = (c) For the Cauchy distribution C (0, ): p(x; ) = where f (y ) = so, we can apply the result from (a): f (y ) = and as a result: 1 Finally, from (a): I () = and transforming y = tan(t): I () = = 1 2 1 22
2
1 () ( )
2
2
I () I ( )
1+y
f (y ) f (y )
2
f (y )dy
1 x 1 =f x2 + 2 11 1 + y2
2y 1 (1 + y 2 )2 1 y2 1 + y2
2y 2 yf (y ) =1 f (y ) 1 + y2
=
1 2
1 y2 dy (1 + y 2 )2
2
cos2 (2t)dt
Problem 4.3 We compute the Fisher information matrix as: I ( ) = E =E log q (x, )
2
log p (x, ( )) 2
2
Now, using the chain rule:
p (x, ( )) log p (x, ( )) = ( ) p (x, ( ))
= So: I ( ) = = () ()
2
()
1
p (x, ( ))
p (x, ( ))
E I ()
log p (x, )
2
2
Now, to prove that the information bound is invariant, apply the chain rule again to get: g (( )) = ( )g (( )) = so it follows that: B ( ) = = =
g ( ( ))
()
1
g (( ))
I ( ) ()1 g (( )) [ ()]2 I () (g (( ))2 I ()
2
Notice that I ( ) can be interpreted as the expected value of the curvature of the loglikelihood function at a given model. It follows that it is not surprising that it is sensitive to the parameterization (think of rescaling the parameters). The information bound, however, corresponds to a quantity (the variance of (X )) that is not dependent on the parameterization. Problem 4.4 (a) p(Y0 = yo , . . . , Yn = yn ) = p(Y0 = y0 ) p(Y1 = y1 |Y0 = y0 ) p(Yn = yn |Y0 = y0 , . . . , Yn1 = yn1 ) = p(Y0 = y0 ) p(Y1 = y1 |Y0 = y0 ) p(Yn = yn |Yn1 = yn1 ) e y0 ey0 (y0 )y1 eyn1 (yn1 )yn = y0 ! y1 ! yn ! 3
l(; y0 , . . . , yn ) = log(p(Y0 = yo , . . . , Yn = yn ))
n n
= l() 2 l ( ) 2 Therefore, =
i=0 n
yi1 + log
i=0 n
yi + g (y0 , . . . , yn ) where y1 = 1 yi
=
i=0
1 yi1 +
n
i=0
1 = 2 =
yi 0
i=0 n j =0 Yj . n 1 j =0 Yj
n j =0 Yj 1
n j =0 Yj
1+
(b) Note that E(Yi ) = E(E(Yi |Yi1 )) = E(Yi1 ) = E(Yi1 ) for i = 1, . . . , n and E(Y0 ) = . Thus, E(Yi ) = i+1 . 2 l() 1 I () = E =E 2 2 If < 1, I ()
n
Yi =
i=0
1 1 n+1 ( + 2 + . . . + n+1 ) = 2 (1 )
1 4 (1 ) Intuitively, as time goes on, the information we can get is decreasing because Yi depends on the previous Yi1 and < 1. The they information carry depends on each other and < 1 means, the information we can get is decreasing. Problem 4.5 p(X ; ) for . Then p(X ; )
(a) Let V = (X ) and W = E (W ) = E (V W ) =
p(x; )dx =
p(x; )dx = 1
(x)p(x; )dx =
(x)p(x; )dx = E ( (X )) =
V ar (W ) = E (W 1)2 C ov (V, W ) = Cov (V, W )2 ( )2 = . Because V ar (W ) E [(W 1)2 ] V ar (V ) does not depend on , the inequality holds. By Cauchy-Schwartz inequality, V ar (V )
2 n
(b) E
p(x; ) p(x; )
= e2n n
ex dx
= en( ) . (RHS) in (a) is sup
( )2 n( ) 1 e
x2 for x 0. Then f (x) 0 x 0 and lim f (x) = lim f (x) = 0 Let f (x) = nx x x0 e 1 4
2 nenx nx h (x) = x e 1 2 n2 enx 2 2 2 + nx . For the such that h () = 0, h () = 2 + 2 (2 n) = 2 x (e 1) 2 (1 n) < 0. (Note that 1 < n < 1.6) Thus, such is local maxima and one 2 of is global maxima, because f (x) 0 x 0 and lim f (x) = lim f (x) = 0. Let h(x) = log f (x) = 2 log x log(enx 1) for x > 0. h (x) = (Actually, such is unique, except = 0. It will be a good exercise to show why it 2 has unique solution except 0, analytically, not numerically.) Then, f () = n = e 1 2 (2 n) = (2 n) 2 for n > 2. So, = attains the inequality. n n n (c) There exists t0 such that 2 et0 t0 t0 = 0. Then a = . Then V ar ( (X )) t0 e 1 n 2 t t0 1 . Actually Sharp bound for V ar ( (X )) is n0 (2 t0 ) = O n2 which dier from 2 n3
x 0 x
this scaling. The distribution in this problem violate one of regularity conditions (the distribution share common support) to guarantee CR bound. Thats why the lower bound is better than O(1/n). (d) Direct Calculation
n
P(X(1) t) =
i=1
P(Xi t) = en(t) . Thus, fX(1) (t) = nen(t) .
E(X(1) ) =
2 E(X(1) )
nen tent dt = +
2
1 n
E Thus, V ar(a (X )) = Fact
X(1)
1 . n2
1 n
2
2 2 + 2 n n 1 = 2 + 2 n = +
Using exponential spacing If Y1 , . . . , Yn i.i.d. Exp(), then (n r + 1)(Y(r) Y(r1) ) i.i.d. Exp() where X(0) = 0. Let Yi = Xi . Then Y1 , . . . , Yn i.i.d. Exp(1). By Fact, nY(1) Exp(1). E(X(1) ) = E(Y(1) ) + = V ar X(1) 1 n 1 + n 1 n2
= V ar(X(1) ) = V ar(Y(1) ) =
5
Problem 4.6 For the Poisson, we have p(y, ) = (y !)1 exp [ + k log()]. Conditional on Y 0, we notice that P (y 1) = 1 P (Y = 0) = 1 exp() the the probability mass function becomes: q (y, ) = I(y 1)(y !)1 exp [ log(1 exp()) + y log()]
e y log(q (y, )) = 1 + 1e 2 y e log(q (y, )) = 2 + 2 (1 e )2 Now, for a single observation (because q (y, ) is full rank exponential family) : I () = E = 2 log(q (y, )) 2 E Y e 2 (1 e )2
, 1e
We can compute E Y from the cumulant generating function as I () = = e 1 (1 e ) (1 e )2 1 exp() exp() (1 e )2
so:
Additionally for n independent samples, we have: I () = E 2 log( 2
n
q (yi , ))
i=1
= nI () n (1 exp() exp()) = (1 e )2 For an unbiased estimate of , g () = , so g () = 1 and the information lower bound is then given by 1 I () = (1 e )2 n (1 exp() exp())
If you have any question about grading or solutions, please come to see me(GSI, Choongsoon Bae). I cant be perfect. :-)
6