This preview shows pages 1–3. Sign up to view the full content.
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: EE 376A/Stat 376A Handout #22 Information Theory Tuesday, February 17, 2009 Prof. T. Cover Solutions to Homework Set #5 1. Conditional Entropy Let ( X,Y ) ∼ p ( x,y ). (a) Express H ( X  X + Y ) in terms of H ( X,Y ) and H ( X + Y ). (b) Suppose H ( X ) > H ( Y ). Is H ( X  X + Y ) > H ( Y  X + Y )? Solution: Conditional Entropy. (a) H ( X  X + Y ) = H ( X,X + Y ) − H ( X + Y ) = H ( X,Y ) − H ( X + Y ). (b) From (a) we can see that H ( X  X + Y ) is symmetric in X,Y , hence H ( X  X + Y ) = H ( Y  Y + X ), regardless of H ( X ) and H ( Y ). 2. Can side information make a bad situation worse? Suppose we have a horse race with outcome X ∈ { 1 , 2 ,...,m } and side information Y , where ( X,Y ) ∼ p ( x,y ) = p ( x ) p ( y  x ). The odds are m for 1. (a) Find the growth optimal strategy b ( x ) and the associated growth rate of wealth max b ( · ) W ( b ( x ) ,p ( x )) for the gambler. (b) Given side information, what is the growth optimal b ( x  y ) and the associated growth rate? What is the improvement Δ W ? Call it Δ W p . Now suppose another gambler believes (incorrectly) that X ∼ q ( x ), and that ( X,Y ) ∼ q ( x ) p ( y  x ), i.e. he believes the joint distribution is q ( x,y ) = q ( x ) p ( y  x ). Note that the conditional distribution q ( y  x ) = p ( y  x ) is the same as in parts (a) and (b). Thus the noise in the observation of Y given X is the same in each version. Only the estimate of the true distribution of X is different. (c) The q gambler now gambles to maximize the growth rate as if q ( x ) is true, without using side information Y . What is the growth rate W ? (d) The q gambler is now given side information Y , still believing q ( x,y ) is the true distribution. Find his optimal b ( x  y ) and associated growth rate. (e) Now calculate Δ W for the q gambler (call it Δ W q ). 1 (f) Express the difference Δ W q − Δ W p . Is Δ W p or Δ W q larger? This difference has a nice expression. Side information helps more when you are wrong than when you are right. Solution: Can side information make a bad situation worse? Without side information, it is optimal to set b ( x ) to be the distribution of X ; while with side information it is optimal to set b ( x  y ) to be the conditional distribution of X given Y . (a) b ( x ) = p ( x ). W = summationdisplay x p ( x ) log b ( x ) o ( x ) = − H ( X ) + log m. (b) b ( x  y ) = p ( x  y ). W = − H ( X  Y ) + log m. Δ W p = I ( X ; Y ) . (c) b ( x ) = q ( x ). W = summationdisplay x p ( x ) log b ( x ) o ( x ) = − H ( X ) − D ( p ( x )  q ( x )) + log m....
View Full
Document
 Spring '09
 B.K.Dey

Click to edit the document details