This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: EE 376A/Stat 376A Handout #22 Information Theory Tuesday, February 17, 2009 Prof. T. Cover Solutions to Homework Set #5 1. Conditional Entropy Let ( X,Y ) ∼ p ( x,y ). (a) Express H ( X  X + Y ) in terms of H ( X,Y ) and H ( X + Y ). (b) Suppose H ( X ) > H ( Y ). Is H ( X  X + Y ) > H ( Y  X + Y )? Solution: Conditional Entropy. (a) H ( X  X + Y ) = H ( X,X + Y ) − H ( X + Y ) = H ( X,Y ) − H ( X + Y ). (b) From (a) we can see that H ( X  X + Y ) is symmetric in X,Y , hence H ( X  X + Y ) = H ( Y  Y + X ), regardless of H ( X ) and H ( Y ). 2. Can side information make a bad situation worse? Suppose we have a horse race with outcome X ∈ { 1 , 2 ,...,m } and side information Y , where ( X,Y ) ∼ p ( x,y ) = p ( x ) p ( y  x ). The odds are m for 1. (a) Find the growth optimal strategy b ( x ) and the associated growth rate of wealth max b ( · ) W ( b ( x ) ,p ( x )) for the gambler. (b) Given side information, what is the growth optimal b ( x  y ) and the associated growth rate? What is the improvement Δ W ? Call it Δ W p . Now suppose another gambler believes (incorrectly) that X ∼ q ( x ), and that ( X,Y ) ∼ q ( x ) p ( y  x ), i.e. he believes the joint distribution is q ( x,y ) = q ( x ) p ( y  x ). Note that the conditional distribution q ( y  x ) = p ( y  x ) is the same as in parts (a) and (b). Thus the noise in the observation of Y given X is the same in each version. Only the estimate of the true distribution of X is different. (c) The q gambler now gambles to maximize the growth rate as if q ( x ) is true, without using side information Y . What is the growth rate W ? (d) The q gambler is now given side information Y , still believing q ( x,y ) is the true distribution. Find his optimal b ( x  y ) and associated growth rate. (e) Now calculate Δ W for the q gambler (call it Δ W q ). 1 (f) Express the difference Δ W q − Δ W p . Is Δ W p or Δ W q larger? This difference has a nice expression. Side information helps more when you are wrong than when you are right. Solution: Can side information make a bad situation worse? Without side information, it is optimal to set b ( x ) to be the distribution of X ; while with side information it is optimal to set b ( x  y ) to be the conditional distribution of X given Y . (a) b ( x ) = p ( x ). W = summationdisplay x p ( x ) log b ( x ) o ( x ) = − H ( X ) + log m. (b) b ( x  y ) = p ( x  y ). W = − H ( X  Y ) + log m. Δ W p = I ( X ; Y ) . (c) b ( x ) = q ( x ). W = summationdisplay x p ( x ) log b ( x ) o ( x ) = − H ( X ) − D ( p ( x )  q ( x )) + log m....
View
Full Document
 Spring '09
 B.K.Dey
 arithmetic coding, Prefix code, huffman code

Click to edit the document details