god2 - EE 376A/Stat 376A Handout #22 Information Theory...

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: EE 376A/Stat 376A Handout #22 Information Theory Tuesday, February 17, 2009 Prof. T. Cover Solutions to Homework Set #5 1. Conditional Entropy Let ( X,Y ) ∼ p ( x,y ). (a) Express H ( X | X + Y ) in terms of H ( X,Y ) and H ( X + Y ). (b) Suppose H ( X ) > H ( Y ). Is H ( X | X + Y ) > H ( Y | X + Y )? Solution: Conditional Entropy. (a) H ( X | X + Y ) = H ( X,X + Y ) − H ( X + Y ) = H ( X,Y ) − H ( X + Y ). (b) From (a) we can see that H ( X | X + Y ) is symmetric in X,Y , hence H ( X | X + Y ) = H ( Y | Y + X ), regardless of H ( X ) and H ( Y ). 2. Can side information make a bad situation worse? Suppose we have a horse race with outcome X ∈ { 1 , 2 ,...,m } and side information Y , where ( X,Y ) ∼ p ( x,y ) = p ( x ) p ( y | x ). The odds are m for 1. (a) Find the growth optimal strategy b ( x ) and the associated growth rate of wealth max b ( · ) W ( b ( x ) ,p ( x )) for the gambler. (b) Given side information, what is the growth optimal b ( x | y ) and the associated growth rate? What is the improvement Δ W ? Call it Δ W p . Now suppose another gambler believes (incorrectly) that X ∼ q ( x ), and that ( X,Y ) ∼ q ( x ) p ( y | x ), i.e. he believes the joint distribution is q ( x,y ) = q ( x ) p ( y | x ). Note that the conditional distribution q ( y | x ) = p ( y | x ) is the same as in parts (a) and (b). Thus the noise in the observation of Y given X is the same in each version. Only the estimate of the true distribution of X is different. (c) The q gambler now gambles to maximize the growth rate as if q ( x ) is true, without using side information Y . What is the growth rate W ? (d) The q gambler is now given side information Y , still believing q ( x,y ) is the true distribution. Find his optimal b ( x | y ) and associated growth rate. (e) Now calculate Δ W for the q gambler (call it Δ W q ). 1 (f) Express the difference Δ W q − Δ W p . Is Δ W p or Δ W q larger? This difference has a nice expression. Side information helps more when you are wrong than when you are right. Solution: Can side information make a bad situation worse? Without side information, it is optimal to set b ( x ) to be the distribution of X ; while with side information it is optimal to set b ( x | y ) to be the conditional distribution of X given Y . (a) b ( x ) = p ( x ). W = summationdisplay x p ( x ) log b ( x ) o ( x ) = − H ( X ) + log m. (b) b ( x | y ) = p ( x | y ). W = − H ( X | Y ) + log m. Δ W p = I ( X ; Y ) . (c) b ( x ) = q ( x ). W = summationdisplay x p ( x ) log b ( x ) o ( x ) = − H ( X ) − D ( p ( x ) || q ( x )) + log m....
View Full Document

Page1 / 9

god2 - EE 376A/Stat 376A Handout #22 Information Theory...

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online