HW1_ES250_sol - Harvard SEAS ES250 Information Theory Homework 1 Solution 1 Let p(x y be given by XY 0 1 0 1/3 0 1 1/3 1/3 Evaluate the following

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon
Harvard SEAS ES250 – Information Theory Homework 1 Solution 1. Let p ( x,y ) be given by X ± Y 0 1 0 1/3 1/3 1 0 1/3 Evaluate the following expressions: (a) H ( X ), H ( Y ) (b) H ( X | Y ), H ( Y | X ) (c) H ( X,Y ) (d) H ( Y ) - H ( Y | X ) (e) I ( X ; Y ) (f) Draw a Venn diagram for the quantities in (a) through (e) Solution : (a) H ( X ) = 2 3 log 3 2 + 1 3 log 3 = 0 . 918 bits = H ( Y ) (b) H ( X | Y ) = 1 3 H ( X | Y = 0) + 2 3 H ( X | Y = 1) = 0 . 667 bits = H ( Y | X ) (c) H ( X,Y ) = 3 × 1 3 log 3 = 1 . 585 bits (d) H ( Y ) - H ( Y | X ) = 0 . 251 bits (e) I ( X ; Y ) = H ( Y ) - H ( Y | X ) = 0 . 251 bits (f) 2. Entropy of functions of a random variable (a) Let X be a discrete random variable. Show that the entropy of a function of X is less than or equal to the entropy of X by justifying the following steps: H ( X,g ( X )) ( a ) = H ( X ) + H ( g ( X ) | X ) ( b ) = H ( X ) H ( X,g ( X )) ( c ) = H ( g ( X )) + H ( X | g ( X )) ( d ) H ( g ( X )) Thus H ( g ( X )) H ( X ). (b) Let Y = X 7 , where X is a random variable taking in positive and negative integer values. What is the relationship of H ( X ) and H ( Y )? What if Y = cos( πX/ 3) ? Solution : (a) STEP (a) : H ( X,g ( X )) = H ( X ) + H ( g ( X ) | X ) by the chain rule for entropies. 1
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Harvard SEAS ES250 – Information Theory STEP (b) : H ( g ( X ) | X ) = 0 since for any particular value of X , g ( X ) is fixed, and hence H ( g ( X ) | X ) = x p ( x ) H ( g ( X ) | X = x ) = x 0 = 0. STEP (c) : H ( X,g ( X )) = H ( g ( X )) + H ( X | g ( X )) again by the chain rule. STEP (d) : H ( X | g ( X )) 0, with equality iff X is a function of g ( X ), i.e., g ( . ) is one-to-one. Hence H ( X,g ( X )) H ( g ( X )). Combining STEP (b) and (d), we obtain H ( X ) H ( g ( X )). (b) By the part (a), we know that passing a random variable through a function can only reduce the entropy or leave it unchanged, but never increase it. That is, H ( g ( X )) H ( X ), for any function g . The reason for this is simply that if the function g is not one-to-one, then it will merge some states, reducing entropy. The trick, then, for this problem, is simply to determine whether or not the mappings are one-to-one. If so, then entropy is unchanged. If the mappings are not one-to-one, then the entropy is necessarily decreased. Note that whether the function is one-to-one or not is only meaningful for the support of X , i.e. for all x with p ( x ) > 0. Y
Background image of page 2
Image of page 3
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 11/19/2010 for the course ANTH 122 taught by Professor 323 during the Spring '10 term at Centennial College.

Page1 / 7

HW1_ES250_sol - Harvard SEAS ES250 Information Theory Homework 1 Solution 1 Let p(x y be given by XY 0 1 0 1/3 0 1 1/3 1/3 Evaluate the following

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online