{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

midterm_s1 - EE 376A/Stat 376A Prof T Weissman Information...

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon
EE 376A/Stat 376A Information Theory Prof. T. Weissman Tuesday, February 6, 2007 Sample Midterm Solution 1. ( 25 points ) True or False? If the inequality is true, prove it, otherwise, give a counterexample: (a) H ( X, Y | Z ) H ( X | Z ) (b) H ( X | Z ) H ( Z ) (c) H ( X, Y, Z ) - H ( X, Y ) H ( X, Z ) - H ( X ) (d) H ( X | Z ) H ( X ) - H ( Z ) Solution: (a) Inequality H ( X, Y | Z ) H ( X | Z ) is true. H ( X, Y | Z ) = H ( X | Z ) + H ( X | Z, Y ) H ( X | Z ) The equality follows the chain rule. (b) Inequality H ( X | Z ) H ( Z ) is false. Let Z be a deterministic variable and X be a Bernoulli(0.5) random variable independent of Z . Hence H ( X | Z ) = H ( X ) = 1 and H ( Z )=0. (c) Inequality H ( X, Y, Z ) - H ( X, Y ) H ( X, Z ) - H ( X ) is false. The right side is H ( X, Y, Z ) - H ( X, Y ) = H ( X, Y ) + H ( Z | X, Y ) - H ( X, Y ) = H ( Z | X, Y ) and left side is H ( X, Z ) - H ( X ) = H ( Z | X ) . Hence if we take the random variables Z, Y to be Bernoulli(0.5), Z = Y , and X to be Bernoulli(0.5) independent of Y then the left side equals zero, because H ( Z | X, Y ) = H ( Z | X, Z ) = 0, and the right side equals 1, because H ( Z | X ) = H ( Z ) = 1. (d) The inequality H ( X | Z ) H ( X ) - H ( Z ) is false. Let X and Z be random variables Bernoulli(0.5) independent of each other. H ( X | Z ) = H ( X ) = 1 and H ( X ) - H ( Z ) = 1 - 1 = 0. 1
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
2. ( 25 points ) Variable length coding (a) Suppose X p , where p = 10 34 , 8 34 , 7 34 , 5 34 , 3 34 , 1 34 .
Background image of page 2
Image of page 3
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}