midterm_s1

# midterm_s1 - EE 376A/Stat 376A Prof T Weissman Information...

• Notes
• 5

This preview shows pages 1–3. Sign up to view the full content.

EE 376A/Stat 376A Information Theory Prof. T. Weissman Tuesday, February 6, 2007 Sample Midterm Solution 1. ( 25 points ) True or False? If the inequality is true, prove it, otherwise, give a counterexample: (a) H ( X, Y | Z ) H ( X | Z ) (b) H ( X | Z ) H ( Z ) (c) H ( X, Y, Z ) - H ( X, Y ) H ( X, Z ) - H ( X ) (d) H ( X | Z ) H ( X ) - H ( Z ) Solution: (a) Inequality H ( X, Y | Z ) H ( X | Z ) is true. H ( X, Y | Z ) = H ( X | Z ) + H ( X | Z, Y ) H ( X | Z ) The equality follows the chain rule. (b) Inequality H ( X | Z ) H ( Z ) is false. Let Z be a deterministic variable and X be a Bernoulli(0.5) random variable independent of Z . Hence H ( X | Z ) = H ( X ) = 1 and H ( Z )=0. (c) Inequality H ( X, Y, Z ) - H ( X, Y ) H ( X, Z ) - H ( X ) is false. The right side is H ( X, Y, Z ) - H ( X, Y ) = H ( X, Y ) + H ( Z | X, Y ) - H ( X, Y ) = H ( Z | X, Y ) and left side is H ( X, Z ) - H ( X ) = H ( Z | X ) . Hence if we take the random variables Z, Y to be Bernoulli(0.5), Z = Y , and X to be Bernoulli(0.5) independent of Y then the left side equals zero, because H ( Z | X, Y ) = H ( Z | X, Z ) = 0, and the right side equals 1, because H ( Z | X ) = H ( Z ) = 1. (d) The inequality H ( X | Z ) H ( X ) - H ( Z ) is false. Let X and Z be random variables Bernoulli(0.5) independent of each other. H ( X | Z ) = H ( X ) = 1 and H ( X ) - H ( Z ) = 1 - 1 = 0. 1

This preview has intentionally blurred sections. Sign up to view the full version.

2. ( 25 points ) Variable length coding (a) Suppose X p , where p = 10 34 , 8 34 , 7 34 , 5 34 , 3 34 , 1 34 .
This is the end of the preview. Sign up to access the rest of the document.
• Fall '09
• Probability theory, y1, AEP, Mutual Information, Y1 |X

{[ snackBarMessage ]}

### What students are saying

• As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

Kiran Temple University Fox School of Business ‘17, Course Hero Intern

• I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

Dana University of Pennsylvania ‘17, Course Hero Intern

• The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

Jill Tulane University ‘16, Course Hero Intern