{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

finalSol - EE 376A/Stat 376A Information Theory Prof T...

Info icon This preview shows pages 1–4. Sign up to view the full content.

View Full Document Right Arrow Icon
EE 376A/Stat 376A Handout #35 Information Theory Monday, March 16, 2009 Prof. T. Cover Solution to Final Examination 1. ( 10 points ) Huffman code Find the Huffman binary codeword lengths for the probability assignment p = (1 / 45 , 1 / 45 , 6 / 45 , 6 / 45 , 7 / 45 , 8 / 45 , 8 / 45 , 8 / 45) . Solution to Huffman code Here is one possible solution. code 0000 x 1 1 2 6 8 8 13 29 45 0001 x 2 1 6 7 8 8 16 16 001 x 3 6 6 8 8 13 16 100 x 4 6 7 8 8 16 101 x 5 7 8 8 13 110 x 6 8 8 8 111 x 7 8 8 01 x 8 8 The codeword lengths are (4 , 4 , 3 , 3 , 3 , 3 , 3 , 2), and the average codeword length is 43 15 2 . 8667. Different solutions are possible depending on how one goes about building the tree. Another solution, with the same average codeword length, has codeword lengths of (5 , 5 , 4 , 3 , 3 , 3 , 2 , 2). 2. ( 20 points ) Random walk obscured by clouds A random walk takes place on this undirected connected graph. Part of the graph is obscured. We observe that 1 6 th of the time, in the limit as n → ∞ , is spent in A . (a) What fraction of the time is spent in the cloud? (b) What is the entropy rate of the random walk? Solution to Random walk obscured by clouds (a) Using the formula for the stationary distribution of a random walk on an undi- 1
Image of page 1

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
C A B D E rected connected graph, the fraction of time spent at node A is E A 2 E , where E is the total number of edges in the graph, and E A is the degree of A . Since E A = 2, and E A 2 E = 1 / 6, it follows that E = 6. From what is visible in the picture, there are either 5 or 6 edges which are not obscured, since we are not sure whether the edge that enters the cloud from A is the same as the edge that enters the cloud from E . However, since E = 6, we know that the edges entering the cloud from A and E must be distinct, because if they were not, then the graph would only have 5 edges in total. So there is only one node in the cloud. The fraction of time spent in the cloud is thus the fraction of time spent at this hidden node, and using the formula for the stationary distribution, the answer is 1/6. (b) The entropy rate is H ( X ) = log 2 (2 E ) H ( 2 12 , 1 12 , 2 12 , 2 12 , 3 12 , 2 12 ) = 2 3 + 1 4 log 2 3 = 1 . 06291 . 3. ( 20 points ) Entropy inequality Let ( X, Y ) p ( x, y ) be discrete real valued random variables. (a) Compare H ( X | X · Y ) and H ( X, Y ) H ( X · Y ) . Is the left-hand side , , or = to the right-hand side? (b) Find conditions for equality. Solution to Entropy inequality The answer for part (a) is , and the answer for part (b) is twofold: either Pr { X = 0 } = 2
Image of page 2
0, or H ( Y | X = 0) = 0. Here are two different ways of arriving at these same conclusions. Solution I (Concise): H ( X | XY ) + H ( XY ) ( a ) = H ( X, XY ) ( b ) H ( X, Y ) where each is step is justified as follows: (a) chain rule (b) ( X, XY ) is a function of ( X, Y ) Thus, moving H ( XY ) to the other side, H ( X | XY ) H ( X, Y ) H ( XY ) . Equality holds if and only if either Pr { X = 0 } = 0, or Y is a constant (degenerate random variable) when X = 0.
Image of page 3

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Image of page 4
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern