This preview shows pages 1–4. Sign up to view the full content.
This preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: EE 376A/Stat 376A Handout #35 Information Theory Monday, March 16, 2009 Prof. T. Cover Solution to Final Examination 1. ( 10 points ) Huffman code Find the Huffman binary codeword lengths for the probability assignment p = (1 / 45 , 1 / 45 , 6 / 45 , 6 / 45 , 7 / 45 , 8 / 45 , 8 / 45 , 8 / 45) . Solution to Huffman code Here is one possible solution. code 0000 x 1 1 2 6 8 8 13 29 45 0001 x 2 1 6 7 8 8 16 16 001 x 3 6 6 8 8 13 16 100 x 4 6 7 8 8 16 101 x 5 7 8 8 13 110 x 6 8 8 8 111 x 7 8 8 01 x 8 8 The codeword lengths are (4 , 4 , 3 , 3 , 3 , 3 , 3 , 2), and the average codeword length is 43 15 ≅ 2 . 8667. Different solutions are possible depending on how one goes about building the tree. Another solution, with the same average codeword length, has codeword lengths of (5 , 5 , 4 , 3 , 3 , 3 , 2 , 2). 2. ( 20 points ) Random walk obscured by clouds A random walk takes place on this undirected connected graph. Part of the graph is obscured. We observe that 1 6 th of the time, in the limit as n → ∞ , is spent in A . (a) What fraction of the time is spent in the cloud? (b) What is the entropy rate of the random walk? Solution to Random walk obscured by clouds (a) Using the formula for the stationary distribution of a random walk on an undi 1 C A B D E rected connected graph, the fraction of time spent at node A is E A 2 E , where E is the total number of edges in the graph, and E A is the degree of A . Since E A = 2, and E A 2 E = 1 / 6, it follows that E = 6. From what is visible in the picture, there are either 5 or 6 edges which are not obscured, since we are not sure whether the edge that enters the cloud from A is the same as the edge that enters the cloud from E . However, since E = 6, we know that the edges entering the cloud from A and E must be distinct, because if they were not, then the graph would only have 5 edges in total. So there is only one node in the cloud. The fraction of time spent in the cloud is thus the fraction of time spent at this hidden node, and using the formula for the stationary distribution, the answer is 1/6. (b) The entropy rate is H ( X ) = log 2 (2 E ) − H ( 2 12 , 1 12 , 2 12 , 2 12 , 3 12 , 2 12 ) = 2 3 + 1 4 log 2 3 = 1 . 06291 . 3. ( 20 points ) Entropy inequality Let ( X,Y ) ∼ p ( x,y ) be discrete real valued random variables. (a) Compare H ( X  X · Y ) and H ( X,Y ) − H ( X · Y ) . Is the lefthand side ≤ , ≥ , or = to the righthand side? (b) Find conditions for equality. Solution to Entropy inequality The answer for part (a) is ≤ , and the answer for part (b) is twofold: either Pr { X = 0 } = 2 0, or H ( Y  X = 0) = 0. Here are two different ways of arriving at these same conclusions. • Solution I (Concise): H ( X  XY ) + H ( XY ) ( a ) = H ( X,XY ) ( b ) ≤ H ( X,Y ) where each is step is justified as follows: (a) chain rule (b) ( X,XY ) is a function of ( X,Y ) Thus, moving H ( XY ) to the other side, H ( X  XY ) ≤ H ( X,Y ) − H ( XY ) ....
View
Full
Document
This note was uploaded on 02/05/2012 for the course EE EE308 taught by Professor B.k.dey during the Spring '09 term at IIT Bombay.
 Spring '09
 B.K.Dey

Click to edit the document details