This preview shows pages 1–4. Sign up to view the full content.
This preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: EE 376A/Stat 376A Handout #32 Information Theory Thursday, March 12, 2009 Prof. T. Cover Solutions to Practice Final Examination (Note: When the solutions refer to particular homework problems, it is speaking with respect to homework problems that were assigned in the year in which this exam was given – not the ones assigned during the Winter 20082009 instantiation of EE376A.) 1. Graph. What is the entropy rate of a random walk on the star graph with a central hub node and n edges: Solution: Graph. Note that the stationary distribution is (1 / 2 n, . . ., 1 / 2 n bracehtipupleft bracehtipdownrightbracehtipdownleft bracehtipupright n , n/ 2 n ) and only the central hub node has nonzero conditional entropy. Hence, the entropy rate is 1 2 log n . 2. Optimal code when entropy is infinite. Let X be integer valued with H ( X ) = ∑ ∞ i =1 p i log p i = ∞ . Thus the expected (binary) 1 description length L = ∑ p i l i is infinite, even for the Shannon ideal codeword length l * i = log 1 p i . Show nonetheless that { l * i } is better than { l i } for any other instantaneous code in the sense that ∑ ∞ i =1 p i ( l i l * i ) ≥ 0, for all { l i } satisfying the Kraft inequality. Solution: Optimal code when entropy is infinite. ∞ summationdisplay i =1 p i ( l i l * i ) = summationdisplay p i log p i 2 l i = summationdisplay p i log p i r i log( summationdisplay 2 l i ) ≥ , where r i = 2 l i ∑ 2 l i and the last inequality follows from the nonnegativity of D ( p  r ) and Kraft inequality, ∑ 2 l i ≤ 1. 3. Huffman code. (a) Find the binary Huffman code for p = { 7 20 , 4 20 , 4 20 , 3 20 , 2 20 } . (b) Guess the optimal (minimal expected description length) binary code for the integer valued random variable X , where Pr { X = i } = p i q, i = 0 , 1 , . . ., and p = . 6. Solution: Huffman code. (a) The Huffman tree for this distribution is Codeword 00 7 7 8 12 20 10 4 5 7 8 11 4 4 5 010 3 4 011 2 (b) Since p i 1 q ≥ p i q ≥ ∑ ∞ k = i +1 p k q = p i +1 for all i , the Huffman tree is 2 Codeword q q . . . q q . . . q q 1 10 pq pq . . . pq pq . . . pq p 110 p 2 q p 2 q . . . p 2 q p 2 q . . . p 2 ....
View
Full
Document
This document was uploaded on 12/01/2010.
 Fall '09

Click to edit the document details