practiceFinalSol

practiceFinalSol - EE 376A/Stat 376A Handout #32...

Info iconThis preview shows pages 1–4. Sign up to view the full content.

View Full Document Right Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: EE 376A/Stat 376A Handout #32 Information Theory Thursday, March 12, 2009 Prof. T. Cover Solutions to Practice Final Examination (Note: When the solutions refer to particular homework problems, it is speaking with respect to homework problems that were assigned in the year in which this exam was given – not the ones assigned during the Winter 2008-2009 instantiation of EE376A.) 1. Graph. What is the entropy rate of a random walk on the star graph with a central hub node and n edges: Solution: Graph. Note that the stationary distribution is (1 / 2 n, . . ., 1 / 2 n bracehtipupleft bracehtipdownrightbracehtipdownleft bracehtipupright n , n/ 2 n ) and only the central hub node has non-zero conditional entropy. Hence, the entropy rate is 1 2 log n . 2. Optimal code when entropy is infinite. Let X be integer valued with H ( X ) =- ∑ ∞ i =1 p i log p i = ∞ . Thus the expected (binary) 1 description length L = ∑ p i l i is infinite, even for the Shannon ideal codeword length l * i = log 1 p i . Show nonetheless that { l * i } is better than { l i } for any other instantaneous code in the sense that ∑ ∞ i =1 p i ( l i- l * i ) ≥ 0, for all { l i } satisfying the Kraft inequality. Solution: Optimal code when entropy is infinite. ∞ summationdisplay i =1 p i ( l i- l * i ) = summationdisplay p i log p i 2- l i = summationdisplay p i log p i r i- log( summationdisplay 2- l i ) ≥ , where r i = 2- l i ∑ 2- l i and the last inequality follows from the nonnegativity of D ( p || r ) and Kraft inequality, ∑ 2- l i ≤ 1. 3. Huffman code. (a) Find the binary Huffman code for p = { 7 20 , 4 20 , 4 20 , 3 20 , 2 20 } . (b) Guess the optimal (minimal expected description length) binary code for the integer valued random variable X , where Pr { X = i } = p i q, i = 0 , 1 , . . ., and p = . 6. Solution: Huffman code. (a) The Huffman tree for this distribution is Codeword 00 7 7 8 12 20 10 4 5 7 8 11 4 4 5 010 3 4 011 2 (b) Since p i- 1 q ≥ p i q ≥ ∑ ∞ k = i +1 p k q = p i +1 for all i , the Huffman tree is 2 Codeword q q . . . q q . . . q q 1 10 pq pq . . . pq pq . . . pq p 110 p 2 q p 2 q . . . p 2 q p 2 q . . . p 2 ....
View Full Document

This document was uploaded on 12/01/2010.

Page1 / 8

practiceFinalSol - EE 376A/Stat 376A Handout #32...

This preview shows document pages 1 - 4. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online