spring2007 - 1 ECE 534 Elements of Information Theory...

Info icon This preview shows pages 1–2. Sign up to view the full content.

View Full Document Right Arrow Icon
1 ECE 534: Elements of Information Theory Midterm Exam I (Spring 2007) Problem 1 [20 pts] What are the relations ( , = , ) between the following pairs of expressions? Explain why. 1) H (5 X ) and H ( X ) ; 2) H ( X 0 | X - 1 ) and H ( X 0 | X - 1 , X 1 ) ; 3) H ( X 1 , X 2 , . . . , X n ) and H ( c ( X 1 , X 2 , . . . , X n )) , where c ( x 1 , x 2 , . . . , x n ) is the Huffman codeword assigned to ( x 1 , x 2 , . . . , x n ) ; 4) H ( X, Y ) and H ( X ) + H ( Y ) . 5) H ( X n | X 1 , . . . , X n - 1 ) and H ( X n +1 | X 1 , . . . , X n ) , where ( X 1 , X 2 , . . . , X n , . . . ) is a stationary random process. Solution: (Provided by Bhanu Kamandla and Ishita Basu) 1) H (5 X ) = H ( X ) , since entropy depends on the probability of the variable not on the value of the variable. Another explanation is that f ( x ) = 5 x is an invertible function of x . 2) H ( X 0 | X - 1 ) H ( X 0 | X - 1 , X 1 ) , since conditioning reduces entropy. 3) H ( X 1 , X 2 , . . . , Xn ) = H ( c ( X 1 , X 2 , . . . , X n )) , since Huffman code is uniquely decodable, that is c ( X 1 , . . . , X n ) is an invertible function. 4) H ( X, Y ) = H ( X ) + H ( Y | X ) H ( X ) + H ( Y ) (chain rule and conditioning reduces entropy) 5) H ( X n | X 1 , . . . , X n - 1 ) = H ( X n +1 | X 2 , . . . , X n ) Since X 1 , X 2 , . . . , X n is a stationary random pro- cess. H ( X n +1 | X 1 , X 2 , . . . , X n ) H ( X n +1 | X 2 , . . . , X n ) , since conditioning reduces entropy. Therefore H ( X n | X 1 , . . . , X n - 1 ) H ( X n +1 | X 1 , X 2 , . . . , X n ) . Problem 2 [20 pts] (a) Find a binary Huffman code for the following random variable: X = parenleftBigg x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 0 . 35 0 . 26 0 . 19 0 . 07 0 . 04 0 . 04 0 . 03 0 . 02 parenrightBigg (b) Find a ternary Huffman code for the above source.
Image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Image of page 2
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern