2006final - EE 376A/Stat 376A Prof. T. Weissman Information...

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon
EE 376A/Stat 376A Information Theory Prof. T. Weissman Thursday, March 23, 2006 Final 1. ( 20 points ) Three Shannon codes Let { U i } i 1 be a stationary 1st-order Markov source whose alphabet size is r . Note that the stationarity property implies that P ( u i ) ,P ( u i | u i - 1 ) does not depend on i . Assume that - log P ( u i ) and - log P ( u i | u i - 1 ) are integers for all ( u i ,u i - 1 ). Three students decided to use the Shannon coding scheme, based on different pmfs as follows: Alice suggested to code each symbol separately, i.e., code by using the pmf { P ( u i ) } . Bob suggested to code in pairs by using the joint pmf { P ( u i ,u i +1 ) } ,i = 1 , 3 , 5 ,... . In other words, first code ( U 1 ,U 2 ), then code ( U 3 ,U 4 ), and so on. Charlie suggested to code each symbol given the previous symbol by using the conditional pmf { P ( u i | u i - 1 ) } . In other words, first code U 1 , then code U 2 given U 1 , then code U 3 given U 2 , and so on. (a) If the source is memoryless, compare the expected codeword length per symbol, i.e., 1 n E‘ ( U n ), of each scheme, assuming n > 2 is even. (b) Repeat Part (a) for the case where the source has memory. Solutions (a) Because the source is memoryless Alice’s and Charlie’s coding schemes are identical. In addition it is given that - log P ( u i ) is an integer and hence Shanon coding for block length 1 has an average length of H ( U 1 ) = u 1 P ( u 1 )log( P ( u 1 )). Bob’s coding scheme is over pairs that their log of the PMF is also an integer, and therefore the average length of his code is 1 2 H ( U 1 ,U 2 ) and because of the memoryless property 1 2 H ( U 1 ,U 2 ) = H ( U 1 ). The conclusion is that all three codes have the same average length. (b) Alice’s Average code is 1 n E‘ ( U n ) = 1 n " n X i =1 X u i P ( u i )log P ( u i ) # = H ( U 1 ) 1
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
1 n E‘ ( U n ) = 1 n n/ 2 X i =1 X u 2 i ,u 2 i - 1 P ( u 2 i ,u 2 i - 1 )log P ( u 2 i ,u 2 i - 1 ) a = 1 n n/ 2 X i =1 X u 2 ,u 1 P ( u 2 ,u 1 )log P ( u 2 ,u 1 ) b = 1 2 [ H ( U 2 ,U 1 )] Charlie’s average code is 1 n E‘ ( U n ) = 1 n " X u 1 P ( u 1 )log P ( u 1 ) + n X i =2 X u i P ( U i - 1 P ( u i | u i - 1 )log P ( u i | u i - 1 ) # a = 1 n " X u 1 P ( u 1 )log P ( u 1 ) + ( n - 1) X u 2 P ( U 1 P ( u 2 | u 1 )log P ( u 2 | u 1 ) # b = 1 n [ H ( U 1 ) + ( n - 1) H ( U 2 | U 1 )] Equality (a) follows the stationary property and equality (b) follows the definition of entropy. Alice’s average code is the longest because H ( U 2 ) H ( U 2 | U 1 ). For n = 2 Bob’s and Charlie’s code are the same but for n 4 Bob’s average code is longer because H ( U 2 ,U 1 ) = H ( U 1 ) + H ( U 2 | U 1 ) 2 H ( U 2 | U 1 ) where U is a stationary source. Notice that equality holds iff the source is memoryless. 2
Background image of page 2
Image of page 3
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 12/01/2010 for the course ADLAC 1023 at Stanford.

Page1 / 11

2006final - EE 376A/Stat 376A Prof. T. Weissman Information...

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online