2006final

# 2006final - EE 376A/Stat 376A Prof. T. Weissman Information...

This preview shows pages 1–3. Sign up to view the full content.

EE 376A/Stat 376A Information Theory Prof. T. Weissman Thursday, March 23, 2006 Final 1. ( 20 points ) Three Shannon codes Let { U i } i 1 be a stationary 1st-order Markov source whose alphabet size is r . Note that the stationarity property implies that P ( u i ) ,P ( u i | u i - 1 ) does not depend on i . Assume that - log P ( u i ) and - log P ( u i | u i - 1 ) are integers for all ( u i ,u i - 1 ). Three students decided to use the Shannon coding scheme, based on diﬀerent pmfs as follows: Alice suggested to code each symbol separately, i.e., code by using the pmf { P ( u i ) } . Bob suggested to code in pairs by using the joint pmf { P ( u i ,u i +1 ) } ,i = 1 , 3 , 5 ,... . In other words, ﬁrst code ( U 1 ,U 2 ), then code ( U 3 ,U 4 ), and so on. Charlie suggested to code each symbol given the previous symbol by using the conditional pmf { P ( u i | u i - 1 ) } . In other words, ﬁrst code U 1 , then code U 2 given U 1 , then code U 3 given U 2 , and so on. (a) If the source is memoryless, compare the expected codeword length per symbol, i.e., 1 n E‘ ( U n ), of each scheme, assuming n > 2 is even. (b) Repeat Part (a) for the case where the source has memory. Solutions (a) Because the source is memoryless Alice’s and Charlie’s coding schemes are identical. In addition it is given that - log P ( u i ) is an integer and hence Shanon coding for block length 1 has an average length of H ( U 1 ) = u 1 P ( u 1 )log( P ( u 1 )). Bob’s coding scheme is over pairs that their log of the PMF is also an integer, and therefore the average length of his code is 1 2 H ( U 1 ,U 2 ) and because of the memoryless property 1 2 H ( U 1 ,U 2 ) = H ( U 1 ). The conclusion is that all three codes have the same average length. (b) Alice’s Average code is 1 n E‘ ( U n ) = 1 n " n X i =1 X u i P ( u i )log P ( u i ) # = H ( U 1 ) 1

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
1 n E‘ ( U n ) = 1 n n/ 2 X i =1 X u 2 i ,u 2 i - 1 P ( u 2 i ,u 2 i - 1 )log P ( u 2 i ,u 2 i - 1 ) a = 1 n n/ 2 X i =1 X u 2 ,u 1 P ( u 2 ,u 1 )log P ( u 2 ,u 1 ) b = 1 2 [ H ( U 2 ,U 1 )] Charlie’s average code is 1 n E‘ ( U n ) = 1 n " X u 1 P ( u 1 )log P ( u 1 ) + n X i =2 X u i P ( U i - 1 P ( u i | u i - 1 )log P ( u i | u i - 1 ) # a = 1 n " X u 1 P ( u 1 )log P ( u 1 ) + ( n - 1) X u 2 P ( U 1 P ( u 2 | u 1 )log P ( u 2 | u 1 ) # b = 1 n [ H ( U 1 ) + ( n - 1) H ( U 2 | U 1 )] Equality (a) follows the stationary property and equality (b) follows the deﬁnition of entropy. Alice’s average code is the longest because H ( U 2 ) H ( U 2 | U 1 ). For n = 2 Bob’s and Charlie’s code are the same but for n 4 Bob’s average code is longer because H ( U 2 ,U 1 ) = H ( U 1 ) + H ( U 2 | U 1 ) 2 H ( U 2 | U 1 ) where U is a stationary source. Notice that equality holds iﬀ the source is memoryless. 2
This is the end of the preview. Sign up to access the rest of the document.

## This note was uploaded on 12/01/2010 for the course ADLAC 1023 at Stanford.

### Page1 / 11

2006final - EE 376A/Stat 376A Prof. T. Weissman Information...

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online