This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: ECE 255AN Fall 2011 Homework set 3  solutions 1 Entropy of P is 7 / 4. Hence T 4 . 25 contains all sequences of length four with probabilities between 2 6 and 2 8 . All sequences except five sequences satisfy this property. The five sequences are (1 / 2 , 1 / 2 , 1 / 2 , 1 / 2) and (1 / 2 , 1 / 2 , 1 / 2 , 1 / 4) and their permutations. Hence  T 4 . 25  = 251 and Pr ( T 4 . 25 ) = 13 / 16. 2 a) T n inf is the set of all possible sequences. T n is the set of all sequences with probability 2 nH ( X ) . Two of them are equal, when all sequences have same probability. This is possible if and only if the distribution is uniform. b) T n 1 is the set of all sequences with probability less than 2 n nH ( P ) and greater than 2 n nH ( P ) . Let p min be the lowest probability of an alphabet and p max be that of the highest. Highest probability of a sequence is p n max and the lowest probability is p n min . Therefore for the given condition, it is necessary that p n max ≤ 2 1 nH ( P ) and p n min ≥ 2 n nH ( P ) . Rearranging, p max ≤ 2 1 H ( P ) p min ≥ 2 1 H ( P ) 3 a) We prove by induction that Pr ( X i = 1) = 1 2 for all i . Clearly it is true for i = 1 , 2. Assume its true for n ≥ 2. For n + 1, Pr ( X n +1 ) = 1) = 1 n n X i =1 Pr ( X i ) = 1) = 1 n n 2 = 1 2 Hence by induction, Pr ( X i = 1) = 1 2 for all i , therefore H ( X i ) = 1. b) lim n →∞ H ( X n 1 ) n = lim n →∞ H ( X n  X n 1 1 ) ≤ H ( X n  X 2 1 ), as conditioning increases the en tropy....
View
Full Document
 Fall '09
 Information Theory, Entropy, 1 k, 2 m, Xn

Click to edit the document details