{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

hw4sol - EE 376B Information Theory Prof T Cover Handout#11...

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon
EE 376B Handout #11 Information Theory Tuesday, May 17, 2011 Prof. T. Cover Prepared by T.A. Gowtham Kumar Solutions to Homework Set #4 1. Maximum Entropy and Counting Let X = { 1 , 2 , . . . , m } . Show that the number of sequences x n ∈ X n satisfying 1 n n i =1 g ( x i ) α is approximately equal to 2 nH * , to first order in the exponent, for n sufficiently large, where H * = max P : P ( i ) g ( i ) α H ( P ) . Solution: Maximum Entropy and Counting The key here is that there are only a polynomial number of types whereas there are exponential number of sequences of each type. Let P * be the entropy-maximizing distribution. Lower Bound: Consider the rational type with denominator n P * n , satisfying the given constraint, that minimizes | P * n - P * | . There are atleast 2 n ( H ( P * n ) - 0 n ) sequences of type P * n . Also, as n → ∞ , | P * n - P * | = 0. Since H ( p ) is a continuous function of p , it follows that | H ( P * n ) - H ( P * ) | → 0. Hence, we conclude there are atleast 2 n ( H ( P * ) - n ) sequences satisfying the constraint. (where n 0 as n → ∞ ). Upper Bound: There are only a polynomial number of types. Thus the total number of types is less than a polynomial, ( n + 1) m . Therefore, the total number of sequences is upper bounded by X P n : E P n g ( X ) α 2 n ( H ( P n )+ 0 n ) ( n + 1) m 2 n ( H ( P * )+ 0 n ) 2 n 00 n 2 n ( H ( P * )+ 0 n ) = 2 n ( H ( P * )+ n ) This completes the proof. 2. Counting states Suppose a die X takes on values in { 1 , 2 , . . . , 6 } and EX = 5. 1
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
(a) Find the maximum entropy pdf p * ( x ) subject to this condition. (b) Let X 1 , X 2 , . . . be i.i.d. p * ( x ). It is now observed that the frequency of occurrence of 1 is twice the frequency of occurrence of 6. Assume n large, find the conditional distribution of the state of the first die X 1 , given this observation. Solution: Counting states (a) The maximum entropy distribution is given by p * ( x ) = exp( λ 0 + λ 1 x ) = exp( λ 1 x ) x 0 exp( λ 1 x 0 ) We determine the constant λ 1 such that the constraint EX = 5 is satisfied, i.e. We solve 6 x 0 =1 x 0 exp( λ 1 x 0 ) 6 x 0 =1 exp( λ 1 x 0 ) = 5 . Using a numerical solver (for instance fsolve on Matlab), we find the solution λ 1 = 0 . 6296, which corresponds to the distribution p * = 0 . 0205 0 . 0385 0 . 0723 0 . 1357 0 . 2548 0 . 4781 (b) According to the Conditional Limit theorem, the most likely distribution is the distribution q * that minimizes D ( q || p * ) over all q that satisfies the given con- straint. As in the previous homework, we express the linear constraint on q ( x ) as an expected value constraint. We have q (1) = 2 q (6) or q (1) - 2 q (6) = 0. This can be written as Eg ( X ) = 0 where g (1) = 1, g (6) = - 2 and g ( i ) = 0 for i = 2 , 3 , 4 , 5. The theorem also tells how to compute the distribution for an expected value constraint: q * ( x ) = p * ( x ) exp( μg ( x ) x 0 p * ( x 0 ) exp ( μg ( x 0 )) We can substitute for p * ( x ) from part (a) to obtain q * ( x ) = exp( λx ) exp( μg ( x )) x 0 exp( λx 0 ) exp( μg ( x 0 )) = exp( λx + μg ( x )) x 0 exp( λx 0 + μg ( x 0 )) where λ is obtained in part (a) and μ
Background image of page 2
Image of page 3
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}