hw3sol - EE 376B/Stat 376B Information Theory Prof. T....

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon
EE 376B/Stat 376B Handout #11 Information Theory Thursday, April 27, 2006 Prof. T. Cover Solutions to Homework Set #3 1. Maximum entropy discrete processes. (a) Find the maximum entropy rate binary stochastic process { X i } i = -∞ , X i ∈ { 0 , 1 } , satisfying Pr { X i = X i +1 } = 1 3 , for all i . (b) What is the resulting entropy rate? Solution: Maximum entropy discrete processes Our first hope may be an i.i.d. Bern( p ) process that is consistent with the constraint. Unfortunately, there is no such process. (Check!) However, we can still construct an independent non-identically distributed sequence of Bernoulli r.v.’s, such that the entropy rate exists, and the constraints are met. (For example, X i Bern(1) for odd i and X i Bern(1 / 3) for even i .) This process does not yield the maximum entropy rate. This problem is, in fact, a discrete (or more precisely, binary) analogue of Burg’s maximum entropy theorem and we can obtain the maximum entropy process from a similar argument. (a) Let X i be any binary process satisfying the constraint Pr { X i = X i +1 } = 1 / 3. Let Z i be a first order stationary Markov chain, that stays at 0 with probability 1/3, jumps to 1 with probability 2/3, and vice versa. This process obviously meets the constraint. With a slight abuse of notation, we have H ( X i | X i - 1 ) = E H ( X i | X i - 1 = x ) = E H (Pr( X i = x | X i - 1 = x )) H ( E Pr( X i = x | X i - 1 = x )) = H (1 / 3) = H ( Z i | Z i - 1 ) , where the inequallity follows from the concavity of the binary entropy function. 1
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Since H ( X 1 ) 1 = H ( Z 1 ), H ( X 1 ,...,X n ) = H ( X 1 ) + n X i =2 H ( X i | X i - 1 ) H ( X 1 ) + n X i =2 H ( X i | X i - 1 ) H ( Z 1 ) + n X i =2 H ( Z i | Z i - 1 ) = H ( Z 1 ,...,Z n ) , whence { Z i } is the maximum entropy process under the given constraint. (b) The maximum entropy rate
Background image of page 2
Image of page 3
This is the end of the preview. Sign up to access the rest of the document.

Page1 / 6

hw3sol - EE 376B/Stat 376B Information Theory Prof. T....

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online