ITCSolution2

ITCSolution2 - achieved when p 10 = p 01 = 1 / 2, in which...

Info iconThis preview shows pages 1–5. Sign up to view the full content.

View Full Document Right Arrow Icon
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Background image of page 2
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Background image of page 4
2. Problem 3.5 (a) Since X 1 ,...,X n are i.i.d, so are q ( X 1 ) ,...,q ( X n ). Then, applying the strong law of large numers, lim - 1 n log q ( X 1 ,...,X n ) = lim - 1 n n s i =1 log q ( X i ) = - E { log q ( X ) } = - s p ( x ) log q ( x ) = s p ( x ) log p ( x ) q ( x ) - s p ( x ) log p ( x ) = D ( p || q ) + H ( p ) (b) Again, by the strong law of large numbers, lim - 1 n log q ( X 1 ,...,X n ) p ( X 1 ,...,X n ) = lim - 1 n n s i =1 log q ( X i ) p ( X i ) = - E b log q ( X ) p ( X ) B = - s p ( x ) log q ( x ) p ( x ) = D ( p || q ) 3. Problem 4.2 By chain rule of entropy and stationarity assumption, H ( X 0 | X - 1 ,...,X - n ) = H ( X 0 ,X - 1 ,...,X - n ) - H ( X - 1 ,...,X - n ) = H ( X 0 ,X 1 ,...,X n ) - H ( X 1 ,...,X n ) = H ( X 0 | X 1 ,...,X n ) 4. Problem 4.5 (a) The stationary distribution of the Markov chain is given by π 0 = p 10 p 01 + p 10 , π 1 = p 01 p 01 + p 10 . Therefore, the entropy rate is H ( X ) = H ( X 2 | X 1 ) = π 0 H ( p 01 ) + π 1 H ( p 10 ) . (b) The entropy rate H ( X ) is at most 1, since the process has only two states. This entropy rate is
Background image of page 5
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: achieved when p 10 = p 01 = 1 / 2, in which case the process is actually i.i.d. (c) The stationary distribution of the Markov chain is given by = 1 p + 1 , 1 = p p + 1 . For this two-state Markov chain, H ( X ) = H ( X 2 | X 1 ) = H ( p ) + 1 H (1) = H ( p ) / (1 + p ) . (d) Dierentiating H ( X ) = H ( p ) / (1 + p ), we get p max = (3- 5) 2 and H ( X ) = log 5+1 2 . 2...
View Full Document

Page1 / 5

ITCSolution2 - achieved when p 10 = p 01 = 1 / 2, in which...

This preview shows document pages 1 - 5. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online