2-AEP - Harvard SEAS ES250 Information Theory Asymptotic...

Info iconThis preview shows pages 1–2. Sign up to view the full content.

View Full Document Right Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Harvard SEAS ES250 Information Theory Asymptotic Equipartition Property (AEP) and Entropy rates * 1 Asymptotic Equipartition Property 1.1 Preliminaries Definition (Convergence of random variables) We say that a sequence of random variables X 1 , X 2 , . . . , converges to a random variable X : 1. In probability if for every > 0, Pr {| X n- X | > } 2. In mean square if E [( X- X n ) 2 ] 3. With probability 1 (also called almost surely) if Pr { lim n X n = X } = 1 1.2 Asymptotic Equipartition Property Theorem Theorem (AEP) If X 1 , X 2 , . . . iid p ( x ), then- 1 n log p ( X 1 , X 2 , . . . , X n ) H ( X ) in probability . Definition The typical set A ( n ) with respect to p ( x ) is the set of sequences ( x 1 , x 2 , . . . , x n ) X n with the property 2- n ( H ( X )+ ) p ( x 1 , x 2 , . . . , x n ) 2- n ( H ( X )- ) . Theorem 1. If ( x 1 , x 2 , . . . , x n ) A ( n ) , then H ( X )- - 1 n log p ( x 1 , x 2 , . . . , x n ) H ( X ) + . 2. Pr { A ( n ) } > 1- for n sufficiently large. 3. | A ( n ) | 2 n ( H ( X )+ ) 4. | A ( n ) | (1- )2 n ( H ( X )- ) The typical set has probability nearly one, cardinality nearly 2 nH , and nearly equiprobable elements. 1.3 Consequences of the AEP A Coding Scheme Recall our definition of A ( n ) as the typical set with respect to p ( x ). Can we assign codewords such that sequences ( x 1 , x 2 , . . . , x n ) X n can be represented using nH bits on average? Let x n denote ( x 1 , x 2 , . . . , x n ), and let l ( x n ) be the length of the codeword corresponding to x n . If n is sufficiently large so that Pr { A ( n ) } 1- , the expected codeword length is E [ l ( X n )] = X x n p ( x n ) l ( x n ) n ( H + ) + n (log |X| ) + 2 = n ( H + ) * Based on Cover & Thomas, Chapter 3,4...
View Full Document

This note was uploaded on 12/01/2010 for the course ADLAC 1023 at Stanford.

Page1 / 4

2-AEP - Harvard SEAS ES250 Information Theory Asymptotic...

This preview shows document pages 1 - 2. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online