2-AEP - Harvard SEAS ES250 Information Theory Asymptotic...

Info icon This preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon
Harvard SEAS ES250 – Information Theory Asymptotic Equipartition Property (AEP) and Entropy rates * 1 Asymptotic Equipartition Property 1.1 Preliminaries Definition (Convergence of random variables) We say that a sequence of random variables X 1 , X 2 , . . . , converges to a random variable X : 1. In probability if for every ² > 0, Pr {| X n - X | > ² } → 0 2. In mean square if E [( X - X n ) 2 ] 0 3. With probability 1 (also called almost surely) if Pr { lim n →∞ X n = X } = 1 1.2 Asymptotic Equipartition Property Theorem Theorem (AEP) If X 1 , X 2 , . . . iid p ( x ), then - 1 n log p ( X 1 , X 2 , . . . , X n ) H ( X ) in probability . Definition The typical set A ( n ) ² with respect to p ( x ) is the set of sequences ( x 1 , x 2 , . . . , x n ) ∈ X n with the property 2 - n ( H ( X )+ ² ) p ( x 1 , x 2 , . . . , x n ) 2 - n ( H ( X ) - ² ) . Theorem 1. If ( x 1 , x 2 , . . . , x n ) A ( n ) ² , then H ( X ) - ² ≤ - 1 n log p ( x 1 , x 2 , . . . , x n ) H ( X ) + ² . 2. Pr { A ( n ) ² } > 1 - ² for n sufficiently large. 3. | A ( n ) ² | ≤ 2 n ( H ( X )+ ² ) 4. | A ( n ) ² | ≥ (1 - ² )2 n ( H ( X ) - ² ) The typical set has probability nearly one, cardinality nearly 2 nH , and nearly equiprobable elements. 1.3 Consequences of the AEP A Coding Scheme Recall our definition of A ( n ) ² as the typical set with respect to p ( x ). Can we assign codewords such that sequences ( x 1 , x 2 , . . . , x n ) ∈ X n can be represented using nH bits on average? Let x n denote ( x 1 , x 2 , . . . , x n ), and let l ( x n ) be the length of the codeword corresponding to x n . If n is sufficiently large so that Pr { A ( n ) ² } ≥ 1 - ² , the expected codeword length is E [ l ( X n )] = X x n p ( x n ) l ( x n ) n ( H + ² ) + ²n (log |X| ) + 2 = n ( H + ² 0 ) * Based on Cover & Thomas, Chapter 3,4 1
Image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Harvard SEAS ES250 – Information Theory Data Compression Theorem Let X n iid p ( x ) and let ² > 0. Then there exists a code that maps sequences x n of length n
Image of page 2
Image of page 3
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern