Lecture7 - Colorado State University, Ft. Collins ECE 516:...

Info iconThis preview shows pages 1–4. Sign up to view the full content.

View Full Document Right Arrow Icon
1 Colorado State University, Ft. Collins Fall 2008 ECE 516: Information Theory Lecture 7 September 18, 2008 Recap: 3 The Asymptotic Equipartition Property (Chapter 3) Markov Inequality : For a nonnegative r.v., X and a constant 0 > c [] c X E c x P Chebyshev Inequality : For a r.v., Y and a constant 0 > c [ ] 2 c Y Var c Y E y P Sample Mean : Let n X X , , 1 L be n iid r.v.’s with mean X μ and variance 2 X σ . Then n X X Y n n + + = L 1 is a r.v. with mean X and variance n X 2 Weak Law of Large Numbers (WLLN) : X n n n X X Y + + = L 1 in probability i.e. 1 lim = < ε X n n y P for any 0 > Theorem (AEP) If L , , 2 1 X X are iid with ( ) X p , then () ( ) X H X X p n n , , log 1 1 L in probability Definition: The typical set ( ) n A with respect to ( ) x p is the set of sequences n n x x X , , 1 L with the following property: ( ) ( ) + X H n n X H n x x p 2 , , 2 1 L
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
2 Theorem: (Properties of the typical set) 1. If () n n A x x ε , , 1 L then ( ) + X H x x p n X H n , , log 1 1 L 2. [ ] > 1 n A P for sufficiently large n . 3. + X H n n A 2 4. X H n n A 2 1 Consequences of AEP: Data compression Let n X X , , 1 L be an iid sequence with ( ) x p and ( ) H x H = . We want to compress (find short description for) sequences ( ) n n x x X , , 1 L . There are X X log 2 n n = possible sequences. i) + H n 2 are in n A . Use ( ) 1 + + H n buts to index them. Prefix each with a 0. Total bits per sequence ( ) 2 + + = H n . ii) + H n n 2 1 X sequences not in ( ) n A . Just assume n X sequences remaining. Use 1 log + X n bits to index them and prefix each with a 1. Total bits per sequence 2 log + = X n Let ( ) n X l be the length of the codeword. It is random. ( ) [ ] + H n X l E n Where n 2 log + + = X , which can be made arbitrarily small.
Background image of page 2
3 Theorem: Let n X X , , 1 L be iid with ( ) x p and ( ) X H . Let 0 > ε . Then, there exists a code which maps sequences ( ) n x x , , 1 L or length n into binary strings (of variable lengths), such that the mapping is one-to-one (and therefore invertible) and () + X H n X l E n I.e., you can compress each symbol into ( ) + X H bits on average.
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 4
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 03/17/2010 for the course ECE 516 taught by Professor Rocky during the Spring '08 term at Colorado State.

Page1 / 9

Lecture7 - Colorado State University, Ft. Collins ECE 516:...

This preview shows document pages 1 - 4. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online