Lecture6 - Colorado State University, Ft. Collins ECE 516:...

Info iconThis preview shows pages 1–4. Sign up to view the full content.

View Full Document Right Arrow Icon
1 Colorado State University, Ft. Collins Fall 2008 ECE 516: Information Theory Lecture 6 September 16, 2008 Recap: Theorem: (data processing inequality) If Z Y X , then () Z X I Y X I ; ; Corollary: In particular, if ( ) Y g Z = , then ( ) ( ) ( ) Y g X I Y X I ; ; Corollary: If Z Y X ( ) ( ) Z X H Y X H | | Corollary: If Z Y X , then ( ) ( ) Z Y X I Y X I | ; ; 2.10 Fano’s Inequality Theorem: (Fano’s Inequality) ( ) ( ) ( ) Y X H P P H e e | 1 log + X Fano’s inequality lower-bounds e P 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 ( ) X log e e P P H + ( ) Y X H | Minimum e P Theorem: If 0 | = Y X H , then X is a function of Y . Corollary: (Relaxation of Fano’s Inequality) X log 1 | Y X H P e
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
2 Discussions: Let us use the precise version of Fano’s inequality () ( ) Y X H P P H e e | 1 log + X Let () () ( ) 1 log + = X e e e P P H P f . What if ( ) Y X H | crosses e P f at two points? 1. If I do want to minimize probability of error, what is the maximum possible value of e P ? If X is uniformly distributed, I do not know any more information. I make a random guess, the probability of error is X X 1 = e P . (If X is not uniformly distributed, I can do better.) 2. What is the maximum of ( ) e P f ? 1 log 1 1 log 1 1 log 1 log + + = + = X X e e e e e e e e P P P P P P P H P f 1 log 1 log + = X e e e e P P dP P f d X X X 1 1 log 1 log 0 = = = e e e e e P P P dP P f d Therefore, if Y X H | crosses ( ) e P f at two points, only the lower e P value is meaningful. 3 The Asymptotic Equipartition Property (Chapter 3) Mathematical Preliminaries Markov Inequality : For a nonnegative r.v., X and a constant 0 > c [] c X E c x P Proof:
Background image of page 2
3 [] () () () () () [] c x cP dx x f c dx x xf dx x xf dx x xf dx x xf X E c c c c = + = = 0 0 Thus, [ ] c X E c x P Chebyshev Inequality : For a r.v., Y and a constant 0 > c [ ] 2 c Y Var c Y E y P
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 4
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 03/17/2010 for the course ECE 516 taught by Professor Rocky during the Spring '08 term at Colorado State.

Page1 / 8

Lecture6 - Colorado State University, Ft. Collins ECE 516:...

This preview shows document pages 1 - 4. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online