Lecture17

Lecture17 - Colorado State University, Ft. Collins ECE 516:...

This preview shows pages 1–3. Sign up to view the full content.

1 Colorado State University, Ft. Collins Fall 2008 ECE 516: Information Theory Lecture 17 October 28, 2008 Recap: For crit R R , it can be shown that () ( ) () R E P n r n e n log 1 lim . Therefore, ( ) R E r is the maximum achievable error exponent for crit R R . 8 Differential Entropy (Chapter 8) Definition: The differential entropy ( ) X h of a continuous r.v. X with PDF x f is defined as ( ) () () () = = S S dx x f x f dx x f x f X h 1 log log Theorem: Let n X X X , , , 2 1 L be iid r.v.’s with PDF ( ) x f . Then, ( ) [] X h x f E x x f n n = log , , log 1 1 L in prob. Definition: The typical set ( ) n A ε w.r.t. ( ) x f is ( ) = X h x x f n S x x A n n n n , , log 1 : , , 1 1 L L Where ( ) = = n i i n x f x x f 1 1 , , L Definition: The volume ( ) A Vol of a set n R A is defined as = A n dx dx dx A L 2 1 Vol Theorem: The typical set has the following properties. 1. [ ] > 1 n A P , for sufficiently large n . 2. ( ) + X h n n A 2 Vol for sufficiently large n .

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
2 3. () ( ) ε X h n n A 2 1 Vol for sufficiently large n . Joint and conditional differential entropies Definition: Joint differential entropy ( ) ( )( ) = S dxdy y x f y x f Y X h , log , , Conditional differential entropy, ( ) ( ) = S dxdy y x f y x f Y X h | log , | And ( ) ( ) Y h Y X h Y X h = , | Relative entropy and mutual information Definition: The relative entropy (Kullback Leibler distance) between two PDFs x f , x g is = g f f g f D log || Definition: The mutual information between two r.v.’s X and Y is ( ) ( ) ( ) ( ) ( ) ( ) Y h X h Y X h y f x f y x f D Y X I = = , || , ; Theorem: 0 || g f D with equality holds iff g f = almost everywhere (a.e.) Corollary: 0 ; Y X I with equality hold iff X , Y are independent. Corollary: ( ) X h Y X h | with equality hold iff X , Y are independent.
This is the end of the preview. Sign up to access the rest of the document.

Lecture17 - Colorado State University, Ft. Collins ECE 516:...

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online