Lecture3 - Colorado State University, Ft. Collins ECE 516:...

Info iconThis preview shows pages 1–5. Sign up to view the full content.

View Full Document Right Arrow Icon
1 Colorado State University, Ft. Collins Fall 2008 ECE 516: Information Theory Lecture 3 September 2, 2008 Recap: 2.2 Joint Entropy and Conditional Entropy Definition: () ( ) = = ∑∑ ∈∈ Y X p E y x p y x p Y X H xy , 1 log , 1 log , , XY Definition: The conditional entropy ( ) X Y H | is ( ) () ( ) () ( ) = = = = = X Y p E x y p y x p x y p x y p x p x X Y H x p X Y H x | 1 log | 1 log , | 1 log | | | X - Chain Rules for Entropy Theorem: ( ) ( ) X Y H X H Y X H | , + = Theorem: Chain rule for more than 2 r.v.s ( ) = = n i i i n X X X H X X X H 1 1 1 2 1 , , | , , , L L Extension: ( ) ( ) Z X Y H Z X H Z Y X H , | | | , + = 2.3 Relative Entropy and Mutual Information Definition: Kullback-Leibler (KL) distance from ( ) x p to ( ) x q is ( ) ( ) ( ) = = X q X p E x q x p x p q p D p x log log || X Definition: The mutual information between two r.v.s X and Y
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
2 () ( ) ( ) ( ) ( ) ()() = = Y p X p Y X p E y p x p y x p D Y X I , log || , ; 2.4 Relationship between Entropy and Mutual Information ()( )( ) ( ) ( ) ( ) ( ) X Y H Y H Y X H X H Y X H Y H X H Y X I | | , ; = = + = Interpretation ( ) ( ) Y X I Y X H X H ; | =
Background image of page 2
3 2.6 Convexity, Jensen’s Inequality, and Its Consequences Definition: A function ( ) x f is convex over ( ) b a , if for every () b a x x , , 2 1 , 1 0 λ ( ) ( ) ( ) 2 1 2 1 1 1 x f x f x x f + + Strictly convex if inequality is strict for 2 1 x x , 0 , 1 , i.e., strictly convex if equality holds only if 0 = or 1 = . 1 x 2 x 2 1 1 x x + 1 x f ( ) 2 x f ()( )() 2 1 1 x f x f + 2 1 1 x x f + x f Not strictly convex. Examples: 2 x x f = , R x -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 0 0.5 1 1.5 2 2.5 3 3.5 4
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
4 () x x f log = , 0 > x 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 -1 0 1 2 3 4 5 6 7 b ax x f + = , R x -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -1 0 1 2 3 4 5 6 7 Definition: A function ( ) x f is concave over ( )
Background image of page 4
Image of page 5
This is the end of the preview. Sign up to access the rest of the document.

Page1 / 10

Lecture3 - Colorado State University, Ft. Collins ECE 516:...

This preview shows document pages 1 - 5. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online