Lecture4 - Colorado State University, Ft. Collins ECE 516:...

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon
1 Colorado State University, Ft. Collins Fall 2008 ECE 516: Information Theory Lecture 4 September 9, 2008 Recap: 2.6 Convexity, Jensen’s Inequality Definition: () x f is convex over ( ) b a , if for every ( ) b a x x , , 2 1 , 1 0 λ ( ) ( ) ( ) 2 1 2 1 1 1 x f x f x x f + + Strictly convex if inequality is strict for 2 1 x x , 0 , 1 , i.e., strictly convex if equality holds only if 0 = or 1 = . Definition: x f is concave over ( ) b a , if ( ) x f is convex If it is twice differentiable 0 2 2 dx x f d convex ( ) 0 2 2 dx x f d concave Theorem: Let 0 , , 1 n p p L such that 1 1 = = n i i p . If ( ) x f is convex, then for any n x x , , 1 L = = n i i i n i i i x f p x p f 1 1 Jensen’s Inequality: If f is a convex and X is a r.v., then [] [ ] ( ) X E f X f E Moreover, if f is strictly convex, then equality implies that X E X = , w.p. 1, i.e., X is constant (deterministic). Theorem: (Information Inequality) Let ( ) x p , ( ) x q be two PMFs on X . Then 0 || q p D with equality iff () () x q x p = for all x .
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
2 Corollary: (Non-negativity of Mutual Information) For any two random variables X , Y , () 0 ; Y X I with equality iff X and Y are independent. Theorem: For any random variable X ( ) X log X H With equality iff X has a uniform distribution over X . Theorem: (conditioning reduces entropy) For any two random variables X , Y ( ) X H Y X H | with equality when X and Y are independent. Theorem: (independence bound on entropy) For r.v.’s n X X , , 1 L ( ) = n i i n X H X X H 1 1 , , L with equality iff all i X are independent. 2.7 Log-sum inequality and its applications Theorem: For non-negative numbers n a a , , 1 L and n b b , , 1 L = = = = n i i n i i n i i n i i i i b a a b a a 1 1 1 1 log log
Background image of page 2
Image of page 3
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 03/17/2010 for the course ECE 516 taught by Professor Rocky during the Spring '08 term at Colorado State.

Page1 / 8

Lecture4 - Colorado State University, Ft. Collins ECE 516:...

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online