# Module2_3 - Module 2 Lecture 3 Fundamental Concepts...

This preview shows pages 1–8. Sign up to view the full content.

Module 2, Lecture 3 Fundamental Concepts: Information Inequalities I G.L. Heileman Module 2, Lecture 3

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
Convex Functions Many useful inequalities in information theory make use of the notion of convexity. Deﬁnition (Convex) A real function f deﬁned on ( a , b ) ⊂ < is called convex if the inequality f ( λ x 1 + (1 - λ ) x 2 ) λ f ( x 1 ) + (1 - λ ) f ( x 2 ) holds whenever a < x 1 , x 2 < b , and 0 λ 1. The function f is called strictly convex if equality holds only if λ = 0 or λ = 1. G.L. Heileman Module 2, Lecture 3
Convex Functions Graphically, convexity implies that if x 1 < y < x 2 , then the point ( y , f ( y )) should lie on or below the line connecting the points ( x , f ( x )) and ( x 2 , f ( x 2 )): f(x) x y 12 x x ( ) f( ) , f( ) 2 x , ( 2 x ) ( x 1 f( ) x 1 , ) y y l l 1 2 G.L. Heileman Module 2, Lecture 3

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
Convex Functions Thus, the inequality in the deﬁnition of convexity is equivalent to the requirement that: f ( y ) - f ( x 1 ) y - x 1 f ( x 2 ) - f ( y ) x 2 - y whenever a < x 1 < Y < x 2 < b . I.e., in the previous ﬁgure, this means that the slope of l 2 must be than the slope of l 1 . Recall the Mean Value Theorem (for derivatives): Suppose that f is a function that is continuous for a x b and has a derivative for a < x < b . Then there is at least one number c , with a < c < b , such that f ( b ) - f ( a ) b - a = f 0 ( c ) Taken together, these previous two results imply that: if f ( x ) is convex (strictly convex), then f 0 ( x ) must be monotonically (strictly) increasing with x . This is equivalent to the condition that if f 00 ( x ) is non-negative (positive) everywhere, f ( x ) is convex (strictly convex). G.L. Heileman Module 2, Lecture 3
Convex Functions Ex. f ( x ) = e x , f 00 ( x ) = e x the exponential function is convex over ( -∞ , ). f ( x ) = x ln x , f 00 ( x ) = 1 x x ln x is convex over (0 , ). f ( x ) = ln x , f 00 ( x ) = - 1 x 2 ln x is not convex. G.L. Heileman Module 2, Lecture 3

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
Concave Functions Deﬁnition (Convex) A real function f concave if - f is convex. Ex. f ( x ) = - ln x , f 00 ( x ) = 1 x 2 ln x is concave over (0 , ). G.L. Heileman Module 2, Lecture 3
Jensen’s Inequality Theorem (Jensen’s inequality) If f is a convex function and X a discrete RV, then E [ f ( x )] f ( E [ X ]) Proof: (by induction) Induction hypothesis: the theorem is true for distributions containing k - 1 mass points.

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

### Page1 / 24

Module2_3 - Module 2 Lecture 3 Fundamental Concepts...

This preview shows document pages 1 - 8. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online