Module2_3 - Module 2, Lecture 3 Fundamental Concepts:...

Info iconThis preview shows pages 1–8. Sign up to view the full content.

View Full Document Right Arrow Icon
Module 2, Lecture 3 Fundamental Concepts: Information Inequalities I G.L. Heileman Module 2, Lecture 3
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Convex Functions Many useful inequalities in information theory make use of the notion of convexity. Definition (Convex) A real function f defined on ( a , b ) ⊂ < is called convex if the inequality f ( λ x 1 + (1 - λ ) x 2 ) λ f ( x 1 ) + (1 - λ ) f ( x 2 ) holds whenever a < x 1 , x 2 < b , and 0 λ 1. The function f is called strictly convex if equality holds only if λ = 0 or λ = 1. G.L. Heileman Module 2, Lecture 3
Background image of page 2
Convex Functions Graphically, convexity implies that if x 1 < y < x 2 , then the point ( y , f ( y )) should lie on or below the line connecting the points ( x , f ( x )) and ( x 2 , f ( x 2 )): f(x) x y 12 x x ( ) f( ) , f( ) 2 x , ( 2 x ) ( x 1 f( ) x 1 , ) y y l l 1 2 G.L. Heileman Module 2, Lecture 3
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Convex Functions Thus, the inequality in the definition of convexity is equivalent to the requirement that: f ( y ) - f ( x 1 ) y - x 1 f ( x 2 ) - f ( y ) x 2 - y whenever a < x 1 < Y < x 2 < b . I.e., in the previous figure, this means that the slope of l 2 must be than the slope of l 1 . Recall the Mean Value Theorem (for derivatives): Suppose that f is a function that is continuous for a x b and has a derivative for a < x < b . Then there is at least one number c , with a < c < b , such that f ( b ) - f ( a ) b - a = f 0 ( c ) Taken together, these previous two results imply that: if f ( x ) is convex (strictly convex), then f 0 ( x ) must be monotonically (strictly) increasing with x . This is equivalent to the condition that if f 00 ( x ) is non-negative (positive) everywhere, f ( x ) is convex (strictly convex). G.L. Heileman Module 2, Lecture 3
Background image of page 4
Convex Functions Ex. f ( x ) = e x , f 00 ( x ) = e x the exponential function is convex over ( -∞ , ). f ( x ) = x ln x , f 00 ( x ) = 1 x x ln x is convex over (0 , ). f ( x ) = ln x , f 00 ( x ) = - 1 x 2 ln x is not convex. G.L. Heileman Module 2, Lecture 3
Background image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Concave Functions Definition (Convex) A real function f concave if - f is convex. Ex. f ( x ) = - ln x , f 00 ( x ) = 1 x 2 ln x is concave over (0 , ). G.L. Heileman Module 2, Lecture 3
Background image of page 6
Jensen’s Inequality Theorem (Jensen’s inequality) If f is a convex function and X a discrete RV, then E [ f ( x )] f ( E [ X ]) Proof: (by induction) Induction hypothesis: the theorem is true for distributions containing k - 1 mass points.
Background image of page 7

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 8
This is the end of the preview. Sign up to access the rest of the document.

Page1 / 24

Module2_3 - Module 2, Lecture 3 Fundamental Concepts:...

This preview shows document pages 1 - 8. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online