1
Colorado State University, Ft. Collins
Fall 2008
ECE 516: Information Theory
Lecture 4 September 9, 2008
Recap:
2.6
Convexity, Jensen’s Inequality
Definition:
()
x
f
is convex over
( )
b
a
,
if for every
( )
b
a
x
x
,
,
2
1
∈
,
1
0
≤
≤
λ
( ) ( ) ( )
2
1
2
1
1
1
x
f
x
f
x
x
f
−
+
≤
−
+
Strictly convex if inequality is strict for
2
1
x
x
≠
,
0
≠
,
1
≠
, i.e., strictly
convex if equality holds only if
0
=
or
1
=
.
Definition:
x
f
is concave over
( )
b
a
,
if
( )
x
f
−
is convex
If it is twice differentiable
0
2
2
≥
dx
x
f
d
convex
( )
0
2
2
≤
dx
x
f
d
concave
Theorem:
Let
0
,
,
1
≥
n
p
p
L
such that
1
1
=
∑
=
n
i
i
p
. If
( )
x
f
is convex, then for any
n
x
x
,
,
1
L
∑
∑
=
=
≤
⎟
⎠
⎞
⎜
⎝
⎛
n
i
i
i
n
i
i
i
x
f
p
x
p
f
1
1
Jensen’s Inequality:
If
f
is a convex and
X
is a r.v., then
[]
[ ]
( )
X
E
f
X
f
E
≥
Moreover, if
f
is strictly convex, then equality implies that
X
E
X
=
, w.p. 1, i.e.,
X
is constant (deterministic).
Theorem:
(Information Inequality) Let
( )
x
p
,
( )
x
q
be two PMFs on
X
. Then
0
||
≥
q
p
D
with equality iff
() ()
x
q
x
p
=
for all
x
.