EE596A Introduction to Information Theory
University of Washington
Winter 2004
Dept. of Electrical Engineering
Handout 3: Problem Set 1, Due Wed, Jan 21st
Prof: Jeff A. Bilmes
<bilmes@ee.washington.edu>
Lecture 3, Jan 12, 2004
3.1
Problems from Text
Do problems 2.16, 2.3, 2.5, 2.6, 2.12, 2.10, 2.20 in Cover and Thomas.
3.2
Additional Problems
Problem 0.
The rest of Jensen.
In class, we proved part of Jensen’s inequality. Specifically, we proved that if
f
is convex and
X
is a random variable,
then
Ef
(
X
)
≥
f
(
EX
)
.
We also mentioned but did not prove that if
f
is strictly convex, then if
Ef
(
X
) =
f
(
EX
)
, then
X
=
EX
meaning
that
X
is a constant.
In this problem, your task is to proof this second condition.
Problem 1.
Matlab, and entropy of images.
Consider a probability distribution
p
(
X
)
over a
32
×
32
pixel image, where
X
is a
32
×
32
random matrix. Assume,
for this discussion, that there are 8 bit pixels which means that in total there are a large number
256
32
×
32
= 10
2466
possible images that are representable (this is more than the number of atoms in the universe which is estimated to be
about
10
78
, and certainly more than the entire human population has viewed throughout the course of its existence).
1
When
p
(
X
)
is a uniform distribution, then when we sample from such a distribution, the images would look entirely
random — the likelihood of ever encountering an “real” image (say of birds, bees, trees, or industrial manufacturing
plants), one we would likely see in our world, is exceedingly small. You can create such images in matlab with the
following command
imagesc(rand(32,32))
. Note that the probability of seeing the image that you see is the
same as that of seeing an image of yourself, when generating such image samples.
Since all the images are equally likely, the entropy of such a distribution is