hw1 - EE596A Introduction to Information Theory Winter 2004...

Info iconThis preview shows pages 1–2. Sign up to view the full content.

View Full Document Right Arrow Icon
EE596A Introduction to Information Theory University of Washington Winter 2004 Dept. of Electrical Engineering Handout 3: Problem Set 1, Due Wed, Jan 21st Prof: Jeff A. Bilmes <bilmes@ee.washington.edu> Lecture 3, Jan 12, 2004 3.1 Problems from Text Do problems 2.16, 2.3, 2.5, 2.6, 2.12, 2.10, 2.20 in Cover and Thomas. 3.2 Additional Problems Problem 0. The rest of Jensen. In class, we proved part of Jensen’s inequality. Specifically, we proved that if f is convex and X is a random variable, then Ef ( X ) f ( EX ) . We also mentioned but did not prove that if f is strictly convex, then if Ef ( X ) = f ( EX ) , then X = EX meaning that X is a constant. In this problem, your task is to proof this second condition. Problem 1. Matlab, and entropy of images. Consider a probability distribution p ( X ) over a 32 × 32 pixel image, where X is a 32 × 32 random matrix. Assume, for this discussion, that there are 8 bit pixels which means that in total there are a large number 256 32 × 32 = 10 2466 possible images that are representable (this is more than the number of atoms in the universe which is estimated to be about 10 78 , and certainly more than the entire human population has viewed throughout the course of its existence). 1 When p ( X ) is a uniform distribution, then when we sample from such a distribution, the images would look entirely random — the likelihood of ever encountering an “real” image (say of birds, bees, trees, or industrial manufacturing plants), one we would likely see in our world, is exceedingly small. You can create such images in matlab with the following command imagesc(rand(32,32)) . Note that the probability of seeing the image that you see is the same as that of seeing an image of yourself, when generating such image samples. Since all the images are equally likely, the entropy of such a distribution is
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 2
This is the end of the preview. Sign up to access the rest of the document.

Page1 / 3

hw1 - EE596A Introduction to Information Theory Winter 2004...

This preview shows document pages 1 - 2. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online