ex1-soln - Entropy, Relative Entropy and Mutual Information...

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon
Entropy, Relative Entropy and Mutual Information Exercises Exercise 2.1 : Coin Flips . A fair coin is Fipped until the ±rst head occurs. Let X denote the number of Fips required. (a) ²ind the entropy H ( X ) in bits. The following expressions may be useful: X n =1 r n = r 1 - r , X n =1 nr n = r (1 - r ) 2 (1) (b) A random variable X is drawn according to this distribution. ²ind an “e³cient” sequence of yes-no questions of the form, “Is X contained in the set S ?” Compare H ( X ) to the expected number of questions required to determine X . Solution : The probability for the random variable is given by P { X = i } = 0 . 5 i . Hence, H ( X ) = - X i p i log p i = - X i 0 . 5 i log(0 . 5 i ) = - log(0 . 5) X i i · 0 . 5 i = 0 . 5 (1 - 0 . 5) 2 = 2 (2) Exercise 2.3 : Minimum entropy . What is the minimum value of H ( p 1 , . . . , p n ) = H ( p ) as p ranges over the set of n -dimensional probability vectors? ²ind all p ’s which achieve this minimum. Solution : Since H ( p ) 0 and i p i = 1, then the minimum value for H ( p ) is 0 which is achieved when p i = 1 and p j = 0 , j 6 = i . Exercise 2.11 : Average entropy . Let H ( p ) = - p log 2 p - (1 - p ) log 2 (1 - p ) be the binary entropy function. (a) Evaluate H (1 / 4). (b) Calculate the average entropy H ( p ) when the probability p is chosen uniformly in the range 0 p 1. 1
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Solution : (a) H (1 / 4) = - 1 / 4 log 2 (1 / 4) - (1 - 1 / 4) log 2 (1 - 1 / 4) = 0 . 8113 (3) (b) ¯ H ( p ) = E [ H ( p )] = Z -∞ H ( p ) f ( p ) dp (4) Now, f ( p ) = ½ 1 , 0 p
Background image of page 2
Image of page 3
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 10/27/2010 for the course ECE 221 taught by Professor Sd during the Spring '10 term at Huston-Tillotson.

Page1 / 6

ex1-soln - Entropy, Relative Entropy and Mutual Information...

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online