This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: ECE 534: Elements of Information Theory, Fall 2010 Homework 1 Solutions Ex. 2.1 (Davide Basilio Bartolini) Text Coin Flips . A fair coin is flipped until the first head occurs. Let X denote the number of flips required. (a) Find the Entropy H(X) in bits (b) A random variable X is drawn according to this distribution. Find an “efficient” sequence of yesno questions of the form, “Is X contained in the set S?”. Compare H(X) to the expected number of questions required to determine X. Solution (a) The random variable X is on the domain X = { 1 , 2 , 3 ,... } and it denotes the number of flips needed to get the first head, i.e. 1 + the number of consecutive tails appeared before the first head. Since the coin is said to be fair, we have p (“ head ”) = p (“ tail ”) = 1 2 and hence (exploiting the independence of the coin flips): p ( X = 1) = p (“ head ”) = 1 2 p ( X = 2) = p (“ tail ”) * p (“ head ”) = 1 2 * 1 2 = 1 2 2 . . . p ( X = n ) = ntimes z } { p (“ tail ”) * ... * p (“ tail ”) * p (“ head ”) = 1 2 * ... * 1 2 * 1 2 = 1 2 n from this, it is clear that the probability mass distribution of X is: p X ( x ) = 1 2 x 1 Once the distribution is known, H ( X ) can be computed from the definition: H ( X ) = X x ∈X p X ( x )log 2 p X ( x ) = ∞ X x =1 1 2 x log 2 1 2 x = ∞ X x =0 1 2 x log 2 1 2 x (since the summed expr. equals 0 for x = 0) = ∞ X x =0 1 2 x x log 2 1 2 (property of logarithms) = log 2 1 2 ∞ X x =0 1 2 x x = ∞ X x =0 1 2 x x = 1 2 ( 1 1 2 ) 2 = 2 [bit] exploiting ∞ X x =0 ( k ) x x = k (1 k ) 2 ! (b) Since the most likely value for X is 1 ( p ( X = 1) = 1 2 ), the most efficient first question is: “Is X = 1?”; the next question will be “Is X = 2?” and so on, until a positive answer is found. If this strategy is used, the random variable Y representing the number of questions will have the same distribution as X and it will be: E [ Y ] = ∞ X y =0 y 1 2 y = 1 2 ( 1 1 2 ) 2 = 2 which is exactly equal to the entropy of X ....
View
Full Document
 Fall '10
 NatashaDevroye
 X, log2, Davide Basilio Bartolini, Kenneth Palacio

Click to edit the document details