*This preview shows
pages
1–3. Sign up
to
view the full content.*

This
** preview**
has intentionally

**sections.**

*blurred***to view the full version.**

*Sign up*
**Unformatted text preview: **Harvard SEAS ES250 Information Theory Homework 2 Solutions 1. An n-dimensional rectangular box with sides X 1 ,X 2 , ,X n is to be constructed. The volume is V n = producttext n i =1 X i . The edge-length l of an n-cube with the same volume as the random box is l = V 1 /n n . Let X 1 ,X 2 , be i.i.d. uniform random variables over the interval [0 ,a ]. Find lim n V 1 /n n , and compare to ( EV n ) 1 /n . Clearly the expected edge length does not capture the idea of the volume of the box. Solution : The volume V n = producttext n i =1 X i is a random variable. Since the X i are i.i.d., uniformly distributed on [0 ,a ], we have: log e V 1 n n = 1 n log e V n = 1 n summationdisplay log e X i E [log e ( X )] by the Strong Law of Large Number, since X i and log e ( X i ) are i.i.d. and E [log e ( X )] < . Now E [log e ( X )] = 1 a integraldisplay a log e ( x ) dx = log e ( a ) 1 Hence, since e x is a continuous function, lim n V 1 n n = e lim n 1 n log e V n = a e < a 2 . Thus the effective edge length of this solid is a e . Note that since the X i s are independent, E ( V n ) = producttext E ( X i ) = ( a 2 ) n . Also a 2 is the arithmetic mean of the random variable, and a e is the geometric mean. 2. Let X 1 ,X 2 , be drawn i.i.d. according to the following distribution: X i = 1 , 1 2 2 , 1 4 3 , 1 4 Find the limiting behavior of the product ( X 1 X 2 X n ) 1 /n Solution : Let P n = ( X 1 X 2 X n ) 1 n Then log P n = 1 n n summationdisplay i =1 log X i E [log X ] , with probability 1, by the strong law of large numbers. Thus P n 2 E [log X ] with prob. 1. We can easily calculate E [log X ] = 1 2 log 1 + 1 4 log 2 + 1 4 log 3 = 1 4 log 6, and therefore P n 2 1 4 log 6 = 1 . 565. 1 Harvard SEAS ES250 Information Theory 3. Let X 1 ,X 2 , be an i.i.d. sequence of discrete random variables with entropy H ( X ). Let C n ( t ) = { x n X n : p ( x n ) 2 nt } denote the subset of n-sequences with probabilities 2 nt . (a) Show | C n ( t ) | 2 nt . (b) For what values of t does P ( { X n C n ( t ) } ) 1? Solution : (a) Since the total probability of all sequences is less than 1, | C n ( t ) | min x n C n ( t ) p ( x n ) 1, and hence | C n ( t ) | 2 nt . (b) Since 1 n log p ( x n ) H , if t < H , the probability that p ( x n ) > 2 nt goes to 0, and if t > H , the probability goes to 1. 4. Let X 1 ,X 2 , be independent, identically distributed random variables drawn according to the probability mass function p ( x ), x { 1 , 2 , ,m } . Thus, p ( x 1 ,x 2 , ,x n ) = producttext n i =1 p ( x i ). We know that 1 n log p ( X 1 ,X 2 , ,X n ) H ( X ) in probability. Let q ( x 1 ,x 2 , ,x n ) = producttext n i =1 q ( x i ), where q is another probability mass function on { 1 , 2 , ,m } ....

View
Full
Document