HW2_ES250_sol_a

HW2_ES250_sol_a - Harvard SEAS ES250 Information Theory...

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Harvard SEAS ES250 Information Theory Homework 2 Solutions 1. An n-dimensional rectangular box with sides X 1 ,X 2 , ,X n is to be constructed. The volume is V n = producttext n i =1 X i . The edge-length l of an n-cube with the same volume as the random box is l = V 1 /n n . Let X 1 ,X 2 , be i.i.d. uniform random variables over the interval [0 ,a ]. Find lim n V 1 /n n , and compare to ( EV n ) 1 /n . Clearly the expected edge length does not capture the idea of the volume of the box. Solution : The volume V n = producttext n i =1 X i is a random variable. Since the X i are i.i.d., uniformly distributed on [0 ,a ], we have: log e V 1 n n = 1 n log e V n = 1 n summationdisplay log e X i E [log e ( X )] by the Strong Law of Large Number, since X i and log e ( X i ) are i.i.d. and E [log e ( X )] < . Now E [log e ( X )] = 1 a integraldisplay a log e ( x ) dx = log e ( a ) 1 Hence, since e x is a continuous function, lim n V 1 n n = e lim n 1 n log e V n = a e < a 2 . Thus the effective edge length of this solid is a e . Note that since the X i s are independent, E ( V n ) = producttext E ( X i ) = ( a 2 ) n . Also a 2 is the arithmetic mean of the random variable, and a e is the geometric mean. 2. Let X 1 ,X 2 , be drawn i.i.d. according to the following distribution: X i = 1 , 1 2 2 , 1 4 3 , 1 4 Find the limiting behavior of the product ( X 1 X 2 X n ) 1 /n Solution : Let P n = ( X 1 X 2 X n ) 1 n Then log P n = 1 n n summationdisplay i =1 log X i E [log X ] , with probability 1, by the strong law of large numbers. Thus P n 2 E [log X ] with prob. 1. We can easily calculate E [log X ] = 1 2 log 1 + 1 4 log 2 + 1 4 log 3 = 1 4 log 6, and therefore P n 2 1 4 log 6 = 1 . 565. 1 Harvard SEAS ES250 Information Theory 3. Let X 1 ,X 2 , be an i.i.d. sequence of discrete random variables with entropy H ( X ). Let C n ( t ) = { x n X n : p ( x n ) 2 nt } denote the subset of n-sequences with probabilities 2 nt . (a) Show | C n ( t ) | 2 nt . (b) For what values of t does P ( { X n C n ( t ) } ) 1? Solution : (a) Since the total probability of all sequences is less than 1, | C n ( t ) | min x n C n ( t ) p ( x n ) 1, and hence | C n ( t ) | 2 nt . (b) Since 1 n log p ( x n ) H , if t < H , the probability that p ( x n ) > 2 nt goes to 0, and if t > H , the probability goes to 1. 4. Let X 1 ,X 2 , be independent, identically distributed random variables drawn according to the probability mass function p ( x ), x { 1 , 2 , ,m } . Thus, p ( x 1 ,x 2 , ,x n ) = producttext n i =1 p ( x i ). We know that 1 n log p ( X 1 ,X 2 , ,X n ) H ( X ) in probability. Let q ( x 1 ,x 2 , ,x n ) = producttext n i =1 q ( x i ), where q is another probability mass function on { 1 , 2 , ,m } ....
View Full Document

Page1 / 8

HW2_ES250_sol_a - Harvard SEAS ES250 Information Theory...

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online