HW2_ES250_sol_a

# HW2_ES250_sol_a - Harvard SEAS ES250 Information Theory...

• Notes
• 8

This preview shows pages 1–3. Sign up to view the full content.

Harvard SEAS ES250 – Information Theory Homework 2 Solutions 1. An n -dimensional rectangular box with sides X 1 ,X 2 , · · · ,X n is to be constructed. The volume is V n = producttext n i =1 X i . The edge-length l of an n -cube with the same volume as the random box is l = V 1 /n n . Let X 1 ,X 2 , · · · be i.i.d. uniform random variables over the interval [0 ,a ]. Find lim n →∞ V 1 /n n , and compare to ( EV n ) 1 /n . Clearly the expected edge length does not capture the idea of the volume of the box. Solution : The volume V n = producttext n i =1 X i is a random variable. Since the X i are i.i.d., uniformly distributed on [0 ,a ], we have: log e V 1 n n = 1 n log e V n = 1 n summationdisplay log e X i E [log e ( X )] by the Strong Law of Large Number, since X i and log e ( X i ) are i.i.d. and E [log e ( X )] < . Now E [log e ( X )] = 1 a integraldisplay a 0 log e ( x ) dx = log e ( a ) 1 Hence, since e x is a continuous function, lim n →∞ V 1 n n = e lim n →∞ 1 n log e V n = a e < a 2 . Thus the ”effective” edge length of this solid is a e . Note that since the X i ’s are independent, E ( V n ) = producttext E ( X i ) = ( a 2 ) n . Also a 2 is the arithmetic mean of the random variable, and a e is the geometric mean. 2. Let X 1 ,X 2 , · · · be drawn i.i.d. according to the following distribution: X i = 1 , 1 2 2 , 1 4 3 , 1 4 Find the limiting behavior of the product ( X 1 X 2 · · · X n ) 1 /n Solution : Let P n = ( X 1 X 2 · · · X n ) 1 n Then log P n = 1 n n summationdisplay i =1 log X i E [log X ] , with probability 1, by the strong law of large numbers. Thus P n 2 E [log X ] with prob. 1. We can easily calculate E [log X ] = 1 2 log1 + 1 4 log2 + 1 4 log3 = 1 4 log6, and therefore P n 2 1 4 log6 = 1 . 565. 1

This preview has intentionally blurred sections. Sign up to view the full version.

Harvard SEAS ES250 – Information Theory 3. Let X 1 ,X 2 , · · · be an i.i.d. sequence of discrete random variables with entropy H ( X ). Let C n ( t ) = { x n ∈ X n : p ( x n ) 2 nt } denote the subset of n -sequences with probabilities 2 nt . (a) Show | C n ( t ) | ≤ 2 nt . (b) For what values of t does P ( { X n C n ( t ) } ) 1? Solution : (a) Since the total probability of all sequences is less than 1, | C n ( t ) | min x n C n ( t ) p ( x n ) 1, and hence | C n ( t ) | ≤ 2 nt . (b) Since 1 n log p ( x n ) H , if t < H , the probability that p ( x n ) > 2 nt goes to 0, and if t > H , the probability goes to 1. 4. Let X 1 ,X 2 , · · · be independent, identically distributed random variables drawn according to the probability mass function p ( x ), x ∈ { 1 , 2 , · · · ,m } . Thus, p ( x 1 ,x 2 , · · · ,x n ) = producttext n i =1 p ( x i ). We know that 1 n log p ( X 1 ,X 2 , · · · ,X n ) H ( X ) in probability. Let q ( x 1 ,x 2 , · · · ,x n ) = producttext n i =1 q ( x i ), where q is another probability mass function on { 1 , 2 , · · · ,m } . (a) Evaluate lim 1 n log q ( X 1 ,X 2 , · · · ,X n ), where X 1 ,X 2 , · · · are i.i.d. p ( x ). (b) Now evaluate the limit of the log likelihood ratio 1 n log q ( X 1 , ··· ,X n ) p ( X 1 , ··· ,X n ) when X 1 ,X 2 , · · · are i.i.d.
This is the end of the preview. Sign up to access the rest of the document.
• Fall '09
• Probability theory, Stochastic process, Xn, Harvard SEAS

{[ snackBarMessage ]}

### What students are saying

• As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

Kiran Temple University Fox School of Business ‘17, Course Hero Intern

• I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

Dana University of Pennsylvania ‘17, Course Hero Intern

• The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

Jill Tulane University ‘16, Course Hero Intern