hw4_sol - EE595A Introduction to Information Theory...

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: EE595A Introduction to Information Theory University of Washington Winter 2004 Dept. of Electrical Engineering Handout 8: Problem Set 4: Solutions Prof: Jeff A. Bilmes <bilmes@ee.washington.edu> Lecture 10, March 10, 2004 Book Problems Do problems 7.4(abcd),7.6,7.9,8.8,8.10,8.12,9.1,10.2 For problem 7.4d, compute the probability of the sequence 1 2 . . . n which is then followed by any arbitrary se- quence, where = X p : U ( p ) halts 2- l ( p ) , and = . 1 2 . . . . Also, do not do 7.4e. (note: a good break in this problem set is to be done with all chapter 7 problems, problem 1 and 2 below, and 1/2 of the chapter 8 problems by Friday). Problem 7.4 Monkeys on a computer. Suppose a random program is typed into a computer. Give a rough estimate of the probability that the computer prints the following sequence: 1. n followed by any arbitrary sequence. 2. 1 2 . . . n followed by any arbitrary sequence, where i is the ith bit in the expansion of . 3. n 1 followed by any arbitrary sequence. 4. (Optional) 1 2 . . . n followed by any arbitrary sequence, where = U ( p ) halts 2- l ( p ) , = . 1 2 . . . . Solution 7.4 The probability that a computer with a random input will print will print out the string x followed by any arbitrary sequence is the sum of the probabilities over all sequences starting with the string x . p U ( x . . . ) = X y { , 1 } * { , 1 } p U ( xy ) , where p U ( x ) = X p : U ( p )= x 2- ( p ) . (8.1) This sum is lower bounded by the largest term, which corresponds to the simplest concatenated sequence. 1. The simplest program to print a sequence that starts with n 0s is Print 0s forever. This program has constant length c and hence the probability of strings starting with n zeroes is p U (0 n . . . ) 2- c . (8.2) 2. Just as in part (a), there is a short program to print the bits of forever. Hence p U ( 1 2 . . . n . . . ) 2- c . (8.3) 3. A program to print out n 0s followed by a 1 must in general specify n . Since most integers n have a complexity log * n , and given n , the program to print out n 1 is simple, we have p U (0 n 1 . . . ) 2- log * n- c , (8.4) 8-1 8-2 4. We know that n bits of are essentially incompressible, i.e., their complexity n- c . Hence, the shortest program to print out n bits of followed by anything must have a length at least n- c , and hence p U ( 1 2 . . . n . . . ) 2- ( n- c ) . (8.5) Problem 7.6 Do computers reduce entropy? Let X = U ( P ) , where P is a Bernoulli(1/2) sequence. Here the binary sequence X is either undefined or is in { , 1 } * . Let H ( X ) be the Shannon entropy of X . Argue that H ( X ) = ....
View Full Document

This note was uploaded on 10/12/2009 for the course EE 596A taught by Professor Jeffa.bilmes during the Winter '04 term at Washington University in St. Louis.

Page1 / 6

hw4_sol - EE595A Introduction to Information Theory...

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online