solns4 - ECE 1502 - Information Theory Problem Set 4...

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon
ECE 1502 — Information Theory Problem Set 4 solutions 1 March 12, 2006 8.1 Preprocessing the output. (a) The statistician calculates ˜ Y = g ( Y ). Since X Y ˜ Y forms a Markov chain, we can apply the data processing inequality. Hence for every distribution on x , I ( X ; Y ) I ( X ; ˜ Y ) . (1) Let ˜ p ( x ) be the distribution on x that maximizes I ( X ; ˜ Y ). Then C = max p ( x ) I ( X ; Y ) I ( X ; Y ) p ( x )=˜ p ( x ) I ( X ; ˜ Y ) p ( x )=˜ p ( x ) = max p ( x ) I ( X ; ˜ Y ) = ˜ C. (2) Thus, the statistician is wrong and processing the output does not increase capacity. (b) We have equality (no decrease in capacity) in the above sequence of inequalities only if we have equality in data processing inequality, i.e., for the distribution that maximizes I ( X ; ˜ Y ), we have X ˜ Y Y forming a Markov chain. 8.2 Maximum likelihood decoding. (a) By Bayes rule, for any events A and B , Pr( A | B ) = Pr( A ) Pr( B | A ) Pr( B ) . (3) In this case, we wish to calculate the conditional probability of a 1 given the channel output. Thus we take the event A to the event that the source produced a 1 , and B to be the event corresponding to one of the 8 possible output sequences. Thus Pr( A ) = 1 / 2, and Pr( B | A ) = ± i (1 - ± ) 3 - i , where ± is the number of ones in the received sequence. Pr( B ) can then be calculated as Pr( B ) = Pr( a 1 ) Pr( B | a 1 ) + Pr( B | a 2 ). Thus we can calculate Pr( a 1 | 000) = 1 2 (1 - ± ) 3 1 2 (1 - ± ) 3 + 1 2 ± 3 (4) Pr( a 1 | 100) = Pr( a 1 | 010) = Pr( a 1 | 001) = 1 2 (1 - ± ) 2 ± 1 2 (1 - ± ) 2 ± + 1 2 ± 2 (1 - ± ) (5) Pr( a 1 | 110) = Pr( a 1 | 011) = Pr( a 1 | 101) = 1 2 (1 - ± ) ± 2 1 2 (1 - ± ) ± 2 + 1 2 ± (1 - ± ) 2 (6) Pr( a 1 | 111) = 1 2 ± 3 1 2 ± 3 + 1 2 (1 - ± ) 3 (7) (b) If ± < 1 / 2, then the probability of a 1 given 000,001,010 or 100 is greater than 1/2, and the probability of a 2 given 110,011,101 or 111 is greater than 1/2. Therefore, the decoding rule above chooses the source symbol that has maximum probability given the observed output. This is the maximum a posteriori decoding rule, and is optimal in that 1 Solutions to problems from the text are supplied courtesy of Joy A. Thomas. 1
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
it minimizes the probability of error. To see that this is true, let the input source symbol be X , let the output of the channel be denoted by Y and the decoded symbol be ˆ X ( Y ). Then Pr( E ) = Pr( X 6 = ˆ X ) (8) = X y Pr( Y = y ) Pr( X 6 = ˆ X | Y = y ) (9) = X y Pr( Y = y ) X x 6 x ( y ) Pr( x | Y = y ) (10) = X y Pr( Y = y ) (1 - Pr(ˆ x ( y ) | Y = y )) (11) = X y Pr( Y = y ) - X y Pr ( Y = y ) Pr(ˆ x ( y ) | Y = y ) (12) = 1 - X y Pr ( Y = y ) Pr(ˆ x ( y ) | Y = y ) (13) and thus to minimize the probability of error, we have to maximize the second term, which is maximized by choosing ˆ x ( y ) to the the symbol that maximizes the conditional probability of the source symbol given the output. (c) The probability of error can also be expanded
Background image of page 2
Image of page 3
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 11/19/2010 for the course ANTH 122 taught by Professor 323 during the Spring '10 term at Centennial College.

Page1 / 7

solns4 - ECE 1502 - Information Theory Problem Set 4...

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online