376A EE Information Theory Prof. T. Cover
Handout #21 Tuesday, March 1, 2011
Solutions to Homework Set #6 1. Postprocessing the output. One is given a communication channel with transition probabilities p(y | x) and channel capacity C = max I(X; Y ). A helpful statistician postprocesses the output by forming
p(x)
~ Y = g(Y ), yielding a channel p(~|x). He claims that this will strictly improve the y capacity. (a) Show that he is wrong. (b) Under what conditions does he not strictly decrease the capacity?
Solution: Preprocessing the output. ~ ~ (a) The statistician calculates Y = g(Y ). Since X Y Y forms a Markov chain, we can apply the data processing inequality. Hence for every distribution on x, ~ I(X; Y ) I(X; Y ). ~ Let p(x) be the distribution on x that maximizes I(X; Y ). Then ~ ~ ~ ~ C = max I(X; Y ) I(X; Y )p(x)=~(x) I(X; Y )p(x)=~(x) = max I(X; Y ) = C. p p
p(x) p(x)
Thus, the helpful suggestion is wrong and processing the output does not increase capacity. (b) We have equality (no decrease in capacity) in the above sequence of inequalities only if we have equality in data processing inequality, i.e., for the distribution ~ ~ that maximizes I(X; Y ), we have X Y Y forming a Markov chain. Thus, ~ should be a sufficient statistic. Y
2. Noisy typewriter. Consider a 26-key typewriter. 1
(a) If pushing a key results in printing the associated letter, what is the capacity C in bits? (b) Now suppose that pushing a key results in printing that letter or the next (with equal probability). Thus A A or B, and Z Z or A. What is the capacity? (c) What is the highest rate code with block length one that you can find that achieves zero probability of error for the channel in part (b) . Solution: Noisy typewriter. (a) If the typewriter prints out whatever key is struck, then the output Y is the same as the input X and C = max I(X; Y ) = max H(X) = log 26, attained by a uniform distribution over the letters.
1 (b) In this case, the output is either equal to the input (with probability 2 ) or equal to the next letter ( with probability 1 ). Hence H(Y |X) = log 2 independent of 2 the distribution of X, and hence
(1)
C = max I(X; Y ) = max H(Y ) - log 2 = log 26 - log 2 = log 13,
(2)
which is attained for a uniform distribution over the output, which in turn is attained by a uniform distribution on the input. (c) A simple zero error block length one code is the one that uses every alternate letter, say A,C,E,. . . ,W,Y. In this case, none of the codewords will be confused, since A will produce either A or B, C will produce C or D, etc. The rate of this code, log(# codewords) log 13 R= = = log 13. (3) Block length 1 In this case, we can achieve capacity with a simple code with zero error. Note that the uniform distribution over the output is attained also by this input distribution.
3. The Z Channel. The Z-channel has binary input and output alphabets and transition probabilities p(y|x) given by the following matrix: Q= 1 0 1/2 1/2 2 x, y {0, 1}
Find the capacity of the Z-channel and the maximizing input probability distribution. Solution: The Z channel. First we express I(X; Y ), the mutual information between the input and output of the Z-channel, as a function of = Pr(X = 1): H(Y |X) = Pr(X = 0) 0 + Pr(X = 1) 1 = H(Y ) = H(Pr(Y = 1)) = H(/2) I(X; Y ) = H(Y ) - H(Y |X) = H(/2) - Since I(X; Y ) is strictly concave on (why?) and I(X; Y ) = 0 when = 0 and = 1, the maximum mutual information is obtained for some value of such that 0 < < 1. Using elementary calculus, we determine that 1 d 1 - /2 I(X; Y ) = log2 - 1, d 2 /2 which is equal to zero for = 2/5. (It is reasonable that Pr(X = 1) < 1/2 since X = 1 is the noisy input to the channel.) So the capacity of the Z-channel in bits is H(1/5) - 2/5 = 0.722 - 0.4 = 0.322.
4. An additive noise channel. Find the channel capacity of the following discrete memoryless channel: Z
?
X
-
Y
1 where Pr{Z = 0} = Pr{Z = a} = 2 . The alphabet for x is X = {0, 1}. Y = X + Z with real addition. Assume that Z is independent of X.
Observe that the channel capacity depends on the value of a. Solution: An additive noise channel. Y =X +Z X {0, 1}, Z {0, a}
We have to distinguish various cases depending on the values of a.
3
a = 0 In this case, Y = X, and max I(X; Y ) = max H(X) = 1. Hence the capacity is 1 bit per transmission. a = 0, 1 In this case, Y has four possible values 0, 1, a and 1 + a. Knowing Y , we know the X which was sent, and hence H(X|Y ) = 0. Hence max I(X; Y ) = max H(X) = 1, which is achieved for an uniform distribution on the input X. a = 1 In this case Y has three possible output values, 0, 1 and 2, and the channel is identical to the binary erasure channel discussed in class, with = 1/2. As derived in class, the capacity of this channel is 1 - = 1/2 bit per transmission. a = -1 This is similar to the case when a = 1 and the capacity here is also 1/2 bit per transmission.
5. Channels with memory have higher capacity. Consider a binary symmetric channel with Yi = Xi Zi , where is mod 2 addition, and Xi , Yi {0, 1}. Suppose that {Zi } has constant marginal probabilities P {Zi = 1} = p = 1-P {Zi = 0}, but that Z1 , Z2 , . . . , Zn are not necessarily independent. Let C = 1 - H(p, 1 - p). Show that max I(X1 , X2 , . . . , Xn ; Y1 , Y2 , . . . , Yn ) nC. Comment on the implications.
p(x1 ,x2 ,...,xn )
Solution: Channels with memory have higher capacity. Y i = Xi Z i , where Zi = and Zi are not independent. When X1 , X2 , . . . , Xn are chosen i.i.d. Bern( 1 ), 2 I(X1 , X2 , . . . , Xn ; Y1 , Y2, . . . , Yn ) = H(X1 , X2 , . . . , Xn ) - H(X1 , X2 , . . . , Xn |Y1 , Y2 , . . . , Yn ) = H(X1 , X2 , . . . , Xn ) - H(Z1, Z2 , . . . , Zn |Y1, Y2 , . . . , Yn ) H(X1 , X2 , . . . , Xn ) - H(Z1, Z2 , . . . , Zn ) H(X1 , X2 , . . . , Xn ) - = n - nH(p). H(Zi ) 1 probability with p 0 with probability 1 - p
4
Hence, the capacity of the channel with memory over n uses of the channel is nC (n) =
p(X1 ,X2 ,...,Xn )
max
I(X1 , X2 , . . . , Xn ; Y1 , Y2 , . . . , Yn )
2
I(X1 , X2 , . . . , Xn ; Y1, Y2 , . . . , Yn )p(x1 ,x2,...,xn )=Bern( 1 ) n(1 - H(p)) = nC. Hence, channels with memory have higher capacity. The intuitive explanation for this result is that the correlation between the noise decreases the effective noise; one could use the information from the past samples of the noise to estimate the present noise.
6. Channel capacity. Consider the channel Y = X + Z (mod 13), where Z= and X {0, 1, . . . , 12}. (a) Find the capacity. (b) What is the maximizing p (x)? Solution: Channel capacity. Y =X +Z where 1 2 Z= 3 4 with with with with (mod 13) 1/4 1/4 . 1/4 1/4 1, 2, 3, 4 1 , 1, 1, 1 4 4 4 4
probability probability probability probability
Then,
H(Y |X) = H(Z|X) = H(Z) = log 4, which is independent of the distribution of X. Hence the capacity of the channel is C = max I(X; Y )
p(x)
= max H(Y ) - H(Y |X)
p(x)
= max H(Y ) - log 4
p(x)
= log 13 - log 4, which is attained when Y has a uniform distribution, which occurs (by symmetry) when X has a uniform distribution. 5
(a) The capacity of the channel is log 13 bits/transmission. 4 (b) The capacity is achieved by a uniform distribution on the inputs, that is, p(X = i) = 1 13 for i = 0, 1, . . . , 12.
7. Using two channels at once. Consider two discrete memoryless channels (X1 , p(y1 | x1 ), Y1) and (X2 , p(y2 | x2 ), Y2 ) with capacities C1 and C2 respectively. A new channel (X1 X2 , p(y1 | x1 ) p(y2 | x2 ), Y1 Y2 ) is formed in which x1 X1 and x2 X2 , are simultaneously sent, resulting in y1 , y2 . Find the mutual information maximizing p (x1 , x2 ) in terms of the individual maximizing distributions p (x1 ) and p (x2 ). Find the capacity of this channel. Solution: Using two channels at once. To find the capacity of the product channel (X1 X2 , p(y1, y2 |x1 , x2 ), Y1 Y2 ), we have to find the distribution p(x1 , x2 ) on the input alphabet X1 X2 that maximizes I(X1 , X2 ; Y1 , Y2 ). Since the transition probabilities are given as p(y1 , y2|x1 , x2 ) = p(y1 |x1 )p(y2 |x2 ), p(x1 , x2 , y1 , y2 ) = p(x1 , x2 )p(y1 , y2 |x1 , x2 ) = p(x1 , x2 )p(y1 |x1 )p(y2 |x2 ), Therefore, Y1 X1 X2 Y2 forms a Markov chain and I(X1 , X2 ; Y1, Y2 ) = = = = H(Y1, Y2 ) - H(Y1, Y2 |X1 , X2 ) H(Y1, Y2 ) - H(Y1|X1 , X2 ) - H(Y2|X1 , X2 ) H(Y1, Y2 ) - H(Y1|X1 ) - H(Y2|X2 ) H(Y1) + H(Y2 ) - H(Y1 |X1 ) - H(Y2|X2 ) I(X1 ; Y1 ) + I(X2 ; Y2 ), (4) (5) (6)
where Eqs. (4) and (5) follow from Markovity, and Eq. (6) is met with equality if X1 and X2 are independent and hence Y1 and Y2 are independent. Therefore C =
p(x1 ,x2 )
max I(X1 , X2 ; Y1, Y2 )
p(x1 ,x2 )
p(x1 ,x2 ) p(x1 )
max I(X1 ; Y1 ) + max I(X2 ; Y2)
p(x2 )
= max I(X1 ; Y1 ) + max I(X2 ; Y2) = C1 + C2 . with equality iff p(x1 , x2 ) = p (x1 )p (x2 ) and p (x1 ) and p (x2 ) are the distributions that maximize C1 and C2 respectively. 6
8. A channel with two independent looks at Y. Let Y1 and Y2 be conditionally independent and conditionally identically distributed given X. Thus p(y1 , y2|x) = p(y1|x)p(y2 |x). (a) Show I(X; Y1, Y2 ) = 2I(X; Y1) - I(Y1 ; Y2). (b) Conclude that the capacity of the channel X
-
(Y1 , Y2 )
is less than twice the capacity of the channel X
-
Y1
(c) How about 3 independent looks? Compare I(X; Y1, Y2 , Y3 ) to 3I(X; Y1). Solution: A channel with two independent looks at Y. (a) First note that Y1 and Y2 are identically distributed since p(y1 |x) = p(y2 |x) and hence p(y1 ) = x p(y1 |x)p(x) = x p(y2 |x)p(x) = p(y2 ). Therefore, I(X; Y1 , Y2 ) = = = = = H(Y1 , Y2 ) - H(Y1 , Y2 |X) H(Y1 ) + H(Y2) - I(Y1 ; Y2 ) - H(Y1 , Y2 |X) H(Y1 ) + H(Y2) - I(Y1 ; Y2 ) - H(Y1 |X) - H(Y2 |X) 2H(Y1 ) - 2H(Y1 |X) - I(Y1 ; Y2) 2I(X; Y1) - I(Y1 ; Y2 ),
(7) (8)
where Eq. (7) follows from the fact that Y1 and Y2 are conditionally independent given X and Eq. (8) follows from the fact that Y1 and Y2 are identically distributed and conditionally identically distributed given X. (b) The capacity of the single look channel X Y1 is C1 = max I(X; Y1 ).
p(x)
The capacity of the channel X (Y1 , Y2) is C2 = max I(X; Y1, Y2 )
p(x) p(x)
= max 2I(X; Y1) - I(Y1 ; Y2 ) max 2I(X; Y1)
p(x)
= 2C1 . Hence, two independent looks cannot be more than twice as good as one look. 7
(c) Observe for Y n conditionally independent given X that I(X; Yk |Y k-1 ) = = = H(Yk |Y k-1 ) - H(Yk |Y k-1 , X) H(Yk |Y k-1 ) - H(Yk |X) H(Yk ) - H(Yk |X) I(X; Yk )
(9) (10)
where Eq. (9) follows from the fact that Yk and Y k-1 are conditionally independent given X and Eq. (10) follows from the fact that conditioning reduces entropy. Using the above relationship, I(X; Y1, Y2 , Y3) = I(X; Y1) + I(X; Y2|Y1 ) + I(X; Y3 |Y1, Y2 ) I(X; Y1) + I(X; Y2) + I(X; Y3) = 3I(X; Y1)
(11) (12)
where Eq. (11) follows from the relationship shown above and Eq. (12) follows from the fact that Y1 , Y2 and Y3 are identically distributed. Thus it can be shown that the capacity C3 of three looks is less than three times the capacity C1 of one look, C3 < 3C1 .
9. Can signal alternatives lower capacity? Show that adding a row to a channel transition matrix does not decrease capacity. Solution: Can signal alternatives lower capacity? Adding a row to the channel transition matrix is equivalent to adding a symbol to the input alphabet X . Suppose there were m symbols and we add an (m + 1)st. We can always choose not to use this extra symbol. Alternatively, let Cm and Cm+1 denote the capacity of the original channel and the new channel, respectively. Since using the original m symbols is same as using m + 1 symbols with the newly added symbol assigned zero probablity, Cm+1 =
p(x1 ,...,xm+1 ) p(x1 ,...,xm ,0)
max
I(X; Y )
max
I(X; Y )
= Cm .
8