376A/Stat EE 376A Information Theory Prof. T. Cover
Handout #17 Tuesday, February 15, 2011
Solutions to Homework Set #5
1. Bad codes. Which of these codes cannot be Huffman codes for any probability assignment? (a) {1, 01, 00}. (b) {00, 01, 10, 110}. (c) {01, 10}. Solution: Bad codes. (a) {1,01,00} is a Huffman code for the distribution ( 1 , 1 , 1 ). 2 4 4 (b) The code {00,01,10, 110} can be shortened to {00,01,10, 11} without losing its instantaneous property, and therefore is not optimal, so it cannot be a Huffman code. Alternatively, it is not a Huffman code because there is a unique longest codeword. (c) The code {01,10} can be shortened to {0,1} without losing its instantaneous property, and therefore is not optimal and not a Huffman code.
2. Huffman coding. Consider the random variable ( ) x1 x2 x3 x4 x5 x6 x7 X= 0.50 0.26 0.11 0.04 0.04 0.03 0.02 (a) Find a binary Huffman code for X. (b) Find the expected codelength for this encoding. (c) Find a ternary Huffman code for X. Solution: Huffman coding. 1
(a) The Huffman tree for this distribution is Codeword 1 x1 0.50 0.50 0.50 0.50 01 x2 0.26 0.26 0.26 0.26 001 x3 0.11 0.11 0.11 0.11 00011 x4 0.04 0.04 0.08 0.13 00010 x5 0.04 0.04 0.05 00001 x6 0.03 0.05 00000 x7 0.02
0.50 0.26 0.24
0.50 0.50
1
(b) The expected length of the codewords for the binary Huffman code is 2 bits. (H(X) = 1.99 bits) (c) The ternary Huffman tree is Codeword 0 x1 0.50 0.50 0.50 1.0 1 x2 0.26 0.26 0.26 20 x3 0.11 0.11 0.24 21 x4 0.04 0.04 222 x5 0.04 0.09 221 x6 0.03 220 x7 0.02 This code has an expected length 1.33 ternary symbols. (H3 (X) = 1.25 ternary symbols).
3. Codes. Let X1 , X2 , . . . , i.i.d. with
1, 2, X= 3,
with probability 1/2 with probability 1/4 with probability 1/4.
Consider the code assignment
0, 01, C(x) = 11,
if x = 1 if x = 2 if x = 3.
(a) Is this code nonsingular? (b) Uniquely decodable? (c) Instantaneous? (d) What is the entropy rate of the process Z1 Z2 Z3 . . . = C(X1 )C(X2 )C(X3 ) . . .? 2
For example, xn = 2311 . . . gives the encoded process z n = 011100 . . . . Solution: Codes. (a) Yes, this code is nonsingular because C(x) is different for every x. (b) Yes, this code is uniquely decodable. Reversing the codewords if x = 1 0, 10, if x = 2 C (x) = 11, if x = 3 gives an instantaneous code, and thus a uniquely decodable code. Therefore the reversed extension is uniquely decodable, and so the extension itself is also uniquely decodable. (c) No, this code is not instantaneous because C(1) is a prefix of C(2). (d) The expected codeword length is 3 L(C(x)) = 0.5 1 + 0.25 2 + 0.25 2 = . 2 Further, the entropy rate of the i.i.d. X n is 3 H(X ) = H(X) = H(.5, .25, .25) = . 2 So the code is a uniquely decodable code with L = H(X ), and therefore the sequence is maximally compressed with H(Z) = 1 bit. If H(Z) were less than its maximum of 1 bit then the Z n sequence could be further compressed to its entropy rate, and X m could also be compressed further by blockcoding. However, this would result in Lm < H(X ) which contradicts theorem 5.4.2 of the text. So H(Z) = 1 bit. Note that the Z n sequence is not i.i.d. Br( 1 ), even though H(Z) = 1 bit. For 2 1 example, P {Z1 = 1} = 4 , and a sequence starting 10 . . . is not allowed. However, once Zi = 0 for some i then Zk is Bernoulli( 1 ) for k > i, so Z n is asymptotically 2 Bernoulli( 1 ) and gives the entropy rate of 1 bit. 2
4. Bad wine. One is given 6 bottles of wine. It is known that precisely one bottle has gone bad (tastes terrible). From inspection of the bottles it is determined that the probability 7 5 4 4 3 3 pi that the ith bottle is bad is given by (p1 , p2 , . . . , p6 ) = ( 26 , 26 , 26 , 26 , 26 , 26 ). Tasting will determine the bad wine. 3
Suppose you taste the wines one at a time. Choose the order of tasting to minimize the expected number of tastings required to determine the bad bottle. Remember, if the first 5 wines pass the test you don't have to taste the last. (a) What is the expected number of tastings required? (b) Which bottle should be tasted first? Now you get smart. For the first sample, you mix some of the wines in a fresh glass and sample the mixture. You proceed, mixing and tasting, stopping when the bad bottle been has determined. (c) What is the minimum expected number of tastings required to determine the bad wine? (d) What mixture should be tasted first? Solution: Bad wine. (a) If we taste one bottle at a time, the corresponding number of tastings are {1, 2, 3, 4, 5, 5} with some order. By the same argument as in Lemma 5.8.1, to minimize the ex pected length pi lk we should have lj lk if pj > pk . Hence, the best order of tasting should be from the most likely wine to be bad to the least. The expected number of tastings required is
6 i=1
p i li = 1 75 26 = 2.88 =
7 5 4 4 3 3 +2 +3 +4 +5 [phone]
(b) The first bottle to be tasted should be the one with probability (c) The idea is to use Huffman coding.
7 . 26
(01) (11) (000) (001) (100) (101)
7 5 4 4 3 3
7 6 5 4 4
8 7 6 5
11 8 7
15 11
26
4
The expected number of tastings required is
6 i=1
p i li = 2 66 26 = 2.54 =
7 5 4 4 3 3 +2 +3 +3 +3 [phone]
Note that H(p) = 2.52 bits. (d) The mixture of the first, third, and forth bottles should be tasted first, (or equivalently the mixture of the second, fifth and sixth).
5. Minimum cost codes. Words like Run! Help! and Fire! are short, not because they are frequently used, but perhaps because time is precious in the situations in which these words are required. Suppose that X = i with probability pi , i = 1, 2, . . . , m. Let li be the number of binary symbols in the codeword associated with X = i, and let ci denote the cost per letter of the codeword when X = i. Thus the average cost C of the description of X is C = m pi ci li . i=1 -li (a) Minimize C over all l1 , l2 , . . . , lm such that 2 1. Ignore any implied integer constraints on li . Exhibit the minimizing l1 , l2 , . . . , lm and the associated minimum value C . (b) How would you use the Huffman code procedure to minimize C over all uniquely decodable codes? Let CHuffman denote this minimum. (c) Show that C CHuffman C + Minimum cost codes. -ni (a) We wish to minimize C = pi ci ni subject to 2 1. We will assume equality in the constraint and let ri = 2-ni and let Q = i pi ci . Let qi = (pi ci )/Q. Then q also forms a probability distribution and we can write C as C = pi ci ni (1) 1 = Q qi log (2) ri ( ) qi = Q qi log - qi log qi (3) ri = Q(D(q||r) + H(q)). (4) 5
m i=1
pi ci .
Since the only freedom is in the choice of ri , we can minimize C by choosing r = q or pi ci n = - log , (5) i pj cj where we have ignored any integer constraints on ni . The minimum cost C for this assignment of codewords is C = QH(q) (6)
(b) If we use q instead of p for the Huffman procedure, we obtain a code minimizing expected cost. (c) Now we can account for the integer constraints. Let ni = - log qi Then - log qi ni < - log qi + 1 Multiplying by pi ci and summing over i, we get the relationship C CHuf f man < C + Q. (9) (8)
(7)
6. Relative entropy is cost of miscoding. Let the random variable X have five possible outcomes {1, 2, 3, 4, 5}. Consider two distributions on this random variable Symbol p(x) q(x) C1 (x) C2 (x) 1 1/2 1/2 0 0 2 1/4 1/8 10 100 3 1/8 1/8 110 101 4 1/16 1/8 1110 110 5 1/16 1/8 1111 111 (a) Calculate H(p), H(q), D(p||q) and D(q||p). (b) The last two columns above represent codes for the random variable. Verify that the average length of C1 under p is equal to the entropy H(p). Thus C1 is optimal for p. Verify that C2 is optimal for q. (c) Now assume that we use code C2 when the distribution is p. What is the average length of the codewords. By how much does it exceed the entropy H(p)? (d) What is the loss if we use code C1 when the distribution is q? Solution: Relative entropy is cost of miscoding. 6
(a) H(p) =
i
-pi log pi
1 1 1 1 1 1 1 1 = - log - log - log - 2 log 2 2 4 4 8 8 16 16 15 = . 8 Similarly, H(q) = 2. D(p||q) =
i
pi log
pi qi
1 1/2 1 1/4 1 1/8 1 1/16 = log + log + log +2 log 2 1/2 4 1/8 8 1/8 16 1/8 1 = . 8
1 Similarly, D(q||p) = 8 .
(b) The average codeword length for C1 is EL1 = 1 1 1 1 15 1+ 2+ 3+2 4= . 2 4 8 16 8
Similarly, the average codeword length for C2 is 2. (c) 1 1 1 1 1+ 3+ 3+2 3 = 2, 2 4 8 16 1 which exceeds H(p) by D(p||q) = 8 . Ep L2 = (d) Similarly, Eq L1 =
17 , 8 1 which exceeds H(q) by D(q||p) = 8 .
7