62
3. By the chain rule and the independence of
X
1
, X
2
, . . . , X
n
1
, we have
H
(
X
1
, X
2
, . . . , X
n
)
=
H
(
X
1
, X
2
, . . . , X
n

1
) +
H
(
X
n

X
n

1
, . . . , X
1
)
(6.10)
=
n

1
X
i
=1
H
(
X
i
) + 0
(6.11)
=
n

1
,
(6.12)
since
X
n
is a function of the previous
X
i
’s. The total entropy is not
n
, which is what would be obtained if
the
X
i
’s were all independent. This example illustrates that pairwise independence does not imply complete
independence.
Problem 5.6
Bad codes.
Which of these codes cannot be Huffman codes for any probability assignment?
1.
{
0
,
10
,
11
}
.
2.
{
00
,
01
,
10
,
110
}
.
3.
{
01
,
10
}
.
Solution 5.6
Bad codes
1.
{
0,10,11
}
is a Huffman code for the distribution (1/2,1/4,1/4).
2. The code
{
00,01,10, 110
}
can be shortened to
{
00,01,10, 11
}
without losing its instantaneous property, and
therefore is not optimal, so it cannot be a Huffman code. Alternatively, it is not a Huffman code because there
is a unique longest codeword.
3. The code