1. (15 points)
True of false and short answer. Brief explanations (rather than lengthy proofs) suffice.
(a) (5 points) Which of the following sequences of codeword lengths cannot be the codeword lengths of a 3

ary
(ternary,
D
= 3) Huffman code?
(i) (1,1,2,2,3,3,3)
(ii) (1,1,2,2,3,3)
(iii) (1,1,2,2,3)
(iv) (1,2,2,2,2,2,2)
(v) (1,2,2,2,2)
Solution:
The easiest way to see which can be ternary Huffman codeword lengths by trying to construct the
Huffman tree.
From the figure, we can see that (iii) and (v) are impossible and could be shortened/pruned to
(1,1,2,2,2) and (1,1,2,2,2) respectively.
(i) Yes
(ii) Yes
(iii) No
(iv) Yes
(v) No
(b) (6 points) We have defined the mutual information
I
(
X
;
Y
) between the two random variables
X
and
Y
.
Let us *try* to define the mutual information between three random variables
X, Y
and
Z
as
I
(
X
;
Y
;
Z
) =
I
(
X
;
Y
)

I
(
X
;
Y

Z
).
(i) Is this definition symmetric in its arguments? Prove why or give an example of why not.
(ii) Is
I
(
X
;
Y
;
Z
) positive? Prove why or give an example of why not.
(iii) True or false:
I
(
X
;
Y
;
Z
) =
H
(
X, Y, Z
)

H
(
X
)

H
(
Y
)

H
(
Z
) +
I
(
X
;
Y
) +
I
(
Y
;
Z
) +
I
(
Z
;
X
).
(iv) True of false:
I
(
X
;
Y
;
Z
) =
H
(
X, Y, Z
)

H
(
X, Y
)

H
(
Y, Z
)

H
(
Z, X
) +
H
(
X
) +
H
(
Y
) +
H
(
Z
).
Solution:
(i) Yes, it is symmetric, as can be seen from (iii) and (iv).
Both expansions will be shown to be true and are
symmetric in their arguments.
(ii) No, it is not necessarily positive.
Take
X
and
Y
independent and identical coin flips and let
Z
=
X
+
Y
mod 2. Then
I
(
X
;
Y
) = 0 but
I
(
X
;
Y

Z
) =
H
(
X

Z
)

H
(
Y

X, Z
) =
H
(
X

Z
)

0 = 1.
2