University of Toronto
Department of Electrical
F. R. Kschischang
ECE1502F — Information Theory
Final Examination Solutions
December 13, 2000
1
.
Short Snappers
(a)
I
(
X
;
X
) =
H
(
X
), the entropy of
X
.
(b) False. Although
I
(
X
;
Y
) =
H
(
Y
)

H
(
Y

X
), maximizing
H
(
Y
) does not necessarily
maximize
I
(
X
;
Y
). In fact, it may not be possible to achieve a uniform output distri
bution. An example is the binary erasure channel, where the capacity achieving input
distribution yields an output distribution ((1

±
)
/
2
,±,
(1

±
)
/
2)), which is not uniform
unless
±
= 1
/
3.
(c) This symmetric channel has capacity
C
= log
2
(3)

H
(1
/
2
,
1
/
4
,
1
/
4)
= log
2
(3)

(1
/
2)log
2
(2)

(1
/
4)log
2
(4)

(1
/
4)log
2
(4)
= log
2
(3)

3
/
2
≈
.
085 bits
/
channel

use
.
(d) The channel transition matrix is
M
=
"
1

± ±
0
0
±
1

±
#
.
For this to deﬁne a weakly symmetric channel, the rows must be permutations of each
other (they are), and the column sums must be equal, which occurs when 1

±
= 2
±
,
i.e., when
±
= 1
/
3.
(e) We have
C
=
W
log
2
±
1 +
P
N
0
W
²
bits/s
.
If the SNR is 20 dB, we have
P/
(
N
0
W
) = 100, so
C
≈
20 Kbps. If the SNR is 30 dB, we
have
P/
(
N
0
W
) = 1000, so
C
≈
30 Kbps. Thus, if the channel is as noisy as indicated,
the telphone line capacity is on the order of 20 to 30 Kbps.
(f) True. Let
B
be a code of length
n
and rate
R < C
that achieves error probability
±
.
(By the coding theorem, we know that such a code exists for some length
n
(or longer)
for any
± >
0.) Form the code
B
0
of length
n
+ 1 by appending an extra symbol to
each codeword of
B
, where the extra symbol is a one if the given codeword has an odd
number of ones, and zero otherwise. Then every codeword of
B
0
has an even number of
ones.
Now, the error probability for
B
0
is no greater than
±
, since one decoding strategy for
B
0
would be to use a decoder for
B
that simply ignores the additional symbol. The rate
of
B
0
is
nR/
(
n
+ 1). By choosing
n
large enough, we can make the rate of
B
0
approach
arbitrarily close to
R
, and hence arbitrarily close to
C
. Hence the given constraint does
not a±ect the capacityachieving ability of this family of codes.
Page 1 of 7