Harvard SEAS
ES250 – Information Theory
Homework 4
(Due Date: Nov. 13, 2007)
1. Consider a binary symmetric channel with
Y
i
=
X
i
⊕
Z
i
, where
⊕
is mod 2 addition, and
X
i
, Y
i
∈
{
0
,
1
}
.
Suppose that
{
Z
i
}
has constant marginal probabilities Pr(
Z
i
= 1) =
p
and Pr(
Z
i
= 0) = 1

p
, but
that
Z
1
, Z
2
,
· · ·
, Z
n
are not necessarily independent. Let
C
= 1

H
(
p
). Show that
max
p
(
x
1
,x
2
,
···
,x
n
)
I
(
X
1
, X
2
,
· · ·
, X
n
;
Y
1
, Y
2
,
· · ·
, Y
n
)
≥
nC
Comment on the implications.
2. Consider the channel
Y
=
X
+
Z
(mod 13), where
Z
=
1
,
with probability
1
3
2
,
with probability
1
3
3
,
with probability
1
3
and
X
∈ {
0
,
1
,
· · ·
,
12
}
.
(a) Find the capacity.
(b) What is the maximizing
p
*
(
x
)?
3. Using two channels.
(a) Consider two discrete memoryless channels (
X
1
, p
(
y
1

x
1
)
,
Y
1
) and (
X
2
, p
(
y
2

x
2
)
,
Y
2
) with capac
ities
C
1
and
C
2
respectively. A new channel (
X
1
× X
2
, p
(
y
1

x
1
)
×
p
(
y
2

x
2
)
,
Y
1
× Y
2
) is formed
in which
x
1
∈ X
1
and
x
2
∈ X
2
, are
simultaneously
sent, resulting in
y
1
, y
2
. Find the capacity of
this channel.