EE 376A
Information Theory
Prof. T. Weissman
Feb 24, 2010
Homework Set #5 Solution
1.
The Z channel.
The Zchannel has binary input and output alphabets and transition
probabilities
p
(
y

x
) given by the following matrix:
Q
=
±
1
0
1
/
2
1
/
2
²
x, y
∈ {
0
,
1
}
Find the capacity of the Zchannel and the maximizing input probability distribution.
Solutions:
The Z channel.
First we express
I
(
X
;
Y
), the mutual information between the input
and output of the Zchannel, as a function of
x
= Pr(
X
= 1):
H
(
Y

X
)
=
Pr(
X
= 0)
·
0 + Pr(
X
= 1)
·
1 =
x
H
(
Y
)
=
H
2
(Pr(
Y
= 1)) =
H
2
(
x/
2)
I
(
X
;
Y
)
=
H
(
Y
)

H
(
Y

X
) =
H
2
(
x/
2)

x
Since
I
(
X
;
Y
) = 0 when
x
= 0 and
x
= 1, the maximum mutual information is
obtained for some value of
x
such that 0
< x <
1.
Using elementary calculus, we determine that
d
dx
I
(
X
;
Y
) =
1
2
log
2
1

x/
2
x/
2

1
,
which is equal to zero for
x
= 2
/
5.
(It is reasonable that Pr(
X
= 1)
<
1
/
2 because
X
= 1 is the noisy input to the channel.) So the capacity of the Zchannel in bits is
H
(1
/
5)

2
/
5 = 0
.
722

0
.
4 = 0
.
322.
2.
Channel capacity:
Calculate the capacity of the following channels with probability
transition matrices:
(a)
X
=
Y
=
{
0
,
1
,
2
}
p
(
y

x
) =
1
/
3
1
/
3
1
/
3
1
/
3
1
/
3
1
/
3
1
/
3
1
/
3
1
/
3
(1)
(b)
X
=
Y
=
{
0
,
1
,
2
}
p
(
y

x
) =
1
/
2
1
/
2
0
0
1
/
2
1
/
2
1
/
2
0
1
/
2
(2)
1
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document(c)
X
=
Y
=
{
0
,
1
,
2
,
3
}
p
(
y

x
) =
p
1

p
0
0
1

p
p
0
0
0
0
q
1

q
0
0
1

q
q
(3)
Channel Capacity:
This is the end of the preview.
Sign up
to
access the rest of the document.
 Spring '10
 sd
 Normal Distribution, y1, py, Prof. T. Weissman

Click to edit the document details