EECS 229A
Spring 2007
*
*
Solutions to Homework 4
1. Problem 7.5 on pg. 224 of the text.
Solution
:
Using two channels at once
To ﬁnd the capacity of the product channel we must ﬁnd the distribution
p
(
x
1
,x
2
) on the
input alphabet
X
1
× X
2
which maximizes
I
(
X
1
,X
2
;
Y
1
,Y
2
). Since the joint distribution
would be
p
(
x
1
,x
2
,y
1
,y
2
) =
p
(
x
1
,x
2
)
p
(
y
1

x
1
)
p
(
y
2

x
2
)
,
we have
I
(
X
1
,X
2
;
Y
1
,Y
2
) =
H
(
Y
1
,Y
2
)

H
(
Y
1
,Y
2

X
1
,X
2
)
=
H
(
Y
1
,Y
2
)

H
(
Y
1

X
1
,X
2
)

H
(
Y
2

X
1
,X
2
)
=
H
(
Y
1
,Y
2
)

H
(
Y
1

X
1
)

H
(
Y
2

X
2
)
≤
H
(
Y
1
) +
H
(
Y
2
)

H
(
Y
1

X
1
)

H
(
Y
2

X
2
)
=
I
(
X
1
;
Y
1
) +
I
(
X
2
;
Y
2
)
,
where we have used the conditional independence of
Y
1
and
Y
2
given (
X
1
,X
2
) to make
the second step, and Markovianity to make the third step. Equality occurs in the fourth
step when
X
1
and
X
2
are independent (so that
Y
1
and
Y
2
are also independent). Now it
follows that we have
C
=
max
p
(
x
1
,x
2
)
I
(
X
1
,X
2
;
Y
1
,Y
2
)
≤
max
p
(
x
1
,x
2
)
I
(
X
1
;
Y
1
) + max
p
(
x
1
,x
2
)
I
(
X
2
;
Y
2
)
= max
p
(
x
1
)
I
(
X
1
;
Y
1
) + max
p
(
x
2
)
I
(
X
2
;
Y
2
)
=
C
1
+
C
2
,
with equality if
p
(
x
1
,x
2
) =
p
*
(
x
1
)
p
*
(
x
2
), where
p
*
(
x
1
) and
p
*
(
x
2
) are the input distribu
tions that achieve the capacity of the ﬁrst and the second channel respectively.
The capacity of the product channel is therefore
C
1
+
C
2
.
2. Problem 7.12 on pg. 226 of the text.
Solution
:
Unused symbols
Let
α,β,γ
respectively denote the probabilities with which the three input symbols are
used,
α
+
β
+
γ
= 1. Then
H
(
Y

X
) = (
α
+
γ
)
h
(
1
3
) +
β
log 3, where
h
(
p
) denotes the
binary entropy function. For this choice of input distribution, we have
I
(
X
;
Y
) =
H
(
Y
)

H
(
Y

X
)
≤
(1

β
)log 3

(1

β
)
h
(
1
3
)
,
1