EE 376A/Stat 376A
Information Theory
Prof. T. Weissman
Friday, March 17, 2006
Solutions to Practice Final Problems
These problems are sampled from a couple of the actual finals in previous years.
1. (
20 points)
Errors and erasures.
Consider a binary symmetric channel (BSC) with crossover probability
p
.


·
·
·
·
·
·
·
·3
Q
Q
Q
Q
Q
Q
Q
Qs
1
0
1
0
1

p
1

p
p
p
A helpful genie who knows the locations of all bit flips offers to convert flipped bits
into erasures. In other words, the genie can transform the BSC into a binary erasure
channel. Would you use his power? Be specific.
Solution: Errors and erasures.
Although it is very tempting to accept the genie’s offer, on a second thought, one
realizes that it is disadvantageous to convert the bit flips into erasures when
p
is large.
For example, when
p
= 1
,
the original BSC is noiseless, while the “helpful” genie will
erase every single bit coming out from the channel.
The capacity
C
1
(
p
) of the binary symmetric channel with crossover probability
p
is
1

H
(
p
) while the capacity
C
2
(
p
) of the binary erasure channel with erasure probability
p
is 1

p
.
One would convert the BSC into a BEC only if
C
1
(
p
)
≤
C
2
(
p
), that is,
p
≤
p
*
=
.
7729
.
(See Figure 1.)
2.
(20 points)
Code constraint.
What is the capacity of a BSC(
p
) under the constraint that each of the codewords has
a proportion of 1’s less than or equal to
α
, i.e.,
1
n
n
X
i
=1
X
i
(
w
)
≤
α,
for
w
∈ {
1
,
2
, . . . ,
2
nR
}
.
(Pay attention when
α >
1
/
2.)
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
p
1H(p)
1p
pstar
Solution: Code constraint.
Using the similar argument for the capacity of Gaussian channels under the power
constraint
P
, we find that the capacity
C
of a BSC(
p
) under the proportion constraint
α
is
C
=
max
p
(
x
):
EX
≤
α
I
(
X
;
Y
)
.
Now under the Bernoulli(
π
) input distribution with
π
≤
α
, we have
I
(
X
;
Y
) =
H
(
Y
)

H
(
Y

X
)
=
H
(
Y
)

H
(
Z

X
)
=
H
(
Y
)

H
(
Z
)
=
H
(
π
*
p
)

H
(
p
)
,
(1)
where
π
*
p
= (1

π
)
p
+
π
(1

p
). (Breaking
I
(
X
;
Y
) =
H
(
X
)

H
(
X

Y
) =
H
(
X
)

H
(
Z

Y
) is way more complicated since
Z
and
Y
are correlated.) Now when
α >
1
/
2
,
we have
max
π
H
(
π
*
p
)

H
(
p
) = 1

H
(
p
)
,
with the capacityachieving
π
*
= 1
/
2
.
On the other hand, when
α
≤
1
/
2
, π
*
=
α
achieves the maximum of (1); hence
C
=
H
(
α
*
p
)

H
(
p
)
.
3. (
20 points)
Partition.
Let (
X, Y
) denote height and weight. Let [
Y
] be
Y
rounded off to the nearest pound.
(a) Which is greater
I
(
X
;
Y
) or
I
(
X
; [
Y
]) ?
(b) Why?
2
Solution: Partition.
(a)
I
(
X
;
Y
)
≥
I
(
X
; [
Y
])
.
(b) Data processing inequality.
4. (
20 points)
Amplify and forward.
We cascade two Gaussian channels by feeding the (scaled) output of the first channel
into the second.
·
¶‡


?

?
·
¶‡
q


·
¶‡

?
X
1
P
X
2
P
Y
1
Y
2
Z
1
∼
N
(0
, N
)
α
Z
2
∼
N
(0
, N
)
Thus noises
Z
1
and
Z
2
are independent and identically distributed according to
N
(0
, N
),
EX
2
1
=
EX
2
2
=
P,
Y
1
=
X
1
+
Z
1
,
Y
2
=
X
2
+
Z
2
,
and
X
2
=
αY
1
,
where the scaling factor
α
is chosen to satisfy the power constraint
EX
2
2
=
P
.
You've reached the end of your free preview.
Want to read all 13 pages?
 Spring '10
 sd
 Information Theory, rate distortion, Zi Xi Yi, rate distortion function