University of Toronto
Department of Electrical
November 9, 2007
& Computer Engineering
ECE1502F — Information Theory
Midterm Test Solution
1
. (
Noisy Gates
)
(
a
) The noisy XOR gate induces a binary symmetric channel (BSC) with crossover
probability
p
.
As shown in class and in the text, the capacity of the BSC is
C
= 1
 H
(
p
); this capacity is achieved (for every
p
) with a uniform distribution.
(
b
) The noisy AND gate with
Z
∼
Bern
(1
/
2) induces a Zchannel:
whenever the
channel input
X
= 0, the channel output
Y
= 0; and whenever the input
X
= 1,
the output
Y
is equally likely to be 0 or 1.
Let
P
[
X
= 1] =
p
. We have
P
[
Y
= 1] =
p/
2, so
H
(
Y
) =
H
(
p/
2). We may write
I
(
X
;
Y
)
=
H
(
Y
)

H
(
Y

X
)
=
H
(
Y
)

H
(
Y

X
= 0)
0
(1

p
)

H
(
Y

X
= 1)
1
p
=
H
(
p/
2)

p.
Recall that
d
dx
H
(
x
) =

1
ln 2
d
dx
(
x
ln
x
+(1

x
) ln(1

x
)) =

1
ln 2
(ln
x
+1

ln(1

x
)

1) = log
2
1

x
x
.
Taking the derivative with respect to
p
, and setting the result to zero, we find
1
2
log((2

p
)
/p
) = 1 or (2

p
)
/p
= 4 or
p
= 2
/
5
.
Thus the capacity of the Zchannel is given by
H
(1
/
5)

2
/
5
≈
0
.
322, achieved
by setting
P
[
X
= 1] =
p
= 2
/
5.
(
c
) The noise OR gate with
Z
∼
Bern
(1
/
2)
also
induces a Zchannel: whenever the
channel input
X
= 1, the channel output
Y
= 1; and whenever the channel input
X
= 0, the output
Y
is equally likely to be 0 or 1. This is the same channel as in
the previous question, except with zeros and ones exchanged. Thus the capacity
is the same, namely
H
(1
/
5)

2
/
5, achieved by setting
P
[
X
= 1] = 3
/
5.
2
. (
Cover’s Cutting Contraption
)
(
a
) Every sequence of cuts induces a binary tree.
Each internal node of the tree
corresponds to a cut and the children of each node correspond to the two pieces
that result from that cut. Associated with each internal node is the cost of cutting
1
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
that node, and this cost is precisely the sum of the lengths of the children of that
node.
For example, the following figure shows the tree that results from applying the
following cuts:
i. cut the 1m rod into lengths (0.6,0.4); cost = 1;
ii. cut the 0.6m rod into two lengths (0.3,0.3); cost = 0.6;
iii. cut the 0.4m rod into lengths (0.2,0.2); cost = 0.4;
iv. cut a 0.2m rod into lengths (0.1,0.1); cost = 0.2.
0.1
0.1
0.2
0.3
0.3
0.6
0.2
0.4
1.0
Our objective is to find a tree whose
m
leaf nodes have values
p
1
,
p
2
, . . . ,
p
m
, and
for which the sum of the values over all internal nodes is minimal.
Let
i
denote the depth in the tree of leaf node
i
having value
p
i
. Let
V
denote the
set of internal nodes, and let
C
(
v
) denote the value associated with an internal
node
v
. We make use of the following observation:
v
∈
V
C
(
v
) =
m
i
=1
p
i
i
.
(1)
To see this, observe that for each node
v
∈
V
,
C
(
v
) is given as the sum of the
This is the end of the preview.
Sign up
to
access the rest of the document.
 Spring '11
 Kelly
 Gate, leaf node, binary symmetric channel, crossover probability, likely typical sequence, Cb > Ca

Click to edit the document details