ECE 255AN
Fall 2011
Homework set 2  solutions
1. For the upper bound, observe that for all distributions over the positive integers,
H
(
X
)

E
(
X
) =
X
x
p
(
x
) log
1
p
(
x
)

X
x
p
(
x
) log 2
x
=
X
x
p
(
x
) log
2

x
p
(
x
)
≤
log
X
x
2

x
≤
log 1 = 0
.
For the lower bound, if
X
is distributed monotonically over the positive integers then
p
(
x
)
≤
1
/x
, for example,
p
(2)
≤
1
/
2. Hence,
H
(
X
) =
E
(log
1
p
(
X
)
)
≥
E
(log
1
1
/X
)
≥
E
(log(
X
))
.
2.
(a) (
X
+
Y,X

Y
) is in 11 correspondence with (
X,Y
), hence
H
(
X
+
Y,X

Y
) =
H
(
X,Y
) =
H
(
X
) +
H
(
Y
) = 2 log
n
.
(b)
2 log
n
=
H
(
X,Y
)
=
H
(min(
X,Y
)
,
max(
X,Y
)) +
H
((
X,Y
)

(min(
X,Y
)
,
max(
X,Y
)))
=
H
(min(
X,Y
)
,
max(
X,Y
)) +
P
(
X,Y
∈ {
n

1
,n
}
X
6
=
Y
)
·
h
(
.
5)
=
H
(min(
X,Y
)
,
max(
X,Y
)) +
2
n
2
.
Solutions to Chapter 2 problems
8.
Drawing with and without replacement.
Intuitively, it is clear that if the balls are drawn
with replacement, the number of possible choices for the
i
th ball is larger, and therefore the
conditional entropy is larger. But computing the conditional distributions is slightly involved.
It is easier to compute the unconditional entropy.
•
With replacement. In this case the conditional distribution of each draw is the same for