Solution: Random questions.
We assume that
X, Q
1
, Q
2
are jointly independent.
We also assume
A
(
X, Q
) is a
deterministic function; i.e., the answer is a function of the question
Q
and the object
X
.
(a)
I
(
X
;
Q, A
)
(
a
)
=
I
(
Q
;
X
) +
I
(
A
;
X

Q
)
(
b
)
=
I
(
A
;
X

Q
)
(
c
)
=
H
(
A

Q
)
−
H
(
A

X, Q
)
(
d
)
=
H
(
A

Q
)
where
(a) Chain rule.
(b)
X
and
Q
are independent.
(c) Definition of mutual information.
(d) Answer is a deterministic function of
X
and
Q
.
The interpretation is as follows. The uncertainty removed in
X
given (
Q, A
) is
the same as the uncertainty in the answer given the question.
(b) Using the result from part (a) and the fact that questions are independent, we
can easily obtain the desired relationship.
I
(
X
;
Q
1
, A
1
, Q
2
, A
2
)
(
a
)
=
I
(
X
;
Q
1
) +
I
(
X
;
A
1

Q
1
) +
I
(
X
;
Q
2

A
1
, Q
1
)
+
I
(
X
;
A
2

A
1
, Q
1
, Q
2
)
(
b
)
=
I
(
X
;
A
1

Q
1
) +
H
(
Q
2

A
1
, Q
1
)
−
H
(
Q
2

X, A
1
, Q
1
)
+
I
(
X
;
A
2

A
1
, Q
1
, Q
2
)
(
c
)
=
I
(
X
;
A
1

Q
1
) +
I
(
X
;
A
2

A
1
, Q
1
, Q
2
)
=
I
(
X
;
A
1

Q
1
) +
H
(
A
2

A
1
, Q
1
, Q
2
)
−
H
(
A
2

X, A
1
, Q
1
, Q
2
)
(
d
)
=
I
(
X
;
A
1

Q
1
) +
H
(
A
2

A
1
, Q
1
, Q
2
)
(
e
)
≤
I
(
X
;
A
1

Q
1
) +
H
(
A
2

Q
2
)
(
f
)
≤
H
(
A
1

Q
1
)
−
H
(
A
1

X, Q
1
) +
H
(
A
2

Q
2
)
(
g
)
≤
H
(
A
1

Q
1
) +
H
(
A
2

Q
2
)
(
h
)
≤
2
H
(
A
1

Q
1
)
(
i
)
=
2
I
(
X
;
A
1
, Q
1
)
ECE 563, Fall 2011
Homework 2 Solution
Parts of the solutions are copied from the solution manual of Cover and Thomas.
Question 1:
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
(a) Chain rule.
(b)
X
and
Q
1
are independent.
(c)
Q
2
is independent of
X
,
Q
1
, and
A
1
. (d)
A
2
is completely determined given
Q
2
and
X
.
(e) Conditioning decreases entropy.
(f) Expand the entropy.
(g)
A
1
is completely determined given
Q
1
and
X
.
(h) Identically distributed.
(i) Invoke the result of part (a).
In the above solution, the mutual information chain rule was applied directly.
Here is an alternative solution which some might prefer. Starting over, we want
to prove that
I
(
X
;
Q
1
, A
1
, Q
2
, A
2
)
≤
2
I
(
X
;
Q
1
, A
1
)
.
This is the end of the preview.
Sign up
to
access the rest of the document.
 Spring '11
 Kelly
 Probability theory, Yi, Pallavolo Modena, Sisley Volley Treviso, Probability mass function

Click to edit the document details