are independent and hence
Y
1
and
Y
2
are independent. Therefore
C
=
max
p
(
x
1
,x
2
)
I
(
X
1
, X
2
;
Y
1
, Y
2
)
≤
max
p
(
x
1
,x
2
)
I
(
X
1
;
Y
1
) + max
p
(
x
1
,x
2
)
I
(
X
2
;
Y
2
)
=
max
p
(
x
1
)
I
(
X
1
;
Y
1
) + max
p
(
x
2
)
I
(
X
2
;
Y
2
)
=
C
1
+
C
2
.
with equality iff
p
(
x
1
, x
2
) =
p
*
(
x
1
)
p
*
(
x
2
) and
p
*
(
x
1
) and
p
*
(
x
2
) are the distributions
that maximize
C
1
and
C
2
respectively.
8.
A channel with two independent looks at Y.
Let
Y
1
and
Y
2
be conditionally independent and conditionally identically distributed
given
X.
Thus
p
(
y
1
, y
2

x
) =
p
(
y
1

x
)
p
(
y
2

x
)
.
(a) Show
I
(
X
;
Y
1
, Y
2
) = 2
I
(
X
;
Y
1
)
−
I
(
Y
1
;
Y
2
)
.
(b) Conclude that the capacity of the channel
a45
a45
X
(
Y
1
, Y
2
)
is less than twice the capacity of the channel
a45
a45
X
Y
1
(c) How about 3 independent looks? Compare
I
(
X
;
Y
1
, Y
2
, Y
3
) to 3
I
(
X
;
Y
1
).
Solution: A channel with two independent looks at Y.
7
(a) First note that
Y
1
and
Y
2
are identically distributed since
p
(
y
1

x
) =
p
(
y
2

x
) and
hence
p
(
y
1
) =
∑
x
p
(
y
1

x
)
p
(
x
) =
∑
x
p
(
y
2

x
)
p
(
x
) =
p
(
y
2
). Therefore,
I
(
X
;
Y
1
, Y
2
)
=
H
(
Y
1
, Y
2
)
−
H
(
Y
1
, Y
2

X
)
=
H
(
Y
1
) +
H
(
Y
2
)
−
I
(
Y
1
;
Y
2
)
−
H
(
Y
1
, Y
2

X
)
=
H
(
Y
1
) +
H
(
Y
2
)
−
I
(
Y
1
;
Y
2
)
−
H
(
Y
1

X
)
−
H
(
Y
2

X
)
(7)
=
2
H
(
Y
1
)
−
2
H
(
Y
1

X
)
−
I
(
Y
1
;
Y
2
)
(8)
=
2
I
(
X
;
Y
1
)
−
I
(
Y
1
;
Y
2
)
,
where Eq. (7) follows from the fact that
Y
1
and
Y
2
are conditionally independent
given
X
and Eq. (8) follows from the fact that
Y
1
and
Y
2
are identically distributed
and conditionally identically distributed given
X
.
(b) The capacity of the single look channel
X
→
Y
1
is
C
1
= max
p
(
x
)
I
(
X
;
Y
1
)
.
The capacity of the channel
X
→
(
Y
1
, Y
2
) is
C
2
=
max
p
(
x
)
I
(
X
;
Y
1
, Y
2
)
=
max
p
(
x
)
2
I
(
X
;
Y
1
)
−
I
(
Y
1
;
Y
2
)
≤
max
p
(
x
)
2
I
(
X
;
Y
1
)
=
2
C
1
.
Hence, two independent looks cannot be more than twice as good as one look.
(c) Observe for
Y
n
conditionally independent given
X
that
I
(
X
;
Y
k

Y
k

1
)
=
H
(
Y
k

Y
k

1
)
−
H
(
Y
k

Y
k

1
, X
)
=
H
(
Y
k

Y
k

1
)
−
H
(
Y
k

X
)
(9)
≤
H
(
Y
k
)
−
H
(
Y
k

X
)
(10)
=
I
(
X
;
Y
k
)
where Eq. (9) follows from the fact that
Y
k
and
Y
k

1
are conditionally independent
given
X
and Eq. (10) follows from the fact that conditioning reduces entropy.
Using the above relationship,
I
(
X
;
Y
1
, Y
2
, Y
3
)
=
I
(
X
;
Y
1
) +
I
(
X
;
Y
2

Y
1
) +
I
(
X
;
Y
3

Y
1
, Y
2
)
≤
I
(
X
;
Y
1
) +
I
(
X
;
Y
2
) +
I
(
X
;
Y
3
)
(11)
=
3
I
(
X
;
Y
1
)
(12)
where Eq. (11) follows from the relationship shown above and Eq. (12) follows
from the fact that
Y
1
,
Y
2
and
Y
3
are identically distributed.
Thus it can be shown that the capacity
C
3
of three looks is less than three times
the capacity
C
1
of one look,
C
3
<
3
C
1
.
8
You've reached the end of your free preview.
Want to read all 9 pages?
 Spring '10
 sd
 Information Theory, Probability theory, y1, NC