1
Colorado State University, Ft. Collins
Fall 2008
ECE 516: Information Theory
Lecture 15 October 21, 2008
Recap:
7.6
Joint Typical Sequences
Definition: The set
()
n
A
ε
of jointly typical sequences with respect to joint PMF
y
x
p
,
is the set of
( )
n
n
y
x
,
whose empirical entropies are
-close to the true
entropies.
{
() ()
()()
⎭
⎬
⎫
<
−
−
<
−
−
<
−
−
×
∈
=
Y
X
H
y
x
p
n
Y
H
y
p
n
X
H
x
p
n
y
x
A
n
n
n
n
n
n
n
n
n
,
,
log
1
log
1
,
log
1
:
,
Y
X
Theorem:
(Joint
AEP)
Let
( )
n
n
Y
X
,
be
drawn
iid
according
to
∏
=
=
n
i
i
i
n
n
y
x
p
y
x
p
1
,
,
1.
( )
( )
1
,
lim
=
∈
∞
→
n
n
n
n
A
y
x
p
, i.e.,
( )
(
)
( )
−
>
∈
∞
→
1
,
lim
n
n
n
n
A
y
x
p
for large enough
n
.
2.
(
)
(
)
(
)
+
−
≤
≤
−
Y
X
H
n
n
Y
X
H
n
A
,
,
2
2
1
3. If
( ) ( ) ( )
n
n
n
n
y
p
x
p
y
x
~
~
~
~
,
~
, i.e.,
n
x
~
and
n
y
~
are independent with the marginals
( )
n
x
p
and
( )
n
y
p
obtained from
( )
n
n
y
x
p
,
, then
( )
(
)
( )
(
)
(
)
3
;
3
;
2
~
,
~
2
1
−
−
+
−
≤
∈
≤
−
Y
X
I
n
n
n
n
Y
X
I
n
A
y
x
p
Achievability (direct part):
All rates below
C
are achievable. That is, if
R
<
C
, then
one can design a code
( )
n
nR
,
2
such that
(
)
n
λ
,
(
)
0
→
n
e
P
as
∞
→
n
.
7.12
Feedback Capacity (of a DMC)
C
C
FB
=
on a DMC