2
7.6
Joint Typical Sequences
Definition: The set
()
n
A
ε
of jointly typical sequences with respect to joint PMF
y
x
p
,
is the set of
( )
n
n
y
x
,
whose empirical entropies are
close to the true
entropies.
{
() ()
()()
⎭
⎬
⎫
<
−
−
<
−
−
<
−
−
×
∈
=
Y
X
H
y
x
p
n
Y
H
y
p
n
X
H
x
p
n
y
x
A
n
n
n
n
n
n
n
n
n
,
,
log
1
log
1
,
log
1
:
,
Y
X
Where
∏
=
=
n
i
i
i
n
n
y
x
p
y
x
p
1
,
,
Theorem:
(Joint
AEP)
Let
( )
n
n
Y
X
,
be
drawn
iid
according
to
∏
=
=
n
i
i
i
n
n
y
x
p
y
x
p
1
,
,
1.
( )
( )
1
,
lim
=
∈
∞
→
n
n
n
n
A
y
x
p
, i.e.,
( )
(
)
( )
−
>
∈
∞
→
1
,
lim
n
n
n
n
A
y
x
p
for large enough
n
.
2.
(
)
(
)
(
)
+
−
≤
≤
−
Y
X
H
n
n
Y
X
H
n
A
,
,
2
2
1
3. If
( ) ( ) ( )
n
n
n
n
y
p
x
p
y
x
~
~
~
~
,
~
, i.e.,
n
x
~
and
n
y
~
are independent with the marginals
( )
n
x
p
and
( )
n
y
p
obtained from
( )
n
n
y
x
p
,
, then
( )
(
)
( )
(
)
(
)
3
;
3
;
2
~
,
~
2
1
−
−
+
−
≤
∈
≤
−
Y
X
I
n
n
n
n
Y
X
I
n
A
y
x
p
The Big Picture
a) For a typical
n
x
( ) ( ) ( )
( )
∑
∑
∑
∉
∈
+
=
=
n
n
n
n
n
A
y
n
n
A
y
n
n
y
n
n
n
y
x
p
y
x
p
y
x
p
x
p
,
,
,
(
)
Y
X
nH
X
nH
L
,
2
2
−
−
=
For each typical
n
x
, there are about
(
)
X
Y
nH
X
nH
Y
X
nH
L

,
2
2
2
=
=
jointly typical
n
y
sequences.