Harvard SEAS
ES250 – Information Theory
2
Jointly Typical Sequences
Definition
The set
A
(
n
)
²
of
²
typical
n
sequences (
x
1
,
x
2
,
· · ·
,
x
k
) is defined by
A
(
n
)
²
(
X
(1)
, X
(2)
,
· · ·
, X
(
k
)
) =
‰
(
x
1
,
x
2
,
· · ·
,
x
k
) :
fl
fl
fl
fl

1
n
log
p
(
s
)

H
(
S
)
fl
fl
fl
fl
< ²,
∀
S
⊆ {
X
(1)
, X
(2)
,
· · ·
, X
(
k
)
}
.
Definition
We will use the notation
a
n
.
= 2
n
(
b
±
²
)
to mean that
fl
fl
fl
fl
1
n
log
a
n

b
fl
fl
fl
fl
< ²
for
n
sufficiently large.
Theorem
For any
² >
0, for sufficiently large
n
,
1.
P
(
A
(
n
)
²
(
S
))
≥
1

²
,
∀
S
⊆ {
X
(1)
, X
(2)
,
· · ·
, X
(
k
)
}
.
2.
s
∈
A
(
n
)
²
(
S
)
⇒
p
(
s
)
.
= 2
n
(
H
(
S
)
±
²
)
.
3.

A
(
n
)
²
(
S
)

.
= 2
n
(
H
(
S
)
±
2
²
)
.
4. Let
S
1
, S
2
⊆ {
X
(1)
, X
(2)
,
· · ·
, X
(
k
)
}
. If (
s
1
,
s
2
)
∈
A
(
n
)
²
(
S
1
, S
2
), then
p
(
s
1
,
s
2
)
.
= 2
n
(
H
(
S
1

S
2
)
±
2
²
)
Theorem
Let
S
1
,
S
2
be two subsets of
X
(1)
, X
(2)
,
· · ·
, X
(
k
)
. For any
² >
0, define
A
(
n
)
²
(
S
1

s
2
) to be the
set of
s
1
sequences that are jointly
²
typical with a particular
s
2
sequence. If
s
2
∈
A
(
n
)
²
(
S
2
), then for
sufficiently large
n
, we have

A
(
n
)
²
(
S
1

s
2
)
 ≤
2
n
(
H
(
S
1

S
2
)+2
²
)
and
(1

²
)2
n
(
H
(
S
1

S
2
)

2
²
)
≤
X
s
2
p
(
s
2
)

A
(
n
)
²
(
S
1

s
2
)

.
Theorem
Let
A
(
n
)
²
denote the typical set for the probability mass function
p
(
s
1
, s
2
, s
3
), and let
P
(
S
0
1
=
s
1
,
S
0
2
=
s
2
,
S
0
3
=
s
3
) =
n
Y
i
=1
p
(
s
1
i

s
3
i
)
p
(
s
2
i

s
3
i
)
p
(
s
3
i
)
.
Then
P
{
(
S
0
1
,
S
0
2
,
S
0
3
)
∈
A
(
n
)
²
}
.
= 2
n
(
I
(
S
1
;
S
2

S
3
)
±
6
²
)
.
3
Multiple Access Channel
Definition
A
discrete memoryless multipleaccess channel
consists of three alphabets,
X
1
,
X
2
, and
Y
, and
a probability transition matrix
p
(
y

x
1
, x
2
).
2