1
Colorado State University, Ft. Collins
Fall 2008
ECE 516: Information Theory
Lecture 15 October 21, 2008
Recap:
7.6
Joint Typical Sequences
Definition: The set
()
n
A
ε
of jointly typical sequences with respect to joint PMF
y
x
p
,
is the set of
( )
n
n
y
x
,
whose empirical entropies are
close to the true
entropies.
{
() ()
()()
⎭
⎬
⎫
<
−
−
<
−
−
<
−
−
×
∈
=
Y
X
H
y
x
p
n
Y
H
y
p
n
X
H
x
p
n
y
x
A
n
n
n
n
n
n
n
n
n
,
,
log
1
log
1
,
log
1
:
,
Y
X
Theorem:
(Joint
AEP)
Let
( )
n
n
Y
X
,
be
drawn
iid
according
to
∏
=
=
n
i
i
i
n
n
y
x
p
y
x
p
1
,
,
1.
( )
( )
1
,
lim
=
∈
∞
→
n
n
n
n
A
y
x
p
, i.e.,
( )
(
)
( )
−
>
∈
∞
→
1
,
lim
n
n
n
n
A
y
x
p
for large enough
n
.
2.
(
)
(
)
(
)
+
−
≤
≤
−
Y
X
H
n
n
Y
X
H
n
A
,
,
2
2
1
3. If
( ) ( ) ( )
n
n
n
n
y
p
x
p
y
x
~
~
~
~
,
~
, i.e.,
n
x
~
and
n
y
~
are independent with the marginals
( )
n
x
p
and
( )
n
y
p
obtained from
( )
n
n
y
x
p
,
, then
( )
(
)
( )
(
)
(
)
3
;
3
;
2
~
,
~
2
1
−
−
+
−
≤
∈
≤
−
Y
X
I
n
n
n
n
Y
X
I
n
A
y
x
p
Achievability (direct part):
All rates below
C
are achievable. That is, if
R
<
C
, then
one can design a code
( )
n
nR
,
2
such that
(
)
n
λ
,
(
)
0
→
n
e
P
as
∞
→
n
.
7.12
Feedback Capacity (of a DMC)
C
C
FB
=
on a DMC
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document2
7.13
Joint Sourcechannel Coding Theorem
So far, we saw two major results
1. Source Coding:
()
( )
ε
+
<
⎥
⎦
⎤
⎢
⎣
⎡
<
X
H
n
X
l
E
X
H
n
O
r
H
R
>
is required for lossless compression
2. Channel Coding:
C
R
<
is required for reliable communication
Two questions:
1.
Is
C
H
<
sufficient and necessary?
2.
Do we lose anything by a two step process? Separate source and channel coding?
Recall source coding:
We work on a block of
n
symbols. With almost probability 1, the sequence belongs
to the typical set. The typical set contains
nH
2
sequences.
[]
( )
+
=
+
×
≈
H
n
n
nH
X
l
E
n
n
X
log
1
1
Recall channel coding:
[] [ ] [ ]
[] []
Y
X
I
R
E
P
E
P
E
E
E
P
W
P
P
R
Y
X
I
n
Y
X
nI
nR
i
i
c
c
nR
nR
;
if
2
2
2
2
1

;
;
2
2
1
2
2
1
<
≈
+
≈
+
≈
+
≤
∪
∪
∪
=
=
=
−
−
−
=
∑
L
If
C
R
<
, then
2
≤
P
Theorem: (Sourcechannel coding theorem) If
n
V
V
V
,
,
,
2
1
L
is a finitealphabet
This is the end of the preview. Sign up
to
access the rest of the document.
 Spring '08
 Rocky
 Information Theory, AEP, sourcechannel coding theorem

Click to edit the document details