This preview shows pages 1–4. Sign up to view the full content.
1
Colorado State University, Ft. Collins
Fall 2008
ECE 516: Information Theory
Lecture 7 September 18, 2008
Recap:
3
The Asymptotic Equipartition Property (Chapter 3)
Markov Inequality
: For a nonnegative r.v.,
X
and a constant
0
>
c
[]
c
X
E
c
x
P
≤
≥
Chebyshev Inequality
: For a r.v.,
Y
and a constant
0
>
c
[ ]
2
c
Y
Var
c
Y
E
y
P
≤
≥
−
Sample Mean
: Let
n
X
X
,
,
1
L
be
n
iid r.v.’s with mean
X
μ
and variance
2
X
σ
. Then
n
X
X
Y
n
n
+
+
=
L
1
is a r.v. with mean
X
and variance
n
X
2
Weak Law of Large Numbers (WLLN)
:
X
n
n
n
X
X
Y
→
+
+
=
L
1
in probability
i.e.
1
lim
=
<
−
∞
→
ε
X
n
n
y
P
for any
0
>
Theorem (AEP)
If
L
,
,
2
1
X
X
are iid with
( )
X
p
, then
()
(
)
X
H
X
X
p
n
n
→
−
,
,
log
1
1
L
in probability
Definition: The typical set
(
)
n
A
with respect to
( )
x
p
is the set of sequences
n
n
x
x
X
∈
,
,
1
L
with the following property:
(
)
(
)
−
−
+
−
≤
≤
X
H
n
n
X
H
n
x
x
p
2
,
,
2
1
L
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document 2
Theorem:
(Properties of the typical set)
1.
If
()
n
n
A
x
x
ε
∈
,
,
1
L
then
(
)
+
≤
−
≤
−
X
H
x
x
p
n
X
H
n
,
,
log
1
1
L
2.
[ ]
−
>
1
n
A
P
for sufficiently large
n
.
3.
+
≤
X
H
n
n
A
2
4.
−
−
≥
X
H
n
n
A
2
1
Consequences of AEP: Data compression
Let
n
X
X
,
,
1
L
be an iid sequence with
( )
x
p
and
( )
H
x
H
=
. We want to compress
(find short description for) sequences
( )
n
n
x
x
X
∈
,
,
1
L
.
There are
X
X
log
2
n
n
=
possible sequences.
i)
+
≤
H
n
2
are in
n
A
. Use
( )
1
+
+
H
n
buts to index them. Prefix each with a 0.
Total bits per sequence
( )
2
+
+
=
H
n
.
ii)
+
−
−
≤
H
n
n
2
1
X
sequences not in
(
)
n
A
. Just assume
n
X
sequences
remaining. Use
1
log
+
X
n
bits to index them and prefix each with a 1. Total bits
per sequence
2
log
+
=
X
n
Let
( )
n
X
l
be the length of the codeword. It is random.
( )
[ ]
′
+
≤
H
n
X
l
E
n
Where
n
2
log
+
+
=
′
X
, which can be made arbitrarily small.
3
Theorem:
Let
n
X
X
,
,
1
L
be iid with
( )
x
p
and
( )
X
H
. Let
0
>
ε
. Then, there
exists a code which maps sequences
( )
n
x
x
,
,
1
L
or length n into binary strings (of
variable lengths), such that the mapping is onetoone (and therefore invertible) and
()
+
≤
⎥
⎦
⎤
⎢
⎣
⎡
X
H
n
X
l
E
n
I.e., you can compress each symbol into
( )
+
X
H
bits on average.
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
This is the end of the preview. Sign up
to
access the rest of the document.
This note was uploaded on 03/17/2010 for the course ECE 516 taught by Professor Rocky during the Spring '08 term at Colorado State.
 Spring '08
 Rocky

Click to edit the document details