This preview shows pages 1–5. Sign up to view the full content.
Module 3, Lecture 1
The Asymptotic Equipartition Property
G.L. Heileman
Module 3, Lecture 1
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document Asymptotic Equipartition Property
Consider an experiment in which you tossed a fair coin 100
times.
Would you be surprised if you observed 100 heads in a row?
Well, since the process is iid, the sequence of 100 heads in a
row has the same probability as any other 100element
sequence.
Intuitively, however, we would know that something is not
“right” if we tossed 100 heads in a row. We expect to see
about the same number of heads and tails in the sequence if
the coin is fair.
What we need is the right probabilistic tool for analyzing this
situation — something that captures the idea that the
number of heads and tails should be about the same.
G.L. Heileman
Module 3, Lecture 1
Asymptotic Equipartition Property
Let
X
1
, . . . ,
X
n
by iid
∼
p
(
x
). If
p
1
, . . . ,
p
X
are the
probabilities associated with the outcomes of this discrete RV,
then if we consider a long sequence (i.e.,
n
large), it should
contain with high probability
p
1
n
occurrences of the ﬁrst
outcome,
p
2
n
occurrences of the second outcome, etc.
Thus, the probability of a particular (long) sequence should be
roughly equal to
p
p
1
n
1
·
p
p
2
n
2
···
p
p
n
.
Ex.
Consider
X
=
{
0
,
1
}
,
p
= Pr
{
X
= 1
}
,
q
= Pr
{
X
= 0
}
= 1

p
.
Then the sequence 10011011 has probability
p
5
q
3
.
Furthermore, we expect that a long sequence of length
n
drawn
according to this distribution will contain roughly
p
·
n
1s and
(1

p
)
·
n
0s.
G.L. Heileman
Module 3, Lecture 1
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document Asymptotic Equipartition Property
The
asymptotic equipartition property (AEP)
relates the
asymptotic behavior of iid sequences to entropy.
We will use the AEP to divide the set of all sequences derived
from an iid process into two sets, those that are typical, and
those that are not.
We will show that almost all of the probability is associated
with the elements in the typical set. Thus, if we prove some
property about the typical set, it will apply with high
probability to
any
long sequence drawn according to the iid
process. In other words, we will be proving something about
the expected behavior of a long sequence.
In this module, we will show how the AEP is related to source
coding.
We will later extend the AEP in order to prove the
joint AEP
for certain “joint” sequences of RVs, and use this result to
prove important results in channel coding.
The AEP is a direct consequence of the Weak LLN.
This is the end of the preview. Sign up
to
access the rest of the document.
This note was uploaded on 05/06/2010 for the course ECE 549 taught by Professor G.l.heileman during the Spring '10 term at University of New Brunswick.
 Spring '10
 G.L.Heileman

Click to edit the document details