1
Colorado State University, Ft. Collins
Fall 2008
ECE 516: Information Theory
Lecture 7 September 18, 2008
Recap:
3
The Asymptotic Equipartition Property (Chapter 3)
Markov Inequality
: For a nonnegative r.v.,
X
and a constant
0
>
c
[
]
[
]
c
X
E
c
x
P
≤
≥
Chebyshev Inequality
: For a r.v.,
Y
and a constant
0
>
c
[ ]
[
]
[ ]
2
c
Y
Var
c
Y
E
y
P
≤
≥
−
Sample Mean
: Let
n
X
X
,
,
1
L
be
n
iid r.v.’s with mean
X
μ
and variance
2
X
σ
. Then
n
X
X
Y
n
n
+
+
=
L
1
is a r.v. with mean
X
μ
and variance
n
X
2
σ
Weak Law of Large Numbers (WLLN)
:
X
n
n
n
X
X
Y
μ
→
+
+
=
L
1
in probability
i.e.
[
]
1
lim
=
<
−
∞
→
ε
μ
X
n
n
y
P
for any
0
>
ε
Theorem (AEP)
If
L
,
,
2
1
X
X
are iid with
(
)
X
p
, then
(
)
(
)
X
H
X
X
p
n
n
→
−
,
,
log
1
1
L
in probability
Definition: The typical set
(
)
n
A
ε
with respect to
( )
x
p
is the set of sequences
(
)
n
n
x
x
X
∈
,
,
1
L
with the following property:
(
)
(
)
(
)
(
)
(
)
ε
ε
−
−
+
−
≤
≤
X
H
n
n
X
H
n
x
x
p
2
,
,
2
1
L

This ** preview** has intentionally

**sections.**

*blurred***to view the full version.**

*Sign up*