This preview shows pages 1–2. Sign up to view the full content.
Class Lecture Notes: Thursday, February 10.
Markov Chains
The Contraction approach to
π
=
π
P
The limit for aperiodic irreducible ﬁnitestate DTMC’s.
There is a nice simple limit for aperiodic irreducible ﬁnitestate Markov chains. For any
initial probability vector
u
≡
(
u
1
,...,u
m
), the probability vector at time
n
is
P
(
X
n
=
j
) = (
uP
n
)
j
=
m
X
i
=1
u
i
P
n
i,j
.
The key limiting result is
Theorem 0.1
If
P
is the transition matrix of an aperiodic irreducible ﬁnitestate Markov
chain with transition matrix
P
, then, for any initial probability vector
u
,
uP
n
→
π
as
n
→ ∞
,
where the limiting probability vector
π
is the unique stationary probability vector, i.e., the
unique solution to the ﬁxedpoint equation
π
=
πP
or
π
j
=
m
X
i
=1
π
i
P
i,j
for all
j ,
where
π
j
≥
0
for all
j
and
∑
j
π
j
= 1
.
Note the conditions: Of course, irreducibility is essential. And aperiodicity is essential to get
full convergence, as opposed to convergence of averages, or convergence through appropriate
subsequences. The method of proof here is designed to apply to ﬁnitestate chains. The
proof extends to inﬁnitestate chains under the condition that there is some state
j
such that
P
i,j
≥
² >
0 for all states
i
, or
P
k
i,j
≥
² >
0 for some
k
. This is a strong extra condition saying
that there is a state
j
such that there is a probability of at least
² >
0 of going to
j
in one step
(or in
k
steps, as a weaker version of the same condition), from any other state. With that
extra condition, we not only get convergence, we get convergence quickly, geometrically fast.
We actually provide a proof without this condition, but we do not get such quick convergence
unless the condition holds.
The Contraction Proof.
One way to prove this result and others is to apply renewal theory. An alternative way to
prove the theorem is to consider the transition matrix
P
as an operator on the space of all
probability vectors, here taken to be of dimension
m
, corresponding to there being
m
states.
An operator on a space maps the space into itself. If
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
This is the end of the preview. Sign up
to
access the rest of the document.
 Spring '11
 WhitWard

Click to edit the document details