Class Lecture Notes: Thursday, February 10.
Markov Chains
The Contraction approach to
π
=
π
P
The limit for aperiodic irreducible ﬁnite-state DTMC’s.
There is a nice simple limit for aperiodic irreducible ﬁnite-state Markov chains. For any
initial probability vector
u
≡
(
u
1
,...,u
m
), the probability vector at time
n
is
P
(
X
n
=
j
) = (
uP
n
)
j
=
m
X
i
=1
u
i
P
n
i,j
.
The key limiting result is
Theorem 0.1
If
P
is the transition matrix of an aperiodic irreducible ﬁnite-state Markov
chain with transition matrix
P
, then, for any initial probability vector
u
,
uP
n
→
π
as
n
→ ∞
,
where the limiting probability vector
π
is the unique stationary probability vector, i.e., the
unique solution to the ﬁxed-point equation
π
=
πP
or
π
j
=
m
X
i
=1
π
i
P
i,j
for all
j ,
where
π
j
≥
0
for all
j
and
∑
j
π
j
= 1
.
Note the conditions: Of course, irreducibility is essential. And aperiodicity is essential to get
full convergence, as opposed to convergence of averages, or convergence through appropriate
subsequences. The method of proof here is designed to apply to ﬁnite-state chains. The
proof extends to inﬁnite-state chains under the condition that there is some state
j
such that
P
i,j
≥
² >
0 for all states
i
, or
P
k
i,j
≥
² >
0 for some
k
. This is a strong extra condition saying
that there is a state
j
such that there is a probability of at least
² >
0 of going to
j
in one step
(or in
k
steps, as a weaker version of the same condition), from any other state. With that
extra condition, we not only get convergence, we get convergence quickly, geometrically fast.
We actually provide a proof without this condition, but we do not get such quick convergence
unless the condition holds.
The Contraction Proof.
One way to prove this result and others is to apply renewal theory. An alternative way to
prove the theorem is to consider the transition matrix
P
as an operator on the space of all
probability vectors, here taken to be of dimension
m
, corresponding to there being
m
states.
An operator on a space maps the space into itself. If