This preview shows page 1. Sign up to view the full content.
Unformatted text preview: + m + 1 from state 1 to k. It follows that if m < d, then
k ∈ Sm+1 and if m = d, then k ∈ S1 , completing the proof.
We have seen that each class of states (for a ﬁnitestate chain) can be classiﬁed both in
terms of its period and in terms of whether or not it is recurrent. The most important case
is that in which a class is both recurrent and aperiodic.
Deﬁnition 4.8. For a ﬁnitestate Markov chain, an ergodic class of states is a class that 4.2. CLASSIFICATION OF STATES 145 ❣ ❣ P
P
❍ P
❑
❆ S2
❆ ❣ ✻ ❍❍ PP ❍❍ PP
PP
❆
PP
PP
❍❍ ❣
▼
q
P
③ ❇
S1 ❆ ❇
PP
PP ❍❍
❆❇
PP ❍
❆❇
PP❍❍
S3 ❆❇
PP❍
P❣
❥
❍
q
P
❇✏
❆❣
✏
✮
✛ Figure 4.3: Structure of a Periodic Markov Chain with d = 3. Note that transitions
only go from one subset Sm to the next subset Sm+1 (or from Sd to S1 ). is both recurrent and aperiodic4 . A Markov chain consisting entirely of one ergodic class is
cal led an ergodic chain.
n
We shall see later that these chains have the desirable property that Pij becomes independent of the starting state i as n → 1. The next theorem establishes the ﬁrst part of this
n
by showing that Pij > 0 for all i and j when n is suﬃciently large. The Markov chain in
Figure 4.4 illustrates the theorem by illustrating how large n must be in the worst case. ✓✏ ✒✑
✒
°
°
✓✏
° 5 ✒✑
■
❅
❅
✓✏
❅
3✛
✒✑ 4 ✓✏
✲6
✒✑
❅
❅ ✓✏
❘
❅ ✒✑
°
✓✏
❄°
✠
° 1 ✒✑ 2 m
Figure 4.4: An ergodic chain with M = 6 states in which Pij > 0 for all m > (M − 1)2
(M−1)2 and all i, j but P11
= 0 The ﬁgure also illustrates that an M state Markov chain
must have a cycle with M − 1 or fewer nodes. To see this, note that an ergodic chain must
have cycles, since each node must have a walk to itself, and any subcycle of repeated
nodes can be omitted from that walk, converting it into a cycle. Such a cycle might
have M nodes, but a chain with only a M node cycle would be periodic. Thus some
nodes must be on smaller cycles, such as the cycle of length 5 in the ﬁgure. m
Theorem 4.4. For an ergodic M state Markov chain, Pij > 0 for al l i, j , and al l m ≥
2 + 1.
(M − 1)
4 For Markov chains with a countably inﬁnite state space, ergodic means that the states are positiverecurrent and aperiodic (see Chapter 5, Section 5.1). 146 CHAPTER 4. FINITESTATE MARKOV CHAINS Proof*:5 As shown in Figure 4.4, the chain must contain a cycle with fewer than M nodes.
Let τ ≤ M − 1 be the number of nodes on a smallest cycle in the chain and let i be any
given state on such a cycle. Deﬁne T (m), m ≥ 1, as the set of states accessible from the
ﬁxed state i in m steps. Thus T (1) = {j : Pij > 0}, and for arbitrary m ≥ 1,
m
T (m) = {j : Pij > 0}. (4.7) τ
Since i is on a cycle of length τ , Pii > 0. For any m ≥ 1 and any j ∈ T (m), we can then
construct an m + τ step walk from i to j by going from i to i in τ steps and then to j in
another m steps. This is true for all j ⊆ T (m), so T (m) ⊆ T (m + τ ). (4.8) By deﬁning T (0) to be the singleton set {i}, (4.8) also holds for m = 0, since i ∈ T (τ ). By
starting with m = 0 and iterating on (4.8),
T (0) ⊆ T (τ ) ⊆ T (2τ ) ⊆ · · · ⊆ T (nτ ) ⊆ · · · . (4.9) We now show that if one of the inclusion relations in (4.9) is satisﬁed with equality, then all
the subsequent relations are satisﬁed with equality. More generally, assume that T (m) =
T (m + s) for some m ≥ 0 and s ≥ 1. Note that T (m + 1) is the set of states that can be
reached in one step from states in T (m), and similarly T (m + s + 1) is the set reachable in
one step from T (m + s) = T (m). Thus T (m + 1) = T (m + 1 + s). Iterating this result,
T (m) = T (m + s) implies T (n) = T (n + s) for all n ≥ m. (4.10) Thus, (4.9) starts with strict inclusions and then continues with strict equalities. Since the
entire set has M members, there can be at most M − 1 strict inclusions in (4.9). Thus
T ((M − 1)τ ) = T (nτ ) for all integers n ≥ M − 1. (4.11) Deﬁne k as (M − 1)τ . We can then rewrite (4.11) as
T (k) = T (k + j τ ) for all j ≥ 1. (4.12) We next show that T (k) consists of all M nodes in the chain. The central part of this is to
t
show that T (k) = T (k + 1). Let t be any positive integer other than τ such that Pii > 0.
Letting m = k in (4.8) and using t in place of τ ,
T (k) ⊆ T (k + t) ⊆ T (k + +2t) ⊆ · · · ⊆ T (k + τ t). (4.13) Since T (k + τ t) = T (k), this shows that
T (k) = T (k + t). (4.14) Now let s be the smallest positive integer such that
T (k) = T (k + s).
5 Proofs marked with an asterisk can be omitted without loss of continuity. (4.15) 4.3. THE MATRIX REPRESENTATION 147 From (4.11), we see that (4.15) holds when s takes the value τ . Thus, the minimizing s must
lie in the range 1 ≤ s ≤ τ . We will show that s = 1 by assuming s > 1 and establishing
a contradiction. Since the chain is aperiodic, there is some t not divisible by s for which
t
Pii > 0. This t can be repre...
View Full
Document
 Spring '09
 R.Srikant

Click to edit the document details