This preview shows page 1. Sign up to view the full content.
Unformatted text preview: a class is transient if there are any directed arcs
going from a node in the class to a node outside the class. Every ﬁnitestate Markov chain
must have at least one recurrent class of states (see Exercise 4.1), and can have arbitrarily
many additional classes of recurrent states and transient states.
States can also be classiﬁed according to their periods (see Figure 4.2). In ﬁg. 4.2(a), given
that X0 = 2, we see that X1 must be either 1 or 3, X2 must then be either 2 or 4, and in
general, Xn must be 2 or 4 for n even and 1 or 3 for n odd. On the other hand, if X0 is 1
or 3, then Xn is 2 or 4 for n odd and 1 or 3 for n even. Thus the eﬀect of the starting state
never dies out. Fig. 4.2(b) illustrates another example in which the state alternates from
odd to even and the memory of the starting state never dies out. The states in both these
Markov chains are said to be periodic with period 2.
❧
1
② ✗
✄ ③
❧
2 ✎
✄
❧
4
② ✎
③✄
❧
3 ✗
✄ (a) ✒
°
° ❧
7 ❧ ✲9
❧
8 ■
❅
❅❧
6✛ ❧
1 ❅
✒
°❅
❘°
❅❧
❘
❅❧
4
2
°❅
°
■
✠
°
✠
°
❅❧
❧
5
3 (b) Figure 4.2: Periodic Markov Chains
Deﬁnition 4.7. The period of a state i, denoted d(i), is the greatest common divisor (g cd)
n
of those values of n for which Pii > 0. If the period is 1, the state is aperiodic, and if the
period is 2 or more, the state is periodic.3
n
For example, in Figure 4.2(a), P11 > 0 for n = 2, 4, 6, . . . . Thus d(1), the period of state
1, is two. Similarly, d(i) = 2 for the other states in Figure 4.2(a). For ﬁg. 4.2(b), we have
2
This theorem is also true for Markov chains with a countable state space, but the proof here is inadequate.
Also recurrent classes with a countable state space are further classiﬁed into either positiverecurrent or nullrecurrent, a distinction that does not appear in the ﬁnitestate case.
3
n
For completeness, we say that the period is inﬁnite if Pii = 0 for all n ≥ 1. Such states do not have the
intuitive characteristics of either periodic or aperiodic states. Such a state cannot communicate with any
other state, and cannot return to itself, so it corresponds to a singleton class of transient states. The notion
of periodicity is of primary interest for recurrent states. 144 CHAPTER 4. FINITESTATE MARKOV CHAINS n
P11 > 0 for n = 4, 8, 10, 12, . . . ; thus d(1) = 2, and it can be seen that d(i) = 2 for all the
states. These examples suggest the following theorem. Theorem 4.2. For any Markov chain (with either a ﬁnite or countably inﬁnite number of
states), al l states in the same class have the same period.
Proof: Let i and j be any distinct pair of states in a class. Then i ↔ j and there is some r
r
such that Pij > 0 and some s such that Pjsi > 0. Since there is a walk of length r + s going
from i to j and back to i, r + s must be divisible by d(i). Let t be any integer such that
Pjtj > 0. Since there is a walk of length r + t + s that goes ﬁrst from i to j , then to j again,
and then back to i, r + t + s is divisible by d(i), and thus t is divisible by d(i). Since this
is true for any t such that Pjtj > 0, d(j ) is divisible by d(i). Reversing the roles of i and j ,
d(i) is divisible by d(j ), so d(i) = d(j ).
Since the states in a class all have the same period and are either all recurrent or all
transient, we refer to the class itself as having the period of its states and as being recurrent
or transient. Similarly if a Markov chain has a single class of states, we refer to the chain
as having the corresponding period and being recurrent or transient.
Theorem 4.3. If a recurrent class in a ﬁnitestate Markov chain has period d, then the
states in the class can be partitioned into d subsets, S1 , S2 , . . . , Sd , such that al l transitions
out of subset Sm go to subset Sm+1 for m < d and to subset S1 for m = d. That is, if
j ∈ Sm and Pj k > 0, then k ∈ Sm+1 for m < d and k ∈ S1 for m = d.
Proof: See Figure 4.3 for an illustration of the theorem. For a given state in the class, say
state 1, deﬁne the sets S1 , . . . , Sd by
nd
Sm = {j : P1j +m > 0 for some n ≥ 0}; 1 ≤ m ≤ d. (4.6) For each given j in the class, we ﬁrst show that there is one and only one value of m such
r
that j ∈ Sm . Since 1 ↔ j , there is some r for which P1j > 0 and some s for which Pjs1 > 0.
Since there is a walk from 1 to 1 (through j ) of length r + s, r + s is divisible by d. Deﬁne
m, 1 ≤ m ≤ d, by r = m + nd, where n is an integer. From (4.6), j ∈ Sm . Now let r0 be any
r0
other integer such that P1j > 0. Then r0 + s is also divisible by d, so that r0 − r is divisible
by d. Thus r0 = m + n0 d for some integer n0 and that same m. Since r0 is any integer such
r0
that P1j > 0, j is in Sm for only that one value of m. Since j is arbitrary, this shows that
the sets Sm are disjoint and partition the class.
Finally, suppose j ∈ Sm and Pj k > 0. Given a walk of length r = nd + m from state 1 to
j , there is a walk of length nd...
View
Full
Document
This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.
 Spring '09
 R.Srikant

Click to edit the document details