Discrete-time stochastic processes

44 perron frobenius theory a real vector x ie a vector

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: a class is transient if there are any directed arcs going from a node in the class to a node outside the class. Every finite-state Markov chain must have at least one recurrent class of states (see Exercise 4.1), and can have arbitrarily many additional classes of recurrent states and transient states. States can also be classified according to their periods (see Figure 4.2). In fig. 4.2(a), given that X0 = 2, we see that X1 must be either 1 or 3, X2 must then be either 2 or 4, and in general, Xn must be 2 or 4 for n even and 1 or 3 for n odd. On the other hand, if X0 is 1 or 3, then Xn is 2 or 4 for n odd and 1 or 3 for n even. Thus the effect of the starting state never dies out. Fig. 4.2(b) illustrates another example in which the state alternates from odd to even and the memory of the starting state never dies out. The states in both these Markov chains are said to be periodic with period 2. ❧ 1 ② ✗ ✄ ③ ❧ 2 ✎ ✄ ❧ 4 ② ✎ ③✄ ❧ 3 ✗ ✄ (a) ✒ ° ° ❧ 7 ❧ ✲9 ❧ 8 ■ ❅ ❅❧ 6✛ ❧ 1 ❅ ✒ °❅ ❘° ❅❧ ❘ ❅❧ 4 2 °❅ ° ■ ✠ ° ✠ ° ❅❧ ❧ 5 3 (b) Figure 4.2: Periodic Markov Chains Definition 4.7. The period of a state i, denoted d(i), is the greatest common divisor (g cd) n of those values of n for which Pii > 0. If the period is 1, the state is aperiodic, and if the period is 2 or more, the state is periodic.3 n For example, in Figure 4.2(a), P11 > 0 for n = 2, 4, 6, . . . . Thus d(1), the period of state 1, is two. Similarly, d(i) = 2 for the other states in Figure 4.2(a). For fig. 4.2(b), we have 2 This theorem is also true for Markov chains with a countable state space, but the proof here is inadequate. Also recurrent classes with a countable state space are further classified into either positive-recurrent or nullrecurrent, a distinction that does not appear in the finite-state case. 3 n For completeness, we say that the period is infinite if Pii = 0 for all n ≥ 1. Such states do not have the intuitive characteristics of either periodic or aperiodic states. Such a state cannot communicate with any other state, and cannot return to itself, so it corresponds to a singleton class of transient states. The notion of periodicity is of primary interest for recurrent states. 144 CHAPTER 4. FINITE-STATE MARKOV CHAINS n P11 > 0 for n = 4, 8, 10, 12, . . . ; thus d(1) = 2, and it can be seen that d(i) = 2 for all the states. These examples suggest the following theorem. Theorem 4.2. For any Markov chain (with either a finite or countably infinite number of states), al l states in the same class have the same period. Proof: Let i and j be any distinct pair of states in a class. Then i ↔ j and there is some r r such that Pij > 0 and some s such that Pjsi > 0. Since there is a walk of length r + s going from i to j and back to i, r + s must be divisible by d(i). Let t be any integer such that Pjtj > 0. Since there is a walk of length r + t + s that goes first from i to j , then to j again, and then back to i, r + t + s is divisible by d(i), and thus t is divisible by d(i). Since this is true for any t such that Pjtj > 0, d(j ) is divisible by d(i). Reversing the roles of i and j , d(i) is divisible by d(j ), so d(i) = d(j ). Since the states in a class all have the same period and are either all recurrent or all transient, we refer to the class itself as having the period of its states and as being recurrent or transient. Similarly if a Markov chain has a single class of states, we refer to the chain as having the corresponding period and being recurrent or transient. Theorem 4.3. If a recurrent class in a finite-state Markov chain has period d, then the states in the class can be partitioned into d subsets, S1 , S2 , . . . , Sd , such that al l transitions out of subset Sm go to subset Sm+1 for m < d and to subset S1 for m = d. That is, if j ∈ Sm and Pj k > 0, then k ∈ Sm+1 for m < d and k ∈ S1 for m = d. Proof: See Figure 4.3 for an illustration of the theorem. For a given state in the class, say state 1, define the sets S1 , . . . , Sd by nd Sm = {j : P1j +m > 0 for some n ≥ 0}; 1 ≤ m ≤ d. (4.6) For each given j in the class, we first show that there is one and only one value of m such r that j ∈ Sm . Since 1 ↔ j , there is some r for which P1j > 0 and some s for which Pjs1 > 0. Since there is a walk from 1 to 1 (through j ) of length r + s, r + s is divisible by d. Define m, 1 ≤ m ≤ d, by r = m + nd, where n is an integer. From (4.6), j ∈ Sm . Now let r0 be any r0 other integer such that P1j > 0. Then r0 + s is also divisible by d, so that r0 − r is divisible by d. Thus r0 = m + n0 d for some integer n0 and that same m. Since r0 is any integer such r0 that P1j > 0, j is in Sm for only that one value of m. Since j is arbitrary, this shows that the sets Sm are disjoint and partition the class. Finally, suppose j ∈ Sm and Pj k > 0. Given a walk of length r = nd + m from state 1 to j , there is a walk of length nd...
View Full Document

This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.

Ask a homework question - tutors are online