Discrete-time stochastic processes

Property 3 let x be any eigenvector of property 2

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: + m + 1 from state 1 to k. It follows that if m < d, then k ∈ Sm+1 and if m = d, then k ∈ S1 , completing the proof. We have seen that each class of states (for a finite-state chain) can be classified both in terms of its period and in terms of whether or not it is recurrent. The most important case is that in which a class is both recurrent and aperiodic. Definition 4.8. For a finite-state Markov chain, an ergodic class of states is a class that 4.2. CLASSIFICATION OF STATES 145 ❣ ❣ P P ❍ P ❑ ❆ S2 ❆ ❣ ✻ ❍❍ PP ❍❍ PP PP ❆ PP PP ❍❍ ❣ ▼ q P ③ ❇ S1 ❆ ❇ PP PP ❍❍ ❆❇ PP ❍ ❆❇ PP❍❍ S3 ❆❇ PP❍ P❣ ❥ ❍ q P ❇✏ ❆❣ ✏ ✮ ✛ Figure 4.3: Structure of a Periodic Markov Chain with d = 3. Note that transitions only go from one subset Sm to the next subset Sm+1 (or from Sd to S1 ). is both recurrent and aperiodic4 . A Markov chain consisting entirely of one ergodic class is cal led an ergodic chain. n We shall see later that these chains have the desirable property that Pij becomes independent of the starting state i as n → 1. The next theorem establishes the first part of this n by showing that Pij > 0 for all i and j when n is sufficiently large. The Markov chain in Figure 4.4 illustrates the theorem by illustrating how large n must be in the worst case. ✓✏ ✒✑ ✒ ° ° ✓✏ ° 5 ✒✑ ■ ❅ ❅ ✓✏ ❅ 3✛ ✒✑ 4 ✓✏ ✲6 ✒✑ ❅ ❅ ✓✏ ❘ ❅ ✒✑ ° ✓✏ ❄° ✠ ° 1 ✒✑ 2 m Figure 4.4: An ergodic chain with M = 6 states in which Pij > 0 for all m > (M − 1)2 (M−1)2 and all i, j but P11 = 0 The figure also illustrates that an M state Markov chain must have a cycle with M − 1 or fewer nodes. To see this, note that an ergodic chain must have cycles, since each node must have a walk to itself, and any subcycle of repeated nodes can be omitted from that walk, converting it into a cycle. Such a cycle might have M nodes, but a chain with only a M node cycle would be periodic. Thus some nodes must be on smaller cycles, such as the cycle of length 5 in the figure. m Theorem 4.4. For an ergodic M state Markov chain, Pij > 0 for al l i, j , and al l m ≥ 2 + 1. (M − 1) 4 For Markov chains with a countably infinite state space, ergodic means that the states are positiverecurrent and aperiodic (see Chapter 5, Section 5.1). 146 CHAPTER 4. FINITE-STATE MARKOV CHAINS Proof*:5 As shown in Figure 4.4, the chain must contain a cycle with fewer than M nodes. Let τ ≤ M − 1 be the number of nodes on a smallest cycle in the chain and let i be any given state on such a cycle. Define T (m), m ≥ 1, as the set of states accessible from the fixed state i in m steps. Thus T (1) = {j : Pij > 0}, and for arbitrary m ≥ 1, m T (m) = {j : Pij > 0}. (4.7) τ Since i is on a cycle of length τ , Pii > 0. For any m ≥ 1 and any j ∈ T (m), we can then construct an m + τ step walk from i to j by going from i to i in τ steps and then to j in another m steps. This is true for all j ⊆ T (m), so T (m) ⊆ T (m + τ ). (4.8) By defining T (0) to be the singleton set {i}, (4.8) also holds for m = 0, since i ∈ T (τ ). By starting with m = 0 and iterating on (4.8), T (0) ⊆ T (τ ) ⊆ T (2τ ) ⊆ · · · ⊆ T (nτ ) ⊆ · · · . (4.9) We now show that if one of the inclusion relations in (4.9) is satisfied with equality, then all the subsequent relations are satisfied with equality. More generally, assume that T (m) = T (m + s) for some m ≥ 0 and s ≥ 1. Note that T (m + 1) is the set of states that can be reached in one step from states in T (m), and similarly T (m + s + 1) is the set reachable in one step from T (m + s) = T (m). Thus T (m + 1) = T (m + 1 + s). Iterating this result, T (m) = T (m + s) implies T (n) = T (n + s) for all n ≥ m. (4.10) Thus, (4.9) starts with strict inclusions and then continues with strict equalities. Since the entire set has M members, there can be at most M − 1 strict inclusions in (4.9). Thus T ((M − 1)τ ) = T (nτ ) for all integers n ≥ M − 1. (4.11) Define k as (M − 1)τ . We can then rewrite (4.11) as T (k) = T (k + j τ ) for all j ≥ 1. (4.12) We next show that T (k) consists of all M nodes in the chain. The central part of this is to t show that T (k) = T (k + 1). Let t be any positive integer other than τ such that Pii > 0. Letting m = k in (4.8) and using t in place of τ , T (k) ⊆ T (k + t) ⊆ T (k + +2t) ⊆ · · · ⊆ T (k + τ t). (4.13) Since T (k + τ t) = T (k), this shows that T (k) = T (k + t). (4.14) Now let s be the smallest positive integer such that T (k) = T (k + s). 5 Proofs marked with an asterisk can be omitted without loss of continuity. (4.15) 4.3. THE MATRIX REPRESENTATION 147 From (4.11), we see that (4.15) holds when s takes the value τ . Thus, the minimizing s must lie in the range 1 ≤ s ≤ τ . We will show that s = 1 by assuming s > 1 and establishing a contradiction. Since the chain is aperiodic, there is some t not divisible by s for which t Pii > 0. This t can be repre...
View Full Document

Ask a homework question - tutors are online