Discrete-time stochastic processes

# This result namely that past and future are

This preview shows page 1. Sign up to view the full content.

This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: h Pj m > 0 and Pmi > 0. For a contradiction, again assume that Fij (1) < 1. From (5.5), X X Fmj (1) = Pmj + Pmk Fkj (1) < Pmj + Pmk = 1, k6=j k6=j where the strict inequality follows since Pmi Fij (1) < Pmi . This is a contradiction, since m is accessible from j in one step, and thus Fmj (1) = 1. It follows that every i accessible from j in two steps satisﬁes Fij (1) = 1. Extending the same argument for successively larger numbers of steps, the conclusion of the lemma follows. Lemma 5.4. Let {Nij (t); t ≥ 0} be the counting process for transitions into state j up to time t for a Markov chain given X0 = i 6= j . Then if i and j are in the same recurrent class, {Nij (t); t ≥ 0} is a delayed renewal process. Proof: From Lemma 5.3, Tij , the time until the ﬁrst transition into j , is a rv. Also Tj j is a rv by deﬁnition of recurrence, and subsequent intervals between occurrences of state j are IID, completing the proof. If Fij (1) = 1, we have seen that the ﬁrst-passage time from i to j is a rv, i.e., is ﬁnite with probability 1. In this case, the mean time T ij to ﬁrst enter state j starting from state i is of interest. Since Tij is a non-negative random variable, its expectation is the integral of its complementary distribution function, T ij = 1 + 1 X (1 − Fij (n)). (5.7) n=1 It is possible to have Fij (1) = 1 but T ij = 1. As will be shown in Section 5.3, the chain in Figure 5.1 satisﬁes Fij (1) = 1 and T ij < 1 for p < 1/2 and Fij (1) = 1 and T ij = 1 for p = 1/2. As discussed before, Fij (1) < 1 for p > 1/2. This leads us to the following deﬁnition. 5.1. INTRODUCTION AND CLASSIFICATION OF STATES 203 Deﬁnition 5.3. A state j in a countable-state Markov chain is positive-recurrent if Fj j (1) = 1 and T j j < 1. It is null-recurrent if Fj j (1) = 1 and T j j = 1. Each state of a Markov chain is thus classiﬁed as one of the following three types — positiverecurrent, null-recurrent, or transient. For the example of Figure 5.1, null-recurrence lies on a boundary between positive-recurrence and transience, and this is often a good way to look at null-recurrence. Part f ) of Exercise 6.1 illustrates another type of situation in which null-recurrence can occur. Assume that state j is recurrent and consider the renewal process {Nj j (t); t ≥ 0}. The limiting theorems for renewal processes can be applied directly. From the strong law for renewal processes, Theorem 3.1, lim Nj j (t)/t = 1/T j j t→1 with probability 1. (5.8) From the elementary renewal theorem, Theorem 3.4, lim E [Nj j (t)/t] = 1/T j j . t→1 (5.9) Equations (5.8) and (5.9) are valid whether j is positive-recurrent or null-recurrent. Next we apply Blackwell’s theorem to {Nj j (t); t ≥ 0}. Recall that the period of a given state j in a Markov chain (whether the chain has a countable or ﬁnite number of states) is the greatest common divisor of the set of integers n > 0 such that Pjn > 0. If this period j is d, then {Nj j (t); t ≥ 0} is arithmetic with span d (i.e., renewals occur only at times that are multiples of d). From Blackwell’s theorem in the arithmetic form of (3.20), lim Pr {Xnd = j | X0 = j } = d/T j j . n→1 (5.10) If state j is aperiodic (i.e., d = 1), this says that limn→1 Pr {Xn = j | X0 = j } = 1/T j j . Equations (5.8) and (5.9) suggest that 1/T j j has some of the properties associated with a steady-state probability of state j , and (5.10) strengthens this if j is aperiodic. For a Markov chain consisting of a single class of states, all positive-recurrent, we will strengthen this association further in Theorem 5.4 by showing that there is a unique P ady-state ste distribution, {P, j ≥ 0} such that πj = 1/T j j for all j and such that πj = i πi Pij for πj all j ≥ 0 and j πj = 1. The following theorem starts this development by showing that (5.8-5.10) are independent of the starting state. Theorem 5.2. Let j be a recurrent state in a Markov chain and let i be any state in the same class as j . Given X0 = i, let Nij (t) be the number of transitions into state j by time t and let T j j be the expected recurrence time of state j (either ﬁnite or inﬁnite). Then lim Nij (t)/t = 1/T j j t→1 with probability 1 lim E [Nij (t)/t] = 1/T j j . t→1 (5.11) (5.12) If j is also aperiodic, then lim Pr {Xn = j | X0 = i} = 1/T j j . n→1 (5.13) 204 CHAPTER 5. COUNTABLE-STATE MARKOV CHAINS Proof: Since i and j are recurrent and in the same class, Lemma 5.4 asserts that {Nij (t); t ≥ 0} is a delayed renewal process for j 6= i. Thus (5.11) and (5.12) follow from Theorems 3.9 and 3.10 of Chapter 3. If j is aperiodic, then {Nij (t); t ≥ 0} is a delayed renewal process for which the inter-renewal intervals Tj j have span 1 and Tij has an integer span. Thus, (5.13) follows from Blackwell’s theorem for delayed renewal processes, Theorem 3.11. For i = j , Equations (5.11-5.13) follow from (5.8-5.10), completing the proof. Theorem 5.3. Al l states in the same class of a Markov chain are of the same type — either al l positive-recurrent, al l nul l-recurrent, o...
View Full Document

## This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.

Ask a homework question - tutors are online