Discrete-time stochastic processes

# Since each individual generates descendants

This preview shows page 1. Sign up to view the full content.

This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: st be calculated for all i before proceeding to calculate them for the next larger value of n. This also gives us fj j (n), although fj j (n) is not used in the iteration. 1 We say ‘essentially forms a renewal process’ because we haven’t yet speciﬁed the exact conditions upon which these returns to a given state form a renewal process. Note, however, that since we start the process in state j at time 0, the time at which the ﬁrst renewal occurs is the same as the interval between successive renewals. 5.1. INTRODUCTION AND CLASSIFICATION OF STATES 199 Let Fij (n), for n ≥ 1, be the probability, given X0 = i, that state j occurs at some time between 1 and n inclusive. Thus, Fij (n) = n X fij (m). (5.3) m=1 For each i, j , Fij (n) is non-decreasing in n and (since it is a probability) is upper bounded by 1. Thus Fij (1), i.e., limn→1 Fij (n) must exist, and is the probability, given X0 = i, that state j will ever occur. If Fij (1) = 1, then, given X0 = i, it is certain (with probability 1) that the chain will eventually enter state j . In this case, we can deﬁne a random variable (rv) Tij , conditional on X0 = i, as the ﬁrst passage time from i to j . Then fij (n) is the probability mass function of Tij and Fij (n) is the distribution function of Tij . If Fij (1) &lt; 1, then Tij is a defective rv, since, with some non-zero probability, there is no ﬁrst passage to j . Defective rv’s are not considered to be rv’s (in the theorems here or elsewhere), but they do have many of the properties of rv’s. The ﬁrst passage time Tj j from a state j back to itself is of particular importance. It has the PMF fj j (n), the distribution function Fj j (n), and is a rv (as opposed to a defective rv) if Fj j (1) = 1, i.e., if the state eventually returns to state j with probability 1 given that it starts in state j . This leads to the deﬁnition of recurrence. Denition 5.2. A state j in a countable-state Markov chain is recurrent if Fj j (1) = 1. It is transient if Fj j (1) &lt; 1. Thus each state j in a countable-state Markov chain is either recurrent or transient, and is recurrent if and only if (iﬀ ) an eventual return to j occurs W.P.1, given that X0 = j . Equivalently, j is recurrent iﬀ Tj j , the time to ﬁrst return to j , is a rv. Note that for the special case of ﬁnite-state Markov chains, this deﬁnition is consistent with the one in Chapter 4. For a countably-inﬁnite state space, however, the earlier deﬁnition is not adequate; for example, i and j communicate for all states i and j in Figure 5.1, but for p &gt; 1/2, each state is transient (this is shown in Exercise 5.2, and further explained in Section 5.3). If state j is recurrent, and if the initial state is speciﬁed to be X0 = j , then Tj j is the integer time of the ﬁrst recurrence of state j . At that recurrence, the Markov chain is in the same state j as it started in, and the discrete interval from Tj j to the next occurence of state j , say Tj j,2 has the same distribution as Tj j and is clearly independent of Tj j . Similarly, the sequence of successive recurrence intervals, Tj j , Tj j,2 , Tj j,3 , . . . is a sequence of IID rv’s. This sequence of recurrence intervals2 is then the sequence of inter-renewal intervals of a renewal process, where each renewal interval has the distribution of Tj j . These inter-renewal intervals have the PMF fj j (n) and the distribution function Fj j (n). Since results about Markov chains depend very heavily on whether states are recurrent or transient, we will look carefully at the probabilities Fij (n). Substituting (5.2) into (5.3), we 2 Note that in Chapter 3 the inter-renewal intervals were denoted X1 , X2 , . . . , whereas here X0 , X1 , . . . , is the sequence of states in the Markov chain and Tj j , Tj j,2 , . . . , is the sequence of inter-renewal intervals. 200 CHAPTER 5. COUNTABLE-STATE MARKOV CHAINS obtain Fij (n) = Pij + X k6=j Pik Fkj (n − 1); n &gt; 1; Fij (1) = Pij . (5.4) P In the expression Pij + k6=j Pik FkjP − 1), note that Pij is the probability that state j is (n entered on the ﬁrst transition, and k6=j Pik Fkj (n − 1) is the sum, over every other state k, of the joint probability that k is entered on the ﬁrst transition and that j is entered on one of the subsequent n − 1 transitions. For each i, Fij (n) is non-decreasing in n and upper bounded by 1 (this can be seen from (5.3), and can also be established directly from (5.4) by induction). Thus it can be shown that the limit as n → 1 exists and satisﬁes Fij (1) = Pij + X k6=j Pik Fkj (1). (5.5) There is not always a unique solution to (5.5). That is, the set of equations xij = Pij + X Pik xkj ; k6=j all i ≥ 0 (5.6) always has a solution in which xij = 1 for all i ≥ 0, but if state j is transient, there is another solution in which xij is the true value of Fij (1) and Fj j (1) &lt; 1. Exercise 5.1 shows that if (5.6) is satisﬁed by a set of non-negative numbers {xij ; 1 ≤ i ≤ J }, then...
View Full Document

Ask a homework question - tutors are online