Discrete-time stochastic processes

# Thus the restriction on the transition probabilities

This preview shows page 1. Sign up to view the full content.

This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Fij (1) ≤ xij for each i. We have deﬁned a state j to be recurrent if Fj j (1) = 1 and have seen that if j is recurrent, then the returns to state j , given X0 = j form a renewal process, and all of the results of renewal theory can then be applied to the random sequence of integer times at which j is entered. Our next ob jective is to show that all states in the same class as a recurrent state are also recurrent. Recall that two states are in the same class if they communicate, i.e., each has a path to the other. For ﬁnite-state Markov chains, the fact that either all states in the same class are recurrent or all transient was relatively obvious, but for countable-state Markov chains, the deﬁnition of recurrence has been changed and the above fact is no longer obvious. We start with a lemma that summarizes some familiar results from Chapter 3. Lemma 5.1. Let {Nj j (t); t ≥ 0} be the counting process for occurrences of state j up to time t in a Markov chain with X0 = j . The fol lowing conditions are then equivalent. 1. state j is recurrent. 2. limt→1 Nj j (t) = 1 with probability 1. 3. limt→1 E [Nj j (t)] = 1. 4. limt→1 P n 1≤n≤t Pj j = 1. 5.1. INTRODUCTION AND CLASSIFICATION OF STATES 201 Proof: First assume that j is recurrent, i.e., that Fj j (1) = 1. This implies that the inter-renewal times between occurrences of j are IID rv’s, and consequently {Nj j (t); t ≥ 1} is a renewal counting process. Recall from Lemma 3.1 of Chapter 3 that, whether or not the expected inter-renewal time E [Tj j ] is ﬁnite, limt→1 Nj j (t) = 1 with probability 1 and limt→1 E [Nj j (t)] = 1. Next assume that state j is transient. In this case, the inter-renewal time Tj j is not a rv, so {Nj j (t); t ≥ 0} is not a renewal process. An eventual return to state j occurs only with probability Fj j (1) < 1, and, since subsequent returns are independent, the total number of returns to state j is a geometric rv with mean Fj j (1)/[1 − Fj j (1)]. Thus the total number of returns is ﬁnite with probability 1 and the expected total number of returns is ﬁnite. This establishes the ﬁrst three equivalences. Finally, note that Pjn , the probability of a transition to state j at integer time n, is equal j to the expectation of a transition to j at integer time n (i.e., a single transition occurs with probability Pjn and 0 occurs otherwise). Since Nj j (t) is the sum of the number of j transitions to j over times 1 to t, we have X E [Nj j (t)] = Pjn , j 1≤n≤t which establishes the ﬁnal equivalence. Next we show that if one state in a class is recurrent, then the others are also. Lemma 5.2. If state j is recurrent and states i and j are in the same class, i.e., i and j communicate, then state i is also recurrent. P Proof: From Lemma 5.1, state j satisﬁes limt→1 1≤n≤t Pjn = 1. Since j and i commuj m nicate, there are integers m and k such that Pij > 0 and Pjki > 0. For every walk from state j to j in n steps, there is a corresponding walk from i to i in m + n + k steps, going from i to j in m steps, j to j in n steps, and j back to i in k steps. Thus m m Pii +n+k ≥ Pij Pjn Pjki j 1 X n=1 n Pii ≥ 1 X n=1 m m Pii +n+k ≥ Pij Pjki 1 X n=1 Pjn = 1. j Thus, from Lemma 5.1, i is recurrent, completing the proof. Since each state in a Markov chain is either recurrent or transient, and since, if one state in a class is recurrent, all states in that class are recurrent, we see that if one state in a class is transient, they all are. Thus we can refer to each class as being recurrent or transient. This result shows that Theorem 4.1 also applies to countable-state Markov chains. We state this theorem separately here to be speciﬁc. Theorem 5.1. For a countable-state Markov chain, either al l states in a class are transient or al l are recurrent. 202 CHAPTER 5. COUNTABLE-STATE MARKOV CHAINS We next look at the delayed counting process {Nij (n); n ≥ 1}. We saw that for ﬁnitestate ergodic Markov chains, the eﬀect of the starting state eventually dies out. In order to ﬁnd the conditions under which this happens with countable-state Markov chains, we compare the counting processes {Nj j (n); n ≥ 1} and {Nij (n); n ≥ 1}. These diﬀer only in the starting state, with X0 = j or X0 = i. In order to use renewal theory to study these counting processes, we must ﬁrst verify that the ﬁrst-passage-time from i to j is a rv. The following lemma establishes the rather intuitive result that if state j is recurrent, then from any state i accessible from j , there must be an eventual return to j . Lemma 5.3. Let states i and j be in the same recurrent class. Then Fij (1) = 1. Proof: First assume that Pj i > 0, and assume for the sake of establishing a contradiction that Fij (1) < 1. Since Fj j (1) = 1, we can apply (5.5) to Fj j (1), getting X X 1 = Fj j (1) = Pj j + Pj k Fkj (1) < Pj j + Pj k = 1, k6=j k6=j where the strict inequality follows since Pj i Fij (1) < Pj i by assumption. This is a contradiction, so Fij (1) = 1 for every i accessible from j in one step. Next assume that Pj2i > 0, say wit...
View Full Document

## This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.

Ask a homework question - tutors are online