Unformatted text preview: Fij (1) ≤ xij for each i.
We have deﬁned a state j to be recurrent if Fj j (1) = 1 and have seen that if j is recurrent,
then the returns to state j , given X0 = j form a renewal process, and all of the results of
renewal theory can then be applied to the random sequence of integer times at which j is
entered.
Our next ob jective is to show that all states in the same class as a recurrent state are
also recurrent. Recall that two states are in the same class if they communicate, i.e., each
has a path to the other. For ﬁnitestate Markov chains, the fact that either all states in
the same class are recurrent or all transient was relatively obvious, but for countablestate
Markov chains, the deﬁnition of recurrence has been changed and the above fact is no longer
obvious. We start with a lemma that summarizes some familiar results from Chapter 3.
Lemma 5.1. Let {Nj j (t); t ≥ 0} be the counting process for occurrences of state j up to
time t in a Markov chain with X0 = j . The fol lowing conditions are then equivalent.
1. state j is recurrent.
2. limt→1 Nj j (t) = 1 with probability 1.
3. limt→1 E [Nj j (t)] = 1.
4. limt→1 P n
1≤n≤t Pj j = 1. 5.1. INTRODUCTION AND CLASSIFICATION OF STATES 201 Proof: First assume that j is recurrent, i.e., that Fj j (1) = 1. This implies that the
interrenewal times between occurrences of j are IID rv’s, and consequently {Nj j (t); t ≥ 1}
is a renewal counting process. Recall from Lemma 3.1 of Chapter 3 that, whether or not
the expected interrenewal time E [Tj j ] is ﬁnite, limt→1 Nj j (t) = 1 with probability 1 and
limt→1 E [Nj j (t)] = 1.
Next assume that state j is transient. In this case, the interrenewal time Tj j is not a rv,
so {Nj j (t); t ≥ 0} is not a renewal process. An eventual return to state j occurs only with
probability Fj j (1) < 1, and, since subsequent returns are independent, the total number of
returns to state j is a geometric rv with mean Fj j (1)/[1 − Fj j (1)]. Thus the total number
of returns is ﬁnite with probability 1 and the expected total number of returns is ﬁnite.
This establishes the ﬁrst three equivalences.
Finally, note that Pjn , the probability of a transition to state j at integer time n, is equal
j
to the expectation of a transition to j at integer time n (i.e., a single transition occurs
with probability Pjn and 0 occurs otherwise). Since Nj j (t) is the sum of the number of
j
transitions to j over times 1 to t, we have
X
E [Nj j (t)] =
Pjn ,
j
1≤n≤t which establishes the ﬁnal equivalence.
Next we show that if one state in a class is recurrent, then the others are also.
Lemma 5.2. If state j is recurrent and states i and j are in the same class, i.e., i and j
communicate, then state i is also recurrent.
P
Proof: From Lemma 5.1, state j satisﬁes limt→1 1≤n≤t Pjn = 1. Since j and i commuj
m
nicate, there are integers m and k such that Pij > 0 and Pjki > 0. For every walk from
state j to j in n steps, there is a corresponding walk from i to i in m + n + k steps, going
from i to j in m steps, j to j in n steps, and j back to i in k steps. Thus
m
m
Pii +n+k ≥ Pij Pjn Pjki
j
1
X n=1 n
Pii ≥ 1
X n=1 m
m
Pii +n+k ≥ Pij Pjki 1
X n=1 Pjn = 1.
j Thus, from Lemma 5.1, i is recurrent, completing the proof.
Since each state in a Markov chain is either recurrent or transient, and since, if one state in
a class is recurrent, all states in that class are recurrent, we see that if one state in a class is
transient, they all are. Thus we can refer to each class as being recurrent or transient. This
result shows that Theorem 4.1 also applies to countablestate Markov chains. We state this
theorem separately here to be speciﬁc.
Theorem 5.1. For a countablestate Markov chain, either al l states in a class are transient
or al l are recurrent. 202 CHAPTER 5. COUNTABLESTATE MARKOV CHAINS We next look at the delayed counting process {Nij (n); n ≥ 1}. We saw that for ﬁnitestate ergodic Markov chains, the eﬀect of the starting state eventually dies out. In order
to ﬁnd the conditions under which this happens with countablestate Markov chains, we
compare the counting processes {Nj j (n); n ≥ 1} and {Nij (n); n ≥ 1}. These diﬀer only in
the starting state, with X0 = j or X0 = i. In order to use renewal theory to study these
counting processes, we must ﬁrst verify that the ﬁrstpassagetime from i to j is a rv. The
following lemma establishes the rather intuitive result that if state j is recurrent, then from
any state i accessible from j , there must be an eventual return to j .
Lemma 5.3. Let states i and j be in the same recurrent class. Then Fij (1) = 1.
Proof: First assume that Pj i > 0, and assume for the sake of establishing a contradiction
that Fij (1) < 1. Since Fj j (1) = 1, we can apply (5.5) to Fj j (1), getting
X
X
1 = Fj j (1) = Pj j +
Pj k Fkj (1) < Pj j +
Pj k = 1,
k6=j k6=j where the strict inequality follows since Pj i Fij (1) < Pj i by assumption. This is a contradiction, so Fij (1) = 1 for every i accessible from j in one step. Next assume that Pj2i > 0,
say wit...
View
Full
Document
This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.
 Spring '09
 R.Srikant

Click to edit the document details