Unformatted text preview: h Pj m > 0 and Pmi > 0. For a contradiction, again assume that Fij (1) < 1. From
(5.5),
X
X
Fmj (1) = Pmj +
Pmk Fkj (1) < Pmj +
Pmk = 1,
k6=j k6=j where the strict inequality follows since Pmi Fij (1) < Pmi . This is a contradiction, since
m is accessible from j in one step, and thus Fmj (1) = 1. It follows that every i accessible
from j in two steps satisﬁes Fij (1) = 1. Extending the same argument for successively
larger numbers of steps, the conclusion of the lemma follows.
Lemma 5.4. Let {Nij (t); t ≥ 0} be the counting process for transitions into state j up to
time t for a Markov chain given X0 = i 6= j . Then if i and j are in the same recurrent
class, {Nij (t); t ≥ 0} is a delayed renewal process.
Proof: From Lemma 5.3, Tij , the time until the ﬁrst transition into j , is a rv. Also Tj j is a
rv by deﬁnition of recurrence, and subsequent intervals between occurrences of state j are
IID, completing the proof.
If Fij (1) = 1, we have seen that the ﬁrstpassage time from i to j is a rv, i.e., is ﬁnite with
probability 1. In this case, the mean time T ij to ﬁrst enter state j starting from state i is
of interest. Since Tij is a nonnegative random variable, its expectation is the integral of its
complementary distribution function,
T ij = 1 + 1
X (1 − Fij (n)). (5.7) n=1 It is possible to have Fij (1) = 1 but T ij = 1. As will be shown in Section 5.3, the chain
in Figure 5.1 satisﬁes Fij (1) = 1 and T ij < 1 for p < 1/2 and Fij (1) = 1 and T ij = 1
for p = 1/2. As discussed before, Fij (1) < 1 for p > 1/2. This leads us to the following
deﬁnition. 5.1. INTRODUCTION AND CLASSIFICATION OF STATES 203 Deﬁnition 5.3. A state j in a countablestate Markov chain is positiverecurrent if Fj j (1) =
1 and T j j < 1. It is nullrecurrent if Fj j (1) = 1 and T j j = 1.
Each state of a Markov chain is thus classiﬁed as one of the following three types — positiverecurrent, nullrecurrent, or transient. For the example of Figure 5.1, nullrecurrence lies
on a boundary between positiverecurrence and transience, and this is often a good way to
look at nullrecurrence. Part f ) of Exercise 6.1 illustrates another type of situation in which
nullrecurrence can occur.
Assume that state j is recurrent and consider the renewal process {Nj j (t); t ≥ 0}. The
limiting theorems for renewal processes can be applied directly. From the strong law for
renewal processes, Theorem 3.1,
lim Nj j (t)/t = 1/T j j t→1 with probability 1. (5.8) From the elementary renewal theorem, Theorem 3.4,
lim E [Nj j (t)/t] = 1/T j j . t→1 (5.9) Equations (5.8) and (5.9) are valid whether j is positiverecurrent or nullrecurrent.
Next we apply Blackwell’s theorem to {Nj j (t); t ≥ 0}. Recall that the period of a given
state j in a Markov chain (whether the chain has a countable or ﬁnite number of states) is
the greatest common divisor of the set of integers n > 0 such that Pjn > 0. If this period
j
is d, then {Nj j (t); t ≥ 0} is arithmetic with span d (i.e., renewals occur only at times that
are multiples of d). From Blackwell’s theorem in the arithmetic form of (3.20),
lim Pr {Xnd = j  X0 = j } = d/T j j . n→1 (5.10) If state j is aperiodic (i.e., d = 1), this says that limn→1 Pr {Xn = j  X0 = j } = 1/T j j .
Equations (5.8) and (5.9) suggest that 1/T j j has some of the properties associated with
a steadystate probability of state j , and (5.10) strengthens this if j is aperiodic. For a
Markov chain consisting of a single class of states, all positiverecurrent, we will strengthen
this association further in Theorem 5.4 by showing that there is a unique P adystate
ste
distribution, {P, j ≥ 0} such that πj = 1/T j j for all j and such that πj = i πi Pij for
πj
all j ≥ 0 and j πj = 1. The following theorem starts this development by showing that
(5.85.10) are independent of the starting state.
Theorem 5.2. Let j be a recurrent state in a Markov chain and let i be any state in the
same class as j . Given X0 = i, let Nij (t) be the number of transitions into state j by time
t and let T j j be the expected recurrence time of state j (either ﬁnite or inﬁnite). Then
lim Nij (t)/t = 1/T j j t→1 with probability 1 lim E [Nij (t)/t] = 1/T j j . t→1 (5.11)
(5.12) If j is also aperiodic, then
lim Pr {Xn = j  X0 = i} = 1/T j j . n→1 (5.13) 204 CHAPTER 5. COUNTABLESTATE MARKOV CHAINS Proof: Since i and j are recurrent and in the same class, Lemma 5.4 asserts that {Nij (t); t ≥
0} is a delayed renewal process for j 6= i. Thus (5.11) and (5.12) follow from Theorems 3.9
and 3.10 of Chapter 3. If j is aperiodic, then {Nij (t); t ≥ 0} is a delayed renewal process
for which the interrenewal intervals Tj j have span 1 and Tij has an integer span. Thus,
(5.13) follows from Blackwell’s theorem for delayed renewal processes, Theorem 3.11. For
i = j , Equations (5.115.13) follow from (5.85.10), completing the proof.
Theorem 5.3. Al l states in the same class of a Markov chain are of the same type —
either al l positiverecurrent, al l nul lrecurrent, o...
View
Full
Document
This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.
 Spring '09
 R.Srikant

Click to edit the document details