This preview shows page 1. Sign up to view the full content.
Unformatted text preview: st
be calculated for all i before proceeding to calculate them for the next larger value of n.
This also gives us fj j (n), although fj j (n) is not used in the iteration.
1
We say ‘essentially forms a renewal process’ because we haven’t yet speciﬁed the exact conditions upon
which these returns to a given state form a renewal process. Note, however, that since we start the process
in state j at time 0, the time at which the ﬁrst renewal occurs is the same as the interval between successive
renewals. 5.1. INTRODUCTION AND CLASSIFICATION OF STATES 199 Let Fij (n), for n ≥ 1, be the probability, given X0 = i, that state j occurs at some time
between 1 and n inclusive. Thus,
Fij (n) = n
X fij (m). (5.3) m=1 For each i, j , Fij (n) is nondecreasing in n and (since it is a probability) is upper bounded
by 1. Thus Fij (1), i.e., limn→1 Fij (n) must exist, and is the probability, given X0 = i,
that state j will ever occur. If Fij (1) = 1, then, given X0 = i, it is certain (with probability
1) that the chain will eventually enter state j . In this case, we can deﬁne a random variable
(rv) Tij , conditional on X0 = i, as the ﬁrst passage time from i to j . Then fij (n) is the
probability mass function of Tij and Fij (n) is the distribution function of Tij . If Fij (1) < 1,
then Tij is a defective rv, since, with some nonzero probability, there is no ﬁrst passage to
j . Defective rv’s are not considered to be rv’s (in the theorems here or elsewhere), but they
do have many of the properties of rv’s.
The ﬁrst passage time Tj j from a state j back to itself is of particular importance. It has
the PMF fj j (n), the distribution function Fj j (n), and is a rv (as opposed to a defective rv)
if Fj j (1) = 1, i.e., if the state eventually returns to state j with probability 1 given that it
starts in state j . This leads to the deﬁnition of recurrence.
Denition 5.2. A state j in a countablestate Markov chain is recurrent if Fj j (1) = 1.
It is transient if Fj j (1) < 1.
Thus each state j in a countablestate Markov chain is either recurrent or transient, and
is recurrent if and only if (iﬀ ) an eventual return to j occurs W.P.1, given that X0 = j .
Equivalently, j is recurrent iﬀ Tj j , the time to ﬁrst return to j , is a rv. Note that for
the special case of ﬁnitestate Markov chains, this deﬁnition is consistent with the one
in Chapter 4. For a countablyinﬁnite state space, however, the earlier deﬁnition is not
adequate; for example, i and j communicate for all states i and j in Figure 5.1, but for
p > 1/2, each state is transient (this is shown in Exercise 5.2, and further explained in
Section 5.3).
If state j is recurrent, and if the initial state is speciﬁed to be X0 = j , then Tj j is the integer
time of the ﬁrst recurrence of state j . At that recurrence, the Markov chain is in the same
state j as it started in, and the discrete interval from Tj j to the next occurence of state
j , say Tj j,2 has the same distribution as Tj j and is clearly independent of Tj j . Similarly,
the sequence of successive recurrence intervals, Tj j , Tj j,2 , Tj j,3 , . . . is a sequence of IID rv’s.
This sequence of recurrence intervals2 is then the sequence of interrenewal intervals of a
renewal process, where each renewal interval has the distribution of Tj j . These interrenewal
intervals have the PMF fj j (n) and the distribution function Fj j (n).
Since results about Markov chains depend very heavily on whether states are recurrent or
transient, we will look carefully at the probabilities Fij (n). Substituting (5.2) into (5.3), we
2 Note that in Chapter 3 the interrenewal intervals were denoted X1 , X2 , . . . , whereas here X0 , X1 , . . . ,
is the sequence of states in the Markov chain and Tj j , Tj j,2 , . . . , is the sequence of interrenewal intervals. 200 CHAPTER 5. COUNTABLESTATE MARKOV CHAINS obtain
Fij (n) = Pij + X
k6=j Pik Fkj (n − 1); n > 1; Fij (1) = Pij . (5.4) P
In the expression Pij + k6=j Pik FkjP − 1), note that Pij is the probability that state j is
(n
entered on the ﬁrst transition, and k6=j Pik Fkj (n − 1) is the sum, over every other state
k, of the joint probability that k is entered on the ﬁrst transition and that j is entered on
one of the subsequent n − 1 transitions. For each i, Fij (n) is nondecreasing in n and upper
bounded by 1 (this can be seen from (5.3), and can also be established directly from (5.4)
by induction). Thus it can be shown that the limit as n → 1 exists and satisﬁes
Fij (1) = Pij + X
k6=j Pik Fkj (1). (5.5) There is not always a unique solution to (5.5). That is, the set of equations
xij = Pij + X Pik xkj ; k6=j all i ≥ 0 (5.6) always has a solution in which xij = 1 for all i ≥ 0, but if state j is transient, there is
another solution in which xij is the true value of Fij (1) and Fj j (1) < 1. Exercise 5.1
shows that if (5.6) is satisﬁed by a set of nonnegative numbers {xij ; 1 ≤ i ≤ J }, then...
View
Full
Document
 Spring '09
 R.Srikant

Click to edit the document details