Unformatted text preview: a solution for {pi ; i ≥ 0} with i pi = 1, but i pi ∫i =
1. This is not possible for a positive recurrent embedded chain, but is possible both if the
embedded Markov chain is transient and if it is null recurrent. A transient chain means
that there is a positive probability that the embedded chain will never return to a state
after leaving it, and thus there can be no sensible kind of steady state behavior for the
process. These processes are characterized by arbitrarily large transition rates from the 246 CHAPTER 6. MARKOV PROCESSES WITH COUNTABLE STATE SPACES ♥
0
② 1 1 0.6 ③
1
②
2−1 0.4
0.6 ③
♥
2
②
2−2 0.4
0.6 ③
♥
3
2−3 ... Figure 6.5: The Markov process for a variation on M/M/1 where arrivals and services
get slower with increasing state. Each node i has a rate ∫i = 2−i . The embedded chain
transition probabilities are Pi,i+1 = 0.4 for i ≥ 1 and Pi,i−1 = 0.6 for i ≥ 1. Note that
qi,i+1 > qi+1,i . various states, and these allow the process to transit through an inﬁnite number of states
in a ﬁnite time.
Processes for which there is a nonzero probability of passing through an inﬁnite number
of states in a ﬁnite time are called irregular. Exercises 6.4 and 6.5 give some insight into
irregular processes. Exercise 6.6 gives an example of a process that is not irregular, but
P
for which (6.20) has a solution with i pi = 1 and the embedded Markov chain is null
recurrent. We restrict P attention in P
our
what follows to irreducible Markov chains for which
(6.20) has a solution,
pi = 1, and
pi ∫i < 1. This is slightly more restrictive than
P
necessary, but processes for which i pi ∫i = 1 (see Exercise 6.6) are not very robust. 6.3 The Kolmogorov diﬀerential equations Let Pij (t) be the probability that a Markov process is in state j at time t given that
X (0) = i,
Pij (t) = P {X (t)=j  X (0)=i}. (6.22) n
Pij (t) is analogous to the nth order transition probabilities Pij for Markov chains. We have
already seen that limt→1 Pij (t) = pj for the case where the embedded chain is positive
P
recurrent and i πi /∫i < 1. Here we want to ﬁnd the transient behavior, and we start
by deriving the ChapmanKolmogorov equations for Markov processes. Let s and t be
arbitrary times, 0 < s < t. By including the state at time s, we can rewrite (6.22) as
X
Pij (t) =
Pr {X (t)=j, X (s)=k  X (0)=i}
k = X
k Pr {X (s)=k  X (0)=i} Pr {X (t)=j  X (s)=k} ; all i, j, (6.23) where we have used the Markov condition, (6.3). Given that X (s) = k, the residual time
until the next transition after s is exponential with rate ∫k , and thus the process starting
at time s in state k is statistically identical to that starting at time 0 in state k. Thus, for
any s ≥ 0, we have
P {X (t)=j  X (s)=k} = Pkj (t − s). 6.3. THE KOLMOGOROV DIFFERENTIAL EQUATIONS 247 Substituting this into (6.23), we have the ChapmanKolmogorov equations for a Markov
process,
Pij (t) = X
k Pik (s)Pkj (t − s). (6.24) These equations correspond to (4.17) for Markov chains. We now use these equations to
derive two types of sets of diﬀerential equations for ﬁnding Pij (t). The ﬁrst type are called
the Kolmogorov backward diﬀerential equations, and the second are called the Kolmogorov
forward diﬀerential equations. The backward equations are obtained by letting s approach 0
from above, and the forward equations are obtained by letting s approach t from below. First
we derive the backward equations. For s small, the probability of two transitions in (0, s] is
o(s) and the probability of a single transition into k is qik s + o(s). Thus Pik (s) = qik s + o(s)
for k 6= i. Now, given that X (0) = i, the state remains at i throughout the interval (0, s]
with probability exp(−∫i s) = 1 − ∫i s + o(s). The probability that the state changes in (0, s)
and returns to i by time s is also of order o(s). Thus Pii (s) = 1 − ∫i s + o(s) and (6.24)
becomes
Pij (t) = X
[qik sPkj (t − s)] + (1 − ∫i s)Pij (t − s) + o(s). (6.25) k6=i Subtracting Pij (t − s) from both sides and dividing by s, we get
§
Pij (t) − Pij (t − s) X £
o(s)
=
qik Pkj (t − s) − ∫i Pij (t − s) +
.
s
s (6.26) k6=i Taking the limit as s → 0, (and assuming that the summation and limit can be interchanged)
we get the Kolmogorov backward equations,
§
dPij (t) X £
=
qik Pkj (t) − ∫i Pij (t).
dt (6.27) k6=i P
For interpretation, the right hand side of (6.27) can be rewritten as k6=i qik [Pkj (t) − Pij (t)].
We see that qik is the rate of moving from i to k in an initial increment of time. If such
a transition occurs in the initial increment, the change in probability of ending in state j
after an additional interval of length t is given by Pkj (t) − Pij (t). Thus the rate of change
of Pij (t) can be interpreted as arising from the initial rates of transition from i to other
states.
The set of equations over i, j in (6.27) is a set of linear diﬀerential ﬁrst order equations
which, for any given j , must be solved simultaneously for all i. For a ﬁnite state space,
this set of equations, for all i and j , can...
View
Full
Document
This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.
 Spring '09
 R.Srikant

Click to edit the document details