Discrete-time stochastic processes

Successive choices between leaving the system and

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: a solution for {pi ; i ≥ 0} with i pi = 1, but i pi ∫i = 1. This is not possible for a positive recurrent embedded chain, but is possible both if the embedded Markov chain is transient and if it is null recurrent. A transient chain means that there is a positive probability that the embedded chain will never return to a state after leaving it, and thus there can be no sensible kind of steady state behavior for the process. These processes are characterized by arbitrarily large transition rates from the 246 CHAPTER 6. MARKOV PROCESSES WITH COUNTABLE STATE SPACES ♥ 0 ② 1 1 0.6 ③ 1 ② 2−1 0.4 0.6 ③ ♥ 2 ② 2−2 0.4 0.6 ③ ♥ 3 2−3 ... Figure 6.5: The Markov process for a variation on M/M/1 where arrivals and services get slower with increasing state. Each node i has a rate ∫i = 2−i . The embedded chain transition probabilities are Pi,i+1 = 0.4 for i ≥ 1 and Pi,i−1 = 0.6 for i ≥ 1. Note that qi,i+1 > qi+1,i . various states, and these allow the process to transit through an infinite number of states in a finite time. Processes for which there is a non-zero probability of passing through an infinite number of states in a finite time are called irregular. Exercises 6.4 and 6.5 give some insight into irregular processes. Exercise 6.6 gives an example of a process that is not irregular, but P for which (6.20) has a solution with i pi = 1 and the embedded Markov chain is null recurrent. We restrict P attention in P our what follows to irreducible Markov chains for which (6.20) has a solution, pi = 1, and pi ∫i < 1. This is slightly more restrictive than P necessary, but processes for which i pi ∫i = 1 (see Exercise 6.6) are not very robust. 6.3 The Kolmogorov differential equations Let Pij (t) be the probability that a Markov process is in state j at time t given that X (0) = i, Pij (t) = P {X (t)=j | X (0)=i}. (6.22) n Pij (t) is analogous to the nth order transition probabilities Pij for Markov chains. We have already seen that limt→1 Pij (t) = pj for the case where the embedded chain is positive P recurrent and i πi /∫i < 1. Here we want to find the transient behavior, and we start by deriving the Chapman-Kolmogorov equations for Markov processes. Let s and t be arbitrary times, 0 < s < t. By including the state at time s, we can rewrite (6.22) as X Pij (t) = Pr {X (t)=j, X (s)=k | X (0)=i} k = X k Pr {X (s)=k | X (0)=i} Pr {X (t)=j | X (s)=k} ; all i, j, (6.23) where we have used the Markov condition, (6.3). Given that X (s) = k, the residual time until the next transition after s is exponential with rate ∫k , and thus the process starting at time s in state k is statistically identical to that starting at time 0 in state k. Thus, for any s ≥ 0, we have P {X (t)=j | X (s)=k} = Pkj (t − s). 6.3. THE KOLMOGOROV DIFFERENTIAL EQUATIONS 247 Substituting this into (6.23), we have the Chapman-Kolmogorov equations for a Markov process, Pij (t) = X k Pik (s)Pkj (t − s). (6.24) These equations correspond to (4.17) for Markov chains. We now use these equations to derive two types of sets of differential equations for finding Pij (t). The first type are called the Kolmogorov backward differential equations, and the second are called the Kolmogorov forward differential equations. The backward equations are obtained by letting s approach 0 from above, and the forward equations are obtained by letting s approach t from below. First we derive the backward equations. For s small, the probability of two transitions in (0, s] is o(s) and the probability of a single transition into k is qik s + o(s). Thus Pik (s) = qik s + o(s) for k 6= i. Now, given that X (0) = i, the state remains at i throughout the interval (0, s] with probability exp(−∫i s) = 1 − ∫i s + o(s). The probability that the state changes in (0, s) and returns to i by time s is also of order o(s). Thus Pii (s) = 1 − ∫i s + o(s) and (6.24) becomes Pij (t) = X [qik sPkj (t − s)] + (1 − ∫i s)Pij (t − s) + o(s). (6.25) k6=i Subtracting Pij (t − s) from both sides and dividing by s, we get § Pij (t) − Pij (t − s) X £ o(s) = qik Pkj (t − s) − ∫i Pij (t − s) + . s s (6.26) k6=i Taking the limit as s → 0, (and assuming that the summation and limit can be interchanged) we get the Kolmogorov backward equations, § dPij (t) X £ = qik Pkj (t) − ∫i Pij (t). dt (6.27) k6=i P For interpretation, the right hand side of (6.27) can be rewritten as k6=i qik [Pkj (t) − Pij (t)]. We see that qik is the rate of moving from i to k in an initial increment of time. If such a transition occurs in the initial increment, the change in probability of ending in state j after an additional interval of length t is given by Pkj (t) − Pij (t). Thus the rate of change of Pij (t) can be interpreted as arising from the initial rates of transition from i to other states. The set of equations over i, j in (6.27) is a set of linear differential first order equations which, for any given j , must be solved simultaneously for all i. For a finite state space, this set of equations, for all i and j , can...
View Full Document

This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.

Ask a homework question - tutors are online