Discrete-time stochastic processes

Thus we have the following useful lemma lemma 64

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: e j . The reward is one whenever the process is in state j . A renewal occurs on each entry to state j , so the reward starts at each such entry and continues until a state transition, assumed to enter a state other than j . The reward then ceases until the next renewal, i.e., the next entry to state j . The figure illustrates the kth interrenewal interval, of duration Wk , which is assumed to start on the n − 1st state transition. The expected interval over which a reward is accumulated is ∫j and the expected duration of the interrewal interval is W (j ). times, the inter-renewal times are non-arithmetic, and thus from Blackwell’s theorem, in the form of (3.77), lim Pr {X (t) = j } = t→1 1 . ∫j W (j ) (6.12) We summarize these results in the following lemma. Lemma 6.3. Consider an irreducible Markov process with a recurrent embedded Markov chain. Then with probability 1, the limiting time-average fraction of time in each state j is 1 given by pj = ∫ W (j ) . This is also the limit, as t → 1 of Pr {X (t) = j }. j Next we must express the mean inter-renewal time, W (j ), in terms of more accessible quantities. We continue to assume an irreducible process and a starting state X0 = i. From the strong law for delayed renewal processes (Theorem 3.9), lim Mij (t)/t = 1/W (j ) t→1 W.P. 1. (6.13) As before, Mij (t) = Nij (Mi (t)). Since limt→1 Mi (t) = 1 with probability 1, lim t→1 Mij (t) Nij (Mi (t)) Nij (n) = lim = lim = πj n→1 t→1 Mi (t) Mi (t) n W.P. 1. (6.14) Combining (6.13) and (6.14), the following equalities hold with probability 1. 1 W (j ) Mij (t) t→1 t Mij (t) Mi (t) = lim t→1 Mi (t) t Mi (t) = πj lim . t→1 t = lim (6.15) (6.16) 6.2. STEADY-STATE BEHAVIOR OF IRREDUCIBLE MARKOV PROCESSES 243 Substituting this in (6.11), we see that pj = πj Mi (t) lim t→1 ∫j t W.P.1. (6.17) This equation does not quite tell us what pj is, since we don’t have a good expression for the time-average number of transitions per unit time, limt→1 Mi (t)/t. It does, however, tell us that this time-average converges to some particular value with probability 1, and also tell us (not surprisingly) that this value is independent of the starting state i. Finally, it tell us that if we can find the single value limt→1 Mi (t)/t (which is independent of i), then we have an expression for all pj in terms of πj , ∫j , and this single constant.3 6.2.3 The strong law for time-average state probabilities In order to evaluate limt→1 Mi (t)/t, we modify the assumption that X0 = i for some arbitrary i to the assumption that the embedded Markov chain starts inP steady state. The expected time until the first transition in the Markov process is then i πi /∫i . The embedded Markov chain remains in steady state after each transition, and therefore the inter-transition time, say Un for the nth transition is a sequence of IID rv’s, each with P mean i πi /∫i . There is a renewal process, say M (t), associated with this sum of interrenewal times, and by the strong law for renewal processes, M (t) 1 =P . t→1 t i πi /∫i lim Now M (t) = P i πi Mi (t), and (6.11) does not depend on i. Thus pj = πj πj /∫j M (t) lim =P t→1 ∫j t i πi /∫i W.P.1. (6.18) The following theorem summarizes these results. Theorem 6.1. Consider an irreducible Markov process with a positive recurrent embedded Markov chain starting in any state X0 = i. Then, with probability 1, the limiting time-average fraction of time spent in an arbitrary state j is given by (6.18). Also pj = limt→1 Pr {X (t)=j }. Final ly, the expected time between returns to state j is W (j ) = P i πi /∫i . πj This has been a great deal of work to validate (6.18), which must seem almost intuitively obvious. However, the fact that these time averages are valid over all sample points with probability 1 is not obvious and the fact that πj W (j ) is independent of j is certainly not obvious. P It is extremely tempting at this point to argue that j pj = 1. This reduces (6.17) to (6.9), and gives the P correct answer in most cases. Mathematically however, pj is defined as a time limit, and asserting that j pj = 1 involves interchanging a limit with a sum. This is not always valid, and we shortly find the conditions under which it is invalid for this problem. 3 244 CHAPTER 6. MARKOV PROCESSES WITH COUNTABLE STATE SPACES P The most subtle thing here, however, is that if i πi /∫i = 1, then pj = 0 for all states j . This is strange because the time-average state probabilities do not add to 1, and also strange because the embedded Markov chain continues to make transitions, and these transitions, in steady state, occur with the probabilities πi . We give an example of this phenomenon for a birth-death process in the next subsection. What is happening is that for very small t, the state is with high probability in the starting state, but as t increases, there is an increasing probability that X (t) is in a state with a P holding time. The effect of these long holding large times builds up with increasing t....
View Full Document

Ask a homework question - tutors are online