Unformatted text preview: e j . The reward is one whenever the process is in state j . A renewal occurs on each entry to
state j , so the reward starts at each such entry and continues until a state transition,
assumed to enter a state other than j . The reward then ceases until the next renewal,
i.e., the next entry to state j . The ﬁgure illustrates the kth interrenewal interval, of
duration Wk , which is assumed to start on the n − 1st state transition. The expected
interval over which a reward is accumulated is ∫j and the expected duration of the
interrewal interval is W (j ). times, the interrenewal times are nonarithmetic, and thus from Blackwell’s theorem, in
the form of (3.77),
lim Pr {X (t) = j } = t→1 1
.
∫j W (j ) (6.12) We summarize these results in the following lemma.
Lemma 6.3. Consider an irreducible Markov process with a recurrent embedded Markov
chain. Then with probability 1, the limiting timeaverage fraction of time in each state j is
1
given by pj = ∫ W (j ) . This is also the limit, as t → 1 of Pr {X (t) = j }.
j Next we must express the mean interrenewal time, W (j ), in terms of more accessible
quantities. We continue to assume an irreducible process and a starting state X0 = i. From
the strong law for delayed renewal processes (Theorem 3.9),
lim Mij (t)/t = 1/W (j ) t→1 W.P. 1. (6.13) As before, Mij (t) = Nij (Mi (t)). Since limt→1 Mi (t) = 1 with probability 1,
lim t→1 Mij (t)
Nij (Mi (t))
Nij (n)
= lim
= lim
= πj
n→1
t→1
Mi (t)
Mi (t)
n W.P. 1. (6.14) Combining (6.13) and (6.14), the following equalities hold with probability 1.
1
W (j ) Mij (t)
t→1
t
Mij (t) Mi (t)
= lim
t→1 Mi (t)
t
Mi (t)
= πj lim
.
t→1
t
= lim (6.15) (6.16) 6.2. STEADYSTATE BEHAVIOR OF IRREDUCIBLE MARKOV PROCESSES 243 Substituting this in (6.11), we see that
pj = πj
Mi (t)
lim
t→1
∫j
t W.P.1. (6.17) This equation does not quite tell us what pj is, since we don’t have a good expression for
the timeaverage number of transitions per unit time, limt→1 Mi (t)/t. It does, however,
tell us that this timeaverage converges to some particular value with probability 1, and
also tell us (not surprisingly) that this value is independent of the starting state i. Finally,
it tell us that if we can ﬁnd the single value limt→1 Mi (t)/t (which is independent of i),
then we have an expression for all pj in terms of πj , ∫j , and this single constant.3 6.2.3 The strong law for timeaverage state probabilities In order to evaluate limt→1 Mi (t)/t, we modify the assumption that X0 = i for some
arbitrary i to the assumption that the embedded Markov chain starts inP
steady state.
The expected time until the ﬁrst transition in the Markov process is then i πi /∫i . The
embedded Markov chain remains in steady state after each transition, and therefore the
intertransition time, say Un for the nth transition is a sequence of IID rv’s, each with
P
mean i πi /∫i . There is a renewal process, say M (t), associated with this sum of interrenewal times, and by the strong law for renewal processes,
M (t)
1
=P
.
t→1
t
i πi /∫i
lim Now M (t) = P i πi Mi (t), and (6.11) does not depend on i. Thus pj = πj
πj /∫j
M (t)
lim
=P
t→1
∫j
t
i πi /∫i W.P.1. (6.18) The following theorem summarizes these results. Theorem 6.1. Consider an irreducible Markov process with a positive recurrent embedded Markov chain starting in any state X0 = i. Then, with probability 1, the limiting timeaverage fraction of time spent in an arbitrary state j is given by (6.18). Also
pj = limt→1 Pr {X (t)=j }. Final ly, the expected time between returns to state j is W (j ) =
P
i πi /∫i
.
πj
This has been a great deal of work to validate (6.18), which must seem almost intuitively
obvious. However, the fact that these time averages are valid over all sample points with
probability 1 is not obvious and the fact that πj W (j ) is independent of j is certainly not
obvious.
P
It is extremely tempting at this point to argue that j pj = 1. This reduces (6.17) to (6.9), and gives
the
P correct answer in most cases. Mathematically however, pj is deﬁned as a time limit, and asserting that
j pj = 1 involves interchanging a limit with a sum. This is not always valid, and we shortly ﬁnd the
conditions under which it is invalid for this problem.
3 244 CHAPTER 6. MARKOV PROCESSES WITH COUNTABLE STATE SPACES P
The most subtle thing here, however, is that if i πi /∫i = 1, then pj = 0 for all states j .
This is strange because the timeaverage state probabilities do not add to 1, and also strange
because the embedded Markov chain continues to make transitions, and these transitions,
in steady state, occur with the probabilities πi . We give an example of this phenomenon for
a birthdeath process in the next subsection. What is happening is that for very small t, the
state is with high probability in the starting state, but as t increases, there is an increasing
probability that X (t) is in a state with a P holding time. The eﬀect of these long holding
large
times builds up with increasing t....
View
Full
Document
This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.
 Spring '09
 R.Srikant

Click to edit the document details