This preview shows page 1. Sign up to view the full content.
Unformatted text preview: tituting this into (5.17), we
P
get πj ≥ 1/T j j i≤M πi . Since M is arbitrary, πj ≥ 1/T j j . Since we already showed
that πj ≤ limt→1 E [Nj j (t)/t] = 1/T j j , we have πj = 1/T j j for all j . This shows both
that πj > 0 for all j and that the solution to (5.14) is unique. Exercise 5.4 completes the
proof by showing that if the states are positiverecurrent, then choosing πj = 1/T j j for all
j satisﬁes (5.14). 206 CHAPTER 5. COUNTABLESTATE MARKOV CHAINS In practice, it is usually easy to see whether a chain is irreducible. We shall also see by a
number of examples that the steadystate distribution can often be calculated from (5.14).
Theorem 5.4 then says that the calculated distribution is unique and that its existence
guarantees that the chain is positive recurrent.
Example 5.1.1. Age of a renewal process: Consider a renewal process {N (t); t ≥ 0}
in which the interrenewal random variables {Wn ; n ≥ 1} are arithmetic with span 1. We
will use a Markov chain to model the age of this process (see Figure 5.2). The probability
that a renewal occurs at a particular integer time depends on the past only through the
integer time back to the last renewal. The state of the Markov chain during a unit interval
will be taken as the age of the renewal process at the beginning of the interval. Thus, each
unit of time, the age either increases by one or a renewal occurs and the age decreases to 0
(i.e., if a renewal occurs at time t, the age at time t is 0).
✿♥
✘0
② ❍ P01 P00 P10 ③
♥
1 ③
♥
2 P12
P20 P23
P30 ③
♥
3 P34
P40 ③
♥
4 ... Figure 5.2: A Markov chain model of the age of a renewal process.
Pr {W > n} is the probability that an interrenewal interval lasts for more than n time
units. We assume that Pr {W > 0} = 1, so that each renewal interval lasts at least one
time unit. The probability Pn,0 in the Markov chain is the probability that a renewal
interval has duration n + 1, given that the interval exceeds n. Thus, for example, P00
is the probability that the renewal interval is equal to 1. Pn,n+1 is 1 − Pn,0 , which is
Pr {W > n + 1} /Pr {W > n}. We can then solve for the steady state probabilities in the
chain: for n > 0,
πn = πn−1 Pn−1,n = πn−2 Pn−2,n−1 Pn−1,n = π0 P0,1 P1,2 . . . Pn−1,n .
The ﬁrst equality above results from the fact that state n, for n > 0 can be entered only
from state n − 1. The subsequent equalities come from substituting in the same expression
for πn−1 , then pn−2 , and so forth.
πn = π0 Pr {W > 1} Pr {W > 2}
Pr {W > n}
...
= π0 Pr {W > n} .
Pr {W > 0} Pr {W > 1}
Pr {W > n − 1} (5.18) We have cancelled out all the cross terms above and used the fact that Pr {W > 0} = 1.
Another way to see that πn = π0 Pr {W > n} is to observe that state 0 occurs exactly once
in each interrenewal interval; state n occurs exactly once in those interrenewal intervals
of duration n or more.
Since the steadystate probabilities must sum to 1, (5.18) can be solved for π0 as
π0 = P1 n=0 1
1
=
.
Pr {W > n}
E [W ] (5.19) 5.2. BRANCHING PROCESSES 207 The second equality follows by expressing E [W ] as the integral of the complementary distribution function of W . Combining this with (5.18), the steadystate probabilities for n ≥ 0
are
πn = Pr {W > n}
.
E [W ] (5.20) In terms of the renewal process, πn is the probability that, at some large integer time, the
age of the process will be n. Note that if the age of the process at an integer time is n,
then the age increases toward n + 1 at the next integer time, at which point it either drops
to 0 or continues to rise. Thus πn can be interpreted as the fraction of time that the age of
the process is between n and n + 1. Recall from (3.37) (and the fact that residual life and
age are equally distributed) that the distribution function of the timeaverage age is given
Rn
by FZ (n) = 0 Pr {W > w} dw/E [W ]. Thus, the probability that the age is between n and
n + 1 is FZ (n + 1) − FZ (n). Since W is an integer random variable, this is Pr {W > n} /E [W ]
in agreement with our result here.
The analysis here gives a new, and intuitively satisfying, explanation of why the age of a
renewal process is so diﬀerent from the interrenewal time. The Markov chain shows the ever
increasing loops that give rise to large expected age when the interrenewal time is heavy
tailed (i.e., has a distribution function that goes to 0 slowly with increasing time). These
loops can be associated with the isosceles triangles of Figure 3.8. The advantage here is that
we can associate the states with steadystate probabilities if the chain is recurrent. Even
when the Markov chain is nullrecurrent (i.e., the associated renewal process has inﬁnite
expected age), it seems easier to visualize the phenomenon of inﬁnite expected age. 5.2 Branching processes Branching processes provide a simple model for studying the population of various types of
individuals from on...
View
Full
Document
 Spring '09
 R.Srikant

Click to edit the document details