Discrete-time stochastic processes

There are two types of correspondence between the

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: tituting this into (5.17), we P get πj ≥ 1/T j j i≤M πi . Since M is arbitrary, πj ≥ 1/T j j . Since we already showed that πj ≤ limt→1 E [Nj j (t)/t] = 1/T j j , we have πj = 1/T j j for all j . This shows both that πj > 0 for all j and that the solution to (5.14) is unique. Exercise 5.4 completes the proof by showing that if the states are positive-recurrent, then choosing πj = 1/T j j for all j satisfies (5.14). 206 CHAPTER 5. COUNTABLE-STATE MARKOV CHAINS In practice, it is usually easy to see whether a chain is irreducible. We shall also see by a number of examples that the steady-state distribution can often be calculated from (5.14). Theorem 5.4 then says that the calculated distribution is unique and that its existence guarantees that the chain is positive recurrent. Example 5.1.1. Age of a renewal process: Consider a renewal process {N (t); t ≥ 0} in which the inter-renewal random variables {Wn ; n ≥ 1} are arithmetic with span 1. We will use a Markov chain to model the age of this process (see Figure 5.2). The probability that a renewal occurs at a particular integer time depends on the past only through the integer time back to the last renewal. The state of the Markov chain during a unit interval will be taken as the age of the renewal process at the beginning of the interval. Thus, each unit of time, the age either increases by one or a renewal occurs and the age decreases to 0 (i.e., if a renewal occurs at time t, the age at time t is 0). ✿♥ ✘0 ② ❍ P01 P00 P10 ③ ♥ 1 ③ ♥ 2 P12 P20 P23 P30 ③ ♥ 3 P34 P40 ③ ♥ 4 ... Figure 5.2: A Markov chain model of the age of a renewal process. Pr {W > n} is the probability that an inter-renewal interval lasts for more than n time units. We assume that Pr {W > 0} = 1, so that each renewal interval lasts at least one time unit. The probability Pn,0 in the Markov chain is the probability that a renewal interval has duration n + 1, given that the interval exceeds n. Thus, for example, P00 is the probability that the renewal interval is equal to 1. Pn,n+1 is 1 − Pn,0 , which is Pr {W > n + 1} /Pr {W > n}. We can then solve for the steady state probabilities in the chain: for n > 0, πn = πn−1 Pn−1,n = πn−2 Pn−2,n−1 Pn−1,n = π0 P0,1 P1,2 . . . Pn−1,n . The first equality above results from the fact that state n, for n > 0 can be entered only from state n − 1. The subsequent equalities come from substituting in the same expression for πn−1 , then pn−2 , and so forth. πn = π0 Pr {W > 1} Pr {W > 2} Pr {W > n} ... = π0 Pr {W > n} . Pr {W > 0} Pr {W > 1} Pr {W > n − 1} (5.18) We have cancelled out all the cross terms above and used the fact that Pr {W > 0} = 1. Another way to see that πn = π0 Pr {W > n} is to observe that state 0 occurs exactly once in each inter-renewal interval; state n occurs exactly once in those inter-renewal intervals of duration n or more. Since the steady-state probabilities must sum to 1, (5.18) can be solved for π0 as π0 = P1 n=0 1 1 = . Pr {W > n} E [W ] (5.19) 5.2. BRANCHING PROCESSES 207 The second equality follows by expressing E [W ] as the integral of the complementary distribution function of W . Combining this with (5.18), the steady-state probabilities for n ≥ 0 are πn = Pr {W > n} . E [W ] (5.20) In terms of the renewal process, πn is the probability that, at some large integer time, the age of the process will be n. Note that if the age of the process at an integer time is n, then the age increases toward n + 1 at the next integer time, at which point it either drops to 0 or continues to rise. Thus πn can be interpreted as the fraction of time that the age of the process is between n and n + 1. Recall from (3.37) (and the fact that residual life and age are equally distributed) that the distribution function of the time-average age is given Rn by FZ (n) = 0 Pr {W > w} dw/E [W ]. Thus, the probability that the age is between n and n + 1 is FZ (n + 1) − FZ (n). Since W is an integer random variable, this is Pr {W > n} /E [W ] in agreement with our result here. The analysis here gives a new, and intuitively satisfying, explanation of why the age of a renewal process is so different from the inter-renewal time. The Markov chain shows the ever increasing loops that give rise to large expected age when the inter-renewal time is heavy tailed (i.e., has a distribution function that goes to 0 slowly with increasing time). These loops can be associated with the isosceles triangles of Figure 3.8. The advantage here is that we can associate the states with steady-state probabilities if the chain is recurrent. Even when the Markov chain is null-recurrent (i.e., the associated renewal process has infinite expected age), it seems easier to visualize the phenomenon of infinite expected age. 5.2 Branching processes Branching processes provide a simple model for studying the population of various types of individuals from on...
View Full Document

Ask a homework question - tutors are online