Discrete-time stochastic processes

In the special case of an idle queue s 0 which we

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: dividuals in a generation increases exponentially with n, and Y gives the rate of growth. Physical processes do not grow exponentially forever, so branching processes are appropriate models of such physical processes only over some finite range of population. Even more important, the model here assumes that the number of offspring of a single member is independent of the total population, which is highly questionable in many areas of population growth. The advantage of an oversimplified model such as this is that it explains what would happen under these idealized conditions, thus providing insight into how the model should be changed for more realistic scenarios. It is important to realize that, for branching processes, the mean number of individuals is not a good measure of the actual number of individuals. For Y = 1 and X0 = 1, the expected number of individuals in each generation is 1, but the probability that Xn = 0 approaches 1 with increasing n; this means that as n gets large, the nth generation contains a large number of individuals with a very small probability and contains no individuals with a very large probability. For Y > 1, we have just seen that there is a positive probability that the population dies out, but the expected number is growing exponentially. A surprising result, which is derived from the theory of martingales in Chapter 7, is that n if X0 = 1 and Y > 1, then the sequence of random variables Xn /Y has a limit with probability 1. This limit is a random variable; it has the value 0 with probability F10 (1), and has larger values with some given distribution. Intuitively, for large n, Xn is either 0 or very large. If it is very large, it tends to grow in an orderly way, increasing by a multiple of Y in each subsequent generation. 210 5.3 CHAPTER 5. COUNTABLE-STATE MARKOV CHAINS Birth-death Markov chains A birth-death Markov chain is a Markov chain in which the state space is the set of nonnegative integers; for all i ≥ 0, the transition probabilities satisfy Pi,i+1 > 0 and Pi+1,i > 0, and for all |i − j | > 1, Pij = 0 (see Figure 5.4). A transition from state i to i + 1 is regarded as a birth and one from i + 1 to i as a death. Thus the restriction on the transition probabilities means that only one birth or death can occur in one unit of time. Many applications of birth-death processes arise in queueing theory, where the state is the number of customers, births are customer arrivals, and deaths are customer departures. The restriction to only one arrival or departure at a time seems rather peculiar, but usually such a chain is a finely sampled approximation to a continuous time process, and the time increments are then small enough that multiple arrivals or departures in a time increment are unlikely and can be ignored in the limit. ✿♥ ✘0 ② ❖ p0 q1 ③ ♥ 1 ② ❖ p1 q2 ③ ♥ 2 ② ❖ p2 q3 ③ ♥ 3 ② ❖ p3 q4 ③ ♥ 4 ... ❖ 1 − p3 − q3 Figure 5.4: Birth-death Markov chain. We denote Pi,i+1 by pi and Pi,i−1 by qi . Thus Pii = 1 − pi − qi . There is an easy way to find the steady-state probabilities of these birth-death chains. In any sample function of the process, note that the number of transitions from state i to i + 1 differs by at most 1 from the number of transitions from i + 1 to i. If the process starts to the left of i and ends to the right, then one more i → i + 1 transition occurs than i + 1 → i, etc. Thus if we visualize a renewal-reward process with renewals on occurrences of state i and unit reward on transitions from state i to i + 1, the limiting time-average number of transitions per unit time is πi pi . Similarly, the limiting time-average number of transitions per unit time from i + 1 to i is πi+1 qi+1 . Since these two must be equal in the limit, πi pi = πi+1 qi+1 for i ≥ 0. (5.27) The intuition in (5.27) is simply that the rate at which downward transitions occur from i + 1 to i must equal the rate of upward transitions. Since this result is very important, both here and in our later study of continuous time birth-death processes, we show that (5.27) also results from using the steady-state equations in (5.14): πi = pi−1 πi−1 + (1 − pi − qi )πi + qi+1 πi+1 ; π0 = (1 − p0 )π0 + q1 π1 . i>0 (5.28) (5.29) From (5.29), p0 π0 = q1 π1 . To see that (5.27) is satisfied for i > 0, we use induction on i, with i = 0 as the base. Thus assume, for a given i, that pi−1 πi−1 = qi πi . Substituting this in (5.28), we get pi πi = qi+1 πi+1 , thus completing the inductive proof. 5.4. REVERSIBLE MARKOV CHAINS 211 It is convenient to define ρi as pi /qi+1 . Then we have πi+1 = ρi πi , and iterating this, πi = π0 i−1 Y ρj ; π0 = j =0 1+ P1 1 i=1 Qi−1 j =0 ρj . (5.30) PQ If i≥1 0≤j <i ρj < 1, then π0 is positive and all the states are positive-recurrent. If this sum of products is infinite, then none of the states are positive-recurrent. If ρj is bounded below 1, say ρj ≤ 1 − ≤ for some fixed e > 0 and all sufficiently l...
View Full Document

Ask a homework question - tutors are online