Discrete-time stochastic processes

# For this same order to be followed in the backward

This preview shows page 1. Sign up to view the full content.

This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: r al l transient. Proof: Let j be a recurrent state. From Theorem 5.1, all states in a class are recurrent or all are transient. Next suppose that j is positive-recurrent, so that 1/T j j > 0. Let i be in the same class as j , and consider the renewal-reward process on {Nj j (t); t ≥ 0} for which R(t) = 1 whenever the process is in state i (i.e., if Xn = i, then R(t) = 1 for n ≤ t < n + 1). The reward is 0 whenever the process is in some state other than i. Let E [Rn ] be the expected reward in an inter-renewal interval; this must be positive since i is accessible from j . From the strong law for renewal-reward processes, Theorem 3.6, Zt 1 E [Rn ] lim R(τ )dτ = with probability 1. t→1 t T jj 0 The term on the left is the time-average number of transitions into state i, given X0 = j , and this is 1/T ii from (5.11). Since E [Rn ] > 0 and T j j < 1, we have 1/T ii > 0, so i is positive-recurrent. Thus if one state is positive-recurrent, the entire class is, completing the proof. If all of the states in a Markov chain are in a null-recurrent class, then 1/T j j = 0 for each state, and one might think of 1/T j j = 0 as a “steady-state” probability for j in the sense that 0 is both the time-average rate of occurrence of j and the limiting probability of j . However, these “probabilities” do not add up to 1, so a steady-state probability distribution does not exist. This appears rather paradoxical at ﬁrst, but the example of Figure 5.1, with p = 1/2 will help to clarify the situation. As time n increases (starting in state i, say), the random variable Xn spreads out over more and more states around i,P thus is less likely and n to be in each individual state. For each j , limn→1 Pij (n) = 0. Thus, j {limn→1 Pij } = 0. Pn On the other hand, for every n, j Pij = 1. This is one of those unusual examples where a limit and a sum cannot be interchanged. In Chapter 4, we deﬁned the steady-state distribution of a ﬁnite-state Markov chain as a probability vector π that satisﬁes π = π [P ]. Here we deﬁne {πi ; i ≥ 0} in the same way, as a set of numbers that satisfy X X πj = πi Pij for all j ; πj ≥ 0 for all j ; πj = 1. (5.14) i j Suppose that a set of numbers {πi ; i ≥ 0} satisfying (5.14) is chosen as the initial probability distribution for a Markov chain, i.e., if Pr {X0 = i} = πi for all i. Then Pr {X1 = j } = P i πi Pij = πj for all j , and, by induction, Pr {Xn = j } = πj for all j and all n ≥ 0. The fact that Pr {Xn = j } = πj for all j motivates the deﬁnition of steady-state distribution 5.1. INTRODUCTION AND CLASSIFICATION OF STATES 205 above. Theorem 5.2 showed that 1/T j j is a ‘steady-state’ probability for state j , both in a time-average and a limiting ensemble-average sense. The following theorem brings these ideas together. An irreducible Markov chain is a Markov chain in which all pairs of states communicate. For ﬁnite-state chains, irreducibility implied a single class of recurrent states, whereas for countably inﬁnite chains, an irreducible chain is a single class that can be transient, null-recurrent, or positive-recurrent. Theorem 5.4. Assume an irreducible Markov chain with transition probabilities {Pij }. If (5.14) has a solution, then the solution is unique, πi = 1/T ii > 0 for al l i ≥ 0, and the states are positive-recurrent. Also, if the states are positive-recurrent then (5.14) has a solution. Proof*: Let {πj ; j ≥ 0} satisfy (5.14) and be the initial distribution of the Markov chain, i.e., Pr {X0 =j } = πj , j ≥ 0. Then, as shown above, Pr {Xn =j } = πj for all n ≥ 0, j ≥ 0. ˜ Let Nj (t) be the number of occurrences of any given state j from time 1 to t. Equating Pr {Xn =j } to the expectation of an occurrence of j at time n, we have, h i X ˜ (1/t)E Nj (t) = (1/t) Pr {Xn =j } = πj for all integers t ≥ 1. 1≤n≤t Conditioning this on the possible starting states i, and using the counting processes {Nij (t); t ≥ 0} deﬁned earlier, h iX e πj = (1/t)E Nj (t) = πi E [Nij (t)/t] for all integer t ≥ 1. (5.15) i For any given state i, let Tij be the time of the ﬁrst occurrence of state j given X0 = i. Then if Tij < 1, we have Nij (t) ≤ Nij (Tij + t). Thus, for all t ≥ 1, E [Nij (t)] ≤ E [Nij (Tij + t)] = 1 + E [Nj j (t)] . (5.16) The last step follows since the process is in state j at time Tij , and the expected number of occurrences of state j in the next t steps is E [Nj j (t)]. Substituting (5.16) in (5.15) for each i, πj ≤ 1/t + E [Nj j (t)/t)]. Taking the limit as P t → 1 and using (5.12), πj ≤ limt→1 E [Nj j (t)/t]. Since i πi = 1, there is at least one value of j for which πj > 0, and for this j , limt→1 E [Nj j (t)/t] > 0, and consequently limt→1 E [Nj j (t)] = 1. Thus, from Lemma 5.1, state j is recurrent, and from Theorem 5.2, j is positive-recurrent. From Theorem 5.3, all states are then positive-recurrent. For any j and any integer M , (5.15) implies that X πj ≥ πi E [Nij (t)/t] for all t. (5.17) i≤M From Theorem 5.2, limt→1 E [Nij (t)/t] = 1/T j j for all i. Subs...
View Full Document

## This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.

Ask a homework question - tutors are online