Discrete-time stochastic processes

This 57 semi markov processes 223 is done in exercise

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: 3 − + e A| 0− ™ = © Much more broadly, any 3 events, say A , X0 , A are − + to b™ Markov−if Pr ™ © X+A ™ © said © Pr A+ | X0 , and this implies the more symmetric form Pr A E | X0 ) = Pr A | X0 ) Pr A | X0 ) . 5.4. REVERSIBLE MARKOV CHAINS 213 and backward running chains, however, to visualize Markov chains running in steady-state from t = −1 to t = +1. If one is uncomfortable with this, one can also visualize starting the Markov chain at some very negative time with the initial distribution equal to the steady-state distribution. ∗ A Markov chain is now defined to be reversible if Pij = Pij for all states i and j . Thus the chain is reversible if, in steady-state, the backward running sequence of states is statistically indistinguishable from the forward running sequence. Comparing (5.38) with the steady-state equations (5.27) that we derived for birth-death chains, we have the important theorem: Theorem 5.5. Every birth-death chain with a steady-state probability distribution is reversible. We saw that for birth-death chains, the equation πi Pij = πj Pj i (which only had to be considered for |i − j | ≤ 1) provided a very simple way of calculating the steady-state probabilities. Unfortunately, it appears that we must first calculate the steady-state probabilities in order to show that a chain is reversible. The following simple theorem gives us a convenient escape from this dilemma. Theorem 5.6. Assume that an irreducible Markov chain has transition probabilities {Pij }. Suppose {πi } is a set of positive numbers summing to 1 and satisfying πi Pij = πj Pj i ; al l i, j. (5.39) then, first, {πi ; i ≥ 0} is the steady-state distribution for the chain, and, second, the chain is reversible. Proof: Given a solution to (5.39) for all i and j , we can sum this equation over i for each j. X X πi Pij = πj Pj i = πj . (5.40) i i P Thus the solution to (5.39), along with the constraints πi > 0, i πi = 1, satisfies the steadystate equations, (5.14), and, from Theorem 5.4, this is the unique steady-state distribution. Since (5.39) is satisfied, the chain is also reversible. It is often possible, sometimes by using an educated guess, to find a solution to (5.39). If this is successful, then we are assured both that the chain is reversible and that the actual steady-state probabilities have been found. Note that the theorem applies to periodic chains as well as to aperiodic chains. If the chain is periodic, then the steady-state probabilities have to be interpreted as average values over the period, but, from Theorem 5.4, (5.40) still has a unique solution (assuming an irreducible chain). On the other hand, for a chain with period d > 1, there are d subclasses of states and the sequence {Xn } must rotate between these classes in a fixed order. For this same order to be followed in the backward chain, the only possibility is d = 2. Thus periodic chains with periods other than 2 cannot be reversible. 214 CHAPTER 5. COUNTABLE-STATE MARKOV CHAINS There are several simple tests that can be used to show that some given irreducible chain is not reversible. First, the steady-state probabilities must satisfy πi > 0 for all i, and thus, if Pij > 0 but Pj i = 0 for some i, j , then (5.39) cannot be satisfied and the chain is not reversible. Second, consider any set of three states, i, j , k. If Pj i Pik Pkj is unequal to Pj k Pki Pij then the chain cannot be reversible. To see this, note that (5.39) requires that πi = πj Pj i /Pij = πk Pki /Pik . Thus, πj Pj i Pik = πk Pki Pij . Equation (5.39) also requires that πj Pj k = πk Pkj . Taking the ratio of these equations, we see that Pj i Pik Pkj = Pj k Pki Pij . Thus if this equation is not satisfied, the chain cannot be reversible. In retrospect, this result is not surprising. What it says is that for any cycle of three states, the probability of three transitions going around the cycle in one direction must be the same as the probability of going around the cycle in the opposite (and therefore backwards) direction. It is also true (see [16] for a proof ), that a necessary and sufficient condition for a chain to be reversible is that the product of transition probabilities around any cycle of arbitrary length must be the same as the product of transition probabilities going around the cycle in the opposite direction. This doesn’t seem to be a widely useful way to demonstrate reversibility. There is another result, similar to Theorem 5.6, for finding the steady-state probabilities of an arbitrary Markov chain and simultaneously finding the transition probabilities of the backward chain. Theorem 5.7. Assume that an irreducible Markov chain has transition probabilities {Pij }. ∗ Suppose {πi } is a set of positive numbers summing to 1 and that {Pij } is a set of transition probabilities satisfying πi Pij = πj Pj∗i ; al l i, j. (5.41) ∗ Then {πi } is the steady-state distribution and {Pij } is the set of transi...
View Full Document

This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.

Ask a homework question - tutors are online