Unformatted text preview: 3
−
+
e
A 0− ™
=
© Much more broadly, any 3 events, say A , X0 , A are − + to b™ Markov−if Pr ™ © X+A
™
© said
©
Pr A+  X0 , and this implies the more symmetric form Pr A E  X0 ) = Pr A  X0 ) Pr A  X0 ) . 5.4. REVERSIBLE MARKOV CHAINS 213 and backward running chains, however, to visualize Markov chains running in steadystate
from t = −1 to t = +1. If one is uncomfortable with this, one can also visualize starting
the Markov chain at some very negative time with the initial distribution equal to the
steadystate distribution.
∗
A Markov chain is now deﬁned to be reversible if Pij = Pij for all states i and j . Thus
the chain is reversible if, in steadystate, the backward running sequence of states is statistically indistinguishable from the forward running sequence. Comparing (5.38) with the
steadystate equations (5.27) that we derived for birthdeath chains, we have the important
theorem: Theorem 5.5. Every birthdeath chain with a steadystate probability distribution is reversible.
We saw that for birthdeath chains, the equation πi Pij = πj Pj i (which only had to be considered for i − j  ≤ 1) provided a very simple way of calculating the steadystate probabilities.
Unfortunately, it appears that we must ﬁrst calculate the steadystate probabilities in order
to show that a chain is reversible. The following simple theorem gives us a convenient
escape from this dilemma.
Theorem 5.6. Assume that an irreducible Markov chain has transition probabilities {Pij }.
Suppose {πi } is a set of positive numbers summing to 1 and satisfying
πi Pij = πj Pj i ; al l i, j. (5.39) then, ﬁrst, {πi ; i ≥ 0} is the steadystate distribution for the chain, and, second, the chain
is reversible.
Proof: Given a solution to (5.39) for all i and j , we can sum this equation over i for each
j.
X
X
πi Pij = πj
Pj i = πj .
(5.40)
i i P
Thus the solution to (5.39), along with the constraints πi > 0, i πi = 1, satisﬁes the steadystate equations, (5.14), and, from Theorem 5.4, this is the unique steadystate distribution.
Since (5.39) is satisﬁed, the chain is also reversible.
It is often possible, sometimes by using an educated guess, to ﬁnd a solution to (5.39). If
this is successful, then we are assured both that the chain is reversible and that the actual
steadystate probabilities have been found.
Note that the theorem applies to periodic chains as well as to aperiodic chains. If the chain
is periodic, then the steadystate probabilities have to be interpreted as average values
over the period, but, from Theorem 5.4, (5.40) still has a unique solution (assuming an
irreducible chain). On the other hand, for a chain with period d > 1, there are d subclasses
of states and the sequence {Xn } must rotate between these classes in a ﬁxed order. For
this same order to be followed in the backward chain, the only possibility is d = 2. Thus
periodic chains with periods other than 2 cannot be reversible. 214 CHAPTER 5. COUNTABLESTATE MARKOV CHAINS There are several simple tests that can be used to show that some given irreducible chain
is not reversible. First, the steadystate probabilities must satisfy πi > 0 for all i, and
thus, if Pij > 0 but Pj i = 0 for some i, j , then (5.39) cannot be satisﬁed and the chain is
not reversible. Second, consider any set of three states, i, j , k. If Pj i Pik Pkj is unequal to
Pj k Pki Pij then the chain cannot be reversible. To see this, note that (5.39) requires that
πi = πj Pj i /Pij = πk Pki /Pik .
Thus, πj Pj i Pik = πk Pki Pij . Equation (5.39) also requires that πj Pj k = πk Pkj . Taking the
ratio of these equations, we see that Pj i Pik Pkj = Pj k Pki Pij . Thus if this equation is not
satisﬁed, the chain cannot be reversible. In retrospect, this result is not surprising. What
it says is that for any cycle of three states, the probability of three transitions going around
the cycle in one direction must be the same as the probability of going around the cycle in
the opposite (and therefore backwards) direction.
It is also true (see [16] for a proof ), that a necessary and suﬃcient condition for a chain
to be reversible is that the product of transition probabilities around any cycle of arbitrary
length must be the same as the product of transition probabilities going around the cycle
in the opposite direction. This doesn’t seem to be a widely useful way to demonstrate
reversibility.
There is another result, similar to Theorem 5.6, for ﬁnding the steadystate probabilities
of an arbitrary Markov chain and simultaneously ﬁnding the transition probabilities of the
backward chain.
Theorem 5.7. Assume that an irreducible Markov chain has transition probabilities {Pij }.
∗
Suppose {πi } is a set of positive numbers summing to 1 and that {Pij } is a set of transition
probabilities satisfying
πi Pij = πj Pj∗i ; al l i, j. (5.41) ∗
Then {πi } is the steadystate distribution and {Pij } is the set of transi...
View
Full
Document
This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.
 Spring '09
 R.Srikant

Click to edit the document details