This preview shows pages 1–2. Sign up to view the full content.
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: Copyright c 2008 by Karl Sigman 1 Communication classes and irreducibility for Markov chains For a Markov chain with state space S , consider a pair of states ( i,j ). We say that j is reachable from i , denoted by i → j , if there exists an integer n ≥ 0 such that P n ij > 0. This means that starting in state i , there is a positive probability (but not necessarily equal to 1) that the chain will be in state j at time n (that is, n steps later); P ( X n = j  X = i ) > 0. If j is reachable from i , and i is reachable from j , then the states i and j are said to communicate , denoted by i ←→ j . The relation defined by communication satisfies the following conditions: 1. All states communicate with themselves: P ii = 1 > 0. 1 2. Symmetry: If i ←→ j , then j ←→ i . 3. Transitivity: If i ←→ k and k ←→ j , then i ←→ j . The above conditions imply that communication is an example of an equivalence relation, meaning that it shares the properties with the more familiar equality relation “ = ”: i = i . If i = j , then j = i . If i = k and k = j , then i = j . Only condition 3 above needs some justification, so we now prove it for completeness: Suppose there exists integers n , m such that P n ik > 0 and P m kj > 0. Letting l = n + m , we conclude that P l ij ≥ P n ik P m kj > 0 where we have formally used the ChapmanKolmogorov equations. The point is that the chain can go from i to j by first going from i to k ( n steps) and then (independent of the past) going from k to j (an additional m steps). If we consider the rat in the open maze, we easily see that the set of states C 1 = { 1 , 2 , 3 , 4 } all communicate with one another, but state 0 only communicates with itself (since it is an absorbing state). Whereas state 0 is reachable from the other states, i → 0, no other state can be reached from state 0. We conclude that the state space S = { , 1 , 2 , 3 , 4 } can be broken up into two disjoint subsets, C 1 = { 1 , 2 , 3 , 4 } and C 2 = { } whose union equals S , and such that each of these subsets has the property that all states within it communicate. Disjoint means that their intersection contains no elements: C 1 ∩ C 2 = ∅ . A little thought reveals that this kind of disjoint breaking can be done with any Markov chain: Proposition 1.1 For each Markov chain, there exists a unique decomposition of the state space S into a sequence of disjoint subsets C 1 ,C 2 ,... , S = ∪ ∞ i =1 C i , in which each subset has the property that all states within it communicate. Each such subset is called a communication class of the Markov chain. If we now consider the rat in the closed maze, S = { 1 , 2 , 3 , 4 } , then we see that there is only one communication class C = { 1 , 2 , 3 , 4 } = S : all states communicate. This is an example of what is called an irreducible Markov chain....
View
Full
Document
This note was uploaded on 10/20/2010 for the course IEOR 4106 taught by Professor Whitt during the Fall '08 term at Columbia.
 Fall '08
 Whitt
 Operations Research

Click to edit the document details