Statistics 150: Spring 2007 February 7, 2007 0-1 1 Markov Chains Let { X ,X 1 , } be a sequence of random variables which take values in some countable set S , called the state space . Each X n is a discrete random variables that takes one of N possible values, where N = | S | ; it may be the case that N = . Definition. The process X is a Markov Chain if it satisfies the Markov condition : P ( X n = s | X = x ,X 1 = x 1 ,...,X n- 1 = x n- 1 ) = P ( X n = s | X n- 1 = x n- 1 ) for all n 1 and all s,x 1 , ,x n- 1 S . 1 Definition. The chain X is called homogeneous if P ( X n +1 = j | X n = i ) = P ( X 1 = j | X = i ) for all n,i,j . The transition matrix P = ( p ij ) is the | S | | S | matrix of transition probabilities p ij = P ( X n +1 = j | X n = i ) . Henceforth, all Markov chains are assumed homogeneous unless otherwise specified. 2 Theorem. The transition matrix P is a stochastic matrix, which is to say that: (a) P has non-negative entries, or p ij for all i,j , (b) P has row sums equal to one, or j p ij = 1 for all i. Proof. An easy exercise. Definition. The n-step transition matrix P ( m,m + n ) = ( p ij ( m,m + n )) is the matrix of n-step transition probabilities p ij ( m,m + n ) = P ( X m + n = j | X m = i ) . 3 Theorem. Chapman-Kolmogorov equations. p ij ( m,m + n + r ) = X k p ik ( m,m + n ) p kj ( m + n,m + n + r ) . Therefore, P ( m,m + n + r ) = P ( m,m + n ) P ( m + n,m + n + r ) , and P ( m,m + n ) = P n , the n th power of P . Proof. We have as required that p ij ( m,m + n + r ) = P ( X m + n + r = j | X m = i ) = X k P ( X m + n + r = j,X m + n = k | X m = i ) 4 = X k P ( X m + n + r = j | X m + n = k,X m = i ) P ( X m + n = k | X m = i ) = X k P ( X m + n + r = j | X m + n = k ) P ( X m + n = k | X m = i ) where we have used the fact that P ( A T B | C ) = P ( A | B T C ) P ( B | C ) , together with the Markov property. The established equation may be written in matrix form as P ( m,m + n + r ) = P ( m,m + n ) P ( m + n,m + n + r ) , and it follows by iteration that P ( m,m + n ) = P n . 5 Let ( n ) i = P ( X n = i ) be the mass function of X n , and write n for the row vector with entries ( n i : i S ) . Lemma ( m + n ) = ( m ) P n , and hence ( n ) = (0) P n . 6 2 Exercises 1) A die is rolled repeatedly. Which of the following are Markov chains? For those that are, supply the transition matrix. (a) The largest number X n shown up to the n th roll. (b) The number N n of sixes in n rolls. (c) At time r , the time C r since the most recent six. (d) At time r , the time B r until the next six. 7 2) Let X be a Markov chain on S , and let T be a random variables taking values in { , 1 , 2 , } with the property that the indicator function 1 { T = n } of the event that T = n is a function of the variables X 1 ,X 2 , ,X n . Such a random variables T is called a stopping time , and the above definition requires that it is decidable whether or not T = n with a knowledge only of the past and present,...
View Full Document