# Stat150_Spring06_Markov_intro - Statistics 150: Spring 2007...

Stat150_Spring06_Markov_intro
Showing pages : 1 - 9 of 45
This preview has blurred sections. Sign up to view the full version!
Statistics 150: Spring 2007 February 7, 2007 0-1
1 Markov Chains Let { X 0 ,X 1 , ···} be a sequence of random variables which take values in some countable set S , called the state space . Each X n is a discrete random variables that takes one of N possible values, where N = | S | ; it may be the case that N = . Deﬁnition. The process X is a Markov Chain if it satisﬁes the Markov condition : P ( X n = s | X 0 = x 0 ,X 1 = x 1 ,...,X n - 1 = x n - 1 ) = P ( X n = s | X n - 1 = x n - 1 ) for all n 1 and all s,x 1 , ··· ,x n - 1 S . 1
Deﬁnition. The chain X is called homogeneous if P ( X n +1 = j | X n = i ) = P ( X 1 = j | X 0 = i ) for all n,i,j . The transition matrix P = ( p ij ) is the | S | × | S | matrix of transition probabilities p ij = P ( X n +1 = j | X n = i ) . Henceforth, all Markov chains are assumed homogeneous unless otherwise speciﬁed. 2
Theorem. The transition matrix P is a stochastic matrix, which is to say that: (a) P has non-negative entries, or p ij 0 for all i,j , (b) P has row sums equal to one, or j p ij = 1 for all i. Proof. An easy exercise. ± Deﬁnition. The n-step transition matrix P ( m,m + n ) = ( p ij ( m,m + n )) is the matrix of n-step transition probabilities p ij ( m,m + n ) = P ( X m + n = j | X m = i ) . 3
Theorem. Chapman-Kolmogorov equations. p ij ( m,m + n + r ) = X k p ik ( m,m + n ) p kj ( m + n,m + n + r ) . Therefore, P ( m,m + n + r ) = P ( m,m + n ) P ( m + n,m + n + r ) , and P ( m,m + n ) = P n , the n th power of P . Proof. We have as required that p ij ( m,m + n + r ) = P ( X m + n + r = j | X m = i ) = X k P ( X m + n + r = j,X m + n = k | X m = i ) 4
= X k P ( X m + n + r = j | X m + n = k,X m = i ) P ( X m + n = k | X m = i ) = X k P ( X m + n + r = j | X m + n = k ) P ( X m + n = k | X m = i ) where we have used the fact that P ( A T B | C ) = P ( A | B T C ) P ( B | C ) , together with the Markov property. The established equation may be written in matrix form as P ( m,m + n + r ) = P ( m,m + n ) P ( m + n,m + n + r ) , and it follows by iteration that P ( m,m + n ) = P n . ± 5
Let μ ( n ) i = P ( X n = i ) be the mass function of X n , and write μ n for the row vector with entries ( μ n i : i S ) . Lemma μ ( m + n ) = μ ( m ) P n , and hence μ ( n ) = μ (0) P n . 6
2 Exercises 1) A die is rolled repeatedly. Which of the following are Markov chains? For those that are, supply the transition matrix. (a) The largest number X n shown up to the n th roll. (b) The number N n of sixes in n rolls. (c) At time r , the time C r since the most recent six. (d) At time r , the time B r until the next six. 7
2) Let X be a Markov chain on S , and let T be a random variables taking values in { 0 , 1 , 2 , ···} with the property that the indicator function 1 { T = n } of the event that T = n is a function of the variables X 1 ,X 2 , ··· ,X n . Such a random variables T is called a stopping time , and the above deﬁnition requires that it is decidable whether or not T = n with a knowledge only of the past and present,
This is the end of the preview. Sign up to access the rest of the document.