lec0210 - IEOR 4106, Spring 2011, Professor Whitt Class...

Info iconThis preview shows pages 1–2. Sign up to view the full content.

View Full Document Right Arrow Icon
Class Lecture Notes: Thursday, February 10. Markov Chains The Contraction approach to π = π P The limit for aperiodic irreducible finite-state DTMC’s. There is a nice simple limit for aperiodic irreducible finite-state Markov chains. For any initial probability vector u ( u 1 ,...,u m ), the probability vector at time n is P ( X n = j ) = ( uP n ) j = m X i =1 u i P n i,j . The key limiting result is Theorem 0.1 If P is the transition matrix of an aperiodic irreducible finite-state Markov chain with transition matrix P , then, for any initial probability vector u , uP n π as n → ∞ , where the limiting probability vector π is the unique stationary probability vector, i.e., the unique solution to the fixed-point equation π = πP or π j = m X i =1 π i P i,j for all j , where π j 0 for all j and j π j = 1 . Note the conditions: Of course, irreducibility is essential. And aperiodicity is essential to get full convergence, as opposed to convergence of averages, or convergence through appropriate subsequences. The method of proof here is designed to apply to finite-state chains. The proof extends to infinite-state chains under the condition that there is some state j such that P i,j ² > 0 for all states i , or P k i,j ² > 0 for some k . This is a strong extra condition saying that there is a state j such that there is a probability of at least ² > 0 of going to j in one step (or in k steps, as a weaker version of the same condition), from any other state. With that extra condition, we not only get convergence, we get convergence quickly, geometrically fast. We actually provide a proof without this condition, but we do not get such quick convergence unless the condition holds. The Contraction Proof. One way to prove this result and others is to apply renewal theory. An alternative way to prove the theorem is to consider the transition matrix P as an operator on the space of all probability vectors, here taken to be of dimension m , corresponding to there being m states. An operator on a space maps the space into itself. If
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 2
This is the end of the preview. Sign up to access the rest of the document.

Page1 / 5

lec0210 - IEOR 4106, Spring 2011, Professor Whitt Class...

This preview shows document pages 1 - 2. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online