# lec0210 - IEOR 4106 Spring 2011 Professor Whitt Class...

This preview shows pages 1–2. Sign up to view the full content.

Class Lecture Notes: Thursday, February 10. Markov Chains The Contraction approach to π = π P The limit for aperiodic irreducible ﬁnite-state DTMC’s. There is a nice simple limit for aperiodic irreducible ﬁnite-state Markov chains. For any initial probability vector u ( u 1 ,...,u m ), the probability vector at time n is P ( X n = j ) = ( uP n ) j = m X i =1 u i P n i,j . The key limiting result is Theorem 0.1 If P is the transition matrix of an aperiodic irreducible ﬁnite-state Markov chain with transition matrix P , then, for any initial probability vector u , uP n π as n → ∞ , where the limiting probability vector π is the unique stationary probability vector, i.e., the unique solution to the ﬁxed-point equation π = πP or π j = m X i =1 π i P i,j for all j , where π j 0 for all j and j π j = 1 . Note the conditions: Of course, irreducibility is essential. And aperiodicity is essential to get full convergence, as opposed to convergence of averages, or convergence through appropriate subsequences. The method of proof here is designed to apply to ﬁnite-state chains. The proof extends to inﬁnite-state chains under the condition that there is some state j such that P i,j ² > 0 for all states i , or P k i,j ² > 0 for some k . This is a strong extra condition saying that there is a state j such that there is a probability of at least ² > 0 of going to j in one step (or in k steps, as a weaker version of the same condition), from any other state. With that extra condition, we not only get convergence, we get convergence quickly, geometrically fast. We actually provide a proof without this condition, but we do not get such quick convergence unless the condition holds. The Contraction Proof. One way to prove this result and others is to apply renewal theory. An alternative way to prove the theorem is to consider the transition matrix P as an operator on the space of all probability vectors, here taken to be of dimension m , corresponding to there being m states. An operator on a space maps the space into itself. If

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

## This note was uploaded on 01/16/2012 for the course IEOR 4106 taught by Professor Whitward during the Spring '11 term at Columbia College.

### Page1 / 5

lec0210 - IEOR 4106 Spring 2011 Professor Whitt Class...

This preview shows document pages 1 - 2. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online