Markov_chainsIIIA-beamer

# Markov_chainsIIIA-beamer - Main Theorem Finite State Spaces...

This preview shows pages 1–6. Sign up to view the full content.

Main Theorem – Finite State Spaces Positive and Null Recurrence Stationary Distributions – Interpretations Computation of a Stationary Distribution Steady State Costs/Rewards Introductory Engineering Stochastic Processes, ORIE 3510 Instructor: Professor Mark E. Lewis School of Operations Research and Information Engineering Cornell University Disclaimer : Notes are only meant as a lecture supplement not substitute! 1

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
Main Theorem – Finite State Spaces Positive and Null Recurrence Stationary Distributions – Interpretations Computation of a Stationary Distribution Steady State Costs/Rewards The Main Theorem (for ﬁnite state spaces) Theorem A ﬁnite state, irreducible, aperiodic Markov chain has a limiting distribution that satisﬁes the steady-state equations. In fact, this distribution is the unique non-negative solution to said equations. The ﬁnite state space, irreducibility assumptions guarantee recurrence The irreducibility (and thus recurrence) result assures that the limiting distribution is independent of the initial state The existence of a limiting distribution guarantees it satisﬁes the steady state equations. 2
Main Theorem – Finite State Spaces Positive and Null Recurrence Stationary Distributions – Interpretations Computation of a Stationary Distribution Steady State Costs/Rewards Inﬁnite State Spaces The next step is to consider inﬁnite state spaces The only thing that changes is to see when a limiting probability is a distribution. .. And (therefore) can be obtained via the steady state equations 3

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
Main Theorem – Finite State Spaces Positive and Null Recurrence Stationary Distributions – Interpretations Computation of a Stationary Distribution Steady State Costs/Rewards Deﬁnition Main Theorem – Inﬁnite State Spaces Positive, Null Recurrence and Ergodicity Recall, a state i is recurrent, if, starting in state i the probability of returning to i is 1. Mathematically, this can be stated another way. Let τ i = min { n > 0 | X n = i } . Then i is recurrent if P ( τ i < ∞| X 0 = i ) = 1 . 4
Positive and Null Recurrence Stationary Distributions – Interpretations Computation of a Stationary Distribution Steady State Costs/Rewards Deﬁnition Main Theorem – Inﬁnite State Spaces Positive and Null Recurrence Of course this does not imply that E ( τ i | X 0 = i ) < (can you come up with a counterexample?). Instead we have the following two deﬁnitions regarding a recurrent state i Deﬁnition Given a recurrent state i (for the DTMC { X n , n 0 } ), if E ( τ i | X 0 = i ) ( < i is positive recurrent , = i is null recurrent . A positive recurrent, aperiodic state is called ergodic Positive and null recurrence are only of importance when the state space is inﬁnite. 5

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

### Page1 / 22

Markov_chainsIIIA-beamer - Main Theorem Finite State Spaces...

This preview shows document pages 1 - 6. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online