Markov_chainsIIIA-beamer

Markov_chainsIIIA-beamer - Main Theorem Finite State Spaces...

Info iconThis preview shows pages 1–6. Sign up to view the full content.

View Full Document Right Arrow Icon
Main Theorem – Finite State Spaces Positive and Null Recurrence Stationary Distributions – Interpretations Computation of a Stationary Distribution Steady State Costs/Rewards Introductory Engineering Stochastic Processes, ORIE 3510 Instructor: Professor Mark E. Lewis School of Operations Research and Information Engineering Cornell University Disclaimer : Notes are only meant as a lecture supplement not substitute! 1
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Main Theorem – Finite State Spaces Positive and Null Recurrence Stationary Distributions – Interpretations Computation of a Stationary Distribution Steady State Costs/Rewards The Main Theorem (for finite state spaces) Theorem A finite state, irreducible, aperiodic Markov chain has a limiting distribution that satisfies the steady-state equations. In fact, this distribution is the unique non-negative solution to said equations. The finite state space, irreducibility assumptions guarantee recurrence The irreducibility (and thus recurrence) result assures that the limiting distribution is independent of the initial state The existence of a limiting distribution guarantees it satisfies the steady state equations. 2
Background image of page 2
Main Theorem – Finite State Spaces Positive and Null Recurrence Stationary Distributions – Interpretations Computation of a Stationary Distribution Steady State Costs/Rewards Infinite State Spaces The next step is to consider infinite state spaces The only thing that changes is to see when a limiting probability is a distribution. .. And (therefore) can be obtained via the steady state equations 3
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Main Theorem – Finite State Spaces Positive and Null Recurrence Stationary Distributions – Interpretations Computation of a Stationary Distribution Steady State Costs/Rewards Definition Main Theorem – Infinite State Spaces Positive, Null Recurrence and Ergodicity Recall, a state i is recurrent, if, starting in state i the probability of returning to i is 1. Mathematically, this can be stated another way. Let τ i = min { n > 0 | X n = i } . Then i is recurrent if P ( τ i < ∞| X 0 = i ) = 1 . 4
Background image of page 4
Positive and Null Recurrence Stationary Distributions – Interpretations Computation of a Stationary Distribution Steady State Costs/Rewards Definition Main Theorem – Infinite State Spaces Positive and Null Recurrence Of course this does not imply that E ( τ i | X 0 = i ) < (can you come up with a counterexample?). Instead we have the following two definitions regarding a recurrent state i Definition Given a recurrent state i (for the DTMC { X n , n 0 } ), if E ( τ i | X 0 = i ) ( < i is positive recurrent , = i is null recurrent . A positive recurrent, aperiodic state is called ergodic Positive and null recurrence are only of importance when the state space is infinite. 5
Background image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Image of page 6
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

Page1 / 22

Markov_chainsIIIA-beamer - Main Theorem Finite State Spaces...

This preview shows document pages 1 - 6. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online