Markov_chains-together-beamer

# Markov_chains-together-beamer - Transition Matrix...

This preview shows pages 1–6. Sign up to view the full content.

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Transition Matrix Transition Diagram Classes and Short Term Calculations Long-Term Behavior Long-term Behavior – Recurrent States Average Costs Introductory Engineering Stochastic Processes, ORIE 3510 Instructor: Professor Mark E. Lewis School of Operations Research and Information Engineering Cornell University Disclaimer : Notes are only meant as a lecture supplement not substitute! 1 Transition Matrix Transition Diagram Classes and Short Term Calculations Long-Term Behavior Long-term Behavior – Recurrent States Average Costs DTMC Example Consider a DTMC with the following one-step transition matrix. P = 1 2 3 4 5 6 7 8 9 1 1 1 2 / 3 1 / 3 1 / 2 1 / 2 1 / 2 1 / 2 0 2 / 3 1 / 3 1 / 2 1 / 2 1 / 4 3 / 4 2 Transition Matrix Transition Diagram Classes and Short Term Calculations Long-Term Behavior Long-term Behavior – Recurrent States Average Costs Transition Diagram 5 2 3 6 7 4 8 9 1 1 1 1 1/2 1/2 2/3 1/3 1/4 3/4 1/3 1/2 1/2 2/3 1/2 1/2 5 2 3 6 7 4 8 9 1 1 1 1 1/2 1/2 2/3 1/3 1/2 1/4 3/4 1/3 1/2 1/2 2/3 1/2 3 Transition Matrix Transition Diagram Classes and Short Term Calculations Long-Term Behavior Long-term Behavior – Recurrent States Average Costs Classes and Short Term Calculations Recurrent: { 1 , 2 , 3 } , { 4 , 5 } Transient: { 6 , 7 } , { 8 } , { 9 } Suppose f ( i ) = i for i = 1 , 2 ,..., 9 Hopefully by now we know that we know how to compute E ( f ( X 3 ) | X = i ) Compute P 3 by matrix multiplication This yields p (3) ij for all i , j ∈ S . Compute E ( f ( X 3 ) | X = i ) = ∑ j ∈ S f ( j ) p (3) ij (this is 1.3891 for i = 7) 4 Transition Matrix Transition Diagram Classes and Short Term Calculations Long-Term Behavior Long-term Behavior – Recurrent States Average Costs Rearrange the state space Cost Until Absorption Probability of Ever Reaching j Probability of Being Absorbed Into...
View Full Document

{[ snackBarMessage ]}

### Page1 / 16

Markov_chains-together-beamer - Transition Matrix...

This preview shows document pages 1 - 6. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online