Stochastic

# 3 classication of states we begin with some important

This preview shows page 1. Sign up to view the full content.

This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: lassiﬁcation of States We begin with some important notation. We are often interested in the behavior of the chain for a ﬁxed initial state, so we will introduce the shorthand Px (A) = P (A|X0 = x) Later we will have to consider expected values for this probability and we will denote them by Ex . Let Ty = min{n 1 : Xn = y } be the time of the ﬁrst return to y (i.e., being there at time 0 doesn’t count), and let ⇢yy = Py (Ty < 1) be the probability Xn returns to y when it starts at y . Note that if we didn’t exclude n = 0 this probability would always be 1. Intuitively, the Markov property implies that the probability Xn will return to y at least twice is ⇢2 , since after the ﬁrst return, the chain is at y , and the yy probability of a second return following the ﬁrst is again ⇢yy . To show that the reasoning in the last paragraph is valid, we have to introduce a deﬁnition and state a theorem. We say that T is a stopping time if the occurrence (or nonoccurrence) of the event “we stop at time n,” {T = n}, can be determined by looking at the values o...
View Full Document

## This document was uploaded on 03/06/2014 for the course MATH 4740 at Cornell.

Ask a homework question - tutors are online