# 173Lecture8 - Lecture 8 Mariana Olvera-Cravioto UC Berkeley...

• Notes
• 22

This preview shows page 1 - 7 out of 22 pages.

Lecture 8 Mariana Olvera-Cravioto UC Berkeley [email protected] February 9th, 2017 IND ENG 173, Introduction to Stochastic Processes Lecture 8 1/13
Convergence theorems I Theorem: Suppose { X n : n 0 } is irreducible, aperiodic, and has a stationary distribution . Then, lim n !1 p n ( x, y ) = y for all x, y 2 S . I Theorem: Suppose { X n : n 0 } is irreducible and all its states are recurrent. Let N n ( y ) be the number of visits to y up to time n , then, lim n !1 N n ( y ) n = 1 E y [ T y ] , where T y is the return time to state y . I Theorem: Suppose { X n : n 0 } is irreducible and has a stationary distribution . Then, y = 1 E y [ T y ] . IND ENG 173, Introduction to Stochastic Processes Lecture 8 2/13
Some remarks about the convergence theorems I Suppose that { X n : n 0 } is an irreducible Markov chain having stationary distribution . I As the Matlab program showed, j has the interpretation of being the long-run proportion of time that the process will be in state j . I Even if the chain is not aperiodic, we still have that lim n !1 N n ( y ) n = y . I If the chain is aperiodic, we also have lim n !1 p n ( x, y ) = y for all x, y 2 S . IND ENG 173, Introduction to Stochastic Processes Lecture 8 3/13
Some remarks about I Theorem: Suppose { X n : n 0 } is irreducible and all its states are recurrent. Then, there exists a solution μ ( x ) 0 to X x μ ( x ) p ( x, y ) = μ ( y ) , y 2 S . If S is finite, μ ( x ) can be normalized to obtain . I The stationary probabilities are such that if we choose X 0 according to { j : j 0 } then P ( X 1 = j ) = 1 X i =0 P ( X 1 = j | X 0 = i ) i = 1 X i =0 p ( i, j ) i = j = P ( X 0 = j ) . I A Markov chain where X 0 has distribution { j : j 0 } is said to be stationary or in stationarity . I For a stationary Markov chain P ( X n = j ) = j for all n 0 . IND ENG 173, Introduction to Stochastic Processes Lecture 8 4/13
Long run cost I Suppose that whenever the chain { X n : n 0 } visits state x it incurs a cost f ( x ) . I We want to compute the long run cost, i.e., lim n !1 1 n n X i =1 f ( X i ) IND ENG 173, Introduction to Stochastic Processes Lecture 8 5/13
Long run cost I Suppose that whenever the chain { X n : n 0 } visits state x it incurs a cost f ( x ) . I We want to compute the long run cost, i.e., lim n !1 1 n n X i =1 f ( X i ) I Assuming a stationary distribution exists, we expect X i to be distributed according to for large i , and since the first few values can be