6.262.Lec8

6.262.Lec8 - DISCRETE STOCHASTIC PROCESSES Lecture 8 Finite...

Info iconThis preview shows pages 1–6. Sign up to view the full content.

View Full Document Right Arrow Icon
Lecture 8 - 2/26/2010 Discrete Stochastic Processes 1 DISCRETE STOCHASTIC PROCESSES Lecture 8 Finite State Markov Chains Reading: Gallager, Chapter 4, pp. 156-181. (Will not emphasize proofs in Sect. 4.4.) Brief Review Markov Property, Graph and Matrix Representations, Classes of States, Transient and Recurrent Classes, Periodic and Aperiodic Classes 6.041 Review – Absorption Probabilities Matrix Approach Chapmann - Kolmogorov Equation Steady-State Behavior: Results from Perron-Frobenius Theory 6.041 Review – Long-Term Frequency Interpretations Markov Chains with Rewards
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Lecture 8 - 2/26/2010 Discrete Stochastic Processes 2 F INITE S TATE M ARKOV C HAINS Definition: A Finite State Markov Chain is an Integer Time Process, X n ; n 0 { } in which X n , for each n 0 is a random variable with possible values {1, 2, . .., J } with the Markov Property Definition : The Markov Property PX n = j X n 1 = i , X n 2 = h ,..., X 0 = m () = n = j X n 1 = i ( ) = P ij for all n , i , j , h , m ,... . A Markov chain is completely described by set of transition probabilities P ij plus initial probabilities 0 . Sometimes view P ij { } graphically, sometimes as matrix. P P .... P P P .... P .... P P .... P 1 2 4 3 5 P P P P P P P P P 11 12 41 24 32 23 35 45 55 11 12 21 22 [P] = 25 15 51 52 55 The graph emphasizes the possible and impossible.
Background image of page 2
Lecture 8 - 2/26/2010 Discrete Stochastic Processes 3 Classification of States Definitions: State j is accessible from i ( i j ) if path from i to j in graph. States i and j communicate if ( i j ) if ( i j ) and ( j i ). If ( i j ) and ( j k ) then ( i k ). Given a walk from i to j , and a walk from j to k , walk from i can be extended to go to k and i k . Similarly k i . Definition: A Class of states is a non-empty set S of states such that ( i j ) for each i S , j S and also for i S , no j S with ( i j ). A class can be though of as a maximal set of communicating states.
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Lecture 8 - 2/26/2010 Discrete Stochastic Processes 4 RECURRENT AND TRANSIENT STATES 1 2 4 3 5 P P P P P P P P P 11 12 41 24 32 23 35 45 55 6 7 {1,2,3,4} {5} {6,7} Classes Definitions: A class S is recurrent if no j S is accessible from any iS . A recurrent class can be thought of as a “trapping” class – once you get in, you never get out. A class that is not recurrent is transient : {5} recurrent, {1, 2, 3, 4}, {6, 7} transient Transient means there is an outgoing edge from the class and no possible return.
Background image of page 4
Lecture 8 - 2/26/2010 Discrete Stochastic Processes 5 ( ) 2, 4 2. d = PERIODIC AND APERIODIC STATES For example, P 44 n > 0 for n = 4, 6, 8, 10, . .. . The greatest common divisor is For state i = 1, P 11 n > 0 for n = 4, 8, 10, 12, . .. . d (1) = 2. If d ( i ) = 1, i is defined to be aperiodic; otherwise it is periodic with period d ( i ). Theorem: All states in the same class have the same period.
Background image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 6
This is the end of the preview. Sign up to access the rest of the document.

Page1 / 22

6.262.Lec8 - DISCRETE STOCHASTIC PROCESSES Lecture 8 Finite...

This preview shows document pages 1 - 6. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online