6.262.Lec8

# 6.262.Lec8 - DISCRETE STOCHASTIC PROCESSES Lecture 8 Finite...

This preview shows pages 1–6. Sign up to view the full content.

Lecture 8 - 2/26/2010 Discrete Stochastic Processes 1 DISCRETE STOCHASTIC PROCESSES Lecture 8 Finite State Markov Chains Reading: Gallager, Chapter 4, pp. 156-181. (Will not emphasize proofs in Sect. 4.4.) Brief Review Markov Property, Graph and Matrix Representations, Classes of States, Transient and Recurrent Classes, Periodic and Aperiodic Classes 6.041 Review – Absorption Probabilities Matrix Approach Chapmann - Kolmogorov Equation Steady-State Behavior: Results from Perron-Frobenius Theory 6.041 Review – Long-Term Frequency Interpretations Markov Chains with Rewards

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
Lecture 8 - 2/26/2010 Discrete Stochastic Processes 2 F INITE S TATE M ARKOV C HAINS Definition: A Finite State Markov Chain is an Integer Time Process, X n ; n 0 { } in which X n , for each n 0 is a random variable with possible values {1, 2, . .., J } with the Markov Property Definition : The Markov Property PX n = j X n 1 = i , X n 2 = h ,..., X 0 = m () = n = j X n 1 = i ( ) = P ij for all n , i , j , h , m ,... . A Markov chain is completely described by set of transition probabilities P ij plus initial probabilities 0 . Sometimes view P ij { } graphically, sometimes as matrix. P P .... P P P .... P .... P P .... P 1 2 4 3 5 P P P P P P P P P 11 12 41 24 32 23 35 45 55 11 12 21 22 [P] = 25 15 51 52 55 The graph emphasizes the possible and impossible.
Lecture 8 - 2/26/2010 Discrete Stochastic Processes 3 Classification of States Definitions: State j is accessible from i ( i j ) if path from i to j in graph. States i and j communicate if ( i j ) if ( i j ) and ( j i ). If ( i j ) and ( j k ) then ( i k ). Given a walk from i to j , and a walk from j to k , walk from i can be extended to go to k and i k . Similarly k i . Definition: A Class of states is a non-empty set S of states such that ( i j ) for each i S , j S and also for i S , no j S with ( i j ). A class can be though of as a maximal set of communicating states.

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
Lecture 8 - 2/26/2010 Discrete Stochastic Processes 4 RECURRENT AND TRANSIENT STATES 1 2 4 3 5 P P P P P P P P P 11 12 41 24 32 23 35 45 55 6 7 {1,2,3,4} {5} {6,7} Classes Definitions: A class S is recurrent if no j S is accessible from any iS . A recurrent class can be thought of as a “trapping” class – once you get in, you never get out. A class that is not recurrent is transient : {5} recurrent, {1, 2, 3, 4}, {6, 7} transient Transient means there is an outgoing edge from the class and no possible return.
Lecture 8 - 2/26/2010 Discrete Stochastic Processes 5 ( ) 2, 4 2. d = PERIODIC AND APERIODIC STATES For example, P 44 n > 0 for n = 4, 6, 8, 10, . .. . The greatest common divisor is For state i = 1, P 11 n > 0 for n = 4, 8, 10, 12, . .. . d (1) = 2. If d ( i ) = 1, i is defined to be aperiodic; otherwise it is periodic with period d ( i ). Theorem: All states in the same class have the same period.

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

### Page1 / 22

6.262.Lec8 - DISCRETE STOCHASTIC PROCESSES Lecture 8 Finite...

This preview shows document pages 1 - 6. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online