6.262.Lec16

6.262.Lec16 - DISCRETE STOCHASTIC PROCESSES Lecture 16...

Info iconThis preview shows pages 1–8. Sign up to view the full content.

View Full Document Right Arrow Icon
Lecture 16 - 3/31/2010 Discrete Stochastic Processes 1 DISCRETE STOCHASTIC PROCESSES Lecture 16 Reversibility in Countable-State Markov Chains Review: Transient, Positive Recurrent and Null Recurrent Classes Steady State Probabilities and Mean Recurrence Times Birth Death Chains Reversible Markov Chains The Markov Chain Model for an M/M/1 Queu and Burke’s Theorem
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Lecture 16 - 3/31/2010 Discrete Stochastic Processes 2 () 0 1 1 since 1 0 1 11 . jj jj n jj jj jj n ET F n F Fn T = = ⎡⎤ = −= ⎣⎦ + Transient and Recurrent States Definition : State j is recurrent if T jj is a non-defective rv (i.e., if chain returns to j in finite steps WP1, which means ( ) lim 1 jj n = →∞ ) A state that is not recurrent is transient . If j is recurrent, then
Background image of page 2
Lecture 16 - 3/31/2010 Discrete Stochastic Processes 3 Alternatively, if j is transient, there is a probability α less than 1 of ever returning to j, () 1 F jj ∞= < , P(n total returns to state j) = 1 αα a f n is geometric and thus: lim tj j Nt →∞ < ∞ a f WP1 and lim j EN t <∞ a f . Also, since P jj n , the probability of arriving back at state j in exactly n steps is precisely the expected number of returns in exactly n steps, it follows that 1 1 [ ( )] (the expected number of returns to state j over the period 1 n t] (the expected number of returns to state j at time n) . [( ) ] . jj t t n jj n n P n P jj jj = = =≤ = = = ∑∑ 1 t n =
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Lecture 16 - 3/31/2010 Discrete Stochastic Processes 4 Lemma 5.1: State j is recurrent N jj t ( ) ; t 0 { } is a renewal process lim tj j Nt →∞ = ∞ a f WP1 lim j EN t =∞ a f lim t jj n nt P ≤≤ 1
Background image of page 4
Lecture 16 - 3/31/2010 Discrete Stochastic Processes 5 . T jj = ∞ Positive Recurrent and Null Recurrent States Definition: State j is positive recurrent if j is recurrent and . T jj < ∞ Definition: State j is null recurrent if j is recurrent and
Background image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Lecture 16 - 3/31/2010 Discrete Stochastic Processes 6 Theorem 5.3 : All states in the same class of a Markov chain are of the same type: either all transient , all positive recurrent , or all null recurrent . The result above lets us classify an entire class as transient or positive recurrent or null-recurrent. Theorem 5.2: Given any two states (i,j) in a recurrent class S in a Markov chain, let N ij (t) be the number of transitions into state j by time t and be the (finite or infinite) expected recurrence time of state j. Then N ij (t) is a delayed renewal process and jj T () [ ] 1 lim lim ij ij tt jj Nt E T →∞ →∞ ==
Background image of page 6
Lecture 16 - 3/31/2010 Discrete Stochastic Processes 7 (Thm. 5.2, cont.) Recall from Blackwell’s theorem that for an arithmetic renewal process with span d, where m(t) = E[N(t)]. Therefore, if state j is aperiodic (and states i and j are in the same recurrent class), then (from Blackwell’s theorem with span d = 1), 0j lim[ ( ) m(n-1)] lim[ (# renewals at time n)] 1 lim ( | ) lim ( = ??) 0.
Background image of page 7

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 8
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 10/21/2010 for the course EE 5581 taught by Professor Moon,j during the Spring '08 term at Minnesota.

Page1 / 23

6.262.Lec16 - DISCRETE STOCHASTIC PROCESSES Lecture 16...

This preview shows document pages 1 - 8. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online