6.262.Lec16

# 6.262.Lec16 - DISCRETE STOCHASTIC PROCESSES Lecture 16...

This preview shows pages 1–8. Sign up to view the full content.

Lecture 16 - 3/31/2010 Discrete Stochastic Processes 1 DISCRETE STOCHASTIC PROCESSES Lecture 16 Reversibility in Countable-State Markov Chains Review: Transient, Positive Recurrent and Null Recurrent Classes Steady State Probabilities and Mean Recurrence Times Birth Death Chains Reversible Markov Chains The Markov Chain Model for an M/M/1 Queu and Burke’s Theorem

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
Lecture 16 - 3/31/2010 Discrete Stochastic Processes 2 () 0 1 1 since 1 0 1 11 . jj jj n jj jj jj n ET F n F Fn T = = ⎡⎤ = −= ⎣⎦ + Transient and Recurrent States Definition : State j is recurrent if T jj is a non-defective rv (i.e., if chain returns to j in finite steps WP1, which means ( ) lim 1 jj n = →∞ ) A state that is not recurrent is transient . If j is recurrent, then
Lecture 16 - 3/31/2010 Discrete Stochastic Processes 3 Alternatively, if j is transient, there is a probability α less than 1 of ever returning to j, () 1 F jj ∞= < , P(n total returns to state j) = 1 αα a f n is geometric and thus: lim tj j Nt →∞ < ∞ a f WP1 and lim j EN t <∞ a f . Also, since P jj n , the probability of arriving back at state j in exactly n steps is precisely the expected number of returns in exactly n steps, it follows that 1 1 [ ( )] (the expected number of returns to state j over the period 1 n t] (the expected number of returns to state j at time n) . [( ) ] . jj t t n jj n n P n P jj jj = = =≤ = = = ∑∑ 1 t n =

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
Lecture 16 - 3/31/2010 Discrete Stochastic Processes 4 Lemma 5.1: State j is recurrent N jj t ( ) ; t 0 { } is a renewal process lim tj j Nt →∞ = ∞ a f WP1 lim j EN t =∞ a f lim t jj n nt P ≤≤ 1
Lecture 16 - 3/31/2010 Discrete Stochastic Processes 5 . T jj = ∞ Positive Recurrent and Null Recurrent States Definition: State j is positive recurrent if j is recurrent and . T jj < ∞ Definition: State j is null recurrent if j is recurrent and

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
Lecture 16 - 3/31/2010 Discrete Stochastic Processes 6 Theorem 5.3 : All states in the same class of a Markov chain are of the same type: either all transient , all positive recurrent , or all null recurrent . The result above lets us classify an entire class as transient or positive recurrent or null-recurrent. Theorem 5.2: Given any two states (i,j) in a recurrent class S in a Markov chain, let N ij (t) be the number of transitions into state j by time t and be the (finite or infinite) expected recurrence time of state j. Then N ij (t) is a delayed renewal process and jj T () [ ] 1 lim lim ij ij tt jj Nt E T →∞ →∞ ==
Lecture 16 - 3/31/2010 Discrete Stochastic Processes 7 (Thm. 5.2, cont.) Recall from Blackwell’s theorem that for an arithmetic renewal process with span d, where m(t) = E[N(t)]. Therefore, if state j is aperiodic (and states i and j are in the same recurrent class), then (from Blackwell’s theorem with span d = 1), 0j lim[ ( ) m(n-1)] lim[ (# renewals at time n)] 1 lim ( | ) lim ( = ??) 0.

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

## This note was uploaded on 10/21/2010 for the course EE 5581 taught by Professor Moon,j during the Spring '08 term at Minnesota.

### Page1 / 23

6.262.Lec16 - DISCRETE STOCHASTIC PROCESSES Lecture 16...

This preview shows document pages 1 - 8. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online