This preview shows pages 1–2. Sign up to view the full content.
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: IEOR 6711, HMWK 3 Solutions, Professor Sigman 1. Consider a positive recurrent Markov chain with limiting stationary distribution π . We know that π i is, by definition, the longrun proportion of time that the chain moves into state i ; π i = lim n →∞ 1 n ∑ n k =1 I { X k = i } . It is also a rate: The longrun rate (number of times per unit time) that the chain moves into state i . (a) Argue that π i is also the longrun proportion of time (rate) that the chain moves out of state i . SOLUTION: Every time the chain visits state i , it must leave state i (at some time soon after) so as to be able to return to it again (which it must by recurrence). Thus there is a onetoone correspondence between visits into state i and visits out of state i . In fact the number of visits into state i during the first n units of time (denote by N + i ( n )) is equal to the number of visits out of state i (denote by N i ( n )), plus or minus 1: N + i ( n ) = N i ( n ) ± 1 Dividing by n and taking the limit as n → ∞ thus yields the two limits (rates) as identical. (This basic fact has nothing to do with Markov chains, it is just a basic fact about functions/paths.) So, “the rate into a state must equal the rate out of that state” (b) Argue that π i P i,j is the longrun rate that the chain moves from state i into state j . SOLUTION: π is the rate out of state i , and independent of the past , whenever the chain is in state i it moves next to state j with probability P i,j by the Markov property. (c) Argue that X i π i P i,j is the longrun rate that the chain moves into state j . SOLUTION: Summing up (b) over all states i yields, for each fixed j ∈ S , that the longrun rate into state j is X i π i P i,j . (d) Use the above to conclude that, for each j , π j = X i π i P i,j ; that is, π = πP . SOLUTION: It says that “The rate out of state j equals the rate into state j , for each j ” which we know must be true from (a). 2. George has r ≥ 3 umbrellas distributed between home and office as follows: When de parting home at the beginning of a day, if it is raining, then he takes an umbrella (if there is one) with him to the office. Similarly, when departing the office at the end of a day, if it is raining, then he takes an umbrella (if there is one) with him to home. Assume that independent of the past there is a fixed probability 0 ≤ p ≤...
View
Full
Document
This note was uploaded on 01/05/2010 for the course ECE 01 taught by Professor All during the Fall '09 term at Aarhus Universitet.
 Fall '09
 all

Click to edit the document details