{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

6711-09-h3-solu - IEOR 6711 HMWK 3 Solutions Professor...

Info iconThis preview shows pages 1–2. Sign up to view the full content.

View Full Document Right Arrow Icon
IEOR 6711, HMWK 3 Solutions, Professor Sigman 1. Consider a positive recurrent Markov chain with limiting stationary distribution π . We know that π i is, by definition, the long-run proportion of time that the chain moves into state i ; π i = lim n →∞ 1 n n k =1 I { X k = i } . It is also a rate: The long-run rate (number of times per unit time) that the chain moves into state i . (a) Argue that π i is also the long-run proportion of time (rate) that the chain moves out of state i . SOLUTION: Every time the chain visits state i , it must leave state i (at some time soon after) so as to be able to return to it again (which it must by recurrence). Thus there is a one-to-one correspondence between visits into state i and visits out of state i . In fact the number of visits into state i during the first n units of time (denote by N + i ( n )) is equal to the number of visits out of state i (denote by N - i ( n )), plus or minus 1: N + i ( n ) = N - i ( n ) ± 1 Dividing by n and taking the limit as n → ∞ thus yields the two limits (rates) as identical. (This basic fact has nothing to do with Markov chains, it is just a basic fact about functions/paths.) So, “the rate into a state must equal the rate out of that state” (b) Argue that π i P i,j is the long-run rate that the chain moves from state i into state j . SOLUTION: π is the rate out of state i , and independent of the past , whenever the chain is in state i it moves next to state j with probability P i,j by the Markov property. (c) Argue that X i π i P i,j is the long-run rate that the chain moves into state j . SOLUTION: Summing up (b) over all states i yields, for each fixed j ∈ S , that the long-run rate into state j is X i π i P i,j . (d) Use the above to conclude that, for each j , π j = X i π i P i,j ; that is, π = πP . SOLUTION: It says that “The rate out of state j equals the rate into state j , for each j ” which we know must be true from (a). 2. George has r 3 umbrellas distributed between home and office as follows: When de- parting home at the beginning of a day, if it is raining, then he takes an umbrella (if there is one) with him to the office. Similarly, when departing the office at the end of a day, if it is raining, then he takes an umbrella (if there is one) with him to home. Assume that independent of the past there is a fixed probability 0 p
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Image of page 2
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}