IEOR 6711, HMWK 3 Solutions, Professor Sigman
1. Consider a positive recurrent Markov chain with limiting stationary distribution
π
. We
know that
π
i
is, by definition, the longrun proportion of time that the chain moves into
state
i
;
π
i
= lim
n
→∞
1
n
∑
n
k
=1
I
{
X
k
=
i
}
.
It is also a rate: The longrun rate (number of
times per unit time) that the chain moves into state
i
.
(a) Argue that
π
i
is also the longrun proportion of time (rate) that the chain moves out
of state
i
.
SOLUTION:
Every time the chain visits state
i
, it must leave state
i
(at some
time soon after) so as to be able to return to it again (which it must by recurrence).
Thus there is a onetoone correspondence between visits into state
i
and visits out
of state
i
. In fact the number of visits into state
i
during the first
n
units of time
(denote by
N
+
i
(
n
)) is equal to the number of visits out of state
i
(denote by
N

i
(
n
)),
plus or minus 1:
N
+
i
(
n
) =
N

i
(
n
)
±
1 Dividing by
n
and taking the limit as
n
→ ∞
thus yields the two limits (rates) as identical.
(This basic fact has nothing to do
with Markov chains, it is just a basic fact about functions/paths.)
So, “the rate into a state must equal the rate out of that state”
(b) Argue that
π
i
P
i,j
is the longrun rate that the chain moves from state
i
into state
j
.
SOLUTION:
π
is the rate out of state
i
, and
independent of the past
, whenever
the chain is in state
i
it moves next to state
j
with probability
P
i,j
by the Markov
property.
(c) Argue that
X
i
π
i
P
i,j
is the longrun rate that the chain moves into state
j
.
SOLUTION:
Summing up (b) over all states
i
yields, for each fixed
j
∈ S
, that
the longrun rate into state
j
is
X
i
π
i
P
i,j
.
(d) Use the above to conclude that, for each
j
,
π
j
=
X
i
π
i
P
i,j
;
that is,
π
=
πP
.
SOLUTION:
It says that “The rate out of state
j
equals the rate into state
j
, for
each
j
” which we know must be true from (a).
2. George has
r
≥
3 umbrellas distributed between home and office as follows: When de
parting home at the beginning of a day, if it is raining, then he takes an umbrella (if there
is one) with him to the office. Similarly, when departing the office at the end of a day, if
it is raining, then he takes an umbrella (if there is one) with him to home. Assume that
independent of the past there is a fixed probability 0
≤
p
≤
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
This is the end of the preview.
Sign up
to
access the rest of the document.
 Fall '09
 all
 Probability theory, Markov chain, stationary distribution

Click to edit the document details