Introduction to Stochastic Processes, Spring 2011
Homework #4
Due: Thursday, February 10th, 2011 in class
Stationary Distributions for Markov Chains
1. Consider a twostate Markov Chain with the states labelled 0 and 1. Let the transition
matrix be
1

α
α
β
1

β
where 0
< α <
1 and 0
< β <
1. The first row is for state 0, the second is for state 1.
(i) Compute the stationary distribution
π
= [
π
0
π
1
].
(ii) For
n
≥
1, let
f
(
n
) =
P
(the first return of the Markov chain to state 0 is at time
n

X
0
= 0)
.
Compute
f
(
n
) for
n
≥
1.
(iii) Let
m
0
be the mean number of time steps it takes for the Markov chain to return
to zero, given that the chain starts at position zero. Use the formula for
f
(
n
) to
compute
m
0
. Check that
π
0
= 1
/m
0
.
2. Recall the umbrella problem of Assignment 3:
A man has a total of
N
umbrellas that he splits between home and work.
Before
travelling from one place to the other, he looks out the window and if it’s raining he
takes an umbrella with him, otherwise he leaves without one. At any given time there
is a 20% chance that it is raining. Suppose that at time zero he is at home, at time one
he is at work, at time two he’s back home and so on, and let
X
j
, j
≥
0 be the number
of umbrellas at his location at time
j
.
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
This is the end of the preview.
Sign up
to
access the rest of the document.
 Spring '11
 nomane
 Markov Chains, Markov chain

Click to edit the document details