ORIE 361 – Homework 2 (Introduction to Discrete Time Markov
Chains)
Instructor: Mark E. Lewis
due February 5, 2007 (drop box)
Answers.
1. The state space is
Ω =
{
0
,
1
, . . . , d
}
and
P
i,j
=
(
2
i
j
)(
2
d

2
i
d

j
)
(
2
d
d
)
j
= 0
,
1
, . . . , d, i
= 1
,
2
, . . . , d

1
0 and d are the two absorbing states.
P
0
,
0
= 1
, P
d,d
= 1
2. Let, the state be the number of umbrellas he has at his present location. The state space =
{
0,1,
. . .
,r
}
. The transition probabilities are
P
0
,r
= 1
, P
0
,i
= 0
i
= 0
,
1
,
2
, . . . , r

1
.
P
i,r

i
= 1

p, P
i,r

i
+1
=
p
i
= 1
,
2
, . . . , r.
P
i,j
= 0
ifj /
∈ {
r

i, r

i
+ 1
}
i
= 1
,
2
, . . . , r.
3. i) In order to have
X
k
= 0, based on our definition of the process, we need to have
2
,
3
,
....
,
k
=
0.
So,
P
(
X
k
= 0) =
P
(
2
= 0
,
3
= 0
,
....
,
k
= 0) =
1
2
k

1
ii)
P
(
X
5
= 1

X
4
= 1
, X
3
= 0) =
P
(
X
4
+
X
3
+
5
= 1

X
4
= 1
, X
3
= 0)
=
P
(
5
= 0

X
4
= 1
, X
3
= 0) =
P
(
5
= 1) =
1
2
iii)
P
(
X
5
= 1

X
4
= 1) =
P
(
X
5
= 1

X
4
= 1
, X
3
= 1)
P
(
X
3
= 1

X
4
= 1)
+
P
(
X
5
= 1

X
4
= 1
, X
3
= 0)
P
(
X
3
= 0

X
4
= 1)
1