ORIE 3510 – Homework 2 Solutions
Instructor: Mark E. Lewis
due Wednesday February 8, 2012 (ORIE Hallway drop box)
1. Diagram for each Markov chain with the given transition matrix:
0.7
1/4
1/3
i)
iii)
0.3
1
2
0.5
1
1/4
2
0.5
1/4
1/4
1/3
0.25
1/3
ii)
3
0.25
3
4
0.5
1/2
0.5
1/2
1
11
2
0.5
2. First, note that each
X
n
can take value in
{
0
,
1
,
2
,
3
,
4
}
. Since the coins have equal probability
of showing head, we have that
X
n
+1
=
(
X
n
+
B
(4

X
n
,
0
.
4) if
X
n
≤
3,
B
(4
,
0
.
4)
if
X
n
= 4,
where
B
(
n,p
) denotes a binomial random variable with parameters
n
(number of trials) and
p
(probability of success). From this, we can see that the distribution of
X
n
+1
given
X
0
,...,X
n
only depends on
X
n
, i.e.
P
(
X
n
+1
=
x
n
+1

X
0
=
x
0
n
=
x
n
) =
P
(
X
n
+1
=
x
n
+1

X
n
=
x
n
)
So
{
X
n
,n
≥
1
}
is a Markov chain with state space
{
0
,
1
,
2
,
3
,
4
}
. Let
b
(
k
;
) =
P
(
B
(
) =
k
) =
±
n
k
²
p
k
(1

p
)
n

k
,
then the transition matrix has the folowing entries
p
ij
=
b
(
j

i
; 4

i,
0
.
4) if
j
≥
i
and
i
≤
3,
b
(
j
; 4
,
0
.
4)
if
i
= 4,
0
otherwise
,
1