IEOR 4106 Midterm practice problems: Spring, 2008
1. Consider the rat in the open maze (state space
S
=
{
0
,
1
,
2
,
3
,
4
}
).
Given that the rat
starts in cell 1, what is the expected total number of visits to cell 2 before the rat escapes?
SOLUTION :
For
i
and
j
any transient states (from 1
,
2
,
3
,
4), let
s
i,j
denote the expected
total number of visits to state
j
, given
X
0
=
i
.
S
= (
s
i,j
) is a 4
×
4 matrix. We are to
find the value of
s
1
,
2
.
If we condition on the first move,
X
1
= 2 or
X
1
= 3, we obtain by the Markov property,
s
1
,
2
=
s
2
,
2
P
1
,
2
+
s
3
,
2
P
1
,
3
= (1
/
2)(
s
2
,
2
+
s
3
,
2
)
.
We can continue this type of conditioning to obtain a set of 4 linear equations for the 4
unknowns
s
i,
2
, i
= 1
,
2
,
3
,
4:
s
1
,
2
=
s
2
,
2
P
1
,
2
+
s
3
,
2
P
1
,
3
= (1
/
2)(
s
2
,
2
+
s
3
,
2
)
s
2
,
2
=
1 +
s
1
,
2
P
2
,
1
+
s
4
,
2
P
2
,
4
= 1 + (1
/
2)(
s
1
,
2
+
s
4
,
2
)
s
3
,
2
=
s
1
,
2
P
3
,
1
+
s
4
,
2
P
3
,
4
= (1
/
2)(
s
1
,
2
+
s
4
,
2
)
s
4
,
2
=
s
2
,
2
P
4
,
2
+
s
3
,
2
P
4
,
3
= (1
/
3)(
s
2
,
2
+
s
3
,
2
)
.
Solving (left to reader) leads to the solution (
s
1
,
2
, s
2
,
2
, s
3
,
2
, s
4
,
2
) = (3
,
3
.
5
,
2
.
5
,
2);
s
1
,
2
= 3.
More elegantly: Letting
P
T
denote the 4
×
4 transition matrix for (only) states 1
,
2
,
3
,
4
(the transient states),
P
T
=
0
1
2
1
2
0
1
2
0
0
1
2
1
2
0
0
1
2
0
1
3
1
3
0
,
we see that in fact the above equations, when expanded to include all 16 of the
s
i,j
, in
matrix form become:
S
=
I
+
SP
T
,
where
I
is the identity matrix. Rewriting this as
S

SP
T
=
I,
or
S
(
I

P
T
) =
I
, we thus
can solve for
S
all a once:
S
= (
I

P
T
)

1
.