Some worked Numerical Examples on DiscreteTime Markov
Chains for ORIE 361
Instructor: Mark E. Lewis
1
Modelling Example
This is a simple problem to get you into modeling DTMC’s. Consider a rat wandering in a maze. The maze
has six rooms labelled
F,
2
,
3
,
4
,
5
and
S
as in Figure 1. If a room has
k
doors, the probability that the rat
Figure 1: The maze.
selects a particular door is
1
/k
. However, if the rat reaches room
F
, which contains the food, or room
S
,
which gives it an electric shock, then it is kept there, and the experiment stops.
The
transition matrix
can be computed from the above description and figure,
P
=
1
0
0
0
0
0
1
/
2
0
0
1
/
2
0
0
1
/
3
0
0
1
/
3
1
/
3
0
0
1
/
3
1
/
3
0
0
1
/
3
0
0
1
/
2
0
0
1
/
2
0
0
0
0
0
1
where the states are labelled in the order
{
F,
2
,
3
,
4
,
5
, S
}
.
It may seem a bit difficult to see the sets of
recurrent
and
transient
states, so a
transition diagram
is
in order. From this it should be clearer that
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
This is the end of the preview.
Sign up
to
access the rest of the document.
 Spring '07
 LEWIS,M.
 Markov chain, DiscreteTime Markov Chains, Mark E. Lewis

Click to edit the document details