IEOR 4106: Professor Whitt
Lecture Notes, Tuesday, February 3
Introduction to Markov Chains
1. Markov Mouse: The Closed Maze
We start by considering how to model a mouse moving around in a maze. The maze is a
closed space containing nine rooms. The space is arranged in a threebythree array of rooms,
with doorways connecting the rooms, as shown in the figure below
The Maze
1
2
3
4
5
6
7
8
9
There are doors leading to adjacent rooms, vertically and horizontally. In particular, there
are doors
from
1
to
2
,
4
from
2
to
1
,
3
,
5
from
3
to
2
,
6
from
4
to
1
,
5
,
7
from
5
to
2
,
4
,
6
,
8
from
6
to
3
,
5
,
9
from
7
to
4
,
8
from
8
to
5
,
7
,
9
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
from
9
to
6
,
8
We assume that the mouse is a Markov mouse; i.e., the mouse moves randomly from room
to room, with the probability distribution of the next room depending only on the current
room, not on the history of how it got to the current room. (This is the
Markov property
.)
Moreover, we assume that the mouse is equally likely to choose each of the available doors in
the room it occupies. (That is a special property beyond the Markov property.)
We now model the movement of the mouse as a Markov chain. The
state
of the Markov
chain is the room occupied by the mouse. We let the time index
n
refer to the
n
th
room visited
by the mouse. So we make a discretetime Markov chain. Specifically, we let
X
n
be the state
(room) occupied by the mouse on step (or time or transition)
n
. The initial room is
X
0
. The
room after the first transition is
X
1
, and so forth. Then
{
X
n
:
n
≥
0
}
is the
discretetime
Markov chain (DTMC)
; it is a discretetime discretestate stochastic process. The mouse
is in room
X
n
( a random variable) after making
n
moves, after having started in room
X
0
,
which could also be a random variable.
We specify the evolution of the Markov chain by specifying the onestep transition prob
abilities. We specify these transition probabilities by specifying the
transition matrix
. For
our example, making a discretetime Markov chain model means that we define a 9
×
9 Markov
transition matrix consistent with the specification above. For the most part,
specifying the
transition matrix is specifying the model.
(We also must say how we start. The starting
point could be random, in which case the initial position would be specified by a probability
vector.)
Notation.
It is common to denote the transition matrix by
P
and its elements by
P
i,j
;
i.e.,
P
i,j
denotes the probability of going to state
j
next when currently in state
i
. (When the
state space has
m
states (is finite),
P
is a square
m
×
m
matrix.)
For example, for our Markov mouse model, we have
P
1
,
2
= 1
/
2 and
P
1
,
4
= 1
/
2, with
P
1
,j
= 0 for all other
j
, 1
≤
j
≤
9. And we have
P
2
,
1
=
P
2
,
3
=
P
2
,
5
= 1
/
3 with
P
2
,j
= 0 for all
other
j
. And so forth.
Here is the total transition matrix:
P
=
1
2
3
4
5
6
7
8
9
0
1
/
2
0
1
/
2
0
0
0
0
0
1
/
3
0
1
/
3
0
1
/
3
0
0
0
0
0
1
/
2
0
0
0
1
/
2
0
0
0
1
/
3
0
0
0
1
/
3
0
1
/
3
0
0
0
1
/
4
0
1
/
4
0
1
/
4
0
1
/
4
0
0
0
1
/
3
1
/
3
0
0
0
1
/
3
0
0
0
1
/
2
0
0
0
1
/
2
0
0
0
0
0
1
/
3
0
1
/
3
0
1
/
3
0
0
0
0
0
1
/
2
0
1
/
2
0
.
This is the end of the preview.
Sign up
to
access the rest of the document.
 Spring '08
 Whitt
 Markov Chains, Markov chain, absorbing Markov Chain

Click to edit the document details