from 9 to 6
,
8
We assume that the mouse is a Markov mouse; i.e., the mouse moves randomly from room
to room, with the probability distribution of the next room depending only on the current
room, not on the history of how it got to the current room. (This is the
Markov property
.)
Moreover, we assume that the mouse is equally likely to choose each of the available doors in
the room it occupies. (That is a special property beyond the Markov property.)
We now model the movement of the mouse as a Markov chain. The
state
of the Markov
chain is the room occupied by the mouse. We let the time index
n
refer to the
n
th
room visited
by the mouse. So we make a discrete-time Markov chain. Speciﬁcally, we let
X
n
be the state
(room) occupied by the mouse on step (or time or transition)
n
. The initial room is
X
0
. The
room after the ﬁrst transition is
X
1
, and so forth. Then
{
X
n
:
n
≥
0
}
is the
discrete-time
Markov chain (DTMC)
; it is a discrete-time discrete-state stochastic process. The mouse
is in room
X
n
( a random variable) after making
n
moves, after having started in room
X
0
,
which could also be a random variable.
We specify the evolution of the Markov chain by specifying the one-step transition prob-
abilities. We specify these transition probabilities by specifying the
transition matrix
. For
our example, making a discrete-time Markov chain model means that we deﬁne a 9
×
9 Markov
transition matrix consistent with the speciﬁcation above. For the most part,
specifying the
transition matrix is specifying the model.
(We also must say how we start. The starting
point could be random, in which case the initial position would be speciﬁed by a probability
vector.)
Notation.
It is common to denote the transition matrix by
P
and its elements by
P
i,j
;
i.e.,
P
i,j
denotes the probability of going to state
j
next when currently in state
i
. (When the
state space has
m
states (is ﬁnite),
P
is a square
m
×
m
matrix.)
For example, for our Markov mouse model, we have
P
1
,
2
= 1
/
2 and
P
1
,
4
= 1
/
2, with
P
1
,j
= 0 for all other
j
, 1
≤
j
≤
9. And we have
P
2
,
1
=
P
2
,
3
=
P
2
,
5
= 1
/
3 with
P
2
,j
= 0 for all
other
j
. And so forth.
Here is the total transition matrix: