Markov Chains, part I
December 8, 2010
1
Introduction
A Markov Chain is a sequence of random variables
X
0
, X
1
, ...
, where each
X
i
∈ S
, such that
P
(
X
i
+1
=
s
i
+1

X
i
=
s
i
, X
i
−
1
=
s
i
−
1
, ..., X
0
=
s
0
) =
P
(
X
i
+1
=
s
i
+1

X
i
=
s
i
);
that is, the value of the next random variable in dependent at most on the
value of the previous random variable.
The set
S
here is what we call the “state space”, and it can be either
continuous or discrete (or a mix); however, in our discussions we will take
S
to be discrete, and in fact we will always take
S
=
{
1
,
2
, ..., N
}
.
Since
X
t
+1
only depends on
X
t
, it makes sense to define “transition prob
abilities”
P
i,j
:=
P
(
X
t
+1
=
j

X
t
=
i
)
,
which completely determine the dynamics of the Markov chain...
well, al
most: we need to either be given
X
0
, or we to choose its value according
to some distribution on the state space.
In the theory of Hidden Markov
Models, one has a set of probabilities
π
1
, ..., π
N
,
π
1
+
· · ·
+
π
N
= 1, such
that
P
(
X
0
=
i
) =
π
i
; however, in some other applications, such as in the
Gambler’s Ruin Problem discussed in another note, we start with the value
for
X
0
.
Ok, so how could we generate a sequence
X
0
, X
1
, ...
, given
X
0
and given
the
P
i,j
’s?
Well, suppose
X
0
=
i
.
Then, we choose
X
1
at random from
{
1
,
2
, ..., N
}
, where
P
(
X
1
=
j
) =
P
i,j
. Next, we select
X
2
at random accord
ing to the distribution
P
(
X
2
=
k
) =
P
j,k
. We then continue the process.
1
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
1.1
Graphical representation
Sometimes, a more convenient way to represent a Markov chain is to use a
transition diagram
, which is a graph on
N
vertices that represent the states.
The edges are directed, and each corresponds to a transition probability
P
i,j
;
however, not all the
N
2
edges are necessarily in the graph – when an edge is
missing, it means that the corresponding
P
i,j
has value 0.
Here is an example: suppose that
N
= 3, and suppose
P
1
,
1
= 1
/
3
, P
1
,
2
= 2
/
3
, P
2
,
1
= 1
/
2
, P
2
,
3
= 1
/
2
, P
3
,
1
= 1
.
Then, the corresponding transition diagram looks like this
d47d46d45d44
d40d41d42d43
1
2
/
3
d47
1
/
3
d45
d47d46d45d44
d40d41d42d43
2
1
/
2
d47
1
/
2
d101
d47d46d45d44
d40d41d42d43
3
1
d116
1.2
Matrix representation, and population distribu
tions
It is also convenient to collect together the
P
i,j
’s into an
N
×
N
matrix; and,
I will do this here a little bit backwards from how you might see it presented
in other books, for reasons that will become clear later on: form the matrix
P
whose (
j, i
) entry is
P
i,j
; so, the
i
th column of the matrix represents all
the transition probabilities
out of
node
i
, while the
j
th row represents all
transition probabilities
into
node
j
. For example, the matrix corresponding
to the example in the previous subsection is
P
=
P
1
,
1
P
2
,
1
P
3
,
1
P
1
,
2
P
2
,
2
P
3
,
2
P
1
,
3
P
2
,
3
P
3
,
3
=
1
/
3
1
/
2
1
2
/
3
0
0
0
1
/
2
0
.
Notice that the sum of entries down a column is 1.
Now we will reinterpret this matrix in terms of population distributions:
suppose that the states 1
, ..., N
represent populations – say state
i
represents
“country
i
”.
Associated to each of these populations, we let
p
i
(
t
) denote
the fraction of some total population residing in country
i
at time
t
.
In
This is the end of the preview.
Sign up
to
access the rest of the document.
 Spring '08
 Staff
 Linear Algebra, Statistics, Matrices, Markov Chains, Probability, Markov chain

Click to edit the document details