An idiosyncratic introduction to stochastic processes
Class notes for Math 216 Notes  Fall 2010
Jonathan C. Mattingly
September 22, 2010
1
Finite State Markov Chains
A discrete time stochastic process (
X
n
)
n
≥
0
is a collection of random variables indexed by the nonnegative
integers
Z
+
=
{
n
∈
Z
:
n
≥
0
}
. The set in which the
X
n
take values is called the state space of the stochastic
process.
Definition.
A stochastic process (
X
n
)
n
≥
0
is a
Markov chain
if
P
(
X
n
+1
=
j

X
n
=
i
n
,
· · ·
, X
0
=
i
0
) =
P
(
X
n
+1
=
j

X
n
=
i
n
)
for all
j, i
n
,
· · ·
, i
0
∈
I
.
Definition.
A Markov chain is
time homogeneous
if for all
k
∈
Z
+
and
i, j
∈
I
P
(
X
k
+1
=
i

X
k
=
j
) =
P
(
X
1
=
i

X
=
j
)
Unless we say otherwise we will always assume that all Markov chains are time homogeneous. In such
cases we will write
p
n
(
i, j
) =
P
(
X
n
=
j

X
0
=
i
)
By the Markov property one has
P
(
X
n
=
x
n
, X
n

1

x
n

1
,
· · ·
X
1
=
x
1

X
0
=
x
0
) =
p
1
(
x
n
, x
n

1
)
p
1
(
x
n

1
, x
n

2
)
· · ·
p
1
(
x
1
, x
0
)
We will begin by concentrating on stochastic processes on a finite state space
I
.
With out loss of
generality, we can take the state space to be
I
=
{
0
,
1
. . . .
, N
}
.
1.1
Markov chains and matrices
There is a very fruitful correspondence between finite state Markov chains and Matrices.
We begin by
considering random variables on a state space
I
=
{
0
, . . . , N

1
}
.
Such a random variable
X
can be
specified completely by
N
nonnegative numbers
{
λ
i
:
i
∈
I
}
such that
P
(
X
=
i
) =
λ
i
.
Clearly we have that
∑
i
∈
I
λ
i
= 1. It is convenient to organize the
λ
i
in a rowvector
λ
= (
λ
0
, . . . , λ
N

1
)
∈
R
N
.
The vector
λ
is called the distribution of the random variable
X
.
With this in mind we make the
following definition.
Definition.
A row vector
λ
= (
λ
0
, . . . , λ
N

1
)
∈
R
N
called a
distribution
if
λ
i
≥
0.
If in addition
∑
N

1
i
=0
λ
i
= 1, it is called a
probability distribution
.
1
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Let
P
∈
R
N,N
be a matrix with nonnegative entries. We will write
P
i,j
for the
i

jth
entry of
P
, that
is to say
P
=
p
0
,
0
· · ·
p
0
,N

1
.
.
.
.
.
.
.
.
.
p
N

1
,
0
· · ·
p
N

1
,N

1
Definition.
A square matrix
P
with nonnegative entries is called a
stochastic matrix
if all rows sum
to one. That is to say, for all
j
,
∑
i
P
ji
= 1.
Stochastic matrices are in onetoone correspondence with time homogeneous Markov processes on a
finite state space. The correspondence is given by
P
ij
=
P
(
X
1
=
j

X
0
=
i
)
It then follows that
P
(
X
n
=
j

X
0
=
i
) = (
P
n
)
ij
Or in other words, the distribution of the random variable
X
n
when conditioned to have
X
0
=
i
is given by
the row vector (
P
n
)
i,
*
by which we mean the
i
th row of the matrix
P
n
. In other words,
(
(
P
n
)
i,
0
, . . . ,
(
P
n
)
i,N

1
)
(1)
If we denote by
e
(
i
) the row vector with 1 in the
i
th slot and 0 in the remaining slots, then (1) can be written
compactly as
e
(
i
)
P
n
.
If instead of starting from a deterministic initial condition, we let
X
0
be random with distribution given
by the probability distribution
λ
= (
λ
1
,
· · ·
, λ
N

1
). This means that
P
(
X
0
=
i
) =
λ
i
.
This is the end of the preview.
Sign up
to
access the rest of the document.
 Fall '08
 Mckinley,S
 Markov Chains, TA, Markov chain

Click to edit the document details