Lecture 21 : Continuous Time Markov Chains
STAT 150 Spring 2006
Lecturer: Jim Pitman
Scribe:
Stephen Bianchi
<>
(
These notes also include material from the subsequent guest lecture given by Ani
Adhikari.
)
Consider a continuous time stochastic process (
X
t
, t
≥
0) taking on values in the
fnite state space
S
=
{
0
,
1
,
2
, . . . , N
}
.
Recall that in discrete time, given a transition matrix
P
=
p
(
i, j
) with
i, j
∈
S
, the
n
step transition matrix is simply
P
n
=
p
n
(
i, j
), and that the Following relationship
holds:
P
n
P
m
=
P
n
+
m
.
Where
p
n
(
i, j
) is the probability that the process moves From state
i
to state
j
in
n
transitions. That is,
p
n
(
i, j
)
=
P
i
(
X
n
=
j
)
,
p
n
(
i, j
)
=
P
(
X
n
=
j

X
0
=
i
)
,
p
n
(
i, j
)
=
P
(
X
m
+
n
=
j

X
m
=
i
)
.
Now moving to continuous time, we say that the process (
X
t
, t
≥
0) is a
continuous
time markov chain
iF the Following properties hold For all
i, j
∈
S
,
t, s
≥
0:
•
P
t
(
i, j
)
≥
0,
•
∑
N
j
=0
P
t
(
i, j
) = 1,
•
P
(
X
t
+
s
=
j

X
0
=
i
) =
∑
N
k
=0
P
(
X
t
+
s
=
j

X
s
=
k
)
P
(
X
s
=
k

X
0
=
i
), or
P
(
X
t
+
s
=
j

X
0
=
i
) =
∑
N
k
=0
P
s
(
i, k
)
P
t
(
k, j
).
This last property can be written in matrix Form as
P
s
+
t
=
P
s
P
t
.
This is known as the
ChapmanKolmogorov
equation (the semigroup property).
Example 21.1 (Poissonize a Discrete Time Markov Chain)
Consider a pois
son process
(
N
t
, t
≥
0)
with rate
λ
, and a jump chain with transition matrix
ˆ
P
, which
211