This preview shows pages 1–3. Sign up to view the full content.
Lecture 21 : Continuous Time Markov Chains
STAT 150 Spring 2006
Lecturer: Jim Pitman
Scribe:
Stephen Bianchi
<>
(
These notes also include material from the subsequent guest lecture given by Ani
Adhikari.
)
Consider a continuous time stochastic process (
X
t
, t
≥
0) taking on values in the
fnite state space
S
=
{
0
,
1
,
2
, . . . , N
}
.
Recall that in discrete time, given a transition matrix
P
=
p
(
i, j
) with
i, j
∈
S
, the
n
step transition matrix is simply
P
n
=
p
n
(
i, j
), and that the Following relationship
holds:
P
n
P
m
=
P
n
+
m
.
Where
p
n
(
i, j
) is the probability that the process moves From state
i
to state
j
in
n
transitions. That is,
p
n
(
i, j
)
=
P
i
(
X
n
=
j
)
,
p
n
(
i, j
)
=
P
(
X
n
=
j

X
0
=
i
)
,
p
n
(
i, j
)
=
P
(
X
m
+
n
=
j

X
m
=
i
)
.
Now moving to continuous time, we say that the process (
X
t
, t
≥
0) is a
continuous
time markov chain
iF the Following properties hold For all
i, j
∈
S
,
t, s
≥
0:
•
P
t
(
i, j
)
≥
0,
•
∑
N
j
=0
P
t
(
i, j
) = 1,
•
P
(
X
t
+
s
=
j

X
0
=
i
) =
∑
N
k
=0
P
(
X
t
+
s
=
j

X
s
=
k
)
P
(
X
s
=
k

X
0
=
i
), or
P
(
X
t
+
s
=
j

X
0
=
i
) =
∑
N
k
=0
P
s
(
i, k
)
P
t
(
k, j
).
This last property can be written in matrix Form as
P
s
+
t
=
P
s
P
t
.
This is known as the
ChapmanKolmogorov
equation (the semigroup property).
Example 21.1 (Poissonize a Discrete Time Markov Chain)
Consider a pois
son process
(
N
t
, t
≥
0)
with rate
λ
, and a jump chain with transition matrix
ˆ
P
, which
211
This preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentLecture 21: Continuous Time Markov Chains
212
makes jumps at times
(
N
t
, t
≥
0)
. Let
X
t
=
Y
N
t
(
Y
N
t
is the value of the jump chain
at time
N
t
), where
X
t
takes values in the discrete state space
S
. Assume
(
Y
0
, Y
1
, . . .
)
and
N
t
are independent. Find
P
t
for the process
X
t
.
By de±nition
P
t
(
i, j
) =
P
(
X
t
=
j

X
0
=
i
)
.
Condition on
N
t
to give,
P
t
(
i, j
)
=
∞
X
n
=0
P
(
N
t
=
n
)
ˆ
P
n
(
i, j
)
=
∞
X
n
=0
e

λt
(
λt
)
n
n
!
ˆ
P
n
(
i, j
)
=
e

λt
∞
X
n
=0
(
λt
ˆ
P
)
n
(
i, j
)
n
!
=
e
This is the end of the preview. Sign up
to
access the rest of the document.
 Spring '09
 Pitman,Jim
 Markov Chains

Click to edit the document details