J. Virtamo
38.3143 Queueing Theory / Markov processes
1
Markov processes
(Continuous time Markov chains)
Consider (stationary) Markov processes with a continuous parameter space (the parameter
usually being time). Transitions from one state to another can occur at any instant of time.
•
Due to the Markov property, the time the system spends in any given state is memoryless:
the distribution of the remaining time depends solely on the state but not on the time
already spent in the state
⇒
the time is exponentially distributed.
A Markov process
X
t
is completely determined by the so called generator matrix
or transition rate matrix
q
i,j
= lim
Δ
t
→
0
P
{
X
t
+Δ
t
=
j

X
t
=
i
}
Δ
t
i
negationslash
=
j
 probability per time unit that the system makes a transition from state
i
to state
j
 transition rate or transition intensity
The total transition rate out of state
i
is
q
i
=
summationdisplay
j
negationslash
=
i
q
i,j

lifetime of the state
∼
Exp(
q
i
)
This is the rate at which the probability of state
i
decreases. Define
q
i,i
=

q
i
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
J. Virtamo
38.3143 Queueing Theory / Markov processes
2
Transition rate matrix and time dependent state probability vector
The transition rate matrix in full is
Q
=
q
0
,
0
q
0
,
1
. . .
q
1
,
0
q
1
,
1
. . .
.
.
.
.
.
.
.
.
.
=

q
0
q
0
,
1
. . .
q
1
,
0

q
1
. . .
.
.
.
.
.
.
.
.
.
row sums equal zero:
the probability mass flowing out of state
i
will go to some other states (is conserved)
State probability vector
π
(
t
) is now a function of time evolving as follows
d
dt
π
(
t
) =
π
(
t
)
·
Q
⇒
π
(
t
+ Δ
t
) =
π
(
t
) +
π
(
t
)
·
Q
Δ
t
+
o
(Δ
t
) =
π
(
t
)(
I
+
Q
Δ
t
) +
o
(Δ
t
)
Transition probability matrix over time interval Δ
t
is
P
(Δ
t
) =
I
+
Q
Δ
t
 tends to the identity matrix
I
as Δ
t
→
0

Q
=
P
prime
(0) is the time derivative of the transition prob. matrix (transition rate matrix)
This is the end of the preview.
Sign up
to
access the rest of the document.
 Spring '09
 all
 Markov chain, Continuoustime Markov process, Xt, J. Virtamo

Click to edit the document details