Unformatted text preview: Probability and Statistics with Reliability,
Queuing and Computer Science
Applications
Second edition
by K.S. Trivedi
PublisherJohn Wiley & Sons Chapter 8 (Part 1) :Continuous Time Markov
Chains: Theory
Dept. of Electrical & Computer engineering
Duke University
Email:[email protected]
URL: www.ee.duke.edu/~kst
Copyright © 2003 by K.S. Trivedi 1 NonState Space Models
• Recall that nonstatespace models like Reliability Block Diagrams
and Fault Trees can easily be formulated and solved for system
reliability, system availability and system MTTF
• Each component can have attached to it
– A probability of failure
– A failure rate
– A distribution of time to failure
– Steadystate and instantaneous unavailability
• Assume
– all the components failure events and repair events are
independent of each other
– simple logical relationships between system and its components
Copyright © 2003 by K.S. Trivedi 2 Statespace Models
• To model complex interactions between system
components, we need Statespace models.
• Example: Markov chains or more general Statespace models.
• Many examples of dependencies among system
components have been observed in practice and
captured by Statespace models.
Copyright © 2003 by K.S. Trivedi 3 StateSpace Models (Contd.)
• Drawn as a directed graph
• State: each state represents a system condition
• Transitions between states indicates occurrence of
events. Each transition can be labeled by
– Probability: homogeneous DiscreteTime Markov
Chain (DTMC); Chapter 7 of the text
– TimeindependentRate: homogeneous ContinuousTime Markov chain (CTMC); Chapter 8 of the text
– Timedependent rate: nonhomogeneous CTMC;
some examples in Chapter 8 of the text
– Distribution function: SemiMarkov process (SMP) ;
some examples in Chapter 7 & Chapter 8 of the text
– Two distribution functions; Markov regenerative
process (MRGP); not treated in the text
Copyright © 2003 by K.S. Trivedi 4 Continuoustime Markov
Chains(CTMC)
• Discrete state space – Markov Chains
• For Continuoustime Markov chains (CTMCs) the
time variable associated with the system
evolution is continuous.
• In this chapter, we will mean a CTMC whenever
we speak of a Markov model (chain).
Copyright © 2003 by K.S. Trivedi 5 Formal Definition
• A discretestate continuoustime stochastic
process
is called a Markov chain if
for t0 < t1 < t2 < …. < tn < t , the conditional
pmf satisfies the following Markov property: • A CTMC is characterized by state changes
that can occur at any arbitrary time
• Index space is continuous.
• The state space is discrete
Copyright © 2003 by K.S. Trivedi 6 CTMC
• A CTMC can be completely described
by:
– Initial state probability vector for X(t0):
– Transition probability functions (over an
interval) Copyright © 2003 by K.S. Trivedi 7 Homogenous CTMC
• is a (time)homogenous CTMC iff • Or, the conditional pmf satisfies:
• State probabilities at a time t or pmf is given
by
• Using the theorem of total probability
If v = 0 in the above equation, we get
Copyright © 2003 by K.S. Trivedi 8 CTMC Dynamics
ChapmanKolmogorov Equation • Unlike the case of DTMC, the transition probabilities
are functions of elapsed time and not of the number of
elapsed steps
• The direct use of the this equation is difficult unlike the
case of DTMC where we could anchor on onestep
transition probabilities
• Hence the notion of rates of transitions which follows
next
Copyright © 2003 by K.S. Trivedi 9 Transition Rates
• Define the rates (probabilities per unit time):
net rate out of state j at time t: • the rate from state i to state j at time t: Copyright © 2003 by K.S. Trivedi 10 Kolmogorov Differential
Equation
• The transition probabilities and transition rates are, • Dividing both sides by h and taking the limit, Copyright © 2003 by K.S. Trivedi 11 Kolmogorov Differential Equation
(contd.)
• Kolmogorov’s backward equation,
• Writing these eqns. in the matrix form, Copyright © 2003 by K.S. Trivedi 12 Homogeneous CTMC
Specialize to HCTMC: all transition rates are time independent
(Kolmogorov diff. equations) : •In the matrix form, (Matrix Q is called the infinitesimal
generator matrix (or simply Generator Matrix)) Copyright © 2003 by K.S. Trivedi 13 Infinitesimal Generator Matrix
• Infinitesimal Generator Matrix Q = [ qij ]
For any offdiagonal element qij , i ≠ j is the transition
rate from state i to state j
• Characteristics of the generator matrix Q
– Row sum equal to zero
– Diagonal elements are all nonpositive
– Offdiagonal elements are all nonnegative Copyright © 2003 by K.S. Trivedi 14 NHCTMC vs. HCTMC
• In the nonhomogeneous case, one or more transition
rate is time dependent where time is measured from the
beginning of system operation (global time) whereas in
a semi Markov process rates are time dependent where
time is measured from the entry into current state (local
time)
• Markov property is satisfied at any time by an NHCTMC
or HCTMC while the Markov property is satisfied by a
semi Markov process only at epochs of entry (or exit)
from a state
• Sojourn times in states of an HCTMC are exponentially
distributed but this is not necessarily true for an
NHCTMC or an SMP
Copyright © 2003 by K.S. Trivedi 15 Definitions
• A CTMC is said to be irreducible if every state can be
reached from every other state, with a nonzero
probability.
• A state is said to be absorbing if no other state can be
reached from it with nonzero probability.
• Notion of transient, recurrent nonnull, recurrent null
are the same as in a DTMC. There is no notion of
periodicity in a CTMC, however.
• Unless otherwise specified, when we say CTMC, we
mean HCTMC Copyright © 2003 by K.S. Trivedi 16 CTMC Steadystate Solution
• Steady state solution of CTMC obtained by solving
the following balance equations (πj is the limiting
value of πj(t)) : • Irreducible CTMCs with all states recurrent nonnull
will have steadystate {πj} values that are unique and
independent of the initial probability vector. All states
of a finite irreducible CTMC will be recurrent nonnull.
• Measures of interest may be computed as weighted
sum of steady state probabilities: Copyright © 2003 by K.S. Trivedi 17 CTMC Measures
• For as a weighted sum of transient probabilities at time t: E[ Z (t )] = ∑ rj π j (t )
j • Expected accumulated quantites (over an interval of time (0,t]) • Lj(t) is the expected time spent in state j during (0,t]
(LTODE)
Copyright © 2003 by K.S. Trivedi 18 Different Analyses of HCTMC
• Steady State Analysis: Solving a linear system of
equations:
• Transient Analysis: Solving a coupled system of
linear first order ordinary diff. Equations, initial
value problem with const. coefficients: • Cumulative Transient Analysis: ODEIVP with a
forcing function:
Copyright © 2003 by K.S. Trivedi 19 Markov Reward Models
(MRMs) Copyright © 2003 by K.S. Trivedi 20 Markov Reward Models
(MRMs)
• Continuous Time Markov Chains are useful
models for performance as well as
availability prediction
• Extension of CTMC to Markov reward
models make them even more useful
• Attach a reward rate (or a weight) ri to state i
of CTMC. Let Z(t)=rX(t) be the instantaneous
reward rate of CTMC at time t
Copyright © 2003 by K.S. Trivedi 21 Markov Reward Models
(MRMs) (Continued)
• Define Y(t) the accumulated reward in the interval
t
(0,t] Y (t ) = ∫ Z (τ )dτ
0 • Computing the expected values of these
measures is easy
• For computing distribution of Y(t) see the
Perfomability monograph edited by Haverkort,
Marie, Rubino & Trivedi
Copyright © 2003 by K.S. Trivedi 22 3State Markov Reward Model with Sample
Paths of X(t) and Z(t) Processes
2λ – r1 = 1.7
– r2 = 1.0
– r3 = 0.0 1 λ
2 µ2 Copyright © 2003 by K.S. Trivedi 3 µ3 23 3State Markov Reward Model with Sample
Paths of X(t) and Z(t) Processes Copyright © 2003 by K.S. Trivedi 24 3State Markov Reward Model with
Sample Path of Y(t) Processes Copyright © 2003 by K.S. Trivedi 25 Markov Reward Models
(MRMs) (Continued)
• Expected instantaneous reward rate at
time t: E[ Z (t )] = ∑ riπ i (t )
i Expected steadystate reward rate: E[ Z ] = ∑ riπ i
i Copyright © 2003 by K.S. Trivedi 26 MRM Measures
• Expected accumulated reward in interval
(0,t] E[Y (t )] = ∑ ri ∫ π i ( x)dx =∑ ri Li (t )
t i 0 i Copyright © 2003 by K.S. Trivedi 27 ...
View
Full Document
 Spring '10
 MohammadAbdolahiAzgomiPh.D
 Markov process, Markov chain, Continuoustime Markov process, K.S. Trivedi

Click to edit the document details