chap8_p1 - Probability and Statistics with Reliability,...

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Probability and Statistics with Reliability, Queuing and Computer Science Applications Second edition by K.S. Trivedi Publisher-John Wiley & Sons Chapter 8 (Part 1) :Continuous Time Markov Chains: Theory Dept. of Electrical & Computer engineering Duke University Email:kst@ee.duke.edu URL: www.ee.duke.edu/~kst Copyright © 2003 by K.S. Trivedi 1 Non-State Space Models • Recall that non-state-space models like Reliability Block Diagrams and Fault Trees can easily be formulated and solved for system reliability, system availability and system MTTF • Each component can have attached to it – A probability of failure – A failure rate – A distribution of time to failure – Steady-state and instantaneous unavailability • Assume – all the components failure events and repair events are independent of each other – simple logical relationships between system and its components Copyright © 2003 by K.S. Trivedi 2 State-space Models • To model complex interactions between system components, we need State-space models. • Example: Markov chains or more general Statespace models. • Many examples of dependencies among system components have been observed in practice and captured by State-space models. Copyright © 2003 by K.S. Trivedi 3 State-Space Models (Contd.) • Drawn as a directed graph • State: each state represents a system condition • Transitions between states indicates occurrence of events. Each transition can be labeled by – Probability: homogeneous Discrete-Time Markov Chain (DTMC); Chapter 7 of the text – Time-independent-Rate: homogeneous ContinuousTime Markov chain (CTMC); Chapter 8 of the text – Time-dependent rate: non-homogeneous CTMC; some examples in Chapter 8 of the text – Distribution function: Semi-Markov process (SMP) ; some examples in Chapter 7 & Chapter 8 of the text – Two distribution functions; Markov regenerative process (MRGP); not treated in the text Copyright © 2003 by K.S. Trivedi 4 Continuous-time Markov Chains(CTMC) • Discrete state space – Markov Chains • For Continuous-time Markov chains (CTMCs) the time variable associated with the system evolution is continuous. • In this chapter, we will mean a CTMC whenever we speak of a Markov model (chain). Copyright © 2003 by K.S. Trivedi 5 Formal Definition • A discrete-state continuous-time stochastic process is called a Markov chain if for t0 < t1 < t2 < …. < tn < t , the conditional pmf satisfies the following Markov property: • A CTMC is characterized by state changes that can occur at any arbitrary time • Index space is continuous. • The state space is discrete Copyright © 2003 by K.S. Trivedi 6 CTMC • A CTMC can be completely described by: – Initial state probability vector for X(t0): – Transition probability functions (over an interval) Copyright © 2003 by K.S. Trivedi 7 Homogenous CTMC • is a (time-)homogenous CTMC iff • Or, the conditional pmf satisfies: • State probabilities at a time t or pmf is given by • Using the theorem of total probability If v = 0 in the above equation, we get Copyright © 2003 by K.S. Trivedi 8 CTMC Dynamics Chapman-Kolmogorov Equation • Unlike the case of DTMC, the transition probabilities are functions of elapsed time and not of the number of elapsed steps • The direct use of the this equation is difficult unlike the case of DTMC where we could anchor on one-step transition probabilities • Hence the notion of rates of transitions which follows next Copyright © 2003 by K.S. Trivedi 9 Transition Rates • Define the rates (probabilities per unit time): net rate out of state j at time t: • the rate from state i to state j at time t: Copyright © 2003 by K.S. Trivedi 10 Kolmogorov Differential Equation • The transition probabilities and transition rates are, • Dividing both sides by h and taking the limit, Copyright © 2003 by K.S. Trivedi 11 Kolmogorov Differential Equation (contd.) • Kolmogorov’s backward equation, • Writing these eqns. in the matrix form, Copyright © 2003 by K.S. Trivedi 12 Homogeneous CTMC Specialize to HCTMC: all transition rates are time independent (Kolmogorov diff. equations) : •In the matrix form, (Matrix Q is called the infinitesimal generator matrix (or simply Generator Matrix)) Copyright © 2003 by K.S. Trivedi 13 Infinitesimal Generator Matrix • Infinitesimal Generator Matrix Q = [ qij ] For any off-diagonal element qij , i ≠ j is the transition rate from state i to state j • Characteristics of the generator matrix Q – Row sum equal to zero – Diagonal elements are all non-positive – Off-diagonal elements are all non-negative Copyright © 2003 by K.S. Trivedi 14 NHCTMC vs. HCTMC • In the non-homogeneous case, one or more transition rate is time dependent where time is measured from the beginning of system operation (global time) whereas in a semi Markov process rates are time dependent where time is measured from the entry into current state (local time) • Markov property is satisfied at any time by an NHCTMC or HCTMC while the Markov property is satisfied by a semi Markov process only at epochs of entry (or exit) from a state • Sojourn times in states of an HCTMC are exponentially distributed but this is not necessarily true for an NHCTMC or an SMP Copyright © 2003 by K.S. Trivedi 15 Definitions • A CTMC is said to be irreducible if every state can be reached from every other state, with a non-zero probability. • A state is said to be absorbing if no other state can be reached from it with non-zero probability. • Notion of transient, recurrent non-null, recurrent null are the same as in a DTMC. There is no notion of periodicity in a CTMC, however. • Unless otherwise specified, when we say CTMC, we mean HCTMC Copyright © 2003 by K.S. Trivedi 16 CTMC Steady-state Solution • Steady state solution of CTMC obtained by solving the following balance equations (πj is the limiting value of πj(t)) : • Irreducible CTMCs with all states recurrent non-null will have steady-state {πj} values that are unique and independent of the initial probability vector. All states of a finite irreducible CTMC will be recurrent non-null. • Measures of interest may be computed as weighted sum of steady state probabilities: Copyright © 2003 by K.S. Trivedi 17 CTMC Measures • For as a weighted sum of transient probabilities at time t: E[ Z (t )] = ∑ rj π j (t ) j • Expected accumulated quantites (over an interval of time (0,t]) • Lj(t) is the expected time spent in state j during (0,t] (LTODE) Copyright © 2003 by K.S. Trivedi 18 Different Analyses of HCTMC • Steady State Analysis: Solving a linear system of equations: • Transient Analysis: Solving a coupled system of linear first order ordinary diff. Equations, initial value problem with const. coefficients: • Cumulative Transient Analysis: ODEIVP with a forcing function: Copyright © 2003 by K.S. Trivedi 19 Markov Reward Models (MRMs) Copyright © 2003 by K.S. Trivedi 20 Markov Reward Models (MRMs) • Continuous Time Markov Chains are useful models for performance as well as availability prediction • Extension of CTMC to Markov reward models make them even more useful • Attach a reward rate (or a weight) ri to state i of CTMC. Let Z(t)=rX(t) be the instantaneous reward rate of CTMC at time t Copyright © 2003 by K.S. Trivedi 21 Markov Reward Models (MRMs) (Continued) • Define Y(t) the accumulated reward in the interval t (0,t] Y (t ) = ∫ Z (τ )dτ 0 • Computing the expected values of these measures is easy • For computing distribution of Y(t) see the Perfomability monograph edited by Haverkort, Marie, Rubino & Trivedi Copyright © 2003 by K.S. Trivedi 22 3-State Markov Reward Model with Sample Paths of X(t) and Z(t) Processes 2λ – r1 = 1.7 – r2 = 1.0 – r3 = 0.0 1 λ 2 µ2 Copyright © 2003 by K.S. Trivedi 3 µ3 23 3-State Markov Reward Model with Sample Paths of X(t) and Z(t) Processes Copyright © 2003 by K.S. Trivedi 24 3-State Markov Reward Model with Sample Path of Y(t) Processes Copyright © 2003 by K.S. Trivedi 25 Markov Reward Models (MRMs) (Continued) • Expected instantaneous reward rate at time t: E[ Z (t )] = ∑ riπ i (t ) i Expected steady-state reward rate: E[ Z ] = ∑ riπ i i Copyright © 2003 by K.S. Trivedi 26 MRM Measures • Expected accumulated reward in interval (0,t] E[Y (t )] = ∑ ri ∫ π i ( x)dx =∑ ri Li (t ) t i 0 i Copyright © 2003 by K.S. Trivedi 27 ...
View Full Document

Ask a homework question - tutors are online