Process
! Markov !
A stochastic process is a function whose values are random variables The classification of a random process depends on different quantities
state space index (time) parameter statistical dependencies among the random variables X(t) for different values of the index parameter t.
2007 A.W. Krings
Page: 1
CS449/549 Fault-Tolerant Systems
Sequence 8
Markov Process
!
State Space
the set of possible values (states) that X(t) might take on. if there are finite states => discrete-state process or chain if there is a continuous interval => continuous process if the times at which changes may take place are finite or countable, then we say we have a discrete-(time) parameter process. if the changes may occur anywhere within a finite or infinite interval on the time axis, then we say we have a continuous-parameter process.
!
Index (Time) Parameter
2007 A.W. Krings
Page: 2
CS449/549 Fault-Tolerant Systems
Sequence 8
Markov Process
!
! !
In 1907 A.A. Markov published a paper in which he defined and investigated the properties of what are now known as Markov processes. A Markov process with a discrete state space is referred to as a Markov Chain A set of random variables forms a Markov chain if the probability that the next state is S(n+1) depends only on the current state S(n), and not on any previous states
2007 A.W. Krings
Page: 3
CS449/549 Fault-Tolerant Systems
Sequence 8
Markov Process
!
States must be
mutually exclusive collectively exhaustive
!
Let Pi(t) = Probability of being in state Si at time t.
!
Markov Properties
future state prob. depends only on current state
independent of time in state path to state
2007 A.W. Krings
Page: 4
CS449/549 Fault-Tolerant Systems
Sequence 8
Markov Process
! !
Assume exponential failure law with failure rate ". Probability system that failed at t + !t, given that is was working at time t is given by with we get
2007 A.W. Krings
Page: 5
CS449/549 Fault-Tolerant Systems
Sequence 8
Markov Process
!
For small !t
1
0
2007 A.W. Krings
Page: 6
CS449/549 Fault-Tolerant Systems
Sequence 8
Markov Process
!
Let P(transition out of state i in !t) =
!
Mean time to transition (exponential holding times)
!
If "s are not functions of time, i.e. if
homogeneous Markov Chain
2007 A.W. Krings
Page: 7
CS449/549 Fault-Tolerant Systems
Sequence 8
Markov Process
!
Accessibility
state Si is accessible from state Sj if there is a sequence of transitions from Sj to Si. state Si is called recurrent, if Si can be returned to from any state which is accessible from Si in one step, i.e. from all immediate neighbor states. if there exists at least one neighbor with no return path.
!
Recurrent State
!
Non Recurrent
2007 A.W. Krings
Page: 8
CS449/549 Fault-Tolerant Systems
Sequence 8
Markov Process
!
sample chain Which states are recurrent or non-recurrent?
2007 A.W. Krings
Page: 9
CS449/5...