03. Constructing and Solving Markov Processes

03. Constructing and Solving Markov Processes - CS4...

Info iconThis preview shows pages 1–2. Sign up to view the full content.

View Full Document Right Arrow Icon
CS4 Modelling and Simulation LN-3 3 Constructing and Solving Markov Processes 3.1 Stochastic Processes Formally, a stochastic model is one represented as a stochastic process ; a stochastic process is a set of random variables { X ( t ) ,t T } . T is called the index set and in all the models we consider T will be taken to represent time. Since we consider continuous time models T = IR, the set of real numbers. The state space of the process is the set of all possible values that the random variables X ( t ) can assume. Each of these values is called a state of the process. These states will correspond to our intuitive notion of states within the system represented, although there will not necessarily be a one-to-one relation. Any set of instances of { X ( t ) T } can be regarded as a path of a particle moving randomly in the state space, S , its position at time t being X ( t ). These paths are called sample paths or realisations of the stochastic process. For example, if we consider the dial-in lines o±ered by CS again, we might associate one random variable with each line and associate the values 1 and 0 with the line being occupied or not. At any time the random variables representing the system will be the set of random variables representing each of the lines. Thus the state of the system is an n -tuple, where n is the number of lines and the entry in each position represents the state of the individual line. The state will change whenever a new connection is made or an existing connection is terminated. Alternatively, we might consider the stochastic process characterised by a single random variable, M , which records the number of lines in use. Now a single state at the stochastic process level will correspond to several states of the system depending on which lines are in use. This example shows how the degree of abstraction influences the modelling process. The stochastic processes we will be considering during the ²rst half of the course will all have the following properties: { X ( t ) } is a Markov process . This implies that { X ( t ) } has the Markov or memoryless property : given the value of X ( t )atsomet ime t T , the future path X ( s )for s>t does not depend on knowledge of the past history X ( u u<t , i.e. for t 1 < ··· <t n n +1 , Pr( X ( t n +1 )= x n +1 | X ( t n x n ,...,X ( t 1 x 1 )=Pr( X ( t n +1 x n +1 | X ( t n x n ) { X ( t ) } is irreducible . This implies that all states in S can be reached from all other states, by following the transitions of the process. { X ( t ) } is stationary : for any t 1 ,...t n T and t 1 + τ,. ..,t n + τ T ( n 1), then the process’s joint distributions are una±ected by the change in the time axis and so, F X ( t 1 + τ ) ...X ( t n + τ ) = F X ( t 1 ) ...X ( t n ) { X ( t ) } is time homogeneous : the behaviour of the system does not depend on when it is observed. In particular, the transition rates between states are independent of the time at which the transitions occur. Thus, for all t and s , it follows that Pr( X ( t + τ x k | X ( t x j X ( s + τ x k | X ( s x j ).
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 2
This is the end of the preview. Sign up to access the rest of the document.

Page1 / 9

03. Constructing and Solving Markov Processes - CS4...

This preview shows document pages 1 - 2. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online