{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

03. Constructing and Solving Markov Processes

# 03. Constructing and Solving Markov Processes - CS4...

This preview shows pages 1–2. Sign up to view the full content.

CS4 Modelling and Simulation LN-3 3 Constructing and Solving Markov Processes 3.1 Stochastic Processes Formally, a stochastic model is one represented as a stochastic process ; a stochastic process is a set of random variables { X ( t ) , t T } . T is called the index set and in all the models we consider T will be taken to represent time. Since we consider continuous time models T = IR, the set of real numbers. The state space of the process is the set of all possible values that the random variables X ( t ) can assume. Each of these values is called a state of the process. These states will correspond to our intuitive notion of states within the system represented, although there will not necessarily be a one-to-one relation. Any set of instances of { X ( t ) , t T } can be regarded as a path of a particle moving randomly in the state space, S , its position at time t being X ( t ). These paths are called sample paths or realisations of the stochastic process. For example, if we consider the dial-in lines offered by CS again, we might associate one random variable with each line and associate the values 1 and 0 with the line being occupied or not. At any time the random variables representing the system will be the set of random variables representing each of the lines. Thus the state of the system is an n -tuple, where n is the number of lines and the entry in each position represents the state of the individual line. The state will change whenever a new connection is made or an existing connection is terminated. Alternatively, we might consider the stochastic process characterised by a single random variable, M , which records the number of lines in use. Now a single state at the stochastic process level will correspond to several states of the system depending on which lines are in use. This example shows how the degree of abstraction inﬂuences the modelling process. The stochastic processes we will be considering during the first half of the course will all have the following properties: { X ( t ) } is a Markov process . This implies that { X ( t ) } has the Markov or memoryless property : given the value of X ( t ) at some time t T , the future path X ( s ) for s > t does not depend on knowledge of the past history X ( u ) for u < t , i.e. for t 1 < · · · < t n < t n +1 , Pr( X ( t n +1 ) = x n +1 | X ( t n ) = x n , . . . , X ( t 1 ) = x 1 ) = Pr( X ( t n +1 ) = x n +1 | X ( t n ) = x n ) { X ( t ) } is irreducible . This implies that all states in S can be reached from all other states, by following the transitions of the process. { X ( t ) } is stationary : for any t 1 , . . . t n T and t 1 + τ, . . . , t n + τ T ( n 1), then the process’s joint distributions are unaffected by the change in the time axis and so, F X ( t 1 + τ ) ...X ( t n + τ ) = F X ( t 1 ) ...X ( t n ) { X ( t ) } is time homogeneous : the behaviour of the system does not depend on when it is observed. In particular, the transition rates between states are independent of the time at which the transitions occur. Thus, for all t and s , it follows that Pr( X ( t + τ ) = x k | X ( t ) = x j ) = Pr( X ( s + τ ) = x k | X ( s ) = x j ).

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}