CS4 Modelling and Simulation
LN3
3
Constructing and Solving Markov Processes
3.1
Stochastic Processes
Formally, a stochastic model is one represented as a
stochastic process
; a stochastic process
is a set of random variables
{
X
(
t
)
, t
∈
T
}
.
T
is called the
index set
and in all the models
we consider
T
will be taken to represent time. Since we consider continuous time models
T
= IR, the set of real numbers. The
state space
of the process is the set of all possible
values that the random variables
X
(
t
) can assume. Each of these values is called a
state
of the process. These states will correspond to our intuitive notion of states within the
system represented, although there will not necessarily be a onetoone relation. Any set
of instances of
{
X
(
t
)
, t
∈
T
}
can be regarded as a path of a particle moving randomly in
the state space,
S
, its position at time
t
being
X
(
t
). These paths are called
sample paths
or
realisations
of the stochastic process.
For example, if we consider the dialin lines offered by CS again, we might associate
one random variable with each line and associate the values 1 and 0 with the line being
occupied or not. At any time the random variables representing the system will be the
set of random variables representing each of the lines.
Thus the state of the system is
an
n
tuple, where
n
is the number of lines and the entry in each position represents the
state of the individual line.
The state will change whenever a new connection is made
or an existing connection is terminated. Alternatively, we might consider the stochastic
process characterised by a single random variable,
M
, which records the number of lines
in use. Now a single state at the stochastic process level will correspond to several states
of the system depending on which lines are in use. This example shows how the degree
of abstraction inﬂuences the modelling process.
The stochastic processes we will be considering during the first half of the course will
all have the following properties:
{
X
(
t
)
}
is a Markov process
. This implies that
{
X
(
t
)
}
has the
Markov
or
memoryless
property
: given the value of
X
(
t
) at some time
t
∈
T
, the future path
X
(
s
) for
s > t
does
not depend on knowledge of the past history
X
(
u
) for
u < t
, i.e. for
t
1
<
· · ·
< t
n
< t
n
+1
,
Pr(
X
(
t
n
+1
) =
x
n
+1

X
(
t
n
) =
x
n
, . . . , X
(
t
1
) =
x
1
) = Pr(
X
(
t
n
+1
) =
x
n
+1

X
(
t
n
) =
x
n
)
{
X
(
t
)
}
is irreducible
. This implies that all states in
S
can be reached from all other
states, by following the transitions of the process.
{
X
(
t
)
}
is stationary
: for any
t
1
, . . . t
n
∈
T
and
t
1
+
τ, . . . , t
n
+
τ
∈
T
(
n
≥
1), then the
process’s joint distributions are unaffected by the change in the time axis and so,
F
X
(
t
1
+
τ
)
...X
(
t
n
+
τ
)
=
F
X
(
t
1
)
...X
(
t
n
)
{
X
(
t
)
}
is time homogeneous
: the behaviour of the system does not depend on when
it is observed. In particular, the transition rates between states are independent of the
time at which the transitions occur. Thus, for all
t
and
s
, it follows that
Pr(
X
(
t
+
τ
) =
x
k

X
(
t
) =
x
j
) = Pr(
X
(
s
+
τ
) =
x
k

X
(
s
) =
x
j
).
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
This is the end of the preview.
Sign up
to
access the rest of the document.
 Spring '10
 MohammadAbdolahiAzgomiPh.D
 Probability theory, Stochastic process, Markov chain, Markovian

Click to edit the document details