Markov Chains
OEM 2009
ESI 6321 –Applied Probability Methods in
Engineering
2
Stochastic processes
The last module of this course deals with
describing and analyzing systems under
uncertainty
In particular, we will study how a
random
variable
changes over time
Examples:
Price of a stock or portfolio of stocks
Inventory level of a good in a warehouse
Career path (workforce planning)
Number of customers present in a store or
bank
etc.

This
** preview**
has intentionally

**sections.**

*blurred***to view the full version.**

*Sign up*
3
Stochastic processes
We will focus on systems that we observe at
discrete
points in time
E.g., we evaluate the value of a stock portfolio at
the end of each trading day
Denote the time points by
t
=0,1,2,…
Denote the value (at time
t
) of the characteristic
of the system that we are interested in by
X
t
X
t
is a
random variable
The sequence
X
0
,
X
1
,
X
2
,… is called a
stochastic
process
4
Example 1
We are interested in tracking the stock
market on a daily basis
Let
X
t
denote the value of the Dow Jones
index at the end of trading day
t
We are currently at the end of trading day 0,
and observe
X
0
=
x
0
Can we model/study the relationship
between the random variables
X
t
(
t
=0,1,2,…)?

5
Example 2
You are visiting Las Vegas, and have a gambling
budget of $
x
0
You participate in a game in which you
repeatedly
bet $1
if you win (which happens with probability
p
) you
receive $2
if you loose (which happens with probability 1-
p
)
you receive nothing
you stop playing as soon as you are broke or have
doubled your initial budget
Denote your total
wealth
at time
t
by
X
t
6
States and state space
The set of values that the random variable
X
t
can
take is called the
state space
of the stochastic
process
Example 1: [0,
∞
)
Example 2: {0,1,2,…2
x
0
}
We will restrict ourselves to situations in which the
state space consists of a finite number of elements
only
often:
S
= {1,2,…,
s
}
If
X
t
=
i
we say that the stochastic process is in
state
i
at time
t
.

This
** preview**
has intentionally

**sections.**

*blurred***to view the full version.**

*Sign up*