Lecture 8
Mariana OlveraCravioto
UC Berkeley
[email protected]
February 9th, 2017
IND ENG 173, Introduction to Stochastic Processes
Lecture 8
1/13
Convergence theorems
I
Theorem:
Suppose
{
X
n
:
n
≥
0
}
is irreducible, aperiodic, and has a
stationary distribution
⇡
. Then,
lim
n
!1
p
n
(
x, y
) =
⇡
y
for all
x, y
2
S
.
I
Theorem:
Suppose
{
X
n
:
n
≥
0
}
is irreducible and all its states are
recurrent. Let
N
n
(
y
)
be the number of visits to
y
up to time
n
, then,
lim
n
!1
N
n
(
y
)
n
=
1
E
y
[
T
y
]
,
where
T
y
is the return time to state
y
.
I
Theorem:
Suppose
{
X
n
:
n
≥
0
}
is irreducible and has a stationary
distribution
⇡
. Then,
⇡
y
=
1
E
y
[
T
y
]
.
IND ENG 173, Introduction to Stochastic Processes
Lecture 8
2/13
Some remarks about the convergence theorems
I
Suppose that
{
X
n
:
n
≥
0
}
is an irreducible Markov chain having
stationary distribution
⇡
.
I
As the Matlab program showed,
⇡
j
has the interpretation of being the
longrun proportion of time
that the process will be in state
j
.
I
Even if the chain is not aperiodic, we still have that
lim
n
!1
N
n
(
y
)
n
=
⇡
y
.
I
If the chain is aperiodic, we also have
lim
n
!1
p
n
(
x, y
) =
⇡
y
for all
x, y
2
S
.
IND ENG 173, Introduction to Stochastic Processes
Lecture 8
3/13
Some remarks about
⇡
I
Theorem:
Suppose
{
X
n
:
n
≥
0
}
is irreducible and all its states are
recurrent. Then, there exists a solution
μ
(
x
)
≥
0
to
X
x
μ
(
x
)
p
(
x, y
) =
μ
(
y
)
,
y
2
S
.
If
S
is finite,
μ
(
x
)
can be normalized to obtain
⇡
.
I
The stationary probabilities are such that if we choose
X
0
according to
{
⇡
j
:
j
≥
0
}
then
P
(
X
1
=
j
) =
1
X
i
=0
P
(
X
1
=
j

X
0
=
i
)
⇡
i
=
1
X
i
=0
p
(
i, j
)
⇡
i
=
⇡
j
=
P
(
X
0
=
j
)
.
I
A Markov chain where
X
0
has distribution
{
⇡
j
:
j
≥
0
}
is said to be
stationary
or
in stationarity
.
I
For a stationary Markov chain
P
(
X
n
=
j
) =
⇡
j
for all
n
≥
0
.
IND ENG 173, Introduction to Stochastic Processes
Lecture 8
4/13
Long run cost
I
Suppose that whenever the chain
{
X
n
:
n
≥
0
}
visits state
x
it incurs a
cost
f
(
x
)
.
I
We want to compute the long run cost, i.e.,
lim
n
!1
1
n
n
X
i
=1
f
(
X
i
)
IND ENG 173, Introduction to Stochastic Processes
Lecture 8
5/13
Long run cost
I
Suppose that whenever the chain
{
X
n
:
n
≥
0
}
visits state
x
it incurs a
cost
f
(
x
)
.
I
We want to compute the long run cost, i.e.,
lim
n
!1
1
n
n
X
i
=1
f
(
X
i
)
I
Assuming a stationary distribution
⇡
exists, we expect
X
i
to be
distributed according to
⇡
for large
i
, and since the first few values can be