Copyright c
±
2010 by Karl Sigman
1
Markov Chain Monte Carlo Methods (MCMC)
There are many applications in which it is desirable to simulate from a probability distribution
π
(say) in which all speciﬁcs of the distribution (cdf, density, etc.) are not known a priori, and
hence standard simulation methods (such as the inverse transform method, acceptance rejection,
etc.) cannot be readily applied. For example, one might know how to simulate a complicated
stochastic process
{
X
n
}
for which a priori it is known to have a limiting (stationary) distribution
π
, but for which explicitly/analytically ﬁnding
π
is not possible. Coming up with an algorithm
that yields a copy of a rv distributed
exactly
as
π
in this context is called
perfect
or
exact
simulation. We of course could
estimate
π
by using the fact that we can simulate the stochastic
process
{
X
n
}
, and use the approximation (for large
N
)
π
i
≈
1
N
N
X
n
=1
I
{
X
n
=
i
}
.
(1)
An important related case is in the framework when a desired distribution
π
is not naturally
part of any stochastic process and is only known up to a multiplicative constant (normalizing
factor
C >
0). For example, consider the discrete case, and suppose we want the following
distribution
π
i
=
a
i
/C, i
∈ S
,
(2)
where each
a
i
>
0 can be computed but the normalizing constant
C
=
X
i
∈S
a
i
,
is uncomputable hence unknown.
π
might, for example, be a uniform distribution over a very
complicated (but ﬁnite) set involving a graph with various edges, weights and arcs. (Combina-
torial objects are notoriously like this.) In such a uniform case,
a
i
= 1 for all
i
, but computing
C
can be hopeless.
The
Markov chain Monte Carlo method (MCMC)
allows us to approximate
π
as deﬁned in
(2) by constructing (and simulating) an irreducible positive recurrent Markov chain
{
X
n
}
with
state space
S
and transition matrix
P
= (
P
ij
)
such that its stationary distribution is the desired
π
,
that is, such that
π
=
πP
. Thus we can simply simulate the Markov chain out
N
steps,
N
large, and use the approximation
π
i
≈
1
N
N
X
n
=1
I
{
X
n
=
i
}
,
as pointed out in (1). (This is not
perfect
simulation, but a perfect simulation might be possible
later applied to the constructed Markov chain.) Perhaps our goal is to compute the mean of
π
in which case we could use the approximation
X
i
∈S
iπ
i
≈
1
N
N
X
n
=1
X
n
.
1