Copyright c
±
2006 by Karl Sigman
1
Discretetime Markov chains
1.1
Stochastic processes in discrete time
A
stochastic process
in discrete time
n
∈
IN =
{
0
,
1
,
2
,...
}
is a sequence of random variables
(rvs)
X
0
,X
1
,X
2
,...
denoted by
X
=
{
X
n
:
n
≥
0
}
(or just
X
=
{
X
n
}
). We refer to the value
X
n
as the
state
of the process at time
n
, with
X
0
denoting the initial state. If the random
variables take values in a discrete space such as the integers ZZ =
{
...,

2
,

1
,
0
,
1
,
2
,...
}
(or
some subset of them), then the stochastic process is said to be discretevalued; we then denote
the states by
i,j
and so on. In general, however, the collection of possible values that the
X
n
can take on is called the
state space
, is denoted by
S
and could be, for example,
d

dimensional
Euclidean space IR
d
, d
≥
1, or a subset of it.
Stochastic processes are meant to model the evolution over time of real phenomina for
which randomness is inherent. For example,
X
n
could denote the price of a stock
n
days from
now, the population size of a given species after
n
years, the amount of bandwidth in use in
a telecommunications network after
n
hours of operation, or the amount of money that an
insurance risk company has right after it pays out its
n
th
claim. The insurance risk example
illustrates how “time”
n
need not really be time, but instead can be a sequential indexing of
some kind of events. Other such examples:
X
n
denotes the amount of water in a reservoir after
the
n
th
rain storm of the year,
X
n
denotes the amount of time that the
n
th
phone caller to an
airline has to wait until speaking to a salesperson, or
X
n
denotes the outcome (heads or tails)
of the
n
th
ﬂip of a coin.
The main challenge in the stochastic modeling of something is in choosing a model that
has – on the one hand – enough complexity to capture the complexity of the phenomina in
question, but has – on the other hand – enough structure and simplicity to allow one to com
pute things of interest. In the context of our examples given above, we may be interested in
computing
P
(
X
30
>
50) for a stock that we bought for
X
0
= $35 per share, or computing
the probability that the insurance risk company eventually gets ruined (runs out of money),
P
(
X
n
<
0
,
for some
n >
0), or computing the longrun average waiting time of a caller to the
airline
lim
N
→∞
1
N
N
X
n
=1
X
n
.
As a very simple example, for the sequential tossing of a “fair” coin, we would naturally
assume that each
X
n
is a 0
.
5 Bernoulli rv,
P
(
X
n
= 0) =
P
(
X
n
= 1) = 0
.
5, and that the
sequence of rvs are independent. This then yields an example of an independent and identically
distributed (iid) sequence of rvs. Such sequences are easy to deal with for they are deﬁned by
a single distribution (in this case Bernoulli), and lend themselves directly to powerful theorems
in probability such as the strong law of large numbers and the central limit theorem.
For the other examples given above, however, an iid sequence would not capture enough