This preview shows page 1. Sign up to view the full content.
Unformatted text preview: es, it will be more convenient to use more illustrative labels
for states.
Deﬁnition 4.1. A Markov chain is an integer time process, {Xn , n ≥ 0} for which each rv
Xn , n ≥ 1, is integer valued and depends on the past only through the most recent rv Xn−1 ,
139 140 CHAPTER 4. FINITESTATE MARKOV CHAINS i.e., for al l integer n ≥ 1 and al l integer i, j, k, . . . , m,
Pr {Xn =j  Xn−1 =i, Xn−2 =k, . . . , X0 =m} = Pr {Xn =j  Xn−1 =i} .. (4.1) Pr {Xn =j  Xn−1 =i} depends only on i and j (not n) and is denoted by
Pr {Xn =j  Xn−1 =i} = Pij . (4.2) The initial state X0 has an arbitrary probability distribution, which is required for a ful l
probabilistic description of the process, but is not needed for most of the results. A Markov
chain in which each Xn has a ﬁnite set of possible sample values is a ﬁnitestate Markov
chain.
The rv Xn is called the state of the chain at time n. The possible values for the state at
time n, namely {1, . . . , M} or {0, 1, . . . } are also generally called states, usually without too
much confusion. Thus Pij is the probability of going to state j given that the previous state
is i; the new state, given the previous state, is independent of all earlier states. The use of
the word state here conforms to the usual idea of the state of a system — the state at a
given time summarizes everything about the past that is relevant to the future. Note that
the transition probabilities, Pij , do not depend on n. Occasionally, a more general model
is required where the transition probabilities do depend on n. In such situations, (4.1) and
(4.2) are replaced by
Pr {Xn =j  Xn−1 =i, Xn−2 =k, . . . , X0 =m} = Pr {Xn =j  Xn−1 =i} = Pij (n). (4.3) A process that obeys (4.3), with a dependence on n, is called a nonhomogeneous Markov
chain. Some people refer to a Markov chain (as deﬁned in (4.1) and (4.2)) as a homogeneous
Markov chain. We will discuss only the homogeneous case (since not much of general
interest can be said about the nonhomogeneous case) and thus omit the word homogeneous
as a qualiﬁer. An initial probability distribution for X0 , combined with the transition
probabilities {Pij } (or {Pij (n)} for the nonhomogeneous case), deﬁne the probabilities for
all events.
Markov chains can be used to model an enormous variety of physical phenomena and can
be used to approximate most other kinds of stochastic processes. To see this, consider sampling a given process at a high rate in time, and then quantizing it, thus converting it into a
discrete time process, {Zn ; −1 < n < 1}, where each Zn takes on a ﬁnite set of possible
values. In this new process, each variable Zn will typically have a statistical dependence on
past values that gradually dies out in time, so we can approximate the process by allowing Zn
to depend on only a ﬁnite number of past variables, say Zn−1 , . . . , Zn−k . Finally, we can deﬁne a Markov process where the state at time n is Xn = (Zn , Zn−1 , . . . , Zn−k+1 ). The state
Xn = (Zn , Zn−1 , . . . , Zn−k+1 ) then depends only on Xn−1 = (Zn−1 , . . . , Zn−k+1 , Zn−k ),
since the new part of Xn , i.e., Zn , is independent of Zn−k−1 , Zn−k−2 , . . . , and the other
variables comprising Xn are speciﬁed by Xn−1 . Thus {Xn } forms a Markov chain approximating the original process. This is not always an insightful or desirable model, but at least
provides one possibility for modeling relatively general stochastic processes.
Markov chains are often described by a directed graph (see Figure 4.1). In the graphical
representation, there is one node for each state and a directed arc for each nonzero transition 4.2. CLASSIFICATION OF STATES 141 probability. If Pij = 0, then the arc from node i to node j is omitted; thus the diﬀerence
between zero and nonzero transition probabilities stands out clearly in the graph. Several
of the most important characteristics of a Markov chain depend only on which transition
probabilities are zero, so the graphical representation is well suited for understanding these
characteristics. A ﬁnitestate Markov chain is also often described by a matrix [P ] (see
Figure 4.1). If the chain has M states, then [P ] is a M by M matrix with elements Pij . The
matrix representation is ideally suited for studying algebraic and computational issues.
P23 ✐
✯②
✟2 ✟✟
✟
P32
✞ ✐ ✟ 12
✟P
P35
1❍
✿❍
✝✘
❍❍
P11
P41❍❍ ❄
P45
✐ 4 ③✛ ✐ P63 ✐
3
6 P35
❄
✲5✘
✐
② P55 (a) ☎
✆ P11 P12 · · · P16
P
P22 · · · P26 [P ] = 21 ................... P61 P62 · · · P66
(b) Figure 4.1: Graphical and Matrix Representation of a 6 state Markov Chain; a directed
arc from i to j is included in the graph if and only if (iﬀ ) Pij > 0. 4.2 Classiﬁcation of states This section, except where indicated otherwise, applies to Markov chains with both ﬁnite
and countable state spaces. We start with several deﬁnitions.
Deﬁnition 4.2....
View
Full
Document
This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.
 Spring '09
 R.Srikant

Click to edit the document details