Discrete-time stochastic processes

# 5 413 414 415 proofs marked with an asterisk can

This preview shows page 1. Sign up to view the full content.

This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: es, it will be more convenient to use more illustrative labels for states. Deﬁnition 4.1. A Markov chain is an integer time process, {Xn , n ≥ 0} for which each rv Xn , n ≥ 1, is integer valued and depends on the past only through the most recent rv Xn−1 , 139 140 CHAPTER 4. FINITE-STATE MARKOV CHAINS i.e., for al l integer n ≥ 1 and al l integer i, j, k, . . . , m, Pr {Xn =j | Xn−1 =i, Xn−2 =k, . . . , X0 =m} = Pr {Xn =j | Xn−1 =i} .. (4.1) Pr {Xn =j | Xn−1 =i} depends only on i and j (not n) and is denoted by Pr {Xn =j | Xn−1 =i} = Pij . (4.2) The initial state X0 has an arbitrary probability distribution, which is required for a ful l probabilistic description of the process, but is not needed for most of the results. A Markov chain in which each Xn has a ﬁnite set of possible sample values is a ﬁnite-state Markov chain. The rv Xn is called the state of the chain at time n. The possible values for the state at time n, namely {1, . . . , M} or {0, 1, . . . } are also generally called states, usually without too much confusion. Thus Pij is the probability of going to state j given that the previous state is i; the new state, given the previous state, is independent of all earlier states. The use of the word state here conforms to the usual idea of the state of a system — the state at a given time summarizes everything about the past that is relevant to the future. Note that the transition probabilities, Pij , do not depend on n. Occasionally, a more general model is required where the transition probabilities do depend on n. In such situations, (4.1) and (4.2) are replaced by Pr {Xn =j | Xn−1 =i, Xn−2 =k, . . . , X0 =m} = Pr {Xn =j | Xn−1 =i} = Pij (n). (4.3) A process that obeys (4.3), with a dependence on n, is called a non-homogeneous Markov chain. Some people refer to a Markov chain (as deﬁned in (4.1) and (4.2)) as a homogeneous Markov chain. We will discuss only the homogeneous case (since not much of general interest can be said about the non-homogeneous case) and thus omit the word homogeneous as a qualiﬁer. An initial probability distribution for X0 , combined with the transition probabilities {Pij } (or {Pij (n)} for the non-homogeneous case), deﬁne the probabilities for all events. Markov chains can be used to model an enormous variety of physical phenomena and can be used to approximate most other kinds of stochastic processes. To see this, consider sampling a given process at a high rate in time, and then quantizing it, thus converting it into a discrete time process, {Zn ; −1 < n < 1}, where each Zn takes on a ﬁnite set of possible values. In this new process, each variable Zn will typically have a statistical dependence on past values that gradually dies out in time, so we can approximate the process by allowing Zn to depend on only a ﬁnite number of past variables, say Zn−1 , . . . , Zn−k . Finally, we can deﬁne a Markov process where the state at time n is Xn = (Zn , Zn−1 , . . . , Zn−k+1 ). The state Xn = (Zn , Zn−1 , . . . , Zn−k+1 ) then depends only on Xn−1 = (Zn−1 , . . . , Zn−k+1 , Zn−k ), since the new part of Xn , i.e., Zn , is independent of Zn−k−1 , Zn−k−2 , . . . , and the other variables comprising Xn are speciﬁed by Xn−1 . Thus {Xn } forms a Markov chain approximating the original process. This is not always an insightful or desirable model, but at least provides one possibility for modeling relatively general stochastic processes. Markov chains are often described by a directed graph (see Figure 4.1). In the graphical representation, there is one node for each state and a directed arc for each non-zero transition 4.2. CLASSIFICATION OF STATES 141 probability. If Pij = 0, then the arc from node i to node j is omitted; thus the diﬀerence between zero and non-zero transition probabilities stands out clearly in the graph. Several of the most important characteristics of a Markov chain depend only on which transition probabilities are zero, so the graphical representation is well suited for understanding these characteristics. A ﬁnite-state Markov chain is also often described by a matrix [P ] (see Figure 4.1). If the chain has M states, then [P ] is a M by M matrix with elements Pij . The matrix representation is ideally suited for studying algebraic and computational issues. P23 ✐ ✯② ✟2 ✟✟ ✟ P32 ✞ ✐ ✟ 12 ✟P P35 1❍ ✿❍ ✝✘ ❍❍ P11 P41❍❍ ❄ P45 ✐ 4 ③✛ ✐ P63 ✐ 3 6 P35 ❄ ✲5✘ ✐ ② P55 (a) ☎ ✆ P11 P12 · · · P16 P P22 · · · P26 [P ] = 21 ................... P61 P62 · · · P66 (b) Figure 4.1: Graphical and Matrix Representation of a 6 state Markov Chain; a directed arc from i to j is included in the graph if and only if (iﬀ ) Pij > 0. 4.2 Classiﬁcation of states This section, except where indicated otherwise, applies to Markov chains with both ﬁnite and countable state spaces. We start with several deﬁnitions. Deﬁnition 4.2....
View Full Document

## This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.

Ask a homework question - tutors are online