Discrete-time stochastic processes

If m then x must be a scalar times e this is

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: -state Markov chain that contains a single recurrent class plus, perhaps, some transient states. An ergodic unichain is a unichain for which the recurrent class is ergodic. A Unichain, as we shall see, is the natural generalization of a recurrent chain to allow for some initial transient behavior without disturbing the long term aymptotic behavior of the underlying recurrent chain. The answer to the second question above is that the solution to (4.18) is unique iff [P] is the transition matrix of a unichain. If there are r recurrent classes, then π = π [P ] has r linearly independent solutions. For the third question, each row of [P ]n converges to the unique solution of (4.18) if [P] is the transition matrix of an ergodic unichain. If there are multiple recurrent classes, but all of them are aperiodic, then [P ]n still converges, but to a matrix with non-identical rows. If the Markov chain has one or more periodic recurrent classes, then [P ]n does not converge. We first look at these answers from the standpoint of matrix theory and then proceed in Chapter 5 to look at the more general problem of Markov chains with a countably infinite number of states. There we use renewal theory to answer these same questions (and to discover the differences that occur for infinite-state Markov chains). The matrix theory approach is useful computationally and also has the advantage of telling us something about rates of convergence. The approach using renewal theory is very simple (given an understanding of renewal processes), but is more abstract. 4.3.1 The eigenvalues and eigenvectors of P A convenient way of dealing with the nth power of a matrix is to find the eigenvalues and eigenvectors of the matrix. Definition 4.11. The row vector π is a left eigenvector of [P ] of eigenvalue ∏ if π 6= 0 and π [P ] = ∏π . The column vector ∫ is a right eigenvector of eigenvalue ∏ if ∫ 6= 0 and [P ]∫ = ∏∫ . We first treat the special case of a Markov chain with two states. Here the eigenvalues and eigenvectors can be found by elementary (but slightly tedious) algebra. The eigenvector equations can be written out as π1 P11 + π2 P21 = ∏π1 π1 P12 + π2 P22 = ∏π2 P11 ∫1 + P12 ∫2 = ∏∫1 . P21 ∫1 + P22 ∫2 = ∏∫2 (4.19) These equations have a non-zero solution iff the matrix [P − ∏I ], where [I ] is the identity matrix, is singular (i.e., there must be a non-zero ∫ for which [P − ∏I ]∫ = 0 ). Thus ∏ must be such that the determinant of [P − ∏I ], namely (P11 − ∏)(P22 − ∏) − P12 P21 , is equal to 0. Solving this quadratic equation in ∏, we find that ∏ has two solutions, ∏1 = 1 and ∏2 = 1 − P12 − P21 . Assume initially that P12 and P21 are not both 0. Then the solution for the left and right eigenvectors, π (1) and ∫ (1) , of ∏1 and π (2) and ∫ (2) of ∏2 , are given by (1) π1 = (2) π1 = P21 P12 +P21 1 (1) π2 = (2) π2 = P12 P12 +P21 −1 (1) ∫1 = (2) ∫1 = 1 P12 P12 +P21 (1) ∫2 = (2) ∫2 = 1 −P21 P12 +P21 . 150 CHAPTER 4. FINITE-STATE MARKOV CHAINS ∑ ∏ ∏1 0 These solutions contain an arbitrary normalization factor. Now let [Λ] = and 0 ∏2 let [U ] be a matrix with columns ∫ (1) and ∫ (2) . Then the two right eigenvector equations in (4.19) can be combined compactly as [P ][U ] = [U ][Λ]. It turns out (given the way we have normalized the eigenvectors) that the inverse of [U ] is just the matrix whose rows are the left eigenvectors of [P ] (this can be verified by direct calculation, and we show later that any right eigenvector of one eigenvalue must be orthogonal to any left eigenvector of another eigenvalue). We then see that [P ] = [U ][Λ][U ]−1 and consequently [P ]n = [U ][Λ]n [U ]−1 . Multiplying this out, we get [P ]n = ∑ π1 + π2 ∏n 2 π1 − π1 ∏n 2 π2 − π2 ∏n 2 π2 + π1 ∏n 2 ∏ where π1 = P21 , π2 = 1 − π1 . P12 + P21 Recalling that ∏2 = 1 − P12 − P21 , we see that |∏2 | ≤ 1. If P12 = P21 = 0, then ∏2 = 1 so that [P ] and [P ]n are simply identity matrices. If P12 = P21 = 1, then ∏2 = −1 so that [P ]n alternates between the identity matrix for n even and [P ] for n odd. In all other cases, |∏2 | < 1 and [P ]n approaches the matrix whose rows are both equal to π . Parts of this special case generalize to an arbitrary finite number of states. In particular, ∏ = 1 is always an eigenvalue and the vector e whose components are all equal to 1 is always a right eigenvector of ∏ = 1 (this follows immediately from the fact that each row of a stochastic matrix sums to 1). Unfortunately, not all stochastic matrices can be represented in the form [P ] = [U ][Λ][U −1 (since M independent right eigenvectors need not exist—see Exercise 4.9) In general, the diagonal matrix of eigenvalues in [P ] = [U ][Λ][U −1 ] must be replaced by something called a Jordan form, which does not easily lead us to the desired results. In what follows, we develop the powerful Perron and Fro...
View Full Document

This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.

Ask a homework question - tutors are online