Discrete-time stochastic processes

Thus a node j receives poisson exogenous arrivals

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: be expressed most compactly in matrix notation. Letting [P (t)] be a matrix for each t whose i, j element is Pij (t), and letting [Q] be a matrix whose i, j term is qij for i 6= j and −∫i for i = j , we have d[P (t)]/dt = [Q][P (t)] ; t ≥ 0. (6.28) 248 CHAPTER 6. MARKOV PROCESSES WITH COUNTABLE STATE SPACES The initial conditions, at t = 0, are [P (t)] = I , where I is the identity matrix. The solution to (6.28), sub ject to these initial conditions, is 1 X tm [Q]m [P (t)] = . m! m=0 (6.29) Equation (6.29) can be verified by substitution into (6.28). We shall return to (6.28) and (6.29) after deriving the Kolmogorov forward equations. For t − s small and positive, (6.24) becomes X Pij (t) = [Pik (s)(t − s)qkj ] + Pij (s)[1 − (t − s)∫j ] + o(t − s) (6.30) k6=j Pij (t) − Pij (s) t−s = dPij (t) dt = X k6=j X k6=j [Pik (s)qkj − Pij (s)∫j (6.31) [Pik (t)qkj ] − Pij (t)∫j , (6.32) (where we have again assumed that the summation and limit can be interchanged). These are the Kolmogorov forward equations; their interpretation is similar to, but slightly simpler than, that of the backward equations. The incremental change in Pij (t) is equal to the difference of two terms. The first is the sum over k of the probability of being in state k at time t and then moving to j in the final incremental interval; the second is the probability of being in state j at time t and then moving away in the final incremental interval. For the finite state case, (6.32), in matrix notation is d[P (t)]/dt = [P (t)][Q]. (6.33) This equation also has the matrix solution in (6.29). Although (6.29) is a solution to these equations, it doesn’t provide much insight into the form of the solution. In order to provide more of this insight, we go back to the sampled time approximation. With an increment of size δ between samples, the probability of a transition from i to j , i 6= j , is qij δ , and the probability of remaining in state i is 1 − ∫i δ . Thus, in terms of the matrix [Q] of transition rates, the transition probability matrix in the sampled time model is I + δ [Q], where I is the identity matrix. We denote this matrix by [Wδ ] = I + δ [Q]. Note that ∏ is an eigenvalue of [Q] iff 1 + ∏δ is an eigenvalue of [Wδ ], and the eigenvectors of these matrices are the same. That is, if ∫ (π ) is a right (left) eigenvector of [Q] with eigenvalue ∏, then ∫ (π ) is a right (left) eigenvector of [Wδ ] with eigenvalue 1 + ∏δ , and conversely. We have already studied the eigenvalues and eigenvectors of transition matrices such as [Wδ ] in Section 4.4, and the following theorem translates some of these results into results about [Q]. Theorem 6.3. Consider an irreducible finite-state Markov process with n states. Then the matrix [Q] for that process has an eigenvalue ∏ equal to 0. That eigenvalue has a right eigenvector e = (1, 1, . . . , 1)T which is unique within a scale factor. It has a left eigenvector p = (p1 , . . . , pn ) that is positive, sums to 1, satisfies (6.20), and is unique within a scale factor. Al l the other eigenvalues of [Q] have strictly negative real parts. 6.3. THE KOLMOGOROV DIFFERENTIAL EQUATIONS 249 Proof: Since all n states communicate, the sampled time chain is recurrent. From corollary 4.3 to the Frobenius theorem (Theorem 4.6), [Wδ ] has an eigenvalue ∏ = 1; the corresponding right eigenvector is e ; the left eigenvector is the steady state probability vector, which is positive. These eigenvectors are unique within a scale factor. From the equivalence of (6.20) and (6.21), p , as given by (6.20), is the steady state probability vector. Each eigenvalue ∏δ of [Wδ ] corresponds to an eigenvalue ∏ of [Q] with the correspondence ∏δ = 1 + ∏δ , i.e., ∏ = (∏δ − 1)/δ . Thus the eigenvalue 1 of [Wδ ] corresponds to the eigenvalue 0 of [Q]. Since |∏δ | ≤ 1 and ∏δ 6= 1 for all other eigenvalues, the other eigenvalues of [Q] all have strictly negative real parts, completing the proof. We continue to look at an irreducible n-state Markov process, and assume for simplicity T T that [Q] has n distinct eigenvalues, ∏1 , . . . , ∏n . Let ∫ 1 , ∫ 2 , . . . , ∫ n and π 1 , . . . , π n be the corresponding right and left eigenvectors respectively. Assume that these eigenvectors are T T scaled so that π i ∫ i = 1 for each i, and recall that π i ∫ j = 0 for i 6= j . Thus, if [V ] is defined as the matrix with columns ∫ 1 , ∫ 2 , . . . , ∫ n , then its inverse, [V ]−1 , is the matrix whose rows T T are π 1 , . . . , π n . Finally, let [Λ] be the diagonal matrix with elements ∏1 , . . . , ∏n . We then have the relationships [Λ] = [V ]−1 [Q][V ] ; [Q] = [V ][Λ][V ]−1 . (6.34) We then also have [Q]m = [V ]−1 [Λ]m [V ], and thus (6.29) can be written as [P (t)] = 1 X m=0 [V ] tm [Λ]m [V ]−1 = [V ][etΛ ][V ]−1 , m! (6.35) where [etΛ ] is the diagonal matrix with elements et∏1 , et∏2 , . . . , , et∏n . Finally, if we br...
View Full Document

Ask a homework question - tutors are online