This preview shows page 1. Sign up to view the full content.
Unformatted text preview: be expressed most compactly in matrix notation.
Letting [P (t)] be a matrix for each t whose i, j element is Pij (t), and letting [Q] be a matrix
whose i, j term is qij for i 6= j and −∫i for i = j , we have
d[P (t)]/dt = [Q][P (t)] ; t ≥ 0. (6.28) 248 CHAPTER 6. MARKOV PROCESSES WITH COUNTABLE STATE SPACES The initial conditions, at t = 0, are [P (t)] = I , where I is the identity matrix. The solution
to (6.28), sub ject to these initial conditions, is
1
X tm [Q]m
[P (t)] =
.
m!
m=0 (6.29) Equation (6.29) can be veriﬁed by substitution into (6.28). We shall return to (6.28) and
(6.29) after deriving the Kolmogorov forward equations.
For t − s small and positive, (6.24) becomes
X
Pij (t) =
[Pik (s)(t − s)qkj ] + Pij (s)[1 − (t − s)∫j ] + o(t − s) (6.30) k6=j Pij (t) − Pij (s)
t−s = dPij (t)
dt = X
k6=j X
k6=j [Pik (s)qkj − Pij (s)∫j (6.31) [Pik (t)qkj ] − Pij (t)∫j , (6.32) (where we have again assumed that the summation and limit can be interchanged). These
are the Kolmogorov forward equations; their interpretation is similar to, but slightly simpler
than, that of the backward equations. The incremental change in Pij (t) is equal to the
diﬀerence of two terms. The ﬁrst is the sum over k of the probability of being in state k at
time t and then moving to j in the ﬁnal incremental interval; the second is the probability
of being in state j at time t and then moving away in the ﬁnal incremental interval. For
the ﬁnite state case, (6.32), in matrix notation is
d[P (t)]/dt = [P (t)][Q]. (6.33) This equation also has the matrix solution in (6.29). Although (6.29) is a solution to these
equations, it doesn’t provide much insight into the form of the solution. In order to provide
more of this insight, we go back to the sampled time approximation. With an increment of
size δ between samples, the probability of a transition from i to j , i 6= j , is qij δ , and the
probability of remaining in state i is 1 − ∫i δ . Thus, in terms of the matrix [Q] of transition
rates, the transition probability matrix in the sampled time model is I + δ [Q], where I is the
identity matrix. We denote this matrix by [Wδ ] = I + δ [Q]. Note that ∏ is an eigenvalue of
[Q] iﬀ 1 + ∏δ is an eigenvalue of [Wδ ], and the eigenvectors of these matrices are the same.
That is, if ∫ (π ) is a right (left) eigenvector of [Q] with eigenvalue ∏, then ∫ (π ) is a right
(left) eigenvector of [Wδ ] with eigenvalue 1 + ∏δ , and conversely. We have already studied
the eigenvalues and eigenvectors of transition matrices such as [Wδ ] in Section 4.4, and the
following theorem translates some of these results into results about [Q].
Theorem 6.3. Consider an irreducible ﬁnitestate Markov process with n states. Then the
matrix [Q] for that process has an eigenvalue ∏ equal to 0. That eigenvalue has a right
eigenvector e = (1, 1, . . . , 1)T which is unique within a scale factor. It has a left eigenvector
p = (p1 , . . . , pn ) that is positive, sums to 1, satisﬁes (6.20), and is unique within a scale
factor. Al l the other eigenvalues of [Q] have strictly negative real parts. 6.3. THE KOLMOGOROV DIFFERENTIAL EQUATIONS 249 Proof: Since all n states communicate, the sampled time chain is recurrent. From corollary
4.3 to the Frobenius theorem (Theorem 4.6), [Wδ ] has an eigenvalue ∏ = 1; the corresponding right eigenvector is e ; the left eigenvector is the steady state probability vector, which is
positive. These eigenvectors are unique within a scale factor. From the equivalence of (6.20)
and (6.21), p , as given by (6.20), is the steady state probability vector. Each eigenvalue
∏δ of [Wδ ] corresponds to an eigenvalue ∏ of [Q] with the correspondence ∏δ = 1 + ∏δ , i.e.,
∏ = (∏δ − 1)/δ . Thus the eigenvalue 1 of [Wδ ] corresponds to the eigenvalue 0 of [Q]. Since
∏δ  ≤ 1 and ∏δ 6= 1 for all other eigenvalues, the other eigenvalues of [Q] all have strictly
negative real parts, completing the proof.
We continue to look at an irreducible nstate Markov process, and assume for simplicity
T
T
that [Q] has n distinct eigenvalues, ∏1 , . . . , ∏n . Let ∫ 1 , ∫ 2 , . . . , ∫ n and π 1 , . . . , π n be the
corresponding right and left eigenvectors respectively. Assume that these eigenvectors are
T
T
scaled so that π i ∫ i = 1 for each i, and recall that π i ∫ j = 0 for i 6= j . Thus, if [V ] is deﬁned
as the matrix with columns ∫ 1 , ∫ 2 , . . . , ∫ n , then its inverse, [V ]−1 , is the matrix whose rows
T
T
are π 1 , . . . , π n . Finally, let [Λ] be the diagonal matrix with elements ∏1 , . . . , ∏n . We then
have the relationships
[Λ] = [V ]−1 [Q][V ] ; [Q] = [V ][Λ][V ]−1 . (6.34) We then also have [Q]m = [V ]−1 [Λ]m [V ], and thus (6.29) can be written as
[P (t)] = 1
X m=0 [V ] tm [Λ]m
[V ]−1 = [V ][etΛ ][V ]−1 ,
m! (6.35) where [etΛ ] is the diagonal matrix with elements et∏1 , et∏2 , . . . , , et∏n . Finally, if we br...
View
Full
Document
 Spring '09
 R.Srikant

Click to edit the document details