Discrete-time stochastic processes

0 1 q 1p p p q 2 p q 3 p q 4 q figure 51

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: ] can be partitioned ∑ ∏ Pr 0 as [P ] = . Ptr Ptt ∑ ∏ [Pr ]n [0] a) Show that [P ]n can be partitioned as [P ]n = . That is, the blocks on n [Pij ] [Ptt ]n the diagonal are simply products of the corresponding blocks of [P ], and the lower left block is whatever it turns out to be. 186 CHAPTER 4. FINITE-STATE MARKOV CHAINS b) Let Qi be the probability that the chain will be in a recurrent state after K transitions, P K starting from state i, i.e., Qi = j ≤M Pij . Show that Qi > 0 for all transient i. nK c) Let Q be the minimum Qi over all transient i and show that Pij ≤ (1 − Q)n for all n approaches the all zero matrix [0] with increasing n). transient i, j (i.e., show that [Ptt ] d) Let π = (π r , π t ) be a left eigenvector of [P ] of eigenvalue 1 (if one exists). Show that π t = 0 and show that π r must be positive and be a left eigenvector of [Pr ]. Thus show that π exists and is unique (within a scale factor). e) Show that e is the unique right eigenvector of [P ] of eigenvalue 1 (within a scale factor). Exercise 4.14. Generalize Exercise 4.13 to the case of a Markov chain [P ] with r recurrent classes and one or more transient classes. In particular, a) Show that [P ] has exactly r linearly independent left eigenvectors, π (1) , π (2) , . . . , π (r) of eigenvalue 1, and that the ith can be taken as a probability vector that is positive on the ith recurrent class and zero elsewhere. b) Show that [P ] has exactly r linearly independent right eigenvectors, ∫ (1) , ∫ (2) , . . . , ∫ (r) (i) of eigenvalue 1, and that the ith can be taken as a vector with ∫j equal to the probability that recurrent class i will ever be entered starting from state j . Exercise 4.15. Prove Theorem 4.8. Hint: Use Theorem 4.7 and the results of Exercise 4.13. Exercise 4.16. Generalize Exercise 4.15 to the case of a Markov chain [P ] with r aperiodic recurrent classes and one or more transient classes. In particular, using the left and right eigenvectors π (1) , π (2) , . . . , π (r) and ∫ (1) , . . . , ∫ (r) of Exercise 4.14, show that X lim [P ]n = ∫ (i)π (i) . n→1 i Exercise 4.17. Suppose a Markov chain with an irreducible matrix [P ] is periodic with period d and let Ti , 1 ≤ i ≤ d, be the ith subset in the sense of Theorem 4.3. Assume the states are numbered so that the first M1 states are in T1 , the next J2 are in T2 , and so forth. Thus [P ] has the block form given by 0 0 [P ] = . . . 0 [Pd ] [P1 ] 0 .. . 0 0 .. . .. . .. . 0 .. . .. . [P2 ] .. .. . . .. .. . . [Pd−1 ] .. .. . . 0 where [Pi ] has dimension Mi by Mi+1 for i < d and Md by M1 for i = d 4.8. EXERCISES 187 a) Show that [P ]d has the form .. . 0 0 [Q1 ] d .. .. [P ] = 0 . . [Q2 ] .. . [Qd ] 0 0 where [Qi ] = [Pi ][Pi+1 ] . . . [Pd ][P1 ] . . . [Pi−1 ]. b) Show that [Qi ] is the matrix of an ergodic Markov chain, so that with the eigenvectors P defined in Exercises 4.14 and 4.16, limn→1 [P ]nd = i ∫ (i)π (i) . ˆ ˆ ˆ c) Show that π (i) , the left eigenvector of [Qi ] of eigenvalue 1 satisfies π (i) [Pi ] = π (i+1) for ˆ ˆ i < d and π (d) [Pd ] = π (1) . √ ˆ ˆ ˆ ˆ d) Let α = 2π d −1 and let π (k) = (π (1) , π (2) eαk , π (3) e2αk , . . . , π (d) e(d−1)αk ). Show that π (k) is a left eigenvector of [P ] of eigenvalue e−αk . Exercise 4.18. (continuation of Exercise 4.17). defined in Exercises 4.14 and 4.16, nd lim [P ] [P ] = n→1 d X a) Show that, with the eigenvectors ∫ (i)π (i+1) i=1 where π (d+1) is taken to be π (1) . b) Show that, for 1 ≤ j < d, lim [P ]nd [P ]j = n→1 d X ∫ (i)π (i+j ) i=1 where π (d+m) is taken to be π (m) for 1 ≤ m < d. c) Show that n o lim [P ]nd I + [P ] + . . . , [P ]d−1 = n→1 √d X i=1 ∫ (i) !√ d X ! π (i+j ) . i=1 d) Show that ¥ 1≥ n [P ] + [P ]n+1 + · · · + [P ]n+d−1 = e π n→1 d lim where π isP steady-state probability vector for [P ]. Hint: Show that e = the π = (1/n) i π (i) . e) Show that the above result is also valid for periodic unichains. P i∫ (i) and 188 CHAPTER 4. FINITE-STATE MARKOV CHAINS Exercise 4.19. Assume a friend has developed an excellent program for finding the steadystate probabilities for finite-state Markov chains. More precisely, given the transition matrix n [P], the program returns limn→1 Pii for each i. Assume all chains are aperiodic. a) You want to find the expected time to first reach a given state k starting from a different state m for a Markov chain with transition matrix [P ]. You modify the matrix to [P 0 ] where 0 0 0 Pkm = 1, Pkj = 0 for j 6= m, and Pij = Pij otherwise. How do you find the desired first passage time from the program output given [P 0 ] as an input? (Hint: The times at which a Markov chain enters any given state can be considered as renewals in a (perhaps delayed) renewal process). b) Using the same [P 0 ] as the program input, how can you find the expected number of returns to state m before the first passage to state k? c) Suppose, for the same Markov chain [P ] and th...
View Full Document

Ask a homework question - tutors are online