{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

Markov

# It should probably be revised and streamlined

This preview shows page 1. Sign up to view the full content.

This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: le knight moves. What is the expected time until he rst returns to the lower left corner square? 1.64 Exercise. Recall the de nition of positive recurrence on page 1-22. Show that positive recurrence is a class property. 1.65 Exercise. Suppose a Markov chain has a stationary distribution  and the state j is null recurrent. Show that j  = 0. 1.66 Exercise Birth-collapse chain . Consider a Markov chain on S = f0; 1; 2; : : : g having P i; i + 1 = pi , P i; 0 = 1 , pi for all i, with p0 = 1 and 0 pi 1 for all i 0. Show that Q i The chain is recurrent if and only if limn!1 n=1 pi = 0. This, in turn, is equivalent to i P the condition 1 1 , pi  = 1. This was just for interest; not a problem or a hint. i=1 PQ ii The chain is positive recurrent if and only if 1 n=1 pi 1. n=1 i iii What is the stationary distribution if pi = 1=i + 1? 1.10 General state space Markov chains So far we have been discussing Markov chains with nite or countably in nite state spaces. But many applications are most naturally modeled as processes moving on more general state spaces, such as the real line or higher dimensional Euclidean spaces. WARNING: This section may be rather long and tiring. It should probably be revised and streamlined... Suggestions welcome. 1.67 Example. Another standard use of the term random walk" is for a sequence of partial sums of iid random variables. For example, we might have Z1 ; Z2; : : : independent and distributed according to the normal distribution N ; 1 with mean  and variance 1, Stochastic Processes J. Chang, March 30, 1999 1.10. GENERAL STATE SPACE MARKOV CHAINS Page 1-35 and de ne the random walk X0 ; X1 ; : : : by Xn = Z1 +    + Zn for n  0. In contrast with the simple symmetric random walk, which moves around on the integers, such a normal random walk has probability 0 of being in any given countable set of numbers at any positive time. 1.68 Example Autoregressive process . Autoregressive processes are the bread and butter of time series analysis. Here is a simple example. Let X0 have a Normal distri2 bution N 0 ; 0 ; and de ne X1 ; X2 ; : : : recursively by Xt = Xt,1 + Zt , where Z1 ; Z2 ; : : : 2 . Then fXt g is an example of an autoregressive process of order 1. are iid N 0; 1.69 Example Reflected random walk . Let X1 ; X2 ; : : : be iid, and de ne the process fWt g by the recursion Wt = maxf0; Wt,1 + Xt g for t 0: and W0 = 0, say. Then fWt g is called a re ected random walk . The W process makes iid increments like a random walk, except when taking such an increment would cause the process to become negative, in which case the process takes the value 0. Re ected random walks arise in diverse contexts, including queueing theory and statistical procedures for quickly detecting a change in a probability distribution. As an example, if the random variables X1 ; X2 ; : : : are iid with distribution N ; 1, with the drift"  0, then the re ected random walk keeps trying to drift downward and...
View Full Document

{[ snackBarMessage ]}

Ask a homework question - tutors are online