Unformatted text preview: le knight moves. What
is the expected time until he rst returns to the lower left corner square? 1.64 Exercise. Recall the de nition of positive recurrence on page 1-22. Show that positive
recurrence is a class property. 1.65 Exercise. Suppose a Markov chain has a stationary distribution and the state j is
null recurrent. Show that j = 0.
1.66 Exercise Birth-collapse chain . Consider a Markov chain on S = f0; 1; 2; : : : g
having P i; i + 1 = pi , P i; 0 = 1 , pi for all i, with p0 = 1 and 0 pi 1 for all i 0.
Show that Q i The chain is recurrent if and only if limn!1 n=1 pi = 0. This, in turn, is equivalent to
the condition 1 1 , pi = 1. This was just for interest; not a problem or a hint.
ii The chain is positive recurrent if and only if 1 n=1 pi 1.
iii What is the stationary distribution if pi = 1=i + 1? 1.10 General state space Markov chains
So far we have been discussing Markov chains with nite or countably in nite state spaces.
But many applications are most naturally modeled as processes moving on more general
state spaces, such as the real line or higher dimensional Euclidean spaces.
WARNING: This section may be rather long and tiring. It should probably be revised
and streamlined... Suggestions welcome.
1.67 Example. Another standard use of the term random walk" is for a sequence of partial sums of iid random variables. For example, we might have Z1 ; Z2; : : : independent
and distributed according to the normal distribution N ; 1 with mean and variance 1,
Stochastic Processes J. Chang, March 30, 1999 1.10. GENERAL STATE SPACE MARKOV CHAINS Page 1-35 and de ne the random walk X0 ; X1 ; : : : by Xn = Z1 + + Zn for n 0. In contrast with the
simple symmetric random walk, which moves around on the integers, such a normal random
walk has probability 0 of being in any given countable set of numbers at any positive time. 1.68 Example Autoregressive process . Autoregressive processes are the bread
and butter of time series analysis. Here is a simple example. Let X0 have a Normal distri2
bution N 0 ; 0 ; and de ne X1 ; X2 ; : : : recursively by Xt = Xt,1 + Zt , where Z1 ; Z2 ; : : :
2 . Then fXt g is an example of an autoregressive process of order 1.
are iid N 0; 1.69 Example Reflected random walk . Let X1 ; X2 ; : : : be iid, and de ne the process fWt g by the recursion Wt = maxf0; Wt,1 + Xt g for t 0:
and W0 = 0, say. Then fWt g is called a re ected random walk . The W process makes
iid increments like a random walk, except when taking such an increment would cause the
process to become negative, in which case the process takes the value 0. Re ected random
walks arise in diverse contexts, including queueing theory and statistical procedures for
quickly detecting a change in a probability distribution. As an example, if the random
variables X1 ; X2 ; : : : are iid with distribution N ; 1, with the drift" 0, then the
re ected random walk keeps trying to drift downward and...
View Full Document