Discrete-time stochastic processes

This is a powerful theorem in more advanced work but

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: alk and the martingale are simply related by Zn = Sn − nX , and thus general results about martingales can easily be applied to random walks. Example 7.6.2 (Sums of dependent zero-mean variables). Let {Xi ; i ≥ 1} be a set of dependent random variables satisfying E [Xi | Xi−1 , . . . , X1 ] = 0. Then {Zn ; n ≥ 1}, where Zn = X1 + · · · + Xn , is a zero mean martingale. To see this, note that E [Zn | Zn−1 , . . . , Z1 ] = E [Xn + Zn−1 | Zn−1 , . . . , Z1 ] = E [Xn | Xn−1 , . . . , X1 ] + E [Zn−1 | Zn−1 , . . . , Z1 ] = Zn−1 . This is a more general example than it appears, since given any martingale {Zn ; n ≥ 1}, we can define Xn = Zn − Zn−1 for n ≥ 2 and define X1 = Z1 . Then E [Xn | Xn−1 , . . . , X1 ] = 0 for n ≥ 2. If the martingale is zero mean (i.e., if E [Z1 ] = 0), then E [X1 ] = 0 also. 7.6. MARTINGALES AND SUBMARTINGALES 301 Example 7.6.3 (Product-form martingales). Another example is a product of unit mean IID random variables. Thus if Zn = X1 X2 . . . Xn , we have E [Zn | Zn−1 , . . . , Z1 ] = E [Xn Zn−1 | Zn−1 , . . . , Z1 ] = E [Xn ] Zn−1 = Zn−1 . (7.47) A particularly simple case of this product example is where Xn = 2 with probability 1/2 and Xn = 0 with probability 1/2. Then Pr {Zn = 2n } = 2−n ; Pr {Zn = 0} = 1 − 2−n ; E [Zn ] = 1. (7.48) Thus limn→1 Zn = 0 with probability 1, but E [Zn ] = 1 for all n and limn→1 E [Zn ] = 1. This is an important example to keep in mind when trying to understand why proofs about martingales are necessary and non-trivial. An important example of a product-form martingale is as follow: let {Xi ; i ≥ 1} be an IID sequence, and let {Sn = X1 + · · · + Xn ; n ≥ 1} be a random walk. Assume that the semi-invariant generating function ∞ (r) = ln{E [exp(rX )]} exists in some region of r around 0. For each n ≥ 1, let Zn be defined as Zn = exp{rSn − n∞ (r)} (7.49) = exp{rXn − ∞ (r)} exp{rSn−1 − (n − 1)∞ (r)} = exp{rXn − ∞ (r)}Zn−1 . (7.50) Taking the conditional expectation of this, E [Zn | Zn−1 , . . . , Z1 ] = E [exp(rXn − ∞ (r))] E [Zn−1 | Zn−1 , . . . , Z1 ] = Zn−1 . (7.51) where we have used the fact that E [exp(rXn )] = exp(∞ (r)). Thus we see that {Zn ; n ≥ 1} is a martingale of the product-form. 7.6.2 Markov modulated random walks Frequently it is useful to generalize random walks to allow some dependence between the variables being summed. The particular form of dependence here is the same as the Markov reward processes of Section 4.5. The treatment in Section 4.5 discussed only expected rewards, whereas the treatment here focuses on the random variables themselves. Let {Ym ; m ≥ 0} be a sequence of (possibly dependent) rv’s, and let {Sn ; n ≥ 1} where Sn = n−1 X Ym . (7.52) m=0 be the process of successive sums of these random variables. Let {Xn ; n ≥ 0} be a Markov chain, and assume that each Yn can depend on Xn and Xn+1 . Conditional on Xn and Xn+1 , however, Yn is independent of Yn−1 , . . . , Y1 , and of Xi for all i 6= n. Assume that Yn , conditional on Xn and Xn+1 has a distribution function Fij (y ) = Pr {Yn ≤ y | Xn = i, Xn+1 = j }. 302 CHAPTER 7. RANDOM WALKS, LARGE DEVIATIONS, AND MARTINGALES Thus each rv Yn depends only on the associated transition in the Markov chain, and this dependence is the same for all n. The process {Sn ; n ≥ 1} is called a Markov modulated random walk. It is also the sequence of epochs in a semi-Markov process. For each m, Ym is associated with the transition in the Markov chain from time m to m + 1, and Sn is the aggregate reward up to but not including time n. Let Y ij denote E [Yn | Xn = i, Xn+1 = j ] and Y i denote E [Yn | Xn = i]. P Let {Pij } be the set of transition probabilities for the Markov chain, so Y i = j Pij Y ij . We may think of the process {Yn ; n ≥ 0} as evolving along with the Markov chain. The distributions of the variables Yn are associated with the transitions from Xn to Xn+1 , but the Yn are otherwise independent random variables. In order to define a martingale related to the process {Sn ; n ≥ 1}, we must subtract the mean reward from {Sn } and must also compensate for the effect of the state of the Markov chain. The appropriate compensation factor turns out to be the relative gain vector defined in Section 4.5. For simplicity, consider only finite-state irreducible Markov chains with M states. Let π = (π1 , . . . , πM ) be the steady state probability vector for the chain, let Y = (Y 1 , . . . , Y M )T be the vector of expected rewards, let g = π Y be the steady state gain per unit time, and let w = (w1 , . . . , wM )T be the relative gain vector. From (4.42), w is the unique solution to w + g e = Y + [P ]w ; w1 = 0. (7.53) We assume a fixed starting state X0 = k. As we now show, the process Zn ; n ≥ 1 given by Zn = Sn − ng + wXn − wk ; n≥1 (7.54) is a martingale. First condition on a given state, Xn−1 = i. E [Zn | Zn−1 , Zn−2 , . . . , Z1 , Xn−1 = i] . (7.55) Since Sn = S...
View Full Document

Ask a homework question - tutors are online