This preview shows page 1. Sign up to view the full content.
Unformatted text preview: alk and the martingale are simply related by Zn = Sn − nX , and thus general results
about martingales can easily be applied to random walks.
Example 7.6.2 (Sums of dependent zeromean variables). Let {Xi ; i ≥ 1} be a set
of dependent random variables satisfying E [Xi  Xi−1 , . . . , X1 ] = 0. Then {Zn ; n ≥ 1},
where Zn = X1 + · · · + Xn , is a zero mean martingale. To see this, note that
E [Zn  Zn−1 , . . . , Z1 ] = E [Xn + Zn−1  Zn−1 , . . . , Z1 ] = E [Xn  Xn−1 , . . . , X1 ] + E [Zn−1  Zn−1 , . . . , Z1 ] = Zn−1 . This is a more general example than it appears, since given any martingale {Zn ; n ≥ 1}, we
can deﬁne Xn = Zn − Zn−1 for n ≥ 2 and deﬁne X1 = Z1 . Then E [Xn  Xn−1 , . . . , X1 ] = 0
for n ≥ 2. If the martingale is zero mean (i.e., if E [Z1 ] = 0), then E [X1 ] = 0 also. 7.6. MARTINGALES AND SUBMARTINGALES 301 Example 7.6.3 (Productform martingales). Another example is a product of unit
mean IID random variables. Thus if Zn = X1 X2 . . . Xn , we have
E [Zn  Zn−1 , . . . , Z1 ] = E [Xn Zn−1  Zn−1 , . . . , Z1 ] = E [Xn ] Zn−1 = Zn−1 . (7.47) A particularly simple case of this product example is where Xn = 2 with probability 1/2
and Xn = 0 with probability 1/2. Then
Pr {Zn = 2n } = 2−n ; Pr {Zn = 0} = 1 − 2−n ; E [Zn ] = 1. (7.48) Thus limn→1 Zn = 0 with probability 1, but E [Zn ] = 1 for all n and limn→1 E [Zn ] = 1.
This is an important example to keep in mind when trying to understand why proofs about
martingales are necessary and nontrivial.
An important example of a productform martingale is as follow: let {Xi ; i ≥ 1} be an
IID sequence, and let {Sn = X1 + · · · + Xn ; n ≥ 1} be a random walk. Assume that the
semiinvariant generating function ∞ (r) = ln{E [exp(rX )]} exists in some region of r around
0. For each n ≥ 1, let Zn be deﬁned as
Zn = exp{rSn − n∞ (r)} (7.49) = exp{rXn − ∞ (r)} exp{rSn−1 − (n − 1)∞ (r)}
= exp{rXn − ∞ (r)}Zn−1 . (7.50) Taking the conditional expectation of this,
E [Zn  Zn−1 , . . . , Z1 ] = E [exp(rXn − ∞ (r))] E [Zn−1  Zn−1 , . . . , Z1 ]
= Zn−1 . (7.51) where we have used the fact that E [exp(rXn )] = exp(∞ (r)). Thus we see that {Zn ; n ≥ 1}
is a martingale of the productform. 7.6.2 Markov modulated random walks Frequently it is useful to generalize random walks to allow some dependence between the
variables being summed. The particular form of dependence here is the same as the Markov
reward processes of Section 4.5. The treatment in Section 4.5 discussed only expected
rewards, whereas the treatment here focuses on the random variables themselves. Let
{Ym ; m ≥ 0} be a sequence of (possibly dependent) rv’s, and let
{Sn ; n ≥ 1} where Sn = n−1
X Ym . (7.52) m=0 be the process of successive sums of these random variables. Let {Xn ; n ≥ 0} be a Markov
chain, and assume that each Yn can depend on Xn and Xn+1 . Conditional on Xn and Xn+1 ,
however, Yn is independent of Yn−1 , . . . , Y1 , and of Xi for all i 6= n. Assume that Yn , conditional on Xn and Xn+1 has a distribution function Fij (y ) = Pr {Yn ≤ y  Xn = i, Xn+1 = j }. 302 CHAPTER 7. RANDOM WALKS, LARGE DEVIATIONS, AND MARTINGALES Thus each rv Yn depends only on the associated transition in the Markov chain, and this
dependence is the same for all n.
The process {Sn ; n ≥ 1} is called a Markov modulated random walk. It is also the sequence
of epochs in a semiMarkov process. For each m, Ym is associated with the transition in
the Markov chain from time m to m + 1, and Sn is the aggregate reward up to but not
including time n. Let Y ij denote E [Yn  Xn = i, Xn+1 = j ] and Y i denote E [Yn  Xn = i].
P
Let {Pij } be the set of transition probabilities for the Markov chain, so Y i = j Pij Y ij .
We may think of the process {Yn ; n ≥ 0} as evolving along with the Markov chain. The
distributions of the variables Yn are associated with the transitions from Xn to Xn+1 , but
the Yn are otherwise independent random variables.
In order to deﬁne a martingale related to the process {Sn ; n ≥ 1}, we must subtract the
mean reward from {Sn } and must also compensate for the eﬀect of the state of the Markov
chain. The appropriate compensation factor turns out to be the relative gain vector deﬁned
in Section 4.5.
For simplicity, consider only ﬁnitestate irreducible Markov chains with M states. Let π =
(π1 , . . . , πM ) be the steady state probability vector for the chain, let Y = (Y 1 , . . . , Y M )T
be the vector of expected rewards, let g = π Y be the steady state gain per unit time, and
let w = (w1 , . . . , wM )T be the relative gain vector. From (4.42), w is the unique solution
to
w + g e = Y + [P ]w ; w1 = 0. (7.53) We assume a ﬁxed starting state X0 = k. As we now show, the process Zn ; n ≥ 1 given by
Zn = Sn − ng + wXn − wk ; n≥1 (7.54) is a martingale. First condition on a given state, Xn−1 = i.
E [Zn  Zn−1 , Zn−2 , . . . , Z1 , Xn−1 = i] . (7.55) Since Sn = S...
View Full
Document
 Spring '09
 R.Srikant

Click to edit the document details