Discrete-time stochastic processes

Discrete-time stochastic processes

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: OUNTABLE STATE SPACES c) Verify that the above hypothesis is correct. d) Find an expression for π0 . e) Find an expression for the steady state probability that an arriving customer is discarded. Chapter 7 RANDOM WALKS, LARGE DEVIATIONS, AND MARTINGALES 7.1 Introduction Definition 7.1. Let {Xi ; i ≥ 1} be a sequence of IID random variables, and let Sn = X1 + X2 + · · · + Xn . The integer-time stochastic process {Sn ; n ≥ 1} is cal led a random walk, or, more precisely, the random walk based on {Xi ; i ≥ 1}. For any given n, Sn is simply a sum of IID random variables, but here the behavior of the entire random walk process, {Sn ; n ≥ 1}, is of interest. Thus, for a given real number α > 0, we might try to find the probabiity that Sn ≥ α for any n, or given that Sn ≥ α for one or more values of n, we might want to find the distribution of the smallest n such that Sn ≥ α. typical questions about random walks are finding the smallest n such that Sn reaches or exceeds a threshold, and finding the probability that the threshold is ever reached or crossed. Since Sn tends to drift downward with increasing n if E [X ] = X < 0, and tends to drift upward if X > 0, the results to be obtained depend critically on whether X < 0, X > 0, or X = 0. Since results for X < 0 can be easily translated into results for X > 0 by considering {−Sn ; n ≥ 0}, we will focus on the case X < 0. As one might expect, both the results and the techniques have a very different flavor when X = 0, since here the random walk does not drift but typically wanders around in a rather aimless fashion. The following three subsections discuss three special cases of random walks. The first two, simple random walks and integer random walks, will be useful throughout as examples, since they can be easily visualized and analyzed. The third special case is that of renewal processes, which we have already studied and which will provide additional insight into the general study of random walks. After this, Sections 3.2 and 3.3 show how two ma jor application areas, G/G/1 queues and 279 280 CHAPTER 7. RANDOM WALKS, LARGE DEVIATIONS, AND MARTINGALES hypothesis testing, can be treated in terms of random walks. These sections also show why questions related to threshold crossings are so important in random walks. Section 3.4 then develops the theory of threshold crossing for general random walks and Section 3.5 extends and in many ways simplifies these results through the use of stopping rules and a powerful generalization of Wald’s equality known as Wald’s identity. The remainder of the chapter is devoted to a rather general type of stochastic process called martingales. The topic of martingales is both a sub ject of interest in its own right and also a tool that provides additional insight into random walks, laws of large numbers, and other basic topics in probability and stochastic processes. 7.1.1 Simple random walks Suppose X1 , X2 , . . . are IID binary random variables, each taking on the value 1 with probability p and −1 with probability q = 1 − p. Letting Sn = X1 + · · · + Xn , the sequence of sums {Sn ; n ≥ 1}, is called a simple random walk. Sn is the difference between positive and negative occurrences in the first n trials. Thus, if there are j positive occurrences for 0 ≤ j ≤ n, then Sn = 2j − n, and Pr {Sn = 2j − n} = n! pj (1 − p)n−j . j !(n − j )! (7.1) This distribution allows us to answer questions about Sn for any given n, but it is not very helpful in answering such questions as the following: for any given integer k > 0, what is the probability that the sequence S1 , S2 , . . . ever reaches or exceeds k? This probability can S be expressed as1 Pr { 1 {Sn ≥ k}} and is referred to as the probability that the random n=1 walk crosses a threshold at k. Exercise 7.1 demonstrates the surprisingly simple result that for a simple random walk with p < 1/2, this threshold crossing probability is (1 )µ ∂k [ p Pr {Sn ≥ k} = . (7.2) 1−p n=1 Sections 7.4 and 7.5 treat this same question for general random walks. They also treat questions such as the overshoot given a threshold crossing, the time at which the threshold is crossed given that it is crossed, and the probability of crossing such a positive threshold before crossing a given negative threshold. 7.1.2 Integer-valued random walks Suppose next that X1 , X2 , . . . are arbitrary IID integer-valued random variables. We can again ask for the probability that such an integer valued random walk crosses a threshold 1 This same probability is often expressed as as Pr {sup1 Sn ≥ k}. For a general random walk, the n=1 S event n≥1 {Sn ≥ k} is slightly different from supn≥1 Sn ≥ k. The latter event can include sample sequences s1 , s2 , . . . in which a subsequence of values sn approach k as a limit but never quite reach k. This is impossible for a simple random walk since all sk must be integers. It is possible, but can be shown to have probability zero, for general random walks. We will avoid this sillin...
View Full Document

This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.

Ask a homework question - tutors are online