Discrete-time stochastic processes

# Thus yn ym is close to 0 with high probability more

This preview shows page 1. Sign up to view the full content.

This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: verage as being essentially equal to the mean is highly appropriate. At the next intuitive level down, the meaning of the word essential ly becomes important and thus involves the details of the above laws. All of the results involve how the rv’s Sn /n change with n and in particular become better and better approximated by X . When we talk about a sequence of rv’s (namely a sequence of functions on the sample space) being approximated by a numerical constant, we are talking about some kind of convergence, but it clearly is not as simple as a sequence of real numbers (such as 1/n for example) converging to some other number (0 for example). The purpose of this section, is to give names and deﬁnitions to these various forms of convergence. This will give us increased understanding of the laws of large numbers already developed, but, more important, we can later apply these convergence results to sequences of random variables other than sample averages of IID rv’s. We discuss four types of convergence in what follows, convergence in distribution, in probability, in mean square, and with probability 1. For each, we ﬁrst recall the type of large number result with that type of convergence and then give the general deﬁnition. The examples are all based on a sequence {Xn ; n ≥ 1} of rv’s with partial sums Sn = X1 + · · · + Xn and the deﬁnitions are given in terms of an arbitrary sequence {Yn ; n ≥ 1} of rv’s . We start with the central limit theorem, which, from (1.56) says Ω æ Zy µ 2∂ Sn − nX 1 −x √ √ exp lim Pr ≤y = dx for every y ∈ R. n→1 2 nσ 2π −1 This is illustrated in Figure 1.10 and says that the sequence (in n) of distribution functions n o √ Pr Sn −nX ≤ y converges at every y to the normal distribution function at y . This is an nσ example of convergence in distribution. 40 CHAPTER 1. INTRODUCTION AND REVIEW OF PROBABILITY Deﬁnition 1.7. A sequence of random variables, Y1 , Y2 , . . . , converges in distribution to a random varible Z if limn→1 FYn (z ) = FZ (z ) at each z for which FZ (z ) is continuous. Convergence in distribution does not say that the rv’s themselves are converging in any sense (see Exercise 1.29), but only that their distribution functions are converging. Note √ that for the CLT, it is the rv’s Sn −nX that are converging in distribution. Convergence in nσ distribution does not say that the PMF’s or densities of the rv’s are converging, and it is easy to see that a sequence of PMF’s can never converge to a density. ★✥ ★✥ MS ✧✦ ✧✦ W.P.1 In probability Distribution Figure 1.12: Relationship between diﬀerent kinds of convergence. Convergence in distribution is the most general and is implied by all the others. Convergence in probability is the next most general and is implied by convergence with probability 1 (W.P.1) and by mean square (MS) convergence, neither of which imply the other. Next, the weak law of large numbers, in the form of (1.57), says that ΩØ æ Ø Ø Sn Ø lim Pr Ø − XØ ≥ ≤ = 0 for every ≤ > 0. n→1 n As illustrated in Figure 1.8, this means that Sn /n converges in distribution to a unit step function at X . This is an example of convergence in probability, Deﬁnition 1.8. A sequence of random variables Y1 , Y2 , . . . , converges in probability to a real number z if limn→1 Pr {|Yn − z | ≥ ε} = 0 for every ε > 0. An equivalent statement, as illustrated in Figure 1.9, is that the sequence Y1 , Y2 , . . . , converges in probability to z if limn→1 FYn (y ) = 0 for all y < z and limn→1 FYn (y ) = 1 for all y > z . This shows (as illustrated in Figure 1.12) that convergence in probability is a special case of convergence in distribution, since with convergence in probability, the sequence FYn of distribution functions converges to a unit step at z . Note that limn→1 FYn (z ) for the z where the step occurs is not speciﬁed. However, the step function is not continuous at z , so the limit there need not be speciﬁied for convergence in distribution. Convergence in probability says quite a bit more than convergence in distribution. As an important example of this, consider the diﬀerence Yn − Ym for n and m both large. If {Yn ; n ≥ 1} converges in probability to z , then Yn − z and Ym − z are close to 0 with high probability for large n and m. Thus Yn − Ym is close to 0 with high probability. More precisely, limm→1,n→1 Pr {|Yn − Ym | > ε} = 0 for every ε > 0. If the sequence {Yn ; n ≥ 1} 1.4. THE LAWS OF LARGE NUMBERS 41 merely converges in distribution to some arbitrary distribution, then Yn − Ym can be large with high probability, even when n and m are large. An example of this is given in Exercise 1.29. In other words, convergence in distribution need not mean that the random variables converge in any sense — it is only their distribution functions that must converge. There is something slightly paradoxical here, since the CLT seems to say a great deal more about how Sn /n approaches X than the weak law, but it...
View Full Document

## This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.

Ask a homework question - tutors are online