Discrete-time stochastic processes

55 but the result is still proportional to 2 1n and 12

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: d 0 as 1/n with increasing n. Since Pr |Sn /n − X | ≥ ≤ is sandwiched between 0 and a quantity approac© hing 0 as 1/n, (1.51) is satisfied. To make this particularly ™ clear, note that for a given ≤, Pr |Sn /n − X | ≥ ≤ represents a sequence of real numbers, one for each positive integer n. The limit of a sequence of real numbers a1 , a2 , . . . is defined to exist17 and equal some number b if for each δ > 0, there is an f (δ ) such that |b − an | ≤ δ © ™ for all n ≥ f (δ ). For the limit here, b = 0, an = Pr |Sn /n − X | ≥ ≤ and, from (1.53), f (δ ) can be taken to satisfy δ = σ 2 /(f (δ )≤2 ). Thus we see that (1.52) simply spells out the definition of the limit in (1.51). One might reasonably ask at this point what (1.52) adds to the more specific statement in σ2 (1.53). In particular (1.53) shows that the function f (≤, δ ) in (1.52) can be taken to be ≤2 δ , 17 Exercise 1.9 is designed to help those unfamiliar with limits to understand this definition. 30 CHAPTER 1. INTRODUCTION AND REVIEW OF PROBABILITY 1 ✻ ✏ ✮ ✏ ✏ FSn /n 1−δ ✲ ✛2≤ δ= σ2 n≤2 ❄ 0 X Figure 1.9: Approximation of the distribution function FSn /n of a sample average by a step function at the©mean. From (1.51), the probability that Sn /n differs from X by ™ more than ≤ (i.e., Pr |Sn /n − X | ≥ ≤ ) approaches 0 with increasing n. This means that the limiting distribution function goes from 0 to 1 as x goes from X − ≤ to X + ≤. Since this is true for every ≤ > 0 (presumably with slower convergence as ≤ gets smaller), FSn /n approaches a unit step at X . Note the rectangle of width 2≤ and height 1 − δ in the figure. The meaning of (1.52) is that the distribution function of Sn /n enters this rectangle from below and exits from above, thus approximating a unit step. Note that there are two ‘fudge factors’ here, ≤ and δ and, since we are approximating an entire distribution function, neither can be omitted, except by going to a limit as n → 1. which provides added information on the rapidity of convergence. The reason for (1.52) is that it remains valid when the theorem is generalized. For variables that are not IID or have an infinite variance, Eq. (1.53) is no longer necessarily valid. For the Bernoulli process of Example 1.3.1, the binomial PMF can be compared with the upper bound in (1.55), and in fact the example in Figure 1.8 uses the binomial PMF. Exact calculation, and bounds based on the binomial PMF, provide a tighter bound than (1.55), but the result is still proportional to σ 2 , 1/n, and 1/≤2 . 1.4.3 Relative frequency We next show that (1.52) and (1.51) can be applied to the relative frequency of an event as well as to the sample average of a random variable. Suppose that A is some event in a single experiment, and that the experiment is independently repeated n times. Then, in the probability model for the n repetitions, let Ai be the event that A occurs at the ith trial, 1 ≤ i ≤ n. The events A1 , A2 , . . . , An are then IID. If we let IAi be the indicator rv for A on the nth trial, then the rv Sn = IA1 + IA2 + · · · + IAn is the number of occurences of A over the n trials. It follows that Pn IA Sn relative frequency of A = = i=1 i . (1.54) n n Thus the relative frequency of A is the sample average of IA and, from (1.52), Pr {| relative frequency of A − Pr {A} |≥ ≤} ≤ σ2 , n≤2 (1.55) 1.4. THE LAWS OF LARGE NUMBERS 31 where σ 2 is the variance of IA , i.e., σ 2 = Pr {A} (1 − Pr {A}). The main point here is everything we learn about sums of IID rv’s applies equally to the relative frequency of IID events. In fact, since the indicator rv’s are binary, everything known about the binomial PMF also applies. Relative frequency is extremely important, but we need not discuss it much in what follows, since it is simply a special case of sample averages. 1.4.4 The central limit theorem The law of large numbers says that Sn /n is close to X with high probability for large n, but this most emphatically does not mean that Sn is close to nX . In fact, the standard √ deviation of Sn is σ n , which increases with n. This leads us to ask about the behavior of √ Sn / n, since its standard deviation is σ and does not vary with n. The celebrated central limit theorem answers this question by stating that if σ 2 < 1, then for every real number y, ∑Ω æ∏ Z y µ 2∂ Sn − nX 1 −x √ √ exp lim Pr ≤y = dx. (1.56) n→1 2 nσ 2π −1 The quantity on the right side of (1.56) is the distribution function of a normalized Gaussian rv, which is known to have mean 0 and variance 1. The sequence of rv’s Zn = (Sn − √ nX )/( n σ ) on the left also have mean 0 and variance 1 for all n. The central limit theorem, as expressed in (1.56), says that the sequence of distribution functions, FZ1 (y ), FZ2 (y ), . . . converges at each value of y to FΦ (y ) as n → 1, where FΦ (y ) is the distribution function on the...
View Full Document

This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.

Ask a homework question - tutors are online