This preview shows page 1. Sign up to view the full content.
Unformatted text preview: d 0 as 1/n with increasing n. Since Pr Sn /n − X  ≥ ≤ is sandwiched
between 0 and a quantity approac©
hing 0 as 1/n, (1.51) is satisﬁed. To make this particularly
™
clear, note that for a given ≤, Pr Sn /n − X  ≥ ≤ represents a sequence of real numbers,
one for each positive integer n. The limit of a sequence of real numbers a1 , a2 , . . . is deﬁned
to exist17 and equal some number b if for each δ > 0, there is an f (δ ) such that b − an  ≤ δ
©
™
for all n ≥ f (δ ). For the limit here, b = 0, an = Pr Sn /n − X  ≥ ≤ and, from (1.53),
f (δ ) can be taken to satisfy δ = σ 2 /(f (δ )≤2 ). Thus we see that (1.52) simply spells out the
deﬁnition of the limit in (1.51).
One might reasonably ask at this point what (1.52) adds to the more speciﬁc statement in
σ2
(1.53). In particular (1.53) shows that the function f (≤, δ ) in (1.52) can be taken to be ≤2 δ ,
17 Exercise 1.9 is designed to help those unfamiliar with limits to understand this deﬁnition. 30 CHAPTER 1. INTRODUCTION AND REVIEW OF PROBABILITY 1 ✻
✏
✮
✏ ✏ FSn /n
1−δ ✲ ✛2≤ δ= σ2
n≤2 ❄
0 X Figure 1.9: Approximation of the distribution function FSn /n of a sample average by
a step function at the©mean. From (1.51), the probability that Sn /n diﬀers from X by
™
more than ≤ (i.e., Pr Sn /n − X  ≥ ≤ ) approaches 0 with increasing n. This means
that the limiting distribution function goes from 0 to 1 as x goes from X − ≤ to X + ≤.
Since this is true for every ≤ > 0 (presumably with slower convergence as ≤ gets smaller),
FSn /n approaches a unit step at X . Note the rectangle of width 2≤ and height 1 − δ in
the ﬁgure. The meaning of (1.52) is that the distribution function of Sn /n enters this
rectangle from below and exits from above, thus approximating a unit step. Note that
there are two ‘fudge factors’ here, ≤ and δ and, since we are approximating an entire
distribution function, neither can be omitted, except by going to a limit as n → 1. which provides added information on the rapidity of convergence. The reason for (1.52) is
that it remains valid when the theorem is generalized. For variables that are not IID or
have an inﬁnite variance, Eq. (1.53) is no longer necessarily valid.
For the Bernoulli process of Example 1.3.1, the binomial PMF can be compared with the
upper bound in (1.55), and in fact the example in Figure 1.8 uses the binomial PMF. Exact
calculation, and bounds based on the binomial PMF, provide a tighter bound than (1.55),
but the result is still proportional to σ 2 , 1/n, and 1/≤2 . 1.4.3 Relative frequency We next show that (1.52) and (1.51) can be applied to the relative frequency of an event
as well as to the sample average of a random variable. Suppose that A is some event in
a single experiment, and that the experiment is independently repeated n times. Then, in
the probability model for the n repetitions, let Ai be the event that A occurs at the ith
trial, 1 ≤ i ≤ n. The events A1 , A2 , . . . , An are then IID. If we let IAi be the indicator rv for A on the nth trial, then the rv Sn = IA1 + IA2 + · · · + IAn
is the number of occurences of A over the n trials. It follows that
Pn
IA
Sn
relative frequency of A =
= i=1 i .
(1.54)
n
n
Thus the relative frequency of A is the sample average of IA and, from (1.52),
Pr { relative frequency of A − Pr {A} ≥ ≤} ≤ σ2
,
n≤2 (1.55) 1.4. THE LAWS OF LARGE NUMBERS 31 where σ 2 is the variance of IA , i.e., σ 2 = Pr {A} (1 − Pr {A}).
The main point here is everything we learn about sums of IID rv’s applies equally to the
relative frequency of IID events. In fact, since the indicator rv’s are binary, everything
known about the binomial PMF also applies. Relative frequency is extremely important,
but we need not discuss it much in what follows, since it is simply a special case of sample
averages. 1.4.4 The central limit theorem The law of large numbers says that Sn /n is close to X with high probability for large n,
but this most emphatically does not mean that Sn is close to nX . In fact, the standard
√
deviation of Sn is σ n , which increases with n. This leads us to ask about the behavior of
√
Sn / n, since its standard deviation is σ and does not vary with n. The celebrated central
limit theorem answers this question by stating that if σ 2 < 1, then for every real number
y,
∑Ω
æ∏ Z y
µ 2∂
Sn − nX
1
−x
√
√ exp
lim Pr
≤y
=
dx.
(1.56)
n→1
2
nσ
2π
−1
The quantity on the right side of (1.56) is the distribution function of a normalized Gaussian
rv, which is known to have mean 0 and variance 1. The sequence of rv’s Zn = (Sn −
√
nX )/( n σ ) on the left also have mean 0 and variance 1 for all n. The central limit theorem,
as expressed in (1.56), says that the sequence of distribution functions, FZ1 (y ), FZ2 (y ), . . .
converges at each value of y to FΦ (y ) as n → 1, where FΦ (y ) is the distribution function
on the...
View
Full
Document
This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.
 Spring '09
 R.Srikant

Click to edit the document details