70 CS Fall 2010
Discrete Mathematics and Probability Theory Tse/Wagner Note 19
A Brief Introduction to Continuous Probability
Up to now we have focused exclusively on discrete probability spaces , where the number of sample points is either nite or countably innite (such as the integers). As a consequence we have only been able to talk about discrete random variables, which take on only a nite (or countably innite) number of values. But in real life many quantities that we wish to model probabilistically are real-valued; examples include the position of a particle in a box, the time at which a certain incident happens, or the direction of travel of a meteorite. In this lecture, we discuss how to extend the concepts weve seen in the discrete setting to this continuous setting. As we shall see, everything translates in a natural way once we have set up the right framework. The framework involves some elementary calculus.
Continuous uniform probability spaces
Suppose we spin a wheel of fortune and record the position of the pointer on the outer circumference of the wheel. Assuming that the circumference is of length and that the wheel is unbiased, the position is presumably equally likely to take on any value in the real interval [0, ]. How do we model this experiment using a probability space? Consider for a moment the (almost) analogous discrete setting, where the pointer can stop only at a nite number m of positions distributed evenly around the wheel. (If m is very large, then presumably this is in some sense similar to the continuous setting.) Then we would model this situation using the discrete sample 1 space = {0, m , 2 , . . . , (m1) }, with uniform probabilities Pr[ ] = m for each . In the continuous m m world, however, we get into trouble if we try the same approach. If we let range over all real numbers in [0, ], what value should we assign to each Pr[ ]? By uniformity this probability should be the same for all , but then if we assign to it any positive value, the sum of all probabilities Pr[ ] for will be ! Thus Pr[ ] must be zero for all . But if all of our sample points have probability zero, then we are unable to assign meaningful probabilities to any events! To rescue this situation, consider instead any non-empty interval [a, b] [0, ]. Can we assign a non-zero probability value to this interval? Since the total probability assigned to [0, ] must be 1, and since we want our probability to be uniform, the logical value for the probability of interval [a, b] is length of [a, b] b a = . length of [0, ] In other words, the probability of an interval is proportional to its length. Note that intervals are subsets of the sample space and are therefore events. So in continuous probability, we are assigning probabilities to certain basic events, in contrast to discrete probability, where we assigned probability to points in the sample space. But what about probabilities of other events? Actually, by specifying the probability of intervals we have also specied the probability of any event E which can be written as the disjoint union of (a nite or countably innite number of) intervals, E = i Ei . For then we can write
CS 70, Fall 2010, Note 19 1
Pr[E ] = i Pr[Ei ], in analogous fashion to the discrete case. Thus for example the probability that the pointer ends up in the rst or third quadrants of the wheel is /4 + /4 = 1 . For all practical purposes, such events 2 are all we really need.1
Continuous random variables
Recall that in the discrete setting we typically work with random variables and their distributions, rather than directly with probability spaces and events. The simplest example of a continuous random variable is the position X of the pointer in the wheel of fortune, as discussed above. This random variable has the uniform distribution on [0, ]. How, precisely, should we dene the distribution of a continuous random variable? In the discrete case the distribution of a r.v. X is described by specifying, for each possible value a, the probability Pr[X = a]. But for the r.v. X corresponding to the position of the pointer, we have Pr[X = a] = 0 for every a, so we run into the same problem as we encountered above in dening the probability space. The resolution is essentially the same: instead of specifying Pr[X = a], we instead specify Pr[a X b] for all intervals [a, b].2 To do this formally, we need to introduce the concept of a probability density function (sometimes referred to just as a density, or a pdf). Denition 19.1 (Density): A probability density function for a random variable X is a function f : R R satisfying
b
Pr[a X b] =
a
f (x)dx
for all a b.
Lets examine this denition. Note that the denite integral is just the area under the curve f between the values a and b. Thus f plays a similar role to the histogram we sometimes draw to picture the distribution of a discrete random variable. In order for the denition to make sense, f must obey certain properties. Some of these are technical in nature, which basically just ensure that the integral is always well dened; we shall not dwell on this issue here since all the densities that we will meet will be well behaved. What about some more basic properties of f ? First, it must be the case that f is a non-negative function; for if f took on negative values we could nd an interval in which the integral is negative, so we would have a negative probability for some event! Second, since the r.v. X must take on some value everywhere in the space, we must have
f (x)dx = Pr[ < X < ] = 1.
(1)
In other words, the total area under the curve f must be 1. A caveat is in order here. Following the histogram analogy above, it is tempting to think of f (x) as a probability. However, f (x) doesnt itself correspond to the probability of anything! For one thing, there is no requirement that f (x) be bounded by 1 (and indeed, we shall see examples of densities in which f (x) is greater than 1 for some x). To connect f (x) with probabilities, we need to look at a very small interval [x, x + ] close to x; then we have
x+
Pr[x X x + ] =
x
f (z)dz f (x).
(2)
1 A formal treatment of which events can be assigned a well-dened probability requires a discussion of measure theory, which is beyond the scope of this course. 2 Note that it does not matter whether or not we include the endpoints a, b; since Pr[X = a] = Pr[X = b] = 0, we have Pr[a < X < b] = Pr[a X b].
CS 70, Fall 2010, Note 19
2
Thus we can interpret f (x) as the probability per unit length in the vicinity of x. Now lets go back and put our wheel-of-fortune r.v. X into this framework. What should be the density of X ? Well, we want X to have non-zero probability only on the interval [0, ], so we should certainly have f (x) = 0 for x < 0 and for x > . Within the interval [0, ] we want the distribution of X to be uniform, which means we should take f (x) = c for 0 x . What should be the value of c? This is determined by the requirement (1) that the total area under f is 1. The area under the above curve is f (x)dx = 0 cdx = c , 1 so we must take c = . Summarizing, then, the density of the uniform distribution on [0, ] is given by 0 f (x) = 1/ 0 for x < 0; for 0 x ; for x > .
Expectation and variance of a continuous random variable
By analogy with the discrete case, we dene the expectation of a continuous r.v. as follows: Denition 19.2 (Expectation): The expectation of a continuous random variable X with probability density function f is
E(X ) =
x f (x)dx.
Note that the integral plays the role of the summation in the discrete formula E(X ) = a a Pr[X = a]. Example: Let X be a uniform r.v. on the interval [0, ]. Then E(X ) = This is certainly what we would expect! We will see more examples of expectations of continuous r.v.s in the next section. Since variance is really just another expectation, we can immediately port its denition to the continuous setting as well: Denition 19.3 (Variance): function f is The variance of a continuous random variable X with probability density
2
0
x2 1 x dx = 2
0
=. 2
Var(X ) = E((X E(X ))2 ) = E(X 2 ) E(X )2 =
x2 f (x)dx
x f (x)dx
.
Example: Lets calculate the variance of the uniform r.v. X on the interval [0, ]. From the above denition, and plugging in our previous value for E(X ), we get Var(X ) =
0
1 x3 x2 d x E(X )2 = 3
2
2
2
2
0
2
=
3
4
=
12
.
2
1 The factor of 12 here is not particularly intuitive, but the fact that the variance is proportional to come as no surprise. Like its discrete counterpart, this distribution has large variance.
should
CS 70, Fall 2010, Note 19
3
An application: Buons needle
Here is a simple application of continuous random variables to the analysis of a classical procedure for estimating the value of known as Buffons needle, after its 18th century inventor Georges-Louis Leclerc, Comte de Buffon. Here we are given a needle of length , and a board ruled with horizontal lines at distance apart. The experiment consists of throwing the needle randomly onto the board and observing whether or not it crosses one of the lines. We shall see below that (assuming a perfectly random throw) the probability of this event is exactly 2/ . This means that, if we perform the experiment many times and record the proportion of throws on which the needle crosses a line, then the Law of Large Numbers (Lecture Note 17) tells us that we will get a good estimate of the quantity 2/ , and therefore also of ; and we can use Chebyshevs inequality as in the other estimation problems we considered in that same Lecture Note to determine how many throws we need in order to achieve specied accuracy and condence. To analyze the experiment, we rst need to specify the probability space. Lets go straight to random variables. Note that the position where the needle lands is completely specied by two random variables: the vertical distance Y between the midpoint of the needle and the closest horizontal line, and the angle between the needle and the vertical. The r.v. Y ranges between 0 and /2, while ranges between /2 and /2. Since we assume a perfectly random throw, we may assume that their joint distribution has density f (y, ) that is uniform over the rectangle [0, /2] [ /2, /2]. Since this rectangle has area , the density 2 should be 2/ for (y, ) [0, /2] [ /2, /2]; f (y, ) = (3) 0 otherwise. As a sanity check, lets verify that the integral of this density over all possible values is indeed 1:
/2 /2
f (y, )dyd =
/2 0
2 d yd =
/2 /2
2y
/2 0
1 d = d = /2
/2
/2
= 1.
/2
This is an analog of equation (1) for our joint distribution; rather than the area under the simple curve f (x), we are now computing the area under the surface f (y, ). But of course the result should again be the total probability mass, which is 1. Now let E denote the event that the needle crosses a line. How can we express this event in terms of the values of Y and ? Well, by elementary geometry the vertical distance of the endpoint of the needle from its midpoint is 2 cos , so the needle will cross the line if and only if Y 2 cos . Therefore we have
/2 ( /2) cos
Pr[E ] = Pr[Y 2 cos ] =
f (y, )dyd .
/2 0
Substituting the density f (y, ) from equation (3) and performing the integration we get
/2 ( /2) cos
Pr[E ] =
/2 0
2 d yd =
/2 /2
2y
( /2) cos
d =
0
1
/2
cos d =
/2
1 2 /2 [sin ] /2 = .
This is exactly what we claimed at the beginning of the section!
Joint distribution and independence for continuous random variables
In analyzing the Buffons needle problem, we used the notion of joint density of two continuous random variables without formally dening the concept. But its denition should be obvious at this point:
CS 70, Fall 2010, Note 19 4
Denition 19.4 (Joint Density): A joint density function for two random variable X and Y is a function f : R2 R satisfying
d b
Pr[a X b, c Y d ] =
c a
f (x, y)dxdy
for all a b and c d .
So in the previous example, a uniform joint density on the rectangle [0, /2] [ /2, /2] simply means that the probability of landing in any small rectangle inside that rectangle is proportional to the area of the small rectangle. In analogy with (2), we can connect f (x, y) with probabilities by looking at a very small square [x, x + ] [y, y + ] close to (x, y); then we have
y+ x+ x
Pr[x X x + , y Y y + ] =
y
f (u, v)dudv 2 f (x, y).
(4)
Thus we can interpret f (x, y) as the probability per unit area in the vicinity (x, of y). Recall that in discrete probability, two r.v.s X and Y are said to be independent if the events X = a and Y = c are independent for every a, c. What about for continuous r.v.s? Denition 19.5 (Independence for Continuous R.V.s): Two continuous r.v.s X , Y are independent if the events a X b and c Y d are independent for all a, b, c, d . What does this denition say about the joint density of independent r.v.s X and Y ? Applying (4) to connect the joint density with probabilities, we get, for small : 2 f (x, y) Pr[x X x + , y Y y + ] = Pr[x X x + ] Pr[y Y y + ] by independence f1 (x) f2 (y) = 2 f1 (x) f2 (y), where f1 and f2 are the (marginal) densities of X and Y respectively. So we see that f (x, y) = f1 (x) f2 (y), i.e., for independent r.v.s the joint density is the product of the marginal densities (cf. the discrete case, where joint distributions are the product of the marginals). In the Buffons needle problem, it is easy to go back and check that Y and are independent r.v.s, each of which is uniformly distributed in its respective range.
Two more important continuous distributions
We have already seen one important continuous distribution, namely the uniform distribution. In this section we will see two more: the exponential distribution and the normal (or Gaussian) distribution. These three distributions cover the vast majority of continuous random variables arising in applications. Exponential distribution: The exponential distribution is a continuous version of the geometric distribution, which we have already met. Recall that the geometric distribution describes the number of tosses of a coin until the rst Head appears; the distribution has a single parameter p, which is the bias (Heads probability) of the coin. Of course, in real life applications we are usually not waiting for a coin to come up Heads but rather waiting for a system to fail, a clock to ring, an experiment to succeed etc. In such applications we are frequently not dealing with discrete events or discrete time, but rather with continuous time: for example, if we are waiting for an apple to fall off a tree, it can do so at any time at
CS 70, Fall 2010, Note 19 5
all, not necessarily on the tick of a discrete clock. This situation is naturally modeled by the exponential distribution, dened as follows: Denition 19.6 (Exponential distribution): For any > 0, a continuous random variable X with pdf f given by e x if x 0; f (x) = 0 otherwise. is called an exponential random variable with parameter . Like the geometric, the exponential distribution has a single parameter , which characterizes the rate at which events happen. We shall illuminate the connection between the geometric and exponential distributions in a moment. First, lets do some basic computations with the exponential distribution. We should check rst that it is a valid distribution, i.e., that it satises (1):
f (x)dx =
0
e x dx = e x
0
= 1,
as required. Next, what is its expectation? We have
E(X ) =
x f (x)dx =
0
xe
x
dx =
xe x 0 +
e
0
x
e x dx = 0 +
=
0
1 ,
where for the rst integral we used integration by parts. To compute the variance, we need to evaluate E(X 2 ) =
x2 f (x)dx =
0
x2 e x dx = x2 e x
+ 0
0
2xe x dx = 0 +
2 2 E(X ) = 2 ,
where again we used integration by parts. The variance is therefore Var(X ) = E(X 2 ) E(X )2 = 2 1 1 2 = 2. 2
Let us now explore the connection with the geometric distribution. Note rst that the exponential distribution satises, for any t 0,
Pr[X > t ] =
t
e x dx = e x
t
= e t .
(5)
In other words, the probability that we have to wait more than time t for our event to happen is e t , which is an exponential decay with rate . Now consider a discrete-time setting in which we perform one trial every seconds (where is very small in fact, we will take 0 to make time continuous), and where our success probability is p = . Making the success probability proportional to makes sense, as it corresponds to the natural assumption that there is a xed rate of success per unit time, which we denote by = p/ . The number of trials until we get a success has the geometric distribution with parameter p, so if we let the r.v. Y denote the time (in seconds) until we get a success we have Pr[Y > k ] = (1 p)k = (1 )k Hence, for any t > 0, we have
t Pr[Y > t ] = Pr[Y > ( ) ] = (1 )t / e t ,
for any k 0.
CS 70, Fall 2010, Note 19
6
where this nal approximation holds in the limit as 0 with = p/ xed. (We are ignoring the detail t of rounding to an integer since we are taking an approximation anyway.) Comparing this expression with (5) we see that this distribution has the same form as the exponential distribution with parameter , where (the success rate per unit time) plays an analogous role to p (the probability of success on each trial)though note that is not constrained to be 1. Thus we may view the exponential distribution as a continuous time analog of the geometric distribution. Normal Distribution: The last continuous distribution we will look at, and by far the most prevalent in applications, is called the normal or Gaussian distribution. It has two parameters, and . Denition 19.7 (Normal distribution): For any and > 0, a continuous random variable X with pdf f given by 2 2 1 f (x) = e(x ) /2 2 2 is called a normal random variable with parameters and . In the special case = 0 and = 1, X is said to have the standard normal distribution. A plot of the pdf f reveals a classical bell-shaped curve, centered at (and symmetric around) x = , and with width determined by . (The precise meaning of this latter statement will become clear when we discuss the variance below.) Lets run through the usual calculations for this distribution. First, lets check equation (1):
1 f (x)dx = 2 2
e(x )
2 /2 2
dx = 1.
(6)
The fact that this integral evaluates to 1 is a routine exercise in integral calculus, and is left as an exercise (or feel free to look it up in any standard book on probability or on the internet). What are the expectation and variance of a normal r.v. X ? Lets consider rst the standard normal. By denition, its expectation is E(X ) = 1 x f (x)dx = 2
xex
2 /2
1 dx = 2
2
0
xex
2 /2
dx +
0
xex
2 /2
dx = 0.
The last step follows from the fact that the function ex /2 is symmetrical about x = 0, so the two integrals are the same except for the sign. For the variance, we have
2 1 Var(X ) = E(X 2 ) E(X )2 = x2 ex /2 dx 2 2 1 1 = xex /2 + 2 2 2 /2 1 = 0+ ex dx = 1. 2
ex
2 /2
dx
In the rst line here we used the fact that E(X ) = 0; in the second line we used integration by parts; and in the last line we used (6) in the special case = 0, = 1. So the standard normal distribution has expectation E(X ) = 0 = and variance Var(X ) = 1 = 2 . Now suppose X has normal distribution with general parameters , . We claim that the r.v. Y = the standard normal distribution. To see this, note that 1 Pr[a Y b] = Pr[ a + X b + ] = 2 2
CS 70, Fall 2010, Note 19
b+ a+ X
has
e(x )
2 /2 2
1 dx = 2
b a
ey
2 /2
dy,
7
by a simple change of variable in the integral. Hence Y is indeed standard normal. Note that Y is obtained from X just by shifting the origin to and scaling by . (And we shall see in a moment that is the mean and the standard deviation, so this operation is very natural.) Now we can read off the expectation and variance of X from those of Y . For the expectation, using linearity, we have X E(X ) 0 = E(Y ) = E , = and hence E(X ) = . For the variance we have 1 = Var(Y ) = Var and hence Var(X ) = 2 . The bottom line, then, is that the normal distribution has expectation and variance 2 . (This explains the notation for the parameters , .) The fact that the variance is 2 (so that the standard deviation is ) explains our earlier comment that determines the width of the normal distribution. Namely, by Chebyshevs inequality, a constant fraction of the distribution lies within distance (say) 2 of the expectation . Note: The above analysis shows that, by means of a simple origin shift and scaling, we can relate any normal distribution to the standard normal. This means that, when doing computations with normal distributions, its enough to do them for the standard normal. For this reason, books and online sources of mathematical formulas usually contain tables describing the density of the standard normal. From this, one can read off the corresponding information for any normal r.v. X with parameters , 2 , from the formula Pr[X a] = Pr[Y where Y is standard normal. Some basic facts that are useful to know, when dealing with a normal distribution: A normally distributed r.v. falls within of the mean about 68% of the time. (i.e., Pr[ X + ] 0.68) A normally distributed r.v. falls within 2 of the mean about 95% of the time. (i.e., Pr[ 2 X + 2 ] 0.95) A normally distributed r.v. falls within 3 of the mean about 99.7% of the time. (i.e., Pr[ 3 X + 3 ] 0.997) The normal distribution is ubiquitous throughout the sciences and the social sciences, because it is the standard model for any aggregate data that results from a large number of independent observations of the same random variable (such as the heights of females in the US population, or the observational error in a physical experiment). Such data, as is well known, tends to cluster around its mean in a bell-shaped curve, with the correspondence becoming more accurate as the number of observations increases. A theoretical explanation of this phenomenon is the Central Limit Theorem, which we next discuss.
a ],
X
=
Var(X ) , 2
CS 70, Fall 2010, Note 19
8
The Central Limit Theorem
Recall from Lecture Note 17 the Law of Large Numbers for i.i.d. r.v.s Xi s: it says that the probability of 1 any deviation of the sample average An := n n=1 Xi from the mean, however small, tends to zero as the i number of observations n in our average tends to innity. Thus by taking n large enough, we can make the probability of any given deviation as small as we like. Actually we can say something much stronger than the Law of Large Numbers: namely, the distribution of 2 the sample average An , for large enough n, looks like a normal distribution with mean and variance . n (Of course, we already know that these are the mean and variance of An ; the point is that the distribution becomes normal!) The fact that the standard deviation decreases with n (specically, as n ) means that the distribution approaches a sharp spike at . Recall from the last section that the density of the normal distribution is a symmetrical bell-shaped curve centered around the mean . Its height and width are determined by the standard deviation as follows: the height at the mean is about 0.4/ ; 50% of the mass is contained in the interval of width 0.67 either side of the mean, and 99.7% in the interval of width 3 either side of the mean. (Note that, to get the correct scale, deviations are on the order of rather than 2 .) To state the Central Limit Theorem precisely (so that the limiting distribution is a constant rather than something that depends on n), we shift the mean of An to 0 and scale it so that its variance is 1, i.e., we replace An by (An ) n n=1 Xi n An = =i . n The Central Limit Theorem then says that the distribution of An converges to the standard normal distribution. Theorem 19.1: [Central Limit Theorem] Let X1 , X2 , . . . , Xn be i.i.d. random variables with common exn 1 Xi n pectation = E(Xi ) and variance 2 = Var(Xi ) (both assumed to be < ). Dene An = i= n . Then as n , the distribution of An approaches the standard normal distribution in the sense that, for any real , 1 Pr[An ] 2
ex
2 /2
dx
as n .
The Central Limit Theorem is a very striking fact. What it says is the following. If we take an average of n observations of absolutely any r.v. X , then the distribution of that average will be approximately a bellshaped curve centered at = E(X ). Thus all trace of the distribution of X disappears as n gets large: all distributions, no matter how complex,3 look like the normal distribution when they are averaged. The only effect of the original distribution is through the variance 2 , which determines the width of the curve for a given value of n, and hence the rate at which the curve shrinks to a spike.
3 We
do need to assume that the mean and variance of X are nite. 9
CS 70, Fall 2010, Note 19