70 CS Fall 2010
Discrete Mathematics and Probability Theory Tse/Wagner Lecture 17
Polling and the Law of Large Numbers
Polling
Question: We want to estimate the proportion p of Democrats in the US population, by taking a small random sample. How large does our sample have to be to guarantee that our estimate will be within (say) 0.1 of the true value with probability at least 0.95? This is perhaps the most basic statistical estimation problem, and it shows up everywhere. We will develop a simple solution that uses only Chebyshevs inequality. More rened methods can be used to get sharper results. Lets denote the size of our sample by n (to be determined), and the number of Democrats in it by the random variable Sn . (The subscript n just reminds us that the r.v. depends on the size of the sample.) Then 1 our estimate will be the value An = n Sn . Now as has often been the case, we will nd it helpful to write Sn = X1 + X2 + + Xn , where Xi = 1 0 if person i in sample is a Democrat; otherwise.
Note that each Xi can be viewed as a coin toss, with Heads probability p (though of course we do not know the value of p). And the coin tosses are independent.1 Hence, Sn is a binomial random variable with parameters n and p. What is the expectation of our estimate?
1 E(An ) = E( n Sn ) = 1 E(Sn ) = 1 (np) = p. n n
So for any value of n, our estimate will always have the correct expectation p. [Such a r.v. is often called an unbiased estimator of p.] Now presumably, as we increase our sample size n, our estimate should get more and more accurate. This will show up in the fact that the variance decreases with n: i.e., as n increases, the probability that we are far from the mean p will get smaller. To see this, we need to compute Var(An ). But An = 1 Sn , which is just a constant times a binomial random n variable. Theorem 17.1: For any random variable X and constant c, we have Var(cX ) = c2 Var(X ).
are assuming here that the sampling is done with replacement; i.e., we select each person in the sample from the entire population, including those we have already picked. So there is a small chance that we will pick the same person twice.
1 We
CS 70, Fall 2010, Lecture 17
1
The proof of this theorem follows directly from the denition of the variance. (Try it yourself.) Now to compute Var(An ): n 2 Var(An ) = Var( 1 Sn ) = ( 1 )2 Var(Sn ) = ( 1 )2 Var(Xi ) = , n n n n i=1 where we have written 2 for the variance of each of the Xi . The third equality follows from the calculation we did for the binomial random variable in the last lecture note. So we see that the variance of An decreases linearly with n. This fact ensures that, as we take larger and larger sample sizes n, the probability that we deviate much from the expectation p gets smaller and smaller. Lets now use Chebyshevs inequality to gure out how large n has to be to ensure a specied accuracy in our estimate of the proportion of Democrats p. A natural way to measure this is for us to specify two parameters, and , both in the range (0, 1). The parameter controls the error we are prepared to tolerate in our estimate, and controls the condence we want to have in our estimate. A more precise version of our original question is then the following: Question: For the Democrat-estimation problem above, how large does the sample size n have to be in order to ensure that Pr[|An p| ] ? In our original question, we had = 0.1 and = 0.05. Lets apply Chebyshevs inequality to answer our more precise question above. Since we know Var(An ), this will be quite simple. From Chebyshevs inequality, we have Pr[|An p| ] Var(An ) 2 = 2. 2 n
To make this less than the desired value , we need to set n 2 1 2 . (1)
Now recall that 2 = Var(Xi ) is the variance of a single sample Xi . So, since Xi is a 0/1-valued r.v., we have 2 = p(1 p), and inequality (1) becomes n p(1 p) 1 . 2 (2)
Plugging in = 0.1 and = 0.05, we see that a sample size of n = 2000 p(1 p) is sufcient. At this point you should be worried. Why? Because our formula for the sample size contains p, and this is precisely the quantity we are trying to estimate! But we can get around this. The largest value possible of p(1 p) is 1/4 (achieved when p = 1/2.) Hence, if we pick n = 2000 (1/4) = 500, then no matter what the value of p is, the sample size is sufcient.
Estimating a general expectation
What if we wanted to estimate something a little more complex than the proportion of Democrats in the population, such as the average wealth of people in the US? Then we could use exactly the same scheme as above, except that now the r.v. Xi is the wealth of the ith person in our sample. Clearly E(Xi ) = , the 1 average wealth (which is what we are trying to estimate). And our estimate will again be An = n n=1 Xi , for i a suitably chosen sample size n. We again have E(An ) = . And as long as Var(n 1 Xi ) = n=1 Var(Xi ) , we = i
CS 70, Fall 2010, Lecture 17 2
have as before Var(An ) = , where 2 = Var(Xi ) is the common variance of the Xi s. From equation (1), it n is enough for the sample size n to satisfy 1 n 2 2 . (3) Here and are the desired error and condence respectively, as before. Now of course we dont know 2 , appearing in equation (3). In practice, we would use an upper bound on 2 (just as we used the fact that 2 = p(1 p) 1/4 in the Democrats problem). Plugging these bounds into equation (3) will ensure that our sample size is large enough. Let us recapitulate the three properties we used about the random variables Xi s in the above derivation: (1) E(Xi ) = , i = 1, . . . , n. (2) Var(Xi ) = 2 , i = 1, . . . , n. (3) Var(n=1 Xi ) = n=1 Var(Xi ). i i The rst two properties hold if the Xi s have the same distribution. The third property holds if the random variables are mutually independent. We have already dened the notion of independence for events, and we will dene independence for random variables in the next lecture note. Intuitively, two random variables are independent if the events "associated" with them are independent. In the polling example when the Xi s are indicator random variables for ipping Heads, the random variables are independent if the ips are independent. Random variables which have the same distribution and are independent are called independent identically distributed (abbreviated as i.i.d.). As a further example, suppose we are trying to estimate the average rate of emission from a radioactive source, and we are willing to assume that the emissions follow a Poisson distribution with some unknown parameter of course, this is precisely the expectation we are trying to estimate. Now in this case we 2 1 have = and also 2 = (see the previous lecture note). So 2 = . Thus in this case a sample size of n=
1 2
2
sufces. (Again, in practice we would use a lower bound on .)
The Law of Large Numbers
The estimation method we used in the previous two sections is based on a principle that we accept as part of everyday life: namely, the Law of Large Numbers (LLN). This asserts that, if we observe some random variable many times, and take the average of the observations, then this average will converge to a single value, which is of course the expectation of the random variable. In other words, averaging tends to smooth out any large uctuations, and the more averaging we do the better the smoothing. Theorem 17.2: [Law of Large Numbers] Let X1 , X2 , . . . , Xn be i.i.d. random variables with common expectation = E(Xi ). Dene An = 1 n=1 Xi . Then for any > 0, we have ni Pr [|An | ] 0 as n .
Proof: Let Var(Xi ) = 2 be the common variance of the r.v.s; we assume that 2 is nite2 . With this (relatively mild) assumption, the LLN is an immediate consequence of Chebyshevs Inequality. For, as we 2 have seen above, E(An ) = and Var(An ) = , so by Chebyshev we have n Pr [|An | ]
2 If
Var(An ) 2 = 0 2 n 2
as n .
2 is not nite, the LLN still holds but the proof is much trickier. 3
CS 70, Fall 2010, Lecture 17
This completes the proof. 2 Notice that the LLN says that the probability of any deviation from the mean, however small, tends to zero as the number of observations n in our average tends to innity. Thus by taking n large enough, we can make the probability of any given deviation as small as we like.
CS 70, Fall 2010, Lecture 17
4