This preview has intentionally blurred sections. Sign up to view the full version.View Full Document
Unformatted text preview: Consider the sequence of sample averages formed by taking the average of progres- sively more X i ’s. The first entry in the sequence is ¯ X 1 , which is simply equal to X 1 . The next entry is ¯ X 2 , which is the average of X 1 and X 2 . The next entry is the average of the first three X i ’s, and so forth. Continuing in this way, we obtain an infinite sequence of sample averages: ¯ X 1 , ¯ X 2 , ¯ X 3 ,... . The law of large numbers tells us that our sequence of sample averages converges in probability to μ . That is, ¯ X n → p μ as n → ∞ . This is one of the two most important results in all of probability and statistics; the other is the central limit theorem, which we shall come to later. The law of large numbers is used to justify the practice of estimating a population average ( μ ) using a sample average ( ¯ X n ). If our sample size n is large, then we must be a “long way along” the sequence ¯ X 1 , ¯ X 2 , ¯ X 3 ,... . Since this sequence is converging in probability to μ , our sample average ¯ X n should be close to μ with high probability. The key to proving the law of large numbers is a fundamental result in prob- ability theory called Chebyshev’s inequality . Chebyshev’s inequality says that, for any random variable X and any strictly positive number ε > 0, we have P ( | X | > ε ) ≤ 1 ε 2 E ( X 2 ) . Let us prove Chebyshev’s inequality in the special case where X is a continuous random variable with pdf f . We begin by writing P ( | X | > ε ) = P ( X <- ε ) + P ( X > ε ) = Z- ε-∞ f ( x )d x + Z ∞ ε f ( x )d x. Now, if x > ε then we must have x 2 /ε 2 > 1. Therefore, Z ∞ ε f ( x )d x ≤ Z ∞ ε x 2 ε 2 f ( x )d x. Similarly, if x <- ε then x 2 /ε 2 > 1, and so Z- ε-∞ f ( x )d x ≤ Z- ε-∞ x 2 ε 2 f ( x )d x. Combining these two equalities with our expression for P ( | X | > ε ) yields P ( | X | > ε ) ≤ Z- ε-∞ x 2 ε 2 f ( x )d x + Z ∞ ε x 2 ε 2 f ( x )d x. 11 We can make the right-hand side of this inequality larger by including the integral between- ε and ε as an extra term. This yields P ( | X | > ε ) ≤ Z- ε-∞ x 2 ε 2 f ( x )d x + Z ε- ε x 2 ε 2 f ( x )d x + Z ∞ ε x 2 ε 2 f ( x )d x = Z ∞-∞ x 2 ε 2 f ( x )d x = 1 ε 2 Z ∞-∞ x 2 f ( x )d x = 1 ε 2 E ( X 2 ) . This completes the proof of Chebyshev’s inequality for the special case where the random variable X is continuous. A similar proof may be used in the discrete case. Along with Chebyshev’s inequality, we shall employ a few basic facts about the expected value and variance to prove the law of large numbers. You should remember these facts from your previous coursework. Let X and Y be arbitrary random variables, and let c be a constant. Then: 1. E ( cX ) = cE ( X ). 2. E ( X + Y ) = E ( X ) + E ( Y ). 3. Var( cX ) = c 2 Var( X ). 4. Var( X + Y ) = Var( X ) + Var( Y ) + 2Cov( X,Y )....
View Full Document
- Spring '08
- Normal Distribution, Probability theory, probability density function