APPLIED MATH602: STATISTICS Winter 2000
Dr. Srinivas R. Chakravarthy Department of Industrial and Manufacturing Engineering & Business Kettering University (Formerly GMI Engineering & Management Institute) Flint, MI 48504-4898
Phone: [phone]; FAX: [phone]
E-mail: [e-mail] Homepage: http://www.kettering.edu/~schakrav
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 1
Normal (Gaussian) distribution: Characterized by two parameters: location and shape. The PMF is given by
1 f ( x) = e 2 ( x )2 2 2
, < x < ,
where the location parameter is nothing but the mean and the shape parameter is the standard deviation of X.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 2
Bivariate Random variables So far we looked at one random variable at a time. But in many situations we are interested in observing two or more characteristics simultaneously. Brittle hardness index as well as the tensile strength of a piece of steel wire; current flow as well as the temperature.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 3
When dealing with more than one variable, it is of interest to look at the relationship between two or more varibles. Covariance and Correlation are measures used in the study of linear relationship between two variables. Covariance between X and Y, Cov(X,Y) or XY, is XY = E(XY) - X Y
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 4
Covariance is measurements.
sensitive
to
the
units
of
Correlation between X and Y, Cov(X,Y) or XY, is XY = XY /XY -1 XY 1. Note that if X and Y are independent then XY = 0; however if XY = 0 doesnt imply that X and Y are independent.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 5
ESTIMATION THEORY
Referring back to one of the problems dealing with MPG of a new model car, suppose that of the 36 cars tested randomly, the sample yielded a mean of 24.5 MPG. What would be our estimate (based on this sample) for the true (population) mean ?
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 6
This is called a point estimate and is based only on one sample. Will we get the same estimate when we take another sample (of same size)? This leads to interval estimate. What can we say about the error term? (size) This depends on: (a) the sample size: the larger the sample size the better the estimate, the smaller the error and the greater the precision; (b) the
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 7
variability: smaller the variance the better the estimate, the smaller the error and the greater the precision; and (c) the level of confidence: Going back to the car MPG problem the true value of is always a fixed numerical value at any given time. We do not know exactly what this value and so we take a random sample size of 36 so that we
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 8
can specify an interval within which we want to be "reasonably confident" that will lie. By reasonably confident we mean a level of confidence anywhere from 90% to 98% (usually 95%). Note that 100% confidence will be useless for all practical purposes. [Why?].
Srinivas R. Chakravarthy
MATH602: LECTURE 3 (WINTER 2000)
9
In estimation theory we are primarily concerned with finding point and interval estimators for parameters of interest. An estimator is a statistic used to estimate the parameter under study. An estimate, which is a specific value of the estimator, is obtained from the sample taken from the population under study.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 10
There are certain properties that we would like our estimator to possess. We shall see only two properties that we will be using for the remainder of the class. Unbiased estimator: if its mean equals the parameter being estimated. Efficient estimator: if it has the least variance among all estimators of the parameter. Also referred to as a minimum variance estimator.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 11
An estimator that has both the properties is referred to as an UMV (unbiased minimum variance) estimator. Point estimator being a random variable is distributed around the true value of the parameter and so the point estimate obtained from the estimator cannot be taken to be the value of the parameter.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 12
So in order for us to be reasonably confident of including the true value of the parameter, we need to construct an interval. This interval to which we can attach some level of confidence is known as confidence interval. The construction of such an interval requires the knowledge of the sampling distribution of the estimator of the parameter under study.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 13
There are several methods of estimation that are available to find, say, an UMV estimator for a given parameter. Method of Maximum Likelihood, Method of Moments, Method of Minimum variance and Method of Minimum Chi-square. The discussion of these methods falls under the general topic of Mathematical Statistics.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 14
SAMPLING DISTRIBUTIONS: These are nothing but probability distributions of estimators (random variables obtained as functions of observations taken from the population under study). These play an important role in statistical inference, namely in constructing confidence intervals and in tests of hypotheses. One of the most celebrated results in probability and statistics is the Central Limit Theorem.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 15
CENTRAL LIMIT THEOREM
This result has found applications in many fields. The history of the central limit theorem is fascinating and the interested reader will find the book, "The Life and Times of the Central Limit Theorem" by William J. Adams very informative and interesting.
Srinivas R. Chakravarthy
MATH602: LECTURE 3 (WINTER 2000)
16
A great many random phenomena that arise in physical situations result from the combined actions of many individual events. Some examples of this nature are: shot noise from electrons, holes in a vacuum tube or transistor, atmospheric noise, turbulence in a medium, thermal agitation of electrons in a conductor, ocean waves, fluctuations in stock market share values, and many other sources of random disturbances.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 17
Suppose we denote by X1, X2,..., Xn the random variables arising out of n situations and by Sn = X1 + ... + Xn, the combined action of these n situations. Individually the random variables X1, X2,..., Xn may not contribute significantly to Sn but collectively the contribution may be significant. The central limit theorem is concerned with conditions under which the random variable Sn suitably normalized will converge to a standard normal distribution.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 18
Central limit theorem: Suppose that {Xk} is a sequence of mutually independent random variables. Suppose that these random variables all have a common distribution with a finite mean and a finite variance 2. If Sn = X1 + X2 + ... + Xn, then we have
1 Sn n x P 2 n
Srinivas R. Chakravarthy
e
x
z2 / 2
dz, as n
19
MATH602: LECTURE 3 (WINTER 2000)
REMARKS: (1) The above statement can be rephrased as
X n 1 x P 2 / n
e
x
z2 / 2
dz, as n
(2) Note that we need only and . (3) The underlying distribution can be either discrete or continuous.
Srinivas R. Chakravarthy
MATH602: LECTURE 3 (WINTER 2000)
20
EXAMPLE: The thickness X (measured in inches) of cold rolled carbon steel shaft is a continuous random variable with a mean of 0.20 inch and a standard deviation of 0.005 inch. If sixty-four measurements on the thickness of steel shaft are taken randomly and independently, find the probability that the mean of these 64 measurements will be at least 0.202 inch.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 21
Linear Combination of normal random variables
Suppose that X1, , Xk are k independent normal random variables with E(XI) = i and V(Xi) = i2. Then any linear combination, Y, of Xis: Y = a1X1 + + ak Xk Follows a normal distribution with
k k = a and 2 = a 2 2. i i Y Y i =1 i i =1 i
Srinivas R. Chakravarthy
MATH602: LECTURE 3 (WINTER 2000)
22
ESTIMATION THEORY Referring back to the horsepower problem, suppose that 16 cars tested (at 4500 RPM) randomly, the sample yielded a mean of 253.25 hp. What would be our estimate (based on this sample) for the true (population) mean ? This is called a point estimate and is based only on one sample.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 23
Will we get the same estimate when we take another sample (of same size)? This leads to the concept of an interval estimate. What can we say about the error term? That is what can we say about the size of the error? This depends on: (a) the sample size: the larger the sample size the better the estimate, the smaller the error and the greater the precision; (b) the variability in the HP; the smaller the variance the better the estimate, the smaller the error and the
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 24
greater the precision; and (c) the level of confidence: This is better understood through an example. Going back to the car HP problem the true value of is always a fixed numerical value at any given time. We do not know exactly what this value and so we take a random sample size of 16 so that we can specify an interval within which we want to be reasonably confident that will lie.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 25
By reasonably confident we mean a level of confidence anywhere from 90% to 98% (usually 95%). Note that 100% confidence will be useless for all practical purposes. [Why?]. In estimation theory we are primarily concerned with finding point and interval estimators for parameters of interest. An estimator is a statistic used to estimate the parameter under study.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 26
An estimate, which is a specific value of the estimator, is obtained from the sample taken from the population under study. There are certain properties that we would like our estimator to possess. We shall see only two properties that we will be using for the remainder of the class. Unbiased estimator: An estimator is said to be unbiased if its mean equals the parameter being estimated.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 27
Efficient estimator: An estimator is said to be efficient among all estimators of the parameter if it has the least variance. This is also referred to as a minimum variance estimator. An estimator that has both the properties is referred to as an UMV (unbiased minimum variance) estimator. Point estimator being a random variable is distributed around the true value of the parameter
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 28
and so the point estimate obtained from the estimator cannot be taken to be the value of the parameter. So in order for us to be reasonably confident of including the true value of the parameter, we need to construct an interval. This interval to which we can attach some level of confidence is known as confidence interval.
Srinivas R. Chakravarthy
MATH602: LECTURE 3 (WINTER 2000)
29
The construction of such an interval requires the knowledge of the sampling distribution of the estimator of the parameter under study. There are several methods of estimation that are available to find an UMV estimator for a given parameter. Some of these are: Method of Maximum Likelihood, Method of Moments, Method of Minimum variance and Method of Minimum Chi-square.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 30
CONFIDENCE INTERVALS The fundamental idea in the construction of a confidence interval is as follows: suppose that is the parameter under study. Then given a 100(1-)% level, compute the statistics L and U such that P(L U) = (1-). That is, we determine L and U from the sample such that 100(1-)% confidence we can say that the true value of will lie between L and U.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 31
Confidence interval for : In the large sample case using the Central Limit Theorem, 100(1-)% confidence interval for is seen to be
X z / 2 / n ,
X + z / 2 / n
[If is unknown then we use the sample standard deviation, s].
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 32
It is important to know how to interpret this interval. With 100(1-)% confidence we can say that the true mean will lie in the above interval. If 100 samples of fixed size n are taken, then at least 100(1-) of the 100 sample means computed will lie in the above interval. The maximum error involved in estimating :
Srinivas R. Chakravarthy
MATH602: LECTURE 3 (WINTER 2000)
33
HorsePower (HP) example (contd) Confidence Intervals
The assumed sigma = 10.0
Variable N hp@4500 16 hp@5500 16 Mean StDev 253.25 13.51 241.06 23.16 SEMean 95.0 % CI 2.50 ( 248.35, 258.15) 2.50 ( 236.16, 245.96)
Srinivas R. Chakravarthy
MATH602: LECTURE 3 (WINTER 2000)
34
STUDENTS t DISTRIBUTION
Referring to HP example, we assumed that the population standard deviation was known (to be 10). However, in practice, it is usually unknown. Hence, we need to estimate it first. If the sample size is reasonably large (n 30), we can still use the normal distribution for inferential part (as justified by the CLT). What happens if the sample is small (n < 30)?
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 35
In this case we cannot use normal since the sample size is small and by using the sample standard deviation to estimate s, we bring in more variability into the picture the and appropriate distribution to use is the student's t-distribution. In 1908, William S.Gosset, a chemist working for a brewery company, under the pseudonym Student, first deduced this distribution. Students t-distribution, like normal, is bell-shaped. It depends on the sample size. It is more spread than
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 36
normal and approaches normal as n approaches infinity. So in the case when n is small, is unknown and with the assumption that the population is approximately normal, 100(1-)% C.I for is given by
X t / 2 s / n ,
Srinivas R. Chakravarthy
X + t / 2 s / n
37
Note that the assumption of the population being approximately normal is important.
MATH602: LECTURE 3 (WINTER 2000)
If there is a reason to believe that this assumption is not valid in any given problem, then one has to rely on nonparametric methods.
HP Example (contd):
Confidence Intervals
Variable N Mean StDev hp@4500 16 253.25 13.51 hp@5500 16 241.06 23.16 SE Mean 95.0 % CI 3.38 (246.05, 260.45) 5.79 (228.72, 253.40)
Note that in constructing the above confidence interval, we assumed that the populations (for 4500 RPM and 5500 RPM) are normal.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 38
How do we verify that the assumptions are not grossly violated? Before we see the answer to this question, we need to review another pillar of the statistical inference, namely, tests of hypotheses.
TESTS OF HYPOTHESES
In this branch of statistical inference we test the claim (or hypothesis) about the population parameter(s) under study.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 39
Simple: A simple hypothesis is one in which the parameter is completely specified. Composite: A hypothesis, which is not simple, is called a composite hypothesis. Null hypothesis (usually this is the claimed one) and an alternative hypothesis. Setting up null (H0) and alternative hypotheses (H1), is an important step in tests of hypotheses.
Srinivas R. Chakravarthy
MATH602: LECTURE 3 (WINTER 2000)
40
The probability of committing a type 1 error is denoted by and is referred to as the level of significance or tolerance level. The probability of committing a type 2 error is denoted by . Given (otherwise we report a probability called a p-value) to compute a specific value of the parameter under H1 should be given.
Srinivas R. Chakravarthy
MATH602: LECTURE 3 (WINTER 2000)
41
We have control over only on (because we can test the hypotheses at a very low level of significance if the problem on hand dictates) and because of this we set up H0 and H1 in such a way that the costly error is treated as a type 1 error. Note that setting up a two-sided (that is H1 is H1: 0) hypotheses is no problem. When is not given we report a probability value called a p-value; this is the probability, obtained under the assumption that H0 is true, of randomly
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 42
obtaining a test statistic value more extreme than that calculated from the sample. If this p-value is small, we reject H0. In other words, p-value gives the minimum level at which to reject H0. Equivalently this gives the maximum level at which not to reject H0. After setting up H0 and H1 the next step is to determine the appropriate statistic to test the hypotheses.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 43
The appropriate test statistic is usually based on the UMV estimator of the parameter under study. Test on Suppose we need to test H0: = 0 vs H1: 0. Sample mean is an UMV estimator for . We need to use the sampling distribution of the sample mean to test the hypotheses.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 44
In the large sample case, the decision rule (try to set up in terms of the calculated value of the test statistic) for a given is as follows. Otherwise do not reject H0 (which will be abbreviated as DNR H0). If is not specified, as mentioned before we report a probability called a p-value.
Srinivas R. Chakravarthy
MATH602: LECTURE 3 (WINTER 2000)
45
HP Example (contd):
Z-Test Test of mu = 250.00 The assumed sigma = Variable N Mean hp@4500 16 253.25 hp@5500 16 241.06 vs mu not = 250.00 10.0 StDev SE Mean Z P 13.51 2.50 1.30 0.19 23.16 2.50 -3.58 0.0004
T-Test of the Mean
Test of mu = 250.00 vs mu not = 250.00 Variable N hp@4500 16 hp@5500 16
Srinivas R. Chakravarthy
Mean StDev SE Mean T 253.25 13.51 3.38 0.96 241.06 23.16 5.79 -1.54
MATH602: LECTURE 3 (WINTER 2000)
P 0.35 0.14
46
[NOTE: The discussion of the inferential statistics for the population proportion and population variance will be left to you as an exercise. Specific problems dealing with these will be assigned as homework problems]
Srinivas R. Chakravarthy
MATH602: LECTURE 3 (WINTER 2000)
47
TEST FOR NORMALITY Earlier, when we used students t-distribution or Fdistribution, we need normality assumption for the population (s) under study. How do we verify that a population is normal or at least approximately normal? There are a number of ways of testing. Some of these will be discussed here.
Srinivas R. Chakravarthy
MATH602: LECTURE 3 (WINTER 2000)
48
Method 1: (Normal plot). Standardize the sample points and order them y(1) y(2) . . . y(n), denote these ordered values. y(i) is the 100i/n-th percentile of these data points. [We use 100 (i-0.5)/n to take into account the continuity correction factor]. Plot (y(i) , 100 (i-0.5)/n %). If the plot resembles a linear function, the normal assumption is satisfied.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 49
Any serious departure from a straight-line plot indicates the violation of normal assumption. The plot can be done using MINITAB with the help of NSCORE command.
Srinivas R. Chakravarthy
MATH602: LECTURE 3 (WINTER 2000)
50
Method 2: (Goodness of fit test). Here we perform a nonparametric test using Chisquare distribution. First we group the sample data into various classes and obtain a frequency distribution. Under the assumption that population is normal with a specified mean and variance, we calculate the expected value for each of the classes. Use Chi-square statistic to test the hypotheses.
Srinivas R. Chakravarthy
MATH602: LECTURE 3 (WINTER 2000)
51
Method 3: (Anderson-Darling statistic). Let y(i) 1 i n, denote the n ordered (and standardized) sample values. Compute the cumulative probabilities zi = (y(i)), where (y(i)) is the CDF of a std. normal. Compute
2 A = ( 2 i - 1) [ ln( zi ) + ln ( 1 - z n - i + 1 ) ] i =1 - n . n
n
Compare B2 = A2 [1 + 0.75/n + 2.25/n2] to the critical values listed below.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 52
Critical values for the A-D Test- B2 B20.25 B20.20 B20.15 B20.10 0.472 0.509 0.561 0.631
2 B 0.05 2 B 0.025 2 B 0.01 2 B 0.005
0.752 0.873 1.035 1.159 If the calculated value exceeds the table value, reject the null hypothesis of normality assumption.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 53
REMEDIAL MEASURES: If normality assumption seems to be violated, then some suitable transformation on the data should be carried out to satisfy this assumption. BOX-COX Transformation
w=
, 0 x ln(x), = 0
54
Srinivas R. Chakravarthy
MATH602: LECTURE 3 (WINTER 2000)
Normal Probability Plot ( hp@4500 )
.999 .99 .95
Probability
.80 .50 .20 .05 .01 .001 235 245 255 265 275
hp@4500
Average: 253.25 StDev: 13.5130 N: 16 Anderson-Darling Normality Test A-Squared: 0.264 P-Value: 0.648
Srinivas R. Chakravarthy
MATH602: LECTURE 3 (WINTER 2000)
55
Normal Probability Plot ( hp@5500 )
.999 .99 .95
Probability
.80 .50 .20 .05 .01 .001 200 210 220 230 240 250 260 270
hp@5500
Average: 241.062 StDev: 23.1588 N: 16 Anderson-Darling Normality Test A-Squared: 0.282 P-Value: 0.589
Srinivas R. Chakravarthy
MATH602: LECTURE 3 (WINTER 2000)
56
Chi-squared distribution
Suppose that X1, , Xn are n random samples from a normal distribution with mean and standard deviation .
2 = n Z 2 is given by Then the PDF of i =1 i (1/ 2) n / 2 (n / 2) 1 x / 2 f ( x) = x e , x > 0, (n / 2) ((k ) = x k 1e x dx, for any positive number k.
0
Referred to as Chi-squared distribution with (n-1) d.f.
Srinivas R. Chakravarthy MATH602: LECTURE 3 (WINTER 2000) 57
ILLUSTRATIVE EXAMPLE Case Study (Ch...