4
Stat102 Lecture - Spring 2012
Chapter 3.1 3.2:
Introduction to regression analysis
Linear regression as a descriptive technique
The least-squares equations
Chapter 3.3
Sampling distribution of b0, b1.
Continued in next lecture
1
Regression Analysis
Galtons classic data on heights of parents and their child
(952 pairs)
Describes the relationship between childs height (y)
and the parents (mid)height (x).
Predict the childs height given parents height.
75
73
71
child ht
Parent ht Child ht
73.60
72.22
72.69
67.72
72.85
70.46
71.68
65.13
70.62
61.20
70.23
63.10
70.74
64.96
70.73
66.43
69.47
63.10
68.26
62.00
65.88
61.31
64.90
61.36
64.80
61.95
64.21
64.96
And more
69
67
65
63
61
63 64 65 66 67 68 69 70 71 72 73 74
parent ht
2
Uses of Regression Analysis
Description: Describe the relationship
between a dependent variable y (childs
height) and explanatory variables x
(parents height).
Prediction: Predict dependent variable y
based on explanatory variables x.
3
Model for Simple Regression
Consider a population of units on which the
variables (y,x) are recorded.
Let y|x denote the conditional mean of y given x.
The goal of regression analysis is to estimate y|x .
Simple linear regression model:
y|x 0 1x
4
Simple Linear Regression Model
Model (more details later)
y 0 1x e
y = dependent variable
x = independent variable
0 = y-intercept
y x
1 = slope of the line
e = error (normally distributed)
y x 0 1 x
0
0 and 1 are unknown population
parameters, therefore are estimated
from the data.
Rise
1 = Rise/Run
Run
x
5
Interpreting the Coefficients
e.g.,for each extra inch for
parents, the average heights of
the child increases by 0.6 inch.
The intercept is the estimated
mean of y for x=0.
However, when working from data this
interpretation should only be used when the
data contains observations with x near 0.
Otherwise it is an extrapolation of the model,
and so can be unreliable (Section 3.7.2).
73
71
child ht
The slope 1 is the change
in the mean of y that is
associated with a one unit
change in x
75
69
67
65
63
61
63 64 65 66 67 68 69 70 71 72 73 74
parent ht
child ht = 26.46 + 0.6 parent ht
EXAMPLE
6
Least Squares Regression Line
What is a good estimate of the line?
A good estimated line should predict y well based
on x.
We use
Least squares regression line: Line that minimizes the
squared prediction errors in the sample. Good criterion
and easy to compute. Has other nice mathematical
properties.
7
The Least Squares (Regression) Line
;
Here is a scatterplot of data (n = 4) with two possible lines of fit:
Y = X and Y = 2.5 (a horizontal line).
Which is a better fit?
Sum of squared differences = (2 - 1)2 + (4 - 2)2 + (1.5 - 3)2 + (3.2 - 4)2 = 7.89
Sum of squared differences = (2 -2.5)2 + (4 - 2.5)2 + (1.5 - 2.5)2 + (3.2 - 2.5)2 = 3.99
(2,4)
w
4
3
2.5
2
w (4,3.2)
(1,2) w
w (3,1.5)
1
1
2
3
4
The smaller the sum of
squared differences
the better the fit of the
line to the data.
8
The Estimated Coefficients
To calculate the estimates of the
coefficients of the line that minimizes the
sum of the squared differences between
the data points and the line, use the
formulas:
n
b1
(x
i
i 1
x )( yi y )
The regression equation that estimates
the equation of the simple linear regression
model is:
y b 0 b1x
n
( xi x ) 2
i 1
b0 y b1 x
9
Example Heights (cont.)
For simple linear regression analysis in JMP:
Click Analyze, Fit Y by X; then put child ht in
Y and parent ht in X and click OK.
Then click red triangle next to Bivariate Fit and
click Fit Line.
Some commands we will use later can now be
found in the red triangle next to Linear Fit
10
Example: Heights (cont)
Based on our observations, find b1 and b0
The summary statistics for parent hts and child hts:
Child hts
Parent hts
Mean
Std Dev
Std Err Mean
upper 95% Mean
lower 95% Mean
N
68.20
2.60
0.084
68.37
68.04
952
Mean
Std Dev
Std Err Mean
upper 95% Mean
lower 95% Mean
N
68.27
1.79
0.0580
68.38
68.15
952
For the regression line
From JMP :
b1 = 0.61
b0 = Y - b1 X = 68.20 - 0.61 68.27 = 26.55
The LS equation is
y = 26.55 + 0.61x
11
JMP Output
Bivariate Fit of child ht By parent ht
75
73
child ht
71
69
67
65
63
61
63 64 65 66 67 68 69 70 71 72 73 74
parent ht
Linear Fit
child ht = 26.456 + 0.612 parent ht
12
JMP Output (cont)
Note the values of b0, b1 in the parameter estimates table
The other output entries will be explained later
Summary of Fit
RSquare
RSquare Adj
Root Mean Square Error
Mean of Response
Observations
Source
Model
Error
C. Total
DF
1
950
951
Term
Intercept
parent ht
0.177
0.176
2.357
68.202
952
Analysis of Variance
Sum of Squares Mean Square F Ratio
1136.50
1136.50
204.59
5277.28
5.56
Prob > F
6413.78
<.0001
Parameter Estimates
Estimate Std Error t Ratio Prob>|t|
26.456 (b0) 2.920
9.06
<.0001
0.612 (b1)
0.043
14.30 <.0001
13
Ordinary Linear Model Assumptions
Properties of errors under ideal model:
y|x 0 1x for all x.
yi 0 1 xi ei for all xi
The distribution of ei | xi is normal.
e1,, en are independent.
and Var (ei | xi ) e 2
E (ei | xi ) 0
Equivalent definition: For each xi , yi has a normal
0 1xi and variance e 2 .
distribution with mean
Also, y1,, yn are independent.
14
Sampling Distribution of b0 ,b1
The sampling distribution of b0 , b1 is the probability
distribution of the estimates over repeated samples y1 ,.., yn
from the ideal linear regression model with fixed values of
0, 1 and e2 and x1 ,.., xn .
Standardregression.jmp contains a simulation of pairs
x1 , y1 ,.., xn , yn from a simple linear regression model with
0 1, 1 2, e2 1. AND
It contains another simulation labeled x1 , y1 ,.., xn , yn from
the same model.
Notice difference the in the estimated coefficients calculated
from the ys and from the y*s.
15
Bivariate Fit of y By x
Bivariate Fit of y* By x
4.0
5
3.5
4
3.0
3
y*
y
2.5
2.0
2
1.5
1.0
1
0.5
0
0.0
.0
.2
.4
.6
x
Linear Fit
y = 1.200 + 1.521 x
.8
1.0
.0
.2
.4
.6
.8
1.0
x
Linear Fit
y* = 1.851 + 0.512 x
Two outcomes from standardregression.jmp
Each data set comes from the model with b 0 = 1, b1 = 2, s e2 = 1
The values of x1 , ..., x20 are the same in both data sets .
See next lecture for more plots and simulations
16
Sampling Distribution (Formulas)
b0 and b1 have easily described normal distributions
Sampling distribution of b0 is normal with
E b0 0
(Hence the estimate is unbiased)
1
x2
1
2
where sx
Var b0 e2
xi x 2
2
n 1 i
n n 1 sx
Sampling distribution of b1 is normal with
E b1 1
Var b1
(Hence the estimate is unbiased)
e2
n 1 sx2
17
Typical Regression Analysis
1. Observe pairs of data (x1,y1),,(xn,yn) that are a
sample from population of interest.
2. Plot the data.
3. Assume simple linear regression model
assumptions hold.
4. Estimate the true regression line y|x 0 1x by
the least squares line y|x b0 b1x
5. Check whether the assumptions of the ideal
model are reasonable (Chapter 6, and next lecture)
6. Make inferences concerning coefficients 0 , 1
and make predictions ( y b0 b1x )
18
Notes
Formulas for the least squares equations:
1. The equations for b0 and b1 are easy to derive. Here is a derivation that involves
a little bit of calculus:
It is desired to minimize the sum of squared errors. Symbolically, this is
2
SSE b0 , b1 yi b0 b1 xi .
i
The minimum occurs when 0
SSE b0 , b1 and 0
SSE b0 , b1 .
b1
b0
Hence we need
SSE b0 , b1 2 xi yi b0 b1 xi and
b1
0
SSE b0 , b1 2 yi b0 b1 xi .
b0
These are two linear equations in the two unknowns b0 and b1 . Some algebraic
manipulation shows that the solution can be written in the desired form
xi x yi y and b y b x .
b1
0
1
2
xi x
0
19
A NICE FACT thats sometimes useful:
a. The least squares line passes through the point x , y .
To see this note that if x x then the corresponding point on the least squares line is
y b0 b1 x . Substituting the definition of b0 yields y y b1 x b1 x y , as claimed.
b. The equation for the least squares line can be re-written in the form
y y b1 x x .
3. There are other useful ways to write the equations for b0 and b1 . Recall that
the sample covariance is defined as
1
Cov xi , yi
xi x yi y S xy , say.
n 1 i
Similarly, the sample correlation coefficient is
S xy
R , say.
22
Sx S y
2.
2
2
2
[ S x sx is defined on overhead 18, and S y is defined similarly.]
Thus,
S xy S y S xy
Sy
b1 2
R.
22
Sx Sx Sx S y Sx
20
History of Galtons Data:
4. Francis Galton gathered data about heights of parents and their children, and
published the analysis in 1886 in a paper entitled Regression towards mediocrity [sic] in
hereditary stature. In the process he coined the term Regression to describe the
straight line that summarizes the type of relational data that may appear in a scatterplot.
He did not use our current least-squares technique for finding this line; instead he
used a clever analysis whose final step is to fit the line by eye. He estimated the slope of
the regression line as 2 3
Further work in the next decades by Galton and by K. Pearson, Gossett ( writing as
A. Student) and others connected Galtons analysis to the least squares technique
earlier invented by Gauss (1809), and also derived the relevant sampling distributions
needed to create a statistical regression analysis.
5. The data we use for our analysis is packaged with the JMP program disk.
It is not exactly Galtons original data. We believe it is a version of the data set prepared
by S. Stigler (1986) as a minor modification of Galtons data. In order for the data to plot
nicely, Stigler jittered the data. He also included some data that Galton did not. The
data listed as Parent height in this data set is actually the average of both parents
heights, after adjusting the mothers heights as discussed in the next note.
21
6. Galton did not know how to separately treat mens and womens heights in
order to produce the kind of results he wanted to look at. SO (after looking at the
structure of the data) he multiplied all female heights by 1.08. This puts all the heights
on very nearly the same scale, and allowed him to treat mens and womens heights
together, without regard to sex.
[Instead of doing this Galton could have divided the mens heights by 1.08; or he
could have achieved a similar effect by dividing the male heights by 1.04 and
multiplying the female ones by 1.04. Why didnt he use one of these other schemes?]
7. Galton did not use modern random-sampling methods to obtain his data.
Instead, he obtained his data through the offer of prizes for the best extracts from their
own family records obtained from individual family correspondents. He summarized the
data in a journal that is now in the Library of the University College of London. Here is
what the first half of p. 4 looks like. (According to Galtons notations one should add 60
inches to every entry in the Table.)
22
Half of p4 of Galtons Journal
(note the approximate heights for some records, and the entries tall and deformed)
This photocopy, as well as much of the above discussion is taken from Hanley, J. A. (2004), Transmuting women into men: Galtons
family data on human stature, The Amer. Statistician, 58, p237-243. Another excellent reference is Stigler, S. (1986) The English
breakthrough: Galton in The History of Statistics: The Measurement of Uncertainty before 1900, Harvard Univ. Press.
23