100
Lecture STA 25
Irina Udaltsova
Department of Statistics
University of California, Davis
March 9th, 2012
Admin for the Day
Homework 7 posted, due Wednesday March 14th
Project Instructions posted, Projects due Monday, March 19th
Project Proposal are due today!
Proposal feedback will be available by Monday:
No feedback nothing wrong with your proposal
Feedback changes needed (usually minor or wording)
Oce Hours project questions please!
References for Today: Samuels 12.(1-6)
References for Monday: Samuels 12.(7-9), 11.(1-4)
Practice Problems: Samuels 12.5.(1-2), 12.5.8, 12.6.5-7
Linear Regression
We can investigate the relationship between two variables using
linear regression:
Yi N + Xi , 2
Given explanatory variable xi the expected value of Yi is
+ xi .
As we vary xi this traces out a straight-line relationship
between x and y .
Variations above and below the straight-line should be
normally distributed with variance 2 .
Note, sometimes we will use a dierent notation ( = 0 , = 1 ):
Y i = 0 + 1 x i +
i
Understanding Assumptions
Scatterplot of x and y
5
0
5
y
10
15
20
0
2
4
6
8
x
10
12
14
Conditional Distributions
Scatterplot of x and y
5
0
5
y
10
15
20
0
2
4
6
8
10
12
14
x
We can take a look at all the data points with x = 5.
0
20
40
60
80
100
Histogram of points with x=5
0
5
Y given X=5
10
Conditional Distributions
Scatterplot of x and y
5
0
5
y
10
15
20
0
2
4
6
8
10
12
14
x
We can take a look at all the data points with x = 12.
0
20
40
60
80
100
Histogram of points with x=12
6
8
10
12
14
Y given X=12
16
18
20
Conditional Distributions
Linear regression assumes that the histograms conditional on
dierent values of x will have:
Dierent means! (Mean = (x ) = + x )
The same variance!
Approximately normal distributions.
Often we dont have enough data to make those histograms for
real problems, but they help us understand what we are assuming.
Key Assumptions
There are ve very fundamental assumptions we make:
1. The mean of Y is a linear function of X :
This describes the core of the relationship between X and Y i.e.,
the straight line.
2. The (vertical) variability of the data points around the line around
follows a normal distribution:
This describes one aspect of how the points are allowed to vary
around the line.
3. The variability of the data points around the line around does not
vary as X changes (i.e., for all x it is just 2 ):
This describes another aspect of how the points are allowed to vary
around the line.
4. Conditional on X , the Y s are independent: This describes another
aspect of how the points are allowed to vary around the line.
5. The explanatory variable X is observed without error:
Everything we do here is conditional on the X s.
Diagnosing Problems
The more complex the model we build, the more things there are
that can go wrong.
For linear regression, we really to need to know what our
assumptions mean, so that we can know what warning signs to
look for. . .
Next 6 slides:
6 Datasets Which assumptions are reasonable/not for each. . . ?
This exercise is designed to give you some practice deciding when
it is appropriate to use linear regression.
Example 1
Plot of y vs. x: Are the linear regression assumptions reasonable?
6
4
2
0
2
y
4
6
8
1
2
3
4
x
5
6
Example 2
Plot of y vs. x: Are the linear regression assumptions reasonable?
6
5
4
y
3
2
1
1
2
3
4
x
5
6
Example 3
400
Plot of y vs. x: Are the linear regression assumptions reasonable?
300
200
100
y
0
100
200
1
2
3
4
x
5
6
Example 4
14
Plot of y vs. x: Are the linear regression assumptions reasonable?
12
10
y
8
6
4
0
2
1
2
3
4
x
5
6
Example 5
Plot of y vs. x: Are the linear regression assumptions reasonable?
7
6
5
y
3
4
2
1
2
3
4
x
5
6
Example 6
Plot of y vs. x: Are the linear regression assumptions reasonable?
150
100
50
0
50
100
150
y
1
2
3
4
x
5
6
Quiz Time. . .
For each of the 6 plots, either one or none of the linear
regression assumptions are violated.
For each of the 6 plots, choose only one of the following:
(A) All assumptions appear to be valid
(B) The linearity assumption appears to be wrong
(C) The constant variance assumption appears to be wrong
(D) The normality assumption appears to be wrong
(E) The conditional independence assumption appears wrong
Note: For some of the plots it is not possible to distinguish between some of
the possible violations, so there is no way to know the correct answer. This is
the same in real-life we cannot always pin down why things go wrong.
Therefore it does not necessarily matter which assumptions you think are
wrong only whether or not you not thought they were all valid.
(1)
(2)
0
y
4
2
1
1
2
3
4
5
6
1
2
3
4
10
6
150
12
7
50
0
2
2
3
4
x
(4)
5
6
1
2
y
1
50
4
100
150
4
2
3
6
0
y
y
5
8
6
Plot of y vs. x: Are the linear regression assumptions reasonable?
5
x
100
14
Plot of y vs. x: Are the linear regression assumptions reasonable?
6
Plot of y vs. x: Are the linear regression assumptions reasonable?
x
x
200
6
8
5
4
3
2
y
3
2
300
1
200
100
2
y
0
6
Plot of y vs. x: Are the linear regression assumptions reasonable?
100
5
4
4
6
(3)
Plot of y vs. x: Are the linear regression assumptions reasonable?
400
Plot of y vs. x: Are the linear regression assumptions reasonable?
3
4
x
(5)
5
6
1
2
3
4
x
(6)
5
6
Solutions. . .
(1)
(2)
Mean of y = a + bx
0
y
4
2
1
2
3
4
5
6
1
2
3
Example where independence is not true
Mean of y = a + bx
Mean of y = a + bx
7
12
10
6
0
2
2
3
4
x
(4)
5
6
1
2
y
1
50
4
100
150
4
2
3
6
0
y
y
50
5
8
6
Example where all assumptions are correct
5
Mean of y = a + bx
4
x
1
6
Example where normality is not true (harder to see)
x
x
14
200
6
5
4
Mean of y = a + bx + cx^2 + dx^3
3
2
y
8
200
3
Mean of y = a + bx
300
100
2
1
y
0
2
100
150
6
Example where constant variance is not true
100
5
4
4
6
(3)
Example where all assumptions are correct
400
Example where linearity is not true
3
4
x
(5)
5
6
1
2
3
4
x
(6)
5
6
Diagnostics
Lessons learned:
1. We can usually tell whether anything has gone wrong (we will
talk about formal tests soon)
2. It can be hard to tell exactly what has gone wrong (e.g., could
be non-linearity, non-independence etc.)
3. There are certain key things we can learn to look for (linearity,
spread of points above and below the line shouldnt get
drastically bigger or smaller as you move along the x-axis)
Linear Regression: Overview
What is linear regression?
1. Conceptually: Method for exploring linear relationships
between explanatory variables and a continuous response.
2. Formally: Assumptions
When is it appropriate?
1. Eyeball test, transformations
2. Diagnostic plots
What do the results mean?
1.
2.
3.
4.
Interpreting the regression coecients
Hypothesis testing for the regression coecients
Prediction and condence intervals
R 2 , goodness-of-t test
Brain-Body Example
Example: Lets dig deeper into the brain-body weight example.
Yi = log-brain weight of species i
Xi = log-body weight of species i
We assume that:
Y i | X i = xi N + xi , 2 .
Another way of writing the same thing:
Y i = + xi + i ,
i
iid
N 0, 2 .
Perspective
Issues and Problems
To keep you thinking about the big picture (as per exams,
homework, projects). . .
What potential problems are there with the brain-body weight
example?
What is the meaning of each pair, (xi , yi ), of data points?
Each species doesnt have one brain weight and body weight!
(We use the median brain and median body weight of a
sample from each species implications?).
How was the data collected?
Not a random sample.
(Still representative enough?)
Other problems?
Understanding Statistical Modeling
Is our mathematical/statistical description of the brain-body
weight problem a perfect one?
No. Does it need to be?
All models are wrong. Some are useful.
George E.P. Box, Statistician
Least Squares
How do we estimate and ?
In other words: how do we select the best straight line?
The criteria we use is something called maximum likelihood. It is
fairly intuitive: choose the values of and that correspond to
the model your data were most likely to have come from.
In the case of linear regression it actually has a nice simple
explanation. . .
10
Plot of log(Brain) vs. log(Body)
5
0
log(brain)
0
5
log(body)
Which line is better?
10
How do we decide on a best line?
Well, we want it to be close to the data points.
For a given estimate (, ), the vertical distance between the line
and data point is given by:
i = yi ( + xi )
This is positive if yi is above the line
This is negative if yi is below the line
This is zero if the line passes through yi
We call the residual for point i
10
Plot of log(Brain) vs. log(Body)
5
0
log(brain)
0
5
log(body)
10
Least Squares
The residuals will be positive and negative, but we want them to be as
close to zero as possible (i.e., we want the line to be close to the data
points).
We need a criteria by which to select the best and .
As usual, we square everything and add up the squared dierences (where
have we seen this before?):
n
SS (, ) =
i =1
[yi ( + xi )]
2
This is known as the residual sums of squares (RSS). We want to select
and that minimize the residual sums of squares.
n
SS (, ) =
i =1
yi ( + xi )
2
n
2
i
=
i =1
10
Plot of log(Brain) vs. log(Body)
5
log(brain)
0
0
5
10
log(body)
Goal: Find the line that minimizes the sums of the squared vertical
distances to the data points.
10
Plot of log(Brain) vs. log(Body)
5
log(brain)
0
0
5
10
log(body)
Equivalent Goal: Find the line that minimizes the sums of the
areas of the squares.
10
Plot of log(Brain) vs. log(Body)
5
log(brain)
0
0
5
log(body)
Solution: This is the best line.
10
The Line of Best Fit
Finding the values and that minimize the RSS is actually not
too dicult. We can write down the solutions analytically.
First, we need some notation: x and y are the mean of x and y .
The estimates of and are:
=
n
i =1 (yi y )(xi
n
2
i =1 (xi x )
x)
= y x
We also get an estimate of 2 (called the residual mean square):
2 =
1
n2
n
2
i
i =1
The estimate is known as the residual standard error.
Doing Linear Regression
To do this in R you simply click:
Statistics > Fit models > Linear regression
Select the appropriate response and explanatory variable.
Click ok.
To plot the regression line:
Graphs > Scatterplot
Select the appropriate response (y-axis) and explanatory
variable (x-axis). Uncheck all boxes apart from Least-squares
line. Click ok.
Animals: log(body) vs. log(brain) weight
8
6
4
2
0
log(brain)
0
5
log(body)
10
Doing Linear Regression
For the brain-body weight example we get:
> RegModel.2 <- lm(log(brain)~log(body), data=animals)
> summary(RegModel.2)
Call:
lm(formula = log(brain) ~ log(body), data = animals)
Residuals:
Min
1Q
-3.1609 -0.9360
Median
0.3074
3Q
0.7832
Max
3.0206
Coefficients:
Estimate Std. Error t value
(Intercept) 2.99064
0.47051
6.356
log(body)
0.43580
0.08751
4.980
--Signif. codes: 0 *** 0.001 ** 0.01
Pr(>|t|)
1.18e-06 ***
3.93e-05 ***
* 0.05 . 0.1 1
Residual standard error: 1.655 on 25 degrees of freedom
Multiple R-squared: 0.498, Adjusted R-squared: 0.4779
F-statistic: 24.8 on 1 and 25 DF, p-value: 3.926e-05
Reading the Output
First part of output:
Call:
lm(formula = log(brain) ~ log(body), data = animals)
This tells you the model you t (written y x where the lhs is the
response, and the rhs is the explanatory variable).
Next part of the output:
Residuals:
Min
1Q
-3.1609 -0.9360
Median
0.3074
3Q
0.7832
Max
3.0206
These are the residuals from earlier (i.e., the vertical distances
between the points and the line):
i = yi ( + xi )
Furthest points from the line were 3.16 below and 3.02 above,
The median distance of the points the line was 0.30 above,
The lower and upper quartiles of the distances from the line were 0.94
below, and 0.78 above.
Reading the Output
Next part of the output are the estimates for and :
Coefficients:
Estimate Std. Error t value
(Intercept) 2.99064
0.47051
6.356
log(body)
0.43580
0.08751
4.980
--Signif. codes: 0 *** 0.001 ** 0.01
Pr(>|t|)
1.18e-06 ***
3.93e-05 ***
* 0.05 . 0.1 1
The (Intercept) line refers to i.e., = 2.99
The log(body) line refers to i.e., = 0.44
The other columns we will get to shortly.
Most of the last part we will get to on Monday:
Residual standard error: 1.655 on 25 degrees of freedom
Multiple R-squared: 0.498, Adjusted R-squared: 0.4779
F-statistic: 24.8 on 1 and 25 DF, p-value: 3.926e-05
Residual standard error:
1.665 means = 1.665.
Recap: Parameters vs. Statistics
The most important slide in the whole course:
Parameters: Population quantities (what we care about)
Statistics: Sample quantities (what we actually get)
Really understanding the dierences between estimands and
estimators, and their properties is key to understanding many
statistical methods.
Estimating Population Quantities
For the brain-body weight example, we are interested in the
relationship between body and brain weights of all species (on the
log scale).
Parameter of interest:
, the population regression slope.
Statistic used to estimate this:
, the estimated slope from the sample data.
We use statistics to estimate parameters.
i.e., we use data to estimate unknown quantities of interest.
Interpretation
We have:
E [Yi |Xi = xi ] = 2.99 + 0.44xi
The expected log-brain weight of an animal with log-body weight xi is
2.99 + 0.44xi .
What do and mean?
The Intercept, :
Interpretation: The intercept gives the expected value of
response if the explanatory variable is equal to zero.
The Slope, :
Interpretation: For a one unit increase in x , our expected
value of y increases by .
Example: = 0.44.
For each additional 1kg of log body weight, we expect the log
brain weight of an animal to increase by 0.44g.
Fitted Values
Earlier we talked about residuals (i.e., the vertical distances between the
points and the line):
i = yi ( + xi )
We can rewrite this as:
yi =
yi =
Data Point =
We assumed that
( + xi )
yi
Fitted Value
+
+
+ Residual
iid
i
N (0, 2 ).
There should be no relationship between tted values & residuals!
We can make a scatterplot to check this!
We can also make a plot to check normality (Q-Q Plot)
Diagnostic Plots in R
For linear regression, R provides four plots to check your assumptions:
1. Residuals vs. Fitted Values:
What it checks: Linearity and constant variance assumptions.
There should be no pattern in the plot.
2. Normal Q-Q Plot:
What it checks: Whether the residuals are normally distributed.
The points should fall close to a straight line with slope 1.
3. Scale-Location Plot:
What it checks: Whether the variance of residuals changes with x .
The height of the points should be fairly stable as x increases.
4. Residuals vs. Leverage:
What it checks: Whether there are any individual points that have a
particularly large eect on the regression estimates.
The points should fall inside the red dotted lines.
Diagnostic Plots in R
After you have t the linear regression, these plots can be
obtained by:
Models > Graphs > Basic diagnostic plots
Alternatively, you can simply type:
plot(my.lm)
where my.lm is the name of the linear regression you just
performed.
Diagnostic Plots: Pass or Fail?
Normal QQ
2
Residuals vs Fitted
1
3
6
2
3
4
5
6
7
25
1
1
8
6
2
1
0.4
2
3
4
5
Fitted values
2
6
7
8
2
0.5
1
0
1
1
Standardized residuals
2
1.2
0.8
0.0
Standardized residuals
25
0
Residuals vs Leverage
6
1
Theoretical Quantiles
ScaleLocation
25
Fitted values
2
Residuals
1 0
0
2
1
Standardized residuals
3
1
0.00
0.05
0.10
20
25
Cook's distance6
0.15
Leverage
Q: What happens if the diagnostic plots look bad?
Dont just carry on! Need to do something else!
0.20
0.5
Linear Regression
Q: So far, we provided estimates for and . What is missing?
A measure of uncertainty!
As usual we can make condence intervals for and to express
the amount of uncertainty in our estimates.
Doing Linear Regression
For the brain-body weight example we get:
> RegModel.2 <- lm(log(brain)~log(body), data=animals)
> summary(RegModel.2)
[snipped]
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 2.99064
0.47051
6.356 1.18e-06 ***
log(body)
0.43580
0.08751
4.980 3.93e-05 ***
--Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1
[snipped]
= 2.99. How certain are we?
= 0.4358. How certain are we?
Confidence Intervals
Condence intervals for the intercept and slope are:
A 100(1 )% CI for is:
tn2,1 SE ( ),
2
+ tn2,1 SE ( )
2
A 100(1 )% CI for the intercept is:
tn2,1 SE (),
2
+ tn2,1 SE ( )
2
Confidence Intervals Example
Example: 95% for in the animals example:
(Intercept)
log(body)
Estimate Std. Error t value Pr(>|t|)
2.99064
0.47051
6.356 1.18e-06 ***
0.43580
0.08751
4.980 3.93e-05 ***
= 0.43580, SE ( ) = 0.08751, n = 27.
Find tn2,0.975 from table or R:
> qt(0.975,df=25)
[1] 2.059539
Thus, our CI is:
tn2,1 SE ( ), + tn2,1 SE ( )
2
2
= (0.43580 2.059539 0.08751, 0.43580 + 2.059539 0.08751)
= (0.2556, 0.6160) .
Confidence Intervals
Since condence intervals are very useful for linear regression, R will make
them for you.
After you have run the regression using:
Statstics > Fit models > Linear regression
Then go to:
Models > Confidence Intervals
and select the desired condence level (e.g., 0.95).
Example:
> Confint(RegModel.1, level=.95)
Estimate
2.5 %
97.5 %
(Intercept) 2.9906412 2.0216091 3.9596734
log.body
0.4357976 0.2555647 0.6160306
Cartoon of the Day