3 Topic Multiple Regression Analysis
Regression on Several Predictor
Variables
(Chapter 8)
1
Topic Overview
Systolic Blood Pressure Example
Multiple Regression Models
SAS Output for Regression
Multicollinearity
2
Systolic Blood Pressure Data
In this topic we will fully analyze the SBP
dataset described in Problem #5.2 in the text.
This dataset illustrates some excellent points
regarding multiple regression.
The file 03SBP.sas provides the data and all
of the code that has been utilized to produce
output shown in the CLG handout and lecture
notes.
3
Dataset Overview (n = 32)
Response Variable: systolic blood pressure
for an individual
Note: SBP is the maximum pressure exerted
when the heart contracts (top number).
Predictor Variables
Age (measured in years)
Body Size (measured using the quetelet index)
Smoking Status (0 = nonsmoker, 1 = smoker)
4
Multiple Regression Analysis
For the SBP data, our goal is to determine
whether SBP may be reasonably well
predicted by some combination of age, body
size, and smoking status.
Additionally we may want to try to describe
relationships and answer questions such as:
Does the SBP increase (on average) with an
increase in size?
5
Multiple Regression Analysis (2)
The first step in a multiple regression
analysis is to consider the individual
variables and their pairwise (SLR)
relationships.
Identify potential problems (e.g. outliers)
Identify and assess the form, direction, and
strength of pairwise relationships.
6
CLG Activity #1
Please discuss questions 3.1-3.6
from the handout.
Additional slides will be made
available on the website after we
have done the activities in class.
7
Multiple Regression Analysis (3)
The next step in multiple regression analysis
is to consider using more than one predictor
variable in the same model.
You can think of adding variables to the
model in a certain order. Each variable
takes up a portion of the total sums of
squares.
8
Graphical View of MLR
Rectangle represents total SS; Ovals
represent variables; Note OVERLAP!
X1
X3
X2
Total SSY
9
Some Key Points
Must take into account relationships
(correlation) among the potential predictor
variables these relationships are
responsible for the overlap in explained SS.
Interaction between predictors may also
become a consideration. Interaction means
that the effect of one predictor changes
depending on the value of the other. More
on this later...
10
Some Additional Concerns
Dealing with multiple predictors is much
more difficult than SLR:
More difficult to choose the best model (we
now have many more choices). Calculation
of estimates can be problematic generally
we always employ a computer.
Interpretation of the parameter estimates is
usually less clear, and can in fact be
meaningless if highly correlated variables are
used in the same model.
Harder to visualize the multi-dimensional
relationship expressed by the model.
11
Examples of Models
Suppose we have two predictors X1 and X2. Then
possible models include (but are not limited to):
Model 1: Y = 0 + 2 X 2 +
Model 2: Y = 0 + 1 X 1 + 2 X 2 +
Model 3: Y = 0 + 1 X 1 + 2 X 2 + 3 X 1 X 2 +
Model 4: Y = 0 + 1 X 1 + 2 X 2 + 3 X 1 X 2
+ 4 X 12 + 5 X 22 +
12
Best Model
The best model will contain significant
variables and exclude those that are not
significant.
BUT: Because of possible overlaps, it is not
always easy to determine which variables
should be excluded!
Use scatter plots and residual plots to check
the FORM of the model.
In scatter plots of Y-hat vs. individual predictors,
non-linear patterns indicate the need to
transform a given predictor.
13
Best Estimates
The best estimates for a MLR model are still
those that minimize the SSE (least squares)
n
(
SSE = Yi Yi
i =1
)
2
Deviations are measured as before by the
difference between the observed response
and the predicted response. The Xs come
into play only in terms of estimating the
predicted values.
14
Notation
Similar to SLR
Greek Letters represent parameters
Lower Case English represent estimates
Parameter estimates / standard error formulas
are much more complicated due to the fact we
have multiple predictors
15
Matrix Approach
Matrix approach simplifies notation
somewhat. Estimates are:
b = ( X ' X)
1
( X 'Y)
This is discussed in Appendix B, however
we will rely on SAS to do the work for us.
Our goal is to learn how to interpret the
information.
16
Slope Estimates
Some important properties of the estimates
include the following:
Each slope estimate is a linear function of the
Ys (the linear coefficients are based on the
Xs as was the case in SLR). Thus they are
random variables.
Correlation between Y and Y is maximized
(while the sum of squared error is minimized).
17
Assumptions on Errors
These are the same as before as well! A
simple statement of the assumptions is that
i ~ N ( 0,
2
)
Normality
Given a particular set of predictors, the errors
(or equivalently the Y|Xs) have a normal
distribution. This provides justification for the
use of T and F tests.
18
Assumptions on Errors (2)
Constancy of Variance
Variance of errors is the same regardless of
the values of the predictor variables.
Equivalent to say that the variance of Y given
X = (X1,X2,...,Xk) is constant.
Transformations
If needed, we can often transform Y to
achieve normality and/or constancy of
variance.
19
Assumptions on Errors (3)
Independence
Errors (and hence responses) are statistically
independent of one another.
One common violation is repeated measures
(measuring the same individual over time). If
the relationship is linear (or transformable)
then it is possible to include time in the model
as a predictor variable and avoid this issue.
20
Linear Model Assumption
Linearity means:
Mean of Y is a linear function of the Xs
Y = 0 + 1 X 1 + 2 X 2 + ... + k X k +
Can have weird terms from transformations
e.g. X 5 = X 2 X 3 or X 2 = log ( X 2 )
21
Checking Assumptions
Also very much the same as SLR
Epsilon is the error component and is
estimated by the residuals:
ei = Yi Yi
Residual plots can be considered to check
for violations of the assumptions.
May plot residuals versus each single
predictor to assess constant variance and
linearity with respect to that predictor.
22
ANOVA Table
Similar idea to simple linear regression; information
is organized into an ANOVA table.
Still have (Corrected, for the mean) Total Sums of
Squares which is split into SS for the regression
model and SS error.
n
( Y Y )
i =1
i
SSTOT
2
=
(
=
SS R
n
i =1
Yi Y
) + (
n
2
i =1
+
Yi Yi
)
2
SS E
23
Degrees of Freedom
Degrees of freedom change
Always have n 1 total degrees of freedom.
The degrees of freedom for the regression is
the number of slope estimates k (you do not
count the intercept).
Degrees of freedom for error is therefore
n 1 k.
24
Regression Sums of Squares
The SSR can be divided up into parts
according to the addition of variables into
the model.
When this is done, we have an extended
ANOVA table the model line is broken
down into one line for each variable and
each variable has 1 DF associated to it.
IMPORTANT: This breakdown is order
dependent! In SAS we will call this the
TYPE I sums of squares.
25
Graphical View of MLR
Rectangle represents total SS; Ovals represent SS explained
by the different variables. Note that there overlap, is and the
order in which variables are added is important!
SS(X1)
SS(X3|X1,X2)
Total SSY
SS(X2|X1)
26
Coefficient of Determination
R2 is still the coefficient of determination.
But it has a slightly different meaning in the
context of MLR.
R2 can be thought of as the percentage of
variation in Y (as represented by the total
SS) that is explained by the group of
predictor variables in the model.
R2 gives no indication of the importance of
any particular predictor variable.
27
MLR Hypothesis Testing: Basics
There are similarities to SLR, but
there are also some big differences.
28
Parameter Estimates
We will have a slope estimate for each
variable. Each slope estimate will also have
a standard error.
Note that in SAS, the variable name is used
to identify each line. Confidence intervals
may be requested as before by using the
CLB option in PROC REG.
29
Parameter Estimates (2)
These are joint estimates.
They assume other variables in the model.
They will all change some if you remove or
add a variable. (Note: If we are interested in
interpretation, how much the estimates
change in such a case is very important!)
A Bonferroni adjustment may be appropriate
to adjust for the fact we are computing
several intervals.
30
Some Questions of Interest
1.
Does the set of independent variables contribute
anything worthwhile to the prediction of the
response variable?
2.
Given the present model, which variable(s) add
significantly to the prediction of the response (over
and above other predictors already included)?
3.
Given the present model, is there any group of
variables which will add significantly to the
prediction of the response?
31
Answers?
We find the answers to these questions by
considering:
F-test from the ANOVA table
T-tests from the parameter estimates table
Partial F tests for comparing different models;
these are based on Type I and II SS which we
will discuss in Topic 4
32
ANOVA F-test
Null hypothesis is that ALL of the slope
parameters are zero. Written:
H 0 : 1 = 2 = L = k = 0
Alternative hypothesis is that at least one of
the slope parameters is non-zero:
H a : i 0 for some i
Rejecting the null hypothesis says There is
something in the model that is important in
explaining the variation in the response.
33
ANOVA F-test (2)
MSR
The test statistic is F =
MSE
2.
MSE estimates the variance
MSR estimates the variance under H0.
Under HA, MSR will be larger than MSE;
hence we reject H0 if F >> 1. Compare to F
statistic with DF associated to the MSR and
MSE; p-value computed by SAS.
34
ANOVA F-test (3)
Failing to reject means that there is no evidence of
association between the response variable Y and the
group of predictors that are currently in the model.
Failing to reject doesnt guarantee that no
variables in the model are important. We might be
lacking in power; we might also be missing an
important variable that would reduce the total SS.
Rejecting means that at least one predictor variable is
important in predicting Y.
Reject doesnt indicate which predictor(s) are
important.
35
T-tests
Null hypothesis for each t-test is that the slope
parameter associated with the corresponding
variable is zero. The alternative is that the
associated slope parameter is not zero.
H 0 : i = 0
vs. H a : i 0
KEY POINT: Other predictors are considered
to be already in the model. So these tests are
variable added last tests.
36
T-tests (2)
Rejection means: Even with all of the
other variables in the model and taking up
as much of the SS as they can, this
variable Xi is still important!
Failing to reject means: This variable
does not do anything useful when added
to the other variables in the model.
Key Point: This may just be due to
overlap. The variable Xi if considered by
itself may be associated to Y.
37
T-tests General Comments
If only one of the variables has an
insignificant t-test, and if the sample size is
reasonable, then probably that variable is
not important.
Any variable that has a significant t-test is
important and should remain in the model.
If multiple variables have insignificant ttests, it is possible (and even likely) that
some of them may still be important!!!!
38
Collinearity Issues
If predictors are highly associated
to each other, then they will likely
be redundant when we consider
them in the same model.
39
Collinearity
Collinearity occurs when there are strong
relationships (high intercorrelations) among
the predictor variables. Some examples:
X1 highly correlated to X2
X1 not highly correlated to X2, X3, or X4
individually, but together they explain most of
the variation in X1.
Note that when squares or interactions are
used, e.g., X3 = X1*X2 will by default be
correlated to X1 and X2. Centering can help
in this case to decrease correlation.
40
Consequences
Whenever we have collinearity, the predictor
variables will fight to explain the same
portion of the variation in the response (SS).
If X1 and X2 are redundant, then the variable
added last T-test will be insignificant for both
even though we probably do want one of them
in the model.
When collinearity is present, standard errors
will be large (inflated) and interpretation of
parameter estimates is adversely affected.
41
Pictorial Representation
X3 is highly correlated to X1 and X2. So in a regression model
we are likely to use either X3 or (X1,X2), but not both.
SS(X3|X1,X2)
SS(X1)
SS(X2|X1)
Total SSY
42
Finding Intercollinearity
Plots (X1 vs X2, etc.)
Use PROC CORR to determine pairwise
correlations. High values are problematic.
(>0.9 is considered most serious, but even
values between 0.5 and 0.9 can cause
trouble).
Variance Inflation Factors (>10 indicates
multicollinearity) catch intercorrelation of the
form X1 = X2 X3 X4.
43
Pairwise Scatterplots
Can look at plots to find relationships among the predictors.
Can use GPLOT for individual plots; for a first look, SAS
Interactive Data Analysis provides a convenient scatter plot
matrix:
Solutions Analysis Interactive Data Analysis
Select WORK library and the dataset for which you
wish to obtain plots.
Select columns you want to plot (can use control key to
select multiple columns).
Analyze Scatterplot
Clicking a point in a plot will highlight that point in every
plot.
44
PROC CORR
Output contains r and a p-value. Significant
relationships among the Xs can be
problematic.
Note that for both PROC CORR and
scatterplots we are considering only
pairwise correlations. It is possible to have
more complicated correlations (e.g. maybe
X2 and X3 explain 98% of X1 together, but
only 50% individually).
45
Variance Inflation Factors
VIF is related to the variance of the
estimated regression coefficients (you can
think of the SEs being inflated by having
intercorrelation among the predictors)
1
V IFi =
1 - R i2
2
i
R is the coefficient of determination
obtained in regression of Xi on all other
predictors.
46
VIFs in SAS
Obtained by using VIF option in the model
statement of PROC REG.
VIF > 10 suggests multicollinearity. If this
happens, a simplistic strategy is to remove
the variable with the highest VIF and rerun
the analysis.
47
CLG Activity #2
We now look at the SBP data from
the MLR model viewpoint. Please
discuss questions 3.7-3.9 from the
handout.
48
Questions?
49
Upcoming in Topic 4...
Partial F Tests in MLR
Related Reading: Chapter 9
50