Unformatted text preview: Introductory Econometrics
ECON2206/ECON3209 Slides03 Lecturer: Minxian Yang ie_Slides03 my, School of Economics, UNSW 1 3. Multiple Regression Model: Estimation (Ch3) 3. Multiple Regression Model: Estimation
• Lecture plan (largely parallel to Ch2)
–
–
–
–
–
–
–
– Motivation and definitions
ZCM assumption
Estimation method: OLS
Mechanics of OLS
Underlying assumptions of multiple regression model
Expected values and variances of OLS estimators
Omitted and irrelevant variables
GaussMarkov theorem ie_Slides03 my, School of Economics, UNSW 2 3. Multiple Regression Model: Estimation (Ch3) • Motivation
– Example 1. (control observable factors)
wage = β0 + β1educ + error,
where error represents (or contains) exper.
• exper is likely related to educ.
• The ceteris paribus effect of educ on wage cannot be
properly estimated in this model. Why? If exper is available, then we can “hold expr fixed” in
wage = β0 + β1educ + β2exper + u,
where wage is explained by both educ and expr.
• β1 and β2 measure ceteris paribus effects,
properly estimable if u is not “related” to educ and expr.
ie_Slides03 my, School of Economics, UNSW 3 3. Multiple Regression Model: Estimation (Ch3) • Motivation
– Example 2. (flexible functional form)
wage = β0 + β1age + β2age2 +...+ u,
where wage may increase initially and decrease
eventually as age increases.
– In general, regression models with multiple x’s have
the following merits. They
• allow us to explicitly control for (hold fixed) many
factors that affect the dependent variable, in order to
draw ceteris paribus conclusions;
• provide better explanation of the dependent variable by
accommodating flexible functional forms. ie_Slides03 my, School of Economics, UNSW 4 3. Multiple Regression Model: Estimation (Ch3) • Multiple regression model
– Definition
y = β0 + β1x1 +...+ βkxk + u ,
• y : dependent variable (observable)
• x1, ..., xk : independent variables (observable)
• β1, ..., βk : slope parameters, “partial effect”,
(to be estimated)
• β0 : intercept parameter (to be estimated)
• u : error term or disturbance (unobservable)
• k : the number of independent variables – The disturbance u represents factors other than x’s.
– With the intercept β0, the unconditional mean of u can
always be set to zero: E(u) = 0 .
ie_Slides03 my, School of Economics, UNSW 5 3. Multiple Regression Model: Estimation (Ch3) • Zero conditional mean assumption
– The zeroconditionalmean (ZCM) assumption is
E(u  x1, ..., xk) = 0 ,
for the multiple regression model.
• It requires the average of u to be the same irrespective
of the values of x’s.
• It implies that the factors in u are uncorrelated with
x1, ..., xk.
• It is a key condition for the OLS estimators being
unbiased.
• It defines the population regression function (PRF)
E(y  x1, ..., xk) = β0 + β1x1 +...+ βkxk .
ie_Slides03 my, School of Economics, UNSW 6 3. Multiple Regression Model: Estimation (Ch3) • Observations on (x1, ..., xk, y)
– A random sample is a set of independent
observations {(xi1, xi2 , ... , xik , yi), i = 1,2,...,n}.
the ith observation on
the 2nd independent variable – At observation level, the model may be written as
yi = β0 + β1xi1 + ... + βkxik + ui ,
i = 1, 2, ..., n
where i is the observation index.
– Or collectively, y1 1 x11 x1k β 0 u1 y 1 x x β u 2k 1 2 = 21
+ 2 . y n 1 x n1 x nk β k un – Matrix notation:
ie_Slides03 Y= X my, School of Economics, UNSW B + U.
7 3. Multiple Regression Model: Estimation (Ch3) • Estimate multiple regression
– The model:
yi = β0 + β1xi1 + ... + βkxik + ui , i = 1, 2, ..., n. ˆˆ
ˆ
– Let ( β 0 , β1,..., β k ) estimate (β0 , β1, ..., βk).
– Corresponding residual is
ˆ
ˆ
ˆ
ˆ
ui = y i − β 0 − β1 x i 1 − − β k x ik , i = 1,2,..., n. – The sum of squared residuals (SSR)
n n i =1 i =1 ˆ
ˆ
ˆ
ˆ
SSR = ∑ ui2 = ∑ ( y i − β 0 − β1 x i 1 − − β k x ik )2
indicates the goodness of the estimates.
– Good estimates should make SSR small.
ie_Slides03 my, School of Economics, UNSW 8 3. Multiple Regression Model: Estimation (Ch3) • Ordinary least squares (OLS)
ˆˆ
ˆ
– The OLS estimates( β 0 , β1,..., β k ) minimise the SSR:
ˆˆ
ˆ
( β 0 , β1,..., β k ) = minimiser of SSR.
– As the estimates minimise SSR,
the first order conditions lead to
n ∑ (y
i =1 i n ∑ (y
i =1 i ˆ
ˆ
ˆ
− β 0 − β1 x i 1 − − β k x ik ) = 0,
ˆ
ˆ
ˆ
− β 0 − β1 x i 1 − − β k x ik )x ij = 0, j = 1,..., k . the ith observation on
the jth independent variable
ie_Slides03 my, School of Economics, UNSW 9 3. Multiple Regression Model: Estimation (Ch3) • Ordinary least squares (OLS)
– Solving the k+1 equations with k+1 unknowns gives
ˆ
B = ( X ' X )−1 X 'Y
in a matrix notation. (linear in Y) 1st order conditions
in a matrix notation :
ˆ
X ' (Y − XB ) = 0 – OLS only requires the existence of ( X ' X )−1 .
It can be carried out for almost any data set.
(X′X) is a form of sumofsquared x’s.
– Don’t worry, matrix will NOT be tested in this course.
ie_Slides03 my, School of Economics, UNSW 10 3. Multiple Regression Model: Estimation (Ch3) • Sample regression function (SRF)
– Once OLS estimates are obtained,
ˆ
ˆ
ˆ
ˆ
y i = β 0 + β1 x i 1 + + β k x ik
is the fitted value of y when (x1, ..., xk)= (xi1, ..., xik).
– The OLS regression line or sample regression
function (SRF) is
ˆ
ˆ
ˆˆ
y = β 0 + β1 x1 + + β k x k
Unless indicated otherwise,
which is an estimate of the PRF. an intercept is included.
– “Run a regression of y on x1 , ..., xk” : use OLS to
estimate the multiple regression model
y = β0 + β1x1 +...+ βkxk + u.
ie_Slides03 my, School of Economics, UNSW 11 3. Multiple Regression Model: Estimation (Ch3) • Interpretation of SRF
– The OLS regression line or SRF can be written in the
form of changes, holding u fixed:
ˆ
ˆˆ
∆y = β1∆x1 + + β k ∆x k .
• the coefficient on x1 is the partial effect of x1 on y
ˆˆ
holding other x’s fixed: ∆y = β1∆x1.
• we are able to control for (or hold fixed) {x2 , ..., xk}
when considering the effect of x1 on y.
ˆ
• β1 has a ceteris paribus interpretation when ZCM holds
or the factors in u are not “related” to x1.
• other coefficients can be similarly interpreted. – If possible, include relevant independent variables.
ie_Slides03 my, School of Economics, UNSW 12 3. Multiple Regression Model: Estimation (Ch3) • Interpretation of SRF
– Example 3.2. Use educ, exper, tenure (years with
current employer) to explain hourly wage:
log(wage) = .284 + .092 educ + .004 exper + .022 tenure • the coefficient on educ : holding exper and tenure fixed,
an extra year of education is predicted to increase
log(wage) by 0.092 (or 9.2% increase in wage), which
is the ceteris paribus effect under ZCM.
• holding educ fixed, the effect of an individual staying at
the same firm for an extra year on log(wage) :
Δlog(wage) = .004 + .022 = .026 ie_Slides03 my, School of Economics, UNSW 13 3. Multiple Regression Model: Estimation (Ch3) • Predicted value and residual
– The fitted value
ˆ
ˆ
ˆ
ˆ
y i = β 0 + β1 x i 1 + + β k x ik
is also known as predicted value.
ˆ
ˆ
– The residual ui = y i − y i can be regarded as
prediction error.
– Important properties
• the sample mean of the residuals is 0.
• the sample covariance between the residuals and each
independent variable is 0.
• the sample mean point ( x1,..., x k , y ) is on the SRF, ie, ˆ
ˆ
ˆ
y = β 0 + β1 x1 + + β k x k . ie_Slides03 my, School of Economics, UNSW 14 3. Multiple Regression Model: Estimation (Ch3) • Sums of squares
ˆˆ
– Each yi may be decomposed into y i = y i + ui .
We measure variations from y :
• Total sum of squares (total variation in yi ): SST = ∑i =1 ( y i − y )2 ,
n ˆ
• Explained sum of squares (variation in y i ):
ˆ
SSE = ∑i =1 ( y i − y )2 ,
n • sum of squared Residuals (variation in ui ):
ˆ ˆ
SSR = ∑i =1 ui2 .
n • It can be shown that
ie_Slides02 SST = SSE + SSR . my, School of Economics, UNSW 15 3. Multiple Regression Model: Estimation (Ch3) • Rsquared: goodnessoffit
– We may use the fraction of variation in y that is
explained by x’s (or by the SRF) to measure the
goodnessoffit.
– Rsquared (coefficient of determination):
SSR
SSE
2
Note: R2 never decreases
R=
= 1−
.
when additional explanatory
SST
SST
R2, variables are added to the
model. • larger
better fit;
• 0 ≤ R2 ≤ 1.
ˆ
• R2 is the squared correlation between y and y . eg. log(wage) = .284 + .092educ + .004exper + .022tenure
R2 = 0.316.
ie_Slides03 my, School of Economics, UNSW 16 3. Multiple Regression Model: Estimation (Ch3) • Assumptions about multiple regression model
(MLR1 to MLR4) 1. (linear in parameters) In the population model, y is
related to x’s by y = β0 + β1x1 +...+ βkxk + u, where
(β0, β1,..., βk) are population parameters and u is
disturbance.
2. (random sample) {(xi1, xi2,..., xik , yi), i = 1,2,...,n} with
n > k+1 is a random sample from the population.
3. (no perfect collinearity) None of x’s is constant and
there is no perfect linear relationship among x’s.
4. (zero conditional mean) The disturbance u satisfies
E(u  x1, ..., xk) = 0 for any given value of (x1, ..., xk).
ie_Slides03 my, School of Economics, UNSW 17 3. Multiple Regression Model: Estimation (Ch3) • Unbiasedness of OLS estimators
Theorem 3.1
Under MLR1 to MLR4, the OLS estimators are
ˆ
unbiased: E ( β j ) = β j , j = 0,1,..., k .
Unbiased estimators
– are “centred” around (β0, β1, ...,βk).
– correctly estimate (β0, β1, ...,βk) on average.
– will be “near” (β0, β1, ...,βk) for a “typical” sample. ie_Slides03 my, School of Economics, UNSW 18 3. Multiple Regression Model: Estimation (Ch3) • Irrelevant “explanatory” variables
– What if an irrelevant x is included?
• “irrelevant” means the population coefficient of that
variable is 0.
• as the OLS estimators are unbiased,
the estimate of that coefficient will “typically” be near 0.
• the inclusion of irrelevant variables has undesirable
effects on the variances of the OLS estimators. eg. x3 is irrelevant in the model
y = β0 + β1x1 + β2x2 + 0x3 + u,
ˆ
the OLS estimator is unbiased: E ( β j ) = β j ,
ˆ
in particular, E ( β 3 ) = β 3 = 0.
ie_Slides03 my, School of Economics, UNSW j = 0,1,..., k . 19 3. Multiple Regression Model: Estimation (Ch3) • Omitted explanatory variables
– What if a relevant x is omitted?
• the OLS estimators will generally be biased.
• the direction and size of bias depend on how the
omitted is related to the included. ie_Slides03 my, School of Economics, UNSW δ β β β – Example. When x2 is omitted from the true model
y = β0 + β1x1 + β2x2 + u,
it becomes: y = β0 + β1x1 + v with v = β2x2 + u.
~~~
The estimated model is y = β 0 + β1 x1.
It can be shown that OLS is biased: E( ~1 ) = 1 + 2 ~,
~
~
where β 2δ is known as omitted variable bias and δ
is the coefficient of regressing x2 on x1.
20 3. Multiple Regression Model: Estimation (Ch3) • Omitted variable bias
– Example. (continued) the omitted variable bias is zero
in two special cases
• when β2 = 0; or (the case of irrelevant variable)
~
• when δ = 0.
(the case of uncorrelated x variables) In general,
cov(x1, x2) > 0 cov(x1, x2) < 0 β2 > 0 + ve bias − ve bias β2 < 0 − ve bias + ve bias In practice, the knowledge about the signs of β2 and
cov(x1, x2) is useful for interpreting estimation result. ie_Slides03 my, School of Economics, UNSW 21 3. Multiple Regression Model: Estimation (Ch3) ~
2, δ 1+ β 1)= β E( ~ β • Omitted variable bias eg. Suppose the true model is
log(wage) = β0 + β1educ + β2ability + u.
We do not have data on ability but know that ability is
positively affects wage and positively correlated with
educ. The estimated equation is
~
log(wag e ) = .584 + .083educ, n = 526, R 2 = .186.
~
– β1 = .083 is from one sample and it can be either
above or below the population parameter β1.
~
– The average of estimates β1 across many random
samples would be too large (positive bias)
ie_Slides03 my, School of Economics, UNSW 22 3. Multiple Regression Model: Estimation (Ch3) • Variance of OLS estimators
5. (MLR5, homoskedasticity)
Var(ui  xi1, ..., xik) = σ2 for i = 1,2,...,n.
(It implies Var(ui) = σ2.)
– MLR5 requires that the conditional variance of u be
unrelated to x’s.
– MLR15 are collectively known as the GaussMarkov
assumptions.
– MLR5 is needed to derive a “simple” formula for the
variances of the OLS estimators. ie_Slides03 my, School of Economics, UNSW 23 3. Multiple Regression Model: Estimation (Ch3) • Variance of OLS estimators
Strictly, Theorem 3.2 is about the variances of Theorem 3.2
OLS estimators, conditional on given x.
Under MLR1 to MLR5, the variances of the OLS
estimators are given by:
σ2
ˆ
Var ( β j ) =
,
j = 1,..., k,
2
SST j (1 − R j ) where SST j = ∑i =1 ( x ij − x j )2 , x j = n −1 ∑i =1 x ij
and R 2 is the Rsquared from regressing xj on all
j
other independent variables.
ˆ
– the larger is σ2, the greater is Var ( β j ).
ˆ
– the larger is R 2 , the greater is Var ( β j ) .
j
ˆ
– the larger the variation in xj, the smaller Var ( β j ) .
n ie_Slides03 my, School of Economics, UNSW n 24 3. Multiple Regression Model: Estimation (Ch3) • Multicollinearity
ˆ
– The larger is R 2, the greater is Var ( β j ) .
j R 2 is the Rsquared from regressing xj on all other x’s.
j The larger is R 2, the stronger is xj “associated” with
j
other x’s, the less informative is xj. “R 2 = 1”, which is ruled out by MLR3, implies that
j
there is a perfect linear relationship between xj and
other x’s (therefore, xj is redundant).
– High (but not perfect) correlation between two or
more independent variables is known as
multicollinearity, which does not violate MLR3.
eg. In the model y = β0 + β1x1 + β2x2 + β3x3 + u, if β1 is of interest, a
high correlation between x2 and x3 may not be a major concern.
ie_Slides03 my, School of Economics, UNSW 25 3. Multiple Regression Model: Estimation (Ch3) • Omitted variables again
– Example. When x2 is omitted from the true model
y = β0 + β1x1 + β2x2 + u,
it becomes: y = β0 + β1x1 + v with v = β2x2 + u.
~
ˆ
Let β1 and β1 be the OLS estimators from the true and
the misspecified. We find that
σ2
σ 2 + β 22 var( x 2 )
~
ˆ
, Var ( β1 ) =
.
Var ( β1 ) =
2
SST1 (1 − R1 ) SST1 ~ ˆ
1. When β2 = 0 and R12 > 0, bothβ1 and β1are unbiased and
the latter is more efficient. (Irrelevant x2 is not harmless.)
~
ˆ
2. When β2 ≠ 0 and R12 = 0, bothβ1 and β1are unbiased and
the former is more efficient. (Relevant, uncorrelated x2 is beneficial.)
~
ˆ
3. When β2 ≠ 0 and R12 > 0, β1 is unbiased but β1is biased.
ie_Slides03 my, School of Economics, UNSW 26 3. Multiple Regression Model: Estimation (Ch3) • Omitted variables again
– Point 1: the inclusion of an irrelevant variable in the
regression generally leads to inefficient estimator of β1.
– Point 2: the exclusion of a relevant variable from the
regression, when it is uncorrelated with x1, leads to
inefficient estimator of β1.
– Point 3: the exclusion of a relevant variable from the
regression generally leads to biased estimator of β1.
When bias is a major concern, researchers tend to include
many xvariables as controls in practice. ie_Slides03 my, School of Economics, UNSW 27 3. Multiple Regression Model: Estimation (Ch3) • Estimation of σ2
– As the residual approximates u, the estimator of σ2 is
ui2
∑i =1 ˆ
n SSR
.
σ=
ˆ
=
n − (k + 1) n − (k + 1)
2 the number of
estimated coefficients – σ = σ 2 is known as the standard error of the
ˆ
ˆ
regression.
– The degrees of freedom (df) for the regression is
n – (k+1) = # of obs – # of estimated coefs.
Theorem 3.3 (unbiased estimator of σ2)
Under MLR1 to MLR5, E (σ 2 ) = σ 2 .
ˆ
ie_Slides03 my, School of Economics, UNSW 28 3. Multiple Regression Model: Estimation (Ch3) • Efficiency of OLS estimation
Theorem 3.4 (GaussMarkov theorem)
ˆˆ
ˆ
Under MLR1 to MLR5, ( β 0 , β1,..., β k ) are the best
linear unbiased estimators (BLUEs) of (β0, β1, ...,βk).
– Linear estimator : a linear combination of y’s.
– Linear unbiased estimators : a class of linear
estimator that are unbiased.
– BLUEs : ones with smallest variance.
– OLS estimators under MLR1MLR5:
• they are linear;
• they are unbiased;
• they are efficient (have smallest variances).
ie_Slides03 my, School of Economics, UNSW 29 3. Multiple Regression Model: Estimation (Ch3) • Summary
– Multiple regression allows us to hold other
(observable) factors fixed.
– It allows for nonlinear relationships between y and x’s.
– It is linear in the β parameters so that simple OLS
method is applicable.
– Under MRL1MRL4, the OLS estimators are
unbiased.
– Under MRL1MRL5, the OLS estimators are BLUEs
and the estimator of σ2 is unbiased.
– Omitting relevant variable or including irrelevant
variable are generally undesirable.
ie_Slides03 my, School of Economics, UNSW 30 ...
View
Full
Document
This note was uploaded on 06/12/2011 for the course ECONOMICS 3291 taught by Professor Professorsnamespublishedtheyarethesoleowners during the Three '11 term at University of New South Wales.
 Three '11
 professorsnamespublishedtheyarethesoleowners
 Economics, Econometrics

Click to edit the document details