494 1
POLS Lecture 7: Review of the Linear Regression Model (Part I)
Kevin Quinn University of Washington
2
Outline
Least Squares as a Fit Criterion Sampling Properties of the OLS estimator Importance of Assumptions The Least Squares Estimator as a Maximum Likelihood Estimator
3
Example: Relationship Between Household Income and Household Expenditures
At each of 10 household income levels we take a random sample of size 100 from the population of US households, for a total sample size of 1000
3
Example: Relationship Between Household Income and Household Expenditures
At each of 10 household income levels we take a random sample of size 100 from the population of US households, for a total sample size of 1000 We record the level of expenditures for each household
3
Example: Relationship Between Household Income and Household Expenditures
At each of 10 household income levels we take a random sample of size 100 from the population of US households, for a total sample size of 1000 We record the level of expenditures for each household Plotting these data we might see:
Expenditures (Thousands of Dollars) 0 10 20 30 40 50 60
2 4 Income 6 8 10
4
5
It appears that the relationship between income and expenditures is essentially linear
5
It appears that the relationship between income and expenditures is essentially linear As a result, it seems reasonable to express this relationship as: yi = 1 + xi2 + i, i = 1, . . . , 1000
where yi denotes the expenditure level in household i, xi denotes the income level of household i, and i denotes a random disturbance
Expenditures (Thousands of Dollars) 0 10 20 30 40 50 60
2 4 Income 6 8 10
6
7
Questions: What are good estimators of 1 and 2? How can we express our uncertainty about the underlying true values of 1 and 2? What assumptions need to be maintained and how might we know if they dont hold?
8
Least Squares as a Fit Criterion
Consider the following linear model in scalar form: yi = 1 + xi22 + + xik k +
i
i = 1, . . . , n
8
Least Squares as a Fit Criterion
Consider the following linear model in scalar form: yi = 1 + xi22 + + xik k +
i
i = 1, . . . , n
We can write this in matrix form as: y = X +
9
Or in other words: y1 1 x12 y2 1 x22 . = . . . . . yn 1 xn2 y is n 1 X is n k is k 1 and is n 1
x1k x2k ... . . xnk
1 2 . . k
1
2 + . .
n
10
The Ordinary Least Squares (OLS) Estimator
One possible estimator for is to choose the coecient vector that minimizes the sum of squared residuals
11
Advantages:
11
Advantages: Has a closed form solution Can be calculated quickly
11
Advantages: Has a closed form solution Can be calculated quickly This estimator has very desirable sampling properties within the class of linear unbiased estimators as long as some additional assumptions hold
11
Advantages: Has a closed form solution Can be calculated quickly This estimator has very desirable sampling properties within the class of linear unbiased estimators as long as some additional assumptions hold As well see in a second, this estimator is the maximum likelihood estimator if a few additional assumptions are maintained
12
Disadvantages:
12
Disadvantages: Not robust to outliers in either y or (X, y) space
12
Disadvantages: Not robust to outliers in either y or (X, y) space Better estimators exist outside the class of linear unbiased estimators
13
Notation: the true value of the coecient vector an arbitrary value of the coecient vector OLS the OLS estimator of
13
Notation: the true value of the coecient vector an arbitrary value of the coecient vector OLS the OLS estimator of We can write the sum of squared residuals as:
n
S() =
i=1
i2
13
Notation: the true value of the coecient vector an arbitrary value of the coecient vector OLS the OLS estimator of We can write the sum of squared residuals as:
n n
S() =
i=1
i2 =
i=1
(yi xi)2
13
Notation: the true value of the coecient vector an arbitrary value of the coecient vector OLS the OLS estimator of We can write the sum of squared residuals as:
n n
S() =
i=1
i2 =
i=1
(yi xi)2
The OLS estimator ( OLS) is dened to be the value of that minimizes S()
14
We can write S() in matrix form as: S() =
14
We can write S() in matrix form as: S() = = (y X) (y X)
14
We can write S() in matrix form as: S() = = (y X) (y X) = y y X y y X + X X
14
We can write S() in matrix form as: S() = = (y X) (y X) = y y X y y X + X X = y y 2 X y + X X
14
We can write S() in matrix form as: S() = = (y X) (y X) = y y X y y X + X X = y y 2 X y + X X
The necessary condition for a minimum is that the gradient of S() with respect to be equal to 0
15
Calculating the gradient and setting it equal to 0 we have: S() = 2X y + 2X X = 0
15
Calculating the gradient and setting it equal to 0 we have: S() = 2X y + 2X X = 0 This implies that OLS must satisfy the so-called normal equations: X X OLS = X y
16
Assuming that (X X)1 exists (which it will as long as X has full rank) we can solve the normal equations and write our estimator OLS as: OLS = (X X)1X y
17
Example: Regression on a Constant
Consider the following model:
yi = 1 + i,
i = 1, , n
18
The data are: y= 2 1 2 3 2 x= 1 1 1 1 1
19
Note that xx =
n
xi = 5
i=1
19
Note that xx =
n
xi = 5
i=1 1
(x x)
1 = 5
19
Note that xx =
n
xi = 5
i=1 1
(x x) xy =
1 = 5 yi = 10
n
i=1
19
Note that xx =
n
xi = 5
i=1 1
(x x) xy =
1 = 5 yi = 10
n
i=1
So
1 = (x x)1x y = 1 10 = 2 5 Which, as we would expect, is the mean of y
20
Example: A Bivariate Relationship
Consider the following model:
yi = 1 + xi22 +
i
21
The data are: y= 1.5 1.0 2.5 1.0 4.0 3.0 4.5 4.0 X= 1 1 1 1 1 1 1 1 1 1 2 2 3 3 4 4
22
Note that XX = n
n i=1 xi2 n i=1 xi2 n x2 i=1 i2
=
8 20 20 60
22
Note that XX = n
n i=1 xi2 1 n i=1 xi2 n x2 i=1 i2
=
8 20 20 60
(X X)
=
0.75 0.25 0.25 0.10
22
Note that XX = n
n i=1 xi2 1 n i=1 xi2 n x2 i=1 i2
=
8 20 20 60
(X X) Xy =
=
0.75 0.25 0.25 0.10
n i=1 yi
n i=1 xi2yi
=
21.5 64.5
22
Note that XX = n
n i=1 xi2 1 n i=1 xi2 n x2 i=1 i2
=
8 20 20 60
(X X) Xy = So
=
0.75 0.25 0.25 0.10
n i=1 yi
n i=1 xi2yi
=
21.5 64.5 0.0 1.075
= (X X)1X y =
y 1.0 1.0 1.5 2.0 2.5 3.0 3.5 4.0 x 1.5 2.0 2.5 3.0 3.5 4.0 4.5
23
y 1.0 1.0 1.5 2.0 2.5 3.0 3.5 4.0 x 1.5 2.0 2.5 3.0 3.5 4.0 4.5
24
25
Sampling Properties
Frequentist inference focuses on the performance of an estimator over repeated samples
25
Sampling Properties
Frequentist inference focuses on the performance of an estimator over repeated samples In other words, if we were to draw a large number of random samples from some population and calculate for each sample, what would the distribution of look like over these repeated samples?
26
In order to draw some conclusions about the repeated sampling properties of the OLS estimator we need to rely on a number of assumptions
27
The Assumptions
1. (Functional Form) y = X +
27
The Assumptions
1. (Functional Form) y = X + E[ ] = 0
2. (Zero Mean Disturbances) or equivalently E[y] = X
27
The Assumptions
1. (Functional Form) y = X + E[ ] = 0 E[ ] = 2I
2. (Zero Mean Disturbances) or equivalently E[y] = X
3. (Variance-Covariance of Disturbances)
27
The Assumptions
1. (Functional Form) y = X + E[ ] = 0 E[ ] = 2I
2. (Zero Mean Disturbances) or equivalently E[y] = X
3. (Variance-Covariance of Disturbances) 4. (Relationship between X and )
E[ |X] = 0
27
The Assumptions
1. (Functional Form) y = X + E[ ] = 0 E[ ] = 2I
2. (Zero Mean Disturbances) or equivalently E[y] = X
3. (Variance-Covariance of Disturbances) 4. (Relationship between X and ) 5. (Nature of X) column rank
E[ |X] = 0
X is a nonstochastic matrix with full
28
6. (Normality of Disturbances)
N (0, 2I)
29
If these assumptions hold what can we say about the repeated sampling properties of OLS?
30
The mean of OLS
30
The mean of OLS OLS] = E[(X X)1X y] E[
30
The mean of OLS OLS] = E[(X X)1X y] E[ = E[(X X) X (X + )]
1
30
The mean of OLS OLS] = E[(X X)1X y] E[ = E[(X X) X (X + )] = E[(X X)1X X + (X X)1X ]
1
30
The mean of OLS OLS] = E[(X X)1X y] E[ = E[(X X) X (X + )] = E[(X X)1X X + (X X)1X ] = E[(I + (X X)1X ]
1
30
The mean of OLS OLS] = E[(X X)1X y] E[ = E[(X X) X (X + )] = E[(X X)1X X + (X X)1X ] = E[(I + (X X)1X ] = E[ + (X X)1X ]
1
30
The mean of OLS OLS] = E[(X X)1X y] E[ = E[(X X) X (X + )] = E[(X X)1X X + (X X)1X ] = E[(I + (X X)1X ] = E[ + (X X)1X ] = + (X X)1X E[ ]
1
30
The mean of OLS OLS] = E[(X X)1X y] E[ = E[(X X) X (X + )] = E[(X X)1X X + (X X)1X ] = E[(I + (X X)1X ] = E[ + (X X)1X ] = + (X X)1X E[ ] = + (X X)1X 0
1
30
The mean of OLS OLS] = E[(X X)1X y] E[ = E[(X X) X (X + )] = E[(X X)1X X + (X X)1X ] = E[(I + (X ] X)1X = E[ + (X X)1X ] = + (X X)1X E[ ] = + (X X)1X 0 =
1
31
OLS is unbiased
31
OLS is unbiased On average the OLS estimator yields the true value of
31
OLS is unbiased On average the OLS estimator yields the true value of Note, we didnt need to assume normal disturbances to prove unbiasedness Unbiasedness doesnt say anything about the variability of OLS
32
Variance-covariance matrix of OLS Var( OLS) = E[( OLS E[ OLS])( OLS E[ OLS]) ]
32
Variance-covariance matrix of OLS Var( OLS) = E[( OLS E[ OLS])( OLS E[ OLS]) ] = E[( OLS )( OLS ) ]
32
Variance-covariance matrix of OLS Var( OLS) = E[( OLS E[ OLS])( OLS E[ OLS]) ] = E[( OLS )( OLS ) ]
= E[( + (X X)1X )( + (X X)1X ) ]
32
Variance-covariance matrix of OLS Var( OLS) = E[( OLS E[ OLS])( OLS E[ OLS]) ] = E[( OLS )( OLS ) ]
= E[( + (X X)1X )( + (X X)1X ) ]
= E[(X X)1X
X(X X)1]
32
Variance-covariance matrix of OLS Var( OLS) = E[( OLS E[ OLS])( OLS E[ OLS]) ] = E[( OLS )( OLS ) ]
= E[( + (X X)1X )( + (X X)1X ) ]
= E[(X X)1X = (X X)1X E[
X(X X)1] ]X(X X)1
32
Variance-covariance matrix of OLS Var( OLS) = E[( OLS E[ OLS])( OLS E[ OLS]) ] = E[( OLS )( OLS ) ]
= E[( + (X X)1X )( + (X X)1X ) ]
= E[(X X)1X = (X X)1X E[
X(X X)1] ]X(X X)1
= (X X)1X ( 2I)X(X X)1
32
Variance-covariance matrix of OLS Var( OLS) = E[( OLS E[ OLS])( OLS E[ OLS]) ] = E[( OLS )( OLS ) ]
= E[( + (X X)1X )( + (X X)1X ) ]
= E[(X X)1X = (X X)1X E[
X(X X)1] ]X(X X)1
= (X X)1X ( 2I)X(X X)1 = 2(X X)1X X(X X)1
32
Variance-covariance matrix of OLS Var( OLS) = E[( OLS E[ OLS])( OLS E[ OLS]) ] = E[( OLS )( OLS ) ]
= E[( + (X X)1X )( + (X X)1X ) ]
= E[(X X)1X = (X X)1X E[
X(X X)1] ]X(X X)1
= (X X)1X ( 2I)X(X X)1 = 2(X X)1X X(X X)1 = 2(X X)1
33
Once again, we didnt need to assume normality of the disturbances to derive the variance covariance matrix of OLS However, we did assume we know 2 In practice, well estimate 2 as = nk
2
34
If we are willing to assume the disturbances are normally distributed then we can make even stronger statements about the sampling distribution of OLS
34
If we are willing to assume the disturbances are normally distributed then we can make even stronger statements about the sampling distribution of OLS In particular (assuming normal disturbances): OLS| 2 N (, 2(X X)1)
35
Other properties of the OLS estimator (assuming the assumptions hold) OLS is consistent
35
Other properties of the OLS estimator (assuming the assumptions hold) OLS is consistent OLS is the best (minimum mean squared error) linear unbiased estimator
35
Other properties of the OLS estimator (assuming the assumptions hold) OLS is consistent OLS is the best (minimum mean squared error) linear unbiased estimator Assuming normality, OLS is the best (minimum mean squared error) unbiased estimator
36
What Happens if Assumptions Dont Hold?
Functional form assumption (y = X + ) doesnt hold
36
What Happens if Assumptions Dont Hold?
Functional form assumption (y = X + ) doesnt hold If functional form isnt correct all bets are o
36
What Happens if Assumptions Dont Hold?
Functional form assumption (y = X + ) doesnt hold If functional form isnt correct all bets are o Omitted variable bias
37
Assumption of zero mean disturbances (E[ ] = 0) doesnt hold
37
Assumption of zero mean disturbances (E[ ] = 0) doesnt hold As long as a constant term is included this is relatively benign
38
Variance-Covariance of Disturbances is not equal to 2I
38
Variance-Covariance of Disturbances is not equal to 2I Suppose E[ ] = 2
38
Variance-Covariance of Disturbances is not equal to 2I Suppose E[ ] = 2 As long as E[ |X] = 0 OLS is still unbiased
38
Variance-Covariance of Disturbances is not equal to 2I Suppose E[ ] = 2 As long as E[ |X] = 0 OLS is still unbiased Var( OLS) is now Var( OLS) = 2(X X)1(X X)(X X)1 so standard errors will not be correct
38
Variance-Covariance of Disturbances is not equal to 2I Suppose E[ ] = 2 As long as E[ |X] = 0 OLS is still unbiased Var( OLS) is now Var( OLS) = 2(X X)1(X X)(X X)1 so standard errors will not be correct 2 may be a biased estimator of 2
38
Variance-Covariance of Disturbances is not equal to 2I Suppose E[ ] = 2 As long as E[ |X] = 0 OLS is still unbiased Var( OLS) is now Var( OLS) = 2(X X)1(X X)(X X)1 so standard errors will not be correct 2 may be a biased estimator of 2 OLS still oftentimes consistent although this depends on X and
39
E[ |X] = 0
39
E[ |X] = 0 OLS is no longer unbiased OLS is no longer consistent
40
X is stochastic
40
X is stochastic If X is stochastic but perfectly measured then all of the results above still go through ( OLS) is unbiased, consistent, and best linear unbiased
40
X is stochastic If X is stochastic but perfectly measured then all of the results above still go through ( OLS) is unbiased, consistent, and best linear unbiased However, if X is measured imperfectly, i.e. there is measurement error, then things get very bad
40
X is stochastic If X is stochastic but perfectly measured then all of the results above still go through ( OLS) is unbiased, consistent, and best linear unbiased However, if X is measured imperfectly, i.e. there is measurement error, then things get very bad if only o...