3300:
Midterm PAM Review
Where weve been
Overview of Program Evaluation and Program Theory (ln2) Statistics Review (ln3) Causality - FPCI, counterfactual reasoning, statistical approach to solving FPCI, drawbacks of PF causal effect estimator, attributes as causes, SUTVA (ln4) Regression Review, description, uses of R2, causal interpretations of coefficients, omitted variable bias, multiple regression (ln5-7) Research designs and validity, randomized controlled trials (ln8)
Rand Health Insurance Experiment (ln8-9) NYC Voucher Experiment (ln9-10) Munnel et al. Study of Mortgage Lending in Boston (ln 11) Krueger vs. DiNardo & Pischke Studies of Returns to computer use (ln12)
Non-experimental designs (ln10-11)
Assessment of non-experimental designs by LaLonde (ln13)
Regression
Regression as a linear approximation to graph of averages
Summarizing the relationship: Regression of Earnings on Years of Education
E[Yi|Xi]=-19,280+4,157Xi Regression line: linear approximation to graph of averages
Regression Language - Interpreting coefficients (1)
Y and X are continuous (regress wage on years education) Interpretation of constant: The average hourly wage of those with 0 years of school is -15.25. Interpretation of coefficient on school: Each additional year of education is associated with an extra $2.67 in hourly wages on average. Avoid words like each additional year of school causes/increases/leads to unless you believe the relationship is causal.
. reg hrwagely school Source | SS df MS -------------+-----------------------------Model | 53501.4245 1 53501.4245 Residual | 419437.138 998 420.277693 -------------+-----------------------------Total | 472938.562 999 473.411974 Number of obs F( 1, 998) Prob > F R-squared Adj R-squared Root MSE = = = = = = 1000 127.30 0.0000 0.1131 0.1122 20.501
-----------------------------------------------------------------------------hrwagely | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------school | 2.670268 .2366684 11.28 0.000 2.205843 3.134693 _cons | -15.25841 3.230216 -4.72 0.000 -21.5972 -8.919613 ------------------------------------------------------------------------------
Regression Language - Interpreting coefficients (2)
Y and X are continuous, Y in logs (regress log wages on years education) Interpretation of constant: The average log hourly wage of those with 0 years of school is 1.31989. Interpretation of coefficient on school: Each additional year of education is associated with about 10.61% higher hourly wages on average.
. reg lnw school Source | SS df MS -------------+-----------------------------Model | 84.5172482 1 84.5172482 Residual | 399.919216 998 .400720658 -------------+-----------------------------Total | 484.436465 999 .484921386 Number of obs F( 1, 998) Prob > F R-squared Adj R-squared Root MSE = = = = = = 1000 210.91 0.0000 0.1745 0.1736 .63303
-----------------------------------------------------------------------------lnw | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------school | .1061316 .0073079 14.52 0.000 .091791 .1204723 _cons | 1.31989 .0997434 13.23 0.000 1.124159 1.515621 ------------------------------------------------------------------------------
Regression Language - Interpreting coefficients (3)
Y continuous but X is a dummy (regress wages on dummy for finished college or did not finish college) Interpretation of constant: The average hourly wage of those who didnt finish college is $15.62. Interpretation of coefficient on college: Individuals who finish college make an average of $16.70 more than individuals who do not finish college. Note this is just like comparing sample means between two groups (X=1 vs. X=0)
. reg hrwagely college Source | SS df MS -------------+-----------------------------Model | 57301.8122 1 57301.8122 Residual | 415636.75 998 416.469689 -------------+-----------------------------Total | 472938.562 999 473.411974 Number of obs F( 1, 998) Prob > F R-squared Adj R-squared Root MSE = = = = = = 1000 137.59 0.0000 0.1212 0.1203 20.408
-----------------------------------------------------------------------------hrwagely | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------college | 16.69938 1.423666 11.73 0.000 13.90566 19.49311 _cons | 15.61962 .7653444 20.41 0.000 14.11775 17.12149 ------------------------------------------------------------------------------
Regression Language - Interpreting coefficients (4)
Y and X are both dummies (regress female (1 if female, 0 otherwise) on dummy for finished college or did not finish college) Interpretation of constant: 47.96 percent of those who did not go to college are female. Interpretation of coefficient on college: Individuals who finish college are 2.21 percentage points more likely to be female compared to individuals who do not finish college.
. reg female college Source | SS df MS -------------+-----------------------------Model | .100575319 1 .100575319 Residual | 249.703425 998 .250203832 -------------+-----------------------------Total | 249.804 999 .250054054 Number of obs F( 1, 998) Prob > F R-squared Adj R-squared Root MSE = 1000 = 0.40 = 0.5262 = 0.0004 = -0.0006 = .5002
-----------------------------------------------------------------------------female | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------college | .0221239 .034895 0.63 0.526 -.0463521 .0905999 _cons | .4796062 .0187591 25.57 0.000 .4427944 .516418 ------------------------------------------------------------------------------
Regression Language - Interpreting coefficients (5)
Y continuous but X is a dummy, control for other variables. Interpretation of constant: The average hourly wage of those who didnt finish college, who are males, and who are zero years old is $8.39 (awkward!). Interpretation of coefficient on college: Holding gender and age constant, individuals who finish college make an average of $15.66 more than individuals who do not finish college. Interpretation of coefficient on age: Controlling for college completion and gender, each additional year of age is associated with 26.5 cents higher hourly wages.
. reg hrwagely college female age Source | SS df MS -------------+-----------------------------Model | 75993.9497 3 25331.3166 Residual | 396944.612 996 398.538768 -------------+-----------------------------Total | 472938.562 999 473.411974 Number of obs F( 3, 996) Prob > F R-squared Adj R-squared Root MSE = = = = = = 1000 63.56 0.0000 0.1607 0.1582 19.963
-----------------------------------------------------------------------------hrwagely | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------college | 15.65782 1.412692 11.08 0.000 12.88563 18.43002 female | -6.00112 1.263422 -4.75 0.000 -8.480395 -3.521846 age | .2655714 .0532771 4.98 0.000 .1610231 .3701196 _cons | 8.390394 2.238877 3.75 0.000 3.996937 12.78385 ------------------------------------------------------------------------------
Regression Inference
Note t-statistics for hypothesis tests are calculated for you by Stata - test the hypothesis that the particular coefficient is equal to zero. Know what all this output means!
. reg hrwagely college female age Source | SS df MS -------------+-----------------------------Model | 75993.9497 3 25331.3166 Residual | 396944.612 996 398.538768 -------------+-----------------------------Total | 472938.562 999 473.411974 Number of obs F( 3, 996) Prob > F R-squared Adj R-squared Root MSE = = = = = = 1000 63.56 0.0000 0.1607 0.1582 19.963
-----------------------------------------------------------------------------hrwagely | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------college | 15.65782 1.412692 11.08 0.000 12.88563 18.43002 female | -6.00112 1.263422 -4.75 0.000 -8.480395 -3.521846 age | .2655714 .0532771 4.98 0.000 .1610231 .3701196 _cons | 8.390394 2.238877 3.75 0.000 3.996937 12.78385 ------------------------------------------------------------------------------
Regression and prediction
Regression models can be used for prediction
How do earnings vary with education and experience, for different demographic (race and gender) groups? Multivariate regression: Yi=a+bXi +cEi+ dRi+f Gi+ei
Use R2 for evaluating fit of the model. This is just the square of the correlation between Y and predicted values of Y and shows how well the model describes the data.
Causality
Causality is measured as difference in two potential outcomes: the outcome one would get if exposed to the treatment minus the outcome one would get if exposed to the control. Effects of causes always are defined relative to other causes. Statistical solution: ATE = E[Yt] - E[Yc] = E[Yt | S = t] - E[Yc | S = c] if S is independent of the potential outcomes (which would be true if S is randomly assigned). Only in this situation will a simple comparison of sample means provide information about causality - why? The key is whether the control group provides a reasonable proxy for the counterfactual outcome of the treatment group (and vice versa): does their experience capture what would have happened to the treatment group if they had not been treated? Randomization ensures there are no other differences between the two groups on average, so differing treatment status is the only thing that can explain different outcomes. Had the treatment group not been treated, they would have had the same outcome as those in the control group. Not everything can be a cause. Well-posed questions satisfy SUTVA.
Effect of a cause should be the same no matter how it is given One persons outcome doesnt depend on the treatments given to others
When do regression coefficients represent causal effects
Consider an example: Whats the effect of education (X) on earnings (Y)? We suspect education is correlated with , so if we regress only Y on X we will have omitted variables bias. One omitted variable might be race (Z=1 if white, 0 otherwise) - for a variety of factors, race may be related to educational attainment and have an independent association with earnings.
Results of a regression ignoring Z
Do we estimate the true (partial) effect of X on Y? If we omit Z, we worry that bOLS might be biased.
. reg lnw coll Source | SS df MS -------------+-----------------------------Model | 84.4818412 1 84.4818412 Residual | 399.954623 998 .400756136 -------------+-----------------------------Total | 484.436465 999 .484921386 Number of obs F( 1, 998) Prob > F R-squared Adj R-squared Root MSE = = = = = = 1000 210.81 0.0000 0.1744 0.1736 .63305
-----------------------------------------------------------------------------lnw | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------coll | .6412066 .0441628 14.52 0.000 .554544 .7278691 _cons | 2.553667 .0237413 107.56 0.000 2.507079 2.600256 ------------------------------------------------------------------------------
Diagnosing an OVB Problem (1)
1) Is X related to Z? If education is related to other variables that affect wages, bOLS may pick up indirect effects of those variables.
. reg white coll Source | SS df MS -------------+-----------------------------Model | 4.74167091 1 4.74167091 Residual | 214.982329 998 .215413155 -------------+-----------------------------Total | 219.724 999 .219943944 Number of obs F( 1, 998) Prob > F R-squared Adj R-squared Root MSE = = = = = = 1000 22.01 0.0000 0.0216 0.0206 .46413
-----------------------------------------------------------------------------white | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------coll | .1519085 .0323782 4.69 0.000 .0883714 .2154456 _cons | .6300985 .0174061 36.20 0.000 .5959417 .6642552 ------------------------------------------------------------------------------
Diagnosing an OVB Problem (2)
2) Is Z related to Y? If we omit something that doesnt affect Y, no problem. Otherwise, weve got trouble.
. reg lnw coll white Source | SS df MS -------------+-----------------------------Model | 94.3514277 2 47.1757138 Residual | 390.085037 997 .391258813 -------------+-----------------------------Total | 484.436465 999 .484921386 Number of obs F( 2, 997) Prob > F R-squared Adj R-squared Root MSE = = = = = = 1000 120.57 0.0000 0.1948 0.1932 .62551
-----------------------------------------------------------------------------lnw | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------coll | .6086581 .0441149 13.80 0.000 .5220893 .6952269 white | .2142635 .042661 5.02 0.000 .1305479 .297979 _cons | 2.41866 .0356772 67.79 0.000 2.348649 2.488671 ------------------------------------------------------------------------------
Whats the effect of OVB in this case?
It looks like Z belongs in our model for Y. A better model of earnings might look like: Yi = + Xi + Zi + I Regressing only Y on X will yield a biased bOLS
Effect of X on Z
dY bols = = " + #$ dX
Whats the sign of the bias?
Effect of Z on Y
Mutiple Regression approach: controlling for sources of OVB
If we have data on Z, then we can include Z in our regression model and estimate the partial effect of X on Y (holding Z constant). (Also the partial effect of Z on Y, holding X constant). We now estimate , , and from: Yi = + Xi + Zi + I
. reg lnw coll white Source | SS df MS -------------+-----------------------------Model | 94.3514277 2 47.1757138 Residual | 390.085037 997 .391258813 -------------+-----------------------------Total | 484.436465 999 .484921386 Number of obs F( 2, 997) Prob > F R-squared Adj R-squared Root MSE = = = = = = 1000 120.57 0.0000 0.1948 0.1932 .62551
Est() Est() -----------------------------------------------------------------------------lnw | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------coll | .6086581 .0441149 13.80 0.000 .5220893 .6952269 white | .2142635 .042661 5.02 0.000 .1305479 .297979 _cons | 2.41866 .0356772 67.79 0.000 2.348649 2.488671 -----------------------------------------------------------------------------Est()
What just happened?
bOLS
(1) reg y x: E[earningsi|colli] = 2.554 + .641 colli
(2) reg z x: E[whitei|colli] = .630 + .152 colli
(3) reg y x z: E[earningsi|colli,whitei] = 2.419 + .609 colli + .214 whitei NOTICE: bOLS in (3) is lower than in (1): .641-.609 = .032 lower In (1): bOLS = + , so bias =
bias = .152*.214 = .032
A diagram of whats going on:
Yi = " + #X i + $Z i + % i Z i = + "X i + #i
education
Direct effect of Education on earnings
X
"
Y
earnings
!
Difference in race by education
" !
Z
Race
! !
"
Effect of race on earnings
! !
dY bols = = " + #$ dX
When is there no omitted variable bias?
Effect of X on Z
dY bols = = " + #$ dX
Effect of Z on Y
bOLS will be unbiased if: =0 (X and Z are uncorrelated - changes in X arent associated with changes in Z) =0 (Z doesnt have any effect on Y - so Z may vary with X, but this doesnt have any effect on Y) If additional variables are controlled for, then think about whether X and Z are related and Z affects Y holding the other variables constant.
Randomized Controlled Trials
Individuals are assigned randomly to treatment and control group This research design lends itself to causal analysis by ensuring groups differ only by their treatment status on average: the treatment should be independent of other determinants of outcomes, so simple comparisons of average outcomes can yield estimates of the causal effects. The key assumption (that groups are the same on average at the time of randomization) is partially testable: we can check whether this is true using the data we have on characteristics measured before the program started. This is the most convincing approach to address the FPCI, but ultimately we must assume that the unobserved things that influence outcomes are not correlated with treatment status in order to infer a causal effect.
Statistical Control - Multivariable Regression
In situations where the treatment isnt randomly assigned, its likely that treatment and control groups differ in many ways that may affect outcomes To avoid bias, one approach is to control for these differences using regression. Multivariate regression coefficients have causal interpretation if the treatment is not correlated with other determinants of the outcomes (I.e. the error term of the regression equation) once other control variables are held constant. This may be likely in cases where other determinants of outcomes are well known and are all included in the regression as control variables (e.g., Munnel et al. study) In many cases, however, it is difficult to control for all relevant variables so bias is likely to remain (see DiNardo and Pischke or LaLonde studies for illustrations).
The exam
Will begin exactly at 8:40am: arrive early. Might be difficult. Dont worry about it.
The exam
Will begin exactly at 8:40am: arrive early. Might be difficult. Expect a few hard questions - do your best but keep moving.
The exam
Will begin exactly at 8:40am: arrive early. Might be difficult. Expect a few hard questions - do your best but keep moving. Bring only calculator & something to write with. Use a calculator only for basic math operations. No notes. All readings on the syllabus are fair game. Most of the exam will be conceptually similar to PS and previous exams.