120_lecture2_fall09

120_lecture2_fall09 - ECO 120, Lecture 2 Jon Robinson UC...

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: ECO 120, Lecture 2 Jon Robinson UC Santa Cruz September 29, 2009 Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 1 / 68 Administration problem set 1 posted on Thursday, due 1 week from date of posting some other things there is no TA for this class, no sections there is an MSI, however George Davis, gsdavis@ucsc.edu the problem sets will have some Stata. We will cover what you need to know in class. Detail on how to access it are on the syllabus (we’ see it in class too) ll Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 2 / 68 Today cover empirical tools for the course talk a bit about how it relates to development should be mostly review, but please make sure you know this stu¤ Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 3 / 68 Next Time Ray, pp. 272-279, 489-504 (health poverty traps) if you want to plan ahead, after that we will start with Thomas, Duncan et al. (2006), “Causal E¤ect of Health on Labor Market Outcomes: Experimental Evidence.” mimeo, UCLA Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 4 / 68 Today stats overview What is a regression? Interpreting regression output Correlation vs. Causation Omitted variable bias Example: ‡ip charts Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 5 / 68 Program Evaluation In this class, we are interested in understanding what e¤ect a program has on some speci…c, measurable outcomes i.e. what e¤ect does providing textbooks have on test scores? Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 6 / 68 Program Evaluation Statistically: How do we come up with a speci…c number? How con…dent are we in that speci…c number? Logically (and most importantly): What does it mean? Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 7 / 68 Example: textbooks 50 schools Individual textbooks for each kid that they can take home Average test scores are 75/100 on a standardized exam 50 schools Few textbooks, kids must share Average test scores are 68/100 on same standardized exam Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 8 / 68 What is the e¤ect? Simple estimate of the e¤ect of textbooks: Average in schools w/ textbooks – average in schools w/out textbooks Here: 75 - 68 = 7 Later, we’ talk about the precision of this estimate (i.e. how ll con…dent are we that the e¤ect is really around 7? How con…dent are we that it’ even > 0?) s But for now, how con…dent are we that the estimate of 7 means something? Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 9 / 68 Big question: what does this mean? Is this the di¤erence the e¤ect of giving out textbooks? Did the textbooks cause this di¤erence of 7 points? Why might we worry that the di¤erence is not really due to the textbooks themselves? hint: are there any other di¤erences that might exist between schools that have textbooks and those that don’ Are these di¤erence t? "important?" examples? Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 10 / 68 Possible issues Are schools with textbooks exactly the same as those without textbooks, except for the textbooks themselves? Are they richer? Poorer? Better students? Worse students? More involved parents? Less involved? Who knows! Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 11 / 68 Classic Example Compare crime rates in cities with big police forces compared to those with smaller police forces Where will crime be higher? -> Do police cause crime? this particular issue is called reverse causality goal of this class is to …nd evidence that doesn’ su¤er from these t problems Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 12 / 68 Regression Interested in relationship between 2 variables x and y The true relationship could be very complicated i.e. y = ln(x ) + 42x ^2 + 19x or could be impossible to capture in a simple equation But we’ mostly think about a very simple linear case in this class ll i.e. y = a + bx Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 13 / 68 Very simple case Interested in the following: test score = a + b textbook Let’ say textbook can only take 2 values s textbook = 1 if kid got an individual textbook textbook = 0 if kid didn’ t textbook is called a "dummy" or indicator variable Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 14 / 68 Very simple case What is the test score for a kid that didn’ get a textbook? t since test score = a + b textbook , and textbook = 0 for this kid, his test score is just a what about a kid that got a textbook? for her, test score = a + b 1 = a + b di¤erence is a + b easy! a=b in simple example, a = 68, b = 7, a + b = 75 Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 15 / 68 but... Unfortunately, too simple We usually assume that relationship includes some error as well i.e. y = a + bx + ε ε is an error, which is independent of x (don’ worry about the t statistical meaning of that for now - it just means that it’ not s dependent on x in any way) we usually assume that the average of ε is 0, but that it varies. Sometimes it’ high, sometimes it’ low s s Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 16 / 68 meaning of error term In example, a kid with a textbook won’ always get a 75 t Might get a 74 one day, a 76 the next day Might even get a 92, or a 100 sometimes Or a 12 or a 26 On average, he will get a 75, but he could get higher or lower from one day to the next Why? having a good or bad day Test is slightly easier or harder on certain days There are many things that also in‡uence the test score besides textbooks (parent help, quality of school, even IQ). Let’ not worry s about those just yet. Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 17 / 68 average Now, this means that the di¤erence b/w a kid with a textbook and another is not always 7. Could be 6 or 8 or 20 or -5 But, on average, the di¤erence will still be b = 7. Logically, when are we likely to get a good estimate of b ? when the error term is big? or small? when the number of observations is big? or small? Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 18 / 68 Law of Large Numbers (loosely stated) If we take a large sample of individuals with textbooks, the average test score will get arbitrarily close to 75 as the sample size gets bigger. Similarly the mean test score for kids without textbooks will get arbitrarily close to 68 as the sample size gets bigger. The di¤erence will get arbitrarily close to 7 the math behind this is not that important - but the logic is Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 19 / 68 Bit of math in math, for the non-textbook schools, we’ computing (for N kids) re (y1 + y2 + ... + yN )/N this is equal to (a + a + ... + a)/N + (e1 + e2 + ...eN )/N the last term becomes close to 0 as N get big we’ left with aN /N = a re on average, we get it right Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 20 / 68 Bit of math in the textbook schools, we’ computing re (a + a + ... + a)/N + (b + b + ... + b )/N + (e1 + e2 + ...eN )/N or aN /N + bN /N = a + b on average, we get it right here, too Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 21 / 68 Di¤erence so since we get both right on average, we get the di¤erence right too In a big sample, the average will get very close to the truth Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 22 / 68 But how close is close? Unless the sample is in…nitely large, we’ not likely to get exactly the re right answer (this is because (e1 + ... + eN )/N is not going to be exactly 0) When are we more likely to get a close estimate? When sample size (N) is very big When variance of the error is very small Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 23 / 68 Central Limit Theorem (very loosely stated) p we can construct something called the standard error: σ / N σ = standard deviation of y , which is the standard deviation of e in this example std. error is small when σ is small or N is big it turns out that the average big samples y1 +y2 +...+yN N has a normal distribution in p with a std. deviation equal to the std. error σ/ N Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 24 / 68 Normal distribution Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 25 / 68 Properties of Normal With 95% probability, draw we get will be within +1.96 SE and 1.96 SE of the mean With 90% probability, draw we get will be within +1.67 SE and 1.67 SE of the mean This is how “close” we get to the truth Not important where this comes from for this class Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 26 / 68 Regression So let’ go back to the regression problem s We’ interested in estimating re y = a + bX + e This is …tting a line to the data (x can now be continuous rather than just 0 or 1) Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 27 / 68 Regression Let’ say the true relationship is y = 25 + 3x , and x ranges from 0-25 s If there were no errors, relationship would look like Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 28 / 68 Regression Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 29 / 68 Error terms Of course, we need to worry about the errors Observed empirical relationship is y = 25 + 3x + e e is an error with 0 mean the relationship between y and x depends on how big the errors are Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 30 / 68 Diagram, std. deviation=10 Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 31 / 68 std. deviation=20 Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 32 / 68 std. deviation=30 Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 33 / 68 Fitting lines: Ordinary Least Squares (OLS) these graphs all have the same slope, but the …t varies The method that we use to …t these lines is called Ordinary Least Squares. It minimizes the sum of squared di¤erences between the observed value of y and the …tted value We will not talk much about how this works Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 34 / 68 Fitting lines: Ordinary Least Squares (OLS) The presence of these errors means that you’ not going to get re exactly the right estimate Sometimes the line you …t will be too steep, sometimes too ‡at (purely because of the errors) Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 35 / 68 estimates 1 Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 36 / 68 estimates 2 Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 37 / 68 estimates 3 Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 38 / 68 Sampling variation We could also get estimates that are really far o¤, like b=10 or b=-5 But as the sample size gets big, it becomes less and less likely that this happens what do we expect to get? what is likely? what is less likely? Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 39 / 68 Law of Large Numbers, CLT In fact, the law of large numbers applies here as well: As N gets very big, the estimated b will get arbitrarily close to the true b the Central Limit Theorem applies too As N gets very big, the estimated b will be distributed as a normal distribution around the true b let’ call b the estimated slope of the line (our best guess by OLS). s b The true slope is b Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 40 / 68 Law of Large Numbers, CLT Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 41 / 68 CLT with regression (don’ worry about details) t with 95% probability, the estimated value of b is between +1.96*SE b and -1.96*SE of the true b in math Pr(b b Pr(b b b + 1.96 SE ) + Pr(b b 1.96 SE b 1.96 SE ) = 0.05 re-arrange i.e. with 95% probability, the true b is within + or - 1.96*SE of the estimated b b b 1.96 SE , b + 1.96 SE ) a 95% con…dence interval. we call (b b Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 42 / 68 b ) + Pr(b + 1.96 SE b b ) = 0.05 Hypothesis testing (basic) We don’ know what the true value of b is, we only know our t estimated b b We can make a guess as to where the b is, though, from the con…dence interval Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 43 / 68 testing, basic (don’ worry about details) t say we want to test whether b = 0 this means that: if b > 0, then b + 1.96 SE must be > 0. Then the question is b b whether b 1.96 SE > 0 b if b < 0, then b 1.96 SE must be < 0. Then the question is b b whether b + 1.96 SE < 0 b b b SE b b SE or that if b > 0, then b 1.96 SE > 0 implies that b b if b < 0, then b + 1.96 SE > 0 implies that b b > 1.96 < 1.96 Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 44 / 68 testing, basic (don’ worry about details) t b so a result is statistically signi…cant if j SE j > 1.96 compare the t-stat to 1.96 to …gure out if it is statistically signi…cant If t > 1.96 (or 2), then we know that with 95% probability, the real b is in the interval (b 1.96 SE , b + 1.96 SE ) b b this means that there is less than a 5% chance that we could have gotten a b that big if the true b = 0 b b b SE is called the t-statistic b Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 45 / 68 con…dence interval Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 46 / 68 Idea other ways of expressing this: the p-value: what is the probability that we could have observed a b b that big? for a result to be signi…cant, the p-value for b = 0 must be less than 0.05 the con…dence interval: a CI is a range from b b +1.96 SE we know that with 95% prob, the true b is in this range. So the odds of anything out of that range being the true value is less than 5% 1.96 SE , b b Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 47 / 68 to mechanically test for signi…cance 95% CI does not include 0 p-value less than 0.05 t-stat greater than 1.96 (i.e. 2) Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 48 / 68 example: health status on income EXAMPLE: dependent variable = health Regression 1 Coefficient Standard Error t-stat 95% CI Income 4.52 2.12 2.132 (0.37,8.70) Regression 2 Coefficient Standard Error t-stat 95% CI Income 4.52 3 1.507 (-1.36, 7.47) p-value 0.033 p-value 0.132 Significant? YES Significant? NO Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 49 / 68 conclusion Those are the basics Sometimes also interested in testing other hypotheses, like b=1 or b=3.478 Principle is the same Take econometrics to understand at deeper level Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 50 / 68 Now what? these are the basics. More important is understanding what this means. Correlation vs. Causation Omitted variable bias (very important) Example 1: ‡ip charts Example 2: text books Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 51 / 68 Causality vs. Correlation we’ more interested in the meaning of coe¢ cients than the re mechanical relationship For example, assume that we run a regression that shows income is associated with better health Does this mean that increasing income causes better health? Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 52 / 68 Causality vs. Correlation We don’ know t Reverse causality? Better health may increase income But let’ explore another more general issue: omitted variable bias s this is crucial to this class, perhaps the most important topic of all Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 53 / 68 Omitted Variable Bias Health surely depends on many factors other than income For instance, it may be that health = a + b income + c education + ε but we stupidly forgot about education and ran health = a + b income + ε the question is whether this omission will make the estimated b systematically wrong (i.e. not just randomly too high or too low but simply wrong) In which direction? Too big or too small? not going to be enough to just say that it’ wrong - throughout the s class we will have to argue why we believe it’ too high or too low s Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 54 / 68 Omitted Variable Bias the direction of the bias depends on 2 things How is education related to health? How is education related to income? Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 55 / 68 signing bias Probably more education -> better health Probably more income -> more education (or more education -> more income - we don’ care about the causality of that relationship t right now) Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 56 / 68 So... A person with high income will have more education. So their health will be better for BOTH reasons If we don’ include education as a control, we miss that t Our estimated b is o¤ Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 57 / 68 Bias In fact, our estimate is too big in this case we say that our estimate is UPWARD biased This is one example of omitted variable bias go through the logic again to make sure you understand. Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 58 / 68 slide It would be DOWNWARD biased if More education -> worse health OR More education -> lower income (Ph.D...) How? stress from high pressure jobs? what if both more education -> worse health, and more education -> lower income? Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 59 / 68 Formula if truth is y = a + bx1 + cx2 + ε but you run y = a + bx1 + ε and use this to estimate OLS coe¢ cients b and b a b formula for OVB then the bias of your estimate b is the di¤erence between this b coe¢ cient and the truth (b ) sign depends on c corr (x1 , x2 ) i.e. independent e¤ect of x2 times the correlation between x1 and x2 this is IMPORTANT Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 60 / 68 Fixing omitted variable bias Add controls If you just control for everything, there will not be any omitted variable bias, by de…nition. Problem: can we control for everything? some things that we may not have data on (IQ?) some things that may be impossible to observe ("unobservable") culture towards learning? How do you measure that? Other techniques Randomized experiment (gold standard) Later on: natural experiment this class is not a big fan of the "kitchen sink" method Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 61 / 68 Du‡o, Glennerster, and Kremer: Randomization Toolkit this paper is technical at times, but provides a formal framework for thinking about this stu¤ continue with old example: we want to look at e¤ect of a program to give out textbooks a regression in this case would be to look at di¤erences in average test scores between schools that have textbooks and those that do not would this regression be likely to show causality? Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 62 / 68 causality there are some variables that should be in there: income, parent’ education, IQ, etc. s what if we did a randomized experiment? what would be di¤erence in average income? what would be di¤erence in parent’ education? s what would be di¤erence in IQ? Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 63 / 68 slide more formally, as in paper, let: YiT be outcome for individual i if he got a textbook YiC be outcome for individual i if he didn’ get a textbook t problem: can never observe YiT YiC for 1 individual can only observe YiT for students in schools with textbooks can only observe YiC for students in schools w/out textbooks Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 64 / 68 slide regression is to look at average over all kids E [YiT for kids in textbook (T) schools] E [YiC for kids in non-textbook (C) schools] think about E [YiC for kids in textbook (T) schools] what does that represent? Can we observe this? Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 65 / 68 slide add and subtract E [YiC for kids in textbook schools], to get E [YiT for kids in T schools] E [YiC for kids in T schools] E [YiC for kids in T schools]) E [YiC for kids in C schools] ( …rst term: di¤erence in outcomes for kids with and without textbooks in textbook schools program e¤ect second term: di¤erence in outcomes for kids that did NOT get textbooks in C and T schools) why should these be di¤erent? in terms of the regression, what are we talking about here? this is called the selection bias - C and T schools may be di¤erent Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 66 / 68 Now what does this mean? For our estimate to be equal to the treatment e¤ect, we want selection bias=0 Is this likely to hold or not? Why? another way of looking at what we’ seen earlier this lecture. ve Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 67 / 68 Randomized experiment what is (E [YiC for kids in C schools] in a randomized experiment? E [YiC for kids in T schools]) what is di¤erence between kids in the 2 types of schools, in the absence of treatment? this is the main reason why we do experiments. we’ come back later to this paper to discuss attrition, spillovers. So ll just take a glance at that stu¤ too. Jon Robinson (UCSC) ECO 120, Lecture 2 September 29, 2009 68 / 68 Example 1: Glewwe et al (2004) • Look at effects of flip charts on test scores • Flip chart – Visual aid for learning – Might be useful with so few textbooks there • Why might regressing a flip chart on test scores create a bias? 1 • • • • Randomized controlled project 89 treatment schools 89 control schools Charts – 2 science – 1 math – 1 health – Wall map 2 3 4 5 6 Example 2: Glewwe et al. (2007): textbooks • Background: very few kids have textbooks in poor rural Kenyan schools • Before project: – 80% of students had less than 1 English book per 20 students (provided by school) – 78% had less than 1 math book per 20 students – 89% had less than 1 science book per 20 students 7 • Parents will sometimes buy books • ¼ kids have books in this way 8 Background Lit. on Textbooks • Many studies find positive impacts of textbooks on test scores in nonexperimental settings • As we discussed, we might be worried about the accuracy of these estimates due to omitted variable bias 9 • Selected 100 of 333 schools in Busia & Teso Districts for School Assistance Program (SAP) • 25 got books • Early 1996 – English: Grades 3-7 – Math: Grades 3, 5, & 7 – Science: Grade 8 • Early 1997 – Math: Grades 4 & 6 – Agriculture: Grade 8 10 • Cost $2-$3 each -> very expensive for poor people • Ratio of textbooks per student: – 0.6 for English & science – 0.5 for math • Grades 3-5 could not take home • Grades 6-8 put in pairs to share (alternate days) 11 • In 1997, another 25 were given grants of $2.65 per student – Spent 43% on textbooks – 46% on construction – Remainder on equipment & supplies 12 • Year 1: – Compare 25 textbook schools to 75 control • Year 2: – Compare 25 textbook schools & 25 grant schools to 50 control • Administered their own tests to students 13 14 15 16 17 18 19 • Why might the impact of the school books be negative? • What might we conclude by looking at student-owned textbooks? • This is why randomization is important 20 Effect for best students • But the program did have SOME impacts 21 22 • Another way of showing this • Run test score = a + b*textbook + + c*pre-test score + d*textbook*pre-test score + e • If best students benefitted the most, what should be the sign of d? • What should be the sign of c? 23 24 Conclusion • Both papers get different results experimentally than retrospectively • Points to how hard it is to control for unobservable factors • Textbooks also show that the overall impact might be small but that the impact might be big for some subgroups 25 School inputs • More generally, these results suggest that increased inputs are not a panacea • In Kenya, students are often taught in their 3rd language (English), classes are big, teachers often don’t show up, etc. 26 ...
View Full Document

This note was uploaded on 01/12/2010 for the course ECON 120 taught by Professor Robinson during the Fall '08 term at UCSC.

Ask a homework question - tutors are online