This preview shows page 1. Sign up to view the full content.
Unformatted text preview: Experiments and Experiments and Observational Studies Today’s topics Two broad categories of studies about relationships: experimental and observational studies
Terminology Tools for critically evaluating studies Experiments Experiments Terms in Experiments Terms in Experiments Treatment: an explanatory variable that is randomized and assigned Say we are interested in the ads’ effects on individuals Randomly expose people to political ad or not Exposure to the ad is the treatment Terms in randomized experiments Terms in randomized experiments Control group: group of experimental units that does not receive the treatment serves as a baseline, showing what would have happened if individuals were not exposed to the treatment Should be treated identically to the treatment group, aside from receiving the treatment E.g. not exposed to political ad Interacting variable: a variable that is related to differing impacts of the treatment variable The ad may have different effects for Republicans and Democrats Party identification, then, is an interacting variable GOTV Example Green and Gerber (2004) randomly assign households to be contacted by a face to face appeal to turn out to vote in a local campaign Treatment: volunteers went door to door to houses in the treatment group, telling people to vote in an upcoming election Control group: No one attempted to contact houses in the control group Outcome variable: whether or not people in the household voted Interacting variable: whether or not household contained habitual voters GOTV Results GOTV Results Face to face appeals had large impact on turnout Turnout was highest among those who participated in elections somewhat frequently GOTV Experiment v. Observational GOTV Experiment v. Observational Previous studies were not experimental; they just conducted a survey after an election These studies had compared turnout among people who have been contacted by someone asking them to turn out to those who had not Researchers in these previous studies, then, did not administer the treatment; it occurred naturally What is wrong with these previous, nonexperimental studies? Confounding variable: one whose effect on the response cannot be separated from the explanatory variable For example, people who are contacted by the parties naturally may be more likely to vote to begin with! This makes the effects of these contacts look larger than it actually is Confounding variables are a Confounding variables are a problem for observational studies Basic Principles in Experiments Basic Principles in Experiments Randomization: assigning treatments to participants that give everyone a specified chance of receiving any particular treatment To balance confounding variables across treatments Because in the Green and Gerber (2004) studies, households are randomly assigned, the individuals in the treatment and control should be similar with respect to interest in politics, etc. Placebos: treatments made to look like the real thing with no “active ingredient”
eliminate the confounding effects of suggestion In medical experiments, participants in the control group are given something that is similar to the drug, but without the active ingredient Nonmedical experiments as well!! Example: Agenda setting Example: Agenda setting Iyengar and Kinder experiment Authors care about whether or not news coverage of an issue increases the perceived importance of that issue Treatment group: watched a news show with fake stories inserted; these were doctored to emphasize certain issues Control group: watched news program which excluded the fake news portion The groups were compared on their attitudes about the importance of each topic Why a placebo group? Basic principles of experiments Basic principles of experiments Field v. Lab experiments Field experiments are performed in as naturalistic a setting as possible, so that we know that the results can be applied to other similar settings E.g. GOTV study Lab experiments are conducted in a lab, which may be somewhat artificial We don’t know if the results would apply to other, real world settings E.g. agenda setting experiment Double blind experiment: an experiment in which neither the participants nor those collecting the measurements know who received which treatment Most drug trials Single blind: only one of the groups kept blind Pairing or blocking means exposing similar (or the same) individuals to both the treatment or control
eliminates extraneous variability, making estimates more accurate In get out the vote drives, usually individuals with the same turnout history are matched before randomizing them Observational studies Observational studies Observational studies are similar to experiments, except that the researcher does not manipulate the treatment Instead, the treatment condition varies naturally across the population E.g. we collect data after the election, and find out who was contacted or not Observational Study Observational Study Kull et al. (2003) Does watching Fox news have an effect on attitudes about the war? Survey asks where individuals get their news (Fox News, NPR, etc) Also asks about beliefs, including beliefs about Iraq war Compares those who report watching Fox news to those who do not Those who watch Fox news are more likely to have misperceptions about the war (e.g. evidence of close ties between alQaeda and Hussein) Observational Studies Observational Studies
The main problem is that observational studies can’t exclude the effect of confounding variables Therefore, we should be careful about drawing causal conclusions from these studies Some sophisticated statistical techniques for dealing with this An unusual observational study An unusual observational study What about this study: Doherty et al. (2006) compare lottery winners in terms of political attitudes Lottery winners who won $100,000 or less were less likely to favor the elimination of the estate tax than those who won $5 million or more Experimental or observational? Do you trust the results of this study? Survey experiments Survey experiments It is possible to set up an experiment that looks like a survey In this case, some respondents get one version of the survey, others get another version Can check quality of brand names, question wording effects, persuasion effects Survey from start of class is an example Confounding variables Difficulties and disasters in Difficulties and disasters in experiments Interacting variables Solved with randomization Placebo, Hawthorne, experimenter effects Not a major problem if interested in average effects Can be measured External validity (generalizability) Incorporate Placebo conditions; conceal the nature of the study Generalizability Generalizability Ecological validity: whether or not experimental conditions reflect the impact the variables have in everyday life Also, other factors matter for generalizability Do participants reflect the population about whom we are interested? Difficulties and disasters with Difficulties and disasters with observational studies Two major ones: Confounding variables
Try to measure Generalizability (again) or “Extending the results inappropriately” Does the sample reflect the population we are interested in? Note difference between random sample and random assignment!!! Pp. 98100 Any questions? ...
View Full Document
This note was uploaded on 01/27/2011 for the course COM 200 taught by Professor Tamborini during the Fall '09 term at Michigan State University.
- Fall '09