Causal Claims from User Studies Introduction to Computational Thinking and Data Science Gale Lucas [email protected]
Today’s Topics 1. Correlational research 2. Comparing correlational research to experiments 3. Consuming statistical results 4. Ensuring internal validity 2
Correlational research 3
Types of user studies User studies Experiment AB Testing Between-subjects Within-subjects AB+ (one-way) design Between-subjects Within-subjects Factorial design Between-subjects Within-subjects Mixed design 4 Correlational research
Correlational vs. experimental Correlational Measure IV and measure DV Experimental Manipulate IV and measure DV
IVs and DVs IV: Independent Variable Variable that you think causes the effect on another variable Variable is measured in correlational research DV: Dependent Variable The outcome variable (you think it has effect on) Variable is also measured in correlational research PREDICTOR OUTCOME
Two ways of asking a research question “Is NTB associated with social sensitivity?” “Does NTB cause heightened sensitivity?” Correlational Experimental NTB Sensitivity NTB Sensitivity Social sensitivi ty
Is NTB associated with sensitivity? Correlational research Picket, Gardner, and Knowles NTB scale DANVA: Number of facial expressions correctly identified r = .38 What does that mean?
Continuous IV (e.g., how much) Categorical IV (e.g, exp vs control) Continuous DV (e.g., how much) correlation or regression t-test or ANOVA Categorical DV (e.g., yes/no) logistical regression chi-square test or loglinear What Test Do I Run? Yes but…
Problems with correlational research Only shows there’s an association We can’t say NTB causes social sensitivity, why not …? It could be some other variable causing both Third variable problem Sensitivity could cause NTB Directionality problem NTB Sensitivity NTB Sensitivity Third Variable
Problems with correlational research NTB Social sensitivity Motivation Please experimenter …
Problems with correlational research NTB Social sensitivity
Comparing correlational research to experiments 13
Problems with correlational research …and there will always another third variable that you didn’t think of! So, to make a causal claim, we conduct experimental research! What makes an experiment? What do you have to do? Random assignment to conditions that only differ on (thus manipulate) one variable Remove confounds Internal validity!
Correlation vs. Causation
Experimental research Picket, Gardner, and Knowles What did they have to do here? Manipulate belonging needs Measure social sensitivity
Experimental research NTB causes an increase in social sensitivity! Experimental research is better!…?
Benefits of correlational research Real world outcomes that really matter Age-adjusted mortality rate Belonging External validity!
Problems with experimental research Hard to make manipulations powerful enough to impact these kinds of outcomes Would you really want to?
- Fall '17
- Statistics, Null hypothesis, Statistical hypothesis testing, Type I and type II errors, Variable, Random assignment