This preview shows page 1. Sign up to view the full content.
Unformatted text preview: SE 10 Research Design
Lecture 3 continued Threats to Internal Validity 1 Real World Your decision H0 True A does not affect B You don't find evidence that it does CORRECT P = 1 ; called "Power" A does not affect B Your study shows an effect H0 REJCTED TYPE I ERROR P = ; researcher sets H0 False A does affect B You don't find evidence of the effect TYPE II ERROR P = A does affect B Your study shows an effect CORRECT P = 1 H0 NOT rejected 2 What is ? Statistics come with a "pvalue" Researcher decides what probability he/she is willing to accept; typically .05 or .01 "Set" to that level If p will reject the null hypothesis
3 Pvalue is the probability that the findings from your sample are a result of error What is ? the likelihood of accepting the null hypothesis when it is wrong 1 is the "power" of a test: likelihood of finding a true effect is affected by many things: sample size, variability, , effect size If you set higher (more type I error) will also be higher (less type II error) Trade off between type I and II errors
4 Threats to Internal Validity Time Threats: Group Threat Maturation History Instrumentation Test Reactivity Selection Group and Time Threats Mortality/Attrition Selection by Time Selective attrition Regression Towards the Mean
5 Maturation Time threat: something happens between measurements The subjects you are measuring are naturally changing What you're studying (program, intervention, variable of interest) is not causing the change it's just happening
6 History Time threat: something happens between measurements An event occurs between measurements that changes the score Natural disasters, political changes, major case decisions The major event causes the change in measurement, not what you are studying
7 Instrumentation Time threat: something happens between measurements Observers become more trained/better or they become fatigued/worse Physical measurements: calibration changes Measurement error in a predictable direction causes the change in score, not what you're studying
8 Test Reactivity Time threat: something happens between measurements The test you give subjects at the beginning affects their answers later Can happen because it clues them in to what you're studying or what your hypothesis is Change in scores is due to test reactivity, not what you're studying
9 Selection Group Threat: the subjects are causing the problem There is something different about your groups to start with Intervention/variable of interest might have an impact for one group but not the other Or initially different levels of the dependent variable mask results
10 Selection by time A special case of the selection threat You start with groups that are different on an important variable This difference gets larger or smaller over time Can either mask or exaggerate results 11 Mortality/Attrition Group threat: the problem is with your subjects Time threat: mortality not an issue with measurements only at one time Subjects drop out of the study Could be random, but usually not the subjects who drop out may be different Loss of subjects changes second measurement and can mask or exaggerate results
12 Selective Attrition Special case of mortality/attrition Subjects from one group drop out at a higher rate than subjects in another Often due to negative results in the high drop out group These negative results are less observable due to the loss of subjects
13 Regression Towards the Mean Group Threat: has to do with selection of subjects Time Threat: problem comes from repeated measurements Only an issue if subjects are selected based on a score before the study begins 14 Example of Regression Toward the Mean
Risk Level high High risk line target subjects above here For a study of a program targeting high risk boys O = T + E Error likely to be large mean O = T + E low Potential Subjects Error likely to be small
15 Measurement error and regression toward the mean For our high risk subject there is likely to be high measurement error Extreme scores have more error If measurement error is random a second measurement will likely have less error Makes it look like subject's score went down, when really all that changed was amount of measurement error
16 Measurement error and regression toward the mean
If the next measurement of the same true score could fall anywhere between the lines, there's a greater change it falls below the first observed score than above Observed score Range of scores including error True score high risk cutoff mean 17 Dealing with threats to internal validity Research design can help control for most of them Look for ways to make sure groups are equivalent Look for ways to isolate impact of variable of interest from impact of the threat 18 SE 10 Research Design
Lecture 4 Sampling and Experimental Design
19 Pick a concept something subjective that you would need directions to measure Subdivide types or categories of the concept Focus pick one category that you will measure Indicators things that help you identify the existence of your concept But first a review of conceptualizing Operational definition NOT causes, effects, or things associated with the concept 20 Review of operational definitions Gives exact guidelines for how a variable is to be measured If another researcher had your operational definition, they would collect data in the same way you intended Not just "murder rate," but "in 2006, what was the rate of criminal, nonaccidental deaths not related to self defense per 1,000 individuals in the population?"
21 Assignment review 1. Identify a concept: Success 2a. Subdivide: financial success, professional success, personal success, athletic success... Notice these are categories of success. NOT types of people: not successful teens, successful adults, men, women, etc. 22 Assignment review cont. 2b. Focus: personal success 2c. Indicators: 1) having a happy family life, 2) enjoying where you live, 3) satisfaction with how spare time is spent These indicators are how you would tell if someone has "personal success" or not They are NOT precursors to or causes of personal success (having a happy childhood might mean you're more likely to have personal success, but it is NOT an indicator of personal success) 23 Yet more assignment review 3. Operationalize: I'm going to operationalize "happy family life." Survey participants: "in the last seven days, how many times have you argued with your spouse or partner such that you raised your voice?" Survey: "thinking about the last month, have you and your partner participated in a hobby together?"
24 Assignment review operationalization cont. Look at public records: In the last year, have any of the following occurred? Death of a household member Divorce Illness of self or family member within household requiring hospitalization 25 Assignment review research questions 4. Research question: Does higher income cause personal success? Independent: income Dependent: personal success Research questions are NOT just descriptive take the form "does X cause Y?"
26 Midterm info Midterm on Wednesday August 20 About half multiple choice and half short response Closed note and closed book Review sheet to be posted on website We will do some review on Monday Bring a scantron, # 2 pencil, calculator 27 And finally... Sampling Sampling is the method you use to pick the subjects you study There are many approaches We'll start with some terminology Which you use is determined by your project one is not always best 28 Sampling terminology Population: All the potential subjects in your study depends on the purpose Enumeration: A list of all the potential subjects in larger scale projects getting this is rare Census: Study that includes all the members of the population Examples: all males, all UCI students, all US residents, DARE participants 29 More sampling terminology Sampling Frame: Incomplete list of all the possible subjects you are able to identify Sample: The group of subjects who actually participate in the study Element: One subject in the study Cluster: Subcategory your elements can fit into
30 Two main types of sampling Probability sampling: Method of selecting subjects where each element has the same chance of being included in the study, or the chance of inclusion is known Non probability sampling: Method of selecting subjects where the probability of each subject being included in the sample is not equal
31 Probability sampling Simple Random Sampling (SRS) Each subject has an equal chance of being selected Draw a sample using the entire population Helps deal with sampling bias A predetermined number of subjects are randomly picked from the population Random: no subject has a greater or less chance of being selected 32 Probability Sampling Systematic Sampling Used as a more realistic substitute for simple random sampling Instead of selecting truly randomly, will pick every nth subject in the sampling frame, starting with a randomly selected place Should still address bias issues 33 Probability sampling Stratified Sampling: used when there are smaller subgroups of the population that you are interested in Divide the population into groups (ex. by gender or race) Select an SRS from each group Can be proportional to the groups' numbers in the population or use "over sampling"
34 Probability Sampling Stratified Sampling Probability proportionate to size (PPS): The number of subjects from each subgroup is proportionate to the subgroup's portion of the population Used when you want to ensure proportional representation of groups Can still know the probability of selection within each subgroup
35 Probability Sampling Stratified Sampling Oversampling: Number of subjects not proportionate to group's population you pick a subgroup to specifically draw more subjects from than indicated by its population Used when you have small groups you want to make sure are represented Probability of selection within subgroup is known and equal 36 Probability sampling Multistage Cluster Sampling: Used when you want to conveniently sample subjects that are organized in groups Example you want to sample school children in the state.
Make a list of all school districts and take a SRS Make a list of the schools in the selected districts and take a SRS Take a SRS of the children in the selected schools 37 Nonprobability sampling Convenience sampling Subjects selected in this manner may be different on important characteristics than subjects who were not selected One of the weakest sampling designs
38 Typically uses volunteers respondents to ad or people you approach Nonprobability sampling Purposive sampling: you select the subjects based on certain characteristics you are looking for Still typically volunteers, but now more hand selected Can select to attempt to recreate population characteristics Or, can select specific populations 39 Nonprobability sampling Snowball sampling: a type of purposive sampling Frequently used when you are studying a population that is underrepresented or hard to contact You start with a few contacts and ask them to recommend other people they know with the characteristic of interest
40 Nonprobability sampling Quota sampling: you set the characteristics of the group you want to study Ex. 50% males and 50% females selected in a nonprobability frame. Decide you want your sample to be the same as the population on a certain demographic Population not broken into groups and sampled from you just keep selecting from a convenience sample until your quota is met
41 Getting a representative sample Representative: your sample is similar to the population on important characteristics Larger samples capture more of the population If you don't know much about the population, can you hand select a representative sample? Can you even know if you got a representative sample? 42 Probability sampling and representativeness Elements in the population will vary on important characteristics often in a normal distribution If each person has an equal chance of being selected, most samples should have a distribution similar to the population Often don't have an enumeration to randomly select from
43 The majority are closeish to the mean Few are very far from the mean Nonprobability sampling and representativeness Population will vary on their willingness to participate in research Volunteers/those out in public may be different on other important characteristics as well Hard to get a truly representative sample Sometimes, you don't want a representative sample
44 Why is sampling important? You are using your sample as a proxy for the population The results of your study should be applicable to the intended population Unknowingly selecting a sample that is different from the population leads to inappropriate generalizations
45 Sampling Error The difference between the true population value and the value predicted by your sample Could be due to Larger samples have less chance variation
46 Chance/random variations Biased or unrepresentative sample Population value is not correct What do we already know? We know our question We know and have defined our variables We know what to look for when we measure them We know other factors might influence the results We know who we want to study and how to find them
47 What is missing? How do we design the study? Is our study ethical? What to do when our question doesn't fit neatly into an experimental design? 48 True Experiments Experiments allow you to draw the strongest causal conclusions They account for most threats to internal validity Defining characteristic of a true experiment: multiple groups and random assignment to groups
49 A quick note about assignment to experimental groups Assignment to groups occurs after a sample has already been drawn. You can randomly assign subjects even if the sample was not randomly selected Random sampling selecting subjects randomly Random assignment putting subjects into experimental groups randomly
50 Experimental design notation
Random assignment: Observation: Treatment: R O X Subscripts define groups and observation timing Treatment refers to the independent variable
51 Control groups In multigroup designs, one of the groups should get no treatment and serves as a comparison Often this isn't possible for ethical reasons then the comparison group receives the standard treatment In the design diagram, the control group doesn't get an "X"
52 Posttest randomized 2 group design
X1 O1 O1
Group 1: treatment, observation of DV R Group 2: no treatment, observation of DV Allows for comparisons between subjects 53 Pre/Posttest randomized 2 group design
O1 O1 X1 O2 O2 R Allows for comparison between subjects and within subjects
54 Solomon 4 group design
O1 R O1 X1 X1 O2 O2 O2 O2 (grp 1) (grp 2) (grp 3) (grp 4) Considered the strongest experimental design Allows comparison between subjects and within subjects (for first two groups only)
55 What if you have more conditions? What if you want to compare DARE with "intensive DARE" with no DARE? Just add more groups Works for both posttest only and pre/post test designs 56 Multigroup randomized design
O1 O1 O1 X1 X2 O2 O2 O2
Group 1: DARE Group 2: "intensive DARE" Group 3: control group no DARE R Allows for comparisons between subjects and within subjects
57 What if you think more than one variable is important? Maybe it is DARE, maybe it is an after school program, or maybe the combo, that is causing less drug use We can use a factorial design to examine this effect X = DARE Y = After school program
58 2 group factorial design
R X1Y1 X1Y2 X2Y1 X2Y2 O1 O1 O1 O1 X1 DARE X2 No DARE Y1 After school program Y2 No after school program All possible combinations are included Allows comparison between DARE groups and between after school program groups
59 What if you can't randomly assign one of the variables? For juveniles placed in adult facilities, maybe younger juveniles have different outcomes than older ones We can use a factorial design to examine this effect X = age group Y = facility placement
60 2 group factorial design
R X1Y1 X1Y2 X2Y1 X2Y2 O1 O1 O1 O1
X1 age 15 or younger X2 age 16 or older Y1 juvenile placement Y2 adult placement R All possible combinations are included Allows comparison between age groups and facility placement groups
61 Other experimental concepts Placebo effect comes from medical studies. Taking a sugar pill and thinking it is a painkiller may actually reduce pain Double blind: neither the subject nor the researcher taking measurements knows which condition the subjects are assigned to
62 If someone thinks they are getting treatment their outcomes may change Control group solutions What if denying treatment isn't ethical? What if your funder or agency won't go for random assignment? Waitlist control groups: You've got 100 people who want treatment only make 50 spots. Randomly select those given treatment and use others as control. Provide them with treatment next
63 Control group problems Demoralized control group: subjects are upset they are not getting treatment Compensatory control group: subjects get treatment elsewhere, but are still used in the experiment as though they have no treatment Stop trying on DV measurements Intentionally lie or exaggerate results Try harder to show improvement on DV measures Have actual improvements due to outside treatment
View Full Document
This note was uploaded on 09/14/2008 for the course SOCECOL 10 taught by Professor Pazzani-raitt during the Summer '08 term at UC Irvine.
- Summer '08