plan The and structure of investigation so conceived as to obtain answers to research questions Tells how the different parts are related
Nature of Treatments or Programs Samples/Sampling Strategies Observations/Measurement Assignment to Treatments or Programs Level of Analysis/Units Time
Considerations of design are paramount in the elimination of threats to validity Purpose of design is to control variance Experimental QuasiExperimental
NonExperimental
Nonequivalent Group Design RegressionDiscontinuity Design Interrupted Time Series Design
How do we establish cause? How do we minimize threats to validity? How do we collect the data? How do we identify samples?
Symbolized with an "O" Subscripts are used to distinguish different combinations of measures if necessary
Symbolized with an "X" Subscripts are used to indicate different programs or combinations of programs
Each group of interest is on its own line
Assignment to Groups
R = random assignment N = nonequivalent groups C = assignment by blocking (concomitant variables)
Left to right movement denotes the passage of time
Design Notation Example
Time Rs indicate random assignment Os indicate different waves of observation or measurement
R R
O O
X
O O
Design Notation Example
R R
O O
X
O O
Time Vertical alignment of Os indicates that pretest and posttest occur at the same time for both groups X indicates timing of the treatment
Design Notation Example
R R O1A X O1A O2A,B O2A,B
Subscripts indicate different timing and subsets of measures This Design is a
pretestposttest (beforeafter) treatment versus control group randomized experimental design with partially overlapping measures (A) at times 1 and 2
Experimental Designs
An experiment is a study in which at least one variable is manipulated and units are randomly assigned to the different levels or categories of the manipulated variables Internal vs. external validity? Question: Does using sex of a person in a study make it an experiment?
Treatment Only. Postmeasure Only. (Possibly a one sample t test). R X O TreatmentControl. Postmeasure Only. (Independent samples t test / Regression). R O R X O
Treatment. Premeasure and Postmeasure. (Dependent t test). R O1 X O2 Multiple Treatments. Postmeasure only. (ANOVA / Regression). R X1 O R X2 O R X3 O
Control group may not be possible, but should be included whenever appropriate
Types (notation differs slightly from P&S)
Factorial Designs. (Factorial ANOVA). R X11 O R X12 O R X13 O R X21 O R X22 O R X23 O Alternative Notation for Factorial Designs. (Factorial ANOVA). R A1B1 O R A1B2 O R A1B3 O R A2B1 O R A2B2 O R A2B3 O
Types (notation differs slightly from P&S)
TreatmentControl. Premeasures and Postmeasures. (Repeated measures ANOVA / ANCOVA / Regression assumes equal slopes of change). R O1 O2 R O1 X O2 Treatments and Concomitant Variables. (ANCOVA / Regression assumes equal slopes of change). R C X1 O R C X2 O R C X3 O
Types (notation differs slightly from P&S)
Solomon FourGroup Design. (Factorial ANOVA). Groups A) R O1 X O2 B) R O1 O2 C) R X O2 D) R O2 Group A O2 > Group A O1 Group A O2 > Group B O2 Group C O2 > Group D O2 Group C O2 > Group B O1
Sample size killer
Treatment
No Treatment
Pretest
Group A
Group B
No Pretest
Group C
Group D
Want to find a main effect for treatment. Don't want to find a main effect for pretesting because that will indicate pretesting had an effect on the outcome. Don't want an interaction because that will indicate pretesting influenced effectiveness of treatment.
To reduce threats to validity like carryover or order effects, can use counterbalancing Counterbalancing ensure that all possible combinations of IV presentation order are used within subjects (e.g., 3 within variables where an equal number of people or groups get each of the following orders of treatment: 123, 132, 213, 231, 312, 321) Latin square type of partial counterbalancing where each treatment occurs an equal number of times in every position: Treatment Order Subject/Grp1 C A B D Subject/Grp2 B C D A Subject/Grp3 D B A C Subject/Grp4 A D C B
Serious problem Delete full cases for missing data, but concern about threats to validity Use sophisticated techniques
Whenever we do a study, we have to be concerned about the variance we can't predict We also have to be concerned about variance that shows up as treatment effects which are due to unmeasured variables Experimental control is about controlling variance so that we can increase the power and validity of our studies There are tradeoffs:
Randomization is a powerful method for control and increasing the validity of our causal inferences but it may not always be possible or it may leave a large amount of variance which could be predicted Systematic assignment may reduce the amount of unpredicted variance but it can reduce validity of causal inferences
Randomization used for:
Random sampling from a population Random assignment to treatments Draw samples that are representative of a known population within limits of sampling error Draw samples that are comparable to each other within known limits of sampling error
Used to make causal inferences in a study by equating people within groups (e.g., pretest means in a lab study should be the same) Groups may differ by chance Used to make inferences about population values (e.g., surveys)
Randomization used to perform one of two key functions
Proper random assignment will reduce many threats to internal validity Creates tests that are less biased and often more powerful than nonequivalent groups designs Experiments often involve allocation of important or scarce resources
Randomization is a fair and ethical way of determining who will obtain these resources Cannot deal with compensatory rivalry or resentful demoralization
Randomization does not:
You should design randomized experiments with:
Guarantee initial comparability Ensure comparability over time (e.g., attrition, motivation) even if initial comparability exists Theoretically based expectations for of possibility attrition Reliable measures for a fallback analysis in case nonrandom attrition occurs
Difference between random and systematic assigment is in number and plausability of assumptions for causal inference Randomization can't be used for all variables (e.g., sex, age, disasters, etc.) or for archival data
Withholding treatment from control groups Faulty randomization
Complete listing Random numbers
Less of a problem with multiple treatments Treatment placebo groups Use proper randomization
Number and choice of units
Ensure perceptions of participants are accurate regarding randomization When using systematic or stratified assignment, do assignment to blocks first, then randomize Research question determines at which level of aggregation units should be randomized Small numbers of units can be matched before randomization
Treatment related refusals to participate
Treatment related attrition
Obtain agreement sometime after describing study, but before randomization occurs (but limits generalization to volunteers and can create compensatory rivalry or resentful demoralization) Agreement after description particularly important when have a limited number of units from which to sample (e.g., schools) Treatment related attrition may be interesting as a DV (limits inferences regarding other DVs) Analyze for bias in attrition (compare rates, look at pretest scores, causes of dropouts) Use theory regarding attrition patterns
Heterogeneity of treatment implementation Treatment in the No Treatment Control (diffusion of treatment) Unobtrusive Treatment Implementation (ethics, legal issues, low salience)
Use large samples to ensure salience Don't keep identifiable data longer than necessary Pilot studies Closely monitor controls Standardization of treatments Training of people giving implementation
Lotteries are expected Demand outstrips supply When innovation cannot be delivered to all places at once When units can be temporarily isolated When units have low communication or spatial separation When change is mandated, solutions unknown Setting cutpoints When some people have no preference When you can create your own organization When you have control over units
Formal sampling: A process aimed at obtaining a representative portion of some whole, thereby affording valid inferences and generalizations to it. Population: Aggregate of all cases that conform to some designated set of specifications Cases or elements: The basic units of the population Aggregate: The target population to which one wishes to generalize Sampling frame: A list that contains all the elements of a population Sample: A subset of units or elements drawn from the sampling frame
Purposes and Advantages
Nonprobability Sampling
Convenience Volunteer Haphazard Quota
Feasibility Economy Time Ethics Accuracy (reduces nonsampling related errors)
Probability Sampling
Every element in sample has equal nonzero chance of being selected
Systematic Stratified Sampling Cluster Sampling Sampling is relevant for experimental, quasi experimental, and nonexperimental (e.g., survey) designs
Systematic
Stratified Sampling
Every kth element selected from sample frame Highly influenced by the ordering and structuring of the sampling frame (e.g., seasons and cycles can strongly affect the outcome) Simple and easy to do Population of interest is first divided into nonoverlapping subdivisions on the basis of classification variables (strata) Random samples drawn from each strata Intent is to either:
Ensure generalizability "across" by ensuring full sampling domain is captured (e.g., surveys) Reduce sampling variability by creating homogeneo...