18
Evaluation Class Research
Class Outline
Evaluation Basics Approaches to Evaluation Research Types of Evaluation Research Designs Internal Validity in Evaluation Research
Evaluation Research
Evaluation research, or program evaluation, refers to the kind of applied social research that attempts to evaluate the effectiveness of social programs. Appropriate for any study of planned or actual social intervention. Goal is to determine whether a social intervention has produced the intended result. Results are not always well received.
Stakeholders
A stakeholder is someone who has sufficient program knowledge to contribute to the process in meaningful ways, and whose self-defined stake in the program is high (Greene, 1988). Types of stakeholders Agents: those persons involved in producing, using, and implementing the program Beneficiaries: those persons who profit in some way from the use of the program Victims: those persons who are negatively affected by the program
Approaches to Evaluation Research
Black-box evaluation or theory-driven evaluation Black-box evaluation involves determining whether a program has the intended effect. Theory-driven evaluation seeks to understand how the program operates and to identify the program elements that are operational. Researcher or stakeholder orientation Should the evaluators be responsive to program stakeholders or should they emphasize the importance of research expertise and maintain some autonomy in order to develop unbiased evaluation?
Approaches to Evaluation Research
Quantitative or qualitative methods Qualitative methods add more depth, detail, and nuance to complex programs. Simple or complex outcomes Even single-purpose programs may turn out to have multiple outcomes.
A Model for Theory-Driven Evaluation
Implementation Environment Generalizability to Other Situations Treatment Intervening Mechanism Outcome
Cause
Effect
Reference: Chen, Huey-Tsyu (1990). Theory-driven evaluations. Sage Publications, Newbury Park, CA. p. 50
Questions to be Asked in Theory-Driven Evaluation
What is the goal of the program? What is the treatment? Under what circumstances is the program being implemented? Does it work? What is the effect? What other variables could have caused the effect? Can you say that this program will work in another place and time?
Internal Validity in Evaluation Research: The Nave Estimator of Causal Effect
The nave way to estimate treatment effect is to compare units of analysis affected by the program to those unaffected by the program. Say in a community, N1 children attended Head Start, and N2 did not. 27 years later, measure the mean years of schooling of the two groups, y1 (those who attended Head Start) and y2 (those who did not attend Head Start).
Internal in Validity Evaluation Research: The Nave Estimator of Causal Effect
We compute y1 - y2 = 13 - 14 = -1. Should we conclude from this that Head Start has a negative effect on educational attainment? The Westinghouse report (1969). The appropriate research question is not to compare observed y1 and observed y2.
Causal Effect as a Counter-Factual Question
Rather, we should ask the counter-factual question, for those who attended Head Start, what would have happened to them if they hadn't attended? Or, y1t - y1c (t denoting treatment; c denoting control) Note that y1t is observed, but y1c is not. This is a missing data problem. y1t - y1c is the average treatment effect for the treated.
Causal Effect as a Counter-Factual Question
We could also ask: for those who did not attend Head Start, what would have happened to them if they had attended? y2t - y2c Note that y2c is observed, but y2t is not. y2t - y2c is the average treatment effect for the control group.
Causal Effect as a Counter-Factual Question
If received If received Treatment treatment control effect Treatment group (N1) Control group (N2) y1t y1c y1t - y1c
y2t
y2c
y2t - y2c
Assumption for Simple Comparisons
If N1 children are comparable to N2 children, we can assume y1c = y2c y1t = y2t In that case y1t - y1c = y2t - y2c = y1t - y2c That is, we can use the nave method to estimate the treatment effect. In reality, we need to consider selectivity.
Selectivity Bias: Observed Selectivity
If subjects who receive social intervention and those who do not are different in observed characteristics, this type of selectivity is called observed selectivity. This problem can be handled by statistical c...