This preview shows pages 1–3. Sign up to view the full content.
This preview has intentionally blurred sections. Sign up to view the full version.View Full Document
Unformatted text preview: Assessing Program Impact Required reading: RLF, Chapter 8 Impact assessment is, perhaps, the most important topic in this course because the goal for most evaluations is to answer the question Does the program have an impact on the desired outcomes? You can examine the impact of a demonstration project (i.e., a pilot or a first project that is developed to determine if a program concept works); you can examine the impact of a modification or change made to a program (to see if the change has the expected impact); you can examine a more established program (to see if it continues to have an impact); and you can compare one program to another program (to compare the programs differential impacts). In short, impact assessments can be done at virtually any phase of program development. A randomized design is an experiment that has random assignment of participants to the groups (e.g., experimental and control) participating in the study. You will recall that random assignment is the best method for equating groups on all known and unknown extraneous variables at the start of an experiment. Consequently, randomized experiments usually provide the best estimate of net impact, compared to the other available research designs and methods. Randomized experiments are the gold standard of experimental research and offer the best strategy for obtaining evidence concerning cause and effect relationships. A feature of this chapter is that the authors focus on field experiments (i.e. experiments that are conducted in relatively real world settings) rather than on laboratory experiments (i.e., experiments that are conducted in the lab under highly structured conditions). Program evaluations are almost always conducted in the field rather than in the lab. I will provide some comments about each of the major sections in this chapter: When is an Impact Assessment Appropriate? Key Concepts in Impact Assessment. Randomized Field Experiments. Limitations on the Use of Randomized Experiments. Please note that I am going to include in this lecture a good deal of additional material that you should already know from taking your research methods class (so, hopefully, it will be review). This course is meant to build on your research methods course; hence, you might need to review some of the material from that course just in case you have forgotten some of it. In the introductory section, RLF made several points to remember: When determining program effect (did the program cause the observed changes) we need an estimate of the counterfactural (i.e., what would have happened if the program had not been implemented?) In education and the social sciences, what we can determine is probabilistic causation ; that is, our results and the relationships we describe in our results are probabilistic, not necessary or absolute. We obtain degrees of evidence, not proof of causation....
View Full Document
- Spring '11