Class 8 - Program Evaluation (ch 8)

Class 8 - Program Evaluation (ch 8) - Chapter 8 Chapter...

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Chapter 8 Chapter Program Evaluation 1 In Chapter 8 we will discuss… In What Program Evaluation is How you complete a program evaluation Techniques for data collection Stakeholders Types of evaluations The origination of program evaluation evaluation 1950’s began era of great focus on finding ways 1950’s to study societal problems scientifically to Initially, very quantitative, or numerical Operations Research – statistics and Operations mathematical models mathematical Planning programming budget systems (PBSS) Planning – cost and effectiveness comparisons cost Unsuccessful and result was shift to program Unsuccessful evaluation evaluation Program evaluation is Program The use of social science research methods in The an effort to determine the worth of a program an Systematic, quasi-scientific, client-centered Often referred to as ‘applied research’ Schools of thought Should follow social science research methods, Should a.k.a., academic research a.k.a., Grounded in scientific research methods but flexible Grounded to meet needs of client to Key to unbiased evaluation – understanding Key motivation behind it! motivation How to collect empirical data How What is empirical data? Observable through your senses – seen, heard Collected by Surveys In-depth interviews Focus groups Field observations Experimentation Use of existing data How to collect empirical data How Surveys Asking significant number of people structured Asking questions questions • Close-ended questions – answers are from Close-ended specific subset (i.e., yes or no) specific • Open-ended questions – respondents have more Open-ended flexibility in answering flexibility Close-ended questions preferred • • • quicker, more efficient data collection process Easier to organize, summarize and analyze Reduces response fatigue However, responses are limited How to collect empirical data How In-depth interviews Asking broad, open-ended questions to get Asking lengthy and detailed responses lengthy Less structured and involve less people than Less surveys surveys Produce tremendous amounts of data Focus groups Establish dialogue about something specific 6-10 people Broad, open-ended questions Both methods rely on ‘probing’ questions. How to collect empirical data How Field Observations Participant observations • Evaluators immerse in program • Key is building/maintaining relationships, you have Key to get ‘inside’ to Nonparticipant observations • Evaluator completely detached • No participatory role whatsoever, you are the fly on No the wall the Challenge – getting too close, ‘gone native’ Best method is a judgment call How to collect empirical data How Experimentation Manipulation and control of data to test causal Manipulation relationships relationships Results of control group versus non-control Results group group Use of existing data Many programs already collect data, produce Many reports, etc – use them! reports, The Importance of Stakeholders The What are stakeholders? Groups or individuals with a vested interest in the Groups success of a program success Environmental Scan Identification of stakeholders Examples • • • Policy Makers – elected and/or high level officials Program Sponsors – those providing funding Evaluation Sponsors –liaison between evaluator and Evaluation program program • Program Managers/Practitioners Program • Program Targets – recipients of services Snowball sampling – helps avoid bias The Importance of Stakeholders The Obtaining Stakeholder Input Most often through in-depth interviews or focus Most groups groups Opening question must be broad but get person Opening talking..also known as grand question talking..also • Why do you think program x is conducting an evaluation? SWOT Analysis • • • • Strength Weaknesses Opportunities Threats Once done, you must ‘make sense’ of the data Once collected. collected. Recurring Themes and Stories Types of Program Evaluations Types Needs Assessment Assessment of Program Theory Assessment of Program Process Program Impact Assessment Program Efficiency Assessment Types of Program Evaluations Types Needs Assessment Determine if, and to what extent, a problem exists Is there a need for the program? Basic step to starting new or restructuring programs Steps • Define social condition or problem – be as specific as Define possible, utilize evaluation tools for tougher questions possible, • Determine the scope of the social condition or problem Determine – where is the problem and how large/small, use existing data, talk to key informants existing • Define target population – develop criteria to determine Define who is eligible to receive services who • Briefly describe the services that are needed – how will Briefly program address the problem? program Types of Program Evaluations Types Assessment of Program Theory Program Theory – conceptual design of program In a perfect world, how does program operate Components • Impact Theory Outlines assumptions of program on social condition • Organizational Plan Outlines resources and personnel available • Service Delivery Plan Outlines how services will be delivered to targets • Impact Theory and Service Delivery coincide with Impact program goals and objectives program Goals – general direction Objectives Objectives Types of Program Evaluations Types Logic Model Focus on program’s resources, services provided, Focus outcomes intended to produce outcomes Organized around Organized • • • • Inputs – resources (money, staff, supplies, etc) Activities – services program provides Outputs – amount of services delivered Outcomes – anticipated short-term and long-term impacts Types of Program Evaluations Types Assessment of Program Process Focus on extent to which a program’s services align Focus with its targets with Referred to as coverage and bias • Coverage – level of participation; undercoverage thought to Coverage be from recruitment and retention issues be • Bias – certain subgroups of the target population are Bias participating with greater or less frequency than expected participating How many served during specific time period, How proportion of total number of eligible served in time period, degree of satisfaction period, Establish achievable benchmarks Use comparable programs Types of Program Evaluations Types Program Impact Assessment Identify measurable outcomes • An observable characteristic relating to potential benefit • Examine program’s logic model Important not to confuse program outcomes with Important program impact program Outcome linked to population served, impact linked to Outcome program itself program Outcomes must be measureable This method popular because has a pre and post-test This design – stronger when coupled with experimentation design Types of Program Evaluations Types Experimentation Randomized field experiment – classic scientific Randomized method method • Treatment group and control group • Randomness of participant assignment key Quasi-experiment – no random assignment of Quasi-experiment participants participants • Weaker, less reliable results because group characteristics Weaker, not equally distributed not Types of Program Evaluations Types Program Efficiency Assessment Focus on whether money and resources are well Focus spent spent Cost benefit analysis • Program costs compared to tangible and intangible program Program benefits benefits • Benefit-cost ratio>1 = benefits outweigh cost • Challenge – how do you measure intangibles Survey people about how much they would pay for similar service Cost effectiveness analysis • Estimate costs of achieving specific outcomes • More popular, easier to use than cost-benefit Ethical Concerns Ethical Use of human subjects Protect your subjects Obtain informed consent Bias in presentation and collection of data Other consideration Your relationship to/with program evaluating Are you being paid Personal feelings on type of services provided Political influence Next Week… Next GOVERNMENT ASSESSMENT GOVERNMENT PAPER DUE PAPER Chapter 9 – Public Budgeting 21 ...
View Full Document

This document was uploaded on 11/03/2011 for the course PAD 3003 at FSU.

Ask a homework question - tutors are online