Pretest_post-test_design
5 Pages

Pretest_post-test_design

Course Number: OTS 450, Fall 2008

College/University: Old Dominion

Word Count: 1839

Rating:

Document Preview

Evaluation Design There are a number of models for the design and implementation of instructional programs. Most are sound and logical and produce effective learning. Some focus on impact instead of activity; others focus on the design process which emphasizes the evaluation of an instructional program, and specifically, the desired results. All models include one important step that we will explore in this...

Unformatted Document Excerpt
Coursehero >> Virginia >> Old Dominion >> OTS 450

Course Hero has millions of student submitted documents similar to the one
below including study guides, practice problems, reference materials, practice exams, textbook help and tutor support.

Course Hero has millions of student submitted documents similar to the one below including study guides, practice problems, reference materials, practice exams, textbook help and tutor support.

Design There Evaluation are a number of models for the design and implementation of instructional programs. Most are sound and logical and produce effective learning. Some focus on impact instead of activity; others focus on the design process which emphasizes the evaluation of an instructional program, and specifically, the desired results. All models include one important step that we will explore in this lesson: selecting the evaluation method and design. This step is perhaps one of the most important among those directly involved with evaluation. The methods of evaluation are difficult to separate from the evaluation design. However, specific methods are only a part of the process and are discussed in more detail in a later chapter. Evaluation design is a more general view of the process. It requires the development of a complete system to get the desired measurements. Proper design is imperative for effective evaluation. This brief chapter presents the basic elements of design and the various design alternatives. Elements of Design In many instructional program evaluations, comparisons are made in the performance of a group of program participants: To their performance prior to the program. To another group who did not receive the same program. To the remainder of the population of potential participants. There can be various combinations which make up an evaluation design. This section deals with the elements of design with emphasis on the different design approaches. The specific design variations will be presented later. Control Groups Control groups and experimental groups frequently appear in designs. A control group is a group of participants who are as similar as possible as those in the experimental group but are not involved in the instructional program: the experimental group does participate. Ideally, the only difference between the two groups is that one participates in the program and the other does not. Therefore, a performance comparison of the two groups should indicate the impact or success of the instructional program. It is important that the two groups be equivalent in their job settings, skills, abilities, and demographic characteristics. If possible, the identity of the control group should not be revealed. Otherwise, it could affect their performance. With these requirements, control groups may not be practical in real work situations. However, for critical evaluations affecting many participants, control groups are almost essential. Timing of Measurements A critical issue in evaluation is the timing of measurements (or tests). Measurements may be taken before the program, during the program, and at subsequent intervals after the program. The post-test part of a design is never omitted because it directly measures the results of a program. Careful attention needs to be given to determining when and how pretests and posttests are conducted. In each case, participants being tested deserve an explanation about the proposed activity. When conducting pretests, four general guidelines are recommended: 1. Do not use the pretest if it alters the participants' performance in some way. Pretests are intended to measure the state of the situation before the HRD program begins. The test itself should not have any effect on performance. If there is evidence that the testing procedure will affect performance, then possibly the pretest should be omitted, modified, or given far enough in advance of the program to minimize the effect. 2. Do not pretest when it is meaningless. In teaching completely new material or providing information that participants do not have at the present time, pretest results may be meaningless. It would just show an absence of the knowledge, skill, or ability. For example, in a foreign language program, it is meaningless to conduct a pretest in the foreign language if the participants are not expected to know it. Instead, a measurement of proficiency in acquiring the foreign language skills after the program is more appropriate (post-test only). 3. Pretests and post-tests should be identical or approximately equivalent. When scores are compared, they should have a common base for comparison. Identical tests may be used, although they may influence results when they are taken the second time. Similar but equivalent tests may be more appropriate. 4. The testing should be conducted under the same or similar conditions. The length of time and the conditions under which both tests are taken should be approximately the same. In addition to pretesting and post-testing, tests may be conducted at different time intervals during and after a program. These tests are called time series tests and are recommended, when feasible. Time series tests after a program show the long-term effects of the program and in particular detect whether or not benefits are lasting. They are more often used in training than in education programs. Time series tests during the program measure participant progress toward the objectives. Threats to Validity When discussing the merits of various evaluation designs, it is appropriate to return to the subject of validity. There are a number of problems (or threats) which can reduce the validity of an evaluation design. Different designs can counteract or offset the effects of these threats. Time or history. Time has a way of changing things. With time, performance can improve and attitudes can change -- even without an instructional program. When observing output measurements of an instructional program, this question should always be asked: "Would the same results have occurred without the instruction?" This threat to validity can be isolated by modifying the evaluation design. Effects of testing. It is possible that the actual experience of a test or other instrument can have an effect on performance attitude or even if no HRD program is undertaken. This effect is more likely to happen when pre-tests and post-tests are identical. With enough time between testing, participants reflect on the material and possibly even seek answers to questions that attract their curiosity Also, after the participants know the scope of a program, they may give more favorable responses in the post-test based on their knowledge of what is expected. Selection. As presented in Appendix 3, the selection of the group to participate in an instructional program can possibly have an effect on the outcome. Naturally, some individuals will perform better than others. If mostly high achievers are selected, then the results will be distorted. The same is true for underachievers. The problem can be resolved by using random selection, if it is feasible. Mortality. Participants may drop out for various reasons. If pre- and post-tests are used, the number in the group may have changed from one testing to another. This change makes it difficult to compare the results of the two, and it is compounded by the fact that the lower-level performers are the ones who will usually drop out of a program. These are the most common internal threats to validity. The designs presented next will attempt to overcome these threats in varying degrees. Selecting the Appropriate Design Selecting the appropriate evaluation design is a key part of the evaluation strategy. This section presents the common evaluation designs and the relative advantages and disadvantages of each. One-Shot Program Design One of the most common, and unfortunately least valid, designs involves the "one-shot" program. This technique involves a single group which is evaluated only once after an instructional program is completed. No data are collected prior to the program. There are many uncontrolled factors that might influence the measurement and invalidate conclusions based on results achieved through this design. However, the information obtained in this one-shot evaluation is better than no evaluation at all. This design may be useful for measuring the performance of a group when there is no way to measure performance beforehand. Or, possibly, when there is no significant knowledge, skill, or ability existing before the program is conducted. For example, a group of international sales representatives are trained to speak Spanish before conducting a market survey in Mexico. It makes little sense to evaluate their current ability to speak Spanish before the instructional program is conducted. A measurement at the end of the program to measure their ability to understand and speak Spanish is more appropriate as part of the evaluation process. This evaluation design is useful when financial, organizational, or time constraints prohibit the use of preprogram data collection. Figure 6-1 illustrates this type of design. Since it is the least effective, it should be a minimum reference point for evaluation and used only when necessary. Single Group, Pretest and Post-Test Design To remedy the problem of no data to make comparisons, the next proposed design is the single group, pretest and post-test design as shown in the illustration below. This design goes one step beyond the one-step design by collecting data before and after the instructional program. The knowledge, skills, or abilities possessed by the participants before the program can be compared to the knowledge, skills, or abilities after the program to detect improvement...

Find millions of documents on Course Hero - Study Guides, Lecture Notes, Reference Materials, Practice Exams and more. Course Hero has millions of course specific materials providing students with the best way to expand their education.

Below is a small sample set of documents:

Old Dominion - OTS - 450
GRADING STUDENTSPerhaps you have always thought of grades as an inevitable part of instruction and have never considered why they are important or useful. As a student, you may have thought of grades only as something to anticipate or to dread, depe
Old Dominion - OTS - 450
Activity for Lesson 10Using Pre-tests and Posttests and Assigning GradesPart 1 - Pre-tests and Posttests:The answers to these activities should be drawn from www.lions.odu.edu/~dnethert/Courses/ ots450/readings/Pretest_post-test_design.pdf. 1. Rev
Old Dominion - OTS - 450
Training Reaction SheetPlease provide us with your comments and reactions to the training. Your feedback is important and will help us evaluate this and future training programs. Instructor _ Training Program __ Date _ 1. How would you rate the subj
Old Dominion - OTS - 450
Training Reaction SheetPlease complete the form below to inform us of your opinions regarding the training program. Your input is very important, helping us to create training programs that meet your needs. Circle the appropriate number beside each
Wisc Stevens Point - MSMIT - 263
Prep;.lre E,H;h . 111uent To Be Succe~sfil1 hHp:/scweb. wi p. k 12. wi.lIsDan Dobratz. PllLlclpnl }' .I. Jacob~ .Junior High School 2wn Main Street Steveu' PaillL Wl 5-1-181 E'lllail dunbmtz'g \'lspkl2 \1 liS Ph (715) 145-5-1-22 Fa\ (715) 3-15-7:1-W
Uni. Westminster - ECON - 263
Fall 2003 ECON 263 02 Elementary Microeconomics Tong Problem Set #1 (Due: 30 September 2003, 7:30pm) Part I. Multiple Choice Questions (One point each) Identify the letter of choice that best completes the statement or answers the question. 1. The m
Allan Hancock College - LING - 110
Psycholinguistics: AimsS 1. 2. H 3. L 4. R CTo explain what psycholinguists do. To describe some psycholinguistic methods of investigation. To introduce a particular psycholinguistic model of language processing; namely, the `dualroute' model o
Allan Hancock College - LING - 110
Psycholinguistics: AimsS 1. 2. H 3. L R 4. CTo explain what psycholinguists do. To describe some psycholinguistic methods of investigation. To introduce a particular psycholinguistic model of language processing; namely, the `dual-route' model of
Allan Hancock College - LING - 110
1LING110DR JENNIFER PECKSEXIST LANGUAGE: LECTURE NOTES_ Sexist discourse looks at the ways in which conventional language use discriminates against people on the basis of their gender - and it is usually women who receive discriminatory treat
Allan Hancock College - LING - 110
1 LING110 DR JENNIFER PECK SEXIST LANGUAGE: LECTURE NOTES _ Sexist discourse looks at the ways in which conventional language use discriminates against people on the basis of their gender - and it is usually women who receive discriminatory treatment
Allan Hancock College - LING - 110
MORPHOLOGY LECTURE 2 MORPHEMES FREE and BOUND Free morphemes are words by themselves They are FREE in the sense that they don't have to be attached to anything. Boy Desire LOVE These morphemes have a MEANING when they stand alone Bound morphemes Som
Allan Hancock College - LING - 110
MORPHOLOGY LECTURE 2 MORPHEMES FREE and BOUND Free morphemes are words by themselves They are FREE in the sense that they don't have to be attached to anything. Boy Desire LOVE These morphemes have a MEANING when they stand alone Bound morphemes Som
Allan Hancock College - LING - 110
LING110 2003 INTRODUCTORY LECTUREDR JENNNIFER PECKHANDOUTS, COURSE OUTLINE, LECTURE. WHITEBOARD MARKER, OHTS & MARKER WELCOME - NAME ON BOARD TODAY - Discuss structure and aims of the subject - details are on the handout. Then an introduction to
Allan Hancock College - LING - 110
LING110 2003 INTRODUCTORY LECTUREDR JENNNIFER PECKHANDOUTS, COURSE OUTLINE, LECTURE. WHITEBOARD MARKER, OHTS & MARKER WELCOME - NAME ON BOARD TODAY - Discuss structure and aims of the subject - details are on the handout. Then an introduction to
Allan Hancock College - LING - 110
LING110: LECTURE 1b DR JENNIFER PECK LANGUAGE AND SOCIAL IDENTITY1Readings from Crystal 6, 7, 10 Extra readings: Holmes (especially 1, 7,12) Fromkin et al. ch 7 Finegan et al. ch 11 & 12 Language varies according to a range of elements here we
Allan Hancock College - LING - 110
1 MORPHOLOGY LECTURE 1Macro sociolinguistics, Auslan, gender and language, brain and language Micro syntax sentence structure- and now MORPHOLOGY word structure and next phonetics sounds and sound structures MORPHOLOGY is the study of words MO
Allan Hancock College - LING - 110
1 MORPHOLOGY LECTURE 1Macro sociolinguistics, Auslan, gender and language, brain and language Micro syntax sentence structure- and now MORPHOLOGY word structure and next phonetics sounds and sound structures MORPHOLOGY is the study of words MO
Allan Hancock College - LING - 110
Students in this unit should read this unit outline carefully at the start of semester. It contains important information about the unit. If anything in it is unclear, please consult one of the teaching staff in the unit. Lecture Notes
Allan Hancock College - LING - 110
Lecture Notes LING110 Lecture Timetable 2006 Week Week 1 July 30 Week 2 Aug 6 Week 3 Aug 13 Week 4 Aug 20 Week 5 Aug27 Week 6 Sept 3 Week 7 Sept 10 Lect. 1a 1b 2a 2b 3a 3b 4a 4b 5a 5b 6a 6b 7a 7b Lecturer JJP JJP TJ JJP PR RT RT RT RT RT LC LC RT RT
Southern Oregon - CS - 455
OSSTMM 2.2.Open-Source Security Testing Methodology ManualCreated by Pete HerzogCURRENT VERSION: NOTES: FIXES: OSSTMM 2.2 With this version the OSSTMM includes more of the 3.0 methodology. This version includes updatedrules of engagement and rules
Rose-Hulman - ECE - 250
ECE 250 Homework #6 Due 10/18/2007Problem 1: a) Bias the circuit below to allow a 4 volt swing. Design for IC = 1 mA, and use VBE = 0.7 V and =150 in your design calculations.Vcc+R c R 1+b) Verify you bias using PSpice. Use a part called Qbr
Rose-Hulman - ECE - 250
vti_encoding:SR|utf8-nl vti_timelastmodified:TR|08 Oct 2007 16:32:46 -0000 vti_extenderversion:SR|12.0.0.4518 vti_author:SR|HERNITER-3\herniter vti_modifiedby:SR|HERNITER-3\herniter vti_timecreated:TR|08 Oct 2007 16:32:46 -0000 vti_cacheddtm:TX|08 Oc
Sveriges lantbruksuniversitet - C - 192
Atoms - An Historical Review. 400 BC: Greek Philosophers - 4 elements: fire, earth, water and air -Democritus: early atomic theory -divisibility of matter "atomos" - ultimate particle Used shape as atomic distinctions No experiments howeverAtoms -
NYU - AS - 11110
Fall 2004New York University Columbia University The Graduate School and University Center, CUNY CUNY Medieval Studies Certificate Program Homepage CUNY Renaissance Studies Certificate Program HomepageFordham UniversityFordham Center for M
SEMO - MA - 090
REVIEW FOR FINAL EXAM MA090 Fall 2007 PART I: DEFINITIONS: INDICATE IF THE FOLLOWING STATEMENTS ARE TRUE OR FALSE. _1. Additive inverses and opposites are the same thing. _2. Subtraction is defined in algebra as addition of the reciprocal. _3. The s
Michigan State University - POWERPOINT - 210
Adobe Portable Document Format (PDF) is the open de facto standard for electronic document distribution worldwide. Adobe PDF is a universal file format that preserves all the fonts, formatting, graphics, and color of any source document, regardless o
Michigan State University - POWERPOINT - 210
Luke Reese, Associate Professor Community, Agriculture, Recreation and Resource Studies www.msu.edu/~reesel reesel@msu.edu 517-355-6580 x 204Using Microsoft PowerPoint For Effective PresentationsPresentation Objective"to teach you four fundamen
Michigan State University - POWERPOINT - 210
Introduction to Type in Graphic DesignAEE 210 Jim Lucas Modified by Luke ReeseOverviewsAfter this presentation, you should be able to describe: Basic typeface styles Basic terminology as applied to type Basic uses of typeType termsFont-
Michigan State University - POWERPOINT - 210
WS_FTP LEYourPilotLoginHERE YourPilotPasswordHERESelect correct subdirectorySelect correct drive and subdirectorySelect files to transferSelect transfer directionTransfer CompletedFree Download WS_FTP 5.0.8 LE Free to faculty, staff
Johns Hopkins - DATA - 3397
Program Management and Implementation: Assignment MatrixThis matrix shows the distribution of assignments throughout the course. The weeks that assignments are introduced and due are noted, and the detailed directions for the assignments will appea
Washington - CHEM - 484
CHEM 484, MIDTERM EXAM # 1Your name Score /100Show details to receive full/partial credits. You will have 50 minutes to work on it. Please sign below to confirm that all work on this test is yours and yours alone.Signature1. Two-Dimensional La