George Washington University PSYC 144 Class Project Development of a Situational Judgment Test (SJT) Spring 2005 Introduction The purpose of this project is to give you hands-on experience developing a psychological assessment. In completing the project, you will work in a group to create a Situational Judgment Test (SJT) designed to identify high school seniors who are likely to succeed at GWU. The project consists of five key steps. Some steps will be completed independently outside of class, whereas other steps will be completed during class time. Description of Situational Judgment Inventories A SJT is designed to assess an examinee's ability to identify the appropriate behavior for a given set of circumstances. A situational judgment item consists of a scenario (about a paragraph long) describing a situation often experienced by people in the job of interest. After reading a scenario, examinees are asked to select from a number of options the behavior they would most likely perform in response to the situation described in the scenario. Examinees are then asked to identify the behavior they would least likely perform. An example SJT item for the job of customer service representative is presented below. While you are working at the service desk, a customer attempts to return a toaster. According to store policy, a customer cannot return a product without presenting a receipt. When you ask the customer for a receipt, he claims that he accidentally threw it out. The customer assures you that he purchased the toaster two days ago. When you tell the customer that you cannot allow him to return the toaster without a receipt, he becomes enraged and begins to yell at you. 1. Which of the following behaviors would you MOST likely exhibit? 2. Which of the following behaviors would you LEAST likely exhibit? A. B. C. D. E. Call security to have the customer removed from the store. Allow the customer to return the toaster. Ignore the customer until he calms down. Tell the customer you cannot help him unless he calms down. Tell the customer that you will see what you can do to help him get a refund. Page 1 of 6 PSYC 144 Class Project Development of an SJT Spring 2005 Specific responsibilities and deadlines for the SJT project are presented below. Project deadlines are summarized in a table on page 4. 1. Collect Critical Incidents: Due February 22, 2005 (completed individually outside of class) The first step in developing an SJT involves collecting examples of effective and ineffective behavior. This is accomplished by having <a href="/keyword/subject-matter-experts/" >subject matter experts</a> (SMEs) complete critical incident forms. In completing the critical incident form, SMEs are asked to describe (a) the events leading up to the situation in which the behavior was exhibited, (b) the behavior exhibited in the situation, and (c) the consequence of the behavior. Student Responsibilities: You are required to collect five critical incidents describing effective behaviors and five critical incidents describing ineffective behaviors (a total of 10 critical incidents). Critical incidents must come from at least two students at GW who are not registered for this class. Be sure to collect both effective and ineffective critical incidents from each student you ask to serve as a SME. For example, one student gives three effective and two ineffective critical incidents, while the other student gives two effective and three ineffective critical incidents. Critical incident forms for effective and ineffective behaviors are included at the end of this file along with an example of a completed critical incident form. 2. Sort Critical Incidents: Due March 10, 2005 (completed individually outside of class) The next step involves identifying the abilities and/or personal characteristics to be assessed on the SJT. This is accomplished in two steps that involve sorting similar critical incidents into categories. First, each group member individually sorts the critical incidents into categories. Second, the group convenes to discuss the individual, reconcile differences, and come to consensus on the final categories. Note that this process also provides the content necessary to write SJT items. Student Responsibilities: On March 1, 2005, you will be assigned to a group consisting of 4 - 6 members. All members of your group will receive a packet of critical incidents collected by students from another group (i.e., you don't get to use the critical incidents you turned in). You are expected to sort the critical incidents into categories according to a common theme. Next, review the critical incidents in each category and come up with a name and brief definition summarizing the KSAOs reflected in the critical incidents. For example, critical incidents that relate to studying can be grouped into the category &quot;Study Habits.&quot; Use the critical incident category form included at the end of this file to document the name, definition, and critical incidents for each category. It is likely that you will need to use more than one critical incident category form. Page 2 of 6 PSYC 144 Class Project Development of an SJT Spring 2005 3. Identify KSAOs Assessed By The SJT: Completed in-class on March 24, 2005 The next step involves formally defining the abilities and/or personal characteristics measured by the SJT. This is usually accomplished in one of two ways. In one approach, a group of SMEs independently review a set of critical incidents. The SMEs are given a list of categories and are asked to which category each critical incident represents. SMEs have the option of indicating that a critical incident does not belong to any of the listed categories. Statistical analyses are then conducted to assess the level of agreement in placing critical incidents into categories. In the second approach, SMEs independently sort critical incidents and then meet as a group to discuss the sorting. The group discusses disagreements and comes to consensus on the final categories. The second approach is used for this project. Student responsibilities: Groups will convene and each group member will present the categories into which he or she sorted the critical incidents. The group will then discuss the categories and come to a consensus about the abilities and/or personal characteristics that will be assessed by the SJT. After all groups have finished, the remainder of the class time will be spent going over the procedures for writing a situational judgment item. By the end of the class, each group should (1) turn in a critical incidents dimension form documenting the name and definition for each category, and (2) decide which situational judgment items each group members will write. Try to have an equal number of items for each category. All group members are responsible for three situational judgment items. 4. Writing SJT Items: Due on April 7, 2005 (completed individually outside of class) The next step involves writing SJT items. The material for an SJT item is taken from the critical incident forms. Information about what led up to the incident can be used to develop the scenario. Information about the behaviors that occurred during the incident can be used to develop response options. Finally, information about the consequence of the behavior can be used to determine a preliminary scoring key for the response options. Student Responsibilities: You are expected to write three situational judgment items using the critical incidents categorized by your group. Each item should consist of a scenario and 4 response options. To develop the scoring key, read each response option and indicate how effective you feel it is using the following four-point scale: 1 very ineffective 2 Ineffective 3 effective 4 very effective A good item will have both effective and ineffective response options. However, it is not necessary for response options to reflect all of the scale points. For example, an item can have one highly effective response, one ineffective response, and two very ineffective responses. Finally, put the scoring key on a separate piece of paper. This will enable the group come-up with a scoring key for an item without being influenced by creator's scoring. Specific guidelines for writing SJT items are presented on pages 5-6 of this document. Page 3 of 6 PSYC 144 Class Project Development of an SJT Spring 2005 5. Scoring Situational Judgment Items: In-Class on April 21, 2005 The next step involves finalizing the scoring key. This is typically done by a group of SMEs. At first, the SMEs read each SJT item and provide a scoring key. SMEs also review each item for clarity and relevance. This process often results in items being revised. The item writer then leads a review of the item. The group discusses any disagreements in their scoring and comes to a consensus on a final scoring key. An item is discarded if SMEs are unable to come to a consensus. Student responsibilities: Groups will be given a sheet with all of the items written by members of their group. Group members will spend the first 15 to 20 minutes of class independently reading and scoring all of the situational judgment items. Next, the group will convene and discuss the scoring key for each item. Each group member will lead the discussion of the scoring key for his or her items. By the end of the class, each group will hand in a final scoring key for the items developed by group members. SJT Development Project Deadlines Due Date 02/22/05 03/10/05 03/24/05 04/07/05 04/21/05 Task Collect 10 critical incidents Individual sorting of critical incidents Identify KSAOs evaluated by the SJT Write 3 SJT items and provide scoring key Final scoring key developed for SJT Completed In/Out of Class Completed outside of class Completed outside of class Completed in-class Completed outside of class Completed in-class Page 4 of 6 George Washington University PSYC 144 Class Project Steps in Writing Situational Judgment Test Items Spring 2005 INTRODUCTION The following outline presents the steps that are involved in writing items for a situational judgment test 1. Develop scenarios 2. Develop five response options for each scenario 3. Score response options for each scenario 4. Have <a href="/keyword/subject-matter-experts/" >subject matter experts</a> review the scenarios and response options 5. Revise the scenarios and response options according to input from <a href="/keyword/subject-matter-experts/" >subject matter experts</a> 6. Create a final version of the Situational Judgment Test At the end of the third activity for the SJT project, you should know which &quot;categories&quot; you will be writing situational judgment items for. As mentioned above, you are expected to write three items with a preliminary scoring key for each item This corresponds to steps 1-3 from the outline presented above. Note that these items are written outside of class. WRITING A SITUATIONAL JUDGMENT ITEM Step 1 - Develop Scenarios 1. Use the information from Question 1 on the critical incident form to create a theme for the scenario A. Answers to this question provide information about &quot;the circumstances that led up to the incident&quot; B. Ideally, the theme should be reflected in a number of similar critical incidents C. Use your own creativity to make the scenario more realistic, but do not go overboard 2. Draft a scenario that is no more than one paragraph long. At a minimum, the scenario should be three sentences. A. It is important that you clarify any information examinees WOULD NOT be expected to know when they take the test: Examples of this type of information include: 1) Policies specific to the organization 2) Technical procedures that are learned on the job 3) Knowledge that is learned on the job Page 5 of 6 George Washington University PSYC 144 Class Project Steps in Writing Situational Judgment Test Items Spring 2005 Step 2 - Develop Five Response Options for Each Scenario 1. Use the information from Question 2 on the critical incident form to develop response options A. Answers to this question provide information about &quot;what occurred during the incident&quot; B. Response options should come from critical incidents with a common theme C. Include information from effective and ineffective critical incidents D. Can use your own knowledge to make-up some of the options E. Based on knowledge gained during job analysis Step 3 - Score Response Options for Each Scenario 1. Use the information from Question 3 on the critical incident form to develop a preliminary scoring key A. Answers to this question provide information about &quot;the consequence of the behavior&quot; 1) Response options based on effective critical incidents should receive a positive weight 2) Response options based on ineffective critical incidents should receive a negative weight Page 6 of 6 Example Critical Incident Form Depicting Effective Behavior Instructions: Describe an incident in which you observed a student engage in behavior that led to effective academic performance. Indicate what led to the incident, what occurred, and what the consequences of the situation were. Provide enough detail to allow the researchers to completely understand the situation. 1. What circumstances led to the incident? Student came down with the flue. As a result, he was forced to miss two weeks of classes. 2. What occurred during the incident? Be sure to clearly indicate the effective behavior. Once the student had recovered, he contacted his professors to explain why he had missed class. He obtained all of the materials he missed from the classes and spent an extra 3 hours per week studying to make-up for lost time. 3. What were the consequences (for the individual exhibiting the behavior, for other individuals, or for the university)? The student received an &quot;A&quot; in four of his classes, and a &quot;B+&quot; in the other class. When he asked one of his professors for a letter of recommendation, the professor told him he would be happy to write a letter and commented that she was very impressed with his study habits. Name of Student in PSYC 144: __________________________________________________ Example Critical Incident Form Depicting Ineffective Behavior Instructions: Describe an incident in which you observed a student engage in behavior that led to ineffective academic performance. Indicate what led to the incident, what occurred, and what the consequences of the situation were. Provide enough detail to allow the researchers to completely understand the situation. 1. What circumstances led to the incident? Student was invited to a party the night before his final exam. He had originally planned to stay for an hour or two, but ended up staying out until 3:00 a.m. 2. What occurred during the incident? Be sure to clearly indicate the effective behavior. The student woke up one hour before the final exam and studied. He was unable to get a handle on all of the material to be covered on the test. 3. What were the consequences (for the individual exhibiting the behavior, for other individuals, or for the university)? The student received a &quot;D&quot; on the final exam. His performance on the final exam resulted in his getting a B- for the class. The B- prevented him from getting on the honor role for the first time in his two years at GW. Name of Student in PSYC 144: __________________________________________________ Situation Depicting Effective Behavior Instructions: Describe an incident in which you observed a student engage in behavior that led to effective academic performance. Indicate what led to the incident, what occurred, and what the consequences of the situation were. Provide enough detail to allow the researchers to completely understand the situation. 1. What circumstances led to the incident? 2. What occurred during the incident? Be sure to clearly indicate the effective behavior. 3. What were the consequences (for the individual exhibiting the behavior, for other individuals, or for the university)? Name of Student in PSYC 144: __________________________________________________ Situation Depicting Ineffective Behavior Instructions: Describe an incident in which you observed a student engage in behavior that led to ineffective academic performance. Indicate what led to the incident, what occurred, and what the consequences of the situation were. Provide enough detail to allow the researchers to completely understand the situation. 1. What circumstances led to the incident? 2. What occurred during the incident? Be sure to clearly indicate the ineffective behavior. 3. What were the consequences (for the individual exhibiting the behavior, for other individuals, or for the university)? Name of Student in PSYC 144: __________________________________________________ Critical Incidents Category Form Name: _________________________________________ Date: _______________________ Instructions: Use this form to identify the categories reflected in the critical incidents you have collected. After sorting the critical incidents, provide a name and definition for each category you have identified. Category Title: Critical Incidents:* Definition: Category Title: Critical Incidents:* Definition: * List the numbers for all of the critical incidents representing the category