1.4 Assessing Your Class for the AMiGEO Project

How to Assess Your Class for the AMiGEO Project

This is a guidance document to help you, the instructor, complete the necessary steps to evaluate the use of open educational resources (OER) in your class for the Analytical Methods in Geosciences (AMiGEO) project.

Materials Needed Before You Begin:

  1. Enough copies of the Student Consent Form for Observation and Assignments Appendix F for everyone in your class.

    1. This first step is a time-saving one. It prevents you from having to sign all of the forms – just sign one and copy it.  Print and sign ONE copy of the Observation and Assignments consent form (Student Consent Form for Observation and Assignments Appendix F.pdf). You are the researcher; you need to sign the last page under my signature (Name of Researcher).
    2. Make copies of the signed form - one for each student in the class. You must give each student the entire form- it is not OK to just print the signature page.


  2. A printed copy of a code sheet for your class.

    1. Please ask Christie Liu ([email protected]) to create a code sheet for your class well before the start of the semester. She will need to know how many students (approximately) are in your class.
    2. This is an excel spreadsheet that with a table containing unique code numbers that will be assigned to specific students in the class. Each student writes their name on the sheet, and should write down and remember their code number.
    3. The purpose of this sheet is to allow us to track individual students while preserving anonymity. For example, we can compare the pre- and post- class survey for the same student. This means that you must keep the paper copy with student names, and do NOT share it with Christie Liu, Joy Mao, or Liz Johnson.  You are the only one who can correlate name to code.  You will need to do this during the semester to code / remove student names from assignments.


  3. Links to the pre- and post-surveys and the content survey.

    1. Appendix L— Pre-course Geoscience Research Class Experience Questionnaire with Consent Form: http://jmu.co1.qualtrics.com/jfe/form/SV_ctJl0ID9NNIDDEh
    2. Appendix M— Post-course Geoscience Research Class Experience Questionnaire with Consent Form: http://jmu.co1.qualtrics.com/SE/?SID=SV_0MV4ieP8S7vlIot
    3. Appendix N – Usability Test Survey for Students with Consent Form: http://jmu.co1.qualtrics.com/jfe/form/SV_8lfjJYIgvfZRMjz
    4. Add these links to the course in your Learning Management System (I use announcements through Canvas; do whatever makes sense in Blackboard or another system). It also makes sense to write the link for the appropriate survey on the board.


  4. If the students are writing a lab report, download the grading rubric for reports (AmiGeo-Essay-Rubric.docx at https://osf.io/aqubs/).

    1. You do not need to use every row of the rubric for grading. Use the rows that are appropriate for your assignment. Click each purple spot in Figure 1.4.1 below to have a zoom-in view.




Figure 1.4.1 AmiGEO Rubric

Timeline and Checklist for Assessment:

  1. Make sure your institution’s Internal Review Board allows you to use these surveys. [The NVCC IRB previously gave blanket approval, but you may need to check with them again since we completed this new update.The updated IRB and appendices are available on OSF in the IRB: Internal Review Board Documents folder.]

    1. The IRB requirements in each university or college vary. Make sure to check with your institutional internal review board (IRB) about your use of these surveys in your classes. If additional IRB procedures are needed, follow your institution’s guidelines and contact Christie Liu [email protected] if you need JMU IRB documentation about this project.”


  2. The students take the pre- survey before the assignment or class begins.

    1. The timing is dependent upon the activity and class. For example, if you are assessing an extra credit lab assignment, it makes sense to give students the pre-survey at the start of that lab unit, even if it is near the end of the semester.  If you are using microscopes throughout the semester in your class, it makes sense to give the pre-survey on the first day of class or lab.
    2. Best practice is to give students the pre-survey before they start assignments for the lab or course.
    3. Explain to students WHY they must sign a consent form (JMU requires it), and also that this is optional. Our experience is that if you explain that they are helping themselves and other students by participating (because we are trying to help them learn more effectively), and extra effort is minimal, then participation is excellent.
    4. E-mail Christie Liu ([email protected]) to make sure she received responses on Qualitrics.


  3. Students and instructors take the usability test survey immediately after they have used the online materials.

    1. This can be at the end of the instructional unit.
    2. The link for the survey is: http://jmu.co1.qualtrics.com/jfe/form/SV_8c8jPi0k9TerPOR


  4. Students take the post-survey at the end of the experience. It is necessary to do this immediately at the end of the experience! Students (and us) quickly move on to think about other things during the semester, so timing is very important.
  5. You, the instructor, need to code all assignments and grades so they are anonymous.

    1. For grade sheets and grading rubrics, remove names and replace them with the code number for each student.
    2. Download or scan student assignments from the Learning Management System (for instance, JMU uses Canvas.). Replace names with code numbers in both the file names and on the assignment.
    3. After grading the coded written reports/assignments with the coded grading sheets/rubrics, scan both the coded rubric and assignment and upload them to a folder on OSF. See next section for details about uploading.
    4. Best practice is to download and document assignments and assessments as soon as they happen. We instructors have a lot going on, and it is easy to forget or misplace items that are needed for this study.  Make it a practice to complete the assessment documentation as part of the grading process.


  6. Make a folder on OSF for your course and upload all assessment materials into the folder. [We can set up a separate component or project on OSF and enroll the workshop participants with their institutional abbreviations for them to upload files.]

    1. Find the “Assessment Data” component in the Files Window (left side) and expand its contents. Click on the “OSF” icon within the Assessment Data component.  At the top of the Files Window, click “Add Folder.”
    2. Name your folder with the University, Course Number, Semester, Course Name. For example, JMU GEOL 390 Spring 2018 Lab Tech in Geology.
    3. Create one subfolder for each type of assessment or assignment. Load files into these folders.
    4. I am happy to help with organization and uploading into OSF ([email protected]).




Example of Assessment #1: Using the SEM for a 1-week lab assignment in a semester-long course.

Online instructional materials were used as part of a 1-week-long laboratory assignment using the SEM to analyze rocks.  Students worked on samples in pairs, but due to time constraints groups of 4 students spent 1 hour each obtaining images and analyses on their rocks using the SEM.

Assignments:

Formative: Students were provided online instructional materials through the Canvas learning management system as a “reading” assignment.  Each student wrote a response to an open-ended question based upon those materials, and these were shared in a graded discussion forum as a homework assignment.

Summative: Students created mini-posters which included data and interpretations from the SEM.

Summative: About 1 month after the lab, students took a midterm exam. One question on the exam asked students to identify two minerals based upon images and chemical data from the SEM analyses.

Timeline for assessment:

  1. Code numbers were assigned to students when they took the pre-survey in the class period before the lab began.
  2. Students completed the online materials and homework assignment.
  3. Students participated in the lab and turned in their lab assignments.
  4. Instructor downloaded the responses, coded the responses to remove student names, and created a document containing the reading assignment, the homework question, and the coded responses. This document was uploaded to OSF.
  5. Students completed the post-survey and usability test survey at the end of the lab session.
  6. Instructor graded the lab assignments using a rubric.
  7. Instructor downloaded the lab assignments, coded them to remove student names, graded the homework assignment, and scanned the graded and coded assignments. Instructor uploaded coded assignments, the lab assignment sheet, and the grading rubrics to OSF.
  8. After students took the exam (about 1 month later), instructor extracted the SEM exam question and student responses from the exams. This information was coded to remove student names and was uploaded to OSF.


Example of Assessment #2: Using multiple instruments through 8 weeks of a course.

Online instructional materials were used for 8 weeks of a semester-long (15 week) course on laboratory techniques in the geosciences.  Students spent most of face-to-face class time using instrumentation, being trained to run the equipment, or interpreting data.

Assignments:

Formative: Students were provided online instructional materials through the Canvas learning management system as “reading” assignments for each 2-3 week-long module focused on a different analytical technique.  Each student wrote a draft of their paper and turned it in at the end of the first week of a module.

Summative: Students wrote 3 reports – one report for each module.

Timeline for assessment:

  1. Code numbers were assigned to students and they took the pre-survey on the first day of class.
  2. Students completed the online materials and either wrote a draft paper or a final report each week.
  3. Each week, the instructor graded papers and provided feedback to students as a group about ways to improve the reports. Draft papers were graded for completeness. Final papers were graded using the AMiGEO grading rubric.
  4. The instructor downloaded the reports and coded the responses to remove student names.
  5. The instructor uploaded coded reports, the report assignment sheet, and copies of the grading rubrics including instructor grades and comments to OSF.
  6. Students completed the post-survey and usability test survey on the last day of class of the 8-week session (At the end of the unit that used the online materials- we did NOT wait another 7 weeks to the end of the semester).


This document was created by Elizabeth Johnson and Juhong Christie Liu, James Madison University, July 2018.  This work is licensed under a Creative Commons Attribution 4.0 International License.

Licenses and Attributions