and Reliability Validity of a Student Scale for Assessing the Quality of Internet-Based Distance Learning
(in review by the Journal of Distance Learning Administration) Craig L. Scanlan, EdD, RRT, FAARC* Professor and Director MS and PhD in Health Sciences Programs University of Medicine and Dentistry of New Jersey School of Health Related Professions 65 Bergen Street Newark, NJ 07107 Phone: [phone] Fax: [phone] E-mail: [e-mail]
I wish to acknowledge the assistance provided by the UMDNJ-SHRP Technology Task Force in this effort: Drs. Laura Nelson, Joyce OConnor, Julie OSullivan-Maillet, Carlos Pratt, Ann Tucker, and Gail Tuzman
*
Reliability & Validity of a Student Scale for Assessing Internet-Based Distance Learning
About 85% of all U.S. universities and colleges offer distance education courses, up from 62% since 1998 (National Governors Association, 2001). In the period between 1998 and 2002, distance education enrollments were estimated to have quadrupled, from 500,000 to over 2,000,000. By 2004, it is expected that distance learners will constitute about 14% of all those enrolled in degree programs (International Data Corporation, 1999). Most of this growth is attributed to the dramatic upsurge in Internet-based education, which now represents the primary means by which colleges and universities provide distance learning (National Center for Education Statistics, 1999). Concomitant with the rapid growth in Internet-based education has come a flurry of guidelines and standards designed to help assure its quality (American Council on Education, (n.d.).; American Distance Education Consortium, (n.d.).; American Federation of Teachers, 2000; Council of Graduate Schools, 1998; Council of Regional Accrediting Commissions, 2000; Institute for Higher Education Policy, 2000; Quality Assurance Agency for Higher Education, 1999; Western Cooperative for Educational Telecommunication, 1997). Although there is a remarkable degree of congruence among these standards (Twigg, 2001), and while most include specifications regarding the evaluation of Internet-based education, none provide the actual measurement tools needed to conduct quality assessment. Indeed, in its preliminary review of distance learning, the Institute for Higher Education Policy (1998) emphasized the need for reliable and valid performance measurements. The situation at the UMDNJ School of Health-Related Professions (SHRP) mirrors the national picture. SHRP has been offering Internet-based education since 1997, with the number of courses and enrollments growing substantially each year. Although individual course evaluation has always been applied, no overall assessment of the Schools distance learning
-1-
Reliability & Validity of a Student Scale for Assessing Internet-Based Distance Learning
program had ever taken place, nor was there a strategy to do so. To that end, a Technology Task Force (TTF) was created and charged with evaluating the Schools overall distance learning program. Methods Based on a review of the literature, the TTF selected a benchmarking model as the basis for its evaluation. Benchmarking is a quality improvement process that compares actual program or institutional performance to exemplary or best practices (McGregor & Attinasi, 1998). Existing and/or prospectively gathered data are used to determine the actual performance level, with the best practices set as the achievement ideal. Any observed discrepancy between the actual and ideal performance points to a specific need for quality improvement - the greater the discrepancy, the greater the need. In terms of best practices, after a review of the published guidelines and standards cited above, the TTF decided to adopt the 24 benchmarks developed and promulgated by the Institute for Higher Education Policy or IHEP (2000). This decision was based on the strong content validation process employed by the IHEP to derive their benchmarks (discussed subsequently under findings). The TTF reviewed each IHEP benchmark to determine (a) the feasibility and appropriateness of its review; (b) applicable data sources; and (c) data collection methods. Because it became clear that multiple perspectives and approaches would be needed to determine actual performance levels, the TTF adopted a mixed-method strategy (Greene & Caracelli, 1997) that focused on triangulating data obtained from students, faculty and support personnel (Fielding & Fielding, 1986). Student data were gathered primarily via a quantitative survey questionnaire, while faculty and support personnel data were obtained, respectively, via focus
-2-
Reliability & Validity of a Student Scale for Assessing Internet-Based Distance Learning
group and individual interviews. The remainder of this paper focuses on the psychometric properties of the student survey questionnaire, in particular its benchmark scale. The student survey questionnaire consisted of four parts: (1) selected IHEP benchmark statements (hereafter referred to as the benchmark scale); (2) four global rating of students online experience; (3) an open-ended request for recommendations for improvement; and (4) demographic information.* The TTF initially identified 10 of the 24 IHEP benchmarks as applicable for student assessment. To enhance the specificity of data collection and analysis, the TTF broke 3 of the IHEP benchmarks into two each. For example, the single technical assistance benchmark was separated into two, one focusing on availability and the other focusing on the quality of support. Last, based on prior concerns, the TTF added one of its own benchmarks, addressing enrollment and registration processes. As depicted in Figure 1, each of these 14 benchmarks was directly translated into a questionnaire statement applicable for student response and presented together on a 4-point Likert scale. The questionnaire was then pilot tested for clarity among the members of the Task Force, with the final edited version administered anonymously in April 2002 to all 115 students enrolled in the Schools Internet-based courses. After two follow-ups, 77 students completed the questionnaire, for a response rate of 67%.
The full online survey questionnaire is available at: http://www.umdnj.edu/idsweb/student_survey.htm. Because submission of the questionnaire responses used CGIemail, those interested in adapting this tool should also download the accompanying text file http://www.umdnj.edu/idsweb/student_survey.txt. Used with CGIemail, the text file generates a comma-delimited line of data for each respondent that can be saved cumulatively or easily transferred to a spreadsheet or database program.
*
-3-
Reliability & Validity of a Student Scale for Assessing Internet-Based Distance Learning
IHEP Benchmark: Feedback to student is constructive and provided in a timely manner
Student Scale Item: In my online course(s), I always received constructive and timely feedback on my assignments and questions
Strongly Agree Agree Somewhat Disagree Somewhat Strongly Disagree
Figure 1. Example translation of IHEP benchmark into student scale item Findings All quantitative survey responses were coded, entered into the data editor and analyzed using the Statistical Package for the Social Sciences (SPSS), Release 9.0.0 (1998). Scale Reliability Reliability of the benchmark scale was determined by computation of Chronbachs alpha. The standardized alpha for the 14 item scale was 0.94, indicating a high degree of internal consistency (Thorndike, 1996). As depicted in Table 1, individual scale item statistics confirmed this finding, with all items exhibiting both high item-to-scale correlations and high multiple r2 values. Moreover, because deleting any item would have lowered overall scale reliability, all 14 items were justified for retention.
-4-
Reliability & Validity of a Student Scale for Assessing Internet-Based Distance Learning
Table 1 Benchmark Scale Item Statistics (items presented in the order they appeared) Scale Items I found it easy to get the information I needed about my online course(s) and complete the enrollment & registration process I was always able to gain access to my online course(s) and the applicable UMDNJ network resources (library, e-mail, etc) when needed. I was given multiple ways to interact with the teacher and other students (e.g., e-mail, discussion) in all online course(s) In my online course(s), I always received constructive and timely feedback on my assignments and questions Before starting my online course(s), I was well advised about the self-motivation and commitment needed to succeed at distance learning Before starting my online course(s), I was well advised about the technology and skills I would need to fulfill my course requirements My online instructor(s) always provided a clearly written, straightforward statement of course objectives and learning outcomes or expectations I had sufficient access to the online library resources I needed to fulfill my course objectives and complete all my assignments Before starting my online course(s), I received sufficient information about admission requirements or prerequisites, tuition and fees, books and materials, test proctoring or phone conferencing requirements, and student support services My course(s) provided me with the skills I needed to secure outside course materials through electronic databases, interlibrary loans, government archives, news services, and other sources Prior to the beginning my online course(s), I was orientated to WebCT and had the opportunity to practice using it I had convenient access to technical assistance/support whenever needed My technical support questions or problems were answered accurately or solved quickly There is a structured system in place to address student complaints about online learning Item-Scale Multiple Alpha if Corr r2 Deleted .642 .644 .936
.609 .812 .738 .774
.525 .793 .712 .830
.937 .932 .934 .933
.727
.789
.934
.742
.785
.934
.6416
.594
.937
.614
.549
.937
.703
.758
.935
.636 .692 .775 .755
.546 .716 .773 .639
.937 .935 .933 .933
-5-
Reliability & Validity of a Student Scale for Assessing Internet-Based Distance Learning
Scale Validity Validity of the benchmark scale was approached in three ways: content validity, construct validity and criterion-referenced validity (Thorndike, 1996). Content Validity. For purposes of this study, content validity was defined as the degree to which the scale properly reflected student-related dimensions of quality in Internet-based distance education. The content validity of the benchmarks themselves were separately established by the Institute for Higher Education Policy (2000). To do so, IHEP first compiled a set of 45 guidelines on Internet-based educational quality via a comprehensive literature search. It then assessed the degree to which colleges and universities were applying these guidelines by visiting six institutions identified as leaders in Internet-based distance learning. At each site, faculty, administrators and students were surveyed to determine (a) the importance of these guidelines, (b) the extent to which they were being followed, and (c) whether or not they made a difference in academic quality. After eliminating statements for which consensus could not be established, deleting those not considered mandatory, and combining standards that addressed the same issue(s), the IHEP derived its final set of 24 benchmarks. As previously described, the content validity of the scale described here was assured by carefully translating each applicable IHEP benchmark into a statement for student response. Construct Validity. For purposes of this study, construct validity was assessed by identifying the concepts underlying students scores on this scale. To determine if the scale had a meaningful component structure, it was factor analyzed. In addition, factor scores were derived on the identified components and were compared to students global ratings of quality, obtained via Part 2 of the survey questionnaire.
-6-
Reliability & Validity of a Student Scale for Assessing Internet-Based Distance Learning
All 14 scale items were included in an exploratory factor analysis. The initial components solution was rotated using the Varimax procedure, with an Eigenvalue > 1.0 used as the criterion for factor retention. As depicted in Table 2, after three iterations and using a minimum factor loading of 0.60 (Nunnally and Bernstein,1994), a meaningful two-factor solution emerged. Table 2 Factor Loadings of Benchmark Scale Items Scale Item In my online course(s), I always received constructive and timely feedback on my assignments and questions Before starting my online course(s), I was well advised about the self-motivation and commitment needed to succeed at distance learning My online instructor(s) always provided a clearly written, straightforward statement of course objectives and learning outcomes or expectations I was given multiple ways to interact with the teacher and other students (e.g., email, discussion) in online all course(s) There is a structured system in place to address student complaints about online learning My course(s) provided me with the skills I needed to secure outside course materials through electronic databases, interlibrary loans, government archives, news services, and other sources Before starting my online course(s), I was well advised about the technology and skills I would need to fulfill my course requirements Prior to the beginning my online course(s), I was orientated to WebCT and had the opportunity to practice using it I found it easy to get the information I needed about my online course(s) and complete the enrollment & registration process I had convenient access to technical assistance/support whenever needed My technical support questions or problems were answered accurately or solved quickly Before starting my online course(s), I received sufficient information about admission requirements or prerequisites, tuition and fees, books and materials, test proctoring or phone conferencing requirements, and student support services I was always able to gain access to my online course(s) and the applicable UMDNJ network resources (library, e-mail, etc) when needed I had sufficient access to the online library resources I needed to fulfill my course objectives and complete all my assignments Factor Factor 1 2 .825 .793 .791 .749 .747 .745 .259 .345 .315 .437 .409 .329
.716 .265 .245 .388 .537 .296
.377 .797 .754 .743 .692 .674
.338 .452
.608 .537
-7-
Reliability & Validity of a Student Scale for Assessing Internet-Based Distance Learning
In combination, the two factors accounted for 66.7% of the benchmark scale variance. Factor 1 consisted of seven items and accounted for 36.4% of the scale variance. Since Factor 1 items related mainly to the teaching/learning process (prerequisite skills, learning outcomes, teacher interaction and provision of two-way feedback), it was labeled Teaching-Learning Process. Factor 2 consisted of six items and accounted for 30.3% of the scale variance. Since Factor 2 items related mainly to administrative and support services (admissions and enrollment information, registration, orientation, network access, technical support), it was labeled Administrative Support. To further explore the concepts underlying students scores on the benchmark scale, standardized factor scores were computed for each case in the analysis. A Pearson ProductMoment bivariate correlation coefficient (r) was then computed between the students factor scores and their global ratings (Excellent Good Fair Poor) on the following four statements: Global Rating Statements (Survey Questionnaire, Part 2) I would rate the overall administrative process of getting online (registering, initial logon, etc.) as: (Administrative Rating) I would rate the overall quality of online instruction I received as: (Instruction Rating) I would rate the overall ease of use of the delivery technology (WebCT and related support resources such as remote library access) as: (Ease of Use Rating) Considering all factors combined, I would rate my overall online learning experience at UMDNJ-SHRP as: (Overall Rating) As evident in Table 3, both factor scores correlated positively and significantly with all global ratings. The highest correlations were observed between each factor and it most equivalent global rating, e.g., r = 0.811 for the Teaching-Learning Process factor versus the rating of instructional quality. On the other hand, the lowest correlations were observed between each factor and it complement, e.g., r = 0.394 for the Administrative Support factor versus rating of instructional quality. This was taken to indicate good convergent validity (Trochim, 2001).
-8-
Reliability & Validity of a Student Scale for Assessing Internet-Based Distance Learning
Table 3 - Correlation of Factor Scores with Students Global Ratings Factor 1 Teaching-Learning Process 2 Administrative Support **p < 0.001 *p < 0.01 Criterion-Referenced Validity. For the benchmark scale to have criterion-related validity, it should explain or predict students perceptions of the quality of their online experience and/or distinguish between groups of students with differing conceptions of that experience (Trochim, 2001). Although Table 3 provides some perspective on this component of validity, additional analysis was performed. Specifically, (1) students summed benchmark scale scores (all 14 items) were correlated with their four global rating, (2) students factor scores were regressed on their overall ratings of their online learning experience, and (3) students factor scores were used to predict their overall satisfaction/dissatisfaction with their online learning experience (logistic regression). Table 4 provides the Pearson correlation matrix among students overall benchmark scale scores and the four global ratings of their online experience. As is evident, all correlations between the overall benchmark scale scores and the four global ratings were significant and ranked from moderately strong (Admin Rating) to very strong (Overall Rating). In the latter case, students summed benchmark scores explained nearly 80% of the variance in their overall ratings of their online experience. Administrative Rating .423* .564** Instruction Rating .811** .394* Ease of Use Rating .580** .479** Overall Rating .677** .606**
-9-
Reliability & Validity of a Student Scale for Assessing Internet-Based Distance Learning
Table 4 - Pearson Correlation Matrix among Benchmark Scale Scores and Global Ratings Scale Score Scale Score Admin Rating Instruct Rating Ease Rating Overall Rating **p < 0.001 1.00 Admin Rating .661** 1.00 Instruct Rating .857** .460** 1.00 Ease Rating .738** .608** .595** 1.00 Overall Rating .893** .630** .802** .690** 1.00
To ascertain the ability of the scale factors to predict students overall rating of their online learning experience, stepwise linear regression was performed, with independent variable entry and removal criteria set to F < 0.5 and F > 0.10 respectively. Table 5 summarizes the results of this analysis. In combination, the two factors accounted for almost 80% of the variance in the overall rating of students online learning experience (adjusted R2), comparable to that explained by the summed scale score (above). Moreover, the R2 and F change statistics for the steps and the models standardized beta weights for the factors indicated that both components contribute significantly and are roughly equivalent in predicting the overall rating. Table 5 - Regression of Factor Scores on Overall Rating Factor 1 Teaching-Learning Process 2 Administrative Support ** p < 0.001 +model constant = 3.04 .060 R2 F Adjusted 2 Change Change R .450 .459 50.03** .799 .347 103.71** Stand Beta+ .662 .589
Last, to determine whether the benchmark scale factors could distinguish between groups of students with differing conceptions of their online experiences, respondents were divided according to their overall ratings into two groups: satisfied (rating of excellent or good) and less -10-
Reliability & Validity of a Student Scale for Assessing Internet-Based Distance Learning
than satisfied (ratings of fair or poor). The factor scores were entered as a block into a binary logistic regression on these two categories. The resulting model correctly predicted group membership in 95.1% of the cases (chi-square = 48.57, df = 2). Conclusions and Recommendations Psychometric analysis of this benchmark scale indicates high reliability (internal consistency) and good content, construct and criterion-related validity. Results also reveal two different dimensions underlying student conceptions of the quality of their Internet-based distance education one related to teaching-learning processes and the other associated with provision of admin...