change Supporting and scholarship Blackwell Publishing Ltd.Oxford, UKBJETBritish Journal of Educational Technology0007-1013British Educational Communications and Technology Agency, 20052005366957974Articles British Journal of Educational Technology
British Journal of Educational Technology
Vol 36 No 6 2005
957974
Supporting change and scholarship through review of online resources in professional development settings
Jason Ravitz and Christopher Hoadley
Jason Ravitz of the Buck Institute for Education specialises in the design and evaluation of educational technologies and research on problem- and project-based learning. Christopher Hoadley is Assistant Professor in the College of Education and in the School of Information Sciences and Technology at Penn State University. Address for correspondence: Jason Ravitz, Research Director, Buck Institute for Education, 18 Commercial Blvd. Novato, CA 94949. Tel: [phone]; email: [e-mail].
Abstract How can we accelerate innovation and ensure effective dissemination of knowledge about online learning resources? This paper advocates strategies that systematically link online professional development with the research, development and diffusion cycle. The systemic approach we describe can accelerate knowledge advancement and help manage change by improving communication among teachers, trainers, developers and researchers. The examples that are provided are set within two funded projects in the United States that led to the development of two distinct but related strategiesthe Online Site Evaluation Form for educators (a web-based review form used in workshops) and a six-week online course on technology-supported assessments. Both strategies make it easier to give feedback to developers and offer incentives to do so in ways that help teachers to learn about online resources individually and with colleagues. The examples are discussed with analysis of their strengths and weaknesses in supporting different modes of interaction. We highlight implications for instructional development, professional development, research and knowledge management in online communities.
Introduction The problem of ensuring the development and use of online resources for learning is a many-faceted one. It may be viewed from the perspective of accelerating innovation and diffusion of new practices. How can the best ideas for change be generated, shared and evaluated in a more efcient fashion? The problem can be viewed in terms of professional development: how can we help teachers understand, select and use resources? It can also be viewed from a developers perspective: how can we improve feedback on the quality of digital resources? The problem can also be viewed from a scholarly perspec British Educational Communications and Technology Agency, 2005. Published by Blackwell Publishing, 9600 Garsington Road, Oxford, OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA.
958
British Journal of Educational Technology
Vol 36 No 6 2005
tive: how can we ensure the most cumulative understanding of digital resources, their usefulness and ways they work best? In this paper, we examine one way to potentially work towards each of these goals: embedding teacher feedback in the professional development process in a way that is not merely a learning exercise for the teachers but also contributes to the knowledge of others. What we are suggesting is designing systems for online professional development that: 1. enable educators to give feedback to developers about the usefulness and usability of online educational resources, allowing developers to hit the mark more often and to rely upon educators feedback in their work; 2. provide incentives for educators to offer reviews by building these experiences into professional development workshops, teacher education classes and graduate-level courses; and 3. make it easier to create shared and sustainable databases and discussions of online resources that can be accessed and added to by others. The dynamic nature of online resources will be managed as each cohort updates and adds to the resource database. Below, we examine the systemic needs of education with respect to online resources and the prospect that teachers might provide valuable data as part of their professional development activities. This is followed by two examples and a discussion of implications for the future, including changes in how various stakeholders in the educational system can work together to support innovation.
Why do we need more systematic review of digital resources? The need for systemic change towards more effective use of digital resources is felt by a number of stakeholder communities. Teachers want to locate and use the highest quality resources; policymakers and evaluators want to identify proven and tested programs; developers want their tools rened and disseminated to appropriate audiences. These issues map onto three ongoing and related challenges: (1) professional development or training for using online resources, (2) evaluation of resources for purposes of research and development, and (3) dissemination and reuse of knowledge and practices related to knowledge management and metadata. We examine each need below.
Professional development needs A professional development system that allows educators to be introduced to new technologies and resources is needed. As noted by Wiley (2000), it is relatively easy for teachers to pick up and use larger reusable digital resources... entire web pages that combine text, images and other media or applications to deliver complete experiences, such as a complete instructional event (p. 7). What seems to be lacking is a system for helping educators to continually learn about new educational resources, keeping track of valued resources and making recommendations to others about best practices for technology-supported learning.
British Educational Communications and Technology Agency, 2005.
Supporting change and scholarship
959
Teachers and teacher educators require critical evaluation skills in order to assess online resources and make good decisions about technology use (Peck, Augustine & Popp, 2003). Ultimately, their analysis of resources must include not just consideration of basic qualities of web design, but also awareness of the structures and processes that provide opportunities for teacher and student learning, and consideration of artifacts of resource use such as examples of student work, project ideas, lesson plans or rubrics. Walton and Archer (2004) note that it is necessary to provide models and opportunities for practicing evaluation of sources and for developing their own critical frameworks (pp. 174175); this includes exploration and scaffolding in order to facilitate critical and evaluative use of the Web (p. 184). Several researchers of reusable learning objects have noted that locating, reviewing and selecting educational resources via the Internet are authentic problem solving tasks that can be shared with others (Chitwood, May, Bunnow & Langan, 2000; Gilbert & Driscoll, 2002; Orrill, 2000). As noted by Barbera (2004), with the proliferation of online distance learning comes a requirement for rigorous selection and an analysis of quality (and) articulated dimensions to evaluate the quality of virtual environments (p. 17). In short, assessing information resources and their potential usefulness is recognised as an important goal for teachers (National Educational Technology Standards for Teachers, 2002a, 2002b). Evaluation, research and development needs Evaluation is another critical component in the technology research and development process. It is critical for the eld to continually investigate how technology can help achieve educational objectives, in what situations, with which students and over what period of time (CEO Forum, 2001, p. 21). As part of formative evaluation efforts, developers are continually seeking ways to interact with their users in a meaningful way, particularly at a distance (McAndrew, Clow, Taylor & Aczel, 2004). However, data collection on the effectiveness of online resources is often difcult because the developers may have very limited contact with the users. In the area of summative evaluation, we are still a long way from knowing how to assess large-scale technology impacts in general (eg, see Haertel & Means, 2003), let alone impacts from reusable educational technologies across the Internet. It can be difcult to document the impacts of online resources when they are used. A report edited by Frechtling (1995) determined that in order to reach a solution it is critical to create non-threatening synergy between dissemination, feedback, and evaluation functions of project management. Hoadley and Pea (2002) question why there is so little accumulated knowledge with billions of dollars spent on educational technology, and they point to the divide between academic researchers and practitioners and industry (p. 330). A similar conclusion was reached by the CEO Forum (2001):
Technology companies must be willing to participate in external evaluations of their products and services... Schools and districts should pilot new assessment tools and provide feedback on how to improve them... . The federal government, states, institutions of higher learning and foundations should develop mechanisms to share best practices in order to accelerate the national process of learning how to use technology most effectively. (p. 21).
British Educational Communications and Technology Agency, 2005.
960
British Journal of Educational Technology
Vol 36 No 6 2005
In short, systems for collecting feedback from educational users of online resources are critical for both formative and summative research and evaluation of educational technologies. Dissemination and reuse needs Another way to conceive the gap that exists amongst stakeholders is to examine it as a knowledge management and diffusion problem. One gap is between technology users and technology developers, making it more difcult for developers to seek and obtain feedback from users. Another gap is between expert and novice educators, creating a knowledge management and diffusion problem. Early adopters and innovators need ways to reect on and share their technology exploits more effectively (Rogers, 1983). This is to help others avoid having to wade through the Internet quagmire looking for critical information about project after project (Orrill, 2000). Many have attempted to solve this problem with technologies such as metadata tagging and search engines. One currently prominent solution is that of resource repositories, in which digital resources developed specically for teaching purposes, by those who teach, are housed, catalogued and described, in ways that make them accessible across institutions (Malcolm, 2005, p. 34). Examples include projects like MERLOT, the US National Science Digital Library, the Gateway to Educational Materials (GEM) and others. These virtual online libraries provide searchable collections of instructional resources, often judged by expert reviewers to be worthy of inclusion or based on the descriptions of the developer (Recker & Wiley, 2001). However, these diffusion mechanisms have typically lacked a cycle of feedback that is essential for advancing knowledge from use of these online resources and innovations. Ely and Huberman (1994) highlight the fact that structures and processes for dissemination must include an evaluative component in order to support not just spreading innovations, but also to help educators to choose from alternative resources and to exchange information about resources and their implementation. Technology may solve the problem of cataloguing, but it will never solve the problems of curation and building quality collections. It is crucial to nd ways to incorporate teacher-generated metadata from diverse contexts into online libraries. In portals like MERLOT and others, contributions from the user perspective that would inform others are often lacking (Recker & Wiley, 2001). Many online resource libraries have a rate it or offer feedback link, but the discussion areas in these libraries are often unpopulated. Without teachers reviewing the materials and discussing them, online libraries risk becoming empty spaces, devoid of life. Feedback is needed to close the loop among various stakeholders of online educational resources (Shepherd 2000) and catalysing mechanisms are required to generate knowledge from use of these resources (Malcolm, 2005). It is often difcult to generate metadata from teachers perspectives. Major challenges to effective reuse of learning objects noted by Recker and Wiley (2001) include the knowledge elicitation bottleneck and difculty in codifying social context. People are loath to explicitly contribute reviews without some kind of incentive; hence it is difcult
British Educational Communications and Technology Agency, 2005.
Supporting change and scholarship
961
to seed and grow a metadata database (Recker, Walker & Wiley, 2000, p. 18). We argue that these obstacles can be overcome through online professional development efforts that populate databases and that include information about opportunities for interaction with participants. In our view, the existence of an authoritative template could make it easier for teachers to provide non-authoritative reviews by allowing their focus to be on the task of reviewing, without being required to learn and apply cataloguing schemes rst. How can we support more systematic review of digital resources? The case for reviewing as teacher professional development As stated in the introduction, our approach is to connect the dots by linking teachers needs for authentic professional development on locating, selecting and reviewing online resources with evaluation and dissemination. Our approach relies on cohorts of teachers going through professional development workshops and classes each year that address educational technologies. Online courses that deal with educational technology (or face-to-face courses with online components) have the potential to generate a wellspring of knowledge about teaching and learning, not just among individual learners, but also across the eld. Generating feedback from educators as part of their professional development experience is feasible and need not be a distraction from the goals of teachers professional development. When these goals are wedded, it can help full the promise of the Internet as a collaborative learning environment (Moallem, 2003) and help overcome the isolation of educational stakeholders. Glenn and Whitaker (2003) call for a structure for creating, evaluating, and sharing open resource materials for educators. Teacher educators (including workshop leaders) can effectively promote trials of new tools and resources through online activities. They can provide test cases and data about teacher perceptions of various resources (lesson plans, rubrics, tools) including assessments of their value, and how to better facilitate use and reuse in diverse settings. When they test resources in their classrooms, they can provide data that can be used by themselves and others. This approach draws on the traditions of participatory evaluation or action research for purposes of resource evaluation (Riding, Fowell & Levy, 1995). Data generated by educators may be the most relevant possible information for other educators as well as for other interested stakeholders (Weinberger, 2004). And, more importantly, teachers would have strong incentives to produce and share these data if it were incorporated into the professional development system. Feedback on online resources and learning objects has a breadth of audiences and can be obtained from cohorts of educators on an ongoing basis when included as part of teachers professional development. Our notion of feedback goes beyond simple ratings of websites to include more detailed, user-centred and contextually based reviews. What we are recommending is not just to create databases of highly rated materials based on expert use of evaluation rubrics (MERLOT, 2005), but a design that encourages meaningful discussions to occur within
British Educational Communications and Technology Agency, 2005.
962
British Journal of Educational Technology
Vol 36 No 6 2005
and across these infrastructures among educators, intermediaries or liaisons and developers. The shift that has to occur is this: instead of only being accountable to the learning of individual educators who might use these resources, teachers and teacher educators should be enabled and encouraged to contribute to the larger enterprise of educational technology research. Examination and tagging of online resources can create conditions for individual learning and personal knowledge management as well as for diffusion and support of organisational and group learning. Recker and Wiley (2001) note that users of learning objects can generate non-authoritative metadata which is critical in supporting effective discovery and reuse (p. 3). They devised a reection and ranking tool that captures context of use and subjective user ratings (p. 13). In this case, the teachers are not merely unpaid librarians (or unpaid google-bots) cataloguing resources, they are building a rich knowledge base (both for themselves and others) about the implications of electronic resources. The construction of such a knowledge base is not only a valuable professional development activity, but also a valuable scholarly one. Two examples of systematic review for professional development The potential of the general approach described above is demonstrated by the two examples that follow. In each case, teachers learned about different online tools as a part of their professional development. They offer feedback to developers and have the opportunity to discuss their understandings. The work being presented was conducted with support from two large collaborative projects funded by the National Science Foundation: the National School Network (NSN, 199497) and Center for Innovative Learning Technologies (CILT, 19972004). In the NSN project, systematic review took place via a Site Feedback Form. In the CILT project, systematic review took place in an online netcourse. The purpose of the projects was similar. The purpose of NSN was to unite researchers, administrators, teachers, and students in building local infrastructures to improve educational processes (Hunter, 1995). This included promoting knowledge sharing both within and across school communities (Hunter, 2002) and with diverse stakeholders. Similarly, the vision for CILT as described by Hoadley and Pea (2002) was to form a learning community of researchers, developers, and practitioners from academia, government, industry, and education (p. 331). Example 1: online site evaluation form for educators The rst example of teacher feedback on digital resources comes from the National School Network (NSN) project in the US. Many educators expressed the need for evaluative information concerning Internet sites and resources that might be used for educational purposes (eg, Becker, 1995). In response, NSN developed an experiment to create a distributed large-scale peer reviewing system. In this system, members would contribute reviews of websites and benet by having reviews of their sites made available.
British Educational Communications and Technology Agency, 2005.
Supporting change and scholarship
963
A group of NSN participants developed a series of criteria that teachers, students and community partners would use to evaluate and categorise websites and web-based curriculum. The criteria focused on different types of educational usefulness that were considered exemplary. The form (see Figure 1) was designed to provide background information about the reviewers and then teachers would go back and forth between the website being reviewed and the review form using multiple views or windows. Participants completed the form during professional development sessions or university classes in which they examined the qualities of different sites and discussed criteria for judging quality. Each checklist was accompanied by open-ended text boxes for explaining responses or for offering new criteria. Reviews were posted to an online searchable database. Owners of sites could request user feedback without having to build their own data collection tools, and reviewers of sites could request feedback of their own sites. A discussion feature allowed participants to discuss the usefulness of sites with instructors and developers and to respond to each other. The tool was used nationally by the Online Internet Institute (Ravitz & Serim, 1997), in courses at Syracuse University, at nearby LeMoyne College, and within a consortium of schools in Central New York (Ravitz & Lake, 1996). Use of the form was spontaneously adopted by teacher educators at Vermont College and Northern Illinois University (without NSNs knowledge until reviews started coming in). This, coupled with unsolicited reviews from individuals, illustrated the inherent appeal of the project. A next step was for large web providers to request reviews in order to both increase awareness of their sites and to receive helpful feedback. Collings and Pearce (2002) noted the usefulness of receiving reviews from remote users as offering a valuable and different perspective. They concluded that a site could be evaluated by multiple users or user groups, to the advantage of all concerned (p. 276). In summary, the form was useful in several ways: (1) for developing a collection of peerreviewed web resources, (2) as an instructional activity for pre-service or in-service educators, (3) as a means of obtaining feedback on the quality of educational websites, and (4) as a tool for researchers and developers. Teachers who used the form found it to be effective as a professional activity, providing the following comments in their feedback about the form itself: The form was very easy to use and really made the user think about the page he/she was reviewing. There are enough choices to pick from that allow the web page owners to nd out whether or not their page is effective. I am looking forward to the day when I will be able to use this tool regularly to see what other professionals in my eld are using.
British Educational Communications and Technology Agency, 2005.
964
British Journal of Educational Technology
Vol 36 No 6 2005
Figure 1: Example of National School Network site evaluation/feedback form
British Educational Communications and Technology Agency, 2005.
Supporting change and scholarship
965
Table 1: Benets of giving and receiving feedback Teachers giving feedback learn to analyse websites or resources create a list of annotated favourites compare with reviews from others learn from interactions with others easy way to share links with colleagues Developers receiving feedback information about potential adopters and how they perceive the site increased awareness of the site for people who read the reviews
This form provided plenty of opportunities to provide feedback in my own words and also assured that the pertinent questions were addressed. A valuable resource for educators. There were over 300 reviews in the database collected over a period of two years (see accompanying URL). Table 1 outlines benets of giving and receiving feedback. The approaches we describe can help educators experience the collaborative potential of the Internet and open new opportunities for generative learning. A constructivist view of learning supports the idea of providing reviews as an added benet for the learner. Offering such a scaffold or a potentially more detailed rubric can help educators apply a set of criteria in a way that requires reection. Completing the form requires educators to apply a conceptual framework for evaluating online resources. On a daily basis, teachers may be too busy to take these steps, and they may want to see reviews more than contribute them. However, in a professional development setting, teachers can be motivated to systematically complete such forms and share feedback with the community. The resulting evaluation and discussion are noted as a strategy for generative learning (Allert, Richter & Nejdl, 2004). Applying a set of criteria may slow down a teacher initially as she reviews web resources, but there are likely benets in the added thinking that is required, including potentially improved ability to identify and compare key features. This approach to enquiry is consistent with a cognitive view of learning as described in the frames discussion of cognitive psychologists (West, Farmer & Wolff, 1991). The resulting database is useful to individual teachers. McAndrew et al (2004) notes that Despite it being much less glamorous than other e-learning tools, a shared database combining an uncomplicated, familiar interface with a powerful search engine and rich content is arguably one of the most useful knowledge management tools one can provide for motivated independent learners (p. 743). Quality concerns can in part be addressed through aggregation of peer-reviews. The results of multiple reviews will offer a better indication of resource quality than a single review. In addition, reviews that are deemed inappropriate can be addressed through the intervention of the instructor, peers, subsequent cohorts, or the developers who may write a response to a given review, or submit competing reviews. Additional trust in the
British Educational Communications and Technology Agency, 2005.
966
British Journal of Educational Technology
Vol 36 No 6 2005
quality of reviews can be created by establishing a reputation system so that one can obtain reviews from those who become trusted sources (Gauss & Urbas, 2003) perhaps contextualizing these reviews within social networks that allow networks of people to form who share similar perspectives or values. Maintenance of the database might be accomplished more easily using software tools or social arrangements. Bannan-Ritland, Dabbagh and Murphy (2000, p. 40) describe an archival engine to clear the database of unwanted and outdated contributions. Another social solution is having teachers verify or update entries in addition to (or in preparation for) contributing their own. Discussion of differences in evaluations of a website could offer unique insights. At the same time, sloppy, inaccurate or out-of-date reviews or entries consisting of spam might easily be refuted or agged by others. To illustrate this, one negative review in the NSN database was rebutted as follows: This review is not consistent with the enthusiasm of teachers shown in the guestbook... It would be appropriate for the reviewer to review the material more carefully. Potentially, this suggests an approach to decentralised authority that could borrow from the scholarship model, as outlined by Hutchings and Shulman (1999) with reviews being open to critique and evaluation, and in a form that others can build on. Example 2: netcourse on technology-supported assessments The second example of teacher feedback on digital resources comes from the CILT (http://www.cilt.org). Like NSN, CILT also promoted trials of innovations and sharing of results (Pea et al, 1999; Sabelli & Pea, 2004) with one focus being on technologysupported assessments for learning. This example is included because it highlights the relationships that can be built between developers and teacher educators. A major CILT initiative involved designing, offering and moderating a series of netcoursesonline short courses for researchers and educators. The Technology Supported Assessments (Netcourse) was conducted entirely online using Blackboard. It lasted 6 weeks and ran twice (in the spring and summer of 2001) part as of a US Department of Education PT3 Catalyst Grant involving the Concord Consortium in Concord, MA and the University of Virginia. Students in the course represented a range of individuals from teachers to school administrative professionals to higher education professors and instructional technologists; many had the responsibility for advancing technology use at their institutions, offering the potential for catalysing reuse in local settings. Enrolment in the netcourse was 20 for the rst run and 14 for the second run. The course focused on the central role of formative assessment in supporting learners (eg, Black & Wiliam, 1998) and on the available research-based tools on the Internet. Goals for the netcourse included: (1) bringing technology leaders up to speed on the latest technologies for assessment, (2) fostering reuse of the assessments (and potentially reuse of the netcourse itself) in local settings, and (3) informing tool developers and researchers about the results of the activity. Ravitz (2004) provides samples of discussions that illustrate the benets to both participants and developers of this process.
British Educational Communications and Technology Agency, 2005.
Supporting change and scholarship
967
The design of the course provided a unique introduction to using the tools that included readings, activities and discussion. Each week the class read about, tried out and discussed the applicability of these resources to their work. The course was designed to cognitively support educators using methods of Collison, Elbaum, Haavind and Tinker (2000) including a structured asynchronous format that gives students a week to complete the assignments. The syllabus with a list of readings and activities is included with the accompanying URL. Development of the course required a partnership between the course author (who was also instructor) and the assessment tool providers to create customised pathways to introduce participants to the tools. Several providers of the assessment tools did not yet have complete user-documentation and training systems in place for online users. They were keen to see how someone else could teach others to use their tool via the Internet, or how someone might manage with scant documentation. Additional collaboration with developers was required to: (1) obtain or create instructions for using the tool, (2) provide perspective on available readings and research, (3) secure passwords and login access to the tools when necessary, (4) secure technical support during the week scheduled for use, and (5) explain opportunities and costs for reuse to interested educators. Generally, there was someone from each research and development group who was available to help in these areas. Sometimes, there was a high level researcher, a technical support person and a professional development person. They wanted to explore how more educators could become users of their assessment tools. To a considerable extent, tool providers (developers) saw a unique opportunity to interact with educators. They were pleased to participate in discussions as guest experts in Blackboard during their assigned week. They monitored and responded to the progress of class participants, offering insights about their work and seeking suggestions on the best way to share it with educators. Only one tool in the netcourse was used without the developer participating in the class discussions. The instructor corresponded by email with the developer and shared the results in the class discussion area. The results of the online course demonstrate that developers are interested in receiving feedback and that with little effort they can obtain fresh eyes on a recurring basis. Developers who offer their tools in the ways described can potentially obtain feedback from cohorts of educators on an ongoing basis and from endorsements that will be seen by others. This would also support the process of taking pilot data, validating it and using it for research. This approach can support an iterative development cycle for which a crucial part of the prototyping process is the utilisation of the design with potential learners (Tripp & Bichelmeyer, 1990, p. 37). While evidence is limited, the netcourse appeared to foster reuse. One of the weeks in Blackboard was introduced to an audience of 20 faculty members at a distant college for their exploration (Ravitz, 2004). In this way, collaboration with teacher educators can help disseminate online resources and supplement data collection.
British Educational Communications and Technology Agency, 2005.
968
British Journal of Educational Technology
Vol 36 No 6 2005
Comparing the two strategies Both the Site Feedback Form and Netcourse offer generative learning strategies (Allert et al, 2004; Collison et al, 2000) for better understanding of online resources and strategies for promoting interactions that can help developers improve educational offerings. They both illustrate the potential for improved communication among stakeholders in educational change. However, the modes of interaction that are offered are different. One strength of the form was its use of the same evaluative framework. This allowed systematic collection of reviews (searching by criteria or context) and comparison of responses by different educators. A weakness of the form is that it focused more on the design of the resources than their use. In contrast, the Netcourse was particularly strong at pointing to the ways tools could be used in teaching and learning. It allowed discussion to focus not so much on the resources themselves, but on their curriculumbased applications. Some resources have indeterminate uses (eg, McAndrew et al, 2004), so a thorough review must address reasons, strategies or ways of use that are envisioned. The Netcourse enabled discussion of these issues more than the form. A weakness of the Netcourse, however, is that there was no common framework applied to the resources being reviewed and thus there was less opportunity for aggregation, searching or recommending for use by others. In future, we envision adding a common framework to the Netcourse design when possible to allow aggregation of reviews and to build on strengths of both projects. Systematic review in professional development as systemic change If widely adopted, this model of systematic review as professional development holds the potential to change feedback systems among stakeholder groups in online resource development. The strategies we have outlined may be of interest to educators who perceive a benet from interacting with and obtaining information from each other. We expect that, as part of their ongoing professional development, educators will revisit sites that offer interaction with peers and chances to learn about new resources. We think our strategies for review of online resources can support the work of online communities and may come to represent one of the few available systemic organisational frameworks that proved sustainable and scaleable across many years (Levin & Cervantes, 2002, p. 291). Easier opportunities to provide online feedback and to see feedback from others may stimulate participation and promote more meaningful interaction. Use of such a system could promote ongoing use by new cohorts of teachers and provide teacher educators with a constant resource for sustaining virtual interaction (Cole, 2002, p. xxviii). Systematic review within the context of online communities may prove to transcend problems with highly centralised approaches of the past. While centralised digital libraries have tremendous potential, they have been in the past unable to cope with the explosion of new resources arising from many quarters (Downes, 2005; Plass & Salisbury, 2002). By devolving control over which sets of criteria to use for evaluation to communities of use (ie, teachers), online communities may be able to allow different types of review criteria to be applied and evolved as quickly as the resources themselves
British Educational Communications and Technology Agency, 2005.
Supporting change and scholarship
969
change. Indeed, as Cole (2002) argues, nding ways to support communities of interest or choice has become increasingly important in modern life. To a considerable extent, movement towards collaborative knowledge building among educational stakeholders requires a shift in culture and perspective. Barab and Duffy (2000) note that schooling often fails to emphasise how learners contribute to the community: all too frequently school culture accords knowledgeable skill a reied existence, commodifying it, and turning knowledge into something to be acquired (p. 34) as opposed to something that is co-constructed and shared. However, co-construction is essential to group and organisational learning, and for the advancement of scholarship and the open source development model (Harris & Swan, 2003). In our view, functional subsystems of the educational system, such as the teacher professional development system, systems for classroom use of resources, resource evaluation and resource development, can all be connected through a common community by changing the ow of information between these subsystems. For example, the systemic feedback we envision will enable teachers to share knowledge with each other and apply analytical frameworks to resources as they work online both individually and in groups. At the same time, developers will be able to obtain feedback from cohorts of teachers in different contexts through partnerships with teacher trainers or workshop leaders. These trainers or workshop leaders responsible for professional development will help provide feedback to developers, contributing data from multiple cohorts and making this available to other educators and trainers. Researchers will have access to teacher contextual data, perceptions, online learning activities and discussions. Finally, managers of resources will more easily obtain and aggregate feedback, making recommendations to teachers about resources to use based on non-authoritative (user) reviews by teachers in similar settings or with similar goals. Will impact of reviews be limited to inuencing teacher professional development and not other subsystems? The incentive for teachers to contribute reviews during professional development is clear, but what about after their course is over? Professional development tools and websites are often introduced with hopes that teachers will return after they matriculate from workshops or classes and are no longer required to visit the site. However, it is difcult to get people to come back to websites. Using online communities for professional development at a distance can help reduce isolation in the classroom (Schlager, Fusco & Schank, 2002) but it is challenging and expensive. An ongoing problem in the design of online professional development is the sustainability of these structures and incentives to participate, especially after busy professionals return to their careers (Cole, 2002). At the individual level, if useful criteria can be developed and taught, these might be adopted and used for personal knowledge management. Some educators might choose to tag resources for their own purposes using evaluative criteria that is provided (or co-constructed) and make this information available not only to themselves but also to
British Educational Communications and Technology Agency, 2005.
970
British Journal of Educational Technology
Vol 36 No 6 2005
others, as is currently the practice with management and tagging of blogs to support human information mining (Orihuela, 2003). In addition to contributing metadata for their own use (and use by others), graduates who have used the form in the past may take an interest in seeing the data and reviews of subsequent cohorts in order to learn about the latest technologies. Thus, curiosity and self-organisation are two incentives for teachers to continue reviewing or reading reviews outside their formal professional development. The ideas we have presented can also be adopted as part of local school reform efforts that benet from being linked to others. Because the proposed approach supports teacher learning, it is consistent with a dual approach working simultaneously on individual and institutional development (Fullan, 1993, p. 12), with learning occurring at both levels and the ability to let good ideas rise above others (Lamon, Reeve & Scardamalia, 2001). McAndrew et al (2004) indicate that colleagues at their college want to know what colleagues are currently thinking, what methods and approaches are currently being used; and they want the opportunity to discuss ideas with colleagues across the university. But no one has time to attend workshops or other faceto-face events to facilitate these needs (p. 740). Professional development activities can be designed to catalyse communications in these areas. In NSN terms, the goal is for the local information infrastructure to support local needs, while both feeding into and beneting from its connection to a larger infrastructure. The strategies we described create the conditions for sharing of resources noted by Mergendoller and Kajder (2003) including helping developers seek feedback from users, collaborate with users and document pedagogically valuable materials. Summary We envision networks of educational researchers and developers systematically applying and aggregating the results of systematic reviews. They may focus on resources that apply the same design principles, assessment instruments, learning goals or teaching contexts. For instance, the Technology Enhanced Learning in Science (TELS, http:// www.telscenter.org/ centre is an NSF-funded centre for teaching and learning with an explicit mission to form an educational accelerator that seeks to connect researchers, teachers and developers through multiple feedback mechanisms ranging from systematic review of resources to systematic review of research. We anticipate that the development of shared criteria by such groups will greatly enhance the potential for aggregation of results and for broader impact. Technology should allow communities to create and apply shared criteria; for instance, the Buck Institute for Education has experimented with an authorable version of the Web Site Evaluation Form with reusable, congurable criteria (see the accompanying URL). All too often, the knowledge that people develop about online resources is not shared with others who may be interested. As a result, the utility of this knowledge is diminished, claims of utility are difcult to assess and demonstrating results is much less feasible. We hope that this will change. We urge professional developers, online community facilitators, and technology designers to consider closing the loop in
British Educational Communications and Technology Agency, 2005.
Supporting change and scholarship
971
educational systems by ensuring that knowledge gained from teacher experiences with digital resources is captured, shared and used by others.
Accompanying URL http://www.bie.org/Ravitz/bjet.html This URL will be updated with related links and examples from the discussed work.
Acknowledgements The authors thank the following for their critical review and suggestions for this work: Judi Harris, Nathan Bos, Charalambos Vrasidas, Jerry Kemp, Charles Reigeluth, Don Ely, the anonymous reviewers, and the online participants who provided feedback on these ideas during discussion of ITForum Paper #71 and Blogging In The Education World, a forum led by Bryan Alexander and Stephen Downes as part of the Online Social Networks 2005 conference.
References
Allert, H. Richter, C. & Nejdl, W. (2004). Lifelong learning and second-order learning objects. British Journal of Educational Technology, 35, 6, 701715. Bannan-Ritland, B., Dabbagh, N. & Murphy, K. (2000). Learning object systems as constructivist learning environments: related assumptions, theories, and applications. In D. A. Wiley (Ed.), The instructional use of learning objects. Retrieved 21 September, 2005, from http:// reusability.org/read/chapters/bannan-ritland.doc Barab, S. & Duffy, T. (2000). From practice elds to communities of practice. In D. Jonassen & S. Land (Eds), Theoretical foundations of learning environments (pp. 2556). Mahway, NJ: Lawrence Earlbuam Associates. Barbera, E. (2004). Quality in virtual education environments. British Journal of Educational Technology, 35, 1, 1320. Becker, H. J. (1995). National school network testbed phase 2 baseline of testbed-participating schools. Retrieved June 14, 2005, from http://www.bie.org/Ravitz/NSN_Baseline.pdf Black, P. & Wiliam, D. (1998). I...