A Knowtator: Protg plug-in for annotated corpus construction
Philip V. Ogren Division of Biomedical Informatics Mayo Clinic Rochester, MN, USA [e-mail]
Abstract
A general-purpose text annotation tool called Knowtator is introduced. Knowtator facilitates the manual creation of annotated corpora that can be used for evaluating or training a variety of natural language processing systems. Building on the strengths of the widely used Protg knowledge representation system, Knowtator has been developed as a Protg plug-in that leverages Protgs knowledge representation capabilities to specify annotation schemas. Knowtators unique advantage over other annotation tools is the ease with which complex annotation schemas (e.g. schemas which have constrained relationships between annotation types) can be defined and incorporated into use. Knowtator is available under the Mozilla Public License 1.1 at http://bionlp.sourceforge.net/Knowtator.
1
Introduction
Knowtator is a general-purpose text annotation tool for creating annotated corpora suitable for evaluating Natural Language Processing (NLP) systems. Such corpora consist of texts (e.g. documents, abstracts, or sentences) and annotations that associate structured information (e.g. POS tags, named entities, shallow parses) with extents of the texts. An annotation schema is a specification of the kinds of annotations that can be created. Knowtator provides a very flexible mechanism for defining anno-
tation schemas. This allows it to be employed for a large variety of corpus annotation tasks. Protg is a widely used knowledge representation system that facilitates construction and visualization of knowledge-bases (Noy, 2003) 1 . A Protg knowledge-base typically consists of class, instance, slot, and facet frames. Class definitions represent the concepts of a domain and are organized in a subsumption hierarchy. Instances correspond to individuals of a class. Slots define properties of a class or instance and relationships between classes or instances. Facets constrain the values that slots can have. Protg has garnered widespread usage by providing an architecture that facilitates the creation of third-party plug-ins such as visualization tools and inference engines. Knowtator has been implemented as a Protg plug-in and runs in the Protg environment. In Knowtator, an annotation schema is defined with Protg class, instance, slot, and facet definitions using the Protg knowledge-base editing functionality. The defined annotation schema can then be applied to a text annotation task without having to write any task specific software or edit specialized configuration files. Annotation schemas in Knowtator can model both syntactic (e.g. shallow parses) and semantic phenomena (e.g. protein-protein interactions).
2
Related work
There exists a plethora of manual text annotation tools for creating annotated corpora. While it has been common for individual research groups to build customized annotation tools for their specific
1
http://protege.stanford.edu
273
Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume, pages 273275, New York City, June 2006. c 2006 Association for Computational Linguistics
Figure 1 Simple co-reference annotations in Knowtator annotation tasks, several text annotation tools have emerged in the last few years that can be employed to accomplish a wide variety of annotation tasks. Some of the better general-purpose annotation tools include Callisto 2 , WordFreak 3 (Morton and LaCivita, 2003), GATE 4 , and MMAX2 5 . Each of these tools is distributed with a limited number of annotation tasks that can be used out of the box. Many of the tasks that are provided can be customized to a limited extent to suit the requirements of a users annotation task via configuration files. In Callisto, for example, a simple annotation schema can be defined with an XML DTD that allows the creation of an annotation schema that is essentially a tag set augmented with simple (e.g. string) attributes for each tag. In addition to configuration files, WordFreak provides a plug-in architecture for creating task specific code modules that can be integrated into the user interface. A complex annotation schema might include hierarchical relationships between annotation types and constrained relationships between the types. Creating such an annotation schema can be a formidable challenge for the available tools either
http://callisto.mitre.org http://wordfreak.sourceforge.net 4 http://gate.ac.uk/. GATE is a software architecture for NLP that has, as one of its many components, text annotation functionality. 5 http://mmax.eml-research.de/.
3 2
because configuration options are too limiting or because implementing a new plug-in is too expensive or time consuming.
3
3.1
Implementation Annotation schema
Knowtator approaches definition the of an annotation schema as a knowledge engineering task by leveraging Protgs strengths as a knowledgebase editor. Protg has user interface components for defining class, instance, slot, and facet frames. A Knowtator annotation schema is created by defining frames using these user interface components as a knowledge engineer would when creating a conceptual model of some domain. For Knowtator the frame definitions model the phenomena that the annotation task seeks to capture. As a simple example, the co-reference annotation task that comes with Callisto can be modeled in Protg with two class definitions called markable and chain. The chain class has two slots references and primary_reference which are constrained by facets to have values of type markable. This simple annotation schema can now be used to annotate co-reference phenomena occur-
274
ring in text using Knowtator. Annotations in Knowtator created using this simple annotation schema are shown in Figure 1. A key strength of Knowtator is its ability to relate annotations to each other via the slot definitions of the corresponding annotated classes. In the co-reference example, the slot references of the class chain relates the markable annotations for the text extents the cat and It to the chain annotation. The constraints on the slots ensure that the relationships between annotations are consistent. Protg is capable of representing much more sophisticated and complex conceptual models which can be used, in turn, by Knowtator for text annotation. Also, because Protg is often used to create conceptual models of domains relating to biomedical disciplines, Knowtator is especially well suited for capturing named entities and relations between named entities for those domains. 3.2
Features
modified. Annotation data can be exported to a simple XML format. Annotation filters can be used to view a subset of available annotations. This may be important if, for example, viewing only named entity annotations is desired in an annotation project that also contains many part-of-speech annotations. Filters are also used to focus IAA analysis and the export of annotations to XML. Knowtator can be run as a stand-alone system (e.g. on a laptop) without a network connection. For increased scalability, Knowtator can be used with a relational database backend (via JDBC). Knowtator and Protg are provided under the Mozilla Public License 1.1 and are freely available with source code at http://bionlp.sourceforge.net/ Knowtator and http://protege.stanford.edu, respectively. Both applications are implemented in the Java programming language and have been successfully deployed and used in the Windows, MacOS, and Linux environments.
In addition to its flexible annotation schema definition capabilities, Knowtator has many other features that are useful for executing text annotation projects. A consensus set creation mode allows on...