This preview shows pages 1–2. Sign up to view the full content.
This preview has intentionally blurred sections. Sign up to view the full version.View Full Document
Unformatted text preview: The MIT Finite-State Transducer Toolkit for Speech and Language Processing Lee Hetherington Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology Cambridge, MA 02139 USA Abstract We present the MIT Finite-State Transducer Toolkit and briefly de- scribe research that it has benefitted. The toolkit is a collection of command-line tools and associated C++ API for manipulating finite-state transducers (FSTs) and acceptors (FSAs) and has been designed to enable research through its flexibility, yet remain ef- ficient enough to aid real-world computationally demanding ap- plications such as automatic speech recognition. The toolkit sup- ports the construction, combination, optimization, and training of weighted FSTs and FSAs, and as such is useful in many areas of human language technology. 1. Introduction Finite-state transducers (FSTs), possibly weighted, have long been utilized within a wide range of human language technologies in- cluding phonology, morphology, statistical language modeling, part-of-speech tagging, parsing, and speech recognition [1, 2]. FSTs can represent uncertain transformations from one level of representation to another, optionally weighting alternative interpre- tations or realizations probabilistically. FSTs allow a uniform rep- resentation, which in turn allows the use of a mathematical frame- work with powerful operations to construct, combine, and optimize them effectively. The alternative, using different representations, often interferes the effective composition of a total system from its components and subsequent optimization. In this paper, we introduce The MIT FST Toolkit we have de- veloped and are now making publicly available. The toolkit pro- vides for the construction of various types of FSTs, their combina- tion, optimization, and weight training. The operations are avail- able at both the command-line executable level, operating on files or through pipes, and through an object-oriented C++ class library for closer integration with applications. There are other toolkits available, most notably AT&Ts FSM  and GRM  packages. However, at the time of this writing, these packages are publically available only in command-line exe- cutable form for research purposes and not as embeddable libraries or source code. Development of this toolkit began in 1996 when we became frustrated with the use of different representations and algorithms scattered throughout our own speech recognition system. We had different representations for context-dependent model identifica- tion, phonological rules, lexicons, and language models. The dif- ferent representations meant there was some duplication of algo- rithms for optimizing components, and combining different levels using different representations was problematic....
View Full Document
This note was uploaded on 05/08/2010 for the course CS 6.345 taught by Professor Glass during the Spring '10 term at MIT.
- Spring '10
- Artificial Intelligence