jurafsky&martin_3rdEd_17 (1).pdf

The hidden markov model was developed by baum and

Info icon This preview shows pages 140–142. Sign up to view the full content.

The hidden Markov model was developed by Baum and colleagues at the Insti- tute for Defense Analyses in Princeton (Baum and Petrie, 1966; Baum and Eagon, 1967) . The Viterbi algorithm was first applied to speech and language processing in the context of speech recognition by Vintsyuk (1968) but has what Kruskal (1983) calls a “remarkable history of multiple independent discovery and publication”. 3 Kruskal and others give at least the following independently-discovered variants of the algorithm published in four separate fields: Citation Field Viterbi (1967) information theory Vintsyuk (1968) speech processing Needleman and Wunsch (1970) molecular biology Sakoe and Chiba (1971) speech processing Sankoff (1972) molecular biology Reichert et al. (1973) molecular biology Wagner and Fischer (1974) computer science The use of the term Viterbi is now standard for the application of dynamic pro- gramming to any kind of probabilistic maximization problem in speech and language processing. For non-probabilistic problems (such as for minimum edit distance), the plain term dynamic programming is often used. Forney, Jr. (1973) wrote an early survey paper that explores the origin of the Viterbi algorithm in the context of infor- mation and communications theory. Our presentation of the idea that hidden Markov models should be characterized by three fundamental problems was modeled after an influential tutorial by Rabiner (1989) , which was itself based on tutorials by Jack Ferguson of IDA in the 1960s. Jelinek (1997) and Rabiner and Juang (1993) give very complete descriptions of the 3 Seven is pretty remarkable, but see page ?? for a discussion of the prevalence of multiple discovery.
Image of page 140

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

E XERCISES 141 forward-backward algorithm as applied to the speech recognition problem. Jelinek (1997) also shows the relationship between forward-backward and EM. See also the description of HMMs in other textbooks such as Manning and Sch¨utze (1999) . Exercises 9.1 Implement the Forward algorithm and run it with the HMM in Fig. 9.3 to com- pute the probability of the observation sequences 331122313 and 331123312 . Which is more likely? 9.2 Implement the Viterbi algorithm and run it with the HMM in Fig. 9.3 to com- pute the most likely weather sequences for each of the two observation se- quences above, 331122313 and 331123312 . 9.3 Extend the HMM tagger you built in Exercise 10. 5 by adding the ability to make use of some unlabeled data in addition to your labeled training corpus. First acquire a large unlabeled (i.e., no part-of-speech tags) corpus. Next, im- plement the forward-backward training algorithm. Now start with the HMM parameters you trained on the training corpus in Exercise 10. 5 ; call this model M 0 . Run the forward-backward algorithm with these HMM parameters to la- bel the unsupervised corpus. Now you have a new model M 1 . Test the perfor- mance of M 1 on some held-out labeled data.
Image of page 141
Image of page 142
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern