224S/LING CS 281 Speech Recognition, Synthesis, and Dialogue
Dan Jurafsky Lecture 15: ASR: Search (Lattices, N-best lists, A*, etc) and Scoring (sclite)
Evaluation
How to evaluate the word string output by a speech recognizer?
Word Error Rate
Word Error Rate = 100 (Insertions+Substitutions + Deletions) -----------------------------Total Word in Correct Transcript Aligment example: REF: portable **** PHONE UPSTAIRS last night so HYP: portable FORM OF STORES last night so Eval I S S WER = 100 (1+2+0)/6 = 50%
NIST sctk-1.3 scoring software:
Computing WER with sclite
http://www.nist.gov/speech/tools/ Sclite aligns a hypothesized text (HYP) (from the recognizer) with a correct or reference text (REF) (human transcribed)
id: (2347-b-013) Scores: (#C #S #D #I) REF: was an engineer and they HYP: was an engineer and they Eval: 9 3 1 2 SO I i was always with **** **** MEN UM ** AND i was always with THEM THEY ALL THAT D S I I S S
Sclite output for error analysis
CONFUSION PAIRS Total With >= (%hesitation) ==> on the ==> that but ==> that a ==> the four ==> for in ==> and there ==> that (%hesitation) ==> and (%hesitation) ==> the (a-) ==> i and ==> i and ==> in are ==> there as ==> is have ==> that is ==> this (972) 1 occurances (972) 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 6 6 5 4 4 4 4 3 3 3 3 3 3 3 3 3 -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> ->
Sclite output for error analysis
17: 18: 19: 20: 21: 22: 23: 24: 25: 26: 27: 28: 29: 30: 31: 32: 33: 34: 3 3 3 3 3 2 2 2 2 2 2 2 2 2 2 2 2 2 -> it ==> that -> mouse ==> most -> was ==> is -> was ==> this -> you ==> we -> (%hesitation) ==> -> (%hesitation) ==> -> (%hesitation) ==> -> (%hesitation) ==> -> a ==> all -> a ==> know -> a ==> you -> along ==> well -> and ==> it -> and ==> we -> and ==> you -> are ==> i -> are ==> were it that to yeah
Better metrics than WER?
WER has been useful But should we be more concerned with meaning (semantic error rate)?
Good idea, but hard to agree on Has been applied in dialogue systems, where desired semantic output is more clear
Part II: Search (= Decoding)
Speeding things up: Viterbi beam decoding Problems with Viterbi decoding Multipass decoding
N-best lists Lattices Word graphs Meshes/confusion networks
A* search
Speeding things up
Viterbi is O(N2T), where N is total number of HMM states, and T is length This is too large for real-time search A ton of work in ASR search is just to make search faster:
Beam search (pruning) Fast match Tree-based lexicons
Beam search
Instead of retaining all candidates (cells) at every time frame Use a threshold T to keep subset:
At each time t Identify state with lowest cost Dmin Each state with cost > Dmin+ T is discarded (pruned) before moving on to time t+1 Unpruned states are called the active states
Viterbi Beam Search
A:
A
bA(1)
bA(2)
bA(3)
bA(4)
B:
B
bB(1)
bB(2)
bB(3)
bB(4)
C:
C
bC(1)
bC(2)
bC(3)
bC(4)
t=0
t=1
t=2
t=3
t=4
Slide from John-Paul Hosom
Viterbi Beam search
Is the most common and powerful search algorithm for LVCSR Note:
What makes this possible is time-synchronous We are comparing paths of equal length For two different word sequences W1 and W2:
We are comparing P(W1|O0t) and P(W2|O0t) Based on same partial observation sequence O0t So denominator is same, can be ignored
Time-asynchronous search (A*) is harder
Viterbi Beam Search
Empirically, beam size of 5-10% of search space Thus 90-95% of HMM states dont have to be considered at each time t Vast savings in time.
On-line processing
Problem with Viterbi search
Doesnt return best sequence til final frame
This delay is unreasonable for many applications. On-line processing usually smaller delay in determining answer at cost of always increased processing time.
14
On-line processing
At every time interval I (e.g. 1000 msec or 100 frames):
At current time tcurr, for each active state qtcurr, find best path P(qtcurr) that goes from from t0 to tcurr (using backtrace ()) Compare set of best paths P and find last time tmatch at which all paths P have the same state value at that time If tmatchexists {
Output result from t0 to tmatch Reset/Remove values until tmatch Set t0 to tmatch+1 }
Efficiency depends on interval I, beam threshold, and how well the observations match the HMM.
Slide from John-Paul Hosom
15
On-line processing
Example (Interval = 4 frames):
best sequence A:
1(A)
2(A)
3(A)
4(A)
BBAA
B:
1(B)
2(B)
3(B)
4(B)
BBBB
C:
1(C)
2(C)
3(C)
4(C)
BBBC
t=1
t0=1
t=2
t=3
t=4
tcurr=4
In this case, at time 4, all best paths for all states A, B, and C have state B in common at time 2. So, tmatch = 2. Now output states BB for times 1 and 2, because no matter what happens in the future, this will not change. Set t0 to 3
Slide from John-Paul Hosom
On-line processing
Interval=4 A: best sequence
3(A)
4(A)
5(A)
6(A)
7(A)
8(A)
BBABBA
B:
3(B)
4(B)
5(B)
6(B)
7(B)
8(B)
BBABBB
C:
3(C)
4(C)
5(C)
6(C)
7(C)
8(C)
tcurr=8
BBABBC
t=3
t0=3
t=4
t=5
t=6
t=7
t=8
Now tmatch = 7, so output from t=3 to t=7: BBABB, then set t0 to 8. If T=8, then output state with best 8, for example C. Final result (obtained piece-by-piece) is then BBBBABBC
Slide from John-Paul Hosom
17
Problems with Viterbi
Its hard to integrate sophisticated knowledge sources
Trigram grammars Parser-based LM
long-distance dependencies that violate dynamic programming assumptions
Knowledge that isnt left-to-right
Following words can help predict preceding words
Solutions
1. Return multiple hypotheses and use smart knowledge to rescore them 2. Use a different search algorithm, A* Decoding (=Stack decoding)
Multipass Search
Ways to represent multiple hypotheses
N-best list
Instead of single best sentence (word string), return ordered list of N sentence hypotheses
Word lattice
Compact representation of word hypotheses and their times and scores
Word graph
FSA representation of lattice in which times are represented by topology
Another Problem with Viterbi
The forward probability of observation given word string
The Viterbi algorithm makes the Viterbi Approximation
Approximates probability of observation given word, with prob of observation given only best state sequence.
Solving the best-path-notbest-words problem
Viterbi returns best path (state sequence) not best word sequence
Best path can be very different than best word string if words have many possible pronunciations
Two solutions
1) Modify Viterbi to sum over different paths that share the same word string. Do this as part of N-best computation Compute N-best word strings, not N-best phone paths 2) Use a different decoding algorithm (A*) that computes true Forward probability.
Sample N-best list
N-best lists
Again, we dont want the N-best paths That would be trivial
Store N values in each state cell in Viterbi trellis instead of 1 value
But:
Most of the N-best paths will have the same word string
Useless!!!
It turns out that a factor of N is too much to pay
Computing N-best lists
In the worst case, an admissible algorithm for finding the N most likely hypotheses is exponential in the length of the utterance.
S. Young. 1984. Generating Multiple Solutions from Connected Word DP Recognition Algorithms. Proc. of the Institute of Acoustics, 6:4, 351-354.
For example, if AM and LM score were nearly identical for all word sequences, we must consider all permutations of word sequences for whole sentence (all with the same scores). But of course if this is true, cant do ASR at all!
Computing N-best lists
Instead, various non-admissible algorithms:
(Viterbi) Exact N-best (Viterbi) Word Dependent N-best
And one admissible
A* N-best
Exact N-best for timesynchronous Viterbi
Due to Schwartz and Chow; also called sentence-dependent N-best Idea: each state stores multiple paths Idea: maintain separate records for paths with distinct word histories
History: whole word sequence up to current time t and word w When 2 or more paths come to the same state at the same time, merge paths w/same history and sum their probabilities.
i.e. compute the forward probability within words
Otherwise, retain only N-best paths each for state
Exact N-best for timesynchronous Viterbi
Efficiency:
Typical HMM state has 2 or 3 predecessor states within word HMM So for each time frame and state, need to compare/merge 2 or 3 sets of N paths into N new paths. At end of search, N paths in final state of trellis give N-best word sequences Complexity is O(N)
Still too slow for practical systems
N is 100 to 1000
More efficient versions: word-dependent N-best
Word-dependent (bigram) N-best Intuition:
Instead of each state merging all paths from start of sentence We merge all paths that share the same previous word
Details:
This will require us to do a more complex traceback at the end of sentence to generate the N-best list
Word-dependent (bigram) N-best
At each state preserve total probability for each of k << N previous words
K is 3 to 6; N is 100 to 1000
At end of each word, record score for each previous word hypothesis and name of previous word
So each word ending we store alternatives
But, like normal Viterbi, pass on just the best hypothesis At end of sentence, do a traceback
Follow backpointers to get 1-best But as we follow pointers, put on a queue the alternate words ending at same point On next iteration, pop next best
Word Lattice
Each arc annotated with AM and LM logprobs
Word Graph
Timing information removed Overlapping copies of words merged AM information removed Result is a WFST Natural extension to N-gram language model
Converting word lattice to word graph
Word lattice can have range of possible end frames for word Create an edge from (wi,ti) to (wj,tj) if tj-1 is one of the end-times of wi
Bryan Pelloms algorithm and figure, from his slides
Lattices
Some researchers are careful to distinguish between word graphs and word lattices But well follow convention in using lattice to mean both word graphs and word lattices. Two facts about lattices:
Density: the number of word hypotheses or word arcs per uttered word Lattice error rate (also called lower bound error rate): the lowest word error rate for any word sequence in lattice
Lattice error rate is the oracle error rate, the best possible error rate you could get from rescoring the lattice. We can use this as an upper bound
Posterior lattices
We dont actually compute posteriors:
Why do we want posteriors?
Without a posterior, we can choose best hypothesis, but we cant know how good it is! In order to compute posterior, need to
Normalize over all different word hypothesis at a time
Align all the hypotheses, sum over all paths passing through word
Mesh = Sausage = pinched lattice
One-pass vs. multipass
Potential problems with multipass Cant use for real-time (need end of sentence)
(But can keep successive passes really fast)
Each pass can introduce inadmissible pruning
(But one-pass does the same w/beam pruning and fastmatch)
Why multipass
Very expensive KSs. (NL parsing,higher-order n-gram, etc) Spoken language understanding: N-best perfect interface Research: N-best list very powerful offline tools for algorithm development N-best lists needed for discriminant training (MMIE, MCE) to get rival hypotheses
A* Decoding = Stack decoding
Intuition:
Viterbi wastes a lot of time on breadth-first search
Computing lots of paths will never need
If we had good heuristics for guiding decoding We could do depth-first (best-first) search and not waste all our time on computing all those paths at every time step as Viterbi does.
A* decoding, also called stack decoding, is an attempt to do that. A* also does not make the Viterbi assumption Uses the actual forward probability, rather than the Viterbi approximation
The search space for A* is the set of possible sentences
If music be the food of love
(1) Start with NULL as root of sentence tree, set n to 0 (2) Determine every possible word starting at time t=1, adding to the stack the words (now a partial sentence), their end times, and scores (e.g. log probabilities, so closer to zero is better), with a link to the NULL partial sentence.
A* Search
(end=100,score=-33) Alice
If
(end=0,
NULL
(end=280,score=-25) Every
score= -INF)
(end=310,score=-40) In
(end=90,score=-250)
Slide from John-Paul Hosom
A* search
(3) Pop partial sentence with highest score, P, off the stack (keeping the word, time, score, and link information for future use) (4) If P is a complete sentence (end time of last word in partial sentence = T), then (a) output sentence by following links to all previous words in sentence, (b) increment n. (c) If n == N, then stop; otherwise, go to (3) (5)Determine every possible word starting at time t=(end time of last word in P), adding to the stack the new words, their end times, and scores, with a link to the last word in P. (6)Go to step (3)
Slide from John-Paul Hosom
A* Search
(end=100, If
score=-33) score=-25) score=-40)
(end=310,
was
score=-35)
(end=0,
NULL
(end=280,
Alice
(end=360,
wants
score=-41)
score=-INF)
and the next step:
(end=310,
Every
(end=370,
walls
score=-321)
(end=90...