Institute Massachusetts of Technology Department of Electrical Engineering & Computer Science 6.345/HST.728 Automatic Speech Recognition Spring, 2010 2/25/10
Lecture Handouts
Lecture Slides: Dynamic Time Warping and Search Reading: DTW: Rabiner et al., Fundamentals of ASR, Chp 4.7 Search: Huang et al., Spoken Language Processing, Chp 12 Homework: Assignment 2 - Speech Recognition using DTW
MIT
Search
Dynamic Time Warping & Search
Dynamic time warping
Graph search algorithms Dynamic programming algorithms
6.345/HST.728 Automatic Speech Recognition (2010)
Search 1
MIT
-
Word-Based Template Matching
Feature Measurement
-
Pattern Similarity
6
- Decision
Rule
-
Spoken Word
Output Word
Word Reference Templates Whole word representation: No explicit concept of sub-word units (e.g., phones) No across-word sharing Used for both isolated- and connected-word recognition Popular in late 1970s to mid 1980s; recent renewed interest
6.345/HST.728 Automatic Speech Recognition (2010) Search 2
MIT
Template Matching Mechanism
Test pattern, T , and reference patterns, {R1 , . . . , RV }, are represented by sequences of feature measurements Pattern similarity is determined by aligning test pattern, T , with reference pattern, Rv , with distortion D(T , Rv ) Decision rule chooses reference pattern, R , with smallest alignment distortion D(T , R ) R = arg min D(T , Rv )
v
Dynamic time warping (DTW) is used to compute the best possible alignment warp, v , between T and Rv , and the associated distortion D(T , Rv )
6.345/HST.728 Automatic Speech Recognition (2010)
Search 3
MIT
Alignment Example
m M M Reference Warp m
1
1
0
1
n N Test
n 0
6.345/HST.728 Automatic Speech Recognition (2010)
1
N
Search 4
MIT
Digit Alignment Examples
Match
6.345/HST.728 Automatic Speech Recognition (2010)
Mismatch
Search 5
MIT
Dynamic Time Warping (DTW)
Objective: an optimal alignment between variable length sequences T = {t1 , . . . , tN } and R = {r1 , . . . , rM } The overall distortion D(T , R) is based on a sum of local distances between elements d(ti , rj ) A particular alignment warp, , aligns T and R via a point-to-point mapping, = (t , r ), of length K tt (k) rr (k) 1 k K
The optimal alignment minimizes overall distortion D(T , R) = min D (T , R) D (T , R) = 1 M
K
d(tt (k) , rr (k) )mk
k=1
6.345/HST.728 Automatic Speech Recognition (2010)
Search 6
MIT
DTW Issues
Endpoint constraints: t (1) = r (1) = 1 Monotonicity: t (k + 1) t (k) r (k + 1) r (k) t (K) = N r (K) = M
Path weights, mk , can influence shape of optimal path Path normalization factor, M , allows comparison between different warps (e.g., with different lengths)
K
M =
k=1
mk
6.345/HST.728 Automatic Speech Recognition (2010)
Search 7
MIT
DTW Issues: Local Continuity
m
TYPE I m-1
m-2 n-2
[n,m]
m
TYPE III
[n,m]
m-1
m-2
n-1
n
[n,m]
n-2
n-1
n
[n,m]
m
TYPE II
m
TYPE IV
m-1
m-2 n-2
m-1
m-2
n-1
n
n-2
n-1
n
Local constraints determine alignment flexibility
6.345/HST.728 Automatic Speech Recognition (2010)
Search 8
MIT
DTW Issues: Global Constraints
slope = 2 (1,M) (N,M)
slope = Legal range slope = 1 2
1 2
(1,1) slope = 2
(N,1)
Local constraints exclude portions of search space
6.345/HST.728 Automatic Speech Recognition (2010) Search 9
MIT
5 4 3
Computing DTW Alignment
3 4 4 1 H 2 3 E 1 A 2 2 B 1 S 3 1 D 3 C 4 4 5 3 F 1 G 3 4 m-2 n-2 n-1 n 3 m-1 2 m 2 1 1 [m,n]
6.345/HST.728 Automatic Speech Recognition (2010)
Search 10
MIT
Graph Representations of Search Space
Search spaces can be represented as directed graphs A
- E * R - H
S
- B
- D
j - F *
R
C
j - G
Finite-state transducers (FSTs) are a popular graph representation Paths through a graph can be represented with a tree
6.345/HST.728 Automatic Speech Recognition (2010)
Search 11
MIT
Search Space Tree
S
) ? q
A
U =
B
? ~
C
U
E
?
F
?
E
?
D
?
G
?
F
?
G
?
H
H
H
F
?
H
H
H
H
6.345/HST.728 Automatic Speech Recognition (2010)
Search 12
MIT
Graph Search Algorithms
Iterative methods using a queue to store partial paths On each iteration the top partial path is removed from the queue and is extended one level New extensions are put back into the queue Search is complete when goal is reached Depth of queue is potentially unbounded Weighted graphs can be searched to find the best path Admissible algorithms guarantee finding the best path Many speech-based search problems can be configured to proceed time-synchronously
6.345/HST.728 Automatic Speech Recognition (2010)
Search 13
MIT
Depth First Search
Searches space by pursuing one path at a time Path extensions are put on top of queue Queue is not reordered or pruned Not well suited for spaces with long dead-end paths Not generally used to find the best path
6.345/HST.728 Automatic Speech Recognition (2010)
Search 14
MIT
Depth First Search Example
S
) ? q
A
U =
B
? ~
C
U
E
?
F
?
E
?
D
?
G
?
F
?
G
?
H
H
H
F
?
H
H
H
H
6.345/HST.728 Automatic Speech Recognition (2010)
Search 15
MIT
Breadth First Search
Searches space by pursuing all paths in parallel Path extensions are put on bottom of queue Queue is not reordered or pruned Queue can grow rapidly in spaces with many paths Not generally used to find the best path Can be made much more effective with pruning
6.345/HST.728 Automatic Speech Recognition (2010)
Search 16
MIT
Breadth First Search Example
S
) ? q
A
U =
B
? ~
C
U
E
?
F
?
E
?
D
?
G
?
F
?
G
?
H
H
H
F
?
H
H
H
H
6.345/HST.728 Speech Automatic Recognition (2010)
Search 17
MIT
Best First Search
Used to search a weighted graph Uses greedy or step-wise optimal criterion, whereby each iteration expands the current best path On each iteration, the queue is resorted according to the cumulative score of each partial path If path scores exhibit monotonic behavior, (e.g., d(ti , rj ) 0), search can terminate when a complete path has a better score than all active partial paths
6.345/HST.728 Automatic Speech Recognition (2010)
Search 18
MIT
Tree Representation (with node scores)
S
) A 1 U 3 = 6 ? 1 q 2 ~
B
? 3
C 6
U 3
E
?
F
?
E
?
D 1
?
G 2
?
F
?
G 1
?
H 2
H 1
H 2
F
?
3
H 1
H 1
H 1
H 1
6.345/HST.728 Automatic Speech Recognition (2010)
Search 19
MIT
Tree Representation (with cumulative scores)
S
) A 2 U 5 = 8 ? 1 q 3 ~
B
? 6
C 7
U 10
E
?
F
?
E
?
D 4
?
G 5
?
F
?
G 8
?
H 7
H 9
H 8
F
?
7
H 6
H 11
H 9
H 8
6.345/HST.728 Automatic Speech Recognition (2010)
Search 20
MIT
Best First Search Example
S
) A 2 U 5 = 8 ? 1 q 3 ~
B
? 6
C 7
E
?
F
E
?
D 4
?
G 5
?
H 7
H 8
F
7
H 6
6.345/HST.728 Automatic Speech Recognition (2010)
Search 21
MIT
Pruning Partial Paths
Both greedy and dynamic programming algorithms can take advantage of optimal substructure: Let (i, j) be the best path between nodes i and j If k is a node in (i, j): (i, j) = {(i, k), (k, j)} Let (i, j) be the cost of (i, j) (i, j) = min((i, k) + (k, j))
k
Solutions to subproblems need only be computed once Sub-optimal partial paths can be discarded while maintaining admissibility of search
6.345/HST.728 Automatic Speech Recognition (2010) Search 22
MIT
Best First Search with Pruning
S
) A 2 U 5 = 8 ? 1 q 3 ~
B
? 6
C 7
E
?
F
E
D 4
?
G 5
?
H 7
F
7
H 6
6.345/HST.728 Automatic Speech Recognition (2010)
Search 23
MIT
Estimating Future Scores
Partial path scores, (1, i), can be augmented with future ^ estimates, (i), of the remaining cost ^ = (1, i) + (i) ^ If (i) is an underestimate of the remaining cost, additional paths can be pruned while maintaining admissibility of search A search uses Best-first search strategy Pruning Future estimates
6.345/HST.728 Automatic Speech Recognition (2010)
Search 24
MIT
Tree Representation (with future estimates)
S
) A 5 U 7 = 9 ? 5 q 6 ~
B
? 8
C 9
U 10
E
?
F
?
E
?
D 6
?
G 6
?
F
?
G 9
?
H 7
H 9
H 8
F
?
8
H 6
H 11
H 9
H 8
6.345/HST.728 Automatic Speech Recognition (2010)
Search 25
MIT
A Search Example
S
) A 5 U 7 = 9 ? 5 q 6 ~
B
? 8
C 9
E
F
E
D 6
?
G 6
?
F
8
H 6
6.345/HST.728 Automatic Speech Recognition (2010)
Search 26
MIT
N -Best Search
Used to compute top N paths Can be re-scored by more sophisticated techniques Typically used at the sentence level Can use modified A search to rank paths No pruning of partial paths Completed paths are removed from queue Can use a threshold to prune paths, and still identify admissibility violations Can also be used to produce a graph Alternative methods can be used to compute N-best outputs (e.g., asynchronous DP)
6.345/HST.728 Automatic Speech Recognition (2010) Search 27
MIT
N -Best Search Example
S
) A 5 U 7 = 9 ? 5 q 6 ~
B
? 8
C 9
E
?
F
E
?
D 6
?
G 6
?
H 7
H 8
F
?
8
H 6
H 8
6.345/HST.728 Automatic Speech Recognition (2010)
Search 28
MIT
Dynamic Programming (DP)
DP algorithms do not employ a greedy strategy DP algorithms typically take advantage of optimal substructure and overlapping subproblems by arranging search to solve each subproblem only once Can be implemented efficiently: Node j retains only best path cost of all (i, j) Previous best node id needed to recover best path Can be time-synchronous or asynchronous DTW and Viterbi are time-synchronous searches and look like breadth-first with pruning
6.345/HST.728 Automatic Speech Recognition (2010)
Search 29
MIT
Time-Synchronous DP Example
A
- E j - F j - G j - H
S
- B
- D
j
C
6.345/HST.728 Automatic Speech Recognition (2010)
Search 30
MIT
Inadmissible Search Variations
Can use a beam width to prune current hypotheses Beam width can be static or dynamic based on relative score Can use an approximation to a lower bound on A lookahead for N-best computation Search is inadmissible, but may be useful in practice
6.345/HST.728 Automatic Speech Recognition (2010)
Search 31
MIT
Beam Search Example
A
- E j - F j - H
S
- B
- D
j
C
j
G
6.345/HST.728 Automatic Speech Recognition (2010)
Search 32
Search Example: Computing N-best Paths
Use forward Viterbi search in first-pass to find best path
a r Lexical Nodes z m h# t0 t1 t2 t3 t4 Time t5 t6 t7
t8
Relative and absolute thresholds used to speed-up search
6.345/HST.728 Automatic Speech Recognition (2010) Search 33
N-best Computation with Backwards A* Search
Second pass uses backwards A* search to find N-best paths Viterbi backtrace is used as future estimate for path scores
a r Lexical Nodes z m h# t0 t1 t2 t3 t4 Time t5 t6 t7
t8
Block processing enables pipelined computation (optional)
6.345/HST.728 Automatic Speech Recognition (2010) Search 34
1
MIT
Next Steps
Reading: DTW: Rabiner et al., Fundamentals of ASR, Chp 4.7 Search: Huang et al., Spoken Language Processing, Chp 12 Homework: Assignment 2: Speech Recognition using DTW We will revisit search in HMMs & FSTs How to build word lattices, confusion networks Use in multi-pass ASR (a.k.a. successive search)
6.345/HST.728 Automatic Speech Recognition (2010)
Search 35