Set Exact Matching
Lecture 8: September 27, 2005 Algorithms in Biosequence Analysis Nathan Edwards - Fall, 2005
Exact set matching
Exact set matching problem set of patterns: P = { P1, P2, ..., Pz } text T
total length n length m (as before) O( | P1| + m + ... + | Pz| + m ) = O( n + z.m )
2
Use Boyer-Moore:
Not bad, but we can do better!
Exact set matching
Aho-Corasick
O( n + m + k ) time, k is # of occurrences
Not the only way to achieve this bound, but the others preprocess T Generalize Knuth-Morris-Pratt Keyword tree / Trie
3
Keyword Tree
Rooted directed tree: 1. exactly one character on each edge 2. any two edges from the same node have distinct labels 3. every pattern corresponds to a path from the root to some node 4. every leaf corresponds to some pattern
4
Keyword Tree
P={potato, pottery, poetry, science, school} a
t o 1 p o t t e r y 2 e t r y 3 s c h i e n c e 4
5
o
o
l
5
Keyword Tree
Construction in O(n) time
Iteratively add the patterns for each pattern: start at root follow edges until either pattern is exhausted, or end at node v, so new path out of v
Constant work per pattern character
6
Keyword Tree
P={potato, pottery, poetry, science, school} a Add: t poet
o 1 p o t t e t e 6 r r y y 2 3 s c h i e n c e 4
7
o
o
l
5
Keyword Tree
P={potato, pottery, poetry, science, school, a poet} Add: t potent o
1 7
8
p o t e t
s c h i e n c e 4 o o l 5
r t e e 6 y r n y 3 t 2
Nave algorithm
At each position l of T, Follow edges from the root until a leaf is reached.
Any labeled node on the path from the root represents an occurrence of a pattern from P ending at l
Running time O(|Pmax| m) O(n m). Solves dictionary problem effectively!
9
Speedup of search
Generalize Knuth-Morris-Pratt
"Shift" pattern in jumps Avoid recomputing matches Need analogue to spi and failure functions
For now, assume that no pattern in P is a substring (proper or not) of another substring of P
10
Label of node v
Definition: Each node v is labeled with the string L(v) of characters from the root to v Definition: lp(v) is the length of the longest proper suffix of L(v) that is a prefix of some pattern in P. Lemma: If is the lp(v)-length suffix of L(v), then there is a unique node u s.t. L(u) =
11
Failure link
Definition: Let nv be the node labeled with (longest suffix of L(v) that is a prefix of some pattern of P). If lp(v) = 0, then nv is the root. Definition: The "edge" (v, nv) is called the failure link of v.
12
Failure links
P={potato, tattoo, theater, other}
a t o 1 4 p o t h e r t a t o h t e a t e r 3
13
t
o
o
2
Aho-Corasick
c = 1, w = root repeat while edge (w,w') is labeled T(c): if w' holds pattern i Pi occurs in T ending at c w = w', c ++ w = nw until c > m
14
Generalization K-M-P
of Failure links jump back in patterns, just as failure function did for K-M-P Note that we generalize spi rather than spi' [ F(i) rather than F'(i) ] "Shift" doesn't miss any patterns from P O(m) time to search, like K-M-P:
Each character matched at most once On mismatch, we jump backwards towards root so we can't mismatch too many times.
15
Failure link algorithm
Failure links can be computed in O(n) time Iterative: given nv for all nodes v with labels of length l, compute nw for all nodes w with labels of length l+1. Failure links for root and nodes with length 1 labels point to the root.
16
Failure link algorithm
v' is parent of v, x is char on edge (v',v),w = nv' while no edge of w has label x and w root w = nw if an edge (w,w') has label x nv = w' else nv = root Running time proof, and correctness, generalize K-M-P.
17
Failure links
P={potato, tattoo, theater, other}
a t o 1 4 p o t h e r t a t o h t e a t e r 3
18
t
o
o
2
Failure links
P={potato, tattoo, theater, other}
a t o 1 4 p o t h e r t a t o h t e a t e r 3
19
t
o
o
2
Failure links
P={potato, tattoo, theater, other}
a t o 1 4 p o t h e r t a t o h t e a t e r 3
20
t
o
o
2
Failure links
P={potato, tattoo, theater, other}
a t o 1 4 p o t h e r t a t o h t e a t e r 3
21
t
o
o
2
Failure links
P={potato, tattoo, theater, other}
a t o 1 4 p o t h e r t a t o h t e a t e r 3
22
t
o
o
2
Failure links
P={potato, tattoo, theater, other}
a t o 1 4 p o t h e r t a t o h t e a t e r 3
23
t
o
o
2
Relaxing the substring assumption
Recall that we assumed that P contained no substrings. Exact duplicates can be ...