CSE 549: Suffix Tries & Suffix Trees. All slides in this lecture not marked with * of Ben Langmead.

Transcription

1 CSE 549: Suffix Tries & Suffix Trees All slides in this lecture not mrked with * of Ben Lngmed.

2 KMP is gret, ut T = m P = n (note: m,n re opposite from previous lecture) Without preprocessing (KMP) Given preprocessing (KMP) Without preprocessing (ST) Given preprocessing (ST) Find n occurrence of P O(m+n) O(m) O(m+n) O(n) Find ll occurrences of P O(m+n) O(m) O(m + n + k) O(n+k) Find n occurrence of O(l(m+n)) O(l(m)) O( m + ln) O(ln) P1,,P l If the text is constnt over mny ptterns, pre-processing the text rther thn the pttern is etter (nd llows other efficient queries). *

3 Tries A trie (pronounced try ) is rooted tree representing tree representing collection collection of strings of with strings with one node per common prefix Smllest tree such tht: Ech edge is leled with chrcter c Σ A node hs t most one outgoing edge leled c, for c Σ Ech key is spelled out long some pth strting t the root Nturl wy to represent either set or mp where keys re strings This structure is lso known s Σ-tree

4 Tries: exmple Represent this mp with trie: Key Vlue instnt 1 internl 2 s i n t internet 3 t e The smllest tree such tht: Ech edge is leled with chrcter c Σ A node hs t most one outgoing edge leled c, for c Σ n t 1 l r n e t Ech key is spelled out long some pth strting t the root 2 3

5 Tries: exmple t s i n t e Checking for presence of key P, where n = P, is??? O(n) time If totl length of ll keys is N, trie hs O(N)??? nodes r Wht out Σ? n t 1 l n e t Depends how we represent outgoing edges. If we don t ssume Σ is smll constnt, it shows up in one or oth ounds. 2 3

6 Tries: nother exmple We cn index T with trie. The trie mps sustrings to offsets where they occur c 4 g 8 t 14 cc 12 cc 2 ct 6 gt 18 gt 0 t 10 tt 16 root: c g t c g t c t t t , ,

7 Indexing with suffixes Some Until indices now, (e.g. our indexes the inverted hve een index) sed re sed on extrcting on extrcting sustrings sustrings from T from T A very different pproch is to extrct suffixes from T. This will led us to some interesting nd prcticl index dt structures: A ANA ANANA BANANA NA NANA B A N A N A A B A N A N A N A B A N A N A N A B B A N A N A N A B A N A N A N A B A Suffix Trie Suffix Tree Suffix Arry FM Index

8 Trie Definitions A Σ-tree (trie) is rooted tree where ech edge is leled with single chrcter c ϵ Σ, such tht no node hs two outgoing edges leled with the sme chrcter. for node v in T, depth(v) or node-depth(v) is the distnce from v to the root. node-depth(r) = 0 string(v) = conctention of ll chrcters on the pth r v string-depth(v) = string(v) (note: string-depth(v) node-depth(v)) for string x, if node v with string(v) = x, we sy node(x) = v T displys string x if node v nd string y such tht xy = string(v) words(t) = { x T displys x} A suffix trie of string s is Σ-tree such tht words(t) = {s s is sustring of s} An internl/lef edge leds to n internl/lef node Defs. from: *

9 Suffix trie Build trie contining ll suffixes of text T T: GTTATAGCTGATCGCGGCGTAGCGG GTTATAGCTGATCGCGGCGTAGCGG TTATAGCTGATCGCGGCGTAGCGG TATAGCTGATCGCGGCGTAGCGG ATAGCTGATCGCGGCGTAGCGG TAGCTGATCGCGGCGTAGCGG AGCTGATCGCGGCGTAGCGG GCTGATCGCGGCGTAGCGG CTGATCGCGGCGTAGCGG TGATCGCGGCGTAGCGG GATCGCGGCGTAGCGG ATCGCGGCGTAGCGG TCGCGGCGTAGCGG CGCGGCGTAGCGG GCGGCGTAGCGG CGGCGTAGCGG GGCGTAGCGG GCGTAGCGG CGTAGCGG GTAGCGG TAGCGG AGCGG GCGG CGG GG G m(m+1)/2 chrs

10 Suffix trie First dd specil terminl chrcter to the end of T is chrcter tht does not pper elsewhere in T, nd we define it to e less thn other chrcters (for DNA: < A < C < G < T) enforces rule we re ll used to using: e.g. s comes efore sh in the dictionry. lso gurntees no suffix is prefix of ny other suffix. T: GTTATAGCTGATCGCGGCGTAGCGG GTTATAGCTGATCGCGGCGTAGCGG TTATAGCTGATCGCGGCGTAGCGG TATAGCTGATCGCGGCGTAGCGG ATAGCTGATCGCGGCGTAGCGG TAGCTGATCGCGGCGTAGCGG AGCTGATCGCGGCGTAGCGG GCTGATCGCGGCGTAGCGG CTGATCGCGGCGTAGCGG TGATCGCGGCGTAGCGG GATCGCGGCGTAGCGG ATCGCGGCGTAGCGG TCGCGGCGTAGCGG CGCGGCGTAGCGG GCGGCGTAGCGG CGGCGTAGCGG GGCGTAGCGG

11 Suffix trie T: T: Shortest (non-empty) suffix Ech pth from root to lef represents suffix; ech suffix is represented y some pth from root to lef Would this still e the cse if we hdn t dded? Longest suffix

12 Suffix trie T: Ech pth from root to lef represents suffix; ech suffix is represented y some pth from root to lef Would this still e the cse if we hdn t dded? No

13 Suffix trie We cn think of nodes s hving lels, where the lel spells out chrcters on the pth from the root to the node

14 Suffix trie How do we check whether string S is sustring of T? Note: Ech of T s sustrings is spelled out long pth from the root. I.e., every sustring is prefix of some suffix of T. Strt t the root nd follow the edges leled with the chrcters of S S = Yes, it s sustring If we fll off the trie -- i.e. there is no outgoing edge for next chrcter of S, then S is not sustring of T If we exhust S without flling off, S is sustring of T

15 Suffix trie How do we check whether string S is sustring of T? Note: Ech of T s sustrings is spelled out long pth from the root. I.e., every sustring is prefix of some suffix of T. Strt t the root nd follow the edges leled with the chrcters of S S = Yes, it s sustring If we fll off the trie -- i.e. there is no outgoing edge for next chrcter of S, then S is not sustring of T If we exhust S without flling off, S is sustring of T

16 Suffix trie How do we check whether string S is sustring of T? Note: Ech of T s sustrings is spelled out long pth from the root. I.e., every sustring is prefix of some suffix of T. Strt t the root nd follow the edges leled with the chrcters of S If we fll off the trie -- i.e. there is no outgoing edge for next chrcter of S, then S is not sustring of T If we exhust S without flling off, S is sustring of T S = Yes, it s sustring

17 Suffix trie How do we check whether string S is sustring of T? Note: Ech of T s sustrings is spelled out long pth from the root. I.e., every sustring is prefix of some suffix of T. Strt t the root nd follow the edges leled with the chrcters of S If we fll off the trie -- i.e. there is no outgoing edge for next chrcter of S, then S is not sustring of T x S = No, not sustring If we exhust S without flling off, S is sustring of T

18 Suffix trie How do we check whether string S is suffix of T? Sme procedure s for sustring, ut dditionlly check whether the finl node in the wlk hs n outgoing edge leled S = Not suffix

19 Suffix trie How do we check whether string S is suffix of T? Sme procedure s for sustring, ut dditionlly check whether the finl node in the wlk hs n outgoing edge leled S = Not suffix

20 Suffix trie How do we check whether string S is suffix of T? Sme procedure s for sustring, ut dditionlly check whether the finl node in the wlk hs n outgoing edge leled S = Is suffix

21 Suffix trie How do we count the numer of times string S occurs s sustring of T? Follow pth corresponding to S. Either we fll off, in which cse nswer is 0, or we end up t node n nd the nswer = # of lef nodes in the sutree rooted t n. n S = 2 occurrences Leves cn e counted with depth-first trversl.

22 Suffix trie How do we count the numer of times string S occurs s sustring of T? Follow pth corresponding to S. Either we fll off, in which cse nswer is 0, or we end up t node n nd the nswer = # of lef nodes in the sutree rooted t n. n S = 2 occurrences Leves cn e counted with depth-first trversl.

23 Suffix trie How do we find the longest repeted sustring of T? Find the deepest node with more thn one child

24 Suffix trie How do we find the longest repeted sustring of T? Find the deepest node with more thn one child

25 Suffix trie How mny nodes does the suffix trie hve? Is there clss of string where the numer of suffix trie nodes grows linerly with m? T = Yes: e.g. string of m s in row ( m ) 1 Root m nodes with incoming edge m + 1 nodes with incoming edge 2m + 2 nodes

26 Suffix trie How mny nodes does the suffix trie hve? Is there clss of string where the numer of suffix trie nodes grows linerly with m? T = Yes: e.g. string of m s in row ( m ) 1 Root m nodes with incoming edge m + 1 nodes with incoming edge 2m + 2 nodes

27 Suffix trie Is there clss of string where the numer of suffix trie nodes grows with m 2? Yes: n n 1 root n nodes long chin, right n nodes long chin, middle n chins of n nodes hnging off ech chin node 2n + 1 leves (not shown) Figure & exmple y Crl Kingsford n 2 + 4n + 2 nodes, where m = 2n

28 Suffix trie Is there clss of string where the numer of suffix trie nodes grows with m 2? Yes: n n 1 root n nodes long chin, right n nodes long chin, middle n chins of n nodes hnging off ech chin node 2n + 1 leves (not shown) Figure & exmple y Crl Kingsford n 2 + 4n + 2 nodes, where m = 2n

29 Suffix trie: upper ound on size Could worst-cse # nodes e worse thn O(m 2 )? Root Suffix trie Mx # nodes from top to ottom = length of longest suffix + 1 = m + 1 Deepest lef Mx # nodes from left to right = mx # distinct sustrings of ny length m O(m 2 ) is worst cse

30 Suffix trie: ctul growth k Built suffix tries for the first 500 prefixes of the lmd phge virus genome Blck curve shows how # nodes increses with prefix length # suffix trie nodes m^2 ctul m Length prefix over which suffix trie ws uilt

31 Suffix tries -> Suffix trees

32 Suffix Tree Definitions A Σ + -tree is rooted tree, T, where ech edge is leled with non-empty strings, where no node hs two outgoing edges leled with strings hving the sme first chrcter. T is compct if ll internl nodes hve 2 children. for node v in T, depth(v) or node-depth(v) is the distnce from v to the root. node-depth(r) = 0 string(v) = conctention of ll chrcters on the pth r v string-depth(v) = string(v) (note: string-depth(v) node-depth(v)) for string x, if node v with string(v) = x, we sy node(x) = v T displys string x if node v nd string y such tht xy = string(v) words(t) = { x T displys x} A suffix tree of string s is compct Σ + -tree such tht words(t) = {s s is sustring of s} Defs. from: *

33 Suffix trie: mking it smller T = Ide 1: Colesce non-rnching pths into single edge with string lel Reduces # nodes, edges, gurntees internl nodes hve >1 child

34 Suffix tree T = L leves, I internl nodes, E edges E = L + I -1 E 2I (ech internl node rnches) L + I -1 2I I L - 1 With respect to m: ut How mny leves? m L m (t most m suffixes) How mny non-lef nodes? m - 1 2m -1 nodes totl, or O(m) nodes I m -1 E = L + I - 1 2m - 2 E + L + I 4m - 3 O(m) No: totl length of edge Is the totl totl size O(m) size now? O(m) now? lels is qudrtic in m

35 Suffix tree T = L leves, I internl nodes, E edges E = L + I -1 E 2I (ech internl node rnches) L + I -1 2I I L - 1 With respect to m: ut How mny leves? m L m (t most m suffixes) How mny non-lef nodes? m - 1 2m -1 nodes totl, or O(m) nodes I m -1 E = L + I - 1 2m - 2 E + L + I 4m - 3 O(m) Is the totl size O(m) now? Is the totl size O(m) now? No: totl length of edge lels is qudrtic in m NO: The totl length of edge lels is qudrtic in m.

36 Suffix tree T = Ide 2: Store T itself in ddition to the tree. Convert tree s edge lels to (offset, length) pirs with respect to T. T = (0, 1) (6, 1) (1, 2) (1, 2) (6, 1) (6, 1) (3, 4) (6, 1) (3, 4) (3, 4) Spce required for suffix tree is now O(m)

37 Suffix tree: leves hold offsets where suffixes egin T = T = (0, 1) (6, 1) (1, 2) (0, 1) (6, 1) (1, 2) 6 (3, 4) (1, 2) (6, 1) (6, 1) (3, 4) (6, 1) (3, 4) (3, 4) 0 (1, 2) (6, 1) 3 (6, 1) (3, 4) (6, 1) (3, 4) 1

38 Suffix tree: lels T = (0, 1) (6, 1) (1, 2) 6 Agin, ech node s lel equls the conctented edge lels from the root to the node. These ren t stored explicitly. (3, 4) 0 (1, 2) (6, 1) 3 (6, 1) (3, 4) 5 4 (6, 1) (3, 4) 1 Lel = 2 Lel =

39 Suffix tree: lels T = (0, 1) (6, 1) (1, 2) 6 Becuse edges cn hve string lels, we must distinguish two notions of depth Node depth: how mny edges we must follow from the root to rech the node (3, 4) 0 (1, 2) (6, 1) 3 (6, 1) (3, 4) (6, 1) (3, 4) 1 Lel depth: totl length of edge lels for edges on pth from root to node

40 Suffix tree: spce cvet T = (0, 1) (6, 1) (1, 2) 6 Minor point: We sy the spce tken y the edge lels is O(m), ecuse we keep 2 integers per edge nd there re O(m) edges (3, 4) 0 (1, 2) (6, 1) 3 (6, 1) (3, 4) (6, 1) (3, 4) 1 To store one such integer, we need enough its to distinguish m positions in T, i.e. ceil(log 2 m) its. We usully ignore this fctor, since 64 its is plenty for ll prcticl purposes. Similr rgument for the pointers / references used to distinguish tree nodes.

41 Suffix tree: uilding Nive method 1: uild suffix trie, then colesce non-rnching pths nd relel edges Nive method 2: uild single-edge tree representing only the longest suffix, then ugment to include the 2 nd -longest, then ugment to include 3 rd -longest, etc Both re O(m 2 ) time, ut first uses O(m 2 ) spce while second uses O(m) (3, 4) 0 (1, 2) (6, 1) 3 (6, 1) (0, 1) (1, 2) (6, 1) (3, 4) (6, 1) (3, 4) 1 6 Nive method 2 is descried in Gusfield 5.4 Python implementtion t:

42 WOTD (Write-Only Top-Down) Construction Giegerich, Roert, nd Stefn Kurtz. "A comprison of impertive nd purely functionl suffix tree constructions." Science of Computer Progrmming 25.2 (1995): Build suffix tree for string s Recursive construction: For every rnching node node(u), sutree of node(u) is determined y ll suffixes of s where u is prefix. Recursively construct sutree for ll suffixes where u is prefix. Definition: remining suffixes of u R(node(u)) = { v uv is suffix of s } *

43 WOTD (Write-Only Top-Down) Construction Build suffix tree for string s Recursive construction: For every rnching node node(u), sutree of node(u) is determined y ll suffixes of s where u is prefix. Recursively construct sutree for ll suffixes where u is prefix. Definition: remining suffixes of u R(node(u)) = { v uv is suffix of s } Definition: c-group of node(u) group(node(u), c) = { w Σ * cw R(node(u))} *

44 WOTD (Write-Only Top-Down) Construction def WOTD(T : tree, node(u): node): for ech c Σ {}: G = group(node(u), c) non-rnching suffix ucv = lcp(g) if G == 1: dd lef node(ucv) s child of node(u) else: dd inner node(ucv) s child of node(u) WOTD(T, node(ucv)) rnching suffix Strt the lgorithm y clling WOTD(T, node(ε)) root node *

45 s = tttctctt WOTD Exmple suffixes re red top-to-ottom exmple from: *

46 s = tttctctt WOTD Exmple exmple from: *

47 s = tttctctt WOTD Exmple exmple from: *

48 s = tttctctt WOTD Exmple exmple from: *

49 s = tttctctt WOTD Exmple exmple from:

50 WOTD Properties Worst cse time still O( T 2 ) Expected cse time O( T log T ) Write-only property & recursive construction lends itself well to prllelism Good cching properties (loclity of reference for sustrings elonging to sutree) Top-down construction order llows lzy construction s discussed in: Giegerich, Roert, Stefn Kurtz, nd Jens Stoye. "Efficient implementtion of lzy suffix trees." Softwre: Prctice nd Experience (2003):

51 Suffix tree: uilding Other methods for construction: Method of choice: Ukkonen s lgorithm Ukkonen, Esko. "On-line construction of suffix trees." Algorithmic 14.3 (1995): O(m) time nd spce Hs online property: if T rrives one chrcter t time, lgorithm efficiently updtes suffix tree upon ech rrivl We won t cover it here; see Gusfield Ch. 6 for detils Or just Google Ukkonen s lgorithm

52 Suffix tree: ctul growth # suffix trie nodes Built suffix trees for the first 500 prefixes of the lmd phge virus genome Blck curve shows # nodes incresing with prefix length Compre with suffix trie: m^2 ctul m # suffix tree nodes 123 K nodes m ctul m Length prefix over which suffix tree ws uilt Length prefix over which suffix trie ws uilt

53 Suffix tree How do we check whether string S is sustring of T? Essentilly sme procedure s for suffix trie, except we hve to del with colesced edges S = Yes, it s sustring 4 0 3

54 Suffix tree How do we check whether string S is sustring of T? Essentilly sme procedure s for suffix trie, except we hve to del with colesced edges S = Yes, it s sustring 4 0 3

55 Suffix tree How do we check whether string S is suffix of T? Essentilly sme procedure s for suffix trie, except we hve to del with colesced edges 2 S = 5 1 Yes, it s suffix

56 Suffix tree How do we count the numer of times string S occurs s sustring of T? Sme procedure s for suffix trie 6 2 S = 5 1 Occurs twice 4 0 3

57 Suffix tree: pplictions With suffix tree of T, we cn find ll mtches of P to T. Let k = # mtches. E.g., P =, T = O(n) Step 1: wlk down pth If we fll off there re no mtches 6 O(k) Step 2: visit ll lef nodes elow Report ech lef offset s mtch offset O(n + k) time

58 Suffix tree ppliction: find long common sustrings Dots re mximl unique mtches (MUMs), kind of long sustring shred y two sequences Red = mtch ws etween like strnds, green = different strnds Axes show different strins of Helicocter pylori, cterium found in the stomch nd ssocited with gstric ulcers

59 Suffix tree ppliction: find longest common sustring To find the longest common sustring (LCS) of X nd Y, mke new string X#Y where # nd re oth terminl symols. Build suffix tree for X#Y. X = xx Y = x X#Y = xx#x x #x X Y x X Y x X Y 5 #x X Y x 12 Consider leves: offsets in [0, 4] re suffixes of X, offsets in [6, 11] re suffixes of Y 1 4 X Y #x 7 11 X #x x#x Y x 10 X Y 2 #x 8 Trverse the tree nd nnotte ech node ccording to whether leves elow it include suffixes of X, Y or oth The deepest node nnotted with oth X nd Y hs LCS s its lel. O( X + Y ) time nd spce.

60 Suffix tree ppliction: generlized suffix trees This is one exmple of mny pplictions where it is useful to uild suffix tree over mny strings t once Such tree is clled generlized suffix tree. These re introduced in Gusfield 6.4. X Y x #x x #x X Y x X Y 5 X Y x 12 4 X Y 11 X 9 Y X Y #x #x x#x x #x

61 Longest Common Extension Longest common extension: We re given strings S nd T. In the future, mny pirs (i,j) will e provided s queries, nd we wnt to quickly find: the longest sustring of S strting t i tht mtches sustring of T strting t j. S LCE(i,j) T LCE(i,j) i j Build generlized suffix tree for S nd T. Preprocess tree so tht lowest common ncestors (LCA) cn e found in constnt time. This cn e done using rnge-minimum queries (RMQ) Crete n rry mpping suffix numers to lef nodes. O( S + T ) O( S + T ) O( S + T ) LCA(i,j) Given query (i,j): Find the lef nodes for i nd j Return string of LCA for i nd j O(1) O(1) j i j i

62 Suffix trees in the rel world: MUMmer FASTA file contining reference ( text ) FASTA file contining ALU string Indexing phse: ~2 minutes Mtching phse: very fst Columns: 1. Mtch offset in T 2. Mtch offset in P 3. Length of exct mtch...

63 Suffix trees in the rel world: MUMmer MUMmer v3.32 time nd memory scling when indexing incresingly lrger frctions of humn chromosome 1 Pek memory usge (megytes) Time (seconds) Frction of humn chromosome 1 indexed Frction of humn chromosome 1 indexed For whole chromosome 1, took 2m:14s nd used 3.94 GB memory

64 Suffix trees in the rel world: MUMmer Attempt to uild index for whole humn genome reference: mummer:!suffix!tree!construction!filed:!textlen= ! lrger!thn!mximl!textlen= We cn predict it would hve tken out 47 GB of memory

65 Suffix trees in the rel world: the constnt fctor While O(m) is desirle, the constnt in front of the m limits wider use of suffix trees in prctice Constnt fctor vries depending on implementtion: Estimte of MUMmer s constnt fctor = 3.94 GB / 250 million nt ytes per node Literture reports implementtions chieving s little s 8.5 ytes per node, ut no implementtion used in prctice tht I know of is etter thn 12.5 ytes per node Kurtz, Stefn. "Reducing the spce requirement of suffix trees." Softwre Prctice nd Experience (1999):

66 Suffix tree: summry Orgnizes ll suffixes into n incredily useful, flexile dt structure, in O(m) time nd spce A nive method (e.g. suffix trie) could esily e qudrtic or worse Used in prctice for whole genome lignment, repet identifiction, etc 3 (5, 21) (12, 14) 10 (4, 1) (6, 2) 5 (8, 18) (23, 3) (8, 18) (13, 1) (14, 12) (19, 7) (16, 1) (3, 1) (7, 1) (1, 1) (0, 1) (17, 9) (25, 1) 22 (3, 1) (12, 14) (2, 24) 2 11 (4, 22) (6, 2) Actul memory footprint (ytes per node) is quite high, limiting usefulness 4 (8, 18) GTTATAGCTGATCGCGGCGTAGCGG GTTATAGCTGATCGCGGCGTAGCGG TTATAGCTGATCGCGGCGTAGCGG TATAGCTGATCGCGGCGTAGCGG ATAGCTGATCGCGGCGTAGCGG TAGCTGATCGCGGCGTAGCGG AGCTGATCGCGGCGTAGCGG 19 1 (23, 3) (9, 17) (25, 1) 25 (10, 16) (7, 1) (1, 1) (16, 1) (8, 18) (15, 1) 16 (19, 7) (16, 1) 13 (17, 9) GCTGATCGCGGCGTAGCGG CTGATCGCGGCGTAGCGG (20, 6) (2, 24) 18 (25, 1) 21 TGATCGCGGCGTAGCGG GATCGCGGCGTAGCGG ATCGCGGCGTAGCGG TCGCGGCGTAGCGG CGCGGCGTAGCGG GCGGCGTAGCGG 0 15 (25, 1) (17, 9) 24 (25, 1) 23 CGGCGTAGCGG GGCGTAGCGG GCGTAGCGG CGTAGCGG GTAGCGG TAGCGG AGCGG GCGG CGG GG G m chrs m(m+1)/2 chrs