Suffix trees. December Computational Genomics

Transcription

1 Computtionl Genomics Prof Irit Gt-Viks, Prof. Ron Shmir, Prof. Roded Shrn School of Computer Science, Tel Aviv University גנומיקה חישובית פרופ' עירית גת-ויקס, פרופ' רון שמיר, פרופ' רודד שרן ביה"ס למדעי המחשב,אוניברסיטת תל אביב Suffix trees Decemer 2018

2 Suffix Trees Description follows Dn Gusfield s ook Algorithms on Strings, Trees nd Sequences Slides sources: Pvel Shviko, (University of Trento), Him Kpln (Tel Aviv University), Ben Lngmed (JHU)

3 Outline Introduction Suffix Trees (ST) Building STs in liner time: Ukkonen s lgorithm Applictions of ST 3

4 Introduction 4

5 Exct String/Pttern Mtching S = m, n different ptterns p 1 p n Text S eginning end Pttern occurrences cn overlp 5

6 String/Pttern Mtching - I Given text S, nswer queries of the form: is the pttern p i sustring of S? Knuth-Morris-Prtt 1977 (KMP) string mtching lg: O( S + p i ) time per query. O(n S + S i p i ) time for n queries. Suffix tree solution: O( S + S i p i ) time for n queries. 6

7 String/Pttern Mtching - II KMP preprocesses the ptterns p i ; The suffix tree lgorithm: preprocess S in O( S ): uilds dt structure clled suffix tree for S when pttern p is input, the lgorithm serches it in O( p ) time using the suffix tree 7

8 Donld Knuth 8

9 Prefixes & Suffixes Nottion: S[i,j] =S(i), S(i+1),, S(j) Prefix of S: sustring of S eginning t the first position of S S[1,i] Suffix of S: sustring tht ends t lst position S[i,n] S=AACTAG Prefixes: AACTAG,AACTA,AACT,AAC,AA,A Suffixes: AACTAG,ACTAG,CTAG,TAG,AG,G Note: P is sustring of S iff P is prefix of some suffix of S. 9

10 Suffix Trees 10

11 Trie A tree representing set of strings. { } eef d fe fg c f e e c d f e c g 11

12 Trie (Cont) Assume no string is prefix of nother Ech edge is leled y letter, no two edges outgoing from the sme node e re leled the sme. e d f c Ech string corresponds to lef. f e g 12

13 Compressed Trie Compress unry nodes, lel edges y strings c c e e d f eef d f f c e g e g 13

14 Def: Suffix Tree for S S = m 1. A rooted tree T with m leves numered 1,,m. 2. Ech internl node of T, except perhps the root, hs 2 children. 3. Ech edge of T is leled with nonempty sustring of S. 4. All edges out of node must hve lels strting with different chrcters. 5. For ny lef i, the conctention of the edge-lels on the pth from the root to lef i exctly spells out S[i,m]. S=xxc 14

15 Existence of suffix tree S If one suffix S j of S mtches prefix of nother suffix S i of S, then the pth for S j would not end t lef. S = xx S 1 = xx nd S 4 = x How to void this prolem? Mke sure tht the lst chrcter of S ppers nowhere else in S. Add new chrcter not in the lphet to the end of S

16 Exmple: Suffix Tree for S=xx

17 Exmple: Suffix Tree for S=xx Query: P = xc P is sustring of S iff P is prefix of some suffix of S

18 Trivil lgorithm to uild Suffix tree S= Put the lrgest suffix in Put the suffix in 18

19 Put the suffix in 19

20 Put the suffix in 20

21 Put the suffix in 21

22 We will lso lel ech lef with the strting point of the corresponding suffix

23 Anlysis Tkes O(m 2 ) time to uild. Cn e done in O(m) time - we will sketch the proof. See the CG clss notes or Gusfield s ook for the full detils of the proof. 23

24 Building STs in liner time: Ukkonen s lgorithm 24

25 History Weiner s lgorithm [FOCS, 1973] Clled y Knuth The lgorithm of 1973 First liner time lgorithm, ut much spce McCreight s lgorithm [JACM, 1976] Liner time nd qudrtic spce More redle Ukkonen s lgorithm [Algorithmic, 1995] Liner time nd less spce This is wht we will focus on. 25

26 Esko Ukkonen 26

27 Implicit Suffix Trees Ukkonen s lg constructs sequence of implicit STs, the lst of which is converted to true ST of the given string. An implicit suffix tree for string S is tree otined from the suffix tree for S y removing from ll edge lels removing ny edge tht now hs no lel removing ny node with only one child 27

28 Exmple: Construction of the Implicit ST The tree for xx 3 x 6 5 x {xx, x, x, x,, } x x

29 Construction of the Implicit ST: Remove Remove 3 x 6 5 x {xx, x, x, x,, } x x

30 Construction of the Implicit ST: After the Removl of 3 x 6 5 x x 2 4 x {xx, x, x, x, } 1 30

31 Construction of the Implicit ST: Remove unleled edges Remove unleled edges 3 x 6 5 x x 2 4 {xx, x, x, x, } x 1 31

32 Construction of the Implicit ST: After the Removl of Unleled Edges x x x {xx, x, x, x, } x

33 Construction of the Implicit ST: Remove degree 1 nodes Remove internl nodes with only one child x x x {xx, x, x, x, } x

34 Construction of the Implicit ST: Finl implicit tree x x x x {xx, x, x, x, } 3 2 Ech suffix is in the tree, ut my not end t lef. 1 34

35 Implicit Suffix Trees (2) An implicit suffix tree for prefix S[1,i] of S is similrly defined sed on the suffix tree for S[1,i]. I i = the implicit suffix tree for S[1,i]. 35

36 Ukkonen s Algorithm (UA) I i is the implicit suffix tree of the string S[1, i] Construct I 1 /* Construct I i+1 from I i */ for i = 1 to m-1 do /* genertion i+1 */ for j = 1 to i+1 do /* extension j */ Find the end of the pth p from the root whose lel is S[j, i] in I i nd extend p with S(i+1) y suffix extension rules; Convert I m into suffix tree S 36

37 Exmple S = xx (initiliztion step) x (i = 1), i+1 = 2, S(i+1)= extend x to x (j = 1, S[1,1] = x) (j = 2, S[2,1] = ) (i = 2), i+1 = 3, S(i+1)= extend x to x (j = 1, S[1,2] = x) extend to (j = 2, S[2,2] = ) (j = 3, S[3,2] = ) 37

38 S(1) S(i) S(i+1) All suffixes of S[1,i] re lredy in the tree Wnt to extend them to suffixes of S[1,i+1] 38

39 Extension Rules Gol: extend ech S[j,i] into S[j,i+1] Rule 1: S[j,i] ends t lef Add chrcter S(i+1) to the end of the lel on tht lef edge Rule 2: S[j,i] doesn t end t lef, nd the following chrcter is not S(i+1) Split new lef edge for chrcter S(i+1) My need to crete n internl node if S[j,i] ends in the middle of n edge Rule 3: S[j,i+1] is lredy in the tree No updte 39

40 Exmple: Extension Rules Constructing the implicit tree for xx from tree for xx x x 4 5 x 2 x 3 Rule 1: 2: 3: t dd lredy lef lef in node edge tree (nd n interior node) x x 1 40

41 UA for xxc (1) S[1,3]=x E S(j,i) S(i+1) 1 x 2 x 3 41

42 UA for xxc (2) 42

43 UA for xxc (3) 43

44 UA for xxc (4) c 44

45 Oservtions Once S[j,i] is locted in the tree, pplying the extension rule tkes only constnt time Nive implementtion: find the end of suffix S[j,i] in O(i-j) time y wlking from the root of the current tree => I m is creted in O(m 3 ) time. Mking Ukkonen s lgorithm run in O(m) time is chieved y set of shortcuts: Suffix links Skip nd count trick Edge-lel compression A stopper Once lef, lwys lef 45

46 Ukkonen s Algorithm (UA) I i is the implicit suffix tree of the string S[1, i] Construct I 1 /* Construct I i+1 from I i */ for i = 1 to m-1 do /* genertion i+1 */ for j = 1 to i+1 do /* extension j */ Find the end of the pth p from the root whose lel is S[j, i] in I i nd extend p with S(i+1) y suffix extension rules; Convert I m into suffix tree S 46

47 Looking for shortcut After we extend string x, we need to extend. Cn we jump right to its position in the current tree, rther thn going down ll the wy from the root? x

48 Suffix Links Consider the two strings nd x (e.g., x in the exmple elow). Suppose some internl node v of the tree is leled with x (x=chr, = string, possily ) nd nother node s(v) in the tree is leled with The edge (v,s(v)) is clled the suffix link of v Do ll internl nodes hve suffix links? (the root is not considered n internl node) pth lel of v: conctention of the strings leling edges from root to v 48

49 Exmple: Suffix links cxcd ukkonens-suffix-tree-lgorithm-in-plin-english 50

50 Suffix Link Lemm If new internl node v with pth-lel x is dded to the current tree in extension j of some genertion i+1, then either the pth leled lredy ends t n internl node of the tree, or the internl node leled will e creted in extension j+1 in the sme genertion i+1, or string is empty nd s(v) is the root 51

51 Suffix Link Lemm If new internl node v with pth-lel x is dded to the current tree in extension j of some genertion i+1, then either the pth leled lredy ends t n internl node of the tree, or the internl node leled will e creted in extension j+1 in the sme genertion Pf: A new internl node is creted only y extension rule 2 In extension j the pth leled x.. continued with some y S(i+1) => In extension j+1, pth p leled.. p continues with y only ext. rule 2 will crete node s(v) t the end of the pth. p continues with two different chrs s(v) lredy exists. r r v v 52

52 Corollries Every internl node of n implicit suffix tree hs suffix link from it y the end of the next extension Proof y the lemm, using induction. In ny implicit suffix tree I i, if internl node v hs pth lel x, then there is node s(v) of I i with pth lel Proof y the lemm, pplied t the end of genertion 53

53 Building I i+1 with suffix links - 1 Gol: in extension j of genertion i+1, find S[j,i] in the tree nd extend to S[j,i+1]; dd suffix link if needed 54

54 Building I i+1 with suffix links - 2 Gol: in extension j of genertion i+1, find S[j,i] in the tree nd extend to S[j,i+1]; dd suffix link if needed S[1,i] must end t lef since it is the longest string in the implicit tree I i Keep pointer to lef of full string; extend to S[1,i+1] (rule 1) S[2,i] =, S[1,i]=x ; let (v,1) e the edge entering lef 1: If v is the root, descend from the root to find Otherwise, v is internl. Go to s(v) nd descend to find rest of 55

55 Building I i+1 with suffix links - 3 In generl: find first node v t or ove S[j-1,i] tht hs s.l. or is root; Let = string etween v nd end of S[j-1,i] If v is internl, go to s(v) nd descend following the pth of If v is the root, descend from the root to find S[j,i] Extend to S[j,i]S(i+1) (if not lredy in the tree) If new internl node w ws creted in extension j-1, y the lemm S[j,i+1] ends in s(w) => crete the suffix link from w to s(w). 56

56 Skip nd Count Trick (1) Prolem: Moving down from s(v), directly implemented, tkes time proportionl to Solution: mke running time proportionl to the numer of nodes in the pth serched Key: surely exists in the current tree; need to serch only the first chr. in ech outgoing node 57

57 Skip nd Count Trick (2) counter=0; On ech step from s(v), find right edge elow, dd no. of chrs on it to counter nd if still < skip to child After 4 skips, the end of S[j, i] is found. Cn show: with skip & count trick, ny genertion of Ukkonen s lgorithm tkes O(m) time 58

58 Interim conclusion Ukkonen s Algorithm cn e implemented in O(m 2 ) time A few more smrt tricks nd we rech O(m) [see scrie or the end of this presenttion] 59

59 Implementtion Issues (1) When the size of the lphet grows: For lrge trees suffix, links llow to move quickly from one prt of the tree to nother. This is slow if the tree isn't entirely in memory. Efficiently implementing ST to reduce spce in prctice cn e tricky. The min design issues re how to represent nd serch the rnches out of the nodes of the tree. A prcticl design must lnce etween constrints of spce nd need for speed 60

60 Representing the rnches out of v An rry of size ( S ) t ech non-lef node v A linked list of chrcters tht pper t the eginning of the edge-lels out of v. If kept in sorted order it reduces the verge time to serch for given chrcter In the worst cse, it dds time S to every node opertion. If the numer of children k of v is lrge, little spce is sved over the rry, more time A lnced tree implements the list t node v Additions nd serches tke O(logk) time nd O(k) spce. Option mkes sense only when k is firly lrge. A hshing scheme. The chllenge is to find scheme lncing spce with speed. For lrge trees nd lphets hshing is very ttrctive t lest for some of the nodes 61

61 Implementtion Issues (3) When m nd S re lrge enough, good design is often mixture. Guidelines: Nodes ner the root tend to hve most children use rrys. If k very dense levels form lookup tle of ll k-tuples with pointers to the roots of the corresponding sutrees. Nodes in the middle of the tree: hshing or lnced trees. 62

62 Applictions of Suffix Trees 63

63 Wht cn we do with it? Exct string mtching: Given Text T, T = n, preprocess it such tht when pttern P, P =m, rrives we cn quickly decide if it occurs in T. We my lso wnt to find ll occurrences of P in T 64

64 Exct string mtching In preprocessing we just uild suffix tree in O(m) time Given pttern P = we trverse the tree ccording to the pttern. 65

65 If we did not get stuck trversing the pttern then the pttern occurs in the text. Ech lef in the sutree elow the node we rech corresponds to n occurrence. By trversing this sutree we get ll k occurrences in O(n+k) time 66

66 Generlized suffix tree Given set of strings S, generlized suffix tree of S is compressed trie of ll suffixes of s S To ssocite ech suffix with unique string in S dd different specil end chr i to ech s i 67

67 Exmple Let s 1 = nd s 2 = A generlized suffix tree for s 1 nd s 2 : { } # # # # 3 # 2 # 1 2 # 4 #

68 So wht cn we do with it? Mtching pttern ginst dtse of strings 69

69 Longest common sustring (of two strings) Every node tht hs oth lef descendnt from string s 1 nd lef descendnt from string s 2 represents mximl common sustring nd vice vers. # 4 # Find such node with lrgest lel depth 1 3 # 2 # 1 2 O( S 1 + S 2 ) to construct the tree nd serch it. 70

70 Lowest common ncestors A lot more cn e gined from the suffix tree if we preprocess it so tht we cn nswer LCA queries on it 71

71 Why? The LCA of two leves represents the longest common prefix (LCP) of these two suffixes Hrel-Trjn (84), Schieer-Vishkin (88): LCA query in constnt time, with liner pre-processing of the tree. 3 # 2 # 1 2 # 4 #

72 Finding mximl plindromes A plindrome: cc, cc Wnt to find ll mximl plindromes in string s s = c 1 i-1 i m m m-i+2 m-i+1 1 The mximl plindrome with center etween i-1 nd i is the LCP of the suffix t position i of s nd the suffix t position m-i+2 of s r 73

73 Mximl plindromes lgorithm Prepre generlized suffix tree for s = c nd s r = c# For every i find the LCA of suffix i of s nd suffix m-i+2 of s r O(m) time to identify ll plindromes 74

74 Let s = c then s r = c# 6 c # m-i+2= i=

75 SUFFIX ARRAYS 76

76 ST Drwcks Spce is O(m) ut the constnt is quite ig For humn genome, spce >45GB. 77

77 Suffix rrys (U. Mnder, G. Myers 91) We lose some of the functionlity ut sve spce. Sort the suffixes of S lexicogrphiclly The suffix rry: list of strting positions of the sorted suffixes 78

78 Suffix Arry for pnmnns Size: For humn genome, ~4 ytes per se x 3 illion ses 12 GB SuffixArry ( pnmnns )=(13,5,3,1,7,9,11,6,4,2,8,10,0,12) Pevzner, Compeu Bioinfo Algs 14 79

79 How do we uild it? Build suffix tree Trverse the tree in DFS, lexicogrphiclly picking edges outgoing from ech node. SA = lef lel order. O(m) time; direct liner time lgs known Pevzner, Compeu Bioinfo Algs 14 80

80 How do we serch for pttern? If P occurs in S then ll its occurrences re consecutive in the suffix rry. Do inry serch on the suffix rry 81

81 Exmple Let S = mississippi Let P = iss For m= S, n= P : O(log m) isections, O(n) comprisons per isection O(nlogm) L M R i ippi issippi ississippi mississippi pi ppi sippi sissippi ssippi ssissippi Cn ctully show: O(n+logm) time 82

82 Suffix Arrys vs. Suffix Trees - Summry 83

83 Udi Mner Gene Myers 84

84 The end? 85

85 The missing pieces in the proof of Ukkonen s Algorithm 86

86 Edge Lel Representtion Prolem Edge lels my require W(m 2 ) spce W(m 2 ) time Exmple: S = cdefghijklmnopqrstuvwxyz Totl length is S j<m+1 j = m(m+1)/2 Solution Lel ech edge with pir of indices indicting the eginning nd the end positions of tht edge s sustring in S Exmple: insted of lel S = cdefghijklmnopqrstuvwxyz hve lel (11,36) 2m-1 edges, 2 numers per edge O(m) spce 87

87 Modified Extension Rules - with the compct edge lels Rule 1: lef edge extension lel ws (p,i) efore extension (p, i) (p, i + 1) Rule 2: new lef edge (phse i+1) crete edge (i+1, i+1) split edge (p, q) (p, w) nd (w + 1, q) Rule 3: S[j,i+1] is lredy in the tree Do nothing 88

88 Edge-lel Compression String S = xx (6,6) (2,2) (3,6) 6 (6,6) 5 3 x x 6 5 x 4 2 (1,2) (3,6) 3 [lso (4,5)] (6,6) (3,6) x 1 89

89 Erly stopping of phse Os: In ny phse, if rule 3 pplies in extension j, it will lso pply in ll extensions k>j in tht phse. end phse i+1 on the first time rule 3 pplies. The extensions fter the first execution of rule 3 re sid to e done implicitly. Ex: in phse i+1=7, explicitly extend (1,7), (2,7), (3,7) y rule 3; do nothing for (4,7),,(7,7) 90

90 Once lef, lwys lef (1) Os: If t some point lef is creted, rule 1 will lwys pply to it lter it will remin lef in ll susequent phses. its lel j is mintined in ll susequent phses. In ny phse, n initil sequence of consecutive extensions (strting with extension 1) in which only rule 1 or 2 pplies. Denote j i : the lst extension in this sequence in phse i. in the next phse the first j i extensions re of leves nd rule 1 pplies. Note : j i j i+1. 91

91 Once lef, lwys lef (2) Let e = glol symol denoting the current end. e is set to i + 1 t the eginning of phse i + 1 When lef is creted, insted of writing [p,i+1] s the edge lel, write [p, e]. In ll lter phses, we implicitly extend the lef y incrementing e once. Perform explicitly extensions j i +1 nd on, until the first rule 3 extension is found, or phse i+1 is done. 92

92 Single phse lgorithm Phse i+1 Increment e to i+1 (implicitly extending ll existing leves) Explicitly compute successive extensions strting t j i +1 nd continuing until reching the first extension j* where rule 3 pplies or no more extensions re needed Set j i+1 to j*-1, to prepre for the next phse Os: Phse i nd i+1 shre t most 1 explicit extension 93

93 Exmple: S=xx - (1) I 1 (1,e) 1 e = 1, j 1 = 1 I 2 1 (1,e) (2,e) e = 2, x S[1,2] : skip S[2,2] : rule 2, crete(2, e) j 2 = 2 I 3 (1,e) (2,e) e = 3, x S[1,3].. S[2,3] : skip S[3,3] : rule 3 j 3 = 2 94

94 Exmple: S=xx - (2) I 4 (1,e) (2,e) e = 4, xx S[1,4].. S[2,4] : skip S[3,4] : rule 3 S[4,4] : uto skip j 4 = 2 I 5 (1,2) (3,e) (5,e) (2,2) (3,e) 2 (5,e) 4 (5,e) 5 e = 5, xx S[1,5].. S[2,5] : skip S[3,5] : rule 2, split (1,e) (1, 2) nd (3,e), crete (5,e) S[4,5] : rule 2, split (2,e) (2,2) nd (3,e), crete (5,e) S[5,5] : rule 2, crete (5,e) j 5 = 5 95

95 Exmple: S=xx - (3) I 6 (1,2) 1 (2,2) (5,e) e = 6, xx (3,e) (5,e) (3,e) (5,e) 5 S[1,6].. S[5,6] : skip S[6,6] : rule 3 j 6 = I 7 (1,2) 1 (2,2) (7,e) (5,5) 7 e = 7, xx S[1,7].. S[5,7] : skip (3,e) (5,e) 1 3 (3,e) 2 (5,e) 4 (6,e) 5 (7,e) 6 S[6,7] : rule 2, split (5,e) (5,5) nd (6,e), crete (6,e) S[7,7] : rule 2, crete (7,e) j 7 = 7 96

96 Complexity of UA In ny phse, ll the implicit extensions tke constnt time => their totl cost is O(m). Totlly, only 2m explicit extensions re executed. The mx numer of down-wlking skips is O(m). Time-complexity of Ukkonen s lgorithm: O(m) Phse i Phse i+1 Phse i : explicit extension 97

97 Finishing up Convert finl implicit suffix tree to true suffix tree: Add using one more phse Now ll suffixes will e leves Replce e on every lef edge y m A trversl of tree in O(m) time 98

98 The end! 99