On String Matching in Chunked Texts

Transcription

1 On String Mtching in Chunked Texts Hnnu Peltol nd Jorm Trhio {hpeltol, Deprtment of Computer Science nd Engineering Helsinki University of Technology P.O. Box 5400, FI HUT, Finlnd On String Mtching in Chunked Texts p. 1

2 Overview Problem Dt consists of chunks Brief history of previous solutions New lgorithms Some experimentl results Conclusions On String Mtching in Chunked Texts p. 2

3 Problem Exct pttern mtching on strings: find ll positions where given pttern cn be found in text Text of length n: T = t 1 t 2 t n Pttern of length m: P = p 1 p 2 p m Texts re specil: chunked On String Mtching in Chunked Texts p. 3

4 Texts re chunked Now texts consist of consecutive fixed-length chunks:... Ech byte position (,,, ) in every chunk hs chrcter distribution of its own A chunk cn lso be interpreted s chrcter of lrger lphbet q is the probbility tht two rndomly chosen bytes from text nd pttern mtch Thierry Lecroq: Experiments on string mtching in memory structures. SPE, 28(5): , On String Mtching in Chunked Texts p. 4

5 Some pttern mtching lgorithms Boyer Moore (BM) Horspool (Hor) shift is simplified: bsed on chrcter tht is ligned with the end of the pttern Sundy s Quick Serch (QS) shift is bsed on chrcter tht is fter the end of the pttern Zhu Tkok, Bez-Ytes, etc. shift is bsed on two or more chrcters On String Mtching in Chunked Texts p. 5

6 Implementtion Alredy Boyer & Moore noticed tht rndom text chrcter rrely mtches with the corresponding chrcter in pttern So usully lgorithms check one chrcter nd move forwrd skip loop TBM = Tuned Boyer Moore uses ufst skip loop (originl implementtion by Hume & Sundy) Gurd: n dditionl test before comprison of the entire pttern On String Mtching in Chunked Texts p. 6

7 The speed of QS nd Hor should be lmost equl If chrcters re sttisticlly independent of ech other Expected shift length of Hor is 1 (1 q)m q Expected shift length of QS is 1 (1 q)m+1 q When comprison is mde forwrd; n exmple: On String Mtching in Chunked Texts p. 7

8 Exmple of the behvior of QS b b b * * * * * * m(m+1) 2 comprisons per 2m chrcters in text QS works here in O(nm) On String Mtching in Chunked Texts p. 8

9 Peculir behvior When comprison is mde forwrd P = m,t = ( m 1 b) n/m Hor works in O(n/m) nd QS in O(nm) P = m 4 c 3,T = (b m 2 cb) n/m QS works in O(n/m) but Hor in O(nm) On String Mtching in Chunked Texts p. 9

10 Lecroq s dt / short integers Shorts symbols mx.freq. zeros 1/q Overll = bytes Regulrities: 5 + i 32 \x00 ; 21 + i 32 \x40 ; 13 + i i 64 \x10 ; 29 + i i 64 \x90 On String Mtching in Chunked Texts p. 10

11 Lecroq s dt / doubles Doubles symbols mx.freq. zeros 1/q Overll On String Mtching in Chunked Texts p. 11

12 Lecroq s dt nd experiments Dt ws dumps from computer memory On shorts, TBM ws fstest on short ptterns nd QS on long ptterns On doubles, BM ws fstest Lecroq did not consider the effects cused by chunks. He ws more interested in the effect of the lphbet size When potentil mtch ws found, it ws checked tht it ends on chunk border On String Mtching in Chunked Texts p. 12

13 Wht would work better Positions with no or little vrition re chllenging We could use two bytes so tht t lest the other byte would hit position with rich vrying content. We could lso peek forwrd greedily to get longer shifts We could shift in synchronized fshion (with chunk borders) nd check the content of the most rndom byte position in the lst chunk of the pttern On String Mtching in Chunked Texts p. 13

14 Fork(h, P = p 1 p 2 p m, T = t 1 t 2 t n ) /* Preprocessing */ 1: for ll c Σ do tmpd[c] m 2: for i 1 to m 1 do tmpd[p i ] m i 3: shift tmpd[p m ]; tmpd[p m ] 0 4: for ll c1 Σ do 5: if tmpd[c1] < h then 6: for ll c2 Σ do d[c1,c2] tmpd[c1] 7: else 8: for ll c2 Σ do d[c1,c2] m + h 9: for i 1 to h do d[c1,p i ] m + h i 10: for i 1 to m h do 11: if tmpd[p i ] h then d[p i,p i+h ] m i /* Serching is on next slide */ On String Mtching in Chunked Texts p. 14

15 Fork(h, P = p 1 p 2 p m, T = t 1 t 2 t n ) /* Serching */ 12: t n+1 t n+2 m P + P /* Stopper */ 13: j m 14: while j n do 15: repet k d[t j,t j+h ]; j j + k until k = 0 16: if j n then 17: if t j m+1 t j 1 = p 1 p m 1 nd j is multiple of w then Report mtch 18: j j + shift On String Mtching in Chunked Texts p. 15

16 Sync(h, P = p 1 p 2 p m, T = t 1 t 2 t n ) /* Preprocessing */ 1: for ll c Σ do d1[c] m 2: for i w h step w to m h 1 do d1[p i ] (m h) i /* Serching */ 3: s p m h 4: t n+1..t n+m s m /* Stopper for inner while */ 5: j m 6: while j n do 7: while t j h s do j j + d1[t j h ] 8: if t j m+1..t j = P then Report mtch 9: j j + d1[s] On String Mtching in Chunked Texts p. 16

17 Results for shorts Time (sec) Running times per pttern in seconds for shorts Pttern length (bytes) Hor TBM Fork 3 BM QS Sync 3 On String Mtching in Chunked Texts p. 17

18 Results for shorts (long ptterns) Time (sec) Running times per pttern in seconds for shorts Pttern length (bytes) Hor TBM Fork 3 BM QS Sync 3 On String Mtching in Chunked Texts p. 18

19 Results for doubles 1.8 Running times per pttern in seconds for doubles Time (sec) TBM Hor QS BM Fork 4 Sync Pttern length (bytes) On String Mtching in Chunked Texts p. 19

20 Concluding remrks Test runs were repeted on two rchitectures: on Sprc nd on AMD Athlon Thunderbird Librry routine memcmp slower thn explicit comprison On Sync prmeter h corresponding smllest q works usully best On Fork the smll vlues seem to be good for prmeter h On String Mtching in Chunked Texts p. 20

21 Conclusions Choice of test position is sometimes crucil Skip loop improves speed in prctice, if test chrcter is not too common Insted of mximizing the verge shift length, it is often fster to keep the skip loop running String mtching results re dt dependent e.g. chunked dt cn hve very different effect on different lgorithms On String Mtching in Chunked Texts p. 21