Shortest Unique Substring Query Revisited

Transcription

1 Shortest Unique Substring Query Revisited Atalay Mert İleri Bilkent University, Turkey M. Oğuzhan Külekci Istanbul Medipol University, Turkey Bojian Xu Eastern Washington University, USA June 18, 2014

2 Outline 1 Preliminary Information 2 Left Bounded Shortest Unique Substrings 3 Finding Shortest Unique Substring For All Positions 4 Experimental Results 5 Conclusion 6 Acknowledgments 7 Q&A

3 Shortest Unique Substring (SUS) A substring S[i j] is called Shortest Unique Substring covering a position k, if i k j and there is no other unique substring S [i j ] exists such that i k j and j i < j i

4 Shortest Unique Substring (SUS) A substring S[i j] is called Shortest Unique Substring covering a position k, if i k j and there is no other unique substring S [i j ] exists such that i k j and j i < j i Problem 1 (find details in paper) Given a string location k, find the leftmost SUS covering location k. Problem 2 (find details in paper) Given a string location k, find all the SUSes covering location k. Problem 3 (discussed in this talk) Find the leftmost SUS covering every string location 1, 2,..., n. Problem 4 (find details in paper) Find all the SUSes covering every string location 1, 2,..., n.

5 Why SUS queries? Distinguishing results in a text search Finding DNA signatures of organisms Extracting the event context in historical events

6 Preliminary Information Preliminaries: Suffix Array and Rank Array 1 mississippi 2 ississippi 3 ssissippi 4 sissippi 5 issippi 6 ssippi 7 sippi 8 ippi 9 ppi 10 pi 11 i

7 Preliminary Information Preliminaries: Suffix Array and Rank Array 1 mississippi 2 ississippi 3 ssissippi 4 sissippi 5 issippi 6 ssippi 7 sippi 8 ippi 9 ppi 10 pi 11 i 11 i 8 ippi 5 issippi 2 ississippi 1 mississippi 10 pi 9 ppi 7 sippi 4 sissippi 6 ssippi 3 ssissippi

8 Preliminary Information Preliminaries: Suffix Array and Rank Array 1 mississippi 2 ississippi 3 ssissippi 4 sissippi 5 issippi 6 ssippi 7 sippi 8 ippi 9 ppi 10 pi 11 i 11 i 8 ippi 5 issippi 2 ississippi 1 mississippi 10 pi 9 ppi 7 sippi 4 sissippi 6 ssippi 3 ssissippi SA RANK

9 Preliminary Information Preliminaries: Longest Common Prefix Array 11 i 8 ippi 5 issippi 2 ississippi 1 mississippi 10 pi 9 ppi 7 sippi 4 sissippi 6 ssippi 3 ssissippi LCP

10 Left Bounded Shortest Unique Substrings Left Bounded Shortest Unique Substring Definition For a particular string location k {1, 2,..., n}, the left-bounded shortest unique substring (LSUS) starting at location k, denoted as LSUS k, is a unique substring S[k... j], such that either k = j or any proper prefix of S[k... j] is not unique. Examples: mississippi LSUS 1 = S[1, 1] mississippi LSUS 2 = S[2, 6]. mississippi LSUS 10 = S[10, 11] mississippi LSUS 11 does not exist

11 Left Bounded Shortest Unique Substrings Fast computation of LSUS es from LCP array For i = 1, 2,..., n: { S[i... i + Li ], if i + L LSUS i = i n not existing, otherwise where L i = max{lcp[rank[i]], LCP[RANK[i] + 1]}.

12 Left Bounded Shortest Unique Substrings Fast computation of LSUS es from LCP array mississippi RANK LCP L i 0 LSUS i m

13 Left Bounded Shortest Unique Substrings Fast computation of LSUS es from LCP array mississippi RANK LCP L i 0 4 LSUS i m issis

14 Left Bounded Shortest Unique Substrings Fast computation of LSUS es from LCP array mississippi RANK LCP L i LSUS i m issis ssis

15 Left Bounded Shortest Unique Substrings Fast computation of LSUS es from LCP array mississippi RANK LCP L i LSUS i m issis ssis sis issip ssip sip ip pp

16 Left Bounded Shortest Unique Substrings Fast computation of LSUS es from LCP array mississippi RANK LCP L i LSUS i m issis ssis sis issip ssip sip ip pp pi

17 Left Bounded Shortest Unique Substrings Fast computation of LSUS es from LCP array mississippi RANK LCP L i LSUS i m issis ssis sis issip ssip sip ip pp pi -

18 Left Bounded Shortest Unique Substrings Properties of LSUS LSUS 1 always exists, because at least it can be the whole string, which is unique.

19 Left Bounded Shortest Unique Substrings Properties of LSUS LSUS 1 always exists, because at least it can be the whole string, which is unique. LSUS i cannot end before LSUS i 1 Repeat abcadabcdcab... Unique

20 Finding Shortest Unique Substring For All Positions Finding leftmost SUS for all positions: Overview SUS i = { SUSi 1 + S[i], if SUS i SLS i SLS i, if SUS i > SLS i SLS i Leftmost shortest LSUS covering string location i.

21 Finding Shortest Unique Substring For All Positions abcabcbcab

22 Finding Shortest Unique Substring For All Positions abcabcbcab abcabcbcab

23 Finding Shortest Unique Substring For All Positions abcabcbcab abcabcbcab

24 Finding Shortest Unique Substring For All Positions abcabcbcab abcabcbcab

25 Finding Shortest Unique Substring For All Positions abcabcbcab abcabcbcab

26 Finding Shortest Unique Substring For All Positions Finding the candidate: Data structure Chunk Start LSUS Start Prev Next Chunk End LSUS Length abcacbca...

27 Finding Shortest Unique Substring For All Positions Example Run S = mississippi List

28 Finding Shortest Unique Substring For All Positions Example Run m i s s i s s i p p i CS LS List 1 1 CE LL 1 1 SUS 0 = SLS 1 = (1, 1) SUS 1 = (1, 1) CS: Chunk Start CE: Chunk End LS: LSUS Start LL: LSUS Length

29 Finding Shortest Unique Substring For All Positions Example Run m i s s i s s i p p i List CS 2 LS 2 CE 6 LL 5 SUS 1 = (1, 1) SLS 2 = (2, 5) SUS 2 = (1, 2) CS: Chunk Start CE: Chunk End LS: LSUS Start LL: LSUS Length

30 Finding Shortest Unique Substring For All Positions Example Run m i s s i s s i p p i List CS LS 3 3 CE 6 LL 4 SUS 2 = (1, 2) SLS 3 = (3, 4) SUS 3 = (1, 3) CS: Chunk Start CE: Chunk End LS: LSUS Start LL: LSUS Length

31 Finding Shortest Unique Substring For All Positions Example Run m i s s i s s i p p i List CS LS 4 4 CE 6 LL 3 SUS 3 = (1, 3) SLS 4 = (4, 3) SUS 4 = (4, 3) CS: Chunk Start CE: Chunk End LS: LSUS Start LL: LSUS Length

32 Finding Shortest Unique Substring For All Positions Example Run m i s s i s s i p p i CS LS CS LS List CE LL CE LL SUS 4 = (4, 3) SLS 5 = (4, 3) SUS 5 = (4, 3) CS: Chunk Start CE: Chunk End LS: LSUS Start LL: LSUS Length

33 Finding Shortest Unique Substring For All Positions Example Run m i s s i s s i p p i List CS LS CS LS CE LL CE LL SUS 5 = (4, 3) SLS 6 = (4, 3) SUS 6 = (4, 3) CS: Chunk Start CE: Chunk End LS: LSUS Start LL: LSUS Length

34 Finding Shortest Unique Substring For All Positions Example Run m i s s i s s i p p i List CS 7 LS 7 CE 9 LL 3 SUS 6 = (4, 3) SLS 7 = (7, 3) SUS 7 = (7, 3) CS: Chunk Start CE: Chunk End LS: LSUS Start LL: LSUS Length

35 Finding Shortest Unique Substring For All Positions Example Run m i s s i s s i p p i List CS LS 8 8 CE 9 LL 2 SUS 7 = (7, 3) SLS 8 = (8, 2) SUS 8 = (8, 2) CS: Chunk Start CE: Chunk End LS: LSUS Start LL: LSUS Length

36 Finding Shortest Unique Substring For All Positions Example Run m i s s i s s i p p i CS LS CS LS List CE LL CE LL SUS 8 = (8, 2) SLS 9 = (8, 2) SUS 9 = (8, 2) CS: Chunk Start CE: Chunk End LS: LSUS Start LL: LSUS Length

37 Finding Shortest Unique Substring For All Positions Example Run m i s s i s s i p p i CS LS CS LS List CE 10 LL CE LL SUS 9 = (8, 2) SLS 10 = (9, 2) SUS 10 = (9, 2) CS: Chunk Start CE: Chunk End LS: LSUS Start LL: LSUS Length

38 Finding Shortest Unique Substring For All Positions Example Run m i s s i s s i p p i List CS 11 LS 10 CE 11 LL 2 SUS 10 = (9, 2) SLS 11 = (10, 2) SUS 11 = (10, 2) CS: Chunk Start CE: Chunk End LS: LSUS Start LL: LSUS Length

39 Finding Shortest Unique Substring For All Positions Time Complexity Derivation of LCP and Rank array Calculation of LSUS Maintenance of data structure Finding SLS Determining SUS Number of iterations: Time O(n) O(1) O(1) O(1) O(1) n Total Time: O(n) + n (O(1) + O(1) + O(1) + O(1)) = O(n)

40 Finding Shortest Unique Substring For All Positions Space Complexity LCP array Rank array Data structure Previous SUS Memory (in words) O(n) O(n) O(n) O(1) Total Memory: O(n) + O(n) + O(n) + O(1) = O(n)

41 Experimental Results Involved work in experimental study J. Pei, W. C. H. Wu, M. Y. Yeh, On shortest unique substring queries, ICDE 2013 K. Tsuruta, S. Inenaga, H. Bannai, M. Takeda, Shortest unique substrings queries in optimal time, SOFSEM 2014

42 Experimental Results Time Measurements

43 Experimental Results Memory Measurements

44 Conclusion Conclusion We developed O(n) time and space algorithms for finding SUS covering one position and all positions. Our work improved the prior work of Pei et al. 8 times in terms of time and 20 times in terms of memory.

45 Acknowledgments Acknowledgments I want to thank them for their monetary support for my trip to CPM2014. CPM Organizing Commitee Bilim Akademisi

46 Q&A Thanks! Questions?