Distributed Suffix Array Construction

Transcription

1 Distributed Suffix Array Construction Huibin Shen Department of Computer Science University of Helsinki String Processing Project April 25, 2012 Huibin Shen (U.H.) Distributed Suffix Array Construction April 25, / 15

2 Outline 1 Problem 2 Implementation 3 Experiments 4 Conclusion Huibin Shen (U.H.) Distributed Suffix Array Construction April 25, / 15

3 Problem Suffix Array Suffix array defination The suffix array of a text T is a lexicographically orderd array of the set T [0...n] of all suffixes of T. More precisely, the suffix array is an array SA[0... n] of integers containing a permutation of set [0... n] such that T SA[0] < T SA[1] < < T SA[n]. Example: The suffix array of the text T = banana$ Huibin Shen (U.H.) Distributed Suffix Array Construction April 25, / 15

4 Problem Suffix Array Suffix array construction algorithm Existing suffix array construction algotirhm: Prefix Doubling, O(nlogn). DC3, O(n). SAIS, O(n). All on one computer! Huibin Shen (U.H.) Distributed Suffix Array Construction April 25, / 15

5 Problem Suffix Array Distributed suffix array construction Deploy the suffix array construction on clusters! Each node of the cluster becomes a bucket for a subset of all the suffiexes Each node sorts the subset of suffixes independently. Merge the result of each node. Diagram is shown on next slide. Huibin Shen (U.H.) Distributed Suffix Array Construction April 25, / 15

6 Problem Suffix Array Distributed suffix array construction diagram suffixes of T: {banana$, anana$, nana$, ana$, na$, a$, $} master node node 1 [$,b$) node 2 [b$, c$)... {anana$, ana$, a$, $} {banana$} {nana$, na$} node [n$,o$) node n [z$, '{') {$, a$, ana$, anana$} {banana$} {na$, nana$} master node sorted suffixes of T: {$, a$, ana$, anana$, banana$, na$, nana$} Huibin Shen (U.H.) Distributed Suffix Array Construction April 25, / 15

7 Implementation Detail Implementation detail Besides the distributed suffix array construction algorithm, a linear time suffix array construction algotirhm DC3 is also implemented for comparison. All codes are written in python. Important pieces of code of distributed suffix array construction are shown here, while codes for DC3 are ignored. Huibin Shen (U.H.) Distributed Suffix Array Construction April 25, / 15

8 Implementation Detail Codes for distributing the tasks Huibin Shen (U.H.) Distributed Suffix Array Construction April 25, / 15

9 Implementation Detail Codes for sorting on one bucket Huibin Shen (U.H.) Distributed Suffix Array Construction April 25, / 15

10 Experiments Time Time comparison between DC3 and distributed sorting with 104 nodes log(time) in s DC3 Distributed Sorting (max) Distributed Sorting (sum) 5KB 10KB 25KB 50KB 100KB 250KB 500KB 1MB input size Huibin Shen (U.H.) Distributed Suffix Array Construction April 25, / 15

11 Experiments Time Time comparison in distributed sorting with different nodes Distributed Sorting 26 nodes Distributed Sorting 52 nodes Distributed Sorting 78 nodes Distributed Sorting 104 nodes Distributed Sorting 130 nodes 5KB 10KB 25KB 50KB 100KB 250KB 500KB 1MB 2.5MB input size Huibin Shen (U.H.) Distributed Suffix Array Construction April 25, / 15

12 Experiments Space Space comparison between DC3 and distributed sorting with 104 nodes virtual memory in MB DC3 Distributed Sorting (max) 5KB 10KB 25KB 50KB 100KB 250KB 500KB 1MB input size Huibin Shen (U.H.) Distributed Suffix Array Construction April 25, / 15

13 Experiments Space Space comparison in distributed sorting with different nodes peak virtual memory in MB Distributed Sorting 26 nodes Distributed Sorting 52 nodes Distributed Sorting 78 nodes Distributed Sorting 104 nodes Distributed Sorting 130 nodes 5KB 10KB 25KB 50KB 100KB 250KB 500KB 1MB 2.5MB 5MB input size Huibin Shen (U.H.) Distributed Suffix Array Construction April 25, / 15

14 Experiments Time distribution timein s Running time distribution of distributed sorting over 104 nodes 5KB 10KB KB 50KB KB 250KB KB 1MB MB 5MB Huibin Shen (U.H.) Distributed Suffix Array Construction April 25, / 15

15 Conclusion Thinking Lesson and future Do not use python when implementing string processing algorithm! Dynamically construct pivots based on string distribution so that every nodes (buckets) receive as equal amount of task as possible. Questions? Huibin Shen (U.H.) Distributed Suffix Array Construction April 25, / 15