Parallelizing Frequent Web Access Pattern Mining with Partial Enumeration for High Speedup

Transcription

1 Parallelizing Frequent Web Aess Pattern Mining with Partial Enumeration for High Peiyi Tang Markus P. Turkia Department of Computer Siene Department of Computer Siene University of Arkansas at Little Rok University of Arkansas at Little Rok Little Rok, AR Little Rok, AR Abstrat Abstrat The maximum speedup of diret parallelization of pattern-growth mining algorithms for long sequenes is limited by the load imbalane among the parallel tasks. In this paper, we present a sheme to parallelize pattern-growth mining algorithms using partial enumeration for high speedup. The experimental results show that partial enumeration inreases the ahievable speedup of parallel mining signifiantly for the databases with long sequenes. 1 Introdution A sequene is the fundamental mehanism to enode information and design as demonstrated by the biologial sequenes. This is the reason why data mining of frequent patterns from sequenes inluding web aess sequenes has attrated signifiant attention in reent years [1, 2, 3, 4, 5]. The omplexity of mining frequent patterns from sequene databases is inherently exponential due to the fat that patterns are sequenes themselves. The searh spae of frequent patterns grows exponentially as the length of the patterns inreases. One way to ope with this omplexity is to use parallel omputers for the mining. In this paper, we fous on parallelizing the pattern-growth frequent pattern mining algorithms [1, 5, 6]. The major problems in diret parallelizing the pattern-growth algorithms are two-fold: (1) Beause the pattern-growth algorithms are reursive algorithms, the number of parallel tasks that an be found is limited by the number of the symbols used in the sequenes. (2) Aording to empirial studies, the exeution times of these parallel tasks vary signifiantly. Supported in part by NASA and the Arkansas Spae Grant Consortium under Grant UALR The maximum speedup ahievable is limited by the longest task, even thought more parallel proessors are available. In this paper, we propose to use partial enumeration to parallelize the pattern-growth mining algorithms to solve the problems above. Partial enumeration is a tehnique to extend the pattern-growth algorithms so that the frequent patterns an grow faster with more than one symbol at a time [7]. It was shown in [7] that the sequential mining time an be redued by partial enumeration for the databases with long sequenes. The purpose of using partial enumeration in this paper is to inrease the number of tasks beyond the number of symbols and, more importantly, to have better load-balaned parallel tasks to ahieve high speedup. Our experimental results show that partial enumeration an inrease the speedup of parallel mining signifiantly for the databases with long sequenes. The rest of the paper is organized as follows. In Setion 2, we introdue the tehnique of partial enumeration and show why it an inrease the maximum speedup ahievable in the parallel mining. In Setion 3, we desribe the sheme to parallelize the patterngrowth mining algorithm [6] using partial enumeration [7]. The experimental results and the onlusion of the paper are presented in Setion 4. 2 Partial Enumeration for High Let Σ be the set of symbols used to onstrut sequenes. A web aess sequene s is a sequene of finite number of symbols from Σ, s = σ 1 σ m (σ i Σ for all 1 i m < and s i and s j are not neessarily different for i j). A web aess database D is a multiset of web aess sequenes. A pattern is also a web aess sequene. A web aess sequene s = σ 1 σ n is a subsequene of sequene s = σ 1 σ m, denoted as s s, if and only if there exist i 1,, i n (n m) suh 163

2 eplaements that 1 i 1 < < i n m and σ j = σ i j for all 1 j n. A web sequene s is said to support a pattern p if p is a subsequene of s. The support of pattern p in database D, denoted as Sup D (p), is the number of sequenes in D that support p. Given a threshold ξ in interval (0, 1], a pattern p is frequent with respet to ξ and D if Sup D (p) ξ D, where D is the number of sequenes in D. ξ D is alled the absolute threshold and denoted by η. The web aess pattern mining problem is to find all the frequent patterns with respet to ξ and D. Figure 1 illustrates the searh spae for the frequent patterns from the symbol set Σ = {a, b,. a b a ɛ a b a b aa ab a ba bb b a b a b a b a b a b aaa a baa b aa b a b a b Figure 1: Searh Spae Tree a b a b a b a b The pattern-growth algorithms [1, 5, 6] grow frequent patterns by one symbol at a time. The abstrat form of the pattern-growth algorithm [6] is shown in Figure 2. The pattern-growth algorithms [1, 5, 6] are funtion Pattern-Grow(pattern q, database D) { F ; for eah σ in Σ do if (Sup D(σ) η) then F F {q σ; Construt projetion database D σ; F Pattern-Grow(q σ, D σ); F F F ; endfor return F ; Figure 2: Pattern-Growth without Enumeration all based on the priniple of onditional searhing. D σ is the σ-projetion database of D. It onsists of all the σ-projetions of the sequenes in D that support σ. The σ-projetion of a sequene is what left after the prefix from the first symbol up to the first ourrene of σ (inlusive) is deleted. For example, the b-projetion of aabba is ba beause its prefix up to the first ourrene of b as underlined is aab. It an be proved that the support of pattern p in D σ is equal to the support of pattern σ p in D, i.e. Sup D (σ p) = Sup Dσ (p). The details for the priniple of onditional searhing an be found in [6]. The set of all frequent patterns in D, thus, an be found by invoking Pattern-Grow(ɛ, D) in Figure 2 (ɛ is the empty pattern). Sine the pattern-growth algorithms are all reursive algorithms as shown in Figure 2, the only way to parallelize them is to turn the for loop of the top level all into a parallel loop and alloate the tasks of mining the frequent patterns starting with individual symbols to parallel proessors. It is obvious that the maximum speedup of this parallelization is bound by the number of symbols in Σ. For example, we an use only three proessors for the parallel mining for the searh spae in Figure 1 and the maximum speedup is 3. Many important appliations do not have large number of symbols (DNA sequenes have four symbols of nuleotides and protein sequenes have 20 symbols of amino aids). The maximum speedup is further limited by the load imbalane of these tasks. We onduted an empirial study about the exeution times of these tasks and found that they vary signifiantly. Task Times in Seonds Task Times of N=12 and threshold=0.005 task times Task Number Figure 3: Task Times for N=12, C=16, D=00 Figure 3 shows the task times of the 12 tasks of the FLWAP-tree mining [6] on a 497 MHtz Pentium III proessor for a dataset with N = 12 symbols generated by the IBM data generator. The dataset has D=00 sequenes and the average length of the sequenes is C = 16. The threshold ξ is The task with the largest exeution time, whih we alled supertask, is task 12. It takes seonds to omplete. The total exeution time of all the tasks is seonds. If we run the mining on parallel proessors, the parallel exeution time is at least the task time of the supertask. The maximum speedup that an be ahieved is, thus, /3.079 = 5.03, even though more parallel proessors are available. In general, if T 1,, T n be the task times of the total n parallel tasks of mining, the maximum speedup 164

3 eplaements ahievable by parallel mining is SP M = T T n max(t 1,, T n ) Maximum with different k k=3 k=2 k= Number of Symbols, N Figure 4: Maximum s SP M (1) We also onduted an empirial study of the task times for the datasets with N = 2,, 12 symbols, C = 16 and ξ = The maximum speedups for these datasets alulated by (1) are plotted on the line labeled k=1 in Figure 4. They are between 1.64 and 5.02, way below the number of the tasks (whih is N), due to the load imbalane of the tasks. In order to raise the maximum speedup for parallel mining, the work loads of parallel tasks need to be better balaned. We have extended the patterngrowth mining with partial enumeration [7] to grow the frequent patterns by more than one symbol at a time 1. Figure 5 [7] shows the re-arranged searh spae ɛ a b bb dd aa d a b aa ab ba bb d d d dd d dd a b a ad... bb dd a bb add aaa aab... aabb aa... aadd da... a dd bb dbb da... dbb d... ddd Figure 5: Searh Spae by Partial Enumeration to illustrate the idea of partial enumeration. The symbol set Σ = {a, b,, d is first partitioned to disjoint subsets Σ 1 = {a, b and Σ 2 = {, d. For eah subset, the enumerations of the non-empty patterns of length no more than the size of the subset are alled partial enumerations. For example, the partial enumerations from Σ 1 = {a, b are a, b, aa, ab, ba, bb. The set of partial enumerations from the symbol subset Σ j 1 The experimental evaluation shows that the pattern-growth mining with partial enumeration [7] outperforms the patterngrowth mining without enumeration [6] for the datasets with long sequenes. funtion Pattern-Grow(pattern q, int i, database D) { F ; for j = 1, l do if (j i) then Construt apriori enumerator Enum(Σ j); 1: while (p Enum.Next()) is not null) do 2: if (Sup) D(p) η) then 3: Construt the p-projetion database D p,σj ; Call Enum.Confirm(p) to report p is frequent; F F {q p; if ( p = Σ j ) then F Pattern-Grow(q p, 0, D p,σj ); F Pattern-Grow(q p, j, D p,σj ); F F F ; Delete apriori enumerator Enum; endfor return F ; Figure 6: Pattern-Growth with Partial Enumeration is denoted by E(Σ j ). The searh spae is re-arranged as follows. The empty pattern ɛ at the root node grows with the partial enumerations from both Σ 1 = {a, b and Σ 2 = {, d. For eah of the other nodes, if it was grown from its parent node with a partial enumeration shorter than the size of the symbol subset, it will not be extended with the same symbol subset. The reason is to prevent repeating the same pattern in the searh spae tree. For example, pattern a is grown from the empty pattern ɛ with partial enumeration a whose length is less than 2, the size of Σ 1 = {a, b. Therefore, pattern a grows with the partial enumerations from Σ 2 = {, d only, namely, d,, d, d, dd. If it grows with the partial enumerations from Σ 1 = {a, b, we would have patterns aa and ab as its hildren. But, aa and ab are already enumerated along with a as the hildren of the root node ɛ. Patterns aa and ab would be mined twie if they are frequent. On the other hand, if a node is grown from its parent with a partial enumeration of the length equal to the size of the symbol subset, it should be extended with the partial enumerations of all symbol subsets. For instane, pattern aa is grown from the parent node (the empty pattern ɛ) with partial enumeration aa whose length is 2, the size of Σ 1 = {a, b. It should ontinue to grow with the partial enumerations from both Σ 1 = {a, b and Σ 2 = {, d. In partiular, the growth from pattern aa with the partial enumerations from Σ 1 = {a, b 165

4 funtion Mine(enum p, int j, database D) { if (Sup D(p) η) then Report to the Master that p of E(Σ j) is frequent; F {p; Construt projetion database D p,σj ; if ( p = Σ j ) then F Pattern-Grow(p, 0, D p,σj ); F Pattern-Grow(p, j, D p,σj ); F F F ; Send F to the Master and request a next task; Report to Master that p of E(Σ j) is infrequent and request a next task; main-slave() { Input the database D; Send a task Request to Master Proessor; Reeive a task (p, j) from Master Proessor; while (task is not null) do Mine(p, j, D); Reeive a task (p, j) from Master Proessor; Figure 7: Algorithm for Slave Proessors will not dupliate any patterns enumerated. When enumerating the partial enumerations from E(Σ j ), the downward enlosure property should be used to prune away the infrequent enumerations if their subsequenes are known to be infrequent. For example, if a from E({a, b) is infrequent, then aa, ab, ba from E({a, b) are also infrequent and should not be enumerated. This is alled apriori partial enumeration. Figure 6 shows the frequent pattern mining algorithm with apriori partial enumeration [7]. The apriori partial enumerations p from E(Σ j ) are obtained by alling the Next() funtion of the apriori partial enumerator Enum, reated for Σ j. If p is frequent in D (Sup D (p) η), the p-projetion database of D, denoted as D p,σj, is onstruted. The apriori partial enumerator Enum is also informed that p is frequent. D p,σj onsists of p-projetions of the sequenes in D that support p. The p-projetion of sequene s supporting p is what left after the p-prefix of s is deleted. The p-prefix of s is the minimal prefix of s that supports p. It is assumed in Figure 6 that the symbol set Σ is partitioned to l subsets, Σ = Σ 1 Σ l. It an be proved that eah frequent pattern will be mined only one by the algorithm in Figure 6 [7]. The set of frequent patterns in the original database D is mined by alling Pattern-Grow(ɛ, 0, D). Eah apriori partial enumeration p obtained in line 1 of Figure 6 represents a task of mining the orresponding projetion database, D p,σj. We onduted an empirial study of the exeution times of these tasks and found that the work loads of these tasks are muh better balaned. As a result, the maximum speedup alulated by (1) is raised. We partition the symbol set Σ to l = N k subsets, where N = Σ. The first l 1 subsets have the size of k and the size of the last subset may be less than k. When k = 1 (and thus l = N), the algorithm in Figure 6 degenerates to the mining algorithm without enumeration as shown in Figure 2. The plots labeled k=2 and k=3 in Figure 4 are the maximum speedups of the tasks with partial enumeration for the same datasets with N = 2,, 12, C=16, and ξ= The maximum speedup for k = 2 is higher than that of k = 1 for every value of N. When k = 3, the maximum speedups are higher than k = 2 exept for the ase of N = 4. These empirial studies show that partial enumeration an raise the maximum speedup ahievable in parallel mining of frequent patterns. 3 Parallelization The parallel tasks represented by apriori partial enumerations p from E(Σ j ) for all j, denoted by task (p,j), are sheduled by the master proessor and exeuted by the slave proessors. The ode for parallel task (p, j) is shown in Figure 7. Eah slave proessor starts by inputting the original database D and sending a task request to the master proessor. Then it reeives a task (p, j) and exeutes it by alling Mine(p, j, D) repeatedly until it reeives a null task signifying that all the tasks have been alloated. The main funtion of slave proessors is main-slave() shown in Figure 7. Slave Master Task Manager T Enum Enum P,W, Q... Sheduler next task urrent state buff P,W, Q... Slave Slave Slave Figure 8: Software Arhiteture of Parallel Proessors Figure 8 shows the software arhiteture of the 166

5 system. The master proessor runs a dynami sheduler. The basi funtion of the sheduler is to interat with the task manager T to get parallel tasks and alloate them to the slave proessors dynamially. The sheduler of the master proessor ommuniates with the slave proessors through the messages as follows: Slave proessor S i sends task request Req(S i ) to the master proessor. The master proessor sends task (p, j) (or a null task signifying that all the tasks are alloated) to slave proessor S i in response to the task request Req(S i ). Slave proessor S i sends a Report to the master proessor about whether p is frequent in D in response to the task (p, j). Slave proessor S i sends the set of frequent patterns it mined for the task (p, j) to the master proessor, if p is frequent. Upon Reeiving task request Req(S i) from S i { if (urrent-state is wait) then Enqueue S i into buffer buff; if (urrent-state is ready) then Send task (p, j) held in next-task to S i; T.Update(urrent-task, urrent-state); // urrent-state is done Send null task to slave proessor S i; Upon Reeiving a report Report(p, j, freq) { Pass the report to the task manager T by alling T.Report(p,j,freq); if (urrent-state is wait) then T.Update(urrent-task, urrent-state); while ( urrent-state is ready and buff is not empty) do S buff.dequeue(); Send task (p, j) held in next-state to slave proessor S; T.Update(urrent-task, urrent-state); while (urrent-state is done and buff is not empty) do S buff.dequeue(); Send the null task to slave proessor S; Figure 9: The Protool of the Sheduler As shown in main-slave() in Figure 7, the slave proessors piggy-bak the request for the next task to the report of infrequent p or the set of frequent patterns mined. However, the report that p is frequent is sent as soon as it is known before the mining starts. This allows the master to generate further parallel tasks at the earliest possible time. The sheduler has two variables next-task and urrent-state (see Figure 8). The next-task holds the next task (p, j) ready to be dispathed to any slave proessor that requests a task. The urrent-state tells the urrent state of the sheduler. It an hold one of the three states as follows: ready: next-task holds a task ready to be dispathed. wait: There is no task ready to be dispathed and further tasks are pending. done: All the tasks have been generated. The task manager T has two funtions to be alled by the sheduler: Update(next-task, urrent-state) to update nexttask and urrent-state. Report(p, j, freq) to report that the pattern p of task (p, j) is frequent(freq being true) or not (freq being false). The main funtion of the sheduler is a finite state mahine protool responding to the task requests and the reports sent from the slave proessors. The sheduler also has a buffer queue, buff, to hold the slave proessors that are waiting for tasks. The protool of the sheduler is shown in Figure 9. The task manager T manages l apriori enumerators Enum j (j = 1,, l), one for eah symbol subset Σ j. The details of the task manager T an be found in [8]. 4 Experimental Results and Conlusion We have implemented our parallel web aess pattern mining with MPI on a luster of workstations. Two datasets, N4C17D1KS4 and N6C16D1KS4, are generated by the IBM data generator for the experiments. In the names of the datasets, we use N for the number of symbols, i.e. Σ, C for the average length of the sequenes of the database, D for the size of the database (in thousands). Both datasets were mined using threshold ξ = We run the parallel mining for the datasets N4C17D1KS4 and N6C16D1KS4 with k = 1, 2, 4 and k = 1, 2, 3, 6, respetively. The number of parallel 167

6 slave proessors varying from 1 to 13. Eah slave proessor is a 497 MHtz Pentium III proessor. The master is a 997 MHtz Pentium III proessor. Figures (a) and (b) show the speedup of the parallel exeutions over the sequential mining without enumeration enumeration [6] of datasets N4C17D1KS4 and N6C16D1KS4, respetively. Figures (a) shows that k = 4 allows the speedup to inrease as the number of proessors inreases. The k = 3 and k = 6 do the same for dataset N6C16D1KS4 as shown in Figure (b). This is beause with large k values the partial enumeration generates more and better loadbalaned parallel tasks and, thus, allows more parallel proessors to be used to redue the exeution time. Note that the speedup of k = 3 for N6C16D1KS4 is higher than that of k = 6. This is beause with k = 6 the partial enumeration generates many smaller parallel tasks and the ommuniation overhead between the master and slave proessors slows down the parallel exeution k=1 k=2 k=4 over k=1 Sequential Exeution Number of Parallel Proessors (a) Dataset N4C17D1KS4 over k=1 Sequential Exeution k=1 k=2 k=3 k= Number of Parallel Proessors (b) Dataset N6C16D1KS4 Figure : over Sequential Mining We have presented a sheme to parallelize the pattern-growth frequent pattern mining algorithms using partial enumeration. We have shown that partial enumeration an inrease the speedup ahievable by providing better load-balaned parallel tasks. Previous works on parallelizing the sequential pattern mining suh as [9] use diret parallelization and only the databases with short sequenes are used. Referenes [1] Jian Pei, Jiawei Han, Behzad Mortazavi-asl, and Hua Zhu. Mining aess patterns effiiently from web logs. In Proeedings of the 4th Paifi-Asia Conferene on Knowledge Disovery and Data Mining (PAKDD 00), pages Leture Notes in Computer Siene, Vol. 1805, [2] J. Ayres, J. Flannik, J. Gehrke, and T. Yiu. Sequential pattern mining using a bitmap representation. In Proeedings of the eighth ACM SIGKDD International Conferene on Knowledge Disovery and Data Mining, pages , [3] J. Wang and J. Han. BIDE: Effiient mining of frequent losed sequenes. In Proeedings of the 20th International Conferene on Data Engineering (ICDE 04), pages 79 90, [4] Ke Wang, Yabo Xu, and Jeffery Xu Yu. Salable sequential pattern mining for biologial sequenes. In In Proeedings of the Thirteenth ACM International Conferene on Information and knowledge management (CIKM 04), pages , November [5] C.I. Ezeife and Yi Lu. Mining web log sequential patterns with position oded pre-order linked WAP-tree. International Journal of Data Mining and Knowledge Disovery, :5 38, [6] Peiyi Tang, Markus P. Turkia, and Kyle A. Gallivan. Mining web aess patterns with first-ourrene linked WAP-trees. In Proeedings of the 16th International Conferene on Software Engineering and Data Engineering (SEDE 07), pages , Las Vegas, USA, July [7] Peiyi Tang and Markus P. Turkia. Mining frequent web aess patterns with partial enumeration. In Proeedings of the 45 th Annual Assoiation for Computing Mahinery Southeast Conferene (ACMSE 07), pages , Winston-Salem, NC, USA, Marh [8] Peiyi Tang and Markus P. Turkia. Parallelizing frequent web aess pattern mining with partial enumeration for high speedup. Tehnial Report titus.ompsi.ualr.edu/~ptang/papers/parwap.pdf, Dept of CS, UALR, [9] Shengnan Cong, Jiawei Han, and David Padua. Parallel mining of losed sequential patterns. In Proeeding of the Eleventh ACM SIGKDD International Conferene on Knowledge Disovery in Data Mining, pages ,