Genetic Algorithms for Parallel Code Optimization

Transcription

1 Genetic Agorithms for Parae Code Optimization Ender Özcan Dept. of Computer Engineering Yeditepe University Kayışdağı, İstanbu, Turkey Emai: Abstract- Determining the optimum data distribution, degree of paraeism and the communication structure on Distributed Memory machines for a given agorithm is not a straightforward task. Assuming that a parae agorithm consists of consecutive stages, a Genetic Agorithm is proposed to find the best number of processors and the best data distribution method to be used for each stage of the parae agorithm. Steady state genetic agorithm is compared with transgenerationa genetic agorithm using different crossover operators. Performance is evauated in terms of the tota execution time of the program incuding communication and computation times. A computation intensive, a communication intensive and a mixed impementation are utiized in the experiments. The performance of GA provides satisfactory resuts for these iustrative exampes. I. INTRODUCTION Data distribution and processor aocation are two important factors that affect the performance of programs written for distributed memory parae architectures. Distribution of the data among the processors affects the communication time. On the other hand, the number of processors used at each step of the parae code (degree of paraeism) affects both the computation time and the communication time. Different approaches have been used to sove the probem of optimizing data distribution in parae programs [1]-[12]. These projects use a variety of optimization methods. There are aso research works that present notation for communication-free distribution of arrays [13], [14]. The probem of finding optima mappings of arrays for parae computers is shown to be NP-compete [15]. As a discrete probem, even if a restricted set of data decomposition patterns is used, the noninear nature of the probem does not change, especiay when combined with seecting the degrees of paraeism for each program stage and attempting to minimize the overa execution time. In this study, Genetic Agorithms (GA) is used to anayze the probem of determining the data distribution and the degree of paraeism for each stage of a parae code in order to minimize the tota execution time. II. PERFORMANCE CHARACTERIZATION OF PARALLEL ALGORITHMS A. Leves A seria agorithm may be composed of a sequence of stages (denoted as eves), where each stage is either a singe oop or a group of nested oops (Figure 1(a)). An Esin Onbaşıoğu Dept. of Computer Engineering Yeditepe University Kayışdağı, İstanbu, Turkey Emai: esin@cse.yeditepe.edu.tr exampe program segment is given in Figure 6, where there are five eves. The entire sequence of eves may aso be encosed by an iterative oop (Figure 1(b)). A genera case is given in Figure 1(c), where some consecutive oops are encosed by iterative oops, which again with adjacent oops may be encosed by other iterative oops and so on, ike a tree structure. Here, to simpify the case, it is assumed that programs are in the form of Figure 1(a) and Figure 1(b), and programs that have structure as in Figure 1(c) are reduced to the form in Figure 1(b) by combining L 2, L 3, L 4 as a singe eve. [ L 1 [ L 1 [ L 1 [ L 2 [ L 2 [ L 2 [ L 3 [ L 3 [ L 3 [ L 4 [ L 4 [ L 4 [ L 5 [ L 5 [ L 5 (a) (b) (c) Figure 1. (a) sequence of eves, (b) sequence of eves encosed by an iterative oop, (c) sequence of oops with a genera structure When the programs are paraeized, each eve is assumed to have a different degree of paraeism that is, each eve may be executed using a different number of processors. Leves requiring short computation may be paraeized on a few processors but those having ong computations may require the use of many processors. Aso, due to the distribution of the data among the processors, when the code is paraeized, communication may be required before the execution of each eve. Athough increasing the number of processors decreases the computation time of a eve, it may cause extra communication between the eves. B. Performance Evauation Performance of a seria agorithm can be expressed in terms of the probem size, but the performance of a parae agorithm, in addition to the probem size, depends on the number of avaiabe processors, the distribution of the data among the processors, and the methods used for transferring data between processors. At some stage of a parae code, execution time (t exec ) can be expressed as the sum of the computation and communication times

2 t = t + t Equation 1., exec comp comm where t comp denotes the computation time, and t comm denotes the communication time at stage. Computation time can be formuated as t = t / p Equation 2., comp seq where tseq is the predicted computation time of stage of the sequentia agorithm on a singe processor, and p is the number of processors used in the parae code for stage. Communication time depends on the number of processors p and the communication structure c used for the transfer of data at that stage. t comm consists of two terms, a term that increases with the size of data to be transferred, represented by f 1, and an overhead, represented by f 2, (a) Horizonta (b) Repicated (c) Rectange Figure 2. Possibe decomposition patterns for a 1D array on 4 processors (a) Horizonta (b) Vertica t = f ( c, p ) d + f ( c, p ) Equation 3., comm 1 2 where d is the data size per processor. Note that both f 1 and f 2 are in the form (a p + b), where a and b are constants that depend on the communication structure. The tota execution time (T) of the program is the sum of the execution times of a stages in the program (c) Repicated (d) Rectange L T = t = 1 exec Equation 4., where L is the tota number of stages (denoted as eves) in the code. In order to determine the execution time of a parae program at compie-time, machine characterization and performance prediction method given in [16] is used. In this method, computation and communication characteristics of the machine are measured and formuated. In order to cacuate the tota execution time of a parae program, its program parameters (i.e. computation time, data size) are substituted in the formuas. C. Data Decomposition and Aignment In most of the parae agorithms, arrays of one or higher dimensions are used. In this study, arrays are assumed to be distributed to the processors in the form of bocks. Bock decompositions of 1 and 2-dimensiona arrays on four processors are iustrated in Figure 2 and Figure 3, respectivey. It is assumed that a scaar vaues are repicated on a processors. Figure 3. Possibe decomposition patterns for a 2D array on 4 processors Arrays are distributed to the processors according to one of the decomposition patterns, presented in Figure 2 and Figure 3. Due to the computationa requirements, arrays existing at a eve, may be distributed to processors using the same or different decomposition patterns. Considering a the arrangements of decomposition patterns for a the arrays at a eve, some of them may not be feasibe. Feasibe arrangements of decomposition patterns of arrays at a eve are referred as aignment. As an exampe, possibe aignments for the third eve of the code in Figure 7 are demonstrated in Tabe I. The arrangement, (Horizonta, Vertica, Horizonta), for arrays c, I and b is not a feasibe arrangement, as it can not satisfy the computationa requirements. Hence, it is not accepted as an aignment. TABLE I. ALLIGNMENTS FOR THE THIRD LEVEL OF THE CODE IN FIGURE 7 Array Aignment 1 Aignment 2 Aignment 3 c Horizonta Vertica Rectanguar I Horizonta Vertica Rectanguar b Horizonta Vertica Rectanguar

3 D. Communication Structures In distributed memory architectures using messagepassing, generay, data is transferred among the processors in a structured way. Different communication structures have been defined for data exchange between processors [17], [18]. In this study, mutiphase (MU), shift (SH), broadcast (BR), scatter (SC) and gather (GA) structures have been utiized (Figure 4). references to the same arrays in the upper and ower eves of the code. Then, a common feasibe communication structure is seected. In this work, instead of seecting the communication structure randomy, the one that produces the minimum transfer time is seected. In order to minimize the program execution time, the best number of processors for each eve of the code and the best aignment for the arrays that are referred to at each eve of the code must be determined. A parae code might consist of many eves. Furthermore, for each eve there might be different possibiities for aignment, degree of paraeism and communication structure. For this reason, the probem of finding the optima parae code configuration becomes highy compex. (a) Mutiphase (c) Scatter (e) Shift (b) Broadcast (d) Gather Figure 4. Five types of communication structures used in the study Performance of the communication structures can be characterized in terms of f 1 and f 2, as shown in Tabe II. For a structures, f 1 and f 2 depend on the number of processors, except for SH, where f 1 and f 2 are constant. The machine parameters M, N, R and Q are measured running benchmarks on the parae machine, as expained in [16]. The hardware patform used in this study consists of 16 Pentium 4, 2GHz processors connected by 100Mbit/s interconnection network. TABLE II. MACHINE PARAMETERS FOR THE COMMUNICATION STRUCTURES c f1 ( c, ) p f2 ( c, ) p MU M 1 p + N 1 R 1 p + Q 1 BR M 3 p + N 3 R 3 p + Q 3 SC M 4 p + N 4 R 4 p + Q 4 GA M 5 p + N 5 R 5 p + Q 5 SH N 2 Q 2 Once the data distribution and the number of processors at each eve are determined, communication structures to be inserted between eves can be seected. For each eve, feasibe communication structures, that are vaid for a arrays at that eve, are identified considering the III. GENETIC ALGORITHMS FOR PARALLEL CODE OPTIMIZATION Genetic Agorithms have been used for soving many difficut probems [19] that were introduced by J. Hoand in 1975 [20]. Given a parae code, consisting of L eves and a distributed memory parae machine with P processors, parae code optimization (PCO) can be stated as finding the best mapping of P processors and D aignments of data decomposition patterns at each eve, reducing the tota expected execution time of the agorithm. The search space ranges up to (PD) L, assuming the best communication structure for the eve. Athough not considered in here, the size of a eve and seecting the eves to combine may aso be used as other optimization parameters. Machine parameters are obtained by Benchmarking as expained in [16] and fed into the GA Sover for execution time optimization as shown in Figure 5. The Aignment Parser parses the given sequentia code to be paraeized, in a C ike anguage. It determines a possibe array aignments in the code and generates the reated data decomposition patterns at a eves, handing them over to the GA sover. Finay, GA sover produces the best assignment of aignments and number of processors at each eve. Sequentia Code Aignment Parser Bechmarking GA Sover Figure 5. Fowchart showing how parae code optimization probem is soved by GA

4 A. GA Components Each individua is a ist of aees, where each ocus represents a eve in the given code. Aees consist of two parts: number of processors to be used at a eve and the data aignment. Figure 6 demonstrates an exampe individua, having 5 eves One iteration of the Hessenberg reduction is represented as foows: A = (I V Z V T ) T A (I V Z V T ), where A, Z, V are NxN matrices. Its parae impementation has 5 eves as shown in Figure 7. Leves 1, 2, 4 and 5 perform matrix mutipication operation where the execution time increases with N 3, and eve 3 consists of a subtraction operation where the execution time increases with N 2. 4 processors and 5 th aignment shoud be used at eve 3 Figure 6. An exampe individua representing a candidate soution for a PCO probem with 5 eves Two types of Genetic Agorithms are impemented as a sover: steady state genetic agorithms (SSGA) and transgenerationa genetic agorithms (TGGA). Popuation size is chosen to be proportiona to the ength of an individua. Fitness function indicates the tota execution time as shown in Equation 4. The best communication structure is chosen at each eve among a possibiities. This step can be considered as a hi cimbing step. This process can be appied, since the contribution of the communication structure to the tota execution time at each eve is independent. SSGA and TGGA both utiize inear ranking strategy for seecting parents and eitist repacement strategies. In SSGA, two worst individuas in the popuation are deeted and both offspring are injected in their paces. In TGGA, the best of the offspring combined with the previous popuation forms the next generation. SSGA visits two new states at each generation, whie the number of states that TGGA visits is two times the individua ength, determining the number of evauations. Different crossover operators are tested: traditiona one point crossover (1PTX), two-point crossover (2PTX) and uniform crossover (UX). Traditiona mutation is used, randomy perturbing an aee, assigning a random vaue to the number of processors and the decomposition pattern. Runs are terminated, whenever the expected fitness is reached or the maximum number of generations is exceeded. IV. EXPERIMENTAL DATA Experiments are performed using three data sets produced from two different agorithms. A. Hessenberg Reduction In the first and second data sets, the parae agorithm in [21] for reducing matrices to Hessenberg form is used. for (i=0; i<n; ++i) /* Leve 1 */ for (j=0; j<n; ++j) { a[i][j]=0; a[i][j]=a[i][j]+v[i][k]*z[k][j]; /* VZ */ for (i=0; i<n; ++i) /* Leve 2 */ for (j=0; j<n; ++j) { b[i][j]=0; b[i][j]=b[i][j]+a[i][k]*v[j][k]; /* VZV^T */ for (i=0; i<n; ++i) /* Leve 3 */ for (j=0; j<n; ++j) c[i][j]=i[i][j]-b[i][j]; /* I- VZV^T */ for (i=0; i<n; ++i) /* Leve 4 */ for (j=0; j<n; ++j) { d[i][j]=0; d[i][j]=d[i][j]+c[k][i]*a[k][j]; /* (I- VZV^T)^T A */ for (i=0; i<n; ++i) /* Leve 5 */ for (j=0; j<n; ++j){ e[i][j]=0; e[i][j]=e[i][j]+d[i][k]*c[k][j]; /* (I- VZV^T)^T A (I- VZV^T) */ Figure 7. Impementation of Hessenberg reduction Hessenberg reduction is chosen due to its simpe nature. At each eve there are three possibe aignments and three choices for the number of processors (1, 4 and 16). In order to be abe to test GA using a known resut, a 9 5 possibe combinations of aignments and the number of processors are computed to determine the optimum configuration. When the probem size N is chosen as 1024, the probem is communication intensive; therefore the best

5 fitness is achieved when the agorithm is run on one processor regardess of the decomposition pattern. However when N is chosen as 10240, the probem becomes computation intensive and the best fitness is observed utiizing 16 processors at each eve, as marked in Figure 8. Note that there are two optima configurations having the best fitness. Above data sets are used during the initia experiments, denoted as H1 and H2, with N=1024 and N=10240, respectivey. Considering the size of the search space and the number of optima points having the same fitness vaue in the search space, H1 is a simper probem than H2 to sove Fitness Aignment at each eve No.of processors at each eve Figure 8. Part of the fitness andscape for H2, N=10240, considering {4, 16 as possibe number of processors and {1, 2 as possibe aignment ids, where arrow points out the best fitness. B. Dongarra s Benchmark The third data set consists of 11 oops seected from the test oops prepared by J. Dongarra for paraeizing compiers, found at site: This probem invoves 12 matrices of sizes n 1 and n 2, and vectors of size n 2. Matrix addition, matrix mutipication, vector-matrix mutipication, assignment, etc. operations are executed on different matrices and vectors (Tabe III). In this case, for a eves, fitness increases with n 1 xn 2, except for eve 1 where fitness increases with n 3 1. When n 1 and n 2 are chosen as 10240, eve 1 becomes computation intensive, and therefore it can be estimated that it wi have the best time for 16 processors. Leve 2 aso has the same arrays as eve 1, therefore to minimize the communication time, 16 processors may be feasibe. Operations in the other eves are not computation intensive and they have a different set of arrays than eves 1 and 2. Therefore, it is expected that 1 processor may be the best choice for these eves. TABLE III. DETAILS FOR SELECTED LOOPS FROM DONGARRA S BENCHMARK SET. Arrays Array Operations 1 X2, Y2, Z2 Matrix Mutipication 2 X2, Y2, Z2 Assignment, 2D-Array Mutipication 3 A2, B2, C2 2D-Array Addition 4 A2, B2, C2, S 2D-Array Addition, Scaar Mutipication 5 A2, B2, C2, D1 1D and 2D-Array Mutipication 6 A2, B2, C2, D2 2D-Array Addition 7 A2, B2, C2, D2, A1 1D and 2D-Array Mutipication 8 A2, B2, C2, D2, E2, F2, A1, B1 1D and 2D-Array Mutipication 9 A2, B2, C2, D2, E2, F2 2D-Array Addition and Subtraction 10 A2, B2, D2, A1 1D and 2D-Array Mutipication 11 A2, B2, D2, A1 1D and 2D-Array Mutipication V. EXPERIMENTAL RESULTS Runs are performed on an Inte Pentium 4, 1.7GHz machine. Each experiment is repeated for 100 times. Success rate, µ, is the ratio of the number of runs returning the optima soution to the tota number of runs. Initia experiments are performed to test crossover operators and different types of GAs using Hessenberg reduction, yieding the experimenta resuts, summarized in Tabe IV. Runs are terminated for H1 and H2, as expained in Section III. A. Since the expected fitness is not known for D1, a run is terminated whenever the maximum number of generations is exceeded. TABLE IV. TEST RESULTS FOR H1 AND H2 USING DIFFERENT GA TYPES AND CROSSOVER (XOVER) OPERATORS, WHERE µ DENOTES SUCCESS RATE, ρ DENOTES AVERAGE NUMBER OF GENERATIONS PER RUN, σ DENOTES STANDARD DEVIATION. Data GA Xover µ (ρ σ) Type H1 SSGA 1PTX H1 SSGA 2PTX H1 SSGA UX H1 TGGA 1PTX H1 TGGA 2PTX H1 TGGA UX H2 SSGA 1PTX H2 SSGA 2PTX H2 SSGA UX H2 TGGA 1PTX H2 TGGA 2PTX , ,61 H2 TGGA UX

6 GA soves H1 much faster than H2, as expected. The soutions are found for H1 and H2 in tens of generations and in seconds as shown in Figure 8. Fitness Resuts supported our predictions about the soution, as expained in Section IV. B. 50% of the runs yied the same soution for D1, emphasizing the probem is computation intensive for the first two eves. Figure 10 shows this fina soution generated by SSGA with 1PTX. Therefore, the first two eves are mapped to 16 processors and the same decomposition patterns are used in order to minimize the communication. Other eves are computed on one processor. As the computation is sequentia on one processor, communication patterns do not have any effect Average Best Generation (a) [ 1] 16, 2 [ 2] 16, 2 [ 3] 1, - [ 4] 1, - [ 5] 1, - [ 6] 1, - [ 7] 1, - [ 8] 1, - [ 9] 1, - [ 10] 1, - [ 11] 1, Figure 10. A soution of D1 obtained by SSGA with 1PTX, where each entry is in the foowing form: [eve] number of processors, aignment id. Fitness Average Best (b) Generation Figure 9. Pot of average and best fitness at each generation averaged over 100 runs for soving H1 using (a) SSGA and (b) TGGA with 1PTX. Whie soving H2, GA sometimes gets stuck at the same oca optimum, arranging expected number of processors and reated aignments at the first two eves and faiing at the rest. Athough, these instances are not that many, the individuas tend to become aike. Parae code optimization probem can be formuized as a mutiobjective probem. Premature convergence is a common issue, whie appying Genetic Agorithms on some mutiobjective probems, as in [22]. As the size of the search space of a PCO probem instances increases and the number of goba optima in the search space decreases, we expect that this issue wi arise and become a burden. Comparing the number of evauations, SSGA performs sighty better than TGGA. For exampe, SSGA with 1PTX requires 136 (68.40*2) evauations on average, whie TGGA with 1PTX requires 171 (17.13*10) for soving H1. Furthermore, crossover operators estabish approximatey the same performance. For this reason, D1 is tested using ony SSGA with 1PTX. VI. CONCLUSIONS AND FUTURE WORK Our experiments show that SSGA is more promising than TGGA for soving the parae code optimization probem. SSGA with 1PTX successfuy generated a soution for Dongarra s benchmarks (D1). Speed is an important issue for this probem. If it takes too ong time to get optima settings for a code, users may prefer utiizing their intuition for paraeizing their codes. Note that success rates for the simper instances of the probem are much better than the success rate for D1. Furthermore, the initia indication of premature convergence is received. Hence, a diversification scheme can be added to improve the GA sover. There is a variety of approaches, such as crowding [23], or hyper mutation [24] for keeping the popuation diversified. Different hi cimbing techniques can aso be deveoped as a part of the GA sover. REFERENCES [1] J. Li, M. Chen, Index Domain Aignment: Minimizing Costs of Cross-referencing between Distributed Arrays, Third Symp. on the Frontiers of Massivey Parae Computation, pp , [2] J. Anderson, M. Lam, Goba Optimizations for Paraeism and Locaity on Scaabe Parae Machines, ACM SIGPLAN Conf. on Programming Language Design and Impementation, pp , [3] R. Bixby, K. Kennedy, U. Kremer, Automatic Data Layout using 0-1 Integer Programming, Proc. Int. Conf. on Parae Architectures and Compiation Techniques (PACT 94), 1994.

7 [4] K. Knobe, J. Lucas, G. Steee, Data Optimizations: Aocation of Arrays to Reduce Communication on SIMD machines, Journa of Parae and Distributed Computing, no. 8, , [5] D. Paermo, Compier Techniques for Optimizing Communication and Data Distribution for Distributed Memory Muticomputers, PhD Thesis, University of Iinois at Urbana-Champaign, [6] J. Ramanujan, P. Sadayappan, Compie-time Techniques for Data Distribution in Distributed Memory Machines, IEEE Tr. on Parae and Distributed Systems, pp , [7] T. Rauber, G. Runger, Deriving Array Distributions by Optimization Techniques, J. Supercomputing, vo. 15, pp , [8] S. Whoey, Automatic Data Mapping for Distributed Memory Parae Computers, Proc. of Int. Conf. on Supercomputing, [9] M. Wofe, High Performance Compiers for Parae Computers, Addison-Wesey, [10] M.E. Wof, M.S. Lam, A Data Locaity Optimizing Agorithm, Proc. ACM SIGPLAN Conf. on Programming Language Design and Impementation, pp , [11] K. Ikudome, G. Fox, A. Koawa, J. Fower, An Automatic and Symboic Paraeization System for Distributed Memory Parae Computers, Proc. of 5th Distributed Memory Computing Conference, pp , [12] E. Onbasiogu, L. Ozdamar, Optimization of Data Distribution and Processor Aocation Probem using Simuated Anneaing, The Journa of Supercomputing, vo.25, pp , [13] T. Chen, J. Sheu, Communication-Free Data Aocation Techniques for Paraeizing Compiers on Muticomputers, IEEE Tr. on Parae and Distributed Systems, pp , [14] C.H. Huang, P. Sadayappan, Communication-Free Hyperpane Partitioning of Nested Loops, J. Parae and Distributed Computing, vo. 19, pp , [15] M. Mace, Memory Storage Patterns in Parae Processing, Kuwer Academic, [16] E. Onbasiogu, Y. Paker, A Comparative Workoadbased Methodoogy for Performance Evauation of Parae Computers, Future Generation Computer Systems, vo. 12, pp , [17] J. Mohan, Performance of Parae Programs, PhD dissertation, Dept. of Computer Science, Carnegie- Meon University, [18] P.Lee, Efficient agorithms for Data Distribution on Distributed Memory Parae Computers, IEEE Tr. on Parae and Distributed Systems, vo. 8, no. 8, pp , [19] D. E. Godberg, Genetic Agorithms in Search, Optimization, and Machine Learning, Addison-Wesey, Reading (MA), [20] J. H. Hoand, Adaptation in Natura and Artificia Systems, Univ. Mich. Press, [21] J. Choi, J.J. Dongarra, D.W. Waker, The design of a parae, Dense Linear Agebra Software Library: Reduction to Hessenberg, Tridiagona and Bidiagona Form, Proceedings of the 2nd Workshop on Environments and Toos for Parae Scientific Computing, pp , [22] A. Akan, E. Ozcan, Memetic Agorithms for Timetabing, Proc. of IEEE Congress on Evoutionary Computation, pp , [23] K.A. De Jong., An Anaysis of the Behavior of a Cass of GeneticAdaptive Systems, PhD Thesis, University of Michigan, Ann Arbour, MI, [24] H.G. Cobb, An investigation into the use of hypermutation as an adaptive operator in Genetic Agorithms Having Continuous, Time-dependent Nonstationary Environment, NRL Memorandum Report 6760, 1990.