An experimental evaluation of a parallel genetic algorithm using MPI

Transcription

1 th Panhellenic Conference on Informatics An experimental evaluation of a parallel genetic algorithm using MPI E. Hadjikyriacou, N. Samaras, K. Margaritis Dept. of Applied Informatics University of Macedonia Thessaloniki, Greece {it06144, samaras, kmarg}@uom.gr Abstract The aim of this paper is to present an experimental evaluation of a parallel genetic algorithm using MPI. The performance of the algorithm is verified by computational experiments on a real world data set, ran in a cluster of workstations. MPI seems to be appropriate for these kind of experiments as the results are reliable and efficient. Index Terms Genetic Algorithms, Parallel Algorithms, Parallel Processing, Distributed Computing. I.INTRODUCTION GENETIC algorithms (GAs) are methods based on the Darwinian natural evolutionary principle [3], [4] and they are designed to find globally optimized solutions. GAs are categorized as global search heuristics and are generally able to find good solutions in reasonable amount of time. A genetic algorithm is an iterative method that evolves a population of elements, which are encoding strings representing a possible selection for a given real-world problem. GAs have been successfully applied to problems that are difficult to solve using conventional techniques or procedures. Common application areas are: scheduling problems [5], graph coloring problems [6], traveling salesman problem, optimization problems [7], network routing problems for circuit-switched networks, financial marketing, bio-informatics and genomics. GAs are easily parallelized algorithms. Parallel genetic algorithms (PGAs) are parallel implementations of GAs that can provide gain in terms of computational performance and scalability. There exist two major kinds of parallelism in genetic algorithms: (1) in the computation of the fitness functions of individuals and (2) in the application of genetic operators (selection, crossover and mutation). An overview of theoretical advances, computing issues, applications and future trends in parallel genetic algorithms can be found in [8], [9]. A more detailed discussion of parallel genetic algorithms can be found in [12]. A general framework for studying and analyzing PGAs was proposed by Alba and Troya [9]. Recently, GAs can be applied for supervised and unsupervised data mining sessions. For data mining purposes we use population individuals as elements defined by attributes and values. These elements or individuals represent candidate production rules. In this paper an experimental evaluation of a parallel genetic algorithm using MPI on a real world data set is presented to establish the practical value of such an implementation. We used the control-parallel singlepopulation model as the parallelization strategy for our implementation. Another approach is the control-parallel distributed-population [13]. To speed-up the whole process, we can split the initial population into several partitions. Each partition is assigned to a different processor which performs a sequential genetic algorithm. Genetic operators (crossover and mutation) involve only individuals within the same partition. The rest of the paper is organized as follows. Following this introduction, in section II we give the fitness function and we briefly present the sequential and the parallel genetic algorithm used in our experimental evaluation. In Sections III and IV we present implementation techniques and computational results. Finally, in Section V we conclude and discuss future work. II.ALGORITHM DESCRIPTION Each iteration of the genetic algorithm is called generation. The current individuals or elements of the population are evaluated by a fitness function. This function measures the quality of the current solution. The fittest individuals tend to reproduce and survive to the next generation. Genetic operators (crossover and mutation) consists the basic search procedure of the GA. Crossover operator takes two individuals and produces two new individuals. Mutation operator takes one individual and produces a single new individual. The most frequently used termination criterion of the GA is a fixed number of generations. The output of the GA is the survival of the fittest population individuals or equivalent the high quality solutions [14]. A. Fitness function One of the most significant parts of a genetic algorithm is the fitness function. Generally we want an attribute, which will be easily separated to groups, two if it is possible. In the encoding phase, the most important attribute which occurs has only two possible values which only one can be inserted at that position. Based on that value, the algorithm then checks all individuals that have the same attribute and then checks for all the other attributes for similarities. If there is a similarity between those elements, then the variable for similarities N increases by one. Similar happens with the differences. A formal description of fitness function is given below: For a single population individual (or element) E the fitness function is defined as follows: /09 $ IEEE DOI /PCI

2 Step 1: Let N be the number of matches of the input attribute values of E within training elements from its own class. Step 2: Let M be the number of input attributes whose value matches to all training elements from the competing classes. Step 3: Add 1 to M (avoid division by zero). Step 4: Divide N by M. B. Sequential algorithm The genetic algorithm starts with an initial population and generates new population individuals by a crossover operation. It produces the fittest population elements that survive and mutate until a population of successful individuals develops. The algorithm stores the whole population for further computations in an mxn population matrix. Each row of the matrix defines a population individual. The first step is to encode the initial population. For example, if an attribute value of a population individual can take 4 different values then those values will be encoded from 0 to 3. The second step is to compute the fitness value for each individual of our initial population, as explained above. If this fitness value is greater or equal to one then the specific element remains in. This means that it is not candidate for the crossover and mutation operations. Otherwise, perform the crossover and mutation operations in the population individual that has not pass the fitness criteria. The crossover point is selected randomly. This happens in a loop until t = 0 or until a large number of iterations is passed, which means that no element has a fitness value less than 1. The result of a supervised genetic learning process is a set of population individuals that best represents the training data set. The termination criterion of the algorithm is a fixed number of iterations (L). Finally, merge sort, sorts the fitness values and the first x values are candidate for crossover and mutation. The next population is now reconstructed. Algorithm Sequential Genetic Input: a population in a form of an mxn matrix Output: the fittest population (mxn matrix) Encode all the attribute values over an alphabet that is devised i:=0 while(i < L) for each individual compute Step 1 and Step 2 of the fitness function compute the fitness function F = N / (M + 1) if F < 1, put individual in a throw matrix if ( t == 0) break for each two sequential individuals in throw matrix crossover the first half attributes of two individuals do mutation on a random 2 nd half attribute of the individual end for i:=i+1 end while sort the fitness values using merge sort select the first x values of the sorted fitness values send x individuals for crossover and mutation return (population) C. Parallel algorithm The parallel algorithm was obviously based on the sequential algorithm. Initially, after the classification phase, the mxn population matrix is portioned into k workers. Each worker takes a subset of the entire population. Also, the computational part of this procedure finishes when t = 0 or when a large number of iterations (L) has already be done. After the computation above, each worker sends back its results. Now the master has a new population. The master then sorts the fitness values using the merge sort algorithm. The first x values of the sorted fitness values are now selected and the master decides whether to send for an iterative crossover and mutation procedure, as described above or perform the crossover and mutation on his own. Finally, if the master decides to send the x values for the crossover and mutation to the workers, he waits to receive their results and reconstruct the new population. In case that the master decides to perform the computation of its own, the final population is constructed automatically. Algorithm Parallel Genetic Input: a population in a form of an mxn matrix Output: the fittest population (mxn matrix) Encode all the attribute values over an alphabet that is devised If (master) partition matrix into k workers receive all the population individuals and construct the new matrix receive the fitness values for each worker sort the fitness values using Merge Sort select the first x values of the sorted fitness values If(x is small) do the same loop, performing crossover and mutation else send a smaller sample of those x elements to the workers receive the new sample return the new population end (master) if (worker) Receive a partition of the matrix while(i < L) for each individual compute Step 1 and Step 2 of the fitness function compute the fitness function F = N / (M + 1) if F < 1, put individual in a throw matrix if ( t == 0) break for each two sequential individuals in throw matrix crossover the first half attributes of two individuals do mutation on a random 2 nd half attribute of the individual end for end while Send individuals back to master Send fitness values back to master If (a small partition of population matrix was received) Perform crossover and mutation Send the small partition of population matrix back to master end (worker) III.IMPLEMENTATION ISSUES Our computational experiments were performed on a university's cluster [2]. There is a total of 17 Intel(R) Xeon(TM) CPU at 3.00GHz and 2MB cache, 4GB ram each, 512MB swap file, connected via 1 Gigabit Ethernet network. Each machine of the cluster has Scientific Linux as an operating system. The compiler used in our code was gcc version and MPICH 2. In our computational study we used a control-parallel single-population approach. All the individuals of the training data set are partitioned into k exclusive data subsets. The value k represents k different processors. Each processor computes the fitness function for 76

3 all individuals of the specific data subset. The data set used for the empirical study was a real one and it was taken from the UCI Machine Learning Repository [1], [11]. Each record is an example of a hand consisting of five playing cards drawn from a standard deck of 52. Each card is described using two attributes (suit and rank). This data set has training individuals, testing individuals and no missing values. Table 1 contains the attributes information and the possible values an attribute can get. Input Attributes name S1, S2, S3, S4, S5 C1, C2, C3, C4, C5 TABLE I ATTRIBUTE INFORMATION Category Mapping Ordinal (1-4) 1. Hearts 2. Spades 3. Diamonds 4. Clubs Numerical (1-13) Ace, 2, 3,,Queen, King The output attribute is the class Poker Hand. It is of type ordinal and it takes 10 different values. Below we show the meaning of these 10 different values. 0: Nothing in hand; not a recognized poker hand 1: One pair; one pair of equal ranks within five cards 2: Two pairs; two pairs of equal ranks within five cards 3: Three of a kind; three equal ranks within five cards 4: Straight; five cards, sequentially ranked with no gaps 5: Flush; five cards with the same suit 6: Full house; pair + different rank three of a kind 7: Four of a kind; four equal ranks within five cards 8: Straight flush; straight + flush 9: Royal flush; {Ace, King, Queen, Jack, Ten} + flush The programming language which was selected for this implementation was C. Thus all we had to do was to program the sequential algorithm and then extend it to succeed the parallelization. The first step was to create a function to read N lines from the file and create the population data set. Being more specific the master reads the file line by line and then uses a function to separate values from commas and encodes them. After this, the master stores the values to the population matrix. The second step was to partition the population dynamically to subsets and send them to the workers using MPI_Send. Each worker gets its subset using MPI_Recv and then he computes fitness function for each individual. For the computation of fitness values, a separate function was created using the subset as in input and as an output were the fitness values. Therefore a function for fitness checking was created, having the set and the number of individuals as input. As an output were the pass matrix for those individuals, which had fitness values greater or equal to 1, a throw matrix for all the others, and also a variable t, which defines how many individuals must crossover and mutate. Functions for crossover and for mutation where created as well. Both techniques work only if the fitness value of an individual is less than 1. The crossover technique works in pairs. There must be at least 2 elements (t 2) to work. In such situation it takes each pair and crossovers their first half attributes from one to other. If t = 0 then this procedure does not execute, because none of the individuals have fitness less than 1. The mutation procedure takes care of the second half of attributes. For a random second half attribute the algorithm finds a random value from a proper range of values and replaces it. This happens for all individuals with fitness value less than 1, although only one random attribute is replaced from the second half of attributes. The fitness check, crossover and mutation phase are executed in a for loop as I < L where L is a number, big enough for our situation (=300). This L number defines that if t never becomes 0 then, at the end of the loop, the 300 th generation will be computed. In that situation, the fittest set is found and the algorithm terminates. After all these, each worker sends back to the master, it's subset and fitness matrix. Master now creates the whole population and the whole fitness matrix. Furthermore master now uses Merge Sort to sort the fitness values and their positions, which refer to the population. After the sorting phase, X values are selected with the smallest fitness values. After that, the master decides whether further computation must be done by him, or by sending them to the workers. If the master is decided to continue the computation, then the same for loop as above is executed. Else if the x sampling for further computation is big enough for executing it locally, then it is partitioned and sent to the workers. The workers in this situation compute their own set (the for loop as described above) and then return their set back. Then the master receives the population and prints it to the screen. Now for 1 CPU, a separate source code was developed, having all functionality as described above, but without the MPI Send and Receive calls. The only MPI commands used were those for time estimation to calculate the computational time and total time. IV.COMPUTATIONAL RESULTS When executing the algorithms it was possible to take an online visualization of the cluster's situation [2]. Using this monitoring tool, the user can watch how many are available, which of them are online, their average load, etc. Figures 1 to 4 show the computational performance of various population sizes over various numbers of. The population sizes are 5000, 10000, and individuals. Each data set was executed 3 times. The times reported are the average computation time, the average communication time and the average total time. 77

4 Population Size: ,205 0,022 29, ,386 0,029 7, ,255 0,043 3, ,541 0,041 2, ,467 0,043 1, ,139 0,028 1, ,947 0,035 0, ,868 0,034 0,903 Population size: Fig. 1. Average total time (in seconds) for various numbers of of population size 5000 individuals. As the computational results show, communication time was too small compared to the computational time. Total time is the sum of the computational time and the communication time. First of all it is well understandable that for 1 CPU, as a separate source code was developed, communication time does not exist, because the master does not send anything to any worker. He does all the computation on his own. It had to be done using this pattern, for having accurate results and the reason was that the same computer architecture should be used. If the code for execution on 1 CPU was executed on a machine with a newer CPU the times should be quite different. As a result this should be executed on the head of the cluster called as master Population size: Fig. 2. Average total time (in seconds) for various numbers of of population size individuals. Looking at Table 2, comparing total time for 1 CPU and total time for 35, we can understand the big difference in time, and how many times faster the results occur with parallelization. To be more specific with 35 and a population size of 5000 individuals, we can achieve a speedup of 512 times (total time cpu[1] /total time cpu[35] ). TABLE2 POPULATION SIZE 5000: TIME (IN SECONDS) computation time communication time total time 1 462, , ,749 0, , Fig. 3. Average total time (in seconds) for various numbers of of population size individuals. Figure 1 show this benchmark even more. Someone may ask, why then not dividing the population to even more to achieve even better times? Well the answer to this, is that as the received partition from the worker keeps getting smaller, the fitness function is not as accurate as in bigger partitions. And this is because the fitness value is calculated by scanning the whole sample. Now, the cluster where the experiments took place consists of 17 dual-core, how is it possible to execute an experiment declaring 35? MPI actually starts, using round robin, as many processes (N) as declared by the command mpirun -np N divided properly to each CPU. In the situation of N=35, the workers one by one will create a worker process until the total number of MPI processes for all workers is 35. Each process then will work on its subset. Figure 2 also shows something similar. For the 2 nd experiment, the sample was doubled, from 5000 to Even the result with 35 here took 4 times more, compared with the results of the 5000 sample. Time for 1 CPU was also increased, almost to 4 times. Another thing to admire is that when the number of decreases, times don't decrease in the same frequency. From 8 to 4, total time didn't increased by 2 times. From seconds rose up to , almost 4 times bigger. So when the number of was divided to 2, total time, as well as computational time, was quadrupled. Something similar happened from 4 to 2. Communication time altered in a very slow frequency, as the communication between master and workers was not a procedure, which took a long time. The most significant factor to this is the 1Gbit Ethernet network, which sits below MPI. Although MPI uses its own protocol for communication, the base for everything is the bus Ethernet 78

5 topology. As the sample from the data set was getting bigger the communication time altered a little bit. Just to refer that for a sample of 5000 for 35 the communication time was 0.03 secs and for a population size of for 35 was 2.37 secs Population size: [8] D. Lim, Y-S. Ong, Y. Jin, B. Sendhoff and B-S. Lee, Efficient hierarchical parallel genetic algorithms using Grid computing, Future Generation Computer Systems, vol. 23, 2007, pp [9] Z. Konfrst, Parallel genetic algorithms: Advances, Computing trends, Applications and Perspectives, Proc. of the 18 th International Parallel and Distributed Processing Symposium (IPDPS 04), [10] E. Alba and J. M. Troya, Analyzing synchronous and asynchronous parallel distributed genetic algorithms, Future Generation Computer Systems, vol. 17(4), 2001, pp [11] R. Cattral, F. Oppacher, and D. Deugo, Evolutionary Data Mining with Automatic Rule Generalization, Recent Advances in Computers, Computing and Communications, 2002, pp [12] E. Cantú-Paz, A survey of parallel genetic algorithms, IllGAL Report 97003, The University of Illinois, 1997, Available on-line at: ftp://ftpilligal.ge.uiuc.edu/pub/papers/illigals/97003.ps.z. [13] I. N. Flockhart and N. J. Radcliffe, A genetic algorithm-based approach to data mining, Proc. of the 2 nd International Conference on Knowledge Discovery and Data Mining, 1996, pp [14] Z. Michalewqicz, Genetic algorithms + Data Structures= Evolution programs. 3 rd Ed., Springer-Verlag, Fig. 4. Average total time (in seconds) for various numbers of of population size individuals. V.CONCLUSION In this paper an experimental evaluation of a parallel genetic algorithm using MPI on a real world data set is presented to establish the practical value of such an implementation. High-performance computing still seems to be the discipline of computer science to find out solutions on such kind of problems. Using several real world data sets, scientists may come to important results. MPI seems to be appropriate for this kind of experiments as the results are reliable and efficient. Additionally, genetic algorithms are proved to be problems where parallel techniques can take place, and our results assure the above statement. These remarkable results are the consequence of the proper fitness function, which in our situation seems to be good enough. The importance of these results has to do with the data set. By using real data sets, an implementation such as this, gains the research interest, as it refers to real-world problems, and how solutions to these may occur. Therefore this can be a point of view for many scientists to contribute to similar projects and combine their knowledge. REFERENCES [1] UCI Machine Learning Repository. Available: [2] Available: [3] J. H. Holland, Adaptation in natural and artificial system, University of Michigan Press, Ann Arbor, MI, [4] S. Forrest, Genetic algorithms: Principles of natural selection applied to computation, Science, vol. 261, 1993, pp [5] W. Bozejko and M. Wodecki, Parallel genetic algorithm for the flow shop scheduling problem, LNCS, vol. 3019, 2004, pp [6] Z. Kokosinski, M. Kolodziej and K. Kwarciany, Parallel genetic algorithm for graph coloring problem, LNCS, vol. 3036, 2004, pp [7] D. E. Goldberg, Genetic algorithms in search, optimization and machine learning, Addison-Wesley, Reading, MA,