PODC: Paradigm-oriented distributed computing

Transcription

1 J. Paallel Distib. Comput. ( ).elsevie.com/locate/jpdc PODC: Paadigm-oiented distibuted computing Haiong Kuang,Lubomi F. Bic,Michael B. Dillencout Infomation and Compute Science, Univesity of Califonia, Ivine, CA , USA Received 4 Febuay 2002; eceived in evised fom 27 Octobe 2004; accepted 26 Novembe 2004 Abstact We descibe an envionment fo distibuted computing that uses the concept of ell-knon paadigms. The main advantage of paadigmoiented distibuted computing (PODC) is that the use only needs to specify application-specific sequential code,hile the undelying infastuctue takes cae of the paallelization and distibution. The main featues of the poposed appoach,called PODC,ae the folloing: (1) It is intended fo loosely coupled netok envionments,not specialized multipocessos; (2) it is based on an infastuctue of mobile agents; (3) it suppots pogamming in C,athe than a functional o special-pupose language,and (4) it povides an inteactive gaphics inteface though hich pogams ae constucted,invoked,and monitoed. We discuss five paadigms pesently suppoted in PODC: the bag-of-tasks,banch-and-bound seach,genetic pogamming,finite diffeence,and individual-based simulation. We demonstate thei use,implementation,and pefomance ithin the mobile agent-based PODC envionment Elsevie Inc. All ights eseved. Keyods: Distibuted computing; Paadigm-oiented computing; Mobile agents; Bag of tasks; Banch and bound; Genetic pogamming; Finite diffeence; Individual-based simulation 1. Intoduction Inceasingly poeful okstations and PCs,inteconnected though vaious netoks,continue to polifeate thoughout the old. Most ae geatly undeutilized and thus epesent a significant computational esouce,hich could be tapped fo unning applications equiing lage amounts of computations. Unfotunately,developing distibuted applications is significantly moe difficult than developing sequential applications. To make the distibuted computational esouces available to a boade class of uses, a numbe of tools and envionments have been poposed, hich attempt to simplify the development of distibuted pogams. One of the pioneeing ideas as to povide pogamming skeletons [3],hich embody specific ell-knon paadigms,such as divide-and-conque o a bag-of-tasks. Coesponding autho. addesses: hkuang@ics.uci.edu (H. Kuang), bic@ics.uci.edu (L.F. Bic), dillenco@ics.uci.edu (M.B. Dillencout). Each skeleton is a high-level template descibing the essential coodination stuctue of the algoithm. Only the poblem-specific data stuctues and functions need to be supplied by the use,and ae used as paametes fo the given pogamming skeleton. This idea has been exploed mostly in the context of sequential pogam development,but can also be applied to distibuted computing [2,4,18,20,22]. One of the main limitations of pogamming skeletons is that they ae not easily potable,but need to be eimplemented fo each ne achitectue. This makes the appoach unsuitable fo exploiting available clustes of okstations and PCs,since thei numbes,thei individual chaacteistics,and thei netok topology ae knon only at untime and may even change dynamically ith the changing availability of individual nodes and/o links. To addess these poblems and make paadigm-based computing feasible in dynamic heteogeneous computing envionments,e exploit the benefits of a mobile agents infastuctue. The autonomous migation ability of agents makes them capable of utilizing a dynamically changing netok. Thei inheent potability allos them to handle /$ - see font matte 2005 Elsevie Inc. All ights eseved. doi: /j.jpdc

2 2 H. Kuang et al. / J. Paallel Distib. Comput. ( ) the distibution of tasks in a heteogeneous envionment in a tanspaent manne. Agent mobility can also be exploited fo load balancing. Because of these featues,mobile agents lend themselves natually to paadigm-oiented distibuted computing. Using a mobile-agent system called MESSENGERS [5,6], hich as developed by ou goup in pevious eseach effots,e have developed an envionment fo paadigmoiented distibuted computing,called PODC. This envionment cuently suppots five common computing paadigms: (1) bag-of-tasks,(2) banch-and-bound,(3) genetic pogamming,(4) finite diffeence,and (5) individual-based simulation. Afte pesenting these paadigms in detail in Section 2, Section 3 illustates the use s point of vie hen employing the paadigms and pesents the oveall system achitectue. Section 4 pesents the undelying implementation using MESSENGERS. Pefomance evaluations ae discussed in Section 5,folloed by geneal conclusions. 2. Paadigms 2.1. Choice of paadigms Cuently PODC suppots five paadigms: bag-of-tasks, banch-and-bound seach,genetic pogamming,finite diffeence,and individual-based simulation. Thee ae thee main easons fo choosing these paadigms: (1) many applications that fit those paadigms ae highly computationally intensive and thus can benefit fom multiple computes to impove thei pefomance though paallelism; (2) an application folloing these paadigms can easily be divided into lage numbes of coase-gained task and (3) thee is limited communication among the tasks. These thee popeties make the chosen paadigms suitable fo execution in a netok envionment,hee the cost of communication is high,and must be offset by lage numbes of coase-gained tasks. We have investigated thee categoies of paadigms,classified by thei inte-task communications. The fist categoy, hich equies no communication at all beteen unning tasks,is typified by the bag-of-tasks paadigm in hich tasks ae highly independent and do not shae any infomation. The second categoy equies only non-essential communications,hich do not affect the coectness although they may have geat impact on the pefomance. The paadigms in this categoy ae banch-and-bound and genetic pogamming. In these paadigms,the tasks ae highly asynchonous and self-contained,and they exchange infomation only fo the pupose of optimization. The thid categoy equies essential communications,meaning that they ae necessay fo coectness. Examples of this type of paadigms ae iteative gid-based paadigms typified by the finite diffeence and the individual-based simulation,in hich nea-neighbo communications must be synchonized at each time step. The difficulty of poviding a distibuted implementation ith 1. BagOfTasks(){ 2. D = InitData(); 3. Until ( Temination_condition ){ 4. T = GenNextTask( D ); 5. add T to the bag of tasks BT; 6. } 7. hile (BT is not empty) { 8. T = emove a task fom BT; 9. R = Compute ( T ); 10. Wite Result ( outfile, R ); 11. }} Fig. 1. The bag-of-tasks paadigm specification. easonable pefomance inceases ith each type of communications. To make the paadigm-oiented appoach to distibuted computing pactical,e must be able to clealy diffeentiate beteen application-specific pogam components, hich must be povided by the use,and paadigm-specific components,hich ae povided by the system. In the emainde of this section,e descibe each paadigm in pseudo-code that makes it clea hich application-specific components ae equied. The identifies in bold face indicate the application-specific functions,hile the identifies in italic indicate the vaiables hose type infomation ae equied Bag-of-tasks paadigm The bag-of-tasks paadigm applies to the situation hen the same function is to be executed a lage numbe of times fo a ange of diffeent paametes. Applying the function to a set of paametes constitutes a task,and the collection of all tasks to be solved is called the bag of tasks. Fig. 1 shos the stuctue of the paadigm. The poblem data and task-geneation state ae initialized at line 2,and the bag of tasks (BT) is ceated by the loop at lines 3 6. The temination condition eithe epesents a fixed numbe of iteations o is given implicitly by eading input values fom a file until the end of file. The hile loop at lines 7 11 epesents the actual computation,hich is epeated until the bag of tasks is empty. Multiple okes may execute the loop independently. All okes have shaed access to the task bag and the output data. Each oke epeatedly emoves a task (line 8),solves it by applying the main function Compute to it (line 9),and ites the esult into a file (line 10) Banch-and-bound paadigm Banch-and-bound is applicable to vaious combinatoial optimization poblems,and is geneally applied hen the goal is to find the exact optimum. The banch-and-bound

3 H. Kuang et al. / J. Paallel Distib. Comput. ( ) 3 1. BanchAndBound(){ 2. B = + ; 3. D = GenPoblemData(); 4. R = GenRootNode( D ); 5. if ( an impove dinitial bound is to be used ) { 6. S = GenInitSol( D ); 7. B = GenBound( S,D ); 8. } 9. L = { R }; 10. hile( L is not empty ) { 11. N = SelectNode ( L ); 12. SN = NextBanch ( N, D ); 13. if( SN is Nil ) L = L - {N}; 14. else { 15. SB = GenBound( SN,D ); 16. if (!IsSol( SN ) ) 17. if ( SB < B ) L = L + {SN}; 18. else if( SB < B ) { 19. B = SB; 20. S = SN; 21. }} } 22. WiteSol ( S ); 23. } Fig. 2. The banch-and-bound paadigm specification. paadigm dynamically constucts and seaches a tee,in hich the oot node epesents the oiginal poblem to be solved,each descendant node epesents a subpoblem of the oiginal poblem,and each leaf epesents a feasible solution. To speed up the seach,a subtee is puned if it can be detemined that it ill not yield a solution bette than the best cuently knon solution. Fig. 2 pesents the basic stuctue of the banch-and-bound paadigm. Without loss of geneality,e assume the goal is to find the minimum value. We assume D is the initial poblem data,hich is passed as a paamete to cetain functions; R is the initial oot node; L is a pool of tee nodes that have yet to be exploed; S is the cuent best solution; and B is the bounding value coesponding to S. The algoithm stats by setting the bound B to infinity (line 2),initializing the poblem data (line 3),and geneating the oot node R of the banch-and-bound tee (line 4). Lines 5 8 ae optional: the stating bound B may be impoved by geneating an initial feasible solution S and the coesponding bound B. The pool L of poblems to be solved is initially set to R (line 9). Lines constitute the main loop. A node N is selected fom the pool L (line 11) and one of the N s subnodes is geneated (line 12). If all subnodes of N have been exploed,then the node N has been completely solved and so is emoved fom the list (line 13). Otheise, 1. Genetic() { 2. D = GenPoblemData(); 3. P = GenInitPop( S, D ); 4. until (Temination_condition) 5. P = CeateNextGen( P,D ); 6. I = BestIndividual( P ); 7. WiteSol( I ); 8. } Fig. 3. The genetic pogamming paadigm specification. a ne bound is computed fo the subpoblem. Lines then accomplish the puning. Afte the pool of subpoblems to be solved is dained,the solution to the poblem is itten (line 22) and the pogam teminates Genetic pogamming paadigm The genetic pogamming paadigm also solves optimization poblems,using the Dainian pinciples of suvival and epoduction of the fittest and genetic inheitance [8]. Unlike banch-and-bound,genetic pogamming is geneally applied to find a good but not necessaily optimal solution. Fig. 3 shos the basic stuctue of a sequential genetic pogamming paadigm. The static poblem data is geneated and an initial population P of size S (lines 2 and 3) is ceated. The hile loop on lines 4 and 5 epesents the evolution pocess. At each iteation,a ne geneation is ceated by applying genetic opeations such as cossove,mutation, and epoduction. The temination condition at line 4 is typically based eithe on the numbe of iteations completed o the quality of the best solution obtained. In ou distibuted vesion of the genetic pogamming paadigm,the population is divided into multiple subpopulations that evolve independently and occasionally exchange individuals. This scheme equies specifying to othe components: SelectEmigant,hich selects an emigant fom a population; and PocessImmigant,hich decides hethe an aiving immigant should be discaded o kept and,in the latte case,selects the individual to be eplaced by the immigant. Seveal contolling paametes ae also equied: the numbe of geneations to be ceated beteen sending out a ave of emigants (EmigationInteval) and the numbe of emigants in a ave (EmigationRate) Finite diffeence paadigm The finite diffeence method [23] is used to solve diffeential equations. The method uses a discete d-dimensional gid of element locations and computes a value u x,t at each gid location x and fo a egula sequence of times t. The value of u x,t+ t is a function of u x,t and the values of u at the neighbos of x at time t.

4 4 H. Kuang et al. / J. Paallel Distib. Comput. ( ) 1. FDM () { 2. double vals1[xsize][ysize], vals2[size][ysize]; 3. double *oldvals, *nevals; 4. double X = XLEN / XSIZE, Y = YLEN/YSIZE; 5. // initialization 6. fo ( evey element X,Y ) 7. vals1[x][y] = Init( X, Y, XSIZE, YSIZE, X, Y ); 8. oldvals = vals1; nevals = vals2; 9. // iteative computation 10. until ( Teminatio_ncondition) { 11. fo (evey element X,Y ) 12. nevals[x][y] = Compute( X, Y, oldvals, X, Y, t ); 13. nevals oldvals; 14. } 15. // output 16. fo ( evey element X,Y ) 17. WiteResult( outfile, X, Y, oldvals ); 18. } Fig. 4. The finite diffeence paadigm specification. Fig. 4 shos the stuctue of the finite diffeence paadigm fo a 2D poblem. The algoithm stats by initializing the values of all elements (lines 6 and 7) and pointes to element buffes (line 8). It then epeatedly computes the ne values of all elements fo each time step until the temination condition is satisfied (lines 10 14). Finally,element values ae itten to the output file (lines 16 and 17) Individual-based simulation paadigm Individual-based simulation pogams ae used to simulate the behavios of a collection of entities such as the movement of paticles [9],the schooling behavio of fish o bids [12,21],and the evolution of an ecology envionment [10]. The simulated entities move in a specified space ove a peiod of time. At each time step,an entity decides its behavio by inteacting ith its neaby envionment and suounding entities. Typically,an entity has an associated adius of visibility and its behavio is affected only by entities ithin this adius of it. Vaiants of the paadigm aise depending on ho the collision poblem is handled. This poblem aises because entities shae esouces (including space),and hence must contend fo esouces. Fo example,to entities may move to the same position,thei paths may intesect,o they may decide to eat the same food. Ou individual-based simulation paadigm suppots to fameoks of collision detection and esolution: delayed state update and immediate state update. In the delayed state update method,each entity makes a tentative decision based on the old state of the pevious time step. Afte all the entities ae finished,possible collisions ae detected and esolved. Thus,the simulation pogam needs to keep to sets of states,the old state and the cuent state. Because the stuctue of individual-based simulation pogams using the delayed state update method is vey simila to that of finite diffeent pogams,e omit its details hee. In the immediate state update method,each entity makes a collision-fee decision based on the cuent state. Afte an entity s on state and its suounding envionment ae updated,the old states ae thon aay. Because an entity uses the most cuent states,collision detection and esolution is much easie to conduct. Collision detection and esolution can be combined ith the pocess of state update,so that a global collision detection and esolution stage is no longe needed. Fig. 5 shos the stuctue of the individual-based simulation paadigm using the immediate state update method. It solves a 2D poblem. Fist,the envionment and all the entities ae initialized (lines 8 16). The main loop (lines 16 29) simulates the behavio of entities fo a fixed numbe of iteations. At each iteation,all the entities update thei states sepaately (lines 20 25). The pogam keeps only the most cuent state. Befoe an entity s state is updated,it is dequeued (line 21). The entity s state is modified in place and the envionment is updated incementally (line 23). An entity can detect and esolve possible collisions hile updating its state. Any updating of the envionment that is independent of the effect of the entities is pefomed on lines Notice that the envionment may get updated in to places. The updates on line 23 ae caused by entities. The updates on lines eflect spontaneous changes to the envionment. 3. PODC system PODC is a gaphics-based envionment that suppots paadigm-oiented distibuted pogamming. To develop an application and execute it in a distibuted manne,the use must fist choose one of the paadigms suppoted by the system,and develop the necessay application-specific functions that pefom the coesponding domain computations. These functions ae all itten using conventional unesticted C,but must obey cetain conventions egading input and output. The use is then guided though a seies of fom-based queies,as illustated in Section 3.1,duing hich s/he supplies the necessay infomation. Using this infomation the system automatically geneates and compiles distibuted code,stats the MESSENGERS system on the set of available nodes,and monitos the execution of the application. In this section,e fist pesent ho uses inteact ith PODC,then e sho the achitectue of the PODC system Poblem specification and the use inteface Uses inteact ith PODC though a gaphical inteface. Thee ae to phases to the inteaction. In the fist phase,the

5 H. Kuang et al. / J. Paallel Distib. Comput. ( ) 5 1. IBS() { 2. Env env[xsize][ysize]; 3. School entitygoup[xsize][ysize]; 4. School *entitylist; 5. Entity entities [NUM_OF_ENTITIES]; 6. Entity *cuentity; 7. double X=XLEN / XSIZE, Y=YLEN/YSIZE; 8. // initialize fish school and envionment 9. fo (all the gid cell i,j ) { 10. neenv( &env[i][j], i, j, XSIZE, YSIZE, X, Y ); 11. neentitylist ( &entitygoup[i][j] ); 12. } 13. fo( i=0; i<numofentities; i++ ) { 14. ne Entity(&entities[i], i); 15. add entity i to the coesponding cuent gid cell; 16. } 17. // iteative simulation 18. hile( t<max_time_steps ) { 19. t++; 20. fo (each gid cell i,j ) { 21. hile( (cuentity = popentity ( &entitygoup[i][j] ))!=NULL ) { 22. entitylist = getneighbos ( cuentity, entitygoup, R); 23. entitylist, env = updateentity (t, cuentity, env, entitylist, X, Y, t); 24. add each entity in entitylist to entitygoup; 25. }} 26. fo ( each gid cell i,j ) { 27. entitylist = updateenv( t,&env[i][j], i, j, X, Y, t); 28. add each entity in entity List to entitygoup; 29. }} } Fig. 5. The individual-based simulation paadigm specification. use specifies and stats the application though a submission indo. In this indo,the use specifies applicationspecific pogam components. In the second phase,the use can monito and inteact ith the unning application though a feedback indo,hich shos the cuent status of the system. The details of the submission and feedback indos depend on the specific paadigm. In this section, e illustate the pocess of specification and inteaction in the context of the banch-and-bound paadigm. Fig. 6 shos the gaphic inteface used to specify a banch-and-bound poblem. At the top of the indo,the use povides the location of the sequential souce pogams, heade file(s),pogam file(s),and the diection of optimization. Next,the use specifies the application-specific functions and data stuctues,as identified in bold font in Section 2.3. The pocedue to geneate an initial feasible solution is optional; if it is not povided,the puning bound is set to a default value of + fo a minimization poblem o fo a maximization poblem. Finally,the use can specify a ecommended numbe of machines to use in the computation. Fig. 7 shos the feedback indo of a banch-and-bound application. An icon at the vey top indicates that the application is still unning. Belo that a gaph shos the histoy of geneated optimal solutions: the x-coodinate epesents elapsed time,and the y-coodinate epesents the best solution achieved at the coesponding point in time. The feedback indo also displays the numbe of machines cuently being used by the distibuted application,a ba shoing the estimated faction of the total ok that has been cuently completed (computed by estimating the numbe of leaf nodes in the banch-and-bound tee that have cuently eithe been visited o puned aay),and eo/status messages System achitectue PODC has a 3-tie client/seve achitectue. The top level is the client,hich inteacts ith the uses though a

6 6 H. Kuang et al. / J. Paallel Distib. Comput. ( ) Fig. 6. Submission indo fo the banch-and-bound paadigm. gaphics inteface and pemits the submission and monitoing of applications. The middle level is the seve,hich builds and supevises the execution of the application. The thid level epesents the undelying MESSENGERS system,in hich the application is implemented as multiple autonomous mobile agents,unning on a given compute netok [14]. Fig. 8 illustates the diffeent stages of developing and unning an application in PODC. Fig. 8(a) shos the submission of an application though a submission indo in the client. By selecting a paadigm fom a list,the use implicitly specifies the paallel algoithm fo the geneated distibuted pogam. The poblem-specific infomation is then used to instantiate the pogam skeleton. When the use instucts the system to stat the application,the seve tanspaently chooses a set of okstations and automatically geneates and compiles the distibuted application codes. The system then stats up the MESSENGERS system on the set of okstations and begins to execute the distibuted pogams. The application is implemented using multiple Messenges,each of hich hops ithin a netok of okstations hee it oks on the task. The undelying MESSENGERS system pefoms the necessay distibution,load balancing,and fault-ecovey automatically and tanspaently [7]. Duing execution of the distibuted application,the undelying netok might change,fo example,as a esults of failue o a change in the system okload. The MESSENGERS can seamlessly adapt to such changes. Fig. 7. Feedback indo fo the banch-and-bound paadigm.

7 H. Kuang et al. / J. Paallel Distib. Comput. ( ) 7 g (a) (b) Fig. 9. Logical netok fo the bag-of-tasks paadigm. Fig. 8. System achitectue. Fig. 8(b) shos the system hile the application is unning. Duing this time,the use is able to monito the pogess using the feedback indo. The feedback infomation is geneated by the distibuted application and sent to the seve,hich passes it to the client and to be displayed on the feedback indo. The seve also saves this infomation in a histoy log fo late pocessing. The use is not equied to emain on-line hile the application is unning. Fig. 8(c) shos the situation hee the system continues opeating hile the use is off-line. Duing this time,the seve continues eceiving feedback infomation fom the distibuted application and ecoding it in the histoy log. A use can late econnect to the seve and check the status of the unning application; eceating the on-line monitoing situation shon in Fig. 8(b). When the application teminates,the MESSENGERS system is shut don and the use is sent a notification via s as illustated in Fig. 8(d). At that time,the use is able to etieve the esults though a Web bose o ftp tool. 4. Distibuted implementation of paadigms 4.1. Bag-of-tasks paadigm The logical netok used to implement the bag-of-tasks paadigm using MESSENGERS is a sta topology as shon in Fig. 9(a). The Meeting Room node is a cental node hee the bag of tasks is stoed. The nodes ae hee okes solves tasks. Since each task can be executed independently,no infomation needs to be exchanged beteen okes,and theefoe no links exist beteen office nodes. Because the numbe of tasks may be lage,it is useful to allo the tasks to be geneated on-the-fly, athe than explicitly at the beginning. A task-geneation messenge, denoted as g in Fig. 9(b),exists fo this pupose and stays at the cental node. When the application stats,the taskgeneation messenge geneates an initial numbe of tasks and injects multiple oke Messenges (),each of hich is assigned an office. Each of them hops back and foth beteen the meeting oom and its office,as shon in Fig. 9(b). On each tip,it bings a ne task to its office to ok on. Afte it finishes executing the task,it caies the esult back to the meeting oom,and pulls anothe task fom the bag of tasks. When the numbe of tasks in the task pool falls belo a cetain theshold,the task-geneation messenge geneates additional ne tasks. A oke teminates hen the bag of tasks becomes empty. The task-geneation messenge teminates hen all the esults have been itten to the output file,thus effecting global temination Banch-and-bound paadigm The logical netok suppoting the distibuted implementation of the banch-and-bound paadigm is shon in Fig. 10(a). The node is hee an initial pool of tasks is stoed,and the nodes ae hee okes exploe a potion of the seach space. When a oke finds a solution that is bette than the best peviously knon solution,othe okes ae notified of the ne puning bound,and the office nodes ae fully connected to facilitate this. This exchange of ne puning bounds is an example of non-essential communication as defined in Section 2: each task ould successfully complete ithout this infomation exchange,but using impoved puning bounds discoveed by othe tasks can impove pefomance significantly. Thee types of Messenges,shon in Fig. 10(b),ae used to implement the banch-and-bound paadigm. An initialization Messenge (g),hich stays in the meeting oom,geneates the initial data static poblem data,the initial puning bound,and the initial task pool (by a patial beadth-fist expansion of the oot node) and also injects the oke messenges. Multiple oke messenges () exist in the system,one pe office node. When a oke messenge finds a bette solution,unne messenges () ae ceated. These

8 8 H. Kuang et al. / J. Paallel Distib. Comput. ( ) g (a) (b) (c) Fig. 10. Logical netok fo the banch-and-bound paadigm. (a) e e Fig. 11. Logical netok fo the genetic pogamming paadigm. unne messages hop to the othe office nodes to update the puning bound. Afte a oke caies a node fom the initial task pool to its office,it exploes the subtee ooted at that node in depth-fist ode. Wheneve it expands a node,it oks on one of the childen and places the emaining childen in a local task pool. It epeatedly exploes nodes in its local task pool. When the local task pool is exhausted,the oke messenge hops to the meeting oom. If the initial task pool has not been dained,the oke caies one of the nodes fom the pool back to its office and continues as befoe. When a oke hops to the meeting oom and finds that the initial task pool has been dained,it attempts to balance the system load by stealing ok fom a andomly chosen office of anothe oke. It hops to the chosen office as shon in Fig. 10(c). If the chosen office has a nonempty task pool,the oke steals one of the tasks by caying it back to the oke s on office and executing the stolen task. If the chosen office has an empty task pool,the oke andomly chooses anothe office to steal a task fom. Afte a cetain numbe of unsuccessful attempts (cuently set as half of the numbe of active okes),the oke teminates. Global temination occus hen the last oke teminates. (b) 4.3. Genetic pogamming paadigm Fig. 11(a) illustates a logical netok fo the implementation of the genetic pogamming paadigm in the MESSENGERS system. The node is hee okes exchange global infomation,hile an node is hee a oke executes the evolution pocess ith a distinct subpopulation pool. Thee types of messenges exist in the system,as illustated in Fig. 11(b). The oke messenges () execute the basic genetic algoithm,one pe office. Each oke is passed a diffeent seed fo its andom numbe geneato hen it is injected. Each oke andomly geneates an initial population pool and then epeatedly applies genetic opeations to ceate subsequent geneations until the temination condition is satisfied. When the best individual in the cuent geneation is bette than the pevious best in the entie system,a unne () messenge sends it to the meeting oom node and epots it to the use. When all the oke messenges finish the specified numbe of geneations o one of them satisfies the temination pedicate,the entie application is teminated. Wokes peiodically exchange individuals ith okes at neighboing nodes,as discussed in Section 2.4. They do this by geneating expote messenges (e) that cay emigants to the neighbos. The fequency and ate of these exchanges ae detemined by the use-specified paametes EmigationInteval and EmigationRate,hich may be modified at untime though the feedback indo. Inceasing the emigation ate o deceasing the emigation inteval causes moe population mixing but inceases the communication cost Finite diffeence paadigm Fig. 12(a) shos a logical netok to suppot the distibuted implementation of the finite diffeence paadigm on a 2D gid. The nodes ae hee element values ae updated. Each node is assigned a potion of the gid. At each iteation,ne values ae computed. Bounday values ae exchanged beteen neighbos hen bounday infomation becomes obsolete. The Meeting oom node is hee the temination condition is evaluated. The implementation uses fou types of messenges as shon in Fig. 12(b). The initialization messenge (i) builds the logical netok and injects the oke messenges ()

9 H. Kuang et al. / J. Paallel Distib. Comput. ( ) 9 i b b b b b (a) (b) Fig. 12. Logical netok fo the finite diffeence paadigm in MESSENGERS System. into offices,one pe office. Each oke messenge initializes its patition and stats bounday messenges (b). The function values in each patition ae epeatedly updated by oke and bounday messenges. At each iteation,the oke messenge updates the inne gid cells (i.e.,the cells hose values do not depend diectly on the values of gid cells in neighboing patitions). Each patition has eight bounday messenges,one pe side and one pe cone. Each bounday messenge shuttles beteen to neighboing nodes. At each iteation,a bounday messenge updates the gid cells on the potion of the bounday fo hich it is esponsible. It then hops to the othe node,caying bounday data,and updates the neighboing office s ghost bounday (i.e.,a shado of its neighbo s bounday). At the next step,it oks at the neighboing office,updates a potion of its bounday,and caies the bounday infomation back to the pevious node. Each node also has a epot messenge () that gathes infomation necessay to evaluate the global temination condition (e.g.,the toleance) and caies it to the. When the temination condition is satisfied,the epot messenges hop back to the nodes and set temination flags. The above implementation uses seveal stategies to impove the pefomance. In pinciple,it attempts to update each bounday and send it to its neighbo as ealy as possible. In this ay the communication and computation can be ovelapped,and the idle aiting time can be avoided. At the beginning of each iteation,the oke messenge sends out a signal,hich akes up incoming bounday messenges and allos them to update the bounday and cay the bounday data to its neighboing node. Hoeve,if some of the neighboing nodes ae sloe and the bounday messenges have not aived yet,the oke messenge does not ait fo the sloe nodes. Instead it updates the inne pat of the patition,inteupting itself peiodically to give late incoming bounday messages a chance to ok. This eliminates the necessity of a baie at each step,hence educing the idle aiting time Individual-based simulation paadigm The distibuted implementation of the individual-based simulation paadigm is simila to that of the finite diffeence paadigm. Both use the same logical netoks,the types of messenges,and the nea neighbo bounday exchange at each time step. Thee ae thee impotant diffeences hee the individual-based simulation paadigm equies additional mechanisms. One majo diffeence aises because of the dynamic migation of the entities in individual-based simulations. Because of this migation,nea-neighbo communication in the distibuted implementation poceeds in to phases: fist the emigating entities move to the neighboing nodes,and then the bounday infomation is exchanged. A second diffeence aises because of the moe dynamic natue of the individual-based simulation paadigm. In the finite diffeence paadigm,once the patition of the use gid is fixed,the load on each machine and the message size exchanged emains constant fo the duation of the simulation. In the individual-based simulation paadigm,because entities ae moving in the space,the load on each machine and the message sizes ae dynamically changing. A thid diffeence is the impotance of epeatability: a use should have the option of eunning a simulation and obtaining exactly the same esults. This can be vey impotant fo validating changes made at the application level o fo tacking don elusive application bugs. Achieving epeatability in distibuted implementations pesents some inteesting challenges,due to the epatitioning of the use gid (i.e.,changes if the mapping of the use gid onto the logical nodes). Repatitioning may occu ithin a un due to load balancing,and it may also occu fom one un to anothe if the use uns the same simulation but changes the configuation (e.g.,changes the logical netok o the numbe of machines). One issue that must be addessed to achieve epeatability is andom numbe geneation. In ode fo to simulations to achieve the same esult,the andom choice made duing the second un must be exactly the same as the co-

10 10 H. Kuang et al. / J. Paallel Distib. Comput. ( ) Fig. 13. Distibuted odd even individual-based simulation paadigm. speedup numbe of okes Fig. 14. Speedup fo the bag-of-tasks paadigm expeiments. esponding andom choice made duing the fist un. This can be achieved in vaious ays: fo example,a steam of andom numbes can be associated ith each entity,o a steam of andom numbes can be associated ith each use gid cell. The pape [15] contains a compaison of these to appoaches and a fe othes as ell. The second issue that affects epeatability is the ode in hich entities ae pocessed. This ode affects the esult of the simulation hen the immediate state update method is used. We intoduce an odd even labeling scheme to specify a paticula ode in hich entities ae updated. This scheme, illustated in Fig. 13,ensues that entities ae pocessed in the same ode,iespective of the patitioning of the use gid. We label each use gid cell ith its index (x, y),and then label the cell ith a numbe in the ange 0 3,computed as (x mod 2) + (y mod 2). At each time step,e update the states of all entities located at use gid cells ith a paticula label value (stating ith 0) befoe poceeding to the next label value. In ode to make this labeling scheme ok coectly,the bounday exchange beteen neighboing patitions needs to be expanded,and a condition must be imposed on the size of the use gid. The eason fo the expanded bounday exchange is illustated in Fig. 13. The hite aea is the collection of use cells that ae allocated to the machine, and the gay aea is the ghost bounday (i.e.,the bounday data obtained fom the neighbo as pat of the bounday exchange). In ode to coectly update the cell ith label 3 in the loe left cone of the aea allocated to the machine, e must have the updated contents of the cell ith label 2 immediately belo it,hich in tun equies the cell ith label 1 belo it and to its left,hich in tun equies the cell ith label 0 belo it. To update this last cell,e need all its neighbos. As this example illustates,ou odd even scheme equies expanding the size of the exchanged bounday by a facto of 4. The odd even labeling scheme epesents a gaph theoetical coloing of the use gid: to cells hose boundaies shae eithe an edge o at a cone ae assigned diffeent labels. If to entities ae located in to diffeent gid cells that ae assigned the same label,the distance beteen the entities is at least as lage as the length of the shotest side of a gid cell. A typical individual-based system model includes paametes v and m,hich,espectively, epesent the adius of visibility and the adius of motion. An entity s behavio in a time step can only be affected by anothe entity if the second entity is ithin a distance of v of the fist,and an entity can move a distance of at most m in one time unit. Geneally it is assumed m v. It must be tue that the length of a use cell must be at least v.if e stengthen this constaint by equiing that the length of the shotest side a use gid cell is at least ( v + m ),then an entity in a gid cell cannot affect an entity in a diffeent gid cell ith the same label. Hence,ith this stengthened constaint,the odd even labeling scheme ill guaantee epeatability hen the immediate-update method is used. 5. Pefomance evaluation 5.1. Bag-of-tasks paadigm Ou expeiments ith the bag-of-tasks paadigm ee based on a Monte Calo simulation of a model of light tanspot in oganic tissue [19]. The sequential pogam as povided by the Beckman Lase Institute and Medical Clinic at UC Ivine. Because the model assumes that the movement of each photon in the tissue is independent of all othe photons,this simulation fits ell in the bag of tasks paadigm. The numbe of photons simulated is 1,000,000, and each task simulates the movements of 1000 photons. The expeimental esults ae shon in Fig. 14. The gaph pesents a nea-linea speedup Banch-and-bound paadigm We tested the banch and bound paadigm on a ellknon combinatoial poblem,the Taveling Salesman Poblem (TSP). In the sequential TSP pogam,the bounding ule is based on a minimum spanning tee [11] of the unvisited cities in a patial tou. We used data fo 24 cities.

11 H. Kuang et al. / J. Paallel Distib. Comput. ( ) 11 speedup numbe of okes Fig. 15. Speedup fo the banch-and-bound paadigm expeiments. Table 1 Summay statistics fo Banch-and-bound execution times ith diffeent seach ode pemutations Mean Max Min StdDev Sequential MESSENGERS 3 okes MESSENGERS 6 okes Duing ou expeiments,e obseved nondeteministic pefomance behavio of the distibuted banch-and-bound pogam [16]. Theefoe,e executed both the sequential and distibuted pogams ten times,each ith diffeent input data. The expeiment esults in Fig. 15 epesent the aveage speedup fom 10 uns. These sho a nea-linea speedup fo the distibuted banch-and-bound pogam. The nondeteministic behavio of the distibuted banchand-bound seach is patly due to the fact that the distibuted banch-and-bound pogam implicitly changes the seach ode by exploing the poblem space concuently. To investigate ho the seach ode effects the pefomance,e conducted anothe type of expeiments. We an both sequential and distibuted pogams ten times,each time ith the same input data,but ith a diffeent pemutation of the initial task pool. The esults of this expeiment,hich ae summaized in Table 1,sho that the pogam execution time is geatly influenced by the seach ode. Fo example,the maximum and minimum execution times fo the sequential pogam ae,espectively, and s,a atio of moe than 20. The table also shos that intoducing multiple okes oking simultaneously has a significant smoothing effect on the execution time,making the execution time moe pedictable Genetic pogamming paadigm We tested the genetic pogamming paadigm also using the taveling salesman poblem. In this case e used a 30-city poblem. The oiginal sequential pogam is due to Lalena [17]. We estuctued the souce code to fit into ou paadigm and added a mutation opeato. The distibuted pogams ee un on a netok of fou okstations. Because genetic pogams ae pobabilistic,e an each pogam 10 times. To ensue a fai pefomance compaison,e defined the quality atio of a paticula computed solution to a minimization poblem as fitness(optimal solution) fitness(computed solution),hee the fitness function evaluates a given solution. Ou sequential pogam attained a quality atio of 99.4% afte unning fo an indefinite peiod of time. We then used this atio as the temination condition fo the distibuted pogams,so that all solutions ould yield the same quality of solution. We an the genetic algoithm fo a sequential pogam ith a population size of 1250,fo a sequential pogam ith a population size of 5000,and fo a distibuted (MESSENGERS) pogam ith a population size of 1250 pe oke (i.e.,a total population of 5000). The espective aveage unning times ee 442.3, ,and s. Thus the MESSENGERS pogam found the solution fastest and as fou times as fast as the sequential pogam ith a population size of We also selected a epesentative un of thee pogams and illustate the evolution pocess in Fig. 16. The squae line is the sequential pogam ith population size The tiangle line is the sequential pogam ith population size 5000 hose population includes that of the fist sequential pogam. The diamond line is the MESSENGERS pogam ith the same initial population as the fist sequential one,but hich exchanges immigants peiodically duing the evolution. The gaph shos that initially the genetic pogam ith the smalle population size can find good solutions the fastest. Hoeve,afte it finds a solution ith a quality atio 97.1%,it takes much longe to impove this solution. The sequential pogam ith the lage population pefoms bette afte this point. The MESSENGERS pogam combines the benefits of the to sequential pogams. Initially it behaves as the sequential pogam ith the smalle population. Late it conveges slightly sloe because of the geate divesity of individuals esulting fom the exchange of immigants. In the final stage,it beats both sequential pogams and finds the nea optimal solution ith the quality atio of 99.4% in the shotest time Finite diffeence paadigm We tested the finite diffeence paadigm using Metopolis Monte Calo algoithm,hich solves the Ising model [1,13]. In ou expeiments,the simulated space is a 2D tooidal gid. We vaied the gid size,and also tied to diffeent patition stategies: stip patition (i.e.,the use gid is patitioned into stips) and ectangula patition (i.e.,the use gid is patitioned into ectangula blocks). Fig. 17 shos the speedup of the distibuted finite diffeence pogams using 9 machines. The speedup of the distibuted pogams inceases ith the computation-tocommunication atio. Also,pogams ith a ectangula

12 12 H. Kuang et al. / J. Paallel Distib. Comput. ( ) 1 The Quality Ratio of the Solution messenge pogam sequential pogam ith population size 1250 sequential pogam ith population size Elapsed time(s) Fig. 16. Repesentative evolution pocesses of the genetic pogams. Speedup Rectangula patition Delayed state update method Immediate state update method Stip patition 540x x x x x2250 Poblem size Speedup Fig. 17. Speedup fo the finite diffeence paadigm expeiments. patition have consistently bette speedup than those ith a stip patition. This is because pogams ith a ectangula patition have a smalle bounday and thus smalle amounts of communication data than pogams ith a stip patition Individual-based simulation paadigm We tested the individual-based simulation paadigm using a modification of the fish schooling model descibed in [12]. This model assumes a 2D space hee each fish peiodically adjusts its position and velocity by coodinating its movement ith up to fou of its neighbos. We tested the paadigm using both the delayed state method and the odd even immediate state update method. The simulation space is a 2D tooid in hich fish move as a single school of fish fo 500 simulation steps. We an each pogam thee times. The execution times pesented ae the aveages of the thee uns. Fig. 18 shos the speedup fo inceasing the numbes of simulated fish. The figue shos that the speedup inceases ith the poblem size. This is due to the inceasing computation-to-communication atio. We can also see that the delayed state update pogam has a bette Fish numbe Fig. 18. Speedup fo the individual-based simulation paadigm expeiments. speedup than the immediate state update pogam. This is because each time the bounday infomation is exchanged, the immediate state update pogam sends and eceives messages fou times the size of the delayed state update pogam. The immediate state update pogam also needs to duplicate edundant computations. 6. Conclusions In this pape e pesented an appoach to distibuted computing that uses the concept of ell-knon paadigms. Its main featues,hich diffeentiate it fom othe appoaches, ae the folloing: (1) It is intended fo loosely-coupled netok envionments,not specialized multipocessos; (2) it is based on an infastuctue of mobile agents; (3) it suppots pogamming in C,athe than a functional o special-

13 H. Kuang et al. / J. Paallel Distib. Comput. ( ) 13 pupose language,and (4) it povides an inteactive gaphics inteface though hich pogams ae submitted,invoked, and monitoed. By implementing five idely used paadigms bag-oftasks,banch-and-bound,genetic pogamming,finite diffeence,and individual-based simulation e have demonstated the viability of this appoach fo use in heteogeneous and dynamically changing clustes of commodity okstations o PCs. One of the main easons fo the flexibility and potability of the PODC envionment is the use of mobile agents,hich povide a vitual envionment ithin hich the given paadigms can be implemented independently of any specific netoking o achitectual constaints. The pefomance tests indicate that,fo the chosen paadigms,the esulting ovehead is minimal,alloing the system to delive nealy linea speedup fo many types of applications. Although ou system equies the uses to patially estuctue thei sequential pogams to adapt them to the PODC envionment,ou expeience shos that the effot equied is vey small. The benefit is a significant impovement in pefomance due to the achieved paallelism. Refeences [1] K. Binde,D.W. Heemann,Monte Calo Simulation in Statistical Physics,Spinge,Belin,1988. [2] G.H. Botoog,H. Kuchen,Skil: an impeative language ith algoithmic skeletons fo efficient distibuted pogamming,in: Poceedings of the Fifth Intenational Symposium on High Pefomance Distibuted Computing (HPDC-5),IEEE Compute Society Pess,Silve Sping,MD,1996,,pp [3] M. Cole,Algoithmic Skeletons: Stuctued Management of Paallel Computation,Reseach Monogaphs in Paallel and Distibuted Computing,Pitman,London,1989. [4] J. Dalington,A.J. Field,P.G. Haison,Paallel pogamming using skeleton functions,in: PARLE 93,Paallel Achitectues and Languages Euope,June [5] M. Fukuda,L.F. Bic,M.B. Dillencout,Messages vesus messenges in distibuted pogamming,j. Paallel Distibuted Comput. 57 (1999) [6] M. Fukuda,L.F. Bic,M.B. Dillencout,F. Mechant,Distibuted coodination ith messenges,sci. Comput. Pogamming 31 (2) (1998). [7] E. Gendelman,L.F. Bic,M.B. Dillencout,An applicationtanspaent,platfom-independent appoach to ollback-ecovey fo mobile agent systems,technical Repot 35,Univesity of Califonia, Ivine,1999. [8] D.E. Goldbeg,Genetic Algoithms in Seach,Optimization,and Machine Leaning,Addison-Wesley,Reading,MA,1989. [9] L. Geengad,V. Rokhlin,A fast algoithm fo paticle simulation, J. Comput. Phys. 73 (1987) [10] G. Hatvigsen,S.A. Levin,Evolution and spatial stuctue inteact to influence plant-hebivoe population and community dynamics, Poc. Roy. Soc. London Se. B 264 (1997) [11] M. Held,R.M. Kap,The taveling-salesman poblem and minimum spanning tees,ope. Res. 18 (1970) [12] A. Huth,C. Wissel,The simulation of the movement of fish schools, J. Theoet. Biol. 156 (1992) [13] M.H. Kalos,P.A. Whitlock,Monte Calo Methods,vol. I,Basics, Wiley,Ne Yok,1986. [14] H. Kuang,L.F. Bic,M.B. Dillencout,Paadigm-oiented distibuted computing using mobile agents,technical Repot 38,Univesity of Califonia at Ivine,1999. [15] H. Kuang,L.F. Bic,M.B. Dillencout,Repeatability,pogammability,and pefomance of iteative gid-based computing,technical Repot,Infomation and Compute Science,Univesity of Califonia, Ivine,2001. [16] T.H. Lai,S. Sahni,Anomalies in paallel banch-and-bound algoithms,comm. Assoc. Comput. Mach. 27 (9) (June 1984) [17] M. LaLena,Tavelling salesman poblem using genetic algoithms, 1996, [18] S. Pelagatti,A Methodology fo the Development and the Suppot of Massively Paallel Pogams,Ph.D. Thesis,Dipatimento di Infomatica,Univesita di Pisa,Mach [19] S.A. Pahl,M. Keijze,S.L. Jacques,A.J. Welch,A Monte Calo model of light popagation in tissue,in: Dosimety of Lase Radiation in Medicine and Biology,SPIE Institutes fo Advanced Optical Technologies Seies,vol. IS 5,SPIE Optical Engineeing Pess,1989, pp [20] F.A. Rabhi,A paallel pogamming methodology based on paadigms,in: Tanspute and Occam Developments,IOS Pess, 1995,,pp [21] C.W. Reynolds,Flocks,heds,and schools: a distibuted behavioal model,comput. Gaphics 21 (4) (July 1987) [22] S. Siu,M. De Simone,D. Gosami,A. Singh,Design pattens fo paallel pogamming,in: PDPTA 96,August [23] E.F. Van de Velde,Concuent Scientific Computing,Spinge,Belin, Haiong Kuang eceived he M.S. degee in Infomation and Compute Science fom the Univesity of Califonia,Ivine,in 1999 and he Ph.D. in Infomation and Compute Science fom the same univesity in She cuently oks as an Assistant Pofesso of Compute Science at the Califonia State Polytechnic Univesity,Pomona. He eseach inteests ae paallel and distibuted computing. Lubomi F. Bic eceived his M.S. degee in Compute Science fom the Technical Univesity Damstadt,Gemany,in 1976 and his Ph.D. in Infomation and Compute Science fom the Univesity of Califonia, Ivine,in He is cuently Pofesso and Co-Chai of the Compute Science Depatment at the Univesity of Califonia,Ivine. His pimay eseach inteests lie in the aeas of paallel and distibuted computing. Cuently he is co-diecting the Messenges Poject,hich exploes the use of self-migating theads to simplify the pogamming of computationally intensive applications and to impove thei pefomance in distibuted compute envionments. Michael B. Dillencout is an Associate Pofesso of Compute Science at the Univesity of Califonia,Ivine. He holds an MA degee in Mathematics (Univesity of Wisconsin,1975),an M.S. degee in Compute Science (Univesity of Wisconsin,1976),and a Ph.D. in Compute Science (Univesity of Mayland,1988). His pimay eseach inteests ae distibuted computing and algoithm design and analysis.