ASYNCHRONOUS PARALLEL PROGRAMMING MODEL FOR SMP CLUSTERS

Transcription

1 ASYNCHRONOUS PARALLEL PROGRAMMING MODEL FOR SMP CLUSTERS Ta Quoc Vet and Tsutomu Yoshnaa Graduate School of Informaton Systems Unversty of Electro-Communcatons Tokyo, Japan ABSTRACT Our study proposes a novel MPI-only parallel prorammn model wth mproved performance for SMP clusters. By rescheduln tasks n a typcal flat MPI soluton, our model forces processors of an SMP node to work n dfferent phases, thereby avodn unneccessary communcaton and computaton bottlenecks. Ths study acheves a snfcant performance mprovement wth a mnmal prorammn effort. In comparson wth a de-facto flat MPI soluton, our alorthm can yeld a % performance mprovement for a 6-node cluster of Xeon dual-processor SMPs whle performn a dstrbuted matrx multplcaton. KEY WORDS Asynchronous Parallel Prorammn Model, Flat MPI, Matrx Multplcaton, SUMMA, SMP cluster Introducton Our study tarets an mprovement n a flat MPI prorammn model for SMP clusters. The typcal flat MPI model treats all the processors of a cluster equally, wthout consdern computaton and communcaton lnks amon processors on the same SMP node. Ths study examnes these lnks and proposes a more effectve prorammn model. Performance of an SMP may ncrease when ts processors work asynchronously,.e., a (number of) processor(s) perform(s) a computaton task whle the remann processor(s) perform(s) a communcaton task. Ths phenomenon can be explaned based on the lmtaton of shared resources for computaton and/or communcaton. When all processors work synchronously, they requre the same knd of resources. Ths may lead to a scarcty of resources, thereby formn an executon bottleneck. In our system, network and memory bus bandwdth lmtatons are the root causes of communcaton and computaton bottlenecks, respectvely. Consequently, we propose an asynchronous parallel prorammn model n whch the processors of a node perform computaton and communcaton tasks asynchronously. Moreover, we rearrane the communcaton tasks to avod unnecessary nternode communcaton. The prorammn effort requred to develop our new soluton from an exstn flat MPI code s farly small. As an example, the new model s appled to perform a dstrbuted matrx multplcaton. The expermental envronment conssts of a cluster of 6 Intel Xeon.8 GHz dual-processor nodes connected va a Gabt Ethernet network. Each node has.5 GB of memory, Red Hat Lnux 9.0 as the operatn system, and MPICH..6 [] as the MPI lbrary. Local matrx multplcaton (demm) s performed by oto-blas []. The snfcant features of ths study are () a clear and easy-to-apply MPI-only asynchronous parallel prorammn model for SMP clusters and () a formula for evaluatn ts performance mprovement usn varables defned for problem nature, problem sze, and system specfcatons. The remann paper s oranzed as follows. Secton ntroduces related studes that also examne prorammn models for SMP clusters. Secton presents our model n detals. Secton descrbes a dstrbuted matrx multplcaton soluton. Secton 5 dscusses the expermental results. Fnally, secton 6 concludes the paper. Related Studes Takn nto account the herarchcal memory nature of SMP clusters, several studes have proposed hybrd MPI- OpenMP prorammn models n whch each SMP node runs only a snle MPI process and computaton parallelzaton nsde a node s acheved by OpenMP. Hybrd models are classfed as a hybrd model wth process-toprocess communcaton (hybrd PC) and a hybrd model wth thread-to-thread communcaton (hybrd TC). The hybrd PC model has been examned earler and could not show a postve result. F. Cappello et. al have frstly shown a common alorthm to develop a fne-ran hybrd PC soluton []. The model then s contnued to be appled for several problems and hardware platforms [, 5]. These studes revealed that n most cases, hybrd PC s nferor to flat MPI despte ts three man advantaes: () low communcaton cost, () dynamc load balancn capablty, and () coarse-ran communcaton capablty [5]. The poor performance of the fne-ran hybrd PC model s prmarly due to ts poor ntranode OpenMP parallelzaton effcency [] resultn from an extremely low cache ht rato [5]. Then, we proposed an enhanced hybrd verson, the hybrd TC model [6, 7], whch was also dscussed by G. Wellen et. al [8] and R. Rabensefner et. al [9, 0]. F-

2 T m T p (a) Flat MPI dt dt T n-phase T Communcaton Computaton (b) Asynchronous MPI out-phase Fure. MPI varatons for a dual-processor SMP node The communcaton speed-up S m s defned by the rato between communcaton speeds durn T outphase and T nphase. The computaton speed-up S p s defned smlarly. Note that the communcaton and computaton speeds durn T nphase are the same as that of the flat MPI model. Durn T outphase, an SMP node has better communcaton and computaton speeds than the flat MPI model has. In other words, S m > and S p >. Wth reard to Fure, T outphase = mn( T m ; T p ); () S m S p and the dfference n the executon tme dt between the two models s ven by On the other hand, dt = T flat T async = dt + dt : dt = T outphase(s m ) nally, we proposed a mddle ran sze approach that allows hybrd TC to acheve an mpressve performance on dfferent platforms n varous types of experments []. However, for problem solvn, hybrd TC requres a consderable prorammn effort to determne an effectve mddle ran task sze. The asynchronous MPI model proposed n ths paper acheves a performance that s smlar to that of the hybrd TC model wthout concernn OpenMP. Morever, t retans the flat MPI computaton pattern, thereby avodn the ran-sze problem. SMP-aware [] s another optmzaton approach that mproves (especally collectve) communcaton performance for SMP clusters. However, snce SMP-aware does not concern computaton tasks, t s not expected to et a comparable performance as that of the asynchronous model. The Asynchronous Model. Flat and Asynchronous MPI Fure shows actvtes of a dual-processor SMP node for the flat and asynchronous MPI solutons. In the flat MPI soluton, all the processors of a node work n the same phase. They execute communcaton or computaton smultaneously. The executon tme T flat s the sum of the communcaton tme T m and the computaton tme T p, whch usually depend on the problem sze. In the asynchronous MPI soluton, processors work n dfferent phases. The tme for whch processors are n an out-of-phase status,.e., a processor executes computaton whle the remann performs communcaton, s denoted by T outphase. Smlarly, the tme for whch processors work smultaneously s denoted by T nphase. The sum of T outphase and T nphase forms the asynchronous model executon tme T async. and Consequently, dt = T outphase(s p ): dt = T outphase(s m + S p ): () The performance speed-up S of the model s evaluated by S = T flat =+ T T outphase (S m + S p ) : () async T outphase + T nphase Equatons (), (), and () defne the tme saved and the speed-up of an asynchronous soluton throuh T m, T p, S m, and S p. In eneral, T m and T p ncrease wth the problem sze, whch also ncreases dt. Ths mples that as the problem sze ncreases, the tme saved by the asynchronous model also ncreases. Speed-up S acheves a maxmum value of (S m + S p )= f T nphase =0,.e., T m =S m = T p =S p.. Communcaton Rearranement The asynchronous model requres a rearranement of communcaton. In ths model, nternode communcaton can be performed amon processes of the same phase only. The ornal flat MPI communcaton pattern does not always satsfy ths requrement. We propose a smple method to perform the requred rearranement, whch s shown n Fure, where squares represent MPI processes, numbers nsde these squares denote the process ID, and dotted rectanles ndcate communcators. The upper and lower parts of the fure descrbe the communcators before and after the rearranement. We assume a flat MPI communcator of eht processes ornatn from four dual-processor nodes. Ths

3 Node 0 Node Node Node C = fc for ( =0;<kk; ++) f bcast(a ;row comm); bcast(b ; col comm); C += ff(a b ); Fure. Flat MPI SUMMA pseudo-code. Fure. Communcaton rearranement Dstrbuted Matrx Multplcaton. SUMMA A a broadcastn scope (communcator) nb kk x nb A b Fure. The SUMMA alorthm communcator s splt nto two separate nternode party communcators the odd and even n terms of the process IDs. There are four addtonal ntranode communcators, whch are not shown n the fure. All communcaton operatons are rearraned to be performed nsde the newly formed communcators only. An nternode communcaton operaton between two processes of dfferent party communcators s replaced by an ntranode operaton and an nternode operaton nsde the same party communcator. For example, communcaton between processes and 7 may be replaced by that between pocesses and toether wth that between processes and 7. B C B C The asynchronous MPI model s demonstrated by performn a dstrbuted matrx multplcaton operaton: C ( ff op(a ) op(b )+fc ; where op(x) =X, X T or X H ; A, B, and C are lobal matrces of szes m k, k n, and m n, respectvely; ff and f are scalars. In ths paper, we focus on the case op(x) = X, but the dea can easly be extended to other cases. The scalable unversal matrx multplcaton alorthm (SUMMA) [] s one of the most effectve alorthms for ths problem. In comparson wth other stron canddates lke PUMMA [] or DIMMA [5], t exhbts a comparable performance and hh scalablty. Moreover, SUMMA s smple and strahtforward. Due to these advantaes, SUMMA s used as the basc for developn an asynchronous soluton. In SUMMA, processes are mapped onto an nprows npcols process rd. Global matrces A, B, and C are splt nto nprows npcols equal submatrces. A process stores the local submatrces A, B and C that are coresspondn to ts locaton n the process rd. Then, the local matrces A and B are dvded nto kk blocks of columns and kk blocks of rows, each of sze nb. Fure shows the data dstrbuton correspondn to a process rd. Shaded cells A, B, and C are the submatrces dstrbuted to a process that s located n the second row, thrd column of the process rd. Fure shows the pseudo-code of the alorthm, n whch functon bcast(data; comm) broadcasts data over the communcator comm.. Task Dependency Problem Independency between communcaton and computaton tasks s another requrement of the asynchronous model. We can arrane the tasks n any order only f they are ndependent. Secton of ref. [] has already demonstrated how to elmnate the dependences. In ths study, we do not descrbe the technque but show the result n Secton.. Communcaton The processor rd s oranzed such that t mnmzes the rearranement. Namely, processors are arraned such that all processors of a snle node belon to the same process row. For a 6-node cluster, we develop a 8 process rd, n whch a row communcator contans processes of the same party; thus, t s not necessary to rearrane bcast(b ;row comm).

4 0 a Even Odd b 8 6 : share(half_a, co_proc) : bcast(half_a, row_party_comm) : bcast(b, col_comm) T : c += alpha a b async (a) Broadcast operatons wthn flat MPI 5: exchane(half_a, co_proc) share(half_a, co_proc): bcast(half_a, row_party_comm): 0 6 Fure 6. Executon schema for the even and odd processes of a node durn a snle teraton. 5 7 exchane(half_a, co_proc): 0 6 wthn the asychronous executon tme. The ntranode communcaton steps should be executed smultaneously by all processes. (b) Rearranement for bcast(a ; row comm) Fure 5. Communcaton rearranement for a 6-node dual-processor cluster. The communcaton rearranement s shown n Fure 5 wth the assumpton that processor 0 s the source of a and b. Fure 5 (a) llustrates both the two broadcast operatons wthn flat MPI. Fure 5 (b) shows the three operatons that replace the ornal bcast(a ;col comm): ffl share(half a ;co proc): ntranode communcaton. In realty, ths step s mplemented by a set of send and recv calls evoked by the source process and co proc the remann process of the node. ffl bcast(half a ;row party comm): nternode communcaton. Even and odd row communcators broadcast dfferent halves of a. After ths step, every process has half a, whle two processes of the same node have dfferent halves. ffl exchane(half a ;co proc): ntranode communcaton. Each process exchanes ts half wth ts co proc. After ths step, all processes have a complete a. The nternode communcaton step bcast(half a ;row party comm) wll be executed 5 7. Elmnaton of Dependency We develop ndependent communcaton and computaton tasks, whch can be executed n any order. A loop teraton shown n Fure s the object of the elmnaton. By applyn one of the technques descrbed n [], we reconstruct the loop such that a new teraton ncludes the computaton part of the ornal teraton and the communcaton part of the next teraton. By ths reconstructon, we obtan a new teraton wth no dependency between the communcaton and computaton parts.. Asynchronous Soluton Fure 6 shows an executon schema for processors nsde a node durn a snle teraton. The correspondn pseudocode s shown n Fure 7. It s evdent that the code s rather smple. Consequently, a small prorammn effort s requred to acheve the asynchronous soluton. Some parts of the proram are unavalable to asynchronous executon. In SUMMA, tasks and 5 represent ntranode communcaton and they should be executed smultaneously by all the processes of a node. However, ntranode communcaton comprses only a small part of the total executon tme. Tasks,, and, whch comprse the man part of the soluton, wll be executed asynchronously. Tasks and represent nternode communcaton. Task represents a computaton. Even processes execute communcaton tasks frst, followed by computaton tasks. In contrast, odd pro-

5 C = fc for ( =0;<kk; ++) f f (s source node) f f (s source proc) send(half a + ;co proc); else recv(half a + ;co proc); f (s even) f bcast(half a + ;row party comm); bcast(b + ; col comm); C += ff(a b ); f (s odd) f C += ff(a b ); bcast(half a + ;row party comm); bcast(b + ; col comm); exchane(half a + ;co proc); Fure 7. Asynchronous SUMMA pseudo-code. Table. Predcted saved tme dt wth dfferent problem szes. m=n=k T m (sec.) T p (sec.) Expected dt (sec.) cesses work n the reverse order. Ths results n the appearance of an out-of-phase tme, thereby mprovn communcaton and computaton performance. 5 Performance Prelmnary examnaton results show that when the szes of local matrces A, B, and C are suffcently lare, the communcaton and computaton speed-ups of the asynchronous model are relatvely stable wth dfferent problem szes. The values of S m and S p can be estmated by smulatn synchronous and asynchronous executons n accordnly applcable condtons and then comparn ther performance. For our cluster, S m ß : and S p ß :8 Expermental results for dfferent problem szes are shown n Fure 8. Square and equal sze matrces A, B, and C are consdered. The block sze nb s fxed to. The values of m=n=k vary between 5000 and 0000, whch are suffcently lare to stablze the local oto-blas demm and small enouh to ft the memory lmt. The results are n ood areement wth those obtaned by usn the formulas ven by Equaton () and shown n Table, althouh there s a small dsparty due to measurement error. The left chart shows the tme saved by the asynchronous soluton n comparson wth the flat MPI one. As expected, the saved tme ncreases toether wth the problem sze. The rht chart shows the absolute performance per processor. The asynchronous model always exhbts a better result. However, the dstance between the two performance lnes reduces radually. At m=n=k=5000, the dfference s approxmately % (50 MFlops). At m=n=k=0000, t decreases by 5.% (0 MFlops). Ths result can be explaned by Equatons () and () toether wth the nature of the matrx multplcaton. The rowth rate of T p s O(n ) whle that of T m s only O(n ). Consequently, the rowth rates of dt and T async are O(n ) and O(n ), respectvely. Ths mples that the ncrease n dt s slower than that n T async, and S becomes smaller wth an ncrease n n. In comparson wth a hybrd TC soluton [], performance of whch s not shown n the fure, the asynchronous model s slhtly slower when the dfference n performance s less than % for all problem szes. The replacement of the ntranode communcaton wth the usae of shared varables saves some tme for the hybrd TC model. However, we beleve that the ease n prorammn of the asynchronous model s more mportant. 6 Conclusons and Dscussons Ths paper proposed a smple and effectve parallel prorammn model and demonstrated ts advantaes over the exstn solutons. It completely outperforms the typcal and wdely accepted flat MPI model. In comparson wth a stron and complcated hybrd TC model, t saves a snfcant amount of prorammn effort wth a nelble reducton n performance. The formulas for evaluatn the ncrease of performance are also remarkable. They predct the behavor of a cluster n varous crcumstances and prove scalablty of the asynchronous model. In fact, scalablty was proved for the SUMMA alorthm applyn the flat MPI model []; and the precedence of the asynchronous model thereby uarantees ts scalablty. For problems n whch the computaton and communcaton tme have dfferent rowth rates (e.., SUMMA), the ncrease of performance decreases wth a lare problem sze. For problems wth constant computaton and communcaton cost rato wth dfferent problem szes (e.., NAS- CG), we expect a stable rate of performance mprovement by the asynchronous model. Thouh the analyses and experments n ths paper were manly for a dual-processor cluster, the conclusons are expandable to other clusters wth more processors per node. In ths case, t s necessary to determne an effectve number of phases and correspondn task-schedules.

6 5 000 Saved tme (Seconds) Performance per processor (MFLops) Asynchronous MPI Flat MPI Matrx Sze (m=n=k) Fure 8. Expermental results. Matrx Sze (m=n=k) We are developn an asynchronous MPI verson of the HPL benchmark, experment results and analyses of whch are soon to be publshed on our homepae, Acknowledment Ths research s partally supported by the Nppon Foundaton of the Japan Scence Socety (JSS) No.7-5. References [] MPICH Team. MPICH, a Portable MPI Implementaton. [] K. Goto. Hh-Performance BLAS by Kazushe Goto. [] F. Cappello and O. Rchard. Intra Node Parallelzaton of MPI Prorams wth OpenMP. TR- CAP-990, techncal report, fc/- onfrewww/96.ps.z, 998. [] F. Cappello and D. Etemble. MPI versus MPI+OpenMP on IBM SP for the NAS Benchmark. Proc. Supercomputn 000, 000. [5] T. Boku, S. Yoshkawa, and M. Sato. Implementaton and Performance Evaluaton of SPAM Artcle Code wth OpenMP-MPI Hybrd Prorammn. Proc. European Workshop on OpenMP 00, 00. [6] T. Q. Vet, T. Yoshnaa, and M. Sowa. A Master- Slaver Alorthm for Hybrd MPI-OpenMP Prorammn on a Cluster of SMPs. IPSJ SIG notes 00- HPC-9-9, 00, 07-. [7] T. Q. Vet, T. Yoshnaa, B. A. Abderazek, and M. Sowa. A Hybrd MPI-OpenMP Soluton for a Lnear System on a Cluster of SMPs. Proc. Symposum on Advanced Computn Systems and Infrastructures, 00. [8] G. Wellen, S. Haer, A. Basermann, and H. Fehske. Fast Sparse Matrx-Vector Mulplcaton for TeraFlop/s Computers. Proc. Vector and Parallel Precessn, 00. [9] R. Rabensefner and G. Wellen. Communcaton and Optmzaton Aspects of Parallel Prorammn Models on Hybrd Archtectures. The Internatonal Journal of Hh Performance Computn Applcaton, 7(), 00, 9-6. [0] R. Rabensefner. Hybrd Parallel Prorammn: Performance Problems and Chances. Proc. The 5th CUG (Cray User Group) Conference 00, 00. [] T. Q. Vet, T. Yoshnaa, B. A. Abderazek, and M. Sowa. Constructon of Hybrd MPI-OpenMP Solutons for SMP Clusters. IPSJ Transactons on Advanced Computn Systems, 6(SIG ), 005, 5-7. [] J. L. Traff. SMP-Aware Messae Passn Prorammn. Proc. The Eht Internatonal Workshop on Hh-Level Parallel Prorammn Models and Supportve Envronments, 00. [] R. A. van de Gejn and J. Watts. SUMMA: Scalable Unversal Matrx Multplcaton Alorthm. LAPACK Workn Note 99, techncal report, Unversty of Tennessee, 995. [] J. Cho, J. J. Donarra, and D. W. Walker. PUMMA: Parallel Unversal Matrx Multplcaton Alorthms on Dstrbuted Memory Concurrent Computers. Concurrency: Practce and Experence, 6(7), 99, [5] J. Cho. A Fast Scalable Unversal Multplcaton Alorthm on Dstrbuted-Memory Concurrrent Computers. Proc. IPPS 97, 997.