Writing Libraries in MPI*

Transcription

1 Writing Libraries in MPI* Anthony Skjellumt Nathan E. Doss Purushotham V. Bangaloret Computer Siene Departmentt & NSF Engineering Researh Center for Computational Field Simulation Mississippi State University Abstrat Abstrat MPI is the new de fato standard for multiomputer and luster message passing. This shod paper desribes the issues underlying the development of parallel libraries with MPI. We motivate the need for parallel libraries, and explain why ommon messagepassing systems do not support them. We desribe added features needed to support libraries, and what these features are alled in MPI. We mention additional features of MPI that support libraries, but whih are beyond the sope of this initial paper on this topi. We provide guidelines in terms of "do's" and UdonYsl' for would-be MPI library writers. We present a linear algebra library in skeletal form on a twodimensional virtual topology, inluding ode fragments. We summarize by mentioning related work, and omment on future ode development of parallel libraries under MPI, as well as porting efforts of existing odes lo the system. 1 Introdution MPI is the new de fato standard for multiomputer and luster mesaage passing [l]. Besides portability and higher performane, one of the primary motivations for MPI is to support parallel libraries, and, onsequently, to support the reation of large-sale multiomputer software. We desribe basi issues in the development of parallel libraries using MPI. We motivate the need for parallel libraries as suh, and explain why ommon message-passing systems do not support them well. We desribe the kind of message-passing features that 'Work supported by the NSF Engineering Rerearh Center for Computational Field Simulation, Missisaippi State University. are needed to support libraries, what MPI alls suh features, and how they are related to one another. We mention additional features of MPI that support litbraries, but whih are beyond the =ope of this initial paper on this topi. We provide guidelines in terms of "do's" and Udon'ts" for would-be MPI library writers. Given MPI features for library safety, MPI library writers will be able to expet reliable message-passing behavior without sarifiing performane. Furthermore, MPI provides a higher level of abstration than ommon message-passing systems, whih will help library writm develop more effiient and understandable odes simultaneously. MPI was designed to support primitives that would map effiiently to the existing and expeted high performane message passing hardware of multiomputers and gigabit luster networks. MPI library writers will be able to aess suh hardware apabilities in portable ode. To onretize the disussion, we present a linear algebra library in skeletal form on a two-dimensional virtual topology, inluding ode fragments. These exa-mples are simplified from other work we have done in the area of salable libraries, so they represent realisti implementations. We think that this extended example will prove instrutive to those who are planning to d.evelop distributed data strutures for mathematial libraries ["I. We summarize by mentioning related work, and omment on future ode development of parallel libraries under MPI, as well as porting efforts of existing library odes to the system. 2 Motivation This setion motivates the need for parallel libraries. It sets expetations for what libraries need to work effetively in a distributed memory environment, and explains why ommon message-passing sys V94 $3.00 Q 1994 IEEE 166

2 term do not support libraries well. MPI's solutions to these needs are disussed. 2.1 Why Parallel Libraries? Parallel libraries are needed to enapsulate the ponderous details assoiated with distributed realizations of key algorithms. The programmer needs: 0 portable, reusable, high quality odes, 0 easily available, reliable odes, 0 higher level portability interfaes, 0 the ability to hide oding omplexities from other programmers and users. Libraries ensure onsistent quality of suh proedures, and provide a "higher level" of portability than MPI itself an or should provide. Libraries prevent eah programmer from repeating the basi work of defining onsistent data strutures, data layouts, and methods that implement key algorithms (suh as matrix operations). Sine the best libraries ome with several variations on parallel systems (different data layouts, different strategies dependent on sirse of system, or type of floating point), this too needs to be hidden from the user. 2.2 What's Missing? The following features are missing from ommon vendor and portability systems: 0 ontexts of message passing (isolation), 0 olletive ommuniation funtionality, defined with proess-group sope, 0 virtual-topology naming of proesses. The subsequent paragraphs elaborate on eah of these topis, and why libraries need these features Communiation Contexts - Isolation In the sequential Fortran and C environments, it is reasonably easy to reate libraries, beause the stak-oriented proedural programming model has well-defined onditions about reasonable versus erroneous programs. However, in the distributed-memory, message-passing environment, it is diffiult to write libraries with either vendor or ommon portability systems. This seemingly strong inditment is baked up as follows: There is no reasonable way for libraries to isolate themselves from the on-going point-to-point message passing present in a running appliation. Message tags, the sole meane for designating restritions on message delivery, ar inadeient for isolation. For instane, more than one library (or invoation of the same library) ould use the mame tags. Systems that lak soure seletivity have to use tags to reate deterministi olletive operations; in suh a situation, olletive and point-to-point messages an be misqueued unless the system reserves tags for this purpose. Finally, wildard reeipt-seletivity on tags destroys any promise of real protetion that tags ould otherwise afford Colletive Operations Colletive operations, suh as redue, broadast, and so on, are not defined in all ommon systems, and when defined are most often only available over the full set of proessors (or proesses) alloated to the user. Libraries need to work on different olletions of proesses, and to have available olletive operations that only impat the proesses that are about the olletive operation; uninvolved proesses want to remain uninvolved. Furthermore, the olletive operations have to work reliably, and without rae onditions Virtual-Topology Naming Libraries don't want to desribe point-to-point ommuniation in terms of hardware-dependent names; in fat, many algorithms are more natural if desribed in terms of point-to-point alls relative to a virtualtopology naming sheme. Virtual-topology naming might reflet row or olumn parallelism in matrix op erations, for instane [SI. So, we have established three key areas that will support the reation of libraries: 0 Safe ommuniation spae (isolation), 0 Support for olletive operation, not just pointto-point; group sope is needed for olletive o p erations, 0 Abstrat names for proesses are based on virtual topologies, or at least rank-in-group names. 2.3 MPI's Support for Libraries In MPI, the features that implement the important onepts disussed in the previous setion are as follows: 167

3 Proess Groups define an ordered olletion of proenses, eah with a rank. Proess groups define the low-level names for inter-proeee ommuniation. For point-to-point message passing, proaa group define the rank of sender and reipient. For olletive ommuniation, proess group define the set of partiipants. Virtual Topologies are the final link in the naming proess for libraries, providing an alternative to simple rank naming for libraries. Cartesian and general graph topologies are supported. The user an also define his or her own topology strategies. Contexts provide the ability to have separate safe "universes" of message-passing for proess group in MPI. (A ontext is a onept made up of a proess group and other information, it does not speify an implementation.) We talk about ontexts in MPI, but they are atually implemented with ommuniators. Communiators enapsulate ontexts, groups, and virtual topologies in an objet that provides the appropriate sope for all ommuniation op erations in MPI. Communiators bind proess group and ontexts together to form a safe ommuniation spae within the group. In summary, the use of separate ommuniation ontexts by distint libraries (or distint library invoations) will insulate ommuniation internal to the library exeution from external ommuniation (group safety). This allows the invoation of the library even if there are pending ommuniations, and avoids the need to synhronize eah entry into and exit from library ode. 2.4 Additional MPI features for libraries MPI also provides several other features whih are useful for building salable libraries: 0 Dupliation of ommuniators allows libraries to get additional ommuniation spae easily. (Good implementations may/will ahe suh additional ommuniation spae, resulting in signifiant performane benefits.) 0 Systemati, aligned subsetting of ommuniators allows ommuniating SPMD proesses to define hierarhies of ommuniating sub-groups. 0 Cahing of attributes on ommuniators allows users to define new operations that have the same oneptual properties as built-in MPI olletive operations. This is important for extending servies not antiipated by MPI. 3 Guidelines for Library Writers In this setion, we bulletize important ideas for library writers based on our previous experiene with the Multiomputer Toolbox and Zipode [3]. Pre grammers who have tried to write libraries with ommon systems will appreiate the simpliity of writing message-passing ode with MPI. 3.1 DO'S 0 Define persistent distributed strutures, based on ommuniators, 0 Dupliate ommuniators for safe ommuniation spae when entering alls to parallel libraries, 0 Use MPI-defined virtual topology information to make algorithms easier to understand, 0 Augment virtual topology tehnology as needed (e.g., topology hierarhies, as in setion 4), 0 Use attribute ahing when adding additional olletive operations to a ommuniator. 3.2 Don'ts 0 Publish how they use the message resoures (e.g., tag allotments), 0 Synhronize exessively at the entry and exit of parallel alls, 0 Guarantee that ommuniators be "quiesent" before they an do olletive operations, 0 Restrit the library programming model to single invoation per proess tehnology, 0 Restrit the library programming model to nonoverlapping proess group assumptions, 0 Write libraries that work only over "all" proessors (proesses) of a user's alloation. 168

4 4 A 2D-Grid Vetor Class Library In this setion we develop a "vetor" library for linear algebra on a twedimensional logial topology and demonstrate two variants of "dot produt" for aligned, dietributed vetors. 4.1 Vetor Layout In a 2D logial grid, vetors may either be olumn or row vetors. We assume that eah row (reap, olumn) vetor is repliated over eah row (resp, olumn) of a logial grid. Figure 1 shows how two row vetors might be dietributed in a grid. Eah vetor is broken into two piees and distributed along the olumns of the grid. Eah proess row has a opy of both vetors. Figure 1: This figure shows the layout of two row vetors over a 2 x 2 logial proess grid. Column vetors are laid out in an analogous, fashion, orthogonal to the row vetors. 4.2 Creating a Logial Grid MPI provides a set of routines whih an be used to reate virtual topologies. In this first example, we show how a library writer might reate a logial 2D grid struture without using the topology routines provided by MPI. Here is the persistent data struture needed to reate a logial grid along with a simple funtion to initialise a grid. /* from tbx-1ibrary.h */ typedef strut tbx-2d-grid < int P, Q; int p, q; MPI-Comm grid-omm; MPI-Comm row-omm; MPI-omm ol-omm; 1 TBX-2D-GRID; /* from tbx,grid. */ /* global shape */ /* position */ TBX-GRID *tbx,build,grid(mpi-comm omm, int P, Q) TBX-2D-GRID *grid; MPI-Comm row, ol; int PD q; /* Determine row and olumn position */ HPI-Comm-rank(omm, hay-rank) ; p = my-rank / 9; q = my-rank X 9; /* Split omm into row and ol omms */ MPI-Comm-split (omm, p, q, &row) ; MPI-Comm-split (omm, q, p, &ol) ; /* Make new grid */ grid = BEW-CLEAR-ITEH(TBX-2D-GRID, 1) ; /* Fill in new grid struture */ grid->grid-omm = omm; grid->row-omm = row; grid->ol-omm = ol; grid->p = P; grid- >q = 9; grid->p = p; grid->q = 9; /* Return the newly built grid */ return (grid); Vetor Struture and Creation Vetors are defined by the following data struture. As a distributed objet, they ontain loal information inluding global and loal lengths, loal real data storage, and a type field that indiates if they are rowor olumn-distributed. In addition, the data struture is based on a logial grid topology data struture, and a, data distribution topology. The former enapsulates 169

5 ommuniators, the latter enapsulates funtions that map global indies to loal indies and vie versa. See Figure 2. /* from tbx-1ibrary.h */ #define TBX-ROW 0 #define TBX-COL 1 typedef strut tbx-dvetor int length; /* full length of vetor */ int loal-length;/* loal vetor length */ int type; /* row or olumn type */ double *data; /* vetor data */ TBX-2D-GRID *grid; TBX-DISTRIB *dis; /* logial grid */ /* how to map data */ /* funtions working on vetors */ double (*dot) (strut tbx-dvetor *x, *y); /*... */ > TBX-DVECTOR; The following funtion is the onstrutor for a die tributed vetor. /* from tbx-dvet0r. */ /* * length = total length of vetor type = row or olumn vetor? * grid = grid whih vetor belongs to * dis = distribution funtions * (how oeffiients map) */ TBX-DVECTOR *new-dvetor(int length, type, TBX-GRID *grid, TBX-DISTRIB *dis) TBX-DVECTOR *ve; int loal-length; /* Create a new vetor */ ve = BEW-CLEAR-ITEM(TBX-DVECTOR,~); /* figure the loal length - i.e., how */ /* many oeffiients (and whih ones) */ /* map to this proess */ loal-length = (*dis -> loal-length) ( grid, dis, length, type); /* Fill in the vetor struture */ ve->length = length; ve->loal-length = loal-length; ve->dat a - HEW-CLEAR-ITEM(double, loal-length) ; ve->type = type; ve->grid = grid; ve->dis = dis; /* assign values to default fns */ ve->dot = ddot-strided; /*... */ 1 Grid to whih vetor belongs Total length of vetor Vetor type -row or olumn I Length of loal part of vetor ~peralions on vetors FUC~~OIW that imp& vetor operations Figure 2: This figure shows the struture of a distributed vetor. 4.4 Dot Produt Funtions High-quality libraries will have to provide more than one version of a funtion that would perform effiiently over different onditions (we all suh a set of methods a poly-algorithm). For example, here we provide two dot produt funtions (strided and nonstrided). The strided version has a redued omputational load at the expense of more ommuniation. Conversely, the non-strided version is omputationally 170

6 more intensive but fewer messages are needed, beause the algorithm exploits data redundany. The most effiient funtion is determined by omparing issues suh aa message lateny and bandwidth, and floating point performane, all with referene to the size of the vetor, and the logial grid dimensions P, Q Non- Strided The non-strided funtion works independently in proess rows (resp, olumns) for row-distributed (resp, olumn-distributed) vetors. First, the loal ontribution to the dot produt is omputed in eah proess, and then, a "ombine" funtion, alled MPI-Allredue() sums the partial sums, leaving the dot produt in eah proess. The olletive ommuniation ours independently in rows (resp, olumns). /* from tbx-dvet0r. */ double ddot-nostride(tbx-dvector *x, *y) int i; double sum = 0.0; /* Chek for ompatible types */ if (x->type!= y->type) C /* erroneous */ 3 /* Sum up my portion of the vetor */ for (i = 0; i < x->loal-length; i++) sum += x->data[i] * y->datacil; /* Get row or olumn sum based on type */ if (x->type == TBX-ROW) MPI-Allredue(Btsum, &sum, 1, MPI-DOUBLE, MPI-SUM, x->grid->rou-omm) ; else MPI-Allredue(&sum, &sum, I, MPI-DOUBLE, MPI-SUM, x->grid->ol-omm); /* Return the result */ return (sum); Strided The strided version of dot produt relies on the fat that there is data repliation. Hene, only a fration of eah sum need be omputed in any proess. However, given striding, the ommuniation required to form the global sum requires MPI-Allredue() to work over all proesses speified by the logial grid, rather than independently in rows or olumns, as above. /* from tbx-dvet0r. */ double ddot-stride(tbx-dvector *x, *y) int i, stride; double sum = 0.0; /* Chek for ompatible types */ if (x->type!= y->type) C /* erroneous */ 3 /* Determine the stride based on type */ if (x->type == TBX-ROW) stride = x->grid->p; else stride = x->grid->q; /* sum up my part */ for ( i = r->grid->p; i < x->loal-length; i += stride) sum += x->data[i] * y->data[il; /* Get the sum of all parts */ MPI-Allredue(Csum, &sum, 1. MPI-DOUBLE, MPI-SUM, x->grid->grid-omm) ; /* Return result */ return( sum) ; 1, 5 The Example Revisited In this setion we revise the "vetor" library for linear algebra on a two-dimensional logial topology and demonstrate two variants of "dot produt" for distributed vetors. This time, we use the MPI-defined Cartesian topologies to help. As before, in a 2D logial grid, vetors may either be olumn or row vetors. Eherything else desribed in the first example remains the same. 5.1 Creating a Logial Grid MPI provides a set of routines whih an be used to reate virtual topologies. The funtion MPI-Make-art () is used for Cartesian topologies. The struture tbr-2d-grid is unhanged from before. However, the funtion tbx-build-grid() makes use of the MPI Cartesian virtual topology funtions, and is somewhat different. 171

7 #define B-DIMS 2 /* from tbr-grid. */ TBX-GRID *tbx,build-grid(mpi-comm omm, int P, 9) I TBX-2D-GRID *grid; NPI-Comm row, ol; int my-r-, P, q; HPI-Comm omm-2d; int dims[lf-dims], /* hold dimensions */ loal[lf-dims], /* loal position */ period[b-dihs] ; /* aperiodi flags */ /* Generate a new ommuniator with virtual topology added: */ dims[o] = P; dims[ll = 9; periodco] = period[l] = FALSE; MPI-Make-art(omm, lf-dims, dims, period, TRUE, Lomm-2d) ; /* Split orm into row and ol omms */ MPI-Comm,rank(omm, &my-rank) ; /* map bak to topology oordinates: */ HPI-Cart-oords(omm~2d,my-ra~k, lf-dims, &loal); p = loal Col ; q = 1oalCil; to send a mail message to netlibooml.gov ontaining the line send index from HPI. Alternatively, one an anonymous ftp the soure from netlib20s.utk.edu. To learn more about full-sale parallel libraries and general distributed data strutures along the lines desribed above, see one or more of the following referenes onerning the Multiomputer Toolbox [4, 5, 6, 7, Conlusions We introdued MPI, the new de fato standard for multiomputer and luster message passing [l]. We desribed the issues underlying the development of parallel libraries with MPI. In fat, we began by motivating the need for parallel libraries per se, and we explained why ommon message-paseing systems do not support them. We desribed added features needed to support suh libraries, and what these features are alled in MPI. We mentioned additional features of MPI that support libraries, but whih are beyond the sope of this initial paper on this topi (see [7]). We provided basi "do's" and Udon'ts" for wouldbe MPI library writers. We presented a linear algebra library in skeletal form on a two-dimensional virtual topology, inluding ode fragments. We summarized by mentioning related work, and omment on future ode development of parallel libraries under MPI, as well as porting efforts of existing odes to the system. /* Make new grid */ grid = lfew-clear-item(tbx-2d-grid, 1); /* Fill in new grid struture */ grid->grid-omm = omm-2d; grid->row-omm = row; grid->ol-omm = ol; grid->p = P; grid->q = 9; grid->p = p; grid->q = 9; /* Return the newly built grid */ return (grid); 1 6 Related Work To find out more about MPI, see the doument [l] available on netlib. A quik way to aess this is Referenes Message Passing Interfae Forum. Doument for a Standard Message-Passing Interfae. Tehnial Report Tehnial Report No. CS , University of Tennessee, November Available on net lib. William Gropp and Ewing Lusk. An Abstrat Devie Definition to Support the Implementation of a High-Level Point-to-Point Message-Passing Interfae. Tehnial Report MCS-P , Mathematis and Computer Siene Division, Argonne National Laboratory, Argonne, IL 60439, Anthony Skjellum. The Design and Evolution of Zipode. Parallel Computing, (Invited Paper, to appear in Speial Issue on Message Passing). Anthony Skjellum. The Multiomputer Toolbox: Current and Future Diretions. In Anthony Skjel- 172

8 lum and Donna S. Reese, editors, Proeedings of the Salable Parallel Libraries Conferene. IEEE Computer Soiety Preee, Otober [5] Anthony Skjellum, Steven F. Ashby, Peter N. Brown, Milo R. Dorr, and Alan C. Hindmarsh. The Multiomputer Toolbox. In G. L. Struble et al., editors, Laboratory Direted Researh and Deuelopment FY91 - LLNL, pages Lawrene Livermore National Laboratory, August UCRL (Rev 1). [6] Anthony Skjellum and Chuk H. Baldwin. The Multiomputer Toolbox: Salable Parallel Libraries for Large-sale Conurrent Appliations. Tehnial Report UCRL-JC , Lawrene Livermore National Laboratory, Deember [7] Anthony Skjellum et al. Salable Libraries Using MPI: Abstration, Performane, Portability. In preparation, [SI Anthony Skjellum, Alvin P. Leung, Charles H. Still Steven G. Smith, Robert D. Falgout, and Chuk H. Baldwin. The Multiomputer Toolbox - First- Generation Salable Libraries. In Proeedings of HICSS-27. IEEE Computer Soiety Press, HICSS-27 Minitrak on Tools and Languages for Transportable Parallel Appliations. [9] Steven G. Smith, Robert D. Falgout, Charles H. Still, and Anthony Skjellum. High-level Messagepassing Construts for Zipode 1.0: Design and Implementation. In Proeedings of the Salable Parallel Libraries Conferene. IEEE Computer Soiety Press,