A few words about MPI (Message Passing Interface) T. Edwald 10 June 2008

Transcription

1 A few words about MPI (Message Passing Interface) T. Edwald 10 June

2 Overview Introduction and very short historical review MPI - as simple as it comes Communications Process Topologies (I have no experience with this) MPI-2, OpenMP (I have no experience with this) Summary 2

3 Intro: MPI (version 1) MPI provides an interface that allows processes in a parallel program to communicate with one another. MPI specifies neither how the processes are created, nor how they establish communication. Moreover, an MPI application is static, that is, no processes can be added to or deleted from an application after it has been started. This is a stumbling block when porting PVM programs, for instance. 3

4 Intro: MPI 1, 2,... history The Message Passing paradigm was widely used and understood by 1992, but vendors had their own variants. It is now the most commonly used model for parallel programming on distributed-memory architectures. Lack of standards hampered progress A group of interested parties defined the MPI 1 Specification (which later was clarified to version 1.2) This was a Specification, intended to be practical, portable, efficient, etc., but not an Implementation, and not language specific. UIs specified for Fortran and C. MPICH was the first available implementation. Around 2004, MPI-2 (2.1) had been solidified, but this is not universally used. Last year (2007) review work started, which may lead to MPI-3. 4

5 Intro: Message Passing libraries On most distributed memory MIMD machines information must be exchanged between processors to accomplish anything Data distribution and communication are managed explicitly by the programmer using SEND and RECEIVE subroutines This can be very tedious, often extremely hard to debug, and requires a lot of thought... Programs are more ad-hoc in this domain, I find 5

6 Intro: Problem Decomposition (program design) Domain Decomposition (data parallelism) where the data is divided into roughly equi-sized parts and farmed out to the available processors. Each processor works only on the area assigned to him, but may need information from neighboring processes, and they may need to communicate periodically to exchange information. Functional Decomposition (task parallelism) where the problem is decomposed into a large number of smaller tasks, and the tasks are assigned to processors as they become available (or have low task load). This is very efficient, obviously, but does not apply to all types of problems (equally obviously). This is usually implemented in a client-server paradigm, where one process takes the role of Master, and assigns subtasks to n-1 Slave nodes, and accumulates their partial results for presentation. 6

7 Intro: Message Passing libraries Programmer writes code consisting of standard serial language (C/ Fortran) + calls to message passing subroutines Programmer s code is linked with (MPI) library Basic message passing usually includes: sends; blocking and non-blocking versions receives; blocking and non-blocking versions packing and unpacking information in buffers grouping functions: broadcasts, and gatherings 7

8 Intro:! MPI 1 (1.2) does have... Point-to-Point communication Collective Communication Routines Support for Process Groups Support for Communication Contexts Support for Process Topologies Bindings for Fortran, C, Environmental inquiry routines 8

9 Intro:! MPI 1 (1.2) does NOT have... Explicit shared-memory operations Interrupt-driven receives One-Sided control over communication Process management Remote memory transfers While MPI does not address these issues, it attempts to remain compatible. Explicit support for threads Debugging facilities I/O functions Initial implementation subset 9

10 Intro:! MPI volume ( required MPI is small (six functions are MPI is large MPI-1.2 has some 125 functions Fortran and C bindings used Typically, some of those are MPI-2.1 has over 500 functions (!) ANSI Fortran, ANSI C, and ANSI C++ bindings MPI is just right... Flexibility can be accessed when it is required We don t need to understand it all before we can use it Now we have bindings for Perl, Python, Java,... 10

11 Simple MPI: start/stop #include mpi.h for basic MPI definitions and types All MPI programs must use: () MPI_Init Starts up MPI system and must be the first MPI call () MPI_Finalize Exits MPI and must be the last MPI call in a program 11

12 Simple MPI: example #include mpi.h #include <stdio.h> int main( int argc, char **argv ) {! MPI_Init( &argc, &argv );! printf( Halló, heimur!\n );! MPI_Finalize();! return(); } # nb: cannot assume printf is available everywhere # This was a bit contrived... 12

13 Simple MPI: Environment The first thing an MPI program wants to know, is How many of us are running? Who am I? Note that these may vary from one run to the next MPI_Comm_size answers the first question MPI_Comm_rank answers the second 13

14 Simple MPI: Example 2 #include mpi.h #include <stdio.h> int main( int argc, char **argv ) {! int rank, size;! MPI_Init( &argc, &argv );! MPI_Comm_rank( MPI_COMM_WORLD, &rank );! MPI_Comm_size( MPI_COMM_WORLD, &size );! printf( Halló, heimur! Ég er %d af %d\n, rank, size );! MPI_Finalize();! return(); } #( MPI_COMM_WORLD is default Communicator and contains everything you need to communicate with default == all possible recipients, ie. the World ) 14

15 Simple MPI: Message Passing MPI has many point-to-point send/receive functions. These are the basic blocking ones: MPI_Send( buf, count, datatype, dest, tag, comm ) MPI_Recv( buf, count, datatype, source, tag, comm ) comm is a 'communicator' that specifies which processes can be addressed by source and dest MPI_COMM_WORLD == all possible processes 15

16 Simple MPI: Six functions suffice MPI is very simple. These six functions allow one to write useful MPI programs. () MPI_Init () MPI_Comm_size () MPI_Comm_rank () MPI_Send () MPI_Recv MPI_Finalize 16

17 Communication: Point-to-Point Messages may be sent between pairs of processes, with message selectivity based on source process, message tag and communication context Processes can execute their own code, sequential or multi-threaded No explicit support for threads, but care has been taken to remain threadsafe. Large range of functionality, but actual number of routines kept manageable High-level functions can be constructed from a small number of primitive operations (around 16) 17

18 Communication: Point-to-Point: blocking SEND MPI_Send( buf, count, datatype, dest, tag, comm ) creates a message, taking its data from the send buffer, buf. The send buffer buf consists of count successive entries of type datatype, starting at address buf Data type may be basic, eg. MPI_INTEGER, MPI_REAL,... or derived Each process within a group has an integer rank, starting from zero. Destination, dest, is the rank of the receiving process. The tag field is an integer value which can be set arbitrarily by the application, to some meaningful value The communicator, comm, designates a process group and context for communication. For simple use, MPI_COMM_WORLD is predefined. Blocks until safe to reuse send buffer 18

19 Communication: Point-to-Point: blocking RECEIVE MPI_Recv( buf, count, datatype, source, tag, comm, status ) Consumes a message and places its data into the receive buffer, which is of size count, and a received message must fit The source, tag and context fields of a message must match those specified The receiver may specify MPI_ANY_SOURCE for the sources, or MPI_ANY_TAG for the tag, but may not wildcard the context field. The type of status is MPI-defined C-structure or Fortran-array, and can be decoded (contains number of elements received among other things) MPI automatically handles data representation issues between architectures (hton,ntoh,..), excluding MPI_BYTE which is not converted, to pass binary data 19

20 Simple MPI: example 3 #include<stdio.h> #include<mpi.h> int main(int argc, char ** argv){ int mynode, totalnodes; int sum,startval,endval,accum; int i,j; MPI_Status status; MPI_Init(&argc,&argv); MPI_Comm_size(MPI_COMM_WORLD, &totalnodes); MPI_Comm_rank(MPI_COMM_WORLD, &mynode); sum = 0; startval = 1000*mynode/totalnodes+1; endval = 1000*(mynode+1)/totalnodes; for( i=startval; i<=endval; i++) { sum += i; } if(mynode!=0) { printf( "[%d] Sending sum of %d to node 0\n", mynode, sum ); MPI_Send(&sum,1,MPI_INT,0,1,MPI_COMM_WORLD); } else { } printf( "[0] Have my own sum: %d...\n", sum ); for( j=1; j<totalnodes; j++ ) { MPI_Recv(&accum,1,MPI_INT,j,1,MPI_COMM_WORLD, &status); printf( "[0]... and receiving %d from node %d\n", accum, j ); sum += accum; } if( mynode == 0 ) { printf( "The (%d)-sum from 1 to 1000 is: %d\n", totalnodes, sum ); } MPI_Finalize(); } 20

21 Simple MPI: Example 3, output bash-3.2$ gcc = -o daemi3 daemi3.c -lmpi bash-3.2$ mpirun -np 8./daemi3 [1] Sending sum of to node 0 [0] Have my own sum: [0]... and receiving from node 1 [0]... and receiving from node 2 [0]... and receiving from node 3 [0]... and receiving from node 4 [0]... and receiving from node 5 [2] Sending sum of to node 0 [3] Sending sum of to node 0 [4] Sending sum of to node 0 [5] Sending sum of to node 0 [7] Sending sum of to node 0 [6] Sending sum of to node 0 [0]... and receiving from node 6 [0]... and receiving from node 7 The (8)-sum from 1 to 1000 is: Where does it go? Which machines? We don t care, in this instance; we assume the configuration of available hosts and transport mechanisms used has been sorted out by the system admins 21

22 Communication: nonblocking SEND and RECEIVE There are non-blocking versions of Send and Recv, MPI_ISEND and MPI_IRECV (I: Immediate execution) which return a handle that can be queried for status. There are four modes to a send operation: Standard, Buffered, Synchronous, and Ready, which vary on the presence or absence of a Recv-call, and buffer space. There are other tools for process status and control, as one might expect. (Wait, test,...) There are ways to check for incoming messages without actually receiving them formally (like peeking on a stack) One can reject or cancel communication (The above needed for graceful shutdown) These concepts are too detailed for a quick intro, but basically you will find what you need in terms of process and communication control 22

23 Communication: Collective Communication Communication that involves all members of a group - MPI_WORLD_COMM- Following operations: Barrier Broadcast Gather Scatter Reduce Scan - across all members of group - from one to all members - from all to one - from one to all members - (sum,max,min,..) of group results - across all members of group All_Broadcast - all members broadcast at once All_gather - all members gather at once (also called Complete Exchange ) 23

24 Communication: Collective Communication Collective communication is layered on top of point-to-point routines Broadcast Process sends data (A0) to all processes which now have a copy 24

25 Communication: Collective Communication 25

26 Communication: Collective Communication All gather All of the processes A through F have gathered all data from all the others 26

27 Communication: Process Groups MPI does not use absolute process names Processes are identified by their rank within a group Universal group including all processes exists at startup A group is an object representing an ordered set of process identifiers A Communication Context (CC) is the MPI mechanism for partitioning communication space A message sent in one context cannot be received in another CCs allow parallel modules, developed separately, to be used together safely without any modifications (think name spaces ) CCs are concealed within communicators and may not be manipulated directly 27

28 Communication: Communicators (Reagan?) Identify the process group and communication context of an operation Are explicit parameters in any p2p and collective routine Accessors exist to obtain group, size, rank from communicator object Communicators may be constructed (-DUP-) and destroyed Enable inter-group communications but then not for collective comms) Either inter- or intra-group communications, not both 28

29 Process Topologies (I have no experience of these myself) Although MPI provides message passing between arbitrary pairs of processes, parallel application programs often have communication patterns as simple as two or 3-D grids MPI allows the user to specify the logical arrangement or virtual topology of processes within a group MPI_GRAPH_CREATE - define general graphs MPI_CART_CREATE - define cartesian structures -of arbi-, trary dimension: rings, grids and torii more 29

30 MPI2 Clarifications to the original MPI standard were released 1995, and the MPI Forum began work on extensions to the standard, culminating in the release of MPI2 in Includes: Process Creation and management One-sided communications Extended Collective operations External interfaces I/O Additional language bindings Miscellaneous topics 30

31 OpenMP, OpenMPI, MVAPICH,... OpenMP!= OpenMPI The emerging standard of OpenMP is a portable base for the development of libraries for shared memory machines. OpenMPI is an Open-Source implementation of MPI-2 Implementations vary on their support for architectures and hardware-related issues, such as transport mechanisms (MVAPICH) Cluster environments may vary in their support for shared memory and other such issues, such as the inclusion of SMP nodes All modern implementations offer APIs for Fortran-77, -90, C and C++ Multi-vendor support, for Unix (of course) and even Win 31

32 Summary MPI has become the standard for message passing programming It is practical, portable, efficient and flexible an has been implemented on a wide range of systems It is freely available, and support is generally as easy as a googlesearch or request. The standard documents are available from: has a plethora of links is also a source self-help course material available at ci-tutor.ncsa.uiuc.edu 32