Parallel Short Course. Distributed memory machines

Transcription

1 Parallel Short Course Message Passing Interface (MPI ) I Introduction and Point-to-point operations Spring 2007 Distributed memory machines local disks Memory Network card 1 Compute node message passing network administrative network Processor Network card 2 2 1

2 Communication between different machines 1st Process (Client) Memory Processor local disks Network card 1 Network card 2 mypc.my-university.edu http request Host Name and Host Address Internet local disks 2nd Process (Server) Memory Processor Network card 1 Network card 2 webserver.provider.com Protocol 3 Communication between different machines on the Internet Addressing: hostname and/or IP Address Communication: based on protocols, e.g. http or TCP/IP Process start-up: every process (= application) has to be started separately 4 2

3 The Message Passing universe Process start-up: Want to start n-processes which shall work on the same problem mechanisms to start n-processes provided by MPI library Addressing: Every process has a unique identifier. The value of the rank is between 0 and n-1. Communication: MPI defines interfaces/routines how to send data to a process and how to receive data from a process. It does not specify a protocol. 5 6 History of MPI Until the early 90 s: all vendors of parallel hardware had their own message passing library Some public domain message passing libraries available all of them being incompatible to each other High efforts for end-users to move code from one architecture to another June 1994: Version 1.0 of MPI presented by the MPI Forum June 1995: Version 1.1 (errata of MPI 1.0) 1997: MPI 2.0 adding new functionality to MPI 3

4 Simple Example (I) MPI command to start process name of the application to start number of processes to be started Number of processes which have been started Rank of the 2nd process Rank of the 1st process 7 Simple example (II) application t1 with rank = 0 PC of the end-user Parallel computer (e.g. PC cluster) mypc> mpirun np 2./t1 application t1 with rank = 1 mpirun starts the application t1 two times (as specified with the np argument) on two currently available processors of the parallel machine telling one process that his rank is 0 and the other that his rank is 1 8 4

5 #include mpi.h Simple Example (III) int main ( int argc, char **argv ) { int rank, size; MPI_Init ( &argc, &argv ); MPI_Comm_rank ( MPI_COMM_WORLD, &rank ); MPI_Comm_size ( MPI_COMM_WORLD, &size ); printf ( Mpi hi von node %d job size %d\n, rank, size); MPI_Finalize (); return (0); 9 MPI basics mpirun starts the required number of processes every process has a unique identifier (rank) which is between 0 and n-1 no identifiers are duplicate, no identifiers are left out all processes which have been started by mpirun are organized in a process group (communicator) called MPI_COMM_WORLD MPI_COMM_WORLD is static number of processes can not change participating processes can not change 10 5

6 MPI basics (II) The rank of a process is always related to the process group e.g. a process is uniquely identified by a tuple (rank, process group) A process can be part of the several groups i.e. a process has in each group a rank MPI_COMM_WORLD, size= new process group, size = Simple Example (IV) Function returns the rank of a process within a process group Rank of a process within the process group MPI_COMM_WORLD ---snip--- MPI_Comm_rank ( MPI_COMM_WORLD, &rank ); MPI_Comm_size ( MPI_COMM_WORLD, &size ); ---snip--- Default process group containing all processes started by mpirun Function returns the size of a process group Number of processes in the process group MPI_COMM_WORLD 12 6

7 Simple Example (V) Function sets up parallel environment: processes set up network connection to each other default process group (MPI_COMM_WORLD) is set up should be the first function executed in the application ---snip--- MPI_Init (&argc, &argv ); ---snip--- MPI_Finalize (); ---snip--- Function closes the parallel environment should be the last function called in the application might stop all processes 13 Second example scalar product of two vectors two vectors are distributed on two processors each process holds half of the original vector Process with rank=0 Process with rank=1 a ( 0... N 1) b ( 0... N 1) a ( N... N) b( N... N)

8 Second example (II) Logical/Global view of the data compared to local view of the data Process with rank=0 Process with rank=1 a ( 0... N 1) a( N... N) 2 2 a local a local a local a local ( 0) a(0) ( 1) a(1) ( 2) a(2) ( n) a( N 1) 2 a ( 0) ( N local a ) 2 a ( 1) ( N local a + 1) 2 a ( 2) ( N local a + 2) 2 a local ( n) a( N 1) 15 Second example (III) Scalar product: s = N 1 i= 0 Parallel algorithm s = N / 2 1 i= 0 N / 2 1 a[ i]* b[ i] ( a[ i]* b[ i]) + N 1 requires communication between the processes i= N / 2 ( a[ i]* b[ i]) N / 2 1 = ( alocal[ i]* blocal[ i]) + ( alocal[ i]* blocal[ i]) i= 0 i= rank = 0 rank =

9 #include mpi.h Second example (IV) int main ( int argc, char **argv ) { int i, rank, size; double a_local[n/2], b_local[n/2]; double s_local, s; MPI_Init ( &argc, &argv ); MPI_Comm_rank ( MPI_COMM_WORLD, &rank ); MPI_Comm_size ( MPI_COMM_WORLD, &size ); s_local = 0; for ( i=0; i<n/2; i++ ) { s_local = s_local + a_local[i] * b_local[i]; 17 Second example (V) if ( rank == 0 ) { /* Send the local result to rank 1 */ MPI_Send ( &s_local, 1, MPI_DOUBLE, 1, 0, MPI_COMM_WORLD); if ( rank == 1 ) { MPI_Recv ( &s, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD, &status ); /* Calculate global result */ s = s + s_local; 18 9

10 Second example (VI) /* Rank 1 holds the global result and sends it now to rank 0 */ if ( rank == 0 ) { MPI_Recv (&s, 1, MPI_DOUBLE, 1, 1, MPI_COMM_WORLD, &status ); if ( rank == 1 ) { MPI_Send (&s, 1, MPI_DOUBLE, 0, 1, MPI_COMM_WORLD); /* Close the parallel environment */ MPI_Finalize (); return (0); 19 Sending Data Data element which shall be send Number of elements which shall be send Data Type of the element which shall be send ---snip--- MPI_Send (&s_local, 1, MPI_DOUBLE, 1, 0, MPI_COMM_WORLD ); ---snip--- Process group containing all processes started by mpirun a user defined integer (tag) for uniquely identifying a message Rank of processes in the process group MPI_COMM_WORLD to which the message shall be sent 20 10

11 Receiving Data Data element where the data shall be received Number of elements which shall be recvd Data Type of the element which shall be recvd ---snip--- MPI_Recv (&s_local, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD, &status ); ---snip--- Process group Status information about the message Rank of processes in the process group which sent the message a user defined integer (tag) for uniquely identifying a message 21 Deadlock (I) Question: how can two processes safely exchange data at the same time? Possibility 1 Process 0 Process 1 MPI_Send(buf, ); MPI_Recv(buf, ); MPI_Send(buf, ); MPI_Recv(buf, ); can deadlock, depending on the message length and the capability of the hardware/mpi library to buffer messages 22 11

12 Deadlock (II) Possibility 2: re-order MPI functions on one process Process 0 Process 1 MPI_Recv(rbuf, ); MPI_Send(buf, ); MPI_Send(buf, ); MPI_Recv(rbuf, ); Other possibilities: asynchronous communication shown later use buffered send (MPI_Bsend) not shown here use MPI_Sendrecv not shown here 23 marvin.cs.uh.edu Login name: gabriel Password: haha99/1 mkdir cd cp../helloworldf.f. mpicc o helloworld helloworld.c mpif77 o helloworldf helloworldf.f mpirun np 2./helloworldf 24 12

13 Faulty examples (I) Sender mismatch: if rank does not exist (e.g. rank > size of MPI_COMM_WORLD), the MPI library can recognize it and return an error if rank does exist (0<rank<size of MPI_COMM_WORLD) but does not send a message => MPI_Recv waits forever => deadlock if ( rank == 0 ) { /* Send the local result to rank 1 */ MPI_Send ( &s_local, 1, MPI_DOUBLE, 1, 0, MPI_COMM_WORLD); if ( rank == 1 ) { MPI_Recv ( &s, 1, MPI_DOUBLE, 5, 0, MPI_COMM_WORLD, &status ); 25 Faulty examples (II) Tag mismatch: if tag outside of the allowed range (e.g. 0<tag<MPI_TAG_UB ) the MPI library can recognize it and return an error if tag in MPI_Recv then the tag specified in MPI_Send => MPI_Recv waits forever => deadlock if ( rank == 0 ) { /* Send the local result to rank 1 */ MPI_Send ( &s_local, 1, MPI_DOUBLE, 1, 0, MPI_COMM_WORLD); if ( rank == 1 ) { MPI_Recv ( &s, 1, MPI_DOUBLE, 0, 18, MPI_COMM_WORLD, &status ); 26 13

14 What you ve learned so far Six MPI functions are sufficient for programming a distributed system memory machine MPI_Init(int *argc, char ***argv); MPI_Finalize (); MPI_Comm_rank (MPI_Comm comm, int *rank); MPI_Comm_size (MPI_Comm comm, int *size); MPI_Send (void *buf, int count, MPI_Datatype dat, int dest, int tag, MPI_Comm comm); MPI_Recv (void *buf, int count, MPI_Datatype dat, int source, int tag, MPI_Comm comm, MPI_Status *status); 27 So, why not stop here? Performance need functions which can fully exploit the capabilities of the hardware need functions to abstract typical communication patterns Usability need functions to simplify often recurring tasks need functions to simplify the management of parallel applications 28 14

15 So, why not stop here? Performance asynchronous point-to-point operations one-sided operations collective operations derived data-types parallel I/O hints Usability process grouping functions environmental and process management error handling object attributes language bindings 29 Point-to-point operations Data exchange between two processes both processes are actively participating in the data exchange two-sided communication Large set of functions defined in MPI-1 (50+) Blocking Non-blocking Persistent Standard MPI_Send MPI_Isend MPI_Send_init Buffered MPI_Bsend MPI_Ibsend MPI_Bsend_init Ready MPI_Rsend MPI_Irsend MPI_Rsend_init Synchronous MPI_Ssend MPI_Issend MPI_Ssend_init 30 15

16 A message contains of the data which is to be sent from the sender to the receiver, described by the beginning of the buffer a data-type the number of elements of the data-type the message header (message envelope) rank of the sender process rank of the receiver process the communicator a tag 31 Rules for point-to-point operations Reliability: MPI guarantees, that no message gets lost Non-overtaking rule: MPI guarantees, that two messages posted from process A to process B arrive in the same order as they have been posted Message-based paradigm: MPI specifies, that a single message cannot be received with more than one Recv operation (in contrary to sockets!) Message Recv buffer1 Recv buffer2 Message in the Recv buffers if (rank == 0 ) { MPI_Send(buf, 4, ); if ( rank == 1 ) { MPI_Recv(buf, 3, ); MPI_Recv(&(buf[3],1, ); 32 16

17 Message matching (I) How does the receiver know, whether the message which he just received is the message for which he was waiting? the sender of the arriving message has to match the sender of the expected message the tag of the arriving message has to match the tag of the expected message the communicator of the arriving message has to match the communicator of the expected message? 33 Message matching (II) What happens if the length of the arriving message does not match the length of the expected message? the length of the message is not used for matching if the received message is shorter than the expected message, no problems the received message is longer than the expected message an error code (MPI_ERR_TRUNC) will be returned or your application will be aborted or your application will deadlock or your application writes a core-dump 34 17

18 Message matching (III) Example 1: correct example if (rank == 0 ) { MPI_Send(buf, 3, MPI_INT, 1, 1, MPI_COMM_WORLD); else if ( rank == 1 ) { MPI_Recv(buf, 5, MPI_INT, 0, 1, MPI_COMM_WORLD, &status); Message Recv buffer untouched elements in the recv buffer Message in the Recv buffer 35 Message matching (IV) Example 2: erroneous example if (rank == 0 ) { MPI_Send(buf, 5, MPI_INT, 1, 1, MPI_COMM_WORLD); else if ( rank == 1 ) { MPI_Recv(buf, 3, MPI_INT, 0, 1, MPI_COMM_WORLD, &status); Message Recv buffer potentially writing over the end of the recv buffer Message in the Recv buffer 36 18

19 Example Implementation of a ring using Send/Recv Rank 0 starts the ring MPI_Comm_rank (comm, &rank); MPI_Comm_size (comm, &size); if (rank == 0 ) { MPI_Send(buf, 1, MPI_INT, rank+1, 1,comm); MPI_Recv(buf, 1, MPI_INT, size-1, 1,comm,&status); else if ( rank == size-1 ) { MPI_Recv(buf, 1, MPI_INT, rank-1, 1,comm,&status); MPI_Send(buf, 1, MPI_INT, 0, 1,comm); else { MPI_Recv(buf, 1, MPI_INT, rank-1, 1,comm,&status); MPI_Send(buf, 1, MPI_INT, rank+1, 1,comm); 37 Wildcards Question: can I use wildcards for the arguments in Send/Recv? Answer: for Send: no for Recv: tag: yes, MPI_ANY_TAG source: yes, MPI_ANY_SOURCE communicator: no 38 19

20 Status of a message (I) the MPI status contains directly accessible information who sent the message what was the tag what is the error-code of the message and indirectly accessible information through function calls how long is the message has the message bin cancelled 39 Status of a message (II) usage in C MPI_Status status; MPI_Recv ( buf, cnt, MPI_INT,, &status); /*directly access source, tag, and error */ src = status.mpi_source; tag = status.mpi_tag; err = status.mpi_error; /*determine message length and whether it has been cancelled */ MPI_Get_count (status, MPI_INT, &rcnt); MPI_Test_cancelled (status, &flag); 40 20

21 Status of a message (III) usage in Fortran integer status(mpi_status_size) call MPI_Recv(buf,cnt,MPI_INTEGER,,status,ierr)! directly access source, tag, and error src = status(mpi_source) tag = status(mpi_tag) err = status(mpi_error)! determine message length and whether it has been! cancelled call MPI_Get_count (status,mpi_integer,rcnt,ierr) call MPI_Test_cancelled (status,flag,ierr) 41 Status of a message (IV) If you are not interested in the status, you can pass MPI_STATUS_NULL MPI_STATUSES_NULL to MPI_Recv and all other MPI functions, which return a status 42 21

22 Non-blocking operations (I) A regular MPI_Send returns, when the data is safely stored away A regular MPI_Recv returns, when the data is fully available in the receive-buffer Non-blocking operations initiate the Send and Receive operations, but do not wait for its completion. Functions, which check or wait for completion of an initiated communication have to be called explicitly Since the functions initiating communication return immediately, all MPI-functions have an I prefix (e.g. MPI_Isend or MPI_Irecv). 43 Non-blocking operations (II) MPI_Isend (void *buf, int cnt, MPI_Datatype dat, int dest, int tag, MPI_Comm comm, MPI_Request *req); MPI_Irecv (void *buf, int cnt, MPI_Datatype dat, int src, int tag, MPI_Comm comm, MPI_Request *reqs); 44 22

23 Non-blocking operations (III) After initiating a non-blocking communication, it is not allowed to touch (=modify) the communication buffer until completion you can not make any assumptions about when the message will really be transferred All Immediate functions take an additional argument, a request a request uniquely identifies an ongoing communication, and has to be used, if you want to check/wait for the completion of a posted communication 45 MPI_Wait MPI_Waitall MPI_Waitany Completion functions (I) (MPI_Request *req, MPI_Status *stat); (int cnt, MPI_Request *reqs, MPI_Status *stats); (int cnt, MPI_Request *reqs, int *index, MPI_Status *stat); MPI_Waitsome (int cnt, MPI_Request *reqs, int *indices, MPI_Status *stats); Functions waiting for completion MPI_Wait wait for one communication to finish MPI_Waitall wait for all comm. of a list to finish MPI_Waitany wait for one comm. of a list to finish MPI_Waitsome wait for at least one comm. of a list Content of the status not defined for Send operations 46 23

24 Completion functions (II) MPI_Test (MPI_Request *req, int *flag, MPI_Status *stat); MPI_Testall (int cnt, MPI_Request *reqs, int *flag, MPI_Status *stats); MPI_Testany (int cnt, MPI_Request *reqs, int *index, int *flag, MPI_Status *stat); MPI_Testsome (int cnt, MPI_Request *reqs, int *indices, int *flag, MPI_Status *stats); Test-functions verify, whether a communication is complete MPI_Test check, whether a comm. has finished MPI_Testall check, whether all comm. of a list finished MPI_Testany check, whether one of a list of comm. finished MPI_Testsome check, how many of a list of comm. finished 47 Deadlock problem revisited Question: how can two processes safely exchange data at the same time? Possibility 3: usage of non-blocking operations Process 0 Process 1 MPI_Irecv(rbuf,, &req); MPI_Send (buf, ); MPI_Wait (req, &status); MPI_Irecv(rbuf,,&req); MPI_Send (buf, ); MPI_Wait (req, &status); note: you have to use 2 separate buffers! many different ways for formulating this scenario identical code for both processes 48 24

25 Some Links MPI Forum: My personal MPI home page: Open MPI: MPICH: