Message Passing Interface

Transcription

1 Message Passing Interface by Kuan Lu Scientific researcher at Georg-August-Universität Göttingen and Gesellschaft für wissenschaftliche Datenverarbeitung mbh Göttingen Am Faßberg, Göttingen, Germany Phone: (+49) Fax: (+49)

2 Literature Patterns for Parallel Programming : by Mattson, Sanders, Massingill Parallel Programming / Parallele Programmierung : by Rauber, Rünger Parallel Programming : by Peter S. Pacheco 2

3 Overview Shared Memory vs. Distributed Memory - System-Architecture MPI Message Passing Interface - What is MPI? - The Message-Passing Model - Basis of MPI Program with Example - MPI Communication Modes - Point-to-Point Communication Mode - Collective Communication Mode - Others Summary 3

4 System-Architecture Distributed memory Shared memory P0 Pn-1 P0 Pn-1 M0 Network Mn-1 Memory + high scalability complicated programming MPI + simple programming low scalability OpenMP 4

5 What is MPI? MPI stands for Message Passing Interface Standardized message passing interface specification No special hardware required, rather in OS level. Supports various programming languages (C/C++, Fortran-77/-95, Java) Frontend (API) is same, Backend (Library) different Different distributions Gaining the optimization of corresponding hardware E.g., MPICH, MVARPICH, LAM/MPI, SCAMPI, OpenMPI 5

6 What is MPI? Computer C compiler OpenMP MPI OpenMP library is part of C compiler. On the contrary, MPI library has to be installed into C explicitly. 6

7 What is MPI? MPICH1 MPICH1 MPICH1 MPICH1 MPICH2 MPICH1 MPICH2 MPICH1 MPICH2 Dynamical MPICH1 MPICH2 MPICH2 Dynamical Process I/O Process Dynamical MPICH1 parallel I/O Process Dynamical MPICH1 parallel I/O Process Dynamical parallel I/O Process parallel I/O parallel First Version Second Version 7

8 The Message-Passing Model Process = program counter + address space Message Passing = Communication among processes with separate memory space Core aspects - Data exchanging - Synchronization Distributed memory P0 Pn-1 M0 Mn-1 Network 8

9 The Message-Passing Model In MPI, each of the processes could execute - its own program (Multiple Program Multiple Data) - the same program (Single Program Multiple Data) By rank, each process can execute different parts of the program. For instance, starting node does some initializing work. No restriction for process number in MPI, but depending on the number of processors, which is changing annually. Usually each processor is corresponding to one process. 9

10 Basis of MPI Program All identifiers defined by MPI start with MPI_XXX. MPI_Init(int *argc, char*** argv) - Setup of MPI system, Communicator - Allocate storage for message buffers - Decide which process gets which rank MPI_Finalize(void) - MPI program is finished - Free all MPI related resources - No MPI functions should be called after it! 10

11 Basis of MPI Program MPI_COMM_WORLD - Default communicator in MPI, a group of processes - Additional communicator can be created MPI_Comm_size(MPI_Comm comm, int* size) - Retrieve the number of process of given communicator - e.g., MPI_Comm_size(MPI_COMM_WORLD, &size) MPI_Comm_rank(MPI_Comm comm, int* rank) - Retrieve the rank of current process - 0 rank size-1 11

12 SPMD Example in C #include mpi.h" #include <stdio.h> int main( int argc, char *argv[] ) { int rank, size; MPI_Init( &argc, &argv ); MPI_Comm_rank( MPI_COMM_WORLD, &rank ); MPI_Comm_size( MPI_COMM_WORLD, &size ); printf( "I am %d of %d\n", rank, size ); MPI_Finalize(); return 0; } 12

13 MPMD Example in C #include mpi.h #include <stdio.h> int main( int argc, char *argv[] ) { int rank, size; MPI_Init( &argc, &argv ); MPI_Comm_rank( MPI_COMM_WORLD, &rank ); MPI_Comm_size( MPI_COMM_WORLD, &size ); if (rank == 0) printf("i am the master!\n"); else printf("worker with rank %d\n", rank); MPI_Finalize(); return 0; } 13

14 Compilation and Execution of MPI Program Compilation of MPI program uses mpicc wrapper script Execution of MPI program uses mpirun or mpiexec wrapper scripts, depending on the system Examples: mpicc -o HelloWorld.out helloworld.c mpirun n 8./HelloWorld.out mpiexec n 8./HelloWorld.out 14

15 Execution of MPI in PBS Environment Compilation of MPI program uses mpicc wrapper script Create a shell script, e.g., helloworld.sh, with the following content: #PBS N mpi\_hello #PBS l nodes=8 /usr/bin/mpiexec /var/local/torque/pbsuser7/mpi\_rings/helloworld.out Submit the script to the PBS Cluster with pbsuserx, for instance as following: qsub helloworld.sh 15

16 Semantic Terms in MPI Local sending and receiving buffers - The local buffers that store the send and receive messages System buffer - MPI system decides whether system buffer is used or not - Program should also work, when no system buffer is used 16

17 Blocking and Non-blocking Operations Blocking: The return of the operation on a process ensures that the local resources (buffers) can be reused Non-Blocking: The operation on a process returns without guarantee that the local resources (buffers) can be reused Describe of operations from the local view of the process 17

18 Synchronous and Asynchronous Communications Synchronous communication: both send and receive operations in two processes should start synchronously Asynchronous communication: both send and receive operations can start asynchronously Describe of operations from the global view between the processes 18

19 Combination Blocking Non-blocking Synchronous Asynchronous MPI_Ssend() MPI_Recv()! MPI_Send() MPI_Recv() MPI_Isend() MPI_Irecv() 19

20 MPI Communication Modes Point-to-Point (P2P) Communication Mode - Functions such as MPI_Send()and MPI_Recv() Collective Communication Mode - Communication functions that involve all the processes in a communicator - Abstracted functions that cover MPI_Send()and MPI_Recv()inside, more safe - Always blocking, no tag is used 20

21 P2P Send and Receive Operations Basic Pattern Process 0 Process 1 Send(data) Receive(data) Open questions: - How to define the data? - How to receive the data? - How are the processes identified? - When are the operations finished? 21

22 MPI (Blocking) Send MPI_Send(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) Message(buf, count, datatype) Receiver (rank) dest tag can be used by the receiver to distinguish different messages from the same sender Returns, when the message is passed to the system: Buffer can be reused again (filled with new message) It is however not necessary that receiver must start receiving operation Example MPI_Send(&message, 1, MPI_DOUBLE, rank, 1, MPI_COMM_WORLD) 22

23 Predefined MPI Data Types MPI_CHAR MPI_SHORT MPI_INT MPI_LONG MPI_LONG_LONG MPI_UNSIGNED_CHAR MPI_UNSIGNED_SHORT MPI_UNSIGNED MPI_UNSIGNED_LONG MPI_FLOAT MPI_DOUBLE MPI_LONG_DOUBLE MPI_BYTE MPI_PACKED Signed char Signed short int Signed int Signed long int Signed long long int Unsigned char Unsigned short int Unsigned int Unsigned long int float double long double

24 MPI (Blocking) Receive MPI_Recv(void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Status *status) Message(buf, count, datatype) Returns, when receive buffer completely contains the message (source and tag) - Buffer can be reused again (reading the message) source is sender or wildcard MPI_ANY_SOURCE tag is Tag or wildcard MPI_ANY_TAG count can be smaller than the message (not greater à Error) status Information about the received message. E.g., the size information, rank of sending process Example MPI_Recv(&message,1,MPI_DOUBLE,rank,1,MPI_COMM_WORLD,&status) 24

25 P2P Send and Receive Operations Both standard methods are blocking and asynchronous They are P2P communications MPI_Recv returns, when data buffer completely contains the message MPI_Send is blocked until the message is received completed by system buffer or receiver 25

26 P2P MPI is simple 6 basic functions can handle many parallel programs - int MPI_Init(int *argc, char **argv) - int MPI_Comm_size(MPI_Comm comm, int *size) - int MPI_Comm_rank(MPI_Comm comm, int *rank) - int MPI_Send(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) - int MPI_Recv(void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Status *status) - int MPI_Finalize(void) 26

27 P2P Send and Receive Operations For P2P MPI, the simpler, the more complicated. High frequent problem in P2P: Deadlock! 27

28 Example: Deadlock (Always!!) #include mpi.h #include <stdio.h> P1 int main(int argc, char **argv) { int me, np, q, sendto; MPI_Status status; MPI_Init(&argc, &argv); P0 P2 MPI_Comm_size(MPI_COMM_WORLD, &np); MPI_Comm_rank(MPI_COMM_WORLD, &me); P3 if (np%2==1) return 0; // should be send/recv pair! if (me%2==1) {sendto = me-1;} // the previous neighbor is receiver. else {sendto = me+1;} // the next neighbor is receiver MPI_Recv(&q, 1, MPI_INT, sendto, sendto, MPI_COMM_WORLD, &status); MPI_Send(&me, 1, MPI_INT, sendto, me, MPI_COMM_WORLD); printf( Sent %d to proc %d, received %d from proc %d\n, me, sendto, q, sendto); MPI_Finalize(); return 0; } 28

29 Why? Deadlock (Always) - Because of blocking operation of MPI_Recv() - All processes wait for the corresponding MPI_Send() 29

30 Example: Deadlock (Sometimes) #include mpi.h P1 #include <stdio.h> int main(int argc, char **argv) { int me, np, q, sendto; P0 MPI_Status status; MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD, &np); P3 MPI_Comm_rank(MPI_COMM_WORLD, &me); if (np%2==1) return 0; if (me%2==1) {sendto = me-1;} else {sendto = me+1;} MPI_Send(&me, 1, MPI_INT, sendto, me, MPI_COMM_WORLD); MPI_Recv(&q, 1, MPI_INT, sendto, sendto, MPI_COMM_WORLD, &status); printf( Sent %d to proc %d, received %d from proc %d\n, me, sendto, q, sendto); MPI_Finalize(); return 0; } P2 30

31 Why? Deadlock (Sometimes) - Because of blocking operation of MPI_Send(), which depends on the available space of system buffer - Unsafe - Deadlock? When using MPI_Ssend()? - Using MPI_Ssend() for testing 31

32 Example: Deadlock (Never) #include mpi.h #include <stdio.h> P1 int main(int argc, char **argv) { int me, np, q, sendto; MPI_Status status; MPI_Init(&argc, &argv); P0 MPI_Comm_size(MPI_COMM_WORLD, &np); MPI_Comm_rank(MPI_COMM_WORLD, &me); if (np%2==1) return 0; P3 if (me%2==1) {sendto = me-1;} else {sendto = me+1;} if (me%2 == 0) { MPI_Send(&me, 1, MPI_INT, sendto, me, MPI_COMM_WORLD); MPI_Recv(&q, 1, MPI_INT, sendto, sendto, MPI_COMM_WORLD, &status); } else { MPI_Recv(&q, 1, MPI_INT, sendto, sendto, MPI_COMM_WORLD, &status); MPI_Send(&me, 1, MPI_INT, sendto, me, MPI_COMM_WORLD); } printf( Sent %d to proc %d, received %d from proc %d\n, me, sendto, q, sendto); MPI_Finalize(); return 0; } P2 32

33 Why? Solution - put MPI_Send and MPI_Recv in pairs 33

34 System Buffer Process 0 Process 1 User data Send buffer System buffer Receive buffer Using blocking MPI_Send, MPI_Send is blocked until the message is received completed by system buffer User data 34

35 Avoid System Buffer Process 0 Process 1 User data Send buffer Receive buffer Using blocking MPI_Send, MPI_Send is blocked until the message is received completed by receiver User data 35

36 MPI Non-Blocking Operations Return immediately after calling Handling test and wait MPI_Isend(start, count, datatype, dest, tag, comm, request) MPI_Irecv(start, count, datatype, dest, tag, comm, request) MPI_Wait(&request, &status) Test without wait MPI_Test(&request, &flag, &status) 36

37 More Operations MPI_Waitall(count, array_of_requests, array_of_statuses) MPI_Waitany(count, array_of_requests, &index, &status) MPI_Waitsome(count, array_of_requests, array_of indices, array_of_statuses) 37

38 Collective Communication in MPI The communication of all the processes in a group MPI Collective Functions MPI_Bcast() MPI_Reduce() MPI_Gather() MPI_Scatter() MPI_Allgather() MPI_Alltoall() Remarks Broadcast operations Accumulation operations Gather operations Scatter operations Multi-broadcast operation Total exchange 38

39 Collective Communication in MPI Broadcast In many cases, one processor needs to send (broadcast) some data (either a scalar or vector) to all the processes in a group. MPI provides the broadcast primitive MPI_BCAST to accomplish this task. The syntax of the MPI_BCAST is given by C int MPI_Bcast (void* send_buffer, int count, MPI_Datatype datatype, int src_process, MPI_Comm comm) 39

40 Broadcast P0 P1 A Broadcast A A P2 A P3 A 40

41 Collective Communication in MPI Scatter The root process sends the contents of its send buffer by average to each process. The syntax of the MPI_Scatter is given by C int MPI_Scatter (void* send_buffer, int send_count, MPI_Datatype send_datatype, void* receive_buffer, int receive_count, MPI_Datatype receive_datatype, int src_process, MPI_Comm comm) 41

42 Collective Communication in MPI Gather, reverse of Scatter operation Each process sends the contents of its send buffer to the root process. Then the root process receives the messages and stores them in rank order. The syntax of the MPI_Gather is given by C int MPI_Gather (void* send_buffer, int send_count, MPI_Datatype send_datatype, void* receive_buffer, int receive_count, MPI_Datatype receive_datatype, int dest_process, MPI_Comm comm) 42

43 Scatter, Gather P0 P1 P2 P3 A B C D Scatter Gather A B C D 43

44 Collective Communication in MPI Allgather Same as Gather, except that all processes, not just root, receive the result. The syntax of the MPI_Allgather is given by C int MPI_Allgather (void* send_buffer, int send_count, MPI_Datatype send_datatype, void* receive_buffer, int receive_count, MPI_Datatype receive_datatype, MPI_Comm comm) 44

45 Allgather P0 P1 A B Allgather A B C D A B C D P2 C A B C D P3 D A B C D 45

46 Collective Communication in MPI Alltoall Extension to Allgather Each process sends distinct data to each receiver. Process i sends its jth block of data to process j Process j stores data from process i in ith block The syntax of the MPI_Alltoall is given by C int MPI_Alltoall (void* send_buffer, int send_count, MPI_Datatype send_datatype, void* receive_buffer, int receive_count, MPI_Datatype receive_datatype, MPI_Comm comm) 46

47 Alltoall P0 P1 P2 P3 A0 A1 A2 A3 B0 B1 B2 B3 C0 C1 C2 C3 D0 D1 D2 D3 Alltoall A0 B0 C0 D0 A1 B1 C1 D1 A2 B2 C2 D2 A3 B3 C3 D3 47

48 Collective Communication in MPI Reduce (Misleading) Combine the data of the input buffer of each process using binary reduction operation. Then return result to root process The syntax of the MPI_Reduce is given by C int MPI_Reduce (void* send_buffer, void* receive_buffer, int count, MPI_Datatype send_datatype, MPI_Op op, int send_src, MPI_Comm comm) 48

49 MPI Built-in Operations (Reduce) MPI_Max MPI_Min MPI_Prod MPI_Sum MPI_Land MPI_Lor MPI_Lxor MPI_Band MPI_Bor MPI_Bxor MPI_Maxloc MPI_Minloc Maximum Minimum Product Sum Logical and Logical or Logical exclusive or Binary and Binary or Binary exclusive or Maximum and its location Minimum and its location 49

50 Reduce P0 P1 A B Reduce ABCD P2 C P3 D 50

51 Example: π in C (Part 1) #include "mpi.h" #include <math.h> int main(int argc, char *argv[]) { int done = 0, n, myid, numprocs, i, rc; double PI25DT = ; double mypi, pi, h, sum, x, a; MPI_Init(&argc,&argv); MPI_Comm_size(MPI_COMM_WORLD,&numprocs); MPI_Comm_rank(MPI_COMM_WORLD,&myid); while (!done) { if (myid == 0) { printf("enter the number of intervals: (0 quits) "); scanf("%d",&n); } MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD); if (n == 0) break; 51

52 Example: π in C (Part 2) } h = 1.0 / (double) n; sum = 0.0; for (i = myid + 1; i <= n; i += numprocs) { x = h * ((double)i - 0.5); sum += 4.0 / (1.0 + x*x); } mypi = h * sum; MPI_Reduce(&mypi, &pi, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD); if (myid == 0) printf("pi is approximately %.16f, Error is.16f\n", pi, fabs(pi - PI25DT)); } MPI_Finalize(); return 0; 52

53 Summary Programing model for Distributed Memory System - Message exchange - Blocking / Non-blocking operations - P2P, Collective communication modes - Avoid deadlock - (A)synchronization communications Distributed memory P0 Pn-1 M0 Mn-1 Network 53

54 Exercises Definition and programming. Exercises for Assignment 10: (1) (morning + afternoon) (2) Deadline: Exercises for Assignment 11: (1) afternoon, (morning + afternoon), afternoon (2) Deadline:

55 Thank you very much!

56 The presentations includes figures, trademarks, logos which are properties of third parties. Rights are reserved to the corresponding rights owners.