Message Passing Interface

Transcription

1 MPSoC Architectures MPI Alberto Bosio, Associate Professor UM Microelectronic Departement Message Passing Interface API for distributed-memory programming parallel code that runs across multiple computers (nodes) De facto industry standard available on (almost) every parallel computer for scientific computing Use from C/C++, Fortran, Python, R,... More than 200 routines Using only 10 routines are enough in many cases 2 1

2 Clarification You can mix MPI and OpenMP in one program You could run multiple MPI processes on a single CPU e.g. debug MPI codes on your laptop An MPI job can span across multiple computer nodes (distributed memory) You could run multiple OpenMP threads on a single CPU e.g. debug OpenMP codes on your laptop 3 Install High-quality implementation available for free Easy to install one on your desktop/laptop OpenMPI: 4 2

3 MPI is a library Compilation: mpicc test.c -o test Execution: Compile MPI applications (running with 16 processes) mpiexec np 16./test You can also run the program on a cluster mpiexec hosts h1:4,h2:4,h3:4,h4:4 np 16./test To set up a cluster: 5 Communicator A group of processes processes are numbered 0,1,.. to N-1 Default communicator MPI_COMM_WORLD contains all processes Query functions: How many processes in total? MPI_Comm_size(MPI_COMM_WORLD, &nproc) MPI_Comm_rank(MPI_COMM_WORLD, &rank) What is my process ID?

4 #include "mpi.h #include <stdio.h> main(int argc, char *argv[]) { int np, pid; MPI_Init(&argc, &argv); // MPI header file // initialize MPI MPI_Comm_size(MPI_COMM_WORLD, &np); MPI_Comm_rank(MPI_COMM_WORLD, &pid); printf("n. of procs = %d, proc ID = %d\n", np, pid); } MPI_Finalize(); // clean up 7 P2P communication 8 4

5 Send MPI_Send(buf, count, data_type, dest, tag, comm) Arguments Meanings buf starting address of send buffer count # of elements data_type data type of each send buffer element dest processor ID (rank) destination tag message tag comm communicator 9 Receive MPI_Recv(buf, count, datatype, src, tag, comm, status) Arguments Meanings buf starting address of send buffer count # of elements datatype data type of each send buffer element src processor ID (rank) destination tag message tag comm communicator status status object (an integer array in Fortran) 10 5

6 Datatypes MPI Datatype is very similar to a C datatype intàmpi_int doubleàmpi_double charàmpi_char More complex datatypes are also possible The count in MPI_SEND and MPI_RECV refers to how many datatype elements should be communicated 11 Receive A message is received when the followings are matched: Source (sending process ID/rank) Tag Communicator (e.g. MPI_COMM_WORLD) Wildcard values may be used: MPI_ANY_TAG (don t care what the tag value is) MPI_ANY_SOURCE (don t care where it comes from; always receive) 12 6

7 example int f[n], src=0, dest=1; MPI_Status status; //... MPI_Comm_rank( MPI_COMM_WORLD, &rank); if (rank == src) // process dest ignores this MPI_Send(f, N, MPI_INT, dest, 0, MPI_COMM_WORLD); if (rank == dest) // process src ignores this MPI_Recv(f, N, MPI_INT, src, 0, MPI_COMM_WORLD, &status); // Blocking Function call does not return until the communication is complete MPI_Send and MPI_Recv are blocking calls Calling order matters it is possible to wait indefinitely, called deadlock improper ordering results in serialization (loss of performance) 14 7

8 It works MPI_Comm_rank(comm, &rank); if (rank == 0) { MPI_Send(sendbuf, cnt, MPI_INT, 1, tag, comm); MPI_Recv(recvbuf, cnt, MPI_INT, 1, tag, comm, &stat); } else { // rank==1 MPI_Recv(recvbuf, cnt, MPI_INT, 0, tag, comm, &stat); MPI_Send(sendbuf, cnt, MPI_INT, 0, tag, comm); } 15 Deadlock MPI_Comm_rank(comm, &rank); if (rank == 0) { MPI_Recv(recvbuf, cnt, MPI_INT, 1, tag, comm, &stat); MPI_Send(sendbuf, cnt, MPI_INT, 1, tag, comm); } else { /* rank==1 */ MPI_Recv(recvbuf, cnt, MPI_INT, 0, tag, comm, &stat); MPI_Send(sendbuf, cnt, MPI_INT, 0, tag, comm); } 16 8

9 Collective Communication One to all MPI_Bcast 17 Mpi_bcast MPI_Bcast(buffer, count, datatype, root, comm ) Broadcasts a message from root process to all other processes in the same communicator data data processes 18 9

10 example C/C++ MPI_Comm comm; int array[100]; //... MPI_Bcast( array, 100, MPI_INT, 3, comm); 19 Reduction Reduces values on all processes to a single value int MPI_Reduce( const void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) 20 10

11 Input Parameters Reduction sendbuf address of send buffer (choice) count number of elements in send buffer (integer) datatype data type of elements of send buffer (handle) op reduce operation (handle) root rank of root process (integer) comm communicator (handle) 21 Example 22 11

12 23 Exercise Try the example mpi_hello.c Modify it in order to Have each task determine a unique partner task to send/receive with. Each task sends its partner a single integer message: its taskid Each task receives from its partner a single integer message: the partner's taskid For confirmation, after the send/receive, each task prints something like "Task ## is partner with ##" where ## is the taskid of the task and its partner

13 Exercise 1. Let us come back to matrix multiplication Parallelize it using MPI 2. Download the serial program for computing pi and parallelize it using reduction 3. Generate random numbers and calculate the average using reduction 25 13