MPI: Parallel Programming for Extreme Machines. Si Hammond, High Performance Systems Group

Transcription

1 MPI: Parallel Programming for Extreme Machines Si Hammond, High Performance Systems Group

2 Quick Introduction Si Hammond, WPRF/PhD Research student, High Performance Systems Group, Computer Science Platforms - Cray XT3/4, CSC Francesca, small AMD Opteron/Intel Xeon clusters

3 What s in this talk? Parallel programming methodologies - why MPI? Where can I use MPI? MPI in action Getting MPI to work at Warwick Examples

4 Programming in Parallel OpenMP, Threads MPI, Network Sockets One Computer Multiple Processors/ Multiple Cores Many Computers Multiple Processors/ Multiple Cores Network

5 What is MPI? Message Passing Interface Programming paradigm for writing parallel codes Defines what messages of data are passed between processes - how this happens at the underlying layers doesn t matter. Runs on single multi-core/processor machines to small distributed clusters, right the way up to IBM BlueGene and IBM RoadRunner

6 Why learn MPI? Used in almost every major parallel scientific code Can be used with Fortran, C, C++ (Java almost) So far the only messaging paradigm to scale to 100k+ nodes Highly tuned and optimised - if you want parallel codes that perform you really need this.

7 MPI - The Theory MPI Programs are Single Program Multiple Data Same executable running multiple times but each will have its own separate data - there is no global memory. MPI is programmer driven - you have to write the parallelism, there are no smart tools to do this for you (like OpenMP).

8 MPI in Action

9 MPI in Action First step is assigning each machine a rank. MPI will do this for you automatically. Ranks start at 0 and go to n-1 Usually refer to rank 0 as the root Rank 0 MPI Library Rank 1

10 Sending a Message Lets send a message from 0 to 1. Send(1, Hello ) Recv(1) Rank 0 posts a send to rank 1. MPI Library Rank 1 posts a receive from rank 0. Data is exchanged. Rank 0 Rank 1

11 MPI Program Outline in C #include mpi.h main(int argc, char* argv[]) { } MPI_Init(&argc, &argv); // Program goes here MPI_Finalize(); Must be before any use of MPI functions Must be the last use of MPI

12 What Rank Am I? #include mpi.h main(int argc, char* argv[]) { } MPI_Init(&argc, &argv); int my_rank; MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); if(my_rank == 0) printf( I am the boss\n ); else printf( I am not the boss :( \n ); MPI_Finalize();

13 What Rank Am I? #include mpi.h main(int argc, char* argv[]) { MPI_Init(&argc, &argv); MPI_COMM_WORLD is the communicator group, we ll come back to this int my_rank; MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); if(my_rank == 0) printf( I am the boss\n ); else printf( I am not the boss :( \n ); MPI_Finalize(); }

14 What Rank Am I? #include mpi.h main(int argc, char* argv[]) { MPI_Init(&argc, &argv); Parameters to MPI which usually passed as pointers int my_rank; MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); if(my_rank == 0) printf( I am the boss\n ); else printf( I am not the boss :( \n ); MPI_Finalize(); }

15 What Rank Am I? #include mpi.h main(int argc, char* argv[]) { } MPI_Init(&argc, &argv); int my_rank; int world_size; MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); MPI_Comm_size(MPI_COMM_WORLD, &world_size); if(my_rank == 0) printf( I am the boss\n ); else printf( I am not the boss :( \n ); MPI_Finalize();

16 Send a message #include mpi.h main(int argc, char* argv[]) { MPI_Init(&argc, &argv); int my_rank; int a[1]; a[0] = 42; int tag = 0; Send data from array a, 1 piece of data of type INT to rank 1 MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); if(my_rank == 0) MPI_Send(a,1,MPI_INT,1,tag,MPI_COMM_WORLD); else { MPI_Status status; MPI_Recv(a,1,MPI_INT,0,tag,MPI_COMM_WORLD, &status); } } MPI_Finalize(); Recv data int array a, 1 piece of data of type INT from rank 0

17 MPI - Data Types The purpose of MPI data types are so the programmer can say how many items of data should be transmitted without worrying about how much memory this is. The compiler and MPI will work this out for you! Good idea to stick to these, will be correct for your architecture, e.g. 64-bit, 32-bit etc.

18 MPI - Data Types MPI_CHAR = signed char MPI_SHORT = signed short int MPI_INT = signed int MPI_LONG = signed long int MPI_FLOAT MPI_DOUBLE MPI_LONG_DOUBLE

19 MPI Gets Serious

20 MPI Collectives Real power of MPI is in the advanced data handling functions - these are known as the collectives Situation is: Multiple ranks have a piece of data you need to carry some operation out with. One rank has a big piece of data you need to split up between multiple ranks. Collectives are highly tuned for this = fast performance

21 MPI Broadcast MPI_Bcast(void* msg, int count, MPI_Datatype datatype, int root, MPI_Comm comm); e.g. MPI_Bcast(a, 16, MPI_INT, 0, MPI_COMM_WORLD); Broadcasts data from the rank which matches root to all ranks in the communicator group. Rank = root

22 MPI Reduce MPI_Reduce(void* operand, void* result, int count, MPI_Datatype datatype, MPI_Op operation, int root, MPI_Comm comm); Reduces data from ranks in the world to one single rank Rank = root

23 MPI Reduce Reduces essentially apply a mathematical operation to pieces of data being held remotely. MPI_MAX, MPI_MIN, MPI_SUM, MPI_PROD etc You must not specify the operand and result to be the same location in memory - only the result will be used on the root, for everyone else it will be empty

24 MPI All-Reduce MPI_Allreduce(void* operand, void* result, int count, MPI_Datatype datatype, MPI_Op operation, MPI_Comm comm); Reduces data from ranks to everyone else in the world. Rank = root

25 MPI Collectives and Arrays MPI has a lot of collectives to handle decomposition and recomposition of arrays. Typically very useful for matrix/vector operations which are conducted in parallel.

26 Scatter and Gather a0, a1, a2, a3 a0, a1, a2, a3 Scatter Gather

27 MPI Scatter MPI_Scatter(void* send_data, int send_count, MPI_Datatype send_type, void* recv_data, int recv_count, MPI_Datatype recv_type, int root, MPI_Comm comm) The send parameters are used on the root The recv parameters are used by everyone Decomposes arrays onto ranks (inc. the root)

28 MPI Gather MPI_Gather(void* send_data, int send_count, MPI_Datatype send_type, void* recv_data, int recv_count, MPI_Datatype recv_type, int root, MPI_Comm comm) Recombines sub-arrays on world ranks (inc. root) into one large array on the root. MPI_Allgather allows you to gather with results being copied to all ranks.

29 Non-Blocking MPI

30 Non-Blocking Send/Recv Until now all MPI operations block until they complete. Good if you know your data might be overwritten, bad for performance if you know its safe to proceed. Can we issue the send/recv now? Do some work and then wait for the operation to complete later at some point later?... Yes.

31 Non-Blocking Send/Recv MPI_Isend(void* buffer, int count, MPI_Datatype datatype, int destination, int tag, MPI_Comm comm, MPI_Request* request) The Isend operation issues the send and then returns immediately, the request parameter is a hook to the operation that allows you to monitor it. There is an overhead with issuing an Isend

32 Non-Blocking Send/Recv MPI_Irecv(void* buffer, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Request* request) The Irecv operation issues the recv operation and then returns. Again, there is an overhead for this.

33 Non-Blocking Send/Recv MPI_Wait(MPI_Request* request, MPI_Status* status) The wait operation blocks on the request until it completes, status is updated with the appropriate information. Using Isend/Irecv enables you to issue the operation do some more processing and then wait later (hopefully the operation will have completed). Overlapping computation in this way can seriously improve performance.

34 Using MPI at Warwick

35 Loading the MPI Compilers To deal with the nitty gritty of linking MPI libraries etc, MPI has compiler wrappers - these are the same compilers but will all the command line options enabled. Load compilers: module load intel/intel64 module load intel/ompi64 Loads MPI compiler with underlying compiler set Francesca, Skua etc

36 Compiling MPI Use the MPI compiler equivalent in place of your normal compiler. mpicc (C programs) mpicxx (C++ programs) mpif77 (Fortran 77) mpif90 (Fortran 90)

37 MPI and PBS Pro In your PBS script you must use the command: mpirun <executable> This will load your executable under MPI and ensure all the ranks etc are set up correctly. When you submit under multiple nodes for PBS the system will automatically sort the MPI out (provided you use MPI_Init in your program).

38 MPI Performance

39 MPI Performance CG EP LU MG SP NAS, NPB-MPI 2.4CSC-Francesca

40 Conclusions...

41 Summary In this presentation we ve met the Message Passing Interface (MPI). MPI uses a Single Program Multiple Data programming paradigm (one executable, each one has its own data). No shared-memory - MPI is all about you saying what data to move around the system. MPI is programmer driven, not compiler like OpenMP Used for small clusters right the way to ultra-scale peta-flop computing

42 Summary We have looked at: MPI Point to Point (Send/Recv) Operations MPI Collectives MPI Non-Blocking Sends/Recvs So what can I do now?

43 Whats Next? So what can I do now? MPI has lots more operations for improving performance, making programming easy. MPI I/O Operations for Parallel Data Processing

44 Questions? Thanks for listening