Introduction to MPI, the Message Passing Library

Transcription

1 Chapter 3, p. 1/57 Basics of Basic Messages -To-? Introduction to, the Message Passing Library School of Engineering Sciences Computations for Large-Scale Problems I

2 Chapter 3, p. 2/57 Outline Basics of Basic Messages -To-? 1 Basics of 2 Basic 3 Messages 4 -To- 5 6? 7 8

3 Chapter 3, p. 3/57 Basics of Basic Programming Options For Message Passing Computing Messages -To-? Programming a message passing computer can be achieved by 1 Designing a special parallel programming language 2 Extending the syntax/reserved words of an existing high-level language to handle message-passing 3 Using an existing sequential high-level language and providing a library of external procedures for message-passing

4 Chapter 3, p. 4/57 What Is? Basics of Basic Messages -To-? Message Passing Interface standard The first standard and portable message passing library with good performance "Standard" by consensus of Forum participants from over 40 organizations Finished and published in May 1994, updated in June complete as of July Extends. Documents that define are on the World Wide Web from the Forum. 2.1 released in released in 2012

5 Chapter 3, p. 5/57 A Warning Basics of Basic Messages -To-? In many respects, using is low-level and thus comparable to assembly programming.

6 Chapter 3, p. 6/57 Format of Basics of Basic Messages -To-? C Bindings rc = _Xxxxx(parameter,... ) rc is error code, = _SUCCESS if successful Fortran bindings call _XXXXX(parameter,..., ierror) case insensitive ierror is error code, = _SUCCESS if successful Exception: timing functions return double precision reals Header file required #include "mpi.h" for C programs include mpif.h for Fortran programs

7 Chapter 3, p. 7/57 Basics of Basic Messages -To-? has several hundreds functions, but you can do much with just 6: Initialize for communications _INIT initializes the environment _COMM_SIZE returns the number of processes _COMM_RANK returns this process s number (rank) Communicate to share data between processes _SEND sends a message _RECV receives a message Exit in a "clean" fashion when done _FINALIZE

8 Chapter 3, p. 8/57 Basics of Basic An Sample Program (Fortran) I Messages -To-? Here, we show the 6 basic calls: program hello include mpif.h integer rank, size, ierror, tag, status(_status_size) character(12) message call _INIT (ierror) call _COMM_SIZE (_COMM_WORLD, size, ierror) call _COMM_RANK (_COMM_WORLD, rank, ierror) tag = 100 cont...

9 Chapter 3, p. 9/57 Basics of Basic Messages -To-? & An Sample Program (Fortran) II if (rank.eq. 0) then message = Hello, world do i=1, size-1 call _SEND(message, 12, _CHARACTER, i, tag, _COMM_WORLD, ierror) enddo else call _RECV(message, 12, _CHARACTER, 0, tag, & _COMM_WORLD, status, ierror) endif print*, node, rank, :, message call _FINALIZE (ierror) end Skip C

10 Chapter 3, p. 10/57 Basics of Basic An Sample Program (C version) I Messages -To-? #include <stdio.h> #include "mpi.h" main(int argc, char **argv) { int rank, size, tag, rc, i; _Status status; char message[20]; rc = _Init(&argc, &argv); rc = _Comm_size(_COMM_WORLD, &size); rc = _Comm_rank(_COMM_WORLD, &rank); tag = 100; cont...

11 Chapter 3, p. 11/57 Basics of Basic Messages -To-? } An Sample Program (C version) II if(rank == 0) { strcpy(message, "Hello, world"); for (i=1; i<size; i++) rc = _Send(message, 13, _CHAR, i, tag, _COMM_WORLD); } else rc = _Recv(message, 13, _CHAR, 0, tag, _COMM_WORLD, &status); printf( "node %d : %.13s\n", rank,message); rc = _Finalize();

12 Chapter 3, p. 12/57 What Is a Message? Basics of Basic Messages -To-? Message = data + envelope call _SEND(startbuf,count,datatype, \ / ---DATA--- dest,tag,comm, \ / ENVELOPE ierror)

13 Chapter 3, p. 13/57 Data Basics of Basic Messages -To-? Arguments startbuf (starting location of data) count (number of elements) receive >= send datatype (basic or derived) receiver = send Datatypes Basic Derived (mixed, non-contiguous data etc.)

14 Chapter 3, p. 14/57 Basic Datatypes (Fortran) Basics of Basic Messages -To-? datatype _INTEGER _REAL _DOUBLE_PRECISION _COMPLEX _LOGICAL _CHARACTER _BYTE _PACKED Fortran datatype INTEGER REAL DOUBLE PRECISION COMPLEX LOGICAL CHARACTER(1) The names of the C dataypes are slightly different!

15 Chapter 3, p. 15/57 Envelope Basics of Basic Messages -To-? Destination or source Tag rank in a communicator receiver = sender or _ANY_SOURCE integer chosen by programmer receiver = sender or _ANY_TAG Communicator defines communication "space" receiver = sender Predefined communicator: _COMM_WORLD

16 Chapter 3, p. 16/57 Blocking Send Basics of Basic Messages -To-? C Fortran int _Send(void *buf, int count, _Datatype datatype, int dest, int tag, _Comm comm) _SEND(buf, count, datatype, dest, tag, comm, ierror) buf is the beginning of the buffer containing the data to be sent. For Fortran, this is often the name of an array in your program. For C, it is an address. count is the number of elements to be sent (not bytes) datatype is the type of data dest is the rank of the process which is the destination for the message tag is an arbitrary number which can be used to distinguish among messages comm is the communicator ierror is a return error code

17 Chapter 3, p. 17/57 Blocking Receive Basics of Basic Messages C int _Recv(void *buf, int count, _Datatype datatype, int source, int tag, _Comm comm, _Status *status) -To-? Fortran _RECV(buf, count, datatype, source, tag, comm, status, ierror) buf is the beginning of the buffer where the incoming data are to be stored. For Fortran, this is often the name of an array in your program. For C, it is an address. count is the number of elements (not bytes) in your receive buffer datatype is the type of data source is the rank of the process from which data will be accepted (This can be a wildcard, by specifying the parameter _ANY_SOURCE.) tag is an arbitrary number which can be used to distinguish among messages (This can be a wildcard, by specifying the parameter _ANY_TAG.) comm is the communicator status is an array or structure of information that is returned. ierror is a return error code

18 Chapter 3, p. 18/57 Modes Basics of Basic Messages -To-? Mode Blocking Non-Blocking synchronous _SSEND _ISSEND ready _RSEND _IRSEND buffered _BSEND _IBSEND standard _SEND _ISEND _RECV _IRECV _SENDRECV _SENDRECV_REPLACE

19 Chapter 3, p. 19/57 Standard Mode Basics of Basic Messages -To-? The operation is implementation dependent, not prescribed by the standard Should be the best compromise among different scenarios on a given hardware Usually good, not necessarily optimal

20 Chapter 3, p. 20/57 Basics of Basic Blocking Standard Send (Short Messages) Messages -To-? Exceeding system buffer space causes communication to stall until space is freed.

21 Chapter 3, p. 21/57 Basics of Basic Standard Blocking Send (Long Messages) Messages -To-?

22 Chapter 3, p. 22/57 Syntax of Non-Blocking Calls Basics of Basic Messages -To-? C Fortran _Isend(buf, count, dtype, dest, tag, comm, request) _Wait(request, status) _ISEND(buf, count, dtype, dest, tag, comm, request, ierror) _WAIT(request, status, ierror)

23 Chapter 3, p. 23/57 Basics of Basic Messages -To- Example: Non-Blocking Standard Send Non-blocking standard send, message size <= threshold Non-blocking receive?

24 Chapter 3, p. 24/57 Basics of Basic Messages -To- Example: Non-Blocking Standard Send, Large Message Non-blocking standard send, message size > threshold Non-blocking receive?

25 Chapter 3, p. 25/57 Conclusions: Non-Blocking Calls Basics of Basic Messages -To-? Gains Avoid Deadlock Decrease Synchronization Overhead Some Systems: Reduce Systems Overhead Best to post non-blocking sends and receives as early as possible, and to do waits as late as possible Must avoid writing to send buffer between _Isend and _Wait and must avoid reading and writing in receive buffer between _Irecv and _Wait

26 Chapter 3, p. 26/57 Basic Deadlock: Causes Basics of Basic Messages -To-?

27 Chapter 3, p. 27/57 Basic Deadlock: Solutions Basics of Basic Messages -To-? different ordering of calls between tasks non-blocking calls _Sendrecv _Sendrecv_replace buffered mode

28 Chapter 3, p. 28/57 Basics of Basic Messages -To-? An call made identically by all processes in process group Supports easy manipulation of a "common" piece or set of information Uses an communicator, defined for the process group

29 Chapter 3, p. 29/57 Basics of Basic Messages -To-? Barrier synchronization Broadcast from one member to all other members Gather data from an array spread across processors into one array Scatter data from one member to all members All-to-all exchange of data Global reduction (e.g., sum, min of "common" data elements) Scan across all members of a communicator

30 Chapter 3, p. 30/57 Characteristics Basics of Basic Messages -To-? Performed on a group of processes, identified by a communicator Substitute for a sequence of point-to-point calls s are locally blocking Synchronization is not guaranteed (implementation dependent) Some routines use a root process to originate or receive all data Data amounts must exactly match Many variations to basic categories No message tags are needed

31 Chapter 3, p. 31/57 Barrier Synchronization Basics of Basic Messages -To-? Blocks the calling process until all group members have called the routine C Fortran _Barrier(_comm comm) _BARRIER(comm, ierr)

32 Chapter 3, p. 32/57 Data Movement Basics of Basic Messages -To-? Broadcast Gather Scatter Allgather Alltoall

33 Chapter 3, p. 33/57 Broadcast Basics of Basic Messages -To-? Send a message from the root process to all processes in the group, including the root process C Fortran int _Bcast(void* buffer, int count, _Datatype datatype, int root, _Comm comm) _BCAST(buffer, count, datatype, root, comm, ierr)

34 Chapter 3, p. 34/57 Broadcast: Effect Basics of Basic Messages -To-?

35 Chapter 3, p. 35/57 Example: Hello World Basics of Basic Messages -To-?! Prepare... if (rank.eq. 0) then message = Hello, world endif call _BCAST(message, 12, _CHARACTER, root, & _COMM_WORLD, ierr) print*, node, rank, :, message! Finalize...

36 Chapter 3, p. 36/57 _Gather And _Scatter Basics of Basic Messages -To-? Gather Purpose: Each process sends the contents of its send buffer to the root process. The root process receives the messages and stores them in rank order. Our Mandelbrot program is a typical instance of a gather operation. Scatter Purpose: The root process splits a buffer of data into chunks, then sends each process in the group 1 chunk. Reverse of GATHER operation

37 Chapter 3, p. 37/57 Global Computation Basics of Basic Messages -To-? routines that include a computation Computation function included in routine call may be either An predefined routine or A user-supplied function Two types of global computation routines: Reduce Scan

38 Chapter 3, p. 38/57 _Reduce Basics of Basic Messages -To- Combine the elements of the input buffer of each process using a specified operation Return result to root process?

39 Chapter 3, p. 39/57 _Reduce (cont) Basics of Basic Messages -To-? C int _Reduce(void* sbuf, void* rbuf, int count, _Datatype stype, _Op op, int root, _Comm comm) Fortran _REDUCE(sbuf, rbuf, count, stype, op, root, comm, ierr)

40 Chapter 3, p. 40/57 Predefined Operators Basics of Basic Messages -To-? Name _MAX _MIN _SUM _PROD _LAND _BAND _LOR _BOR _LXOR _BXOR _MAXLOC _MINLOC Meaning maximum value minimum value sum product logical and bit-wise and logical or bit-wise or logical xor bit-wise xor max value and location min value and location Each of these operations makes sense for only certain datatypes. The Standard lists the types accepted for each operation.

41 Chapter 3, p. 41/57 Considerations Basics of Basic Messages -To-? A great deal of hidden communication takes place with collective communication depends greatly on the particular implementation of Because there may be forced synchronization, not always best to use collective communication

42 Chapter 3, p. 42/57 at PDC Basics of Basic Messages -To-? The PDC ( Computer Center) runs a number of machines: Machine Implementation Swegrid/SBC Ferlin Ekman Lindgren CH (with PGI or GNU compilers) Open (with Intel or GNU compilers), many Many (many compilers) CH2 (many compilers) Ubuntu workstations in the computer laboratories

43 Chapter 3, p. 43/57 Open On ferlin Basics of Basic Messages -To-? Have a look at the description at the PDC Introduction and In short, after having logged in into an interactive node (at most 8 processes!): module add easy i-compilers mpi mpicc -O -g <source> -o <executable> mpirun -np <number of processes> <executable> exit Interactive nodes are only useful for debugging purposes!

44 Chapter 3, p. 44/57 Low-Level Basics of Basic Messages -To-? Use strategies which are used for sequential programs: program instrumentation (e.g., insert printing statements and compare to expected results) Use low-level debuggers (seldom used since you have many simultaneous processes) Intrinsic Problem of Low-Level Instrumenting a sequential program does not change the functioning. In a parallel program, communication will probably take place in a different order thus changing the behavior.

45 Chapter 3, p. 45/57 Basics of Basic Messages -To-? 1 If possible, run the program as a single process and debug as a normal sequential program. 2 Execute the program using two or four multi-tasked processes on a single computer. Check that messages are sent to the correct places. 3 Execute the program using two or four processes but now across several computers.

46 Chapter 3, p. 46/57 Timing Basics of Basic Messages -To-? A simple way of measuring parallel performance is to use _Wtime to get execution time: _Barrier(_COMM_WORLD); start = _Wtime(); /* Perform your computations */ _Barrier(_COMM_WORLD); end = _Wtime(); if (myrank == 0) printf( Execution time: %e\n,end-start);

47 Chapter 3, p. 47/57 Hints Basics of Basic Messages -To-? measure average time over several executions less sensitive to random events use realistic parameters measurements valid for representative problems time only relevant parts initialization and file IO not likely to be significant for real simulations Use results to choose problem size and number of processors identify parts of the code and algorithm to optimize further

48 Chapter 3, p. 48/57 Appendix: calls Additional Material Sca Calls: _INIT _INIT must be the first routine you call in each process. It can only be called once. It establishes an environment necessary for to run. This environment may be customized for any runtime flags provided by the implementation. The command line arguments are passed to the C version. C int _Init(int *argc, char ***argv) Fortran call _INIT(ierror) integer ierror

49 Chapter 3, p. 49/57 calls: _COMM_SIZE Appendix: calls Additional Material Sca _COMM_SIZE returns the number of processes within a communicator. can determine the number of processes because you specify this when you set the environment variable MP_PROCS. C int _Comm_size(_Comm comm, int *size) Fortran _COMM_SIZE(comm, size, ierror) integer comm, size, ierror

50 Chapter 3, p. 50/57 calls: _COMM_RANK Appendix: calls Additional Material Sca Rank is used to specify a particular process. It is an integer in the range 0 through size-1. _COMM_RANK returns the calling process s rank in the specified communicator. C int _Comm_rank(_Comm comm, int *rank) Fortran _COMM_RANK(comm, rank, ierror) integer comm, rank, ierror

51 Chapter 3, p. 51/57 calls: _SEND Appendix: calls Additional Material Sca A message will be sent to another process. _SEND is a blocking send. C int _Send(void *buf, int count, _Datatype datatype, int dest, int tag, _Comm comm) Fortran _SEND(buf, count, datatype, dest, tag, comm, ierror) <type> buf(*) integer count, datatype, dest, tag, comm, ierror

52 Chapter 3, p. 52/57 calls: _RECV Appendix: calls Additional Material A message will be received from another process. _RECV is blocking. Sca C Fortran int _Recv(void *buf, int count, _Datatype datatype, int source, int tag, _Comm comm, _Status *status) _RECV(buf, count, datatype, source, tag, comm, status, ierror) <type> buf(*) integer count, datatype, source, tag, comm integer status(_status_size), ierror

53 Chapter 3, p. 53/57 calls: _FINALIZE Appendix: calls Additional Material Sca The last call to should be _FINALIZE. This makes to exit cleanly. The code can continue to execute after calling _FINALIZE, but it can longer call routines. C int _Finalize() Fortran _FINALIZE(ierror) integer ierror

54 Chapter 3, p. 54/57 Calls: _Sendrecv Appendix: calls Additional Material Sca C int _Sendrecv (void *sendbuf, int sendcount, _Datatype sendtype, int dest, int sendtag, void *recvbuf, int recvcount, _Datatype recvtype, int source, int recvtag, _Comm comm, _Status *status)

55 Chapter 3, p. 55/57 Datatypes (C) Appendix: calls Additional Material Sca datatype _CHAR _SHORT _INT _LONG _UNSIGNED_CHAR _UNSIGNED_SHORT _UNSIGNED _UNSIGNED_LONG _FLOAT _DOUBLE _LONG_DOUBLE _BYTE _PACKED C datatype signed char signed short int signed int signed long int unsigned char unsigned short int unsigned int unsigned long int float double long double

56 Chapter 3, p. 56/57 Sca: How Does It Work Appendix: calls Additional Material Sca Thresholds for different communication protocols

57 Chapter 3, p. 57/57 Sca: Protocols Appendix: calls Additional Material Sca A Inlining protocol B Eagerbuffering protocol C Transporter protocol Zerocopy protocol (similar to C, but by-passing the ringbuffer)