Buffering in MPI communications

Transcription

1 Buffering in MPI communications Application buffer: specified by the first parameter in MPI_Send/Recv functions System buffer: Hidden from the programmer and managed by the MPI library Is limitted and can be easy to exhaust 1

2 High Performance Computing Course Notes Message Passing Programming III

3 Blocking and non-blocking communications Blocking send The sender doesn t return until the application buffer can be re-used (which often means that the data have been copied from application buffer to system buffer), but doesn t mean that the data will be received Blocking receive The receiver doesn t return until the data have been ready to use by the receiver (which often means that the data have been copied from system buffer to application buffer at the receiving side) Non-blocking send/receive The calling process returns immediately Just request the MPI library to perform the operation, the user cannot predict when that will happen Unsafe to modify the application buffer until you can make sure the requested operation has been performed (MPI provides routines to test this) Can be used to overlap computation with communication and have possible performance gains 3

4 Testing non-blocking communications for completion Completion tests come in two types: WAIT type TEST type WAIT type: the WAIT type testing routines block until the communication has completed. A non-blocking communication immediately followed by a WAITtype test is equivalent to the corresponding blocking communication TEST type: these routines return TRUE or FALSE value The process can perform some other tasks when the communication has not completed 4

5 Testing non-blocking communications for completion The WAIT-type test is: MPI_Wait (request, status) This routine blocks until the communication specified by the handle request has completed. The request handle will have been returned by an earlier call to a non-blocking communication routine. The TEST-type test is: MPI_Test (request, flag, status) In this case the communication specified by the handle request is simply queried to see if the communication has completed and the result of the query (TRUE or FALSE) is returned immediately in flag. 5

6 Testing multiple non-blocking communications for completion Wait for all communications to complete MPI_Waitall (count, array_of_requests, array_of_statuses) This routine blocks until all the communications specified by the request handles, array_of_requests, have completed. The statuses of the communications are returned in the array array_of_statuses and each can be queried in the usual way for the source and tag if required Test if all communications have completed MPI_Testall (count, array_of_requests, flag, array_of_statuses) If all the communications have completed, flag is set to TRUE, and information about each of the communications is returned in array_of_statuses. Otherwise flag is set to FALSE and array_of_statuses is undefined. 6

7 Testing multiple non-blocking communications for completion Query a number of communications at a time to find out if any of them have completed Wait: MPI_Waitany (count, array_of_requests, index, status) MPI_WAITANY blocks until one or more of the communications associated with the array of request handles, array_of_requests, has completed. The index of the completed communication in the array_of_requests handles is returned in index, and its status is returned in status. Should more than one communication have completed, the choice of which is returned is arbitrary. Test: MPI_Testany (count, array_of_requests, index, flag, status) The result of the test (TRUE or FALSE) is returned immediately in flag. 7

8 Communication modes Synchronous mode The communication is considered complete when the sender receives the acknowledgement from the receiver that the data have been received Buffered mode The sender uses the user-defined buffer instead of system buffer (the system buffer is limited) Communication is considered complete when the application buffer can be reused, which means that the data has been copied from the application buffer to the user-defined buffer Ready mode This mode can be used only when the programmer can make sure that the receive routine will be posted before the corresponding send routine, otherwise, the outcome is undefined. Standard mode Could be synchronous mode or buffered mode, depending on implementations 8

9 Blocking and non-blocking forms for the communication modes All these four communication modes have both blocking and non-blocking forms Standard send: MPI_Send (blocking), MPI_Isend (nonblocking) Synchronous send: MPI_Ssend (blocking), MPI_Issend (non-blocking) Buffered send: MPI_Bsend (blocking), MPI_Ibsend (nonblocking) Ready send: MPI_Rsend (blocking), MPI_Irsend (nonblocking) 9

10 Blocking synchronous send the sender doesn t return until it receives the acknowledgement from the receiver that the message has been received Format: MPI_Ssend (buf, count, datatype, dest, tag, comm) 10

11 Blocking buffered send The sender doesn t return until the application buffer can be reused Format: MPI_Bsend(&buf, count, datatype, dest, tag, comm) Must attach buffer space using: MPI_Buffer_attach(buffer, size) Buffer space is detached using: MPI_Buffer_detach(buffer, size) 11

12 Blocking ready send The sender returns when the application buffer can be reused Format: MPI_Rsend (buf, count, datatype, dest, tag, comm) 12

13 Blocking standard send Format: MPI_Send(buf, count, datatype, dest, tag, comm) 13

14 Non-blocking synchronous send Format: MPI_Issend (buf, count, datatype, dest, tag, comm, request) the system issues a unique request number The request can be used later to determine the completion of the communication Other non-blocking send functions are similar, all have one additional request in the parameter list of the corresponding blocking send functions 14

15 Two Receive routines MPI_Recv() MPI_Irecv() 15

16 Message order Order: If a sender sends two messages (Message 1 and Message 2) in succession to the same destination, and both match the same receive, the receive operation will receive Message 1 before Message 2. If a receiver posts two receives (Receive 1 and Receive 2), in succession, and both are looking for the same message, Receive 1 will receive the message before Receive 2. Order rules do not apply to multiple messages sent by different processes. 16

17 Virtual topology Is a mechanism for naming the processes in a communicator in a way that fits the communication pattern better Can make coding simpler 17

18 Cartesian topology naming the processes in a communicator using Cartesian coordinates 18

19 Cartesian topology Create a Cartesian topology int MPI_Cart_create(MPI_Comm comm_old, int ndims, int *dims, int *periods, int reorder, MPI_Comm *comm_cart) [ IN comm_old] input communicator [ IN ndims] number of dimensions of cartesian grid [ IN dims] integer array of size ndims specifying the number of processes in each dimension [ IN periods] logical array of size ndims specifying whether the grid is periodic ( true) or not ( false) in each dimension [ IN reorder] ranking may be reordered ( true) or not ( false) [ OUT comm_cart] communicator with new cartesian topology (handle) The topology is only accessible through the new communicator returned in comm_cart 19

20 Converting between ranks and coordinates MPI_Cart_rank (comm, coords, rank) converts process grid coordinates to process rank. It might be used to determine the rank of a particular process whose grid coordinates are known, in order to send a message to it or receive a message from it MPI_Cart_coords (comm, rank, ndims, coords) converts process rank to process grid coordinates. It might be used to determine the grid coordinates of a particular process from which a message has just been received. 20

21 Derived datatype Users can construct (derive) their own datatype The memory layout of a datatype in MPI a is expressed as {(type_0, offset_0), (type_1, offset_1),, (type_n, offset_n) 21

22 Derived datatypes MPI_TYPE_CONTIGUOUS(10, MPI_REAL, tenrealtype) // returns a new datatype that represents the concatenation of 10 instances of // MPI_REAL. Allows replication of a datatype into contiguous locations MPI_TYPE_COMMIT(tenrealtype) // commits the datatype, must be done before communication MPI_SEND(data, 1, tenrealtype, dest, tag, MPI_COMM_WORLD) // sends the data at location data to dest MPI_TYPE_FREE(tenrealtype) // frees the datatype This is equivalent to the following single call MPI_SEND(data, 10, MPI_REAL, dest, tag, MPI_COMM_WORLD) Where the elements to be sent are already contiguous 22

23 Derived datatypes MPI_TYPE_VECTOR (count, blocklen, stride, oldtype, newtype) Defines a derived type newtype comprising count consecutive blocks of data elements with datatype oldtype, with each block containing blocklen data elements, and the start of successive blocks separated by stride data elements. E.g. float data [1024]; MPI_Datatype floattype; MPI_TYPE_vector (10, 1, 32, MPI_FLOAT, &floattype); MPI_Type_commit (&floattype); MPI_Send (data, 1, floatype, dest, tag, MPI_COMM_WORLD); MPI_Type_free (&floattype) Is equivalent to the following code float data[1024], buff[10]; for (i=0; i<10; i++) buff[i] = data [i*32]; MPI_Send (buff, 10, MPI_FLOAT, dest, tag, MPI_COMM_WORLD); Both send 10 FP numbers from locations data[0], data[32],, data[288] 23

24 Derived datatypes MPI_Type_Indexed (count, lengths[], offsets[], oldtype, newtype) Used in the case where the elements in the datatype to be constructed have the same type, but different offsets Used to define a type comprising one or more blocks of a primitive or previously defined datatype, where block lengths and the displacement between blocks are specified in arrays The above call defines a type newtype comprising count consecutive blocks of data elements with type oldtype, with block i having a displacement of offsets data elements and containing lengths data elements 24

25 Derived Datatype int MPI_Type_struct(int count, int *array_of_blocklengths, MPI_Aint *array_of_displacements, MPI_Datatype *array_of_types, MPI_Datatype *newtype) The derived datatype includes different datatypes, each with different displacements 25

26 Summary of MPI Point-to-point communication Collective communication Communication modes Virtual topology Derived datatype 26