Fengguang Song Department of Computer Science IUPUI

Transcription

1 CS 590: High Performance Computing MPI (2) Fengguang Song Department of Computer Science IUPUI int MPI_Recv(void *buf,int count,mpi_datatype datatype,int source,int tag, MPI_Comm comm, MPI_Status *status) Message Data Consists of count successive entries of the type indicated by datatype, starting with the entry at the address buf. MPI data types: Basic data types: one for each data type in hosting languages of C/C++, FORTRAN Derived data type: will learn later. 31 1

2 Basic MPI Data Types MPI datatype C datatype MPI datatype FORTRAN datatype MPI_CHAR MPI_SHORT MPI_INT MPI_LONG MPI_UNSIGNED_CHAR MPI_UNSIGNED_SHORT MPI_UNSIGNED signed char signed short signed int signed long unsigned char unsigned short unsigned int MPI_INTEGER MPI_REAL MPI_DOUBLE_PREC ISION MPI_COMPLEX MPI_LOGICAL INTEGER REAL DOUBLE PRECISION COMPLEX LOGICAL MPI_UNSIGNED_LONG unsigned long int MPI_CHARACTER CHARACTER(1) MPI_DOUBLE MPI_FLOAT MPI_LONG_DOUBLE double float long double MPI_BYTE MPI_PACKED MPI_BYTE MPI_PACKED 32 Message Envelope Envelope: Source Destination Tag Communicator MPI_Send: source is the calling process, destination is defined Destination can be MPI_PROC_NULL // will return asap, no effect. MPI_Recv: destination is the calling process, source is defined Source can be a wildcard, MPI_ANY_SOURCE. Source can also be MPI_PROC_NULL // Return asap, no effect, receive buffer not modified. Tag: non-negative number, 0, 1,, UB; UB can be determined by querying MPI environment (UB>=32767). For MPI_Recv, can be a wildcard, MPI_ANY_TAG. Communicator: specified, usually MPI_COMM_WORLD. 34 2

3 In Order for a Message To be Received 1) Message envelopes must match Message must be directed to the receiver process Message source must match the source specified by MPI_Recv, unless MPI_ANY_SOURCE is specified. Message tag must match the tag specified by MPI_Recv, unless MPI_ANY_TAG is specified Message communicator must match that specified by MPI_Recv. 2) Data type must match Datatype specified by MPI_Send and MPI_Recv must match. MPI_PACKED can match any other data type Can be more complicated when derived data types are involved. 35 Example: Sending A from P0 to P1 double A[10], B[15]; int rank, tag = 1001, tag1=1002; MPI_Status status; MPI_Comm_rank(MPI_COMM_WORLD,&rank); if(rank==0) MPI_Send(A, 10, MPI_DOUBLE, 1, tag, MPI_COMM_WORLD); else if(rank==1){ MPI_Recv(B, 15, MPI_DOUBLE, 0, tag, MPI_COMM_WORLD, &status); // OK // MPI_Recv(B, 15, MPI_FLOAT, 0, tag, MPI_COMM_WORLD, &status); ß wrong // MPI_Recv(B,15,MPI_DOUBLE,0,tag1,MPI_COMM_WORLD,&status); ß un-match // MPI_Recv(B,15,MPI_DOUBLE,1,tag,MPI_COMM_WORLD,&status); ß un-match // MPI_Recv(B,15,MPI_DOUBLE,MPI_ANY_SOURCE,tag,MPI_COMM_WORLD,&status); ß OK // MPI_Recv(B,15,MPI_DOUBLE,0,MPI_ANY_TAG,MPI_COMM_WORLD,&status); ß OK } 36 3

4 Blocking Send/Recv MPI_Send is blocking: will not return until the message data and envelope is safely stored away. The message data might be delivered to the matching receive buffer, or copied to some temporary system buffer. After MPI_Send returns, user can safely access or overwrite the send buffer. MPI_Recv is blocking: returns only after the receive buffer has the received message After it returns, the data is here and ready for use. Non-blocking send/recv: will be discussed later. Non-blocking calls will return immediately; however, not safe to access the send/receive buffers. Need to call other functions to complete send/recv, then safe to access/modify send/receive buffers. 37 Buffering Send and its matching receive operations may not be synchronized in reality. MPI implementation must decide what happens when send/recv are out of sync. Consider: Send occurs 5 seconds before receive is ready; where is the message when receive is not ready? Multiple sends arrive at the same receiving task which can receive one send at a time what happens to the messages that are backing up? MPI implementation (not the MPI standard) decides what happens in these cases. Typically a system buffer is used to hold data in transit. 38 4

5 Buffering System buffer: Invisible to users and managed by MPI library Finite resource that can be easily exhausted May exist on sending or receiving side, or both Can improve performance. User can attach his/her own buffer for MPI message buffering Warning: if the system buffer becomes full, MPI_Send will be blocked for a long time 39 Different Communication Modes for Send Standard mode: MPI_Send System decides whether the outgoing message will be buffered or not Usually, small messages à buffering mode; large messages, no buffering, synchronous mode Buffered mode: MPI_Bsend Message will be copied to user-provided buffer; Sender calls then returns Users attach own buffer using int MPI_Buffer_attach(void *buf, int size) Synchronous mode: MPI_Ssend No buffering Will block until a matching receive starts receiving data Ready mode: MPI_Rsend Can be used only if a matching receive is already posted; avoid handshake etc. otherwise erroneous. 40 5

6 Communication Modes MPI_Send(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) MPI_Bsend(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) MPI_Ssend(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) MPI_Rsend(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) There is only one MPI_Recv; will match any send mode 41 Properties Message Order: MPI messages are order-preserved If a sender sends two messages to the same destination, and both match the same receiver, then the receiver will receive the first message no matter which message physically arrives first at the receiver. However, if a receive matches two messages from two different senders, the receive may receive either one (implementation dependent). 42 6

7 Properties Progress: if a pair of matching send/recv are initiated on two processes, at least one of them will complete Send will complete, unless the receive is satisfied by some other message Receive will complete, unless send is consumed by some other matching receive. 43 Properties No guarantee of fairness If a message is sent to a destination, and the destination process repeatedly posts a receive that matches this send, however the message may never be received since it is each time overtaken by another message sent from another source. It is user s responsibility to prevent the starvation in such situations. e.g., specify the source process 44 7

8 Properties Resource limitation: Pending communications consume system resources (e.g. buffer space) Lack of resource may either cause error or prevent execution of MPI call. e.g. MPI_Bsend that cannot complete, due to lack of buffer space, is erroneous. e.g., MPI_Send that cannot complete, due to lack of buffer space, will only block, waiting for buffer space to be available or for a matching receive. 45 Pay Attention to Deadlock Deadlock is a state when the program cannot proceed. Cyclic dependencies cause deadlock MPI_Comm_rank(MPI_COMM_WORLD,&rank); If(rank==0){ MPI_Recv(buf1,count,MPI_DOUBLE,1,tag,comm); MPI_Send(buf2,count,MPI_DOUBLE,1,tag,comm); } else if (rank==1) { MPI_Recv(buf1,count,MPI_DOUBLE,0,tag,comm); MPI_Send(buf2,count,MPI_DOUBLE,0,tag,comm); } 0 1 MPI_Comm_rank(MPI_COMM_WORLD,&rank); If(rank==0){ MPI_Ssend(buf1,count,MPI_DOUBLE,1,tag,comm); MPI_Recv(buf2,count,MPI_DOUBLE,1,tag,comm); } else if (rank==1) { MPI_Ssend(buf1,count,MPI_DOUBLE,0,tag,comm); MPI_Recv(buf2,count,MPI_DOUBLE,0,tag,comm); } 46 8

9 Pay Attention to Deadlock Lack of buffer space may also cause deadlock MPI_Comm_rank(MPI_COMM_WORLD,&rank); If(rank==0){ MPI_Send(buf1,count,MPI_DOUBLE,1,tag,comm); MPI_Recv(buf2,count,MPI_DOUBLE,1,tag,comm); } else if (rank==1) { MPI_Send(buf1,count,MPI_DOUBLE,0,tag,comm); MPI_Recv(buf2,count,MPI_DOUBLE,0,tag,comm); } Deadlock if not enough buffer space! 47 Using Send-receive 2 remedies: 1) non-blocking communication, 2) send-recv MPI_SENDRECV: combine send and recv in one call Useful in shift operations; Avoid possible deadlock with circular shift operations Equivalent to: execute a nonblocking send and a nonblocking recv, and then wait for them to complete int MPI_Sendrecv(void *sendbuf, int sendcount, MPI_Datatype sendtype, int dest, int sendtag, void *recvbuf, int recvcount, MPI_Datatype recvtype, int source, int recvtag, MPI_Comm comm, MPI_Status *status) P0 P1 P2 P3 *sendbuf and *recvbuf may not be the same memory address 48 9

10 #include <mpi.h> #include <stdio.h> int main(int argc, char **argv) { int my_rank, ncpus; int left_neighbor, right_neighbor; int data_received=-1; int send_tag = 101, recv_tag=101; MPI_Status status; mpirun np 4 test_shift Example Among 4 processes, process 3 received from right neighbor: 0 Among 4 processes, process 2 received from right neighbor: 3 Among 4 processes, process 0 received from right neighbor: 1 Among 4 processes, process 1 received from right neighbor: 2 MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); MPI_Comm_size(MPI_COMM_WORLD, &ncpus); left_neighbor = (my_rank-1 + ncpus)%ncpus; right_neighbor = (my_rank+1)%ncpus; MPI_Sendrecv(&my_rank, 1, MPI_INT, left_neighbor, send_tag, &data_received, 1, MPI_INT, right_neighbor, recv_tag, MPI_COMM_WORLD, &status); printf("among %d processes, process %d received from right neighbor: %d\n", ncpus, my_rank, data_received); // clean up MPI_Finalize(); return 0; } 49 Non-Blocking Communications 50 10

11 Semantics Purpose: Important mechanism for overlapping communication and computations. Communication and computation may proceed concurrently. à You hide communications! Deadlock avoidance May avoid system buffering and memory-to-memory copying, and improve performance How to Use it: 1. Post communication requests ß non-blocking call: MPI_Isend 2. // do some useful work 3. Complete communication call ß MPI_Wait, MPI_Test, 51 Non-blocking send Semantics Non-blocking calls: MPI_Isend, MPI_Irecv Will return immediately. Merely post a request to system to initiate communication. However, communication is not completed. Cannot change the memory provided in these calls until the communication is completed by calling MPI_Wait or MPI_Test. Non-blocking receive 52 11

12 The Interface int MPI_Isend(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm, MPI_Request *request) int MPI_Irecv(void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Request *request) MPI_Request request is a handle to an internal MPI object. Everything about that non-blocking communication is through that handle. MPI_Request req1, req2; double A[10], B[10]; MPI_Isend(A, 10, MPI_DOUBLE, rank, tag, MPI_COMM_WORLD, &req1); MPI_Irecv(B, 10, MPI_DOUBLE, rank, tag, MPI_COMM_WORLD, &req2); 53 #include <mpi.h> #include <stdio.h> Example int main(int argc, char **argv) { int my_rank, ncpus; int left_neighbor, right_neighbor; int data_received=-1; int tag = 101; MPI_Status statsend, statrecv; MPI_Request reqsend, reqrecv; mpirun np 4 test_shift Among 4 processes, process 3 received from right neighbor: 0 Among 4 processes, process 2 received from right neighbor: 3 Among 4 processes, process 0 received from right neighbor: 1 Among 4 processes, process 1 received from right neighbor: 2 MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); MPI_Comm_size(MPI_COMM_WORLD, &ncpus); left_neighbor = (my_rank-1 + ncpus)%ncpus; right_neighbor = (my_rank+1)%ncpus; MPI_Isend(&my_rank,1,MPI_INT,left_neighbor,tag,MPI_COMM_WORLD,&reqSend); // comm start MPI_Irecv(&data_received,1,MPI_INT,right_neighbor,tag,MPI_COMM_WORLD,&reqRecv); // Here, you can do some useful computations MPI_Wait(&reqSend, &statsend); // complete comm MPI_Wait(&reqRecv, &statrecv); printf("among %d processes, process %d received from right neighbor: %d\n", ncpus, my_rank, data_received); // clean up MPI_Finalize(); return 0; } 54 12

13 Non-blocking Sends 4 communication modes, same semantics as blocking sends. MPI_ISEND standard mode MPI_IBSEND buffered mode MPI_ISSEND synchronous mode MPI_IRSEND ready mode Identical arguments as MPI_Isend int MPI_Ibsend(void *buf,int count,mpi_datatype datatype,int dest, int tag, MPI_Comm comm, MPI_Request *request) int MPI_Issend(void *buf,int count,mpi_datatype datatype,int dest, int tag, MPI_Comm comm, MPI_Request *request) int MPI_Irsend(void *buf,int count,mpi_datatype datatype,int dest, int tag, MPI_Comm comm, MPI_Request *request) 55 Completion Use MPI_Wait or MPI_Test to complete non-blocking communication Semantics: after MPI_Wait returns For send, message data has been safely stored away, and safe to access buffer. For receive, data has been received

14 MPI_Wait int MPI_Wait(MPI_Request *request, MPI_Status *status) *request is a handle returned from MPI_Isend, MPI_Irecv etc Will block until the communication completes (or fails) If request is from MPI_Isend, MPI_Irecv etc Will deallocate the request object (set request to MPI_REQUEST_NULL) Return in status the status information. for MPI_Irecv, hold additional information. For MPI_Isend, not much to be used MPI_Request req; MPI_Status status; MPI_Irecv(, &req); MPI_Wait(&req, &status); 57 MPI_Test int MPI_Test(MPI_Request *request, int *flag, MPI_Status *status) request MPI_Request object from MPI_Isend, etc flag true if communication complete; false if not yet If true, request object will be de-allocated, and set to MPI_REQUEST_NULL status contain status information if complete Does not block, return immediately Provide a mechanism for overlapping communication and computation Do useful computation; periodically check communication status; if not complete, go back to computation

15 MPI_Wait Variants Deal with an array of MPI_Requests: MPI_Request req[4]; MPI_Waitall: MPI_Waitall(int count, MPI_Request *request, MPI_Status *status) Blocks until all active requests in array complete; return status of all communications Deallocate request objects, set to MPI_REQUEST_NULL MPI_Waitany: MPI_Waitany(int count,mpi_request *req, int *index, MPI_Status *stat) Blocks until one of the active requests in array completes; return THE index and the status of completing request; deallocate that request object. If none completes, return index=mpi_undefined. MPI_Waitsome: MPI_Waitsome(int incount, MPI_Request *req, int *outcount, int *array_indices, MPI_Status *array_status) Blocks until at least one of the active communications completes; return associated indices and status of completed communications; deallocate objects. If none, outcount=mpi_undefined. MPI_Request req[2]; MPI_Status stat[2]; MPI_Isend(, &req[0]); MPI_Isend(, &req[1]); MPI_Waitall(2, req, stat); MPI_Request req[2]; MPI_Status stat; Int index; MPI_Isend(, &req[0]); MPI_Isend(, &req[1]); MPI_Waitany(2, req, &index, &stat); 60 MPI_Test Variants MPI_Testall: MPI_Testall(int count, MPI_Request *array_req, int *flag, MPI_Status *array_stat) Return flag=true if all active requests complete; return flag=false otherwise. If true, will de-allocate request objects, set to MPI_REQUEST_NULL. MPI_Testany: MPI_Testany(int count, MPI_Request *array_req, int *index, int *flag, MPI_Status *stat) If one of active comm completes, return flag=true the index and status of completing comm; deallocate that object. Return flag=false, index=mpi_undefined if none completes Return flag=true, index=mpi_undefined if none active requests. MPI_Testsome: MPI_Testsome(int incount, MPI_Request *array_req, int *outcount, int *array_indices, MPI_Status *array_stat) Return in outcount the number of completed active comm and associated indices and status of completing comm. If none completes, return outcount=0 if none active comm, outcount=mpi_undefined

16 CS 590: High Performance Computing MPI (3) Fengguang Song Department of Computer Science IUPUI 16

17 Collective Communications 74 Overview of Collective Communications All processes in a group participate in a communication, by calling the same function with matching arguments. Types of collective operations: Synchronization: MPI_Barrier Collective data movement: MPI_Bcast, MPI_Scatter, MPI_Gather, MPI_Allgather, MPI_Alltoall Collective computation: MPI_Reduce, MPI_Allreduce, MPI_Scan Collective routines are blocking Completion of function call means the communication buffer can be accessed 75 17

18 Overview of Collective Communications MPI guarantees messages from collective communications will not be confused with P2P communications. All processes must participate in communication If you want only a sub-group of processes involved in collective communication, need to create a sub-group/subcommunicator from MPI_COMM_WORLD 76 Barrier int MPI_Barrier(MPI_Comm comm) Blocks the calling process until all group members have called it. Affect program performance Refrain from using it. MPI_Barrier(MPI_COMM_WORLD); // synchronization point 77 18

19 Broadcast int MPI_Bcast(void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm) Broadcasts a message from process with rank root to all processes in group, including itself. comm and root must be the same in all processes count and datatype must be the same for all processes Exception: could be different when derived datatypes are involved. int num=-1; If(my_rank==0) num=100; //the ROOT MPI_Bcast(&num, 1, MPI_INT, 0/*root*/, MPI_COMM_WORLD); 78 Scatter Int MPI_Scatter(void *sendbuf, int sendcount, MPI_Datatype *sendtype, void *recvbuf, int recvcount, MPI_Datatype *recvtype, int root, MPI_Comm comm) Inverse to MPI_Gather Split message into ncpus equal segments i-th segment to i-th process sendbuf, sendcount, sendtype important only at root, ignored in other processes. sendcount is the number of items sent to each process, not the total number of items in sendbuf 79 19

20 Scatter Example int A[1000], B[100]; //10 processes in total... // initializa A etc MPI_Scatter(A, 100/*not 1000*/, MPI_INT, B, 100, MPI_INT, 0, MPI_COMM_WORLD); 80 Gather Int MPI_Gather(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm) Gathers message to root; concatenated based on rank order at root process Recvbuf, recvcount, recvtype are only important at root; ignored in other processes. root and comm must be identical on all processes. recvbuf and sendbuf cannot be the same on root process. recvcount=sendcount, recvtype=sendtype. recvcount is the number of items received from each process, not the total number of items received, and not the size of receive buffer! 81 20

21 Gather Example int rank, ncous; int root = 0; int *data_received=null, data_send[100]; // assume running with 10 cpus MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &ncpus); if(rank==root) data_received = new int[100*ncpus]; // 100*10 MPI_Gather(data_send, 100, MPI_INT, data_received, 100 /*not 100*ncpus!*/, MPI_INT, root, MPI_COMM_WORLD); 82 All Gather (i.e., everyone gathers) Int MPI_Allgather(void *sendbuf, int sendcount, MPI_Datatype *sendtype, void *recvbuf, int recvcount, MPI_Datatype *recvtype, root, MPI_Comm comm) Concatenated messages according to rank order received by all processes Again, recvcount is the number of items from each process, not the total number of items received Again, sendcount=recvcount,sendtype=recvtype int A[100], B[1000]; // assume 10 processors MPI_Allgather(A, 100, MPI_INT, B, 100, MPI_INT, MPI_COMM_WORLD); // ok 83 21

22 All-to-All MPI_Alltoall can be seen as a global transposition operation, acting on chunks of data. int MPI_Alltoall(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm) Most stressful communication. Important for distributed matrix transposition, and critical to FFT-based algorithms sendcount is the number of items sent to each process, not the total number of items in sendbuf. recvcount is the number of items received from each process, not the total number of items received. For now, sendcount=recvcount, sendtype=recvtype 84 All-to-All Example double A[4], B[4];... // assume 4 cpus for(i=0;i<4;i++) A[i] = my_rank*4 + i; MPI_Alltoall(A, 1, MPI_DOUBLE, B, 1, MPI_DOUBLE, MPI_COMM_WORLD); Cpu 0 Cpu 1 Cpu 2 Cpu

23 Reduction Perform global reduction operations (sum, max, min, and, etc) across processors. MPI_Reduce return result to one processor MPI_Allreduce return result to all processors MPI_Scan parallel prefix operation 86 Reduction Int MPI_Reduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) Element-wise combine data from input buffers across processors using operation op store results in output buffer on processor root. All processes must provide input/output buffers of the same length and data type. Operation op must be associative: Pre-defined operations User can define own operations int rank, res; MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Reduce(&rank, &res, 1, MPI_INT, MPI_MAX, 0, MPI_COMM_WORLD); 87 23

24 Pre-Defined Operations MPI_MAX MPI_MIN MPI_SUM MPI_PROD MPI_LAND MPI_LOR MPI_BAND MPI_BOR MPI_LXOR MPI_BXOR MPI_MAXLOC MPI_MINLOC Logical AND Logical OR Bitwise AND Bitwise OR max + location min + location 88 All Reduce int MPI_Allreduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) Reduction result stored on all processors

25 Scan Int MPI_Scan(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) Prefix reduction To process j, return results of reduction on input buffers of processes 0, 1,, j. int MPI_Op_create(MPI_User_function *function, int commute, MPI_Op *op) 90 25

26 Process Subgroups and Subcommunicators Process Virtual Topology Process Groups & Communicators You can create sub-groups of processes, or subcommunicators Must create sub-groups or sub-communicators from some existing groups or communicators e.g., start from MPI_COMM_WORLD (or MPI_COMM_SELF) 99 26

27 Sub-communicators: How? MPI_Comm Communicator MPI_Group Process Group Sub-communicator Process sub-group 100 Facts about groups and communicators Group: ordered set of processes each process in group has a unique rank within that group process can belong to more than one group Rank is always relative to a group Communicators: from the programming viewpoint, groups and communicators are equivalent why distinguish between groups and communicators? sometimes you want to create two communicators for the same group useful for supporting libraries Groups and communicators are dynamic objects and can be created and destroyed during the execution of the program 27

28 Typical Usage 1. Extract handle of the global group from MPI_COMM_WORLD using MPI_Comm_group 2. Form new group as a subset of global group using MPI_Group_incl or MPI_Group_excl 3. Create new communicator for the new group using MPI_Comm_create 4. Determine new rank in new communicator using MPI_Comm_rank 5. Conduct communications using any MPI message passing routine 6. When finished, free up new communicator and group (optional) using MPI_Comm_free and MPI_Group_free Communicator à Process Group MPI_Comm_group gets group associated with communicator int MPI_Comm_group(MPI_Comm comm, MPI_Group *group)... MPI_Group group; MPI_Comm_group(MPI_COMM_WORLD, &group);

29 Details MPI_Group_excl: create a new group by removing certain processes from existing group int MPI_Group_excl(MPI_Group group, int n, int *ranks, MPI_Group *newgroup); ranks: containing ranks to be excluded MPI_Group_incl: create a new group from certain selected processes from existing group int MPI_Group_incl(MPI_Group group, int n, int *ranks, MPI_Group *newgroup); ranks: containing ranks to be included Process Group à Communicator MPI_Comm_create(): create a communicator from a process group group is a sub-group of that associated with comm Collective operation int MPI_Comm_create(MPI_Comm comm, MPI_Group group, MPI_Comm *newcomm) MPI_Group MPI_GROUP_WORLD, slave; MPI_Comm slave_comm; int ranks[1]; Ranks[0] = 0; MPI_Comm_group(MPI_COMM_WORLD, &MPI_GROUP_WORLD); MPI_Group_excl(MPI_GROUP_WORLD, 1, &ranks, &slave); MPI_Comm_create(MPI_COMM_WORLD, slave, &slave_comm);... MPI_Group_free(&slave); MPI_Comm_free(&slave_comm);

30 main(int argc, char **argv) { int me, count, count2; void *send_buf, *recv_buf, *send_buf2, *recv_buf2; MPI_Group MPI_GROUP_WORLD, grprem; MPI_Comm commslave; static int ranks[] = {0}; MPI_Init(&argc, &argv); MPI_Comm_group(MPI_COMM_WORLD, &MPI_GROUP_WORLD); MPI_Comm_rank(MPI_COMM_WORLD, &me); MPI_Group_excl(MPI_GROUP_WORLD, 1, ranks, &grprem); MPI_Comm_create(MPI_COMM_WORLD, grprem, &commslave); if(me!= 0){ /* compute on slave */ MPI_Reduce(send_buf,recv_buff,count, MPI_INT, MPI_SUM, 1, commslave); } /* zero falls through immediately to this reduce, others do later... */ MPI_Reduce(send_buf2, recv_buff2, count2, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD); MPI_Comm_free(&commslave); MPI_Group_free(&MPI_GROUP_WORLD); MPI_Group_free(&grprem); MPI_Finalize(); } Communicator Splitter int MPI_Comm_split(MPI_Comm comm, int color, int key, MPI_Comm *newcomm) Partition comm into disjoint sub-communicators, one for each value color Processes providing same color values will belong to the same new communicator newcomm Within newcomm, processes ranked based on key values that are provided (optional) color must be non-negative. If MPI_UNDEFINED is provided in color, MPI_COMM_NULL will be returned in newcomm. MPI_Comm_rank(MPI_COMM_WORLD, &rank); myrow = (int)(rank/ncol); //answer is 0, 1,... nrows-1 MPI_Comm_split(MPI_COMM_WORLD, myrow, rank, row_comm);

31 Virtual Topology 1-D Ranking (0, 1,, N-1) often does not reflect logical communication patterns of processes Desirable to arrange processes to reflect the underlying problem geometry or numerical algorithm, e.g. 2D or 3D grids Virtual topology: logical process arrangement Virtual topology does not reflect machine s physical topology 110 Cartesian Topology n-dimensional Cartesian topology (m1, m2, m3,, mn) grid; mi is the number of processes in i-th direction A process can be represented by its coordinate (j1, j2, j3,, jn) Constructor: MPI_Cart_create()

32 Cartesian Constructor int MPI_Cart_create(MPI_Comm comm_old, int ndims, int *dims, int *periods, int reorder, MPI_Comm *comm_cart) comm_old: existing communicator ndims: dimension of Cartesian topology dims: dimension [ndims], number of processes in each direction periods: dimension [ndims], true or false, periodic or not in each direction reorder: trueàmay reorder rank; false, no re-ordering of ranks comm_cart: new communicator with Cartesian topology MPI_Comm comm_new; int dims[2], periods[2], ncpus; MPI_Comm_size(MPI_COMM_WORLD, &ncpus); dims[0] = 2; dims[1] = ncpus/2; // assume ncpus dividable by 2 periods[0] = periods[1] = 1; MPI_Cart_create(MPI_COMM_WORLD, 2, dims, periods, 0, &comm_new); Topology Inquiry Functions int MPI_Cartdim_get(MPI_Comm comm, int *ndims) int MPI_Cart_coords(MPI_Comm comm, int rank, int maxdims, int *coords) int MPI_Cart_rank(MPI_Comm, int *coords, int *rank) MPI_Cartdim_get returns the dimension of Cartesian topology MPI_Cart_coords returns the coordinates of a process of rank MPI_Cart_rank returns of the rank of a process with coordinate *coords MPI_Comm comm_cart; int ndims, rank, *coords; // create Cartesian topology on comm_cart MPI_Comm_rank(comm_cart, &rank); MPI_Cartdim_get(comm_cart, &ndims); coords = new int[ndims]; MPI_Cart_coords(comm_cart, rank, ndims, coords); // coords contains coord MPI_Cart_rank(comm_cart, coords, &rank);