CPS 303 High Performance Computing

Transcription

1 CPS 303 High Performance Computing Wensheng Shen Department of Computational Science SUNY Brockport

2 Chapter 5: Collective communication The numerical integration problem in Chapter 4 is not very efficient. After process 0 reads the input data, it sends the data to other processes in an increasing order in rank. In other words, lower ranked processes receive the data early then higher ranked processes, that results in an idle time of higher ranked processes. This is not a desired feature of parallel computing. Similar situation happens when process 0 does all the work of collecting and adding. The main point of parallel computing is to get multiple processes to collaborate on solving a problem. We will deal with this issue in Chapter 5.

3 5.1 Tree-structured communication Stage 1: 0 1 Stage 2: Last stage: number of stages: log 2 p If we have 1024 processes, how many stages do we need to send a message from the root to all other processes? Great savings in time.

4 The non tree-structured communication To send data from 1 process to all other processes: If (my_rank == 0) for (i=1; i<p; i++) send data to process i Else for(i=1; i<p; i++) receive data from process o There are p-1 stages of send and receive!

5 Modify the Get_data function to use a treestructured distribution scheme for (stage = first; stage <= last; stage++) { if(i_receive(stage, my_rank, &source) ) Receive(data, source); else if (I_send(stage, my_rank, p, &dest) ) Send(data, dest); If during the current stage the calling process receives data, then I_receive function returns 1, otherwise it returns 0. To implement this, we have to know (1) Whether a process receives and if so, the source. (2) Whether a process sends and if so, the destination.

6 To implement tree-structured communication (1) 0 sends to 1; (2) 0 sends to 2; 1 sends to 3; (3) 0 sends to 4; 1 sends to 5; 2 sends to 6; 3 sends to 7. If 2 stage my _ rank < Then I receive from If my _ rank < 2 2 stage+ 1 my _ rank 2 stage stage Then I send to my _ rank + 2 stage Note: using C convention, the first stage is stage 0, the second is stage 1

7 Verifications Stage 0: my_rank < 2 0 (process 0) 0 sends to my_rank =1 2 0 =<my_rank<2 1 (prcocess 1) 1 receives from =0 Stage 1: my_rank < 2 1 (processes 0, 1) 0 sends to my_rank =2 1 sends to my_rank =3 2 1 =<my_rank<2 2 2 receives from =0 3 receives from =1

8 int Celing_log2(int x ) { /* use unsigned so that right shift will fill leftmost bit with 0 */ unsigned temp=(unsigned) x-1; int result = 0; while (temp!= 0) { temp = temp >> 1; result = result + 1; return result; Int I_receive(int stage, int my_rank, int* source_ptr) { int power_2_stage; power_2_stage = 1 << stage; if ((power_2_stage<=my_rank) && (my_rank < 2*power_2_stage)) { *source_ptr = my_rank power_2_stage; return 1; else return 0; void Send(float a, float b, int n, int dest) { MPI_Send(&a,1,MPI_FLOAT,dest,0,MPI_COMM_WORLD); MPI_Send(&b,1,MPI_FLOAT,dest,1,MPI_COMM_WORLD); MPI_Send(&n,1,MPI_INT,dest,2,MPI_COMM_WORLD); Void Receive(float* a_ptr, float* b_ptr, int* n_ptr, int source) { MPI_Status status; MPI_Recv(a_ptr,1,MPI_FLOAT,source,0,MPI_COMM_WORLD,&status); MPI_Recv(b_ptr,1,MPI_FLOAT,source,1,MPI_COMM_WORLD,&status); MPI_Recv(n_ptr,1,MPI_FLOAT,source,2,MPI_COMM_WORLD,&status); Void Get_data1(float* a_ptr, float* b_ptr, int* n_ptr, int my_rank, int p) { int source; int dest; int stage; if(my_rank==0) { printf( enter a, b, and n\n ); scanf( %f %f %d, a_ptr, b_ptr, c_ptr); for(stage=0; stage<ceiling_log2(p); stage++) { if(i_receive(stage, my_rank, &source); Receive(a_ptr, b_ptr, n_ptr, source); else if(i_send(stage, my_rank, p, &dest) Send(*a_ptr, *b_ptr, *n_ptr, dest);

9 5.2 Broadcast Collective communication: a communication pattern that involves all the processes in a communicator. Broadcast: a broadcast is a collective communication in which a single process sends the same data to every process in the communicator. Hand-coded broadcast is very complicated and the programmer has to know the system topology. Root: the process that has data to send and initiates the broadcast function. MPI has a built-in broadcast function, MPI_Bcast();

10 The syntax of MPI_Bcast() int MPI_Bcast(void * message /* in/out */, int count /* in */, MPI_Datatype datatype /* in */, int root /* in */, MPI_Comm comm /* in */) MPI_Bcase sends a copy to the data in message on the process with rank root to each process in the communicator comm. It is called by all the processes in the communicator. A broadcast message cannot be received with MPI_Recv. Count and datatype are the same so all processes in the communicator. Message is identified as an in/out parameter, in the case of MPI_Bcast, message is in on the root process, and out on other processes.

11 New version of Get_data() Void Get_data2() { float* a_ptr /* out */, float* b_ptr /* out */, int* n_ptr /* in */, int my_rank /* in */ if(my_rank == 0) { printf( enter a, b, and n\n ); scanf( %f %f %d, a_ptr, b_ptr, n_ptr); MPI_Bcast(a_ptr, 1, MPI_FLOAT, 0, MPI_COMM_WORLD); MPI_Bcast(b_ptr, 1, MPI_FLOAT, 0, MPI_COMM_WORLD); MPI_Bcast(n_ptr, 1, MPI_INT, 0, MPI_COMM_WORLD);

12 Performance comparison processes ncube2 Paragon SP2 Version 1 Version 2 Version 1 Version 2 Version1 Version Broadcast time (times are in milliseconds; version 1 uses a linear loop of sends from process 0, version 2 uses MPI_Bcast)

13

14 5.3 Tags, safety, buffering, and synchronization MPI_Bcast does not use tags, all other collective communication functions on MPI do not use tags. Why? Look at some examples.

15 Time Process A MPI_Send to B, tag=0 MPI_Send to B, tag=1 Local work Local work Process B Local work Local work MPI_Recv for A, tag=1 MPI_Recv from A, tag=0 For the above send and receive sequences, process A sends multiple messages to process B, and B decided how to handle these messages on the basis of the tag. The send/receive sequence of events requires that the system buffers the messages being sent to process B. Memory must be set aside for storing messages before a receive has been executed. The message envelop contains: (1) the rank of the receiver, (2) the rank of the sender, (3) a tag, (4) a communicator. There is no information specifying where the message should be stored by the receiving process, until B calls MPI_Recv When B calls MPI_Recv, the system software, after executing the receive function, can see which buffered message has an envelop that matches the parameters specified by the receive. If there isn t one, it waits until one arrives. In the case of no buffering, A must wait until it receives a RTR (ready to receive) signal from B with tag 1, similarly, B must wait until it receives a RTS (ready to send) signal from A with tag 0. The program will deadlock. A waits for B, and B waits for A.

16 The case of broadcast Time Process A Process B Process C 1 MPI_Bcast &x Local work Local work 2 MPI_Bcast &y Local work Local work 3 Local work MPI_Bcast &y MPI_Bcast &x 4 Local work MPI_Bcast &x MPI_Bcast &y Suppose we have three processes, A, B, and C, and A is broadcasting two floats, x and y, to B and C. Suppose that on process A, x=5 and y=10. When the broadcasting is completed on all three processes, x=5 and y=10 on processes A and C, but x=10 and y=5 on process B. why? Broadcasting operation was designed to be synchronized, i.e., a broadcast would not return until every process has received the broadcast data. This restriction has been relaxed in current systems. If the system has enough buffering, A can complete two broadcasts before the other processes even begin their call. However, the effect must be the same as if the processes are synchronized. In other words, the first MPI_Bcast on B matches the first MPI_Bcast on A. Therefore B stores the first value it receives, 5, in y.

17 5.4 Reduce In the final summation part of trapezoidal integration, process 0 has to do lots of extra work. Similar to sending input data to different processes, We can distribute the summation among all processes: 1 (a) 4 sends to 0; 5 sends to 1; 6 sends to 2; 7 sends to 3; (b) 0 adds its integral to that of 4; 1 adds its integral to that of 5; etc. 2 (a) 2 sends to 0; 3 sends to 1. (b) 0 adds; 1 adds. 3 (a) 1 sends to 0 (b) 0 adds This can be done using a reduction function MPI_Reduce();

18 Int MPI_Reduce( void* operand /* in */, void* result /* out */, int count /* in */, MPI_Datatype datatype /* in */, MPI_Op operator /* in */, int root /* in */, MPI_Comm comm /* in */) MPI_Reduce combines the operands stored in the memory referenced by operand using operation operator and stores the result in *result on root process. Both operand and result refer to count memory locations with type datatype. MPI_Reduce must be called by all processes in the communicator comm, and count, datatype, operator, and root must be the same on each process. The parameter operator can take on one of the predefined values listed in a table in the next slide.

19 Operation Name MPI_MAX MPI_MIN MPI_SUM MPI_PROD MPI_LAND MPI_BAND MPI_LOR MPI_BOR MPI_LXOR MPI_BXOR MPI_MAXLOC MPI_MINLOC Meaning Maximum Minimum Sum Product Logical and Bitwise and Logical or Bitwise or Logical exclusive or Bitwise exclusive or Maximum and location of maximum Minimum and location of minimum

20 The last several lines of the trapezoidal rule program can be rewritten as: /* add up the integrals calculated by each process */ MPI_Reduce(&integral, &total, 1, MPI_FLOAT, MPI_SUM, 0, MPI_COMM_WORLD); /* print the result */ Note: do not pass the same arguments to both operand and result. /* attempt to store the result in the same location as the operand. Illegal call */ MPI_Reduce(&integral, &integral, 1, MPI_FLOAT, MPI_SUM, 0, MPI_COMM_WORLD); /* print the result */ Reason: without the extra argument, an implementation may be forced to provide large temporary buffers.

21 5.5 Dot product If x=(x 0,x 1,..,x n-1 ) T and y=(y 0,y 1,,y n-1 ) T are n-dimensional vectors of real numbers, then their dot product is x y = x 0 y 0 + x 1 y x n-1 y n-1 Float Serial_dot( float x[] /* in */, float y[] /* in */, int n /* in */) { int i; float sum=0.0; for(i=0;i<n;i++) { sum = sum + x[i]*y[i]; return;

22

23 process 0 1 k p-1 components x 0,x 1,,x N-1 x N,x N+1,,x 2N-1 x kn,x kn+1,,x (k+1)n-1 x (p-1)n,x (p-1)n+1,,x n-1 If we have p processes, and n is divisible by p, we can divide the vectors among the processes so that each process has N=n/p components of the vector.

24 Parallel dot product Float paralle_dot( float local_x[] /* in */, float local_y[] /* in */, int n_bar /* in */) { float local_dot; float dot=0.0; float Serial_dot(float x[], float y[], int m); local_dot=serial_dot(local_x, local_y, n_bar); MPI_Reduce(&local_dot, &dot, 1, MPI_FLOAT, MPI_SUM, 0, MPI_COMM_WORLD); return dot;

25 5.6 Allreduce If we want each process to return the correct dot product? Call MPI_Bcast after MPI_Reduce A more efficient way: MPI_Allreduce()

26 A butterfly communication structure for eight processes

27 (1) a. processes 0 and 4 exchange their local results, process 1 and 5 exchange, processes 2 and 6 change, processes 3 and 7 exchange. b. each process adds its result to the result just received. (2) a. processes 0 and 2 exchange their intermediate results, processes 1 and 3 exchange theirs, processes 4 and 6 exchange, and processes 5 and 7 exchange. b. each process adds. (3) a. processes 0 and 1 exchange, processes 2 and 3 exchange, processes 4 and 5 exchange, and processes 6 and 7 exchange. b. each process adds.

28 int MPI_Allreduce( void* operand /* in */, void* result /* out */, int count /* in */, MPI_Datatype datatype /* in */, MPI_Op operator /* in */, MPI_Comm comm /* in */) MPI_Allreduce is used in exactly the same way as MPI_Reduce. The only difference is that the result of he reduction is returned in result on all the processes. Hence there is no root parameter.

29 5.7 Gather and scatter For an m n matrix A=(a ij ) m n and an n-dimensional vector x=(x 0,x 1,,x n-1 ), the matrix vector product y=ax can be performed by taking the dot product of each row of A with x. if A has m rows, we will then form m dot products. The product vector y will consist of m entries: y=(y 0,y 1,,y m-1 ) and y k =a k,0 x 0 +a k,1 x 1 + +a k,n-1 x n-1

30

31 Void Serial_matrix_vector_prod( MATRIX_T A /* in */, int m /* in */, int n /* in */, float x[] /* in */, float y[] /* out */) { int k, j; for (k=0; k<m; k++) { y[k]=0.0; for(j=0; j<n; j++) y[k] = y[k] + A[k][j]*x[j];

32 Block-row partition In block-row partition, we partition the matrix into blocks of consecutive rows, and assign a block to each process. Example m=8, n=4, and p=4: process a 00 a 20 a 40 a 60 a 01 a 11 a 21 a 31 a 41 a 51 a 61 a 70 a 63 a 71 Elemens of A a 02 a 12 a 22 a 32 a 42 a 52 a 62 a 72 a 13 a 10 a 03 a 33 a 30 a 23 a 53 a 50 a 43 a 73

33 A x y Process 0 Process 1 Process 2 = Process 3 Mappings of A, x and y for matrix-vector product

34 In order to form the dot product of each row of A with x, we need to either gather all of x onto each process or scatter each row of A across the processes. For example, suppose, m=n=p=4, then, before we form the dot product, a 00, a 01, a 03, and x 0 are assigned to process 0, while x 1 is assigned to process 1, x 2 is assigned to process 2, and x 3 is assigned to process 3. To form the dot product of the first row of A with x, we can either send x 1, x 2, and x 3 to process gather or we can send a 01 to process 1, a 02 to process 2, and a 03 to process scatter.

35 Process 0 x0 Process 1 Process 2 x1 x2 Gather Process 3 x3 /* assumes n is divisible by p */ MPI_Gather(local_x, n/p, MPI_FLOAT, global_x, n/p, MPI_FLOAT, 0, MPI_COMM_WORLD);

36 int MPI_Gather( void* send_data /* in */, int send_count /* in */, MPI_Datatype send_type /* in */, void* recv_data /* out */, int recv_count /* in */, MPI_Datatype recv_type /* in */, int root /* in */, MPI_Comm comm /* in */) MPI_Gather collects the data referenced by send_data from each process in the communicator comm and stores the data in the root process in the memory referenced by recv_data.

37

38 Process 0 a 00 a 01 a 02 a 03 Process 1 Scatter Process 2 Process 3 /* Assumes n is divisible by p */ MPI_Scatter(&(localA[0][0]), n/p, MPI_FLOAT, row_segment, n/p, MPI_FLOAT, 0, MPI_COMM_WORLD);

39 int MPI_Scatter( void* send_data /* in */, int send_count /* in */, MPI_Datatype send_type /* in */, void* recv_data /* out */, int recv_count /* in */, MPI_Datatype recv_type /* in */, int root /* in */, MPI_Comm comm /* in */) MPI_Scatter splits the data referenced by send_data on the root process into p segments, each of which consists of send_count elements of type send_type. The first segment is sent to process 0, the second to process 1, etc. The send parameters are significant only on the root process. In most cases send_count will be the same as recv_count and send_type will be the same as recv_type. The parameters root and comm must be the same on all processes.

40

41 5.8 Allgather How to carry parallel matrix-vector multiplication using MPI_Gather() and MPI_Scatter()? MPI_Scatter: suppose we scatter each row of A, then after the scatter of row 0, process q will compute a 0q x q, i.e., the individual terms in the dot product will be assigned to different processes. Thus we will need to call MPI_Reduce to finish the dot product. More communication is needed. Process Products a 00 x 0 a 01 x 1 a 02 x 2 a 03 x 3

42 MPI_Gather(): we can gather x onto each process and form the dot product of each local row of A with x. The kth element of y is assigned to the same process as the kth row of A, and the kth element of y is computed by forming the dot product of the kth row of A with x. To do this, we need to call MPI_Gather() p times. for( root=0; root<p; root++) { MPI_Gather(local_x, n/p, MPI_FLOAT, global_x, n/p, MPI_FLOAT, root, MPI_COMM_WORLD);

43 A better approach based on butterfly scheme is used to simultaneously gather all of x onto each process, the MPI_Allgather() function. int MPI_Allgather( void* send_data /* in */, int send_count /* in */, MPI_Datatype send_type /* in */, void* recv_data /* out */, int recv_count /* in */, MPI_Datatype recv_type /* in */, MPI_comm comm /* in */) It gathers the contents of each process s send_data into each process s recv_data.

44 /* all arrays are allocated in calling program */ Void Parallel_matrix_vector_prod( LOCAL_MATRIX_T local_a /* in */, int m /* in */, int n /* in */, float local_x[] /* in */, float global_x[]/* in */, float local_y[] /* out */, int local_m /* in */, int local_n /* in */) { /* local_m = m/p, local_n = n/p */ int i, j; MPI_Allgather(local_x, local_n, MPI_FLOAT, global_x, local_n, MPI_FLOAT, MPI_COMM_WORLD); for(i=0; i<local_m; i++) { local_y[i] = 0.0; for (j=0; j<n; j++) { local_y[i] = local_y[i] + local_a[i][j]*global_x[j];

45

46 5.9 Summary Collective communication is the communication that involves all the processes in a communicator. In broadcast, a single process sends the same data to all the other processes in a communicator. In reduction, an operation works on a collection of numbers distributed across the process. In gather, a distributed data structure is collected onto a single process. In scatter, a data structure that is stored on a single process is distributed across the processes.

47 #include <stdio.h> #include "mpi.h" #define MAX_ORDER 100 typedef float LOCAL_MATRIX_T[MAX_ORDER][MAX_ORDER]; main(int argc, char* argv[]) {int my_rank; int p; LOCAL_MATRIX_T local_a; float global_x[max_order]; float local_x[max_order]; float local_y[max_order]; int m, n; int local_m, local_n; void Read_matrix(char* prompt, LOCAL_MATRIX_T local_a, int local_m, int n, int my_rank, int p); void Read_vector(char* prompt, float local_x[], int local_n, int my_rank, int p); void Parallel_matrix_vector_prod( LOCAL_MATRIX_T local_a, int m, int n, float local_x[], float global_x[], float local_y[], intlocal_m, intlocal_n); void Print_matrix(char* title, LOCAL_MATRIX_T local_a, int local_m, int n, int my_rank, int p); void Print_vector(char* title, float local_y[], int local_m, int my_rank, int p); MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD, &p); MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); if (my_rank == 0) { printf("enter the order of the matrix (m x n)\n"); scanf("%d %d", &m, &n); MPI_Bcast(&m, 1, MPI_INT, 0, MPI_COMM_WORLD); MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD); local_m = m/p; local_n = n/p; Read_matrix("Enter the matrix", local_a, local_m, n, my_rank, p); Print_matrix("We read", local_a, local_m, n, my_rank, p); Read_vector("Enter the vector", local_x, local_n, my_rank, p); Print_vector("We read", local_x, local_n, my_rank, p); Parallel_matrix_vector_prod(local_A, m, n, local_x, global_x, local_y, local_m, local_n); Print_vector("The product is", local_y, local_m, my_rank, p); MPI_Finalize();

48 /**********************************************************************/ void Read_matrix( char* prompt /* in */, LOCAL_MATRIX_T local_a /* out */, int local_m /* in */, int n /* in */, int my_rank /* in */, int p /* in */) { int i, j; LOCAL_MATRIX_T temp; /* Fill dummy entries in temp with zeroes */ for (i = 0; i < p*local_m; i++) for (j = n; j < MAX_ORDER; j++) temp[i][j] = 0.0; if (my_rank == 0) { printf("%s\n", prompt); for (i = 0; i < p*local_m; i++) for (j = 0; j < n; j++) scanf("%f",&temp[i][j]); MPI_Scatter(temp, local_m*max_order, MPI_FLOAT, local_a, local_m*max_order, MPI_FLOAT, 0, MPI_COMM_WORLD); /* Read_matrix */

49 /**********************************************************************/ void Read_vector( char* prompt /* in */, float local_x[] /* out */, int local_n /* in */, int my_rank /* in */, int p /* in */) { int i; float temp[max_order]; if (my_rank == 0) { printf("%s\n", prompt); for (i = 0; i < p*local_n; i++) scanf("%f", &temp[i]); MPI_Scatter(temp, local_n, MPI_FLOAT, local_x, local_n, MPI_FLOAT, 0, MPI_COMM_WORLD); /* Read_vector */ /**********************************************************************/

50 /**********************************************************************/ /* Note that argument m is unused */ void Parallel_matrix_vector_prod( LOCAL_MATRIX_T local_a /* in */, int m /* in */, int n /* in */, float local_x[] /* in */, float global_x[] /* in */, float local_y[] /* out */, int local_m /* in */, int local_n /* in */) { /* local_m = m/p, local_n = n/p */ int i, j; MPI_Allgather(local_x, local_n, MPI_FLOAT, global_x, local_n, MPI_FLOAT, MPI_COMM_WORLD); for (i = 0; i < local_m; i++) { local_y[i] = 0.0; for (j = 0; j < n; j++) local_y[i] = local_y[i] + local_a[i][j]*global_x[j];