MPI Groups and Communicators

Transcription

1 MPI Groups nd Communictors We now lern techniques to modulrize communictions t more locl level so tht messges do not interfere with ech other We do this by setting up Groups of processes We then crete new communictor bsed on the subgroup structure All MPI communiction commnds involve communictor, so fr we hve only used MPI_COMM_WORLD; we cn now use other communictors bsed on group structures

2 MPI Group Commnds MPI_Group_size returns number of processes in group MPI_Group_rnk returns rnk of clling process in group MPI_Group_compre compres group members nd group order MPI_ Group_ trnslte_ rnks trnsltes rnks of processes in one group to those in nother group MPI_Comm_group returns the group ssocited with communictor MPI_Group_union cretes group by combining two groups MPI_Group_intersection cretes group from the intersection of two groups MPI_Group_difference cretes group from the difference between two groups MPI_Group_incl cretes group from listed members of n existing group MPI_Group_excl cretes group excluding listed members of n existing group MPI_Group_rnge_incl cretes group ccording to first rnk, stride, lst rnk MPI_Group_rnge_excl cretes group by deleting ccording to first rnk, stride, lst rnk MPI_Group_free mrks group for delloction These re the most common of the commnds Detils on the prmeters will be given s prt of the exercises

3 MPI Communictor Commnds MPI_Comm_size returns number of processes in communictor's group MPI_Comm_rnk returns rnk of clling process in communictor's group MPI_ Comm_ compre compres two communictors MPI_Comm_dup duplictes communictor MPI_Comm_crete cretes new communictor for group MPI_Comm_split splits communictor into multiple, non-overlpping overlpping communictors MPI_Comm_free mrks communictor for delloction These re the most common of the commnds Detils on the prmeters will be given s prt of the exercises

4 Exmple: Eight Subgroups of Size 4 Assume 32 processes re vilble Here re the commnds to crete these groups rnks1[4]={0,1,2,3}, rnks2[4]={4,5,6,7}; rnks3[4]={8,9,10,11}, rnks4[4]={12,13,14,15}; rnks5[4]={16,17,18,19}, rnks6[4]={20,21,22,23}; rnks7[4]={24,25,26,27}, rnks8[4]={28,29,30,31}; MPI_Group orig_group, new_group; MPI_Comm new_comm; : : : /* initilize, get size, get rnks, etc. here /* Extrct the originl group hndle */ MPI_Comm_group(MPI_COMM_WORLD, &orig_group); /* Divide tsks into eight distinct groups bsed upon rnk */ if (rnk < NPROCS/8) { MPI_ Group_ incl(orig g_group, NPROCS/8, rnks1, &new_group); } else. /* other seven cses */ /* Crete new new communictor */ MPI_Comm_crete(MPI_COMM_WORLD, new_group, &new_comm);

5 In-Clss Lb You will use eight processes Process 0, 1, 2, 3 will be ssigned to the first group Process 4, 5, 6, 7 will be ssigned to the second group Ech process will know its originl rnk nd its new rnk within the new groups You will crete one new communictor bsed on these groups Using the new communictor Hve ech process sum up the originl rnks in its group using reduce ll The result for the first group is 6, the result for the second group is 22 The lb sheet describes this lb in detil; check off with the lb ssistnt when finished

6 MPI_COMM_DUP nd MPI_COMM_FREE A progrm my crete new communictor comprising the sme process group Uses new context to ensure tht communictions performed for different purposes re not confused. This mechnism supports sequentil composition. A communictor is discrded fter it is done

7 A Simple Exmple We wnt to trnspose mtrix using locl communictor tht will not interfere with other communictions integer comm, newcomm, ierr! Hndles re integers... cll MPI_COMM_DUP(comm, newcomm, ierr)! Crete new context cll trnspose(newcomm, A)! Pss mtrix A to librry cll MPI_COMM_FREE(newcomm, ierr)! Free new context This exmple is in Fortrn This exmple is in Fortrn, but you get the ide

8 MPI_COMM_SPLIT A progrm my crete new communictor comprising just subset of given group of processes. These processes cn then communicte mong themselves without fer of conflict with other concurrent computtions. This mechnism supports prllel composition.

9 A Simple Exmple The gol is to split eight processes into three communictors with the ID of ech process defined by myid mod 3 Notice tht in the subcommunictors the numbering strts t 0 nd increses by 1 MPI_Comm comm, newcomm; int myid, color; MPI_Comm_rnk(comm, &myid); color = myid%3; MPI_Comm_split(comm, color, myid, &newcomm);

10 Progrm 12 Wht will be printed by this progrm? (ssume n 8) #include <stdio.h> #include <mpi.h> min(int rgc, chr* rgv[]) { MPI_Comm MY_COMM; int world_rnk,my_rnk,my_size; sttic int vlue[12]={0,0,1,1,1,2,2,2,3,3,3,3}; int color; MPI_Init(&rgc,&rgv); MPI_Comm_rnk(MPI_COMM_WORLD,&world_rnk); COMM k) color=vlue[world_rnk]; MPI_Comm_split(MPI_COMM_WORLD,color,world_rnk,&MY_COMM); MPI_Comm_rnk(MY_COMM,&my_rnk); COMM k) MPI_Comm_size(MY_COMM,&my_size); printf("world_rnk=%d my_rnk=%d color=%d my_size=%d\n", world_rnk,my_rnk,color,my_size); k l } MPI_Finlize();

11 prog12]$ mpirun -n 8./prog12 world_rnk=0 my_rnk=0 color=0 my_size=2 world_rnk=2 my_rnk=0 color=1 my_size=3 world_rnk=6 my_rnk=1 color=2 my_size=3 world_rnk=5 my_rnk=0 color=2 my_size=3 world_rnk=1 my_rnk=1 color=0 my_size=2 world_rnk=4 my_rnk=2 color=1 my_size=3 world_ rnk=3 my_ rnk=1 color=1 my_ size=3 world_rnk=7 my_rnk=2 color=2 my_size=3 prog12]$

12 MPI_INTERCOM_CREATE A progrm my construct n intercommunictor This links processes in two groups. This mechnism supports prllel composition.

13 A Simple Exmple Suppose MPI_COMM_SPLIT hs tken 8 processes nd creted two distinct communictors: the first contins the even elements nd the second contins the odd elements The gol is the set up intercommunictions between these two communictors

14 The Progrm Code this exmple is in FORTRAN, the intent should be cler

15 #include <stdio.h> #include <mpi.h> Progrm 13-1 min(int rgc, chr* rgv[]) { MPI_Comm MY_COMM; MPI_ Comm THE_ INTER; int world_rnk,locl_rnk, inter_rnk; int color, i; int rr[5]; MPI_Sttus sttus; MPI_Init(&rgc,&rgv); MPI_Comm_rnk(MPI_COMM_WORLD,&world_rnk); color=world_rnk%2; MPI_Comm_split(MPI_COMM_WORLD,color,0,&MY_COMM); if (color==0) { MPI_Intercomm_crete(MY_COMM,0,MPI_COMM_WORLD,1,33,&THE_INTER); MPI_Comm_rnk(MY_COMM, &locl_rnk); MPI_Comm_rnk(THE_INTER, &inter_rnk); if(locl_rnk == 0){ printf("color=%d inter_rnk=%d locl rnk %d: ", color, inter_rnk, locl_rnk); for(i = 0; i < 5; i++){ printf("%d ", rr[i]); } printf("\n"); ") MPI_Send(rr, 5, MPI_INT, 0, 0, THE_INTER); } }

16 Progrm 13-2 } } else if (color==1) { MPI_Intercomm_crete(MY_COMM,0,MPI_COMM_WORLD,0,33,&THE_INTER); MPI_Comm_rnk(MY_COMM, &locl_rnk); MPI_ Comm_ rnk(the _ INTER, &inter_ rnk); if(locl_rnk == 0){ printf("about to recieve. color=%d inter_rnk=%d P%d: ", color, inter_rnk, locl_rnk); MPI_Recv(rr, 5, MPI_INT, 0, 0, THE_INTER, &sttus); for(i = 0; i<5;i++){ printf("%d ", rr[i]); } printf("\n"); } } MPI_Comm_free(&THE_INTER); MPI_Finlize(); Wht is printed by this progrm?

17 prog13]$ mpirun -n 2./prog13 color=0 inter_rnk=0 locl rnk 0: About tto recieve. color=1 1inter_rnk=0 0P0: [blk@virtul prog13]$

18 Exercise: Using Intercommunictors Split the processes s shown: 0,2,4,6 in one group nd d1357i 1,3,5,7 in one group Set up intercommunictor between corresponding elements in subgroups Use send nd receive between pired processors to exchnge their world rnks; print the sums For the originl rnks 0..7 the results re 1, 1, 5, 5, 9, 9, 13, 13; the order will vry due to prllelism Hve the lb ssistnt check off your progrm when completed

19 MPI_Crt_crete Mkes new communictor to which h topology informtion hs been ttched Synopsis #include "mpi.h" int MPI_Crt_crete ( MPI_Comm comm_old, int ndims, int *dims, int *periods, int reorder, MPI_Comm *comm_crt ) Input Prmeters comm_old input communictor (hndle) ndims number of dimensions of crtesin grid (integer) dims integer rry of size ndims specifying i the number of processes in ech dimension periods logicl rry of size ndims specifying whether the grid is periodic (true) or not (flse) in ech dimension reorder rnking my be reordered (true) or not (flse) (logicl) Output Prmeter Output Prmeter comm_crt communictor with new crtesin topology (hndle)

20 MPI_Crt_coords Determines process coords in crtesin topology given rnk in group Synopsis #include "mpi.h" int MPI_Crt_coords ( MPI_Comm comm, int rnk, int mxdims, int *coords ) Input Prmeters comm communictor with crtesin structure (hndle) rnk rnk of process within group of comm (integer) mxdims length of vector coords in the clling progrm (integer) Output Prmeter coords integer rry (of size ndims) contining the Crtesin coordintes of specified process (integer)

21 MPI_Crt_sub Prtitions communictor into subgroups which h form lower-dimensionl crtesin subgrids Synopsis #include "mpi.h" int MPI_Crt_sub ( MPI_Comm comm, int *remin_dims, MPI_Comm *comm_new ) Input Prmeters comm communictor with crtesin structure (hndle) remin_dims the ith entry of remin_dims specifies whether the ith dimension is kept in the subgrid (true) or is dropped (flse) (logicl vector) Output Prmeter newcomm communictor contining the subgrid tht includes the clling process (hndle)

22 MPI_Crt_rnk Determines process rnk in communictor given Crtesin loction Synopsis #include "mpi.h" int MPI_Crt_rnk _ ( MPI_Comm comm, int *coords, int *rnk ) Input Prmeters comm communictor with crtesin structure t (hndle) coords integer rry (of size ndims) specifying the crtesin coordintes of process Output Prmeter rnk rnk of specified process (integer)

23 Mtrix Multipliction The clssic sequentil lgorithm for (int i = 0 ; i < ROWS ; i++) { for (int j = 0 ; j < COLUMNS ; j++) { double sum = 0.0 ; for (int k = 0 ; k < COLUMNS ; k++) { sum = sum + A[i][k] * B[k][j] ; } C[i][j] = sum ; } } The complexity for n n x n mtrix is O(n 3 )

24 How to Decompose the Mtrix Some choices Which choice do you think will be most productive?

25 Multipliction of Submtrices b b b b b b b b b b b b b b b b A00A A01 A 10 A B01 B B 10 B 11 = C 00 C C10 C C 00 = A00 B00 + A01 B10 C 01 = A00 B01 + A01 B11 C10 = A10 B00 + A11 B10 C11 = A10 B01 + A11 B11

26 Sub-block definition of Mtrix Multiply The entire mtrix is 16 x 16; ech sub block is 4 x 4 Note indices strt t 0 for rows nd columns of mtrices They strt t 1 for rows nd columns of processors

27 Fox s Algorithm (Brodcst, Multiply, nd Roll)

28 The first stge -- index n=0 in sub-block sum -- of the lgorithm on N=16 exmple

29 The second stge -- n=1 in sum over subblock indices -- of the lgorithm on N=16 exmple

30 Second stge, continued

31 Look t the whole lgorithm on one element

32 More Detils in Pseudocode Clculting sub-block C[i,j] q = sqrt(p); dest = ((i-1) mod q, j); source = ((i+1) mod q, j); for(stge = 0; stge < q; stge++) { k_br = (i + stge) mod q; Brodcst A[i, k_br] cross row i; C[i,j] = C[i,j] + A[i, k_ br] * B[k _ br, j] Send B[k_br, j] to dest; Receive B[(k_br+1) mod q, j] from source; } We will use communictors for the Brodcst, Send nd Receivee

33 Holding informtion in struct We will use struct to fcilitte pssing heterogeneous set of informtion esily typedef struct { int p; /* number of processes */ MPI_Comm comm; /* Communictor for entire grid */ MPI_Comm row_comm; /* Communictor for my row */ MPI_ Comm col_ comm; /* Communictor for my col */ int q; /* order of grid */ int my_row; /* my row number */ int my_col; /* my col number */ int my_rnk; /* my rnk in the grid comm */ } GRID_INFO_T;

34 Setting up the grid - 1 Setting up grid informtion MPI_Comm_size(MPI_COMM_WORLD, &(grid->p)); MPI_Comm_rnk(MPI_COMM_WORLD, COMM &old_rnk); Assuming tht p is perfect squre we ssign grid->q = (int) sqrt((double) grid->p); dimensions[0] = dimensions[1] = grid->q; Setting up for verticl circulr shift wrp_round[0] = wrp_round[1] = 1; MPI_Crt_crete(MPI_COMM_WORLD, 2, dimensions, wrp_round, 1, &(grid->comm)); MPI_Comm_rnk(grid->comm, &(grid_rnk)); rnk)); MPI_Crt_coords(grid->comm, grid->my_rnk2, 2, coordintes); grid->my >my_row = coordintes[0]; grid->my_col = coordintes[1];

35 Setting up the grid - 2 Setting up the row communictors free_coords[0] = 0; free_coords[1] = 1; MPI_Comm_sub(grid->comm, free_coords, &(grid->row_comm)); Setting up the column communictors free_coords[0] = 1; free_coords[1] = 0; MPI_Comm_sub(grid->comm, free_coords, &(grid->col_comm));

36 A New MPI Commnd When the row of B re rolled upwrd (with wrp round), you will need to use the MPI commnd int MPI_Sendrecv_replce ( void* buffer /* in/out */, int count /* in */, MPI_Dttype dttype /* in */, int dest /* in */, int send_tg /* in */, int source /* in */, int recv_tg /* in */, MPI_Comm comm /* in */, MPI_Sttus* sttus /* out */ )

37 Set up for Fox s Algorithm - 1 Prmeters int n /* in */, GRID_INFO_T* grid /* in */, LOCAL_MATRIX_T* locl_a /* in */, LOCAL_MATRIX_T* locl_b /* in */, LOCAL_MATRIX_T* T* locl_c l /* out */ Locl Declrtions LOCAL_MATRIX_T* temp_a; int stge; int bcst_root; int n_br; int source; int dest; MPI_Sttus sttus;

38 Set up for Fox s Algorithm - 2 Initil Assignments n_br = n/grid->q; Set_to_zero(locl_C); source = (grid->my_row + 1) % grid->q; dest = (grid->my_row + grid->q 1) % grid->q; temp_a = locl_mtrix_llocte(n_br); Other tsks you need to write method Locl_mtrix_multiply tht multiplies sub-block A times sub-block B nd returns sub-block C s n out prmeter

39 Implementing the Min Loop For the grid->p stges do the following set bcst_root to (grid->my_row + stge) mod grid->q if the root equls my_col then brodcst locl_a cross the row clculte l locl_c l s locl_a l * locl_bl else brodcst temp_a cross the row clculte locl_c s temp_a * locl_b end if cll MPI_Sendrecv_replce to roll the rows of B upwrd

40 Timing Routines MPI_Wtick Returns the resolution of MPI_Wtime Synopsis #include "mpi.h" double MPI_Wtick() Return vlue Time in seconds of resolution of MPI_Wtime MPI_Wtime Returns n elpsed time on the clling processor Synopsis #include "mpi.h" double MPI_Wtime() Return vlue Time in seconds since n rbitrry time in the pst

41 Exmple of Running Fox s Algorithm Number of processes is perfect squre nd divides the size of the mtrix Two mtrices of ll 1 s re generted [blk@virtul fox]$ mpirun -n 4./fox Wht's the order of the mtrices? 8 Enter A We red A =

42 Enter B We red B = The product is time of computtion = [blk@virtul fox]$

43 Progrm: Complete Fox s Algorithm Your tsk is to get Fox s lgorithm fully functionl using MPI Define the method Fox Define the locl mtrix multiply Add timing routines to Fox s lgorithm so tht you cn collect performnce dt Collect dt for Fox s lgorithm compred with the trditionl sequentil lgorith Increse the mtrix size until you see some differences in performnce Include your results in redme file