Transparent neutral-element elimination in MPI reduction operations

Transcription

1 Trnsprent neutrl-element elimintion in MPI reduction opertions Jesper Lrsson Träff Deprtment of Scientific Computing University of Vienn

2 Disclimer Exploiting repetition nd sprsity in input for reducing (communiction) costs is nothing new in MPI librries [Filgueir et l., EuroPVM/MPI 2009] [HofmnnRünger, EuroPVM/MPI 2008] Exploiting lgebric (neutrl element) properties for compression nd computtion nturl for MPI reduction opertions: MPI_Reduce, MPI_Allreduce, MPI_Reduce_sctter, MPI_Scn,

3 Contributions: Neutrl element elimintion cn be incorported smoothly (for MPI implementer) nd trnsprently (to MPI user) in ANY MPI implementtion bsed on point-to-point communiction nd MPI_Reduce_locl with little performnce overhed nd potentilly lrge svings esily extended to messge compression for reduction opertions Missing functionlity in MPI stndrd? Strtegy: Develop bsic librry internl functionlity (3 functions!) Incorporte in two stndrd MPI_Reduce lgorithms Demonstrte smll overhed nd svings by compring to stndrd nd ntive MPI_Reduce implementtions Lrger theme: dt dependent optimiztions, hrd to model,

4 Neutrl elements in prllel reduction opertions Exmple: MPI_Reduce(,b,count,type,MPI_SUM,root,comm); count = b b b b Observtion/clim: sums vector contributions, one per process, result vector t root Reduction opertions (MPI_Allreduce, MPI_Reduce, MPI_Reduce_sctter, MPI_Scn) re mong the (most) used MPI collectives

5 Neutrl elements in prllel reduction opertions Exmple: MPI_Reduce(,b,count,type,MPI_SUM,root,comm); count = b b b b sums vector contributions, one per process, result vector t root Observtion/clim: Typicl pplictions reduce either very short or very long vectors

6 Sclr ddition + hs 0 s neutrl element: 0+ =, +0 = For vectors: = 2 0 count Algebric property: No need to perform ddition in rows with 0 No need to communicte 0 (explicitly) Only need to communicte nd use indices of non-0 es

7 Clim: Long vectors sometimes/often hve mny neutrl-elements (bnded vectors) - must be verified by ppliction study = Note: Vector summnds my hve different structure

8 Clim: Long vectors sometimes hve mny neutrl-elements (bnded vectors) - must be verified by ppliction study Note: + = = Intermedite neutrl elements my rise

9 Exmple: sprse exchnge (counting messges, lso for MPI_Win_fence) Ech process hs n individul messge to some (not ll) of the other processes but receivers do not know. Do this exchnge collectively i MPI_Send neighbors Question: how to set up the corresponding MPI_Recv clls?

10 Process i: [Hoefler et l.: Sclble protocols for dynmic sprse exchnge, 2010] for (j=0; j<p; j++) msgto[j] = 0; for (j in neighbors) msgto[j] = 1; MPI_Reduce_sctter_block(msgto,gets,1,MPI_SUM,comm); for (j=0; j<gets; j++) { MPI_Recv(dt,,MPI_ANY_SOURCE,); } For sprse exchnge (e.g. MPI_Win_fence), messge vector hs few non-zero s, mny, mny zero s Attrctive to do reduction on sy 10-element neighborhoods insted of p=100,000 element full vectors

11 MPI_Reduce(,b,count,dttype,op,root,comm); dttype MPI_INT MPI_SHORT MPI_LONG MPI_FLOAT MPI_DOUBLE op: MPI_MIN MPI_MAX MPI_SUM MPI_PROD MPI_BAND MPI_LAND User-defined x = neutrl: mxint, minint, 0, 0, 1, 1, Most predefined MPI op s hve neutrl element for most bsic dttypes Note: for MPI_LAND the neutrl element is nything nonzero

12 MPI_Reduce(,b,count,dttype,op,root,comm); dttype MPI_INT MPI_SHORT MPI_LONG MPI_FLOAT MPI_DOUBLE op: MPI_MIN MPI_MAX MPI_SUM MPI_PROD MPI_BAND MPI_LAND User-defined x = neutrl: mxint, minint, 0, 0, 1, 1, Most predefined MPI op s hve neutrl element for most bsic dttypes Note: no wy to know whether user-defined opertion hs neutrl element

13 Gol: Find esy, smooth, trnsprent wy not to perform locl, pirwise reduction (MPI_SUM) on neutrl elements send/receive neutrl elements for ll combintions of MPI opertion nd MPI dttype Note: not possible to do this with existing MPI (derived dttype) functionlity

14 Typicl reduction lgorithms 1. MPI_Recv j 2. MPI_Reduce_locl 0. MPI_Send i k Trees (pipelined) 2. MPI_Reduce_locl 1. MPI_Recv Liner pipeline i j k MPI_Send Butterfly pttern

15 Typicl reduction lgorithms with neutrl element elimintoin 1. SPS_Recv_comp j 2. SPS_Reduce_locl_sprse 0. SPS_Send_elim i k Trees (pipelined) 2. SPS_Reduce_locl_sprse 1. SPS_Recv_comp Liner pipeline i j k SPS_Send_elim Butterfly pttern

16 Typicl reduction lgorithms with neutrl element elimintion 1. SPS_Recv_comp j 2. SPS_Reduce_locl_sprse 0. SPS_Send_elim i k Trees (pipelined) 1. Decide on compressed formt 2. SPS_Send_elim sends messge with 0 s eliminted 3. SPS_Reduce_locl_sprse - reduction on received, sprse vector ginst uncompressed vector 4. SPS_Get_neutrl(op,type,*neutrl) lookup of neutrl element for op/type 5. No need for specil SPS_Recv_comp function (see next slides)

17 Typicl reduction lgorithms with neutrl element elimintion 1. SPS_Recv_comp (tempbuf,) j 2 SPS_Reduce_locl_sprse (tempbuf,recvbuf,) 0. SPS_Send_elim i k Trees (pipelined) Smooth, trnsprent mens: further chnges should not be necessry, sme intermedite buffers for MPI_Recv nd MPI_Reduce_locl cn be used

18 Typicl reduction lgorithms with neutrl element elimintion 1. SPS_Recv_comp j 2. SPS_Reduce_locl_sprse 0. SPS_Send_elim i k Trees (pipelined) Note: With this ide, the sme number of messges re sent s in the stndrd, uncompressed lgorithm. Mximum improvement is therefore bound by time to send full (uncompressed) messge-block divided by time to send (fully compressed) messge of size 0

19 1. Compressed formt Vector of indices of non-zeroes followed by vector of vlues y1 yi 0 0 ix vl [0,7,11,1023,],[y1,yi,] (list of index-vlue pirs would give lignment problems for eg. CHAR dttypes) Note: only mkes sense if more thn hlf the elements re 0 es Distinguish uncompressed/compressed messge by new tgs: DENSETAG SPARSETAG - uncompressed vector index vector followed by vlues

20 2. SPS_Send_elim(sendbuf,count,type,dest,tg,neutrl,comm); MPI_Type_size(type,&typesize); spce = 0; j = 0; for (i=0; i<count; i++) { if (sendbuf[i]!=*neutrl) { spce += sizeof(int)+typesize; index[j++] = i; } if (spce>=cutoff) brek; } if (spce>=cutoff) { // no compression possible, send norml MPI_Send(sendbuf,totl,MPI_BYTE,dest,DENSETAG,comm); } else { vlue = (type*)(index+j); // vlues fter indices for (i=0; i<j; i++) vlue[i] = sendbuf[index[i]]; MPI_Send(index,spce,MPI_BYTE,dest,SPARSETAG,comm); } Property: never sends more thn MPI_Send would hve done

21 2. SPS_Send_elim(sendbuf,count,type,dest,tg,neutrl,comm); MPI_Type_size(type,&typesize); spce = 0; j = 0; for (i=0; i<count; i++) { if (sendbuf[i]!=*neutrl) { spce += sizeof(int)+typesize; index[j++] = i; } if (spce>=cutoff) brek; } if (spce>=cutoff) { // no compression possible, send norml MPI_Send(sendbuf,totl,MPI_BYTE,dest,DENSETAG,comm); } else { vlue = (type*)(index+j); // vlues fter indices for (i=0; i<j; i++) vlue[i] = sendbuf[index[i]]; MPI_Send(index,spce,MPI_BYTE,dest,SPARSETAG,comm); } Drwbck: sendbuf is ccessed twice (finding indices, copying non-neutrl elements)

22 2. SPS_Send_elim(sendbuf,count,type,dest,tg,neutrl,comm); MPI_Type_size(type,&typesize); spce = 0; j = 0; for (i=0; i<count; i++) { if (sendbuf[i]!=*neutrl) { spce += sizeof(int)+typesize; index[j++] = i; } if (spce>=cutoff) brek; } if (spce>=cutoff) { // no compression possible, send norml MPI_Send(sendbuf,totl,MPI_BYTE,dest,DENSETAG,comm); } else { vlue = (type*)(index+j); // vlues fter indices for (i=0; i<j; i++) vlue[i] = sendbuf[index[i]]; MPI_Send(index,spce,MPI_BYTE,dest,SPARSETAG,comm); } Remrk: slight misuse of MPI_BYTE

23 Note: extr buffer (llocted in SPS_Send_elim) is needed, in-plce implementtion rrely mkes sense MPI_Reduce(sendbuf,) { MPI_Isend(sendbuf, ); } sendbuf must remin unchnged fter return from (blocking) MPI_Reduce cll (from non-blocking MPI_Ireduce cll, simultneous usge of sendbuf llowed in ddition)

24 3. SPS_Reduce_locl_sprse(index,in,inout,count,type,MPI_SUM); for (i=0; i<count; i++) inout[index[i]] = in[i]+inout[index[i]]; Locl reduction determined by tg, no need for SPS_Recv_comp: MPI_Recv(tempbuf,blocksize,MPI_BYTE,child,MPI_ANY_TAG, comm,&sttus); if (sttus.mpi_tag==densetag) { MPI_Reduce_locl(tempbuf,recvbuf,count,type,op); } else if (sttus.mpi_tag==sparsetag) { MPI_Get_count(&sttus,MPI_BYTE,&sprsecount); sprsecount = sprsecount/(sizeof(int)+typesize); index = (int*)tempbuf; vlue = (void*)(index+sprsecount); MPI_Reduce_locl_index(index,vlue,recvbuf,sprsecount, type,op); } else { // impossible for dt sent with SPS_Send_elim }

25 4. SPS_Get_neutrl(op,type,*neutrl) Return pointer to neutrl element for op/type combintion Fct: neutrl element defined nd known for MPI built-in opertors Missing in MPI 2.2 stndrd: - or extend MPI_Op_crete MPI_Op_set_neutrl(op,type,neutrl): IN op MPI opertion (user-defined) IN type dttype for op, bsic or derived IN neutrl pointer to neutrl element for op on type wy to inform MPI librry bout possible neutrl element for userdefined opertion (+ query function)

26 Performnce some experiments Implemented MPI_Reduce with Binomil tree reduction for short vectors Liner pipeline for long vectors (switch point round 8K bytes) Experiments on new, 64-core, 4x16 Intel Nehlem system with InfiniBnd t University of Vienn Mesure: 1. Gin s function of density of neutrl elements 2. Overhed of elimintion 3. Gin for different dttypes 4. Gin with different intermedite results (more/less neutrl elements)

27 Input: 1. Strided vectors: offset, (block of non-zeroes, stride of zeroes)* 2. Ops: MPI_SUM, MPI_MAX, MPI_MIN, MPI_BOR, MPI_PROD, 3. Types: MPI_INT, MPI_SHORT, MPI_SIGNED_CHAR,MPI_FLOAT, MPI_DOUBLE, By setting offset dependent on process rnk, vectors with different structure nd overlps cn be generted, s well s cses where the sprsity chnges Note: no clim tht implemented lgorithms re best possible; therefore comprison to ntive lso

28 Fctor 3 improvement, close to mximum possible

29 Reminder: Since the sme number of dt blocks re sent nd loclly reduced s in originl, non-elimintion lgorithm, the mximum improvement is the time to send full (non-compressed) block divided by the time to send n empty (fully compressed) block For the pipelined lgorithms, B=8K, nd this rtio is between 3 nd 4 on the Nehlem/Infinibnd system!

30

31 Virtully no overhed for lrge vectors Ntive MPI_Reduce no good

32 Some overhed for short vectors

33 More thn fctor 3

34 Smller improvement, sizeof(chr) < sizeof(int)

35

36 1. Better(?) compressed formt Not in pper Avoid extr scn of sendbuf, exploit repetitions (RLE) Block of sme non-neutrl elements ix single ix-1<0 ix vl ix vl ix ix vl strt ix>=0 end ix>0 strt ix>=0 end ix<0 Block of different non-neutrl elements

37 do { while (i<count&&(n=sender[i])==e) i++; ii = i; /* skip neutrl */ do i++; while (i<count&&sender[i]==n); /* look for repetitions */ if (i-ii>=2) { index[0] = ii; index[1] = -i; buffer = index+2; index = buffer+1; buffer[0] = n; spce += sizeof(int)+sizeof(int)+sizeof(int); } else { /* not enough sme element repetitions */ if (sender[i]==e) { /* single non-neutrl element */ index[0] = -ii; buffer = index+2; index = buffer+1; i++; buffer[0] = n; spce += sizeof(int)+sizeof(int); } else { index[0] = ii; buffer = index+2; spce += (i-ii)*sizeof(int); for (; ii<i; ii++,buffer++) buffer[0] = n; /* repet element */ while (i<count&&sender[i]!=e) { (buffer++)[0] = sender[i++]; } index[1] = i; index = buffer; spce += (i-ii)*sizeof(int)+sizeof(int)+sizeof(int); } } } while (i<_count&&spce<cutoff);

38 Now: Single scn through sendbuf suffices, cutoff when compressed buffer becomes too lrge (t most size of originl sendbuf) SPS_Reduce_locl_sprse need to be chnged ccordingly, not difficult, nd potentilly fster Experiments: pending Advntge of originl formt (s described in pper): esy to implement, vectorizes, double scn probbly does not mtter for smll blocks in cche (pipelined lgorithms)

39 Summry, remrks 1. Simple, trnsprent technique for neutrl element elimintion for MPI reduction opertions 2. Close to mximum possible improvement cn be chieved 3. Very little overhed with creful tuning, recommended for stndrd use 4. Also crried out s student project t the University of Vienn thus implementtion effort is limited 5. Code cn be mde vilble Dt dependent optimiztions in MPI librries? -implementble with smll overhed -or tke hints from ppliction on specil structure -Mkes performnce modeling even more difficult!!