Some more aspects on Automatic Differentiation of MPI programs. Ralf Giering

Transcription

1 Some more aspects on Automatic Differentiation of MPI programs Ralf Giering

2 Outline AD of Message Passing Interface (MPI) overview Non linear reductions Non blocking communication Concept of Derivative Infrastructure Variables Conclusions

3 Previously on AD of MPI (extension from Utke, not complete) Hovland (1997): thesis AD of parallel programs - mostly forward mode Hovland/Bischof (1998): Automatic Differentiation for Message-Passing Parallel Programs association value-derivative Carle/Fagan (2002): Automatically Differentiating MPI-1 Datatypes: The Complete Story Faure/Dutto (1999): Extension of Odyss ee to the MPI library (direct mode, reverse mode) plain send/recv Cheng (2006): A Duality between Forward and Adjoint MPI Communication Routines Carle (2007): in ch. 24 of Sourcebook of Parallel Computing - 4 pgs on analysis Strout/Hovland/Kreaseck (2006): Data flow analysis for MPI programs Heimbach/Hill/Giering (2002): Automatic generation of efficient adjoint code for a parallel Navier- Stokes Solver - hand-written communication adjoints in MITgcm wrapper Griewank: first ed. of book had 2 pages on parallel programs; second edition has more Utke/Hascoët/Heimbach/Hill/Hovland/Naumann (2009): Toward Adjoinable MPI Schanen/Naumann/Hascoët/Utke (2010): Interpretative adjoints for numerical simulation codes using MPI Utke et.al.(2013): Designing and implementing a tool-independent, adjoinable wrapper library AMPI

4 AD of MPI strategies association between value and derivative same communications but on new type (only forward mode) using wrapper routines as black boxes (eg. MITgcm+TAF) hand written routines + flow directives adjoinable MPI library (AMPI) (ADOL-C, Tapenade, dcc, CompAD) use extended argument list subset of MPI direct handling of MPI library calls (TAF) start with a subset of the MPI library send/recv and collective communications might need directives to support tool

5 AD of MPI basics MPI send/receive pairs identification needed for activity anaysis based on tag and communicator (and data type) might be not decidable at compile time ==> conservative assumptions ==> AD directives to assist AD-tool there might be a send with multiple receive statements and vice versa forward and reverse mode communications must be distinguishable from original communications new tag (e.g. tag+999, Tapenade, TAF) new 'derivative' communicator (TAF, experimental)

6 Linear reduction (MPI_Sum) tangent code: call MPI_reduce( sbuf,rbuf,cnt,datat,mpi_sum,root,comm,ierr ) call MPI_reduce( sbuf_tl,rbuf_tl,cnt,datat,mpi_sum,root,comm,ierr ) wrong adjoint code: allocate( mpihi(cnt) ) mpihi = rbuf_ad rbuf_ad = 0. call mpi_bcast( mpihi,cnt,datat,root,comm,ierr ) sbuf_ad = sbuf_ad+mpihi

7 Linear reduction (MPI_Sum) tangent code: call MPI_reduce( sbuf,rbuf,cnt,datat,mpi_sum,root,comm,ierr ) call MPI_reduce( sbuf_tl,rbuf_tl,cnt,datat,mpi_sum,root,comm,ierr ) wrong adjoint code: correct adjoint code?: reset of rbuf_ad: only on root only received part allocate( mpihi(cnt) ) mpihi = rbuf_ad rbuf_ad = 0. call mpi_bcast( mpihi,cnt,datat,root,comm,ierr ) sbuf_ad = sbuf_ad+mpihi call MPI_Comm_rank( comm,rank,ierr ) allocate( mpihi(cnt) ) if (rank.eq. root) then mpihi(:) = rbuf_ad(1:cnt) rbuf_ad(1:cnt) = 0. else mpihi(:) = 0. endif call mpi_bcast( mpihi,cnt,datat,root,comm,ierr ) sbuf_ad(1:cnt) = sbuf_ad(1:cnt) + mpihi(:)

8 AD of MPI pitfalls count value lower than array size FORTRAN-77 actual arguments: call MPI_Xxxx( sbuf(1,j,k) ) MPI_Proc_null src/dest = MPI_Proc_null means no communication MPI_Any_source, MPI_Any_tag no simple send-receive pairs

9 AD of MPI: the easy part (almost) only MPI-1 most collective communications simple blocking send/recv forward mode AD

10 AD of MPI: the difficult part non linear reductions (MPI_Max, MPI_Min, MPI_Prod) non blocking communications vector mode AD

11 AD of MPI: the ugly part testing of non blocking communications (MPI_Testsome) user defined data types (MPI_Type_struct) user defined reduction operators (MPI_Op_create) MPI_Barrier MPI-2 standard MPI-3, MPI-3.1

12 Non linear reduction (MPI_Min, MPI_Max) call MPI_Reduce( sbuf,rbuf,cnt,mpi_real,mpi_max,root,comm,ierr ) tangent code: Get #procs and rank call mpi_comm_size( comm,nproc,ierr ) call mpi_comm_rank( comm,nrank,ierr ) determine locations (ranks) of maxima allocate( mpi1h(2,1:cnt), mpi2h(2,1:cnt) ) mpi1h(1,:) = sbuf mpi1h(2,:) = nrank call mpi_reduce( mpi1h,mpi2h,cnt,mpi_2real,mpi_maxloc,root,comm,ierr ) gather sbuf_tl from all processors allocate( mpi3h(1:cnt,1:nproc) ) call mpi_gather( sbuf_tl,cnt,mpi_real,mpi3h,cnt,mpi_real,root,comm,ierr ) if (nrank.eq. root) then rbuf = mpi2h(1,:) propagate tangent of maxima do im = 1, cnt rbuf_tl(im-1+lbound(rbuf_tl,1)) = mpi3h(im,int(1+mpi2h(2,im))) end do endif deallocate( mpi1h,mpi2h,mpi3h )

13 AD of MPI: non blocking communication MPI_Send_init( buf,cnt,datatype,dest,tag,comm,request,ierr ) MPI_Start( request,ierr ) MPI_Wait( request,status,ierr ) MPI_Request_free( request,ierr ) MPI_Isend( buf,cnt,datatype,dest,tag,comm,request ) MPI_Wait( request,status,ierr ) MPI_Isend ==> MPI_Send_init MPI_Start identify groups of calls establishing one send (receive) bundle generate a 'derivative' request object request_tl / request_ad

14 adjoint of non blocking communication original MPI_Recv_init(buf,request ) MPI_Start( request ) MPI_Wait( request ) MPI_Request_free( request ) transformed to ajoint MPI_Request_free( request_ad ) MPI_Wait( request_ad ) MPI_Start( request_ad ) MPI_Send_init(buf_ad,request_ad ) looks simple but... the MPI_*_init statement has many more arguments mainly buf_ad these arguments are not available if adjoining..._free() some arguments might be computed (src,dest,tag) the MPI calls might be in different scopes

15 adjoint of non blocking communication original MPI_Recv_init(buf,request ) MPI_Start( request ) MPI_Wait( request ) MPI_Request_free( request ) transformed to ajoint MPI_Request_free( request_ad ) MPI_Wait( request_ad ) MPI_Start( request_ad ) sliced for request variable and transformed to MPI_Recv_init(buf,request ) MPI_Send_init( buf_ad, request_ad )

16 adjoint of non blocking communication forward data flow analysis to identify send (receive) groups generation of derivative infrastructure variable for request-object standard Efficient Recomputation Algorithm ERA slicing of code for DIV definition (MPI_*_init) replace MPI_Send_init by MPI_Recv_init and vice versa replace request-object by its DIV replace all active variables by their derivative variable continue slicing to make sure all input arguments of MPI_*_init are defined

17 Adjoint of non blocking communication if (myid.eq. 0) then fc = mypi do i = 1, numprocs-1 original code call MPI_Irecv( res, 1, myreal, i, tag, MPI_COMM_WORLD, iwait, ierr ) call MPI_Wait ( iwait, status, ierr) fc = fc + res enddo else call MPI_Send( mypi, 1, myreal, 0, tag, MPI_COMM_WORLD, ierr ) endif if (myid.eq. 0) then do i = numprocs-1, 1, -1 call mpi_send_init( res_ad,1,myreal,i,tag+999,mpi_comm_world,iwait_ad,ierr ) res_ad = res_ad+fc_ad call mpi_start( iwait_ad,ierr ) call mpi_wait( iwait_ad,statuu,ierr ) call mpi_request_free( iwait_ad,ierr ) res_ad = 0. TAF adjoint code end do mypi_ad = mypi_ad+fc_ad fc_ad = 0. else mpihh = 0. call mpi_recv( mpihh,1,myreal,0,tag+999,mpi_comm_world,statut,ierr ) mypi_ad = mypi_ad+mpihh endif

18 Derivative Infrastructure Variables concept of DIV's can be generalised to handle other constructs pointer assignments DIV = pointer open/close/read/write of active files DIV = unit number MPI communicators DIV = request, communicator properties of DIV's that differs from other derivatives: do not carry derivative information do not need an additional dimension in vector mode usually have a non numeric type i.e. integer, pointer are not relevant in the activity analysis requires additional data flow analysis

19 Points-to-analysis (PTA) Original code, x,y and z are active, independent x, dependent y : real, target :: z real, pointer :: zp zp => z z = x y = zp zp is required z is only detected required by PTA adjoint code: real, target :: z_ad real, pointer :: zp_ad z_ad = 0. zp_ad => z_ad zp_ad = zp_ad + y_ad y_ad = 0. x_ad = x_ad + z_ad z_ad = 0.! adjoint pointer assignment

20 Conclusions Many pitfall exits even for plain MPI send/recv Challenge of AD of MPI-1 is currently non blocking communication AD of non blocking communication can be handled directly with the concept of Derivative Infrastructure Variables requires additonal data flow analysis slicing with transformation for DIVs TAF handles MPI calls directly (option -mpi), currently support for collective communications (forward,reverse) (not MPI_Prod) plain send/recv (forward,reverse) non blocking communication (forward, partly reverse) work on pittfalls still in progress