An Open64-based Compiler and Runtime Implementation for Coarray Fortran

Transcription

1 An Open64-based Compiler and Runtime Implementation for Coarray Fortran talk by Deepak Eachempati Presented at: Open64 Developer Forum /25/2010 1

2 Outline Motivation Implementation Overview Evaluation Future Work 2

3 Motivation Goals Provide an open, portable implementation of Coarray Fortran (CAF) Explore potential optimizations in compiler and runtime for reducing parallelization costs Prototype new language features Encourage support for CAF - adding to Open64 encourages other vendors to pick it up 3

4 Motivation What is Coarray Fortran (CAF)? Simple language extension to Fortran for explicit parallel programming Principle: smallest change required to make Fortran an effective parallel language Features SPMD model similar to MPI Fixed number of asynchronously executing copies called images coarray abstraction allows indexing into other images using cosubscripts for remote data access new intrinsics and synchronization statements added to language 4

5 Motivation Example: Matrix Multiply real, allocatable :: a(:,:)[:,:], b(:,:)[:,:], c(:,:)[:,:] allocate(a(n,n)[p,*], b(n,n)[p,*], c(n,n)[p,*]) myp = this_image(a,1) myq = this_image(a,2) * a = 1.0 b = 1.0 c = sync all do i=1,n do j=1,n do l=1,p c(i,j) = c(i,j) + sum(a(i,:)[myp,l]*b(:,j)[l,myq]) end do end do end do 3 p allocatable coarrays get cosubscripts for this image based on codimensions a(:,:)[3,4] myp = 3 myq = 4 remote access to array sections; communication costs may be reduced by optimizing compiler 5

6 Implementation UH CAF Implementation Basic translation strategy associate two dope vectors with each declared coarray replace coarray references (having * + cosubscripts) with GET or PUT runtime call identify and translate intrinsic functions What about optimizations? Open64 is a robust optimizing compiler we defer translation until after front-end so that we can explore compiler optimizations based on CAF syntax 6

7 Implementation Comparison to other CAF compilers UH CAF currently a baseline implementation (no optimization) portable, open-source runtime support based current implementations using ARMCI or GASNet G95 can generate optimized program for multi-core can support images run on cluster, using G95 Coarray Console (closed source) gfortran CAF implementation is in progress focus is on shared memory (single node) multi-image cluster execution is not yet supported 7

8 Implementation OpenUH-CAF Compiler CAF Source Code CRAYF90 Fortran Frontend With CAF Support VH-WHIRL Emitter OpenUH Communication Library (ARMCI, GASNet) coarray analysis phase coarray lowering OpenUH Middle-end and Backend VHO - LNO WOPT CG 8

9 Implementation UH CAF Coarray Lowering Coarray Lowering Responsibilities replacement of coarray variables with low-level data structures for describing shape and location of data modify program to create coarray data and track it throughout lifetime of program replacement of coarray references to remote data with 1-sided communication leverage analyses from compiler to optimize: synchronizations communication buffering 9

10 Implementation Data structure for describing coarrays Dope Vector Codope Vector Introduced 10

11 Coarray Lowering Communication Generation (Example) Implementation A(i, j, 1:n)[q] = B(1, j, 1:n)[p] + C(1, j, 1:n)[p] + D[p] A(i, j, 1:n)[q] t4 = t1 B(1, j, 1:n)[p] + t2 C(1, j, 1:n)[p] + t3 D[p]] 11

12 t1 B(1, i:j)[p]! allocate t1 t1%base = 0 t1%flds%el_len = b_dope%flds%el_len t1%flds%assoc = 0 t1%flds%type_code = IOR(ISHFT(2_8,32), ISHFT(32_8,44)) t1%flds%num_dims = ISHFT(2,29) t1%dims(1)%lb = 1 t1%dims(1)%ext = 1 t1%dims(1)%str_m = 1 t1%dims(2)%lb = 1 t1%dims(2)%ext = j-i+1 t1%dims(2)%str_m = 1 call caf_alloc_comm_buf(t1)! set up b_dope and b_codope for transfer xfer_dope = b_dope xfer_dope%base = b_dope%base + ISHFT(b_dope%flds%el_len,-3)*& ( (1-b_dope%dims(1)%lb)*b_dope%dims(1)%str_m + & (j-b_dope%dims(2)%lb)*b_dope%dims(2)%str_m ) xfer_codope = b_codope xfer_codope%base_img = b_codope%base_img + & (p-b_codope%codims(1)%lb)*b_codope%codims(1)%str_m call caf_rma_get(t1, xfer_dope, xfer_codope) B(3:5,i:j)[p] t2 =! allocate t2 t2%base = 0 t2%flds%el_len = b_dope%flds%el_len t2%flds%assoc = 0 t2%flds%type_code = IOR(ISHFT(2_8,32), ISHFT(32_8,44)) t2%flds%num_dims = ISHFT(2,29) t2%dims(1)%lb = 1 t2%dims(1)%ext = 3 t2%dims(1)%str_m = 1 t2%dims(2)%lb = 1 t2%dims(2)%ext = j-i+1 t2%dims(2)%str_m = 3 call caf_alloc_comm_buf(t2) t2 = Implementation! set up b_dope and b_codope for transfer xfer_dope = b_dope xfer_dope%base = b_dope%base + ISHFT(b_dope%flds%el_len,-3)*& ( (3-b_dope%dims(1)%lb)*b_dope%dims(1)%str_m + & (i-b_dope%dims(2)%lb)*b_dope%dims(2)%str_m ) xfer_codope = b_codope xfer_codope%base_img = b_codope%base_img + & (p-b_codope%codims(1)%lb)*b_codope%codims(1)%str_m call caf_rma_put(t2, xfer_dope, xfer_codope) 12

13 Implementation Opportunities for Optimization reduce costs due to allocation of temporary buffers optimize buffer sizes static analysis to generate non-blocking communication as appropriate static analysis to reduce synchronization costs adapt WOPT and CG optimization phases for coarrays 13

14 Implementation UH CAF Runtime Support A communication runtime library, provides: process management manage coarray data 1-sided communication synchronizations collectives communication buffer allocation/deallocation ARMCI or GASnet for 1-sided communication Assumes CRAY dopevector format launch images with mpirun 14

15 Implementation Managing Coarray Data Coarrays may be non-allocable or allocatable. Remote Memory Descriptor Defines a memory region allocated collectively on each executing process (addr i is the base address on image i) size bytes on all images Coarrays are created collectively within these memory regions at next_offset rm_desc 0 rm_desc 1 rm_desc n addr 1 addr 2 addr 3 addr 4... size next offset 15

16 Evaluation Parallelizing using CAF real :: u(xmin-lx:xmax+lx, zmin-lz:zmin+lz) real :: v(xmin-lx:xmax+lx, zmin-lz:zmin+lz) & & do it=1,nt, 2 print *, maxval(u), minval(u) u(xsource,zsource) += & source(it) call cg_fwd_2d( u, v, ) v(xsource,zsource += & source(it+1) call cg_fwd_2d( v, u, ) end do real :: u(xmin-lx:xmax+lx, & zmin-lz:zmin+lz)[npx,*] real :: v(xmin-lx:xmax+lx, & zmin-lz:zmin+lz)[npx,*] do it=1,nt, 2 call co_maxval(maxval(u), u_max) call co_minval( minval(u), u_min) if (this_image()==1) print *, u_max, u_min if (is_center_image()) & u(xsource,zsource) += source(it) call cg_fwd_2d( u, v, )! get bottom rows from top neighbor if (px>1) u(xmin-lx:xmax-1,:) = & u(xmax-lx+1:xmax,:)[px-1,pz] if (is_center_image()) & v(xsource,zsource += source(it+1)! get bottom rows from top neighbor if (px>1) v(xmin-lx:xmax-1,:) = & v(xmax-lx+1:xmax,:)[px-1,pz] call cg_fwd_2d( v, u, ) end do 16

17 Evaluation Results on IB cluster 17

18 Future Work On-going and future work Optimize implementation for execution on SMP node generate non-blocking communication and wait primitives for reduce overheads explore language extensions tune buffering: consider user-, compiler-, and runtime-driven strategies. 18

19 Current Status Supported Features coarrays of basic data types allocatable coarrays intrinsics: this_image, num_images, image_index, lcobound, ucobound synchronizations sync all, sync images, notify/query, critical sections collectives comax, comin, cosum enable performance tracing in runtime For more information: 19

20 Thank you. Questions? 20