First Experiences with Application Development with Fortran Damian Rouson

Transcription

1 First Experiences with Application Development with Fortran 2018 Damian Rouson

2 Overview Fortran 2018 in a Nutshell ICAR & Coarray ICAR WRF-Hydro Results Conclusions

3 Overview Fortran 2018 in a Nutshell Motivation: exascale challenges Background: SPMD & PGAS in Fortran 2018 Beyond CAF WRF-Hydro ICAR & Coarray ICAR Results Conclusions

4 EXASCALE CHALLENGES & Fortran 2018 Response Billion-way concurrency with high levels of on-chip parallelism events collective subroutines richer set of atomic subroutines teams Source: Ang, J.A., Barrett, R.F., Benner, R.E., Burke, D., Chan, C., Cook, J., Donofrio, D., Hammond, S.D., Hemmert, K.S., Kelly, S.M. and Le, H., 2014, November. Abstract machine models and proxy architectures for exascale computing. In Hardware-Software Slide / 01 Co-Design for High Performance Computing (Co-HPC), 2014 (pp ). IEEE. Higher failure rates failed-image detection Expensive data movement one-sided communication teams (locality control) Heterogeneous hardware on processor events

5 Single Program Multiple Data Images (SPMD) Image 1 Image 2 Image N Far far away, behind the word mountains, far from the countries Vokalia and Consonantia, there live the blind texts. Separated they live in Bookmarksgrove. Sign

6 Single Program Multiple Data Images (SPMD) Image 1 Image 2 Image N Far far away, behind the word mountains, far from the countries Vokalia and Consonantia, there live the blind texts. Separated they live in Bookmarksgrove. Sign

7 Single Program Multiple Data Images (SPMD) {processes threads} Image 1 Image 2 Image N... Far far away, behind the word mountains, far from the countries Vokalia and Consonantia, there live the blind texts. Separated they live in Bookmarksgrove. Sign

8 Single Program Multiple Data Images (SPMD) {processes threads} Image 1 Image 2 Image N... Far far away, behind the word mountains, far from the countries Vokalia and Consonantia, there live the blind texts. Separated they live in Bookmarksgrove. Sign

9 Single Program Multiple Data Images (SPMD) {processes threads} Image 1 Image 2 Image N... Far far away, behind the word mountains, far from the countries Vokalia and Consonantia, there live the blind texts. Separated they live in Bookmarksgrove. Sign sync all

10 Single Program Multiple Data Images (SPMD) {processes threads} Image 1 Image 2 Image N... Far far away, behind the word mountains, far from the countries Vokalia and Consonantia, there live the blind texts. Separated they live in Bookmarksgrove. sync all Sign sync all

11 Single Program Multiple Data Far far away, behind the word mountains, far from the countries Vokalia and Consonantia, there live the blind texts. Separated they live in Bookmarksgrove. sync all Images (SPMD) {processes threads} Image 1 Image 2 Image N Sign sync all... sync all Images execute asynchronously up to a programmer-specified synchronization: sync all sync images allocate/deallocate

12 Partitioned Global Address Space (PGAS)

13 Partitioned Global Address Space (PGAS) Coarrays integrate with other languages features: Fortran 90 array syntax works on local data. Communicate objects

14 Partitioned Global Address Space (PGAS) Coarrays integrate with other languages features: Fortran 90 array syntax works on local data. Communicate objects The ability to drop the square brackets harbors two important implications: Easily determine where communication occurs. Parallelize legacy code with minimal revisions.

15 if (me<n) a(1:2) = a(2:3)[me+1] Image 1 Image 2 Image 3 Sign

16 if (me<n) a(1:2) = a(2:3)[me+1] Image 1 Image 2 Image 3 Sign

17 if (me<n) a(1:2) = a(2:3)[me+1] Image 1 Image 2 Image 3 Sign

18 if (me<n) a(1:2) = a(2:3)[me+1] Image 1 Image 2 Image 3 Sign

19 if (me<n) a(1:2) = a(2:3)[me+1] Image 1 Image 2 Image 3 Sign

20 if (me==1) a(5)[2] = a(5)[3] Image 1 Image 2 Image 3 Far far away, behind the word mountains, far from the countries Vokalia and Consonantia, there live the blind texts. Separated they live in Bookmarksgrove. Sign

21 if (me==1) a(5)[2] = a(5)[3] Image 1 Image 2 Image 3 Far far away, behind the word mountains, far from the countries Vokalia and Consonantia, there live the blind texts. Separated they live in Bookmarksgrove. Sign

22 Segment Ordering: An intrinsic module provides the derived type event_type, which encapsulates an atomic_int_kind integer component default-initialized to zero. Events An image increments the event count on a remote image by executing event post. The remote image obtains the post count by executing event_query. Image Control Side Effect event post x atomic_add 1 event_query defines count event wait x atomic_add -1

23 vents ello, world!. greeting_ready(2:n)[1] ok_to_overwrite[3]

24 vents ello, world!. greeting_ready(2:n)[1] ok_to_overwrite[3]

25 vents ello, world! query. greeting_ready(2:n)[1] ok_to_overwrite[3]

26 vents ello, world! query. greeting_ready(2:n)[1] ok_to_overwrite[3]

27 vents ello, world! query post. greeting_ready(2:n)[1] ok_to_overwrite[3]

28 vents ello, world! query post. greeting_ready(2:n)[1] ok_to_overwrite[3]

29 vents ello, world! query post. wait greeting_ready(2:n)[1] ok_to_overwrite[3]

30 vents ello, world! query post. wait greeting_ready(2:n)[1] ok_to_overwrite[3]

31 vents ello, world! query post. post wait greeting_ready(2:n)[1] ok_to_overwrite[3]

32 vents ello, world! query post. post wait greeting_ready(2:n)[1] ok_to_overwrite[3] Performance-oriented constraints: Query and wait must be local. Post and wait are disallowed in do concurrent constructs.

33 vents ello, world! query post. post wait greeting_ready(2:n)[1] ok_to_overwrite[3] Performance-oriented constraints: Query and wait must be local. Post and wait are disallowed in do concurrent constructs. Pro tips: Overlap communication and computation. Wherever safety permits, query without waiting. Write a spin-query-work loop & build a logical mask describing the remaining work.

34 TEAM A set of images that can readily execute independently of other images Team 2 Team 1 Team 2 Image 1 Image 2 Image 3 Image 4 Image 5 Image 6 a(1:4)[1] a(1:4)[2] a(1:4)[2] a(1:4)[4] a(1:4)[5] a(1:4)[6] a(1:4)[4] a(1:4)[3]

35 Collective Subroutines Each non-failed image of the current team must invoke the collective. After parallel calculation/communication, the result is placed on one or all images. Optional arguments: stat, errmsg, result_image, source_image All collectives have intent(inout) argument A holding the input data and may hold the result on return, depending on result_image No implicit synchronization at beginning/end, which allows for overlap with other actions. No image s data is accessed before that image invokes the collective subroutine.

36 Extensible Set of Collective Subroutines

37 Performance User-Defined vs Intrinsic Collectives NERSC Hopper: Xeon nodes on Cray XE6 KNL nodes on a Cray XC Execution Time (sec) co_sum_int32 sum_bin_tree_int32 sum_bin_tree_real64 sum_rec_doubling_int32 sum_rec_doubling_real64 sum_alpha_tree_real e Num. of images

38 Failure Detection Image 1 Image 2 Image 3 Image 4 Image 5 Image n a(1:4)[1] a(1:4)[2] a(1:4)[3] a(1:4)[4] a(1:4)[5] a(1:4)[n] use iso_fortran_env, only : STAT_FAILED_IMAGE integer :: status sync all(stat==status) if (status==stat_failed_image) call fault_tolerant_algorithm()

39 Failure Detection Image 1 Image 2 Image 3 Image 4 Image 5 Image n a(1:4)[1] a(1:4)[2] a(1:4)[3] a(1:4)[4] a(1:4)[5] a(1:4)[n] use iso_fortran_env, only : STAT_FAILED_IMAGE integer :: status sync all(stat==status) if (status==stat_failed_image) call fault_tolerant_algorithm()

40 Failure Detection Image 1 Image 2 Image 3 Image 4 Image 5 Image n a(1:4)[1] a(1:4)[2] a(1:4)[3] a(1:4)[4] a(1:4)[5] a(1:4)[n]

41 FORTRAN 2018 Failed-Image Detection FAIL IMAGE (simulates a failures) IMAGE_STATUS (checks the status of a specific image) FAILED_IMAGES (provides the list of failed images) Coarray operations may fail. STAT= attribute used to check correct behavior.

42 Overview Fortran 2018 in a Nutshell ICAR & Coarray ICAR Results Conclusions

43 The Climate Downscaling Problem Climate model (top row) and application needs (bottom row) Topography Computational Cost High-res regional model >10 billion core hours (CONUS 4km 150yrs 40 scenarios) Precipitation

44 The Intermediate Complexity Atmospheric Research Model (ICAR) Analytical solution for flow over topography (right) 90% of the information for 1% of the computational cost ICAR Wind Field over Topography Core Numerics: 90% of cost = Cloud physics grid-cells independent 5% of cost = Advection fully explicit, requires local neighbor communication Gutmann et al (2016) J. of Hydrometeorology, 17 p957. doi: /jhm-d

45 ICAR Intermediate Complexity Atmospheric Research Animation of Water vapor (blue) and Precipitation (Green to Red) over the contiguous United States Output timestep : 1hr Integration timestep : Variable (~30s) Boundary Conditions: ERA-interim (historical) Run on 16 cores with OpenMP Limited scalability

46 ICAR Intermediate Complexity Atmospheric Research Animation of Water vapor (blue) and Precipitation (Green to Red) over the contiguous United States Output timestep : 1hr Integration timestep : Variable (~30s) Boundary Conditions: ERA-interim (historical) Run on 16 cores with OpenMP Limited scalability

47 ICAR comparison to Full-physics atmospheric model (WRF) Ideal Hill case ICAR (red) and WRF (blue) precipitation over an idealized hill (green) Real Simulation WRF (left) and ICAR (right) Season total precipitation over Colorado Rockies

48 ICAR Users and Applications

49 Coarray ICAR Mini-App Object-oriented design Collective broadcast of initial data Overlaps communication & computation via one-sided puts. Coarray halo exchanges with 3D, 1st-order upwind advection (~2000 lines of new code) Cloud microphysics (~5000 lines of preexisting code)

50 Overview Fortran 2018 in a Nutshell ICAR & Coarray ICAR WRF-Hydro Results Conclusions

51 WRF-Hydro Weather Research Forecast Hydrological Model Currently, ensemble runs involve redundant initialization calculations that occupy approximately 30% of the runtime and use only the processes allocated to one ensemble member. Our use of Fortran 2018 teams will eliminate the redundancy and the amortization of that effort across all of the processes allocated for the full ensemble of simulations.

52 ICAR comparison to Full-physics atmospheric model (WRF) Ideal Hill case ICAR (red) and WRF (blue) precipitation over an idealized hill (green) Real Simulation WRF (left) and ICAR (right) Season total precipitation over Colorado Rockies

53 Mapping Fortran Teams onto MPI Fortran statement or procedure MPI procedure or variable form team( ) MPI_Barrier( ) MPI_Comm_Split( ) time team_number( ) end team color MPI_Barrier( ) 1 2 3

54 Communicator Hand-off (CAF to MPI)

55 Communicator Hand-off (CAF to MPI) CAF in the driver seat: CAF compiler embeds MPI_Init, MPI_Finalize Language extension exposes MPI communicator

56 Communicator Hand-off (CAF to MPI) CAF in the driver seat: CAF compiler embeds MPI_Init, MPI_Finalize MPI under the hood: Language extension exposes MPI communicator Only trivial modification of MPI application Future work: amortize & speed up initialization

57 Overview Fortran 2018 in a Nutshell ICAR & Coarray ICAR WRF-Hydro Results Conclusions

58 OpenSHMEM vs MPI Puts vs gets Coarray ICAR Simulation Time 500 x 500 x 20 grid 2000 x 2000 x 20 grid Number of processes Number of processes GCC 6.3 on NCAR Cheyenne SGI ICE XA Cluster: 4032 nodes, 2 x 18-core Xeon 2.3 GHz

59 Coarray ICAR Speedup 500 x 500 x 20 grid 2000 x 2000 x 20 grid Number of processes Number of processes GCC 6.3 on NCAR Cheyenne SGI ICE XA Cluster: 4032 nodes, 2 x 18-core Xeon 2.3 GHz

60 Coarray ICAR Xeon vs KNL Number of processes Number of processes Compilers & platforms: GNU on Cheyenne; Cray compiler on Edison (Broadwell), Cori (KNL).

61 Coarray ICAR At Scale Number of processes Cray compiler on Edison. Number of processes

62 Conclusions Fortran 2018 is a PGAS language supporting SPMD programming at scale. Programming-model agnosticism is a lifesaver. High productivity pays off: from sharedmemory parallelism to 100,000 cores in ~100 person-hours. CAF interoperates seamlessly with MPI

63 Acknowledgements & References Ethan Gutman Alessandro Fanfarillo James McCreight Brian Friesen Rouson, D., Gutmann, E. D., Fanfarillo, A., & Friesen, B. (2017, November). Performance portability of an intermediatecomplexity atmospheric research model in coarray Fortran. In Proceedings of the Second Annual PGAS Applications Workshop (p. 4). ACM. Rouson, D., McCreight, J. L., & Fanfarillo, A. (2017, November). Incremental caffeination of a terrestrial hydrological modeling framework using Fortran 2018 teams. In Proceedings of the Second Annual PGAS Applications Workshop (p. 6). ACM.