A dynamic load-balancing strategy for large scale CFD-applications

Size: px

Start display at page:

Download "A dynamic load-balancing strategy for large scale CFD-applications"

Hester Wendy King
5 years ago
Views:

A dynamic load-balancing strategy for large scale CFD-applications Philipp Offenhäuser 10.

1 A dynamic load-balancing strategy for large scale CFD-applications Philipp Offenhäuser /20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

2 Outline Motivation and Background Load-Balancing strategy Results Further Challenges Conclusion 2/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

3 Motivation - Top500 from 1993 to Rmax Rank 1 Rmax Rank 10 #Cores Rank 1 #Cores Rank Performance [GF lop/s] # Cores Year 10 0 (Source: top500.org) 3/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

4 Motivation HPC systems generate their power by facilitating hundreds of thousands of cores Massive parallel algorithms are implemented Computational effort must be distributed evenly across all cores Initial domain decomposition techniques are well know Well balanced applications for scientific use-cases are developed Performance [GF lop/s] Rmax Rank Rmax Rank #Cores Rank #Cores Rank Year # Cores (Source: top500.org) 4/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

5 Definitions Element-weight: Specific computational effort of an element. 5/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

6 Definitions Element-weight: Specific computational effort of an element. Load of a MPI-domain: Sum of the element-weights of a MPI-domain. 5/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

7 Definitions Element-weight: Specific computational effort of an element. Load of a MPI-domain: Sum of the element-weights of a MPI-domain. Load-imbalance: Load-imbalance L = Load-imbalance T = maximum load average load maximum computation time average computation time 5/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

8 Example 1 - Finite-Volume-Code FS3D References: Institute of Aerospace Thermodynamics - University of Stuttgart 6/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

9 Examples for load-imbalance - Volume of Fluid Method y ,7 0, , , ,7 0 Treatment of multiple phases 0 liquid f (x, t) = (0; 1) interface 1 solid Reconstruction of the interface Additional computational effort for the interface elements Source: ITLR x 7/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

10 Examples for load-imbalance - Volume of Fluid Method y ,7 0, , , ,7 0 Treatment of multiple phases 0 liquid f (x, t) = (0; 1) interface 1 solid Reconstruction of the interface Additional computational effort for the interface elements Source: ITLR x 7/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

11 Example 2 - Discontinuous Galerkin Spectral Element Method (DG SEM) based code FLEXI References: Institute of Aerodynamics and Gas Dynamics - University of Stuttgart Numerical research group FLEXI on github: 8/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

12 FLEXI - a high-order numerical framework Hight-order numerical approach Massively parallel CFD-code designed for HPC-systems DG SEM enables efficient communication-pattern (communication hiding) FLEXI tested on different HPC-systems: Hornet, HazelHen, JUQueen, Marconi FLEXI is used for very large academic use-cases 9/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

13 Example - gas injection Numerical approach was extended to simulate complex fluid-flow The FV-Sub-Cell-Method is used for shock capturing Code is used for industrial use cases Depending on the local solution the FV-Sub-Cell-Method is turned on/off FV-Sub-Cell-Method adds local computational effort 10/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

14 Local computational effort T = 0,00 ms T = 0,25 ms T = 0,50 ms 11/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

Motivation for dynamic load-balancing 1.30 1.25 load / average load 1.20 1.15 1.10 1.05 1.00 0.95 10 20 30 40 50 60 70 80 90 core id Load distribution of a test-case on 96 cores.

15 Motivation for dynamic load-balancing load / average load core id Load distribution of a test-case on 96 cores. Small number of cores are over-loaded Huge number of cores are under-loaded Performance of the application suffers from the overload on a small number of cores 12/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

16 A dynamic load-balancing strategy Step 1: Calculate a new optimized load distribution Step 2: Distribute the load between MPI-processes 13/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

17 A dynamic load-balancing strategy Step 1: Calculate a new optimized load distribution Step 2: Distribute the load between MPI-processes Challenges: Global optimization problem Simple and fast calculation of the optimized load distribution Keep the communication overhead small 13/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

18 A dynamic load-balancing strategy Step 1: Calculate a new optimized load distribution Step 2: Distribute the load between MPI-processes Challenges: Global optimization problem Simple and fast calculation of the optimized load distribution Keep the communication overhead small Challenges: Communication-structure originates during run-time Communication-structure changes during run-time Keep the communication overhead small 13/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

19 A dynamic load-balancing strategy - approach w2 w1 Elementweight Core Core Using a space-filling-curve for the domain-decomposition Mapping the 3D-problem to a 1D-problem Minimal communication (only one time) Simple communication pattern 14/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

20 1. Calculate a new optimized load distribution An optimization algorithm based on the element specific computational effort is used for the load-distribution. cavg Pn = min( c P n 1 avg + Pn 1, cp n 1 avg + Pn 2 ) load / average load load / average load core id core id Load distribution without load-balancing Load distribution with load-balancing Load distribution of a test-case on 96 cores. 15/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

Step 2: Distributing the load between MPI-processes Using again the 1D-structure of the space-filling-curve Intra-node-communication via MPI-Shared-Memory-Window Shared-Memory used for

21 Step 2: Distributing the load between MPI-processes Using again the 1D-structure of the space-filling-curve Intra-node-communication via MPI-Shared-Memory-Window Shared-Memory used for communication and to store the results during the load-balancing MPI-communication only between nodes 16/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

the elements every 1,000 timesteps 10% reduction of the wall-time 17/20 ::

22 Results - Element distribution T 0 = 0,00ms T 1 = 0,25ms T 2 = 0,50ms Element distribution depends on the computational effort Redistrubution of the elements every 1,000 timesteps 10% reduction of the wall-time 17/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

23 Dynamic load distribution 18/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

24 Further Challenges - detection of load-imbalance at run-time State of the art: I Different powerful performance measuring tools are available I All tools need an instrumentation and produce overhead I Results of the measurement only available after simulation I No feedback loop to the application 19/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

Further Challenges - detection of load-imbalance at run-time State of the art: I Different powerful performance measuring tools are available I All tools need an instrumentation and produce overhead

25 Further Challenges - detection of load-imbalance at run-time State of the art: I Different powerful performance measuring tools are available I All tools need an instrumentation and produce overhead I Results of the measurement only available after simulation I No feedback loop to the application Challenges: I Measurement has to be integrated in the application I Feedback loop to the application I Measurement of the load-imbalance with a minimal overhead 19/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

26 Conclusion HPC systems generate their power by facilitating hundreds of thousands of cores Massively parallel CFD-codes are implemented Initial domain decomposition techniques are well know Well balanced CFD-applications Simulation of complex fluid flow varying load distribution during run-time occurrence of the load distribution hard to predict spatially and chronologically simulation specific load-imbalance occurs For efficient application execution dynamic load-balance strategies are indispensable Efficient redistribution of the computational load Detection of load-imbalance with a very low overhead 20/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

27 Thank you for your attention! 20/20 :: A dynamic load-balancing strategy for large scale CFD-applications :: ::

Unstructured Grid Numbering Schemes for GPU Coalescing Requirements

Unstructured Grid Numbering Schemes for GPU Coalescing Requirements Andrew Corrigan 1 and Johann Dahm 2 Laboratories for Computational Physics and Fluid Dynamics Naval Research Laboratory 1 Department