Multi-Physics Multi-Code Coupling On Supercomputers

Transcription

1 Multi-Physics Multi-Code Coupling On Supercomputers J.C. Cajas 1, G. Houzeaux 1, M. Zavala 1, M. Vázquez 1, B. Uekermann 2, B. Gatzhammer 2, M. Mehl 2, Y. Fournier 3, C. Moulinec 4 1) er, Edificio NEXUS I, Office 204 Gran Capitán 2-4, Barcelona, Spain. 2) Institute for Advanced Study (IAS), Technische Universität München, Lichtenbergstraβe 2a, Garching b. München, Germany. 3) EDF-R&D, Fluid Dynamics, Power Generation, and Environment, 6 quai Waiter BP Chatou Cedex, France. 4) Scientific Computing Department, Science and Technology Facilities Council Daresbury Laboratory, Sci- Tech Daresbury Warrington. WA4 4AD, UK.

2 Why Multiphysics? Multiphysics problems are everywhere. In our environment, in our technology, and in our bodies Through years of research, we have codes to simulate, in an efficient way, the components of multiphysics problems independently

3 Multiphysics strategies Monolithic approach The coupled problem is casted as a whole, resulting in a single algebraic (matrix) problem. The coupling conditions are part of the solution of the problem. You can t use pre-existing code for each component of the coupling Partitioned approach Each component of the problem is treated individually, resulting in as many algebraic problems as physical theories involved. The coupling conditions appear as source terms or boundary conditions, and you can use pre-existing codes. However, instabilities or/and inaccuracy may arise.

4 Example of Multiphysics problems - Fluid-structure interaction (interface coupling) - Fluid-structure-thermal interaction (interface+field interaction) - Fluid-structure-combustion interaction - Fluid-fluid interaction (moving subdomains) - Fluid-particle interaction (point coupling) - Structure-thermal interaction - Electro-mechanical-fluid-1D fluid model (our goal )

5 Multiphysics overview Multicode: Legacy code Research code Coupling technologies: Direct coupling Top-level interface Interfaces: library: change code easily (PLE, PRECICE, TDT) In-house: take advantage of internal data structure

6 HPC context ALYA Multiphysics code Finite Element Method (nodes) Solids mechanics + Electrophysiology Incompressible flow + ALE formulation Code_Saturne CFD code Finite Volume Method (cell centered) Coupling library PLE Challenges => Coupling strategy cannot be the bottleneck => Multiphysics in exascale

7 HPC Simulation Tools: Alya Alya is one of the two CFD codes of the PRACE benchmark suite Lindgren (Sweden), Cray XE system at PDC, incompressible flow CPU s (collaboration with Jing Gong from PDC) Huygens, (The Netherlands), IBM power 6, incompressible flow, 2128 CPU's Jugene BG (Germany): CPU's, incompressible flow (Prace project for Mesh multiplication) and, running first tests of FSI in collaboration with Paolo Crosetto (Julich) Fermi BG (Italy): CPU's, incompressible flow + species transport + Lagrangian particles (Prace project for nose) Curie Bullx (France): CPU s, incompressible flow (collaboration with Jing Gong - PDC) Marenostrum: 5000 CPU s compressible flow, incompressible flow, thermal flow (scalability test) BlueWaters: 100,000 CPU s combustion, elctro-mechanical model (collaboraiton with NCSA)

8 HPC context (cont.) Speedup Alya: CFD-Combustion-Chemistry Blue Waters (USA) Collaboration with NCSA 4.2B elements 525M elements 525M elements

9 HPC context (cont.) Speedup Alya: Electro-mechanical model Blue Waters (USA) Collaboration with NCSA 3.4B elements 427M elements 427M elements

10 The ingredients 1. What The coupling variable to exchange 2. Where The coupling interface or volume 3. How The coupling algorithm 4. When The section of the code where the exchange will be performed

11 What Unknown Derived The unknowns of the equations are the coupling variables (velocity, temperature, pressure, displacement) Quantities derived from the unknowns appearing in the physical equations are the coupling variables (heat flux, shear stress)

12 1. On Volume (Low Mach) 2. On Surface (FSI) 3. On points (Fluid/particle) Navier-Stokes Navier-Stokes Navier-Stokes Where Temperature Solid mechanics Particles

13 Gauss-Seidel approach displacement CFD idle idle force CSD No converged Yes Next time step How Jacobi approach CFD force displacement No converged Yes Next time step CSD

14 When Pre-processing Initialisation of coupling Time loop Non-linear iterations loop Assemble matrix Solve linear problem End non-linear iteration loop End time loop Post-processing

15 Pre-processing Initialize coupling Time loop Non-linear iterations loop Assemble matrix Solve linear problem End non-linear iteration loop End time loop Post-processing When Pre-processing Initialize coupling Time loop Non-linear iterations loop Assemble matrix Solve linear problem End non-linear iteration loop End time loop Post-processing

16 Pre-processing Initialize coupling Time loop Non-linear iterations loop Assemble matrix Solve linear problem End non-linear iteration loop End time loop Post-processing When Pre-processing Initialize coupling Time loop Non-linear iterations loop Assemble matrix Solve linear problem End non-linear iteration loop End time loop Post-processing

17 MPI_COMM_WORLD problem1 CFD Alya coupling library Parallel context ALYA coupling/ple/precice are library linked to the code Codes share the same MPI_COMM_WORLD Codes communicate through MPI messages problem2 SM Alya coupling library 10 CPUs 5 CPUs mpirun -n 10 Alya.x problem1 : -n 5 Alya.x problem2

18 Initialize coupling 1. Wet entity (volume, surface, point) => define a set of target points (nodes, Gauss points) 2. Identify candidate host CPUs for each target point Send my target points to candidates Receive answer from candidates Decide who s the owner CPUs 3. Set communication strategy and arrays wet surface wet surface target source

19 Interpolate value MPI_Send value Put Get and put MPI_Recv value Get

20 At organ level,the heart can be seen as the following coupled problem: Electrical Propagation Coupling Mechanical Deformation The heart as a physical system Boundaries Particular features Macro-micro models and bridging scales Live tissue: metabolism, perfusion... Geometrical problems Coupling: volume and boundaries Large variability Uncertainties Personalized models Blood Flow Boundaries 1D Arterial Network

21 The multiphysics problem Four coupled problems: Electrophysiology, Solid Mechanics, 3D Blood flow (with ALE), 1D Blood flow One single mesh for EP and SM (solid), ALE (fluid), network for 1D Non-structured mesh coming from medical image processing Anisotropic media, complex geometries One parallel code to simulate the full problem Staggered strategy: solving the problems sequentially for each time step

22 Barcelona Supercomputing Cen Biomechanics: Cardiac computational model

23 ADAN - Anatomically Detailed Arterial Network (P. Blanco) Complete high-resolution cardiac geometries (D. Einstein) Fluid-electro-mechanical model Synthetic Purkinje system (R. Sebastian) Barcelona Supercomputing Cen Biomechanics: Cardiac computational model

24 ALYA

25 Arterial 1D network Pablo Blanco, LNCC, Brazil ADAN: Anatomically Detailed Arterial Network

26 Internal coupling of ALYA NASTIN + ALEFOR ALYA WORLD MPI_COMM_split SOLIDZ + EXMEDI MPI framework MPI_COMM_WORLD MPI_COMM_split Intra-code coupling adapter ADAN WORLD

27 FSI-Electrophysiology coupling ALYA

28 Upper and lower airways Fluid-Fluid coupling Imperial College London BSC Jackson State University Barcelona Supercomputing Cen Computational Respiratory System

29 Sniff Barcelona Supercomputing Cen Computational Respiratory System

30 Conclusions Multiphysics problems are exciting and defying (let s have fun) We have the means to address multiphysics (in a partitioned way) There exists interfaces for inter-code coupling Multi-code coupling enhance collaboration, define a standard in the data structures would be desirable For intra-code coupling, an internal library is more efficient