Computational Fluid Dynamics with the Lattice Boltzmann Method KTH SCI, Stockholm

Size: px

Start display at page:

Download "Computational Fluid Dynamics with the Lattice Boltzmann Method KTH SCI, Stockholm"

Stanley Manning
6 years ago
Views:

1 Computational Fluid Dynamics with the Lattice Boltzmann Method KTH SCI, Stockholm March 17 March 21, 2014 Florian Schornbaum, Martin Bauer, Simon Bogner Chair for System Simulation Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

2 Introductory Lecture Monday, March 17, 2014

3 Outline Introduction (of us / of participants) The walberla Framework What is the walberla Framework? Examples / Applications Course Schedule Introduction to the Lattice Boltzmann Method Introduction to the walberla Framework Data Structures / Underlying Concepts Domain Decomposition & Parallelization A Prototypical Simulation 3

4 Introduction (of us / of participants) Who are we? Who are you?

Introduction (of us / of participants) Martin Bauer [computational engineering @ FAU] PhD

this compact course: Monday Friday Simon Bogner [math & computer science @ FAU] PhD student @

on Tuesday & Wednesday Florian Schornbaum [computer science @ FAU] PhD this compact course:

5 Introduction (of us / of participants) Martin Bauer [computational FAU] PhD LSS (chair for system simulation) core developer of walberla framework tutor for this compact course: Monday Friday Simon Bogner [math & computer FAU] PhD LSS (chair for system simulation) models and methods for lattice Boltzmann tutor for lectures on Tuesday & Wednesday Florian Schornbaum [computer FAU] PhD LSS (chair for system simulation) core developer of walberla framework tutor for this compact course: Monday Friday 5

6 Introduction (of us / of participants) Chair for System Simulation (1 st floor) Erlangen The University of Erlangen-Nuremberg [Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)] has five faculties with over 270 chairs. 6

7 Introduction (of us / of participants) Chair for System Simulation (1 st floor) Faculty of Engineering (marked: Chair for System Simulation) FAU Erlangen-Nürnberg Faculty of Engineering Department of Computer Science Chair for System Simulation 7

8 Introduction (of us / of participants) Who are you?? Name? Field of study? Semester? Know-how in (computational) fluid mechanics? Programming Experience? (especially C++? C? Java?) 8

9 The walberla Framework What is the walberla Framework? Examples / Applications

The walberla Simulation Framework written in C++(11) main focus on CFD simulations based on the lattice Boltzmann method (LBM) but generally suitable for all kinds of numeric codes working with

10 The walberla Simulation Framework written in C++(11) main focus on CFD simulations based on the lattice Boltzmann method (LBM) but generally suitable for all kinds of numeric codes working with uniform domain decompositions (LSE solvers, phase field method) at its very core designed as an HPC software framework: scales from laptops to current petascale supercomputers largest simulation: 1,835,008 processes (IBM Blue Jülich) hybrid parallelization: MPI + OpenMP vectorization of compute kernels open source widely applicable lattice Boltzmann framework from Erlangen 10

11 The walberla Simulation Framework written in C++(11) main focus on CFD simulations based on the lattice Boltzmann method (LBM) but generally suitable for all kinds of numeric codes working with uniform domain decompositions (LSE solvers, phase field method) at its very core designed as an HPC software framework: scales from laptops to current petascale supercomputers largest simulation: 1,835,008 processes (IBM Blue Jülich) hybrid parallelization: MPI + OpenMP vectorization of compute kernels open source 11

The walberla Simulation Framework coupling

automated build and test system: New git

on different systems with different compilers

OpenMP on/off, Debug/Release, float/double, ).

12 The walberla Simulation Framework coupling with in-house rigid body physics engine pe automated build and test system: New git commit to central repository triggers builds on different systems with different compilers and different configurations (MPI on/off, OpenMP on/off, Debug/Release, float/double, ). After successful build, unit tests & set of applications are executed. support for different platforms (Linux, Windows) and compilers llvm/clang 12

The walberla Simulation Framework blood flow through coronary artery tree (C. Godenschwager) [ C. Godenschwager, F. Schornbaum, M. Bauer, H. Köstler, and U.

13 The walberla Simulation Framework blood flow through coronary artery tree (C. Godenschwager) [ C. Godenschwager, F. Schornbaum, M. Bauer, H. Köstler, and U. Rüde, A Framework for Hybrid Parallel Flow Simulations with a Trillion Cells in Complex Geometries, SC13, Denver] domain decomposition of complex geometry 13

14 The walberla Simulation Framework electron beam melting metal powder: 3D printing (R. Ammer, M. Markl) [ liquid: walberla (LBM) metal powder: pe (fully resolved rigid bodies) ] POV-Ray rendering of actual simulation 14

15 The walberla Simulation Framework electron beam melting metal powder: 3D printing (R. Ammer, M. Markl) [ liquid: walberla (LBM) metal powder: pe (fully resolved rigid bodies) ] camera footage of real application 15

16 The walberla Simulation Framework liquid-gas-solid flow simulation: stable floating positions of boxes with different densities (S. Bogner) [ liquid: walberla (LBM) boxes: pe (fully resolved rigid bodies) ] POV-Ray rendering of actual simulation 16

17 The walberla Simulation Framework liquid-gas-solid flow simulation: bubble rising through pearls hovering under water (S. Bogner) [ liquid: walberla (LBM) pearls: pe (fully resolved rigid bodies) ] POV-Ray rendering of actual simulation 17

18 The walberla Simulation Framework study of the vocal fold LBM with grid refinement (F. Schornbaum, E. Fattahi) THVZs 18

19 The walberla Simulation Framework study of the vocal fold LBM with grid refinement (F. Schornbaum, E. Fattahi) DNS (direct numerical simulation) Reynolds number: 1000 / D3Q19 TRT 4300 SuperMUC 101,466,432 fluid cells 25,800 blocks with 16 x 16 x 16 cells 19

20 The walberla Simulation Framework study of the vocal fold LBM with grid refinement (F. Schornbaum, E. Fattahi) number of different grid levels: time steps / sec (finest grid) total number of time steps (finest grid): 864,000 without refinement: 55.2 times more memory and 98.6 times the workload 20

21 Course Schedule

22 Course Schedule Monday Tuesday Wednesday Thursday Friday Introduction [Lecture] LB [Lecture] LB [Lecture] HPC (MPI) [Lecture] Project Presentations C++ [Lecture] C++ [Lecture] LB [Lecture] Work on Projects Project Presentations Lunch Break Lunch Break Lunch Break Lunch Break Lunch Break Getting to Know the Environment [Hands-On] walberla Tutorials [Hands-On] Introduction of Projects Work on Projects Work on Projects Work on Projects Morning time slots: and Lunch Break: Afternoon: (Friday: ) 22

23 Introduction to the Lattice Boltzmann Method

24 Introduction to the LBM regular grid with multiple particle distribution functions per cell particle distribution function = scalar value different models: D2Q9, D3Q19, D3Q27, Macroscopic quantities (velocity, density, ) can be calculated from the particle distribution functions. 24

25 Introduction to the LBM regular grid with multiple particle distribution functions per cell particle distribution function = scalar value different models: D2Q9, D3Q19, D3Q27, Macroscopic quantities (velocity, density, ) can be calculated from the particle distribution functions. 25

26 Introduction to the LBM regular grid with multiple particle distribution functions per cell particle distribution function = scalar value different models: D2Q9, D3Q19, D3Q27, Every particle distribution function is associated with a certain direction. 26

27 Introduction to the LBM explicit method time stepping (separated into two steps) two steps: stream & collide streaming: values are copied to neighboring cells collision: All values of each cell are updated using only the values of this cell. [cell-local operation!] 27

28 Introduction to the LBM explicit method time stepping (separated into two steps) two steps: stream & collide streaming: values are copied to neighboring cells collision: All values of each cell are updated using only the values of this cell. [cell-local operation!] 28

29 Introduction to the LBM explicit method time stepping (separated into two steps) two steps: stream & collide streaming: values are copied to neighboring cells collision: All values of each cell are updated using only the values of this cell. [cell-local operation!] 29

30 Introduction to the LBM explicit method time stepping (separated into two steps) two steps: stream & collide streaming: values are copied to neighboring cells collision: All values of each cell are updated using only the values of this cell. [cell-local operation!] For the collision, different operators exist: SRT, TRT, MRT 30

31 Introduction to the LBM boundary treatment pre-streaming step boundary Particle distribution functions are calculated for boundary cells which are neighboring fluid cells. These calculated values depend on the underlying boundary condition type. 31

32 Introduction to the LBM boundary treatment pre-streaming step boundary Particle distribution functions are calculated for boundary cells which are neighboring fluid cells. During streaming, these values are transferred/copied into the neighboring fluid cells. 32

33 Introduction to the LBM boundary treatment pre-streaming step boundary Particle distribution functions are calculated for boundary cells which are neighboring fluid cells. During streaming, these values are transferred/copied into the neighboring fluid cells. 33

34 Introduction to the LBM boundary treatment pre-streaming step boundary Particle distribution functions are calculated for boundary cells which are neighboring fluid cells. During streaming, these values are transferred/copied into the neighboring fluid cells. 34

35 Introduction to the LBM boundary treatment pre-streaming step boundary The lattice Boltzmann method allows to handle very complex geometries easily! Cells are typically just marked as being a boundary cell of a certain type: no-slip, free slip, velocity bounce-back, pressure, open boundary, [ flag field = special grid that stores these markings] 35

36 Introduction to the LBM A typical time step of the LBM looks like as follows: 1. boundary treatment is performed (= values in the boundary cells are prepared for streaming) 2. the streaming step is executed (requires nearest neighbor access) 3. the collision operator is evaluated (cell-local operation) A typical parallel time step includes communication: 1. all values of cells on the border of a process s partition of the domain are exchanged between neighboring processes (involves so-called ghost layer cells) 2. boundary treatment is performed (= values in the boundary cells are prepared for streaming) 3. the streaming step is executed (requires nearest neighbor access) 4. the collision operator is evaluated (cell-local operation) 36

37 Introduction to walberla Data Structures / Underlying Concepts Domain Decomposition & Parallelization A Prototypical Simulation

38 Introduction to walberla geometry given by surface mesh domain decomposition into blocks flow simulation only in here (example: complex geometry of an artery) load balancing empty blocks are discarded Load balancing can be based on either space-filling curves (Z-order/Morton order, Hilbert curve) using the underlying forest of octrees or graph partitioning (METIS, ). Whatever fits best the needs of the simulation. 38

39 Introduction to walberla geometry given by surface mesh domain decomposition into blocks load balancing empty blocks are discarded Load balancing can be based on either space-filling curves (Z-order/Morton order, Hilbert curve) using the underlying forest of octrees or graph partitioning (METIS, ). Whatever fits best the needs of the simulation. 39

40 Introduction to walberla geometry given by surface mesh domain decomposition into blocks load balancing empty blocks are discarded allocation of block data ( grids) The domain decomposition and load balancing can be performed during the actual simulation OR 40

41 Introduction to walberla geometry given by surface mesh domain decomposition into blocks load balancing empty blocks are discarded separation of domain partitioning from simulation DISK DISK file size: kilobytes to few megabytes allocation of block data ( grids) 41

42 Introduction to walberla geometry given by surface mesh domain decomposition into blocks All of this (the entire pipeline) works just the same when grid refinement is used. load balancing empty blocks are discarded separation of domain partitioning from simulation DISK DISK file size: kilobytes to few megabytes allocation of block data ( grids) 42

43 Introduction to walberla Domain Decomposition: regular decomposition into blocks containing uniform grids grid refinement: octree-like decomposition special case of the much more general forest of octrees data structure non-uniform/refined grids forest of octrees: each block contains a uniform grid of the same size 2:1 balance between neighboring cells on level transitions (enables LB refinement algorithms) 43

44 Introduction to walberla Domain Decomposition: regular decomposition into blocks containing uniform grids special case of the much more general forest of octrees data structure non-uniform/refined grids octree-like decomposition of a globally uniform grid forest of octrees: each block contains a uniform grid with cells of the same size performance of uniform grids & good fitting to complex geometries 44

45 Introduction to walberla Domain Decomposition: regular decomposition into blocks containing uniform grids octree-like decomposition of the simulation space?????????????????????????????? special case of the much more general forest of octrees data structure non-uniform/refined grids forest of octrees: In general, each block contains whatever data was registered by the application: can be any arbitrary (sub)grid, scalar values, any object (in terms of C++), 45

46 Introduction to walberla Parallelization : data exchange on borders between blocks via ghost layers sender process receiver process (slightly more complicated for non-uniform domain decompositions, but the same general ideas still apply) typical parallelization of the underlying algorithm (e.g., LB): 1. perform communication: copy outermost layer of inner cells of perform communication: sender to ghost layer of receiver 2. perform one time step of the computation algorithm only on all inner cells of each block this computation algorithm is only allowed to access nearest neighbor cells during calculations! 46

47 Introduction to walberla A Prototypical Simulation: 1. construct domain decomposition (includes load balancing = distribution of blocks to all available processes) 2. add data to blocks (LB: at least one grid that stores the particle distribution functions and one grid that is used for marking cells as either being fluid or obstacle additional grids/classes may be required/useful [e.g., an object that is responsible for the boundary treatment]) 3. set-up geometry/boundaries (= mark cells as either being fluid or a boundary of a certain type information about the geometry/boundaries can be loaded from a mesh file, read in from a configuration file, set in the source code, etc.) 4. specify the algorithms/functions that are supposed to be executed in each time step (basic LB: communication, boundary treatment, stream & collide, output[optional]) 5. run the simulation = execute fixed number of time steps 47

48 THANK YOU FOR YOUR ATTENTION! QUESTIONS? (visit

simulation framework for piecewise regular grids

WALBERLA, an ultra-scalable multiphysics simulation framework for piecewise regular grids ParCo 2015, Edinburgh September 3rd, 2015 Christian Godenschwager, Florian Schornbaum, Martin Bauer, Harald Köstler