Large Scale Simulation of Cloud Cavitation Collapse

Transcription

1 PRACEdays17 Barcelona, May 16-18, 2017 Large Scale Simulation of Cloud Cavitation Collapse Ursula Rasthofer with: Fabian Wermelinger, Petr Karnakov, Panagiotis Hadjidoukas, Petros Koumoutsakos CSElab Computational Science & Engineering Laboratory

2 Power of Cavitation 2

3 Power of Cavitation engineering extracted from: Brennen, Hydrodynamics of pumps, Oxford University Press,

4 Power of Cavitation extracted from: Bazan-Peregrino et al., Cavitation-enhanced delivery of a replicating oncolytic adenovirus to tumors using focused ultrasound, Journal of Controlled Release Volume 169, Issues 1 2, 2013, engineering biomedical applications Extracorporeal_shock_wave_lithotripsy extracted from: Brennen, Hydrodynamics of pumps, Oxford University Press,

5 Power of Cavitation extracted from: Bazan-Peregrino et al., Cavitation-enhanced delivery of a replicating oncolytic adenovirus to tumors using focused ultrasound, Journal of Controlled Release Volume 169, Issues 1 2, 2013, engineering nature biomedical applications Extracorporeal_shock_wave_lithotripsy extracted from: Brennen, Hydrodynamics of pumps, Oxford University Press, Knallkrebse herve.cochard.free.fr 2

6 Cloud Cavitation Collapse esource.alstrbology.com 3

7 Cloud Cavitation Collapse 3

8 Cloud Cavitation Collapse thousands of bubbles 3

9 Cloud Cavitation Collapse thousands of bubbles cloud dynamics dominated by bubble-bubble interactions 3

10 Cloud Cavitation Collapse thousands of bubbles cloud dynamics dominated by bubble-bubble interactions experiments - challenging high frequencies and microscopic length scales - risk of damage 3

11 Cloud Cavitation Collapse thousands of bubbles cloud dynamics dominated by bubble-bubble interactions experiments - challenging high frequencies and microscopic length scales - risk of damage theory - based on Rayleigh-Plesset-like equations - spherical bubbles usually assumed 3

12 Cloud Cavitation Collapse thousands of bubbles cloud dynamics dominated by bubble-bubble interactions experiments - challenging high frequencies and microscopic length scales - risk of damage theory - based on Rayleigh-Plesset-like equations - spherical bubbles usually assumed simulations - Lagrangian approaches for bubbles - fully resolved bubble clouds usually restricted to small clouds 3

13 Outline Computational Approach Simulations Conclusions 4

14 Outline Computational Approach Simulations Conclusions 5

15 Modeling Strategies complexity/fidelity computational efficiency 6

16 Modeling Strategies two-fluid models Lagrangian methods as well as Eulerian methods, e.g., based on level-set description complexity/fidelity computational efficiency 6

17 Modeling Strategies two-fluid models Lagrangian methods as well as Eulerian methods, e.g., based on level-set description mixture approches e.g., thermodynamic equilibrium cavitation model complexity/fidelity computational efficiency 6

18 Modeling Strategies two-fluid models Lagrangian methods as well as Eulerian methods, e.g., based on level-set description single-fluid models e.g., diffuse interface methods mixture approches e.g., thermodynamic equilibrium cavitation model complexity/fidelity computational efficiency 6

19 Diffuse Interface Method FLUID 1 O(h) INTERFACE FLUID 2 [Kapila et al. 2001, Murrone & Guillard 2005, Saurel et al. 2009, Tiwari et al. 2013, ] 7

20 Diffuse Interface Method where r ( r ( 2 ( u) + r ( u u + + r ((E + p) + u r 2 = K r u K = 1 2 ( 1 c c 2 2) 1 2 c c =1 O(h) FLUID 2 FLUID 1 INTERFACE [Kapila et al. 2001, Murrone & Guillard 2005, Saurel et al. 2009, Tiwari et al. 2013, ] 7

21 Thermodynamic Closure 8

22 Thermodynamic Closure stiffened equation-ofstate (EoS) for pure fluids p k =( k 1) k e k k p c,k [Menikoff & Plohr 1989, ] 8

23 Thermodynamic Closure stiffened equation-ofstate (EoS) for pure fluids p k =( k 1) k e k k p c,k [Menikoff & Plohr 1989, ] mixture relations = e = 1 1 e e 2 8

24 Thermodynamic Closure stiffened equation-ofstate (EoS) for pure fluids p k =( k 1) k e k k p c,k [Menikoff & Plohr 1989, ] mixture relations = e = 1 1 e e 2 pressure/velocity equilibrium p=pk/u=uk 8

25 Thermodynamic Closure stiffened equation-ofstate (EoS) for pure fluids p k =( k 1) k e k k p c,k [Menikoff & Plohr 1989, ] mixture relations = e = 1 1 e e 2 pressure/velocity equilibrium p=pk/u=uk mixture speed of sound 1 c 2 = 1 1 c 2 1 [Wood 1930] c 2 2 8

26 Godunov-Type Finite Volume Method 9

27 Godunov-Type Finite Volume Method reformulation of gas-volumefraction + r ( 2u) = 2 1 c c c 2 1 r u [Johnsen & Colonius 2006] 9

28 Godunov-Type Finite Volume Method reformulation of gas-volumefraction equation [Johnsen & Colonius 2006] high-order reconstruction of face values of primitive variables using WENO3/5 [Liu et al. 1994, Jiang & Shu + r ( 2u) = Ui-1 ci-1 WL WR Fi-1/2 xi-1/2 2 1 c c c 2 1 Ui h ci Fi+1/2 xi+1/2 Ui+1 ci+1 r u 9

29 Godunov-Type Finite Volume Method reformulation of gas-volumefraction equation [Johnsen & Colonius 2006] high-order reconstruction of face values of primitive variables using WENO3/5 [Liu et al. 1994, Jiang & Shu 1998] approximate HLLC Riemann solver for flux reconstruction [Toro et al. + r ( 2u) = r Ui-1 ci-1 WL WR Fi-1/2 xi-1/2 U1 UL* 2 1 c c c 2 1 y Ui h ci Fi+1/2 xi+1/2 UR* c U2 Ui+1 ci+1 s r u U- U+ x 9

30 Godunov-Type Finite Volume Method reformulation of gas-volumefraction equation [Johnsen & Colonius 2006] high-order reconstruction of face values of primitive variables using WENO3/5 [Liu et al. 1994, Jiang & Shu 1998] approximate HLLC Riemann solver for flux reconstruction [Toro et al. 1994] low-storage 3rd-order Runge- Kutta scheme for time discretization [Gottlieb et al. + r ( 2u) = r U- Ui-1 ci-1 WL WR Fi-1/2 xi-1/2 U1 UL* 2 1 c c c 2 1 y Ui h ci Fi+1/2 xi+1/2 UR* c U2 Ui+1 U+ ci+1 s r u x 9

31 Cubism-MPCF compressible multicomponent flow solver tailored to HPC systems [Rossinelli et al. 2013, P. E. Hadjidoukas et al. 2015] 3D Cartesian-grid finite volume solver wavelet-based compression of simulation data 10

32 Cubism-MPCF compressible multicomponent flow solver tailored to HPC systems [Rossinelli et al. 2013, P. E. Hadjidoukas et al. 2015] 3D Cartesian-grid finite volume solver wavelet-based compression of simulation data TIME TO SOLUTION (no I/O) Tw = 1.8 Tw= 29.7 (TUM) Tw= (Stanford) PFLOPS (% Peak) 14.4 PFLOPS (72%) 0.1-3% (TUM) % (Stanford) I/O COMPRESSION X - - SIZE 1.3 E13-15K bubbles 1.2 E K bubbles (TUM) 0.4 E13 - Turbulence (Stanford) 10

33 Cubism-MPCF: Software Layout

34 Cubism-MPCF: Software Layout computational domain Node Cluster subdomain Core cluster (MPI) node (OpenMP) core (SIMD): WENO, HLLC block grid cell 11

35 Node and Cluster Layer OpenMP parallelization - dynamic work scheduling - 1 thread exclusively processes 1 block node i-1 node i node i+1 MPI parallelization - non-blocking P2P communication for ghost blocks - communication time ~ O(time for processing 1-2 blocks) 12

36 Core Layer block-based memory layout - increases spatial locality instruction and data level parallelism - explicit vectorization - code fusion techniques temporal locality - ring buffers for active data slices, e.g., in WENO, HLLC 13

37 Validation [Rasthofer et al. 2017] liquid-gas shock tube single-bubble collapse 14

38 Outline Computational Approach Simulations Conclusions 15

39 50K Bubbles, 64 Billion Cells 25K time steps, 72h x 32K cores 16

40 50K Bubbles, 64 Billion Cells 25K time steps, 72h x 32K cores 16

41 Cloud Setup R B R C y d G r B R B x z 17

42 Cloud Setup R B R C y d G r B R B x z cloud generation - locations: random - radii: constant or random collapse driven by increase of ambient pressure 17

43 Cloud Setup R B R C y d G r B R B x z cloud generation - locations: random - radii: constant or random collapse driven by increase of ambient pressure gas-volume fraction = 1 R 3 C cloud interaction parameter = Req Xn B RB,i 3 i=1 R avg 2 17

44 Large-Scale Cloud air bubbles in water pc = 1 bar p = 10 bar cloud radius: 45 mm average bubble radius: 0.69 mm gas-volume faction: 5% cloud interaction parameter: 28 18

45 Large-Scale Cloud air bubbles in water pc = 1 bar p = 10 bar cloud radius: 45 mm average bubble radius: 0.69 mm gas-volume faction: 5% cloud interaction parameter: 28 18

46 Cloud Collapse p S p S,max V 2 V 2 (0) t [µs] 19

47 Cloud Collapse p S p S,max V 2 V 2 (0) t [µs] 19

48 Micro-Jet Formation y x 20

49 Micro-Jet Formation y x 20

50 Micro-Jet Formation due to pressure gradient along interface directed toward cloud center y x 20

51 Further Results The discussion of the results has been removed from the online version of the presentation. We refer to the publications provided at the end of the presentation for discussion of the results. 21

52 Outline Computational Approach Simulations Conclusions 22

53 Summary & Outlook 23

54 Summary & Outlook diffuse interface method and Cubism-MPCF important to account for gas expansion and compression in interface compressible multicomponent flow solver capable of processing up to 7x10 11 cells per second 23

55 Summary & Outlook diffuse interface method and Cubism-MPCF important to account for gas expansion and compression in interface compressible multicomponent flow solver capable of processing up to 7x10 11 cells per second HPC for collapsing clouds with up to 50K bubbles comparison with bubble-particle approaches insights into induced pressure and velocity fields, wave dynamics, bubble shape evolution as well as damage potential 23

56 Summary & Outlook diffuse interface method and Cubism-MPCF important to account for gas expansion and compression in interface compressible multicomponent flow solver capable of processing up to 7x10 11 cells per second HPC for collapsing clouds with up to 50K bubbles comparison with bubble-particle approaches insights into induced pressure and velocity fields, wave dynamics, bubble shape evolution as well as damage potential integration of mass transfer and application to turbulent cavitating flow 23

57 Thank you for your attention! 1. U. Rasthofer, F. Wermelinger, P. Hadijdoukas, P. Koumoutsakos, Large-scale simulation of cloud cavitation collapse, ICCS Proceedings 2017, accepted for publication. 2. U. Rasthofer, F. Wermelinger, P. Koumoutsakos, A comparative study on cloud cavitation collapse: Rayleigh-Plesset-like equations versus full three-dimensional simulations, 2017, in preparation. 3. U. Rasthofer, F. Wermelinger, P. Karnakov, J. Sukys, P. Hadijdoukas, P. Koumoutsakos, Numerical investigation of collapsing clouds with up to O(10 4 ) gas bubbles, 2017, in preparation. 4. J. Sukys, U. Rasthofer, F. Wermelinger, P. Hadijdoukas, P. Koumoutsakos, Optimal fidelity multi-level Monte Carlo for quantification of uncertainty in simulations of cloud cavitation collapse, 2017, submitted for publication (preprint available online: 5. P. Hadijdoukas, D. Rossinelli, F. Wermelinger, U. Rasthofer, P. Koumoutsakos, A high-performance compression framework for extreme-scale CFD data, 2017, in preparation. 24