Towards Large Scale Predictive Flow Simulations

Transcription

1 Towards Large Scale Predictive Flow Simulations using ITAPS Tools and Services Onkar Sahni a a Present address: Center for Predictive Engineering and Computational Sciences (PECOS), ICES, UT Austin ollaborators: Mark Shephard (RPI), Ken Jansen (UColo), Miki Amitay (RPI), Karen Devine (SNL), Lori Diachin (LLNL) 3 rd March 2011, SIAM CSE11, Reno, NV. 1

2 Outline of the Presentation Problems of interest: real world flow applications Selected ITAPS features: Anisotropic adaptivity (including semi structured/boundary layer meshes) Complex curved geometry (including high order discretizations) Massively parallel computation (including dynamic load balancing) Selected results (on parallel scaling and production simulations): Strong scaling up to 288k cores (including meshes with up to 8B elements) Simulation of active flow control (including comparison with experiments) Closing remarks 2

3 Flow Applications of Interest Cardiovascular flows (e.g., virtual surgical planning) Aerodynamic flows (e.g., aero shaping ) Two phase flows (e.g., burnout prediction) Human coarctation Abdominal aortic aneurysm (AAA) Active flow control via synthetic jets on wings Two phase annular flow Plunging liquid jet Active flow control in inlet duct 3

4 Require Adaptive and Parallel Techniques (ITAPS Tools and Services) Applications of interest involve: solution features that are unknown in terms of location and evolution length scales with strong anisotropy (e.g., boundary layers, shock waves) geometries involve high curvature and sharp corners thousands of implicit solves with O(100M 1B) degrees of freedom To perform these simulations in a relevant time frame, we use adaptive and parallel techniques provided by ITAPS tools and services 4

5 Unstructured Adaptivity (including Boundary Layer Meshes) Need to use general unstructured, anisotropic meshes for complex geometries/solutions Healthy human aorta (with adapted boundary layer mesh) Plunging liquid jet (colors indicate parts of a partitioned mesh) 5 Sahni et. al., EWC 2008 Galimov et. al., AMS 2010

6 Boundary Layer Mesh Adaptivity for Viscous Flows Porcine aorta (anisotropic, fully unstructured and boundary layer meshes) Boundary layer mesh Fully unstructured mesh 6 Sahni et. al., EWC 2008

7 Example of Adapted BL Mesh Counter rotating pair of synthetic jets Initial BL mesh Adapted BL mesh 7

8 Example of Mesh Curving (High order Analysis) Curving procedures for BL mesh for high order numerical analysis 8 Sahni et. al., FEAD 2010

9 Parallel Mesh Adaptation Edge split on partitioned mesh Edge collapse on partitioned mesh Simulation of air entrainment in the case of a plunging jet: (colors indicate different parts in partition) Air entrainment for a plunging jet Clip view of adapted (partitioned) mesh Magnified view of mesh (near air entrainment) 9 Galimov et. al., AMS 2010

10 Graph based Partitioning Both global and local graphs can be used to create mesh partitions at extreme scale (>=100K parts): Global graph Global partition Initial mesh partition (partition from 3 parts to 6 parts) Local graph Local partition 10 Zhou et. al., SIAM JScC 2010

11 Strong Scaling 1B Mesh up to 288k Cores AAA 1B elements: three supercomputer systems up to full scale 11 Sahni et. al., SC on XT5 and on BGP secs for 20 time steps

12 Scaling Analysis 1B Mesh up to 160k Cores AAA 1B elements: scaling analysis (t tot =t comm +t comp ) comm to comp ratio increases 12 Sahni et. al., SC09

13 Strong Scaling 8B Mesh up to 128k Cores AAA 8B elements: full scale on IBM BG/P and Cray XT5 systems 13 Sahni et. al., SC09; Zhou et. al., JofSc 2010; Zhou et. al., CM 2010

14 Joint Experimental and Numerical Studies (Two Selected Cases with Active Flow Control) Complementary experimental and numerical investigations Tangential control jets in short inlet ducts Finite span synthetic jets on a wing section Exp setup Exp setup Case 1 (internal flow) Case 2 (external flow) CFD setup CFD setup 14

15 Case 1: Tangential Control Jets in a Short Inlet Duct Initial mesh Adapted mesh Time averaged in plane velocity for forced cases Exp CFD CFD 15 Vaccaro et. al., JFM (submitted)

16 Case 2: Finite span Synthetic Jets on a Wing Section Adapt 0 Adapt 1 Adapt 2 Iso surface of time averaged total velocity for forced cases Exp CFD CFD 16 Sahni et. al., JFM 2011 (published)

17 Large Parametric Space and Significant Uncertainties Blowing ratio and layout of jets? As designed, as built/realized or as operated? Closed loop flow control? Tangential control jets in short inlet ducts Finite span synthetic jets on a wing Case 1 Sanded Baseline (h=1mm) Case 2 Baseline (h=0.5mm) Forced (h=1mm+deform.) 17

18 Closing Remarks Real world problems at extreme scale using ITAPS tools/services: Anisotropic adaptivity for arbitrarily complex curved geometries Strong scaling up to 288k cores including full scale on several systems Simulation of aerodynamic, cardiovascular and two phase flows Even with these advances, significant challenges remain: Large parametric space (Re, α, C b /C μ, layout of jets, etc.) Many uncertainties in inputs and modeling Closed loop flow control 18

19 Acknowledgements Grants: NSF grant OCI (PetaApps) and DOE grant DE FC02 06ER25769 (ITAPS under SciDAC) Computing resources: Ranger & Spur at TACC, Kraken at NICS and BigBen at PSC (NSF TeraGrid systems), Intrepid at ANL and Franklin at NERSC (DOE systems), CCNI at RPI, JuGene at JSC (Germany), Software cpmonents: PHASTA, ITAPS, Zoltan, and also, Acusim (Altair), Simmetrix, Kitware, Thanks to: SCOREC team, FCRL team, ITAPS team, various computer systems support teams, 19

20 UQ and HPC related mini symposium at USNCCM11, July 25 29, 2011 Computational Methodologies for Uncertainty Quantification and Stochastic PDE based Models for Predictive Simulations (Patra, Ghattas, Prudencio, Sahni, Schulz) Mini symposium number: 13.6 (see Abstracts can be submitted until March 15 th

21 Thank you for your attention! Questions? 21