Automated Adaptive Finite Element Methods with Applications in Massively Parallel Simulation of Turbulent flow and Fluid-structure Interaction

Transcription

1 Automated Adaptive Finite Element Methods with Applications in Massively Parallel Simulation of Turbulent flow and Fluid-structure Interaction Johan Jansson12 Basque Center for Applied Mathematics (BCAM), Bilbao [1] Computational Technology Laboratory, HPCViz, CSC, KTH, Stockholm [2] (m2 op)iii, BCAM, 24 May 2014

2 Automated High-Performance Scientific Computing (HPSC) FEniCS(-HPC) framework for automated solution of general PDE, give PDE in high-level mathematical notation as input, do Automated discretization (generate linear system from PDE): Automated error control (incl. parallel adaptive mesh refinement): + M(e) TOL with M(e) a functional of the computational error e = u U. Goal: Autom. generate both the solution and program from PDE

3 Turbulent flow and fluid-structure interaction (FSI) Computationa of high and low Reynolds number turbulent flow, also interacting with elastic structures. Open scientific problem. Applications in aerodynamics (aircraft, cars, wind/ocean energy), biomechanics (heart, vocal folds)

4 Automated HPSC Weak form of stationary incompressible Navier-Stokes: r(û, ˆv) = ((u )u + p, v) + (ν u, v) + ( u, q) = 0, ˆv V û = (u, p) (Solution) ˆv = (v, q) (Test function) Stabilized FEM for stationary incompressible Navier-Stokes: r G (Û, ˆv) = ((U )U + P, v) + (ν U, v) + ( U, q) r S (Û, ˆv) = (δr(û), R(ˆv)) (Least-squares stabilization) r(û, ˆv) + rs(û, ˆv) = 0 ˆv V h, U V h ˆv = (v, q) (Test function) r_g = ( grad ( u ) u + inner ( grad ( p ), v ) + \ nu inner ( grad ( u ), grad ( v ) ) + div ( u ) q ) dx r_s = delta inner ( R ( uhat, uhat ), R ( vhat, uhat ) ) dx r = r_g + r_s solve ( r == 0, uhat ) [Jansson, PRACE course notes, 2013]

5 Representation adaptivity - automated adaptive error control We want to adaptively control the error ê = û Û in the functional M(ê) given only the weak residual r and the primal FEM solution Û as input. Adaptive error control directly using the error representation: M(ê) = a(ê, φ) = r(û, φ) = M j=1 r(û, φ) Kj dual problem a(ˆv, φ) = M(ˆv) for the dual solution φ Automated generation (including linearization) of the dual problem: M = q n [ 0 ] ds # Drag ( o n l y p r e s s u r e h e r e ) a_dual = adjoint ( derivative ( r, uhat ) ) L_dual = M solve ( a_dual == L_dual, phi )

6 Representation adaptivity - automated error control With FEM primal and dual in the same space Û, Φ ˆV h we get: r(u, Φ) = 0 (1) Historical approach: work-around by Cauchy-Schwartz (and subtracting an interpolant) to get: hr(û) Φ (2) (other similar approaches also possible, see e.g. Darrigrand) We show that (1) gives an excellent and simple error indicator: E K = r(u, Φ) K (3) We represent (3) as a DG(0) function in space and prove an a priori error estimate for the error indicator: E(U, φ)(x) Ē cg(1) (U, Φ)(x) L2 Ch 2+d (4) E(U, φ)(x) E cg(1) (U, Φ)(x) L2 Ch 1+d (5) [Jansson, Hoffman, Spühler, Degirmenci, preprint 2014]

7 Representation adaptivity - automated error control Test with elliptic model problem comparing between the orthogonal Φ cg(1) (left) and non-orthogonal Φ cg(2) (right) as reference. E 10 K rep E10 K quad T 0 rep T 0 quad

8 Representation adaptivity - automated error control log10(error estimate) log10(eff. index) Representation adaptivity for diffusion, ɛ = 10 1 Error estimate (est) 1 jump 2 jump2 3 rep (P1 dual) 4 quad (P2 dual) log10(number of vertices) Eff. index (est/e) (log10(1)=0 is optimal) jump jump2 rep (P1 dual) quad (P2 dual) log10(number of vertices) log10(error) log10(index) 6 Error (e) jump jump2 rep (P1 dual) rep (P2 dual) uniform log10(number of vertices) (rep) P2 error indicator eff. index rel. diff. abs. diff log10(number of vertices)

9 Representation adaptivity - automated error control Advantages: Generality and simplicity The derivation and source code generation can be completely automated. Simple to understand, teach and implement. Reliability We show that the error indicator converges to the exact error indicator with order 2 + d/1 + d. For the stabilized case we no longer have orthogonality, and the error estimate is also good. Efficiency The method does not require computation of additional constructs such as facet integrals, local problems or modified finite element spaces which can be costly, especially in a parallel framework.

10 General Galerkin (G2) method / Direct FEM Developed over a 20+ year period by Johnson, Hoffman, Jansson, etc. Space-time FEM with Galerkin/least squares stabilization (R(Û), ˆv) + (δr(û), R(ˆv)) = 0, δ = h, ˆv ˆV h, Û ˆV h Adaptive error control and mesh refinement M(ê) = (R(Û, φ) hr(û) K ˆφ K TOL K T Slip/friction boundary condition as boundary layer model u n = 0 Implicit parameter-free turbulence model based on stabilization Dissipation: D = δ 1/2 R(Û) 2 (similar approach to ILES, VMS) Moving mesh, fluid-structure interaction, shock-capturing, etc.

11 Time-resolved adaptive simulation of aircraft HiLiftPW-2 NASA/Boeing-organized benchmark challenge. Participated with good results, and our adaptive methodology was highlighted in summary. Published paper in proceedings of AIAA SciTech Working on updated paper for AIAA JOA. Our computational results capture both pre-stall and stall well quantitatively (aoa=22.4 (pre-stall) and aoa=24 (stall) in right movie above) at Re Collectively, how well is CFD capturing the stall breaks?, Chris Rumsey, NASA, HiLiftPW-2 organizer

12 Flight simulation: our results Unicorn/FEniCS-HPC (our) results from NASA/Boeing HiLiftPW-2 workshop for full aircraft: CL C L and C D vs. angle of attack num. cfg4 num. cfg5 exp. cfg5 CD angle of attack num. cfg4 num. cfg5 exp. cfg angle of attack [Hoffman, Jansson, Jansson, Vilela, Proc. AIAA 2014]

13 Adaptive mesh refinement - adjoint velocity Goal quantity: drag and lift for right half of airplane

14 Adaptive mesh refinement - mesh sequence mesh0: 700k vertices, note upstream refinement Mesh on surface Mesh slice on target side

15 Adaptive mesh refinement - mesh sequence mesh1: 1.1M vertices, note upstream refinement Mesh on surface Mesh slice on target side

16 Adaptive mesh refinement - mesh sequence mesh2: 1.6M vertices, note upstream refinement Mesh on surface Mesh slice on target side

17 Adaptive mesh refinement - mesh sequence mesh3: 2.2M vertices, note upstream refinement Mesh on surface Mesh slice on target side

18 Adaptive mesh refinement - mesh sequence mesh4: 3.1M vertices, note upstream refinement Mesh on surface Mesh slice on target side

19 Aerodynamic forces α = 12 Lift within 5% of exp., Drag within 0.1% of exp. Use 1536 cores on Lindgren supercomputer, ca. 300k core hours total Would like a few more adaptive iterations C_l degrees angle of attack Unicorn adaptive mesh refinement experiment No. mesh points Unicorn adaptive mesh refinement experiment C_d No. mesh points

20 Adaptive mesh refinement - adjoint velocity Goal quantity: drag and lift for right half of airplane. Left (non-target) side Right (target) side

21 Adaptive mesh refinement - adjoint velocity Residual, recall: M(ê) = ( R(Û), ˆφ)

22 FEniCS-HPC components FEniCS: fenicsproject.org Open source software project for automated solution of PDE with many involved universities/individuals. (see fenicsproject.org for developer info/papers). FEniCS-HPC: branch with focus on good scaling on massively parallel architectures, targeting open/challenging problems in turbulent flow/fsi. Developed mainly by CTL group. Automated generation of finite elements/basis functions (FIAT) e = (K, P, N ) Automated evaluation of variational forms on one cell based on code generation (FFC+UFL) A K = a K (v, U) = v Udx = r(v, U)dx K K Automated assembly of discrete systems on a mesh T Ω (DOLFIN-HPC) A = 0 for all elements K T Ω A += A K Automated turbulent Unified Continuum modeling (Unicorn) r UC (W, v) = (ρ( tu + (u )u) + σ g, v) + SD(W, v) = 0

23 FEniCS-HPC FEniCS-HPC tool chain: FIAT FFC DOLFIN(-HPC) Unicorn Fully distributed unstructured mesh, parallel I/O with MPI Hybrid parallel model MPI-OpenMP/MPI-PGAS LA back-end: PETSc, Janpack (MPI+PGAS) A posteriori error estimation based on adjoint problems Parallel adaptive mesh refinement, ALE moving mesh Turbulent flow, fluid-structure interaction, contact models Installed at supercomputing centers in Sweden, Spain, Germany,... Available for download at: fenicsproject.org or at CTL Forge [ Our research on FEniCS-HPC was awarded a PRACE Tier-0 grant for 20M core hours (ca. 2M EUR) in Florian Rathgeber will present another HPC branch of FEniCS: Firedrake, targeting more architectures in a clever way.

24 Strong scaling verification Strong linear scaling of FEniCS-HPC full adaptive solve of incompressible flow up to at least ca cores on Lindgren (PDC) supercomputer. execution time (seconds) ideal vcc fsi cores [Hoffman, Jansson, et. al., 2012 C&F], [Hoffman, Jansson, Jansson, 2011, SISC]

25 Unified Continuum (UC) formulation for FSI Conservation equations (momentum, mass) in Euler (laboratory) coordinates, stress σ as data, track discontinuous phase function θ with moving mesh/basis functions: ρ( t u + (u )u) σ u t θ + (u )θ u(, 0) = = = = u0 in in in in Q, Q, Q, Ω, Different constitutive equations for phases: σ = σ pi Dt σ s = 2µs + uσ s + σ s u > σ = θσ f + (1 θ)σ s σ f = 2µf Discretize as one domain (exploited in error estimation, robustness):

26 EUNISON: modeling of human voice Assess feasibility of complete model of voice production by a Unified Continuum fluid-structure simulation of the vocal fold mechanics in Unicorn/FEniCS (our software framework). Reynolds number up to : turbulent/transition flow, contact EU FP7 3 year project KTH (Stockholm), Gipsa-Lab (Grenoble), FAU (Erlangen), UPC (Barcelona), lasalle (Barcelona). Formulate the vocal fold FSI system in the Unified Continuum model and FEniCS-HPC framework. Started comparing against [Becker, et. al. JASA 2009] model. [Jansson, et. al., ICA2013]

27 EUNISON: current work General parallel contact detection by computation of distance function based on solving an Eikonal equation Active work on adaptive error control for FSI.

28 Computational Technology research group Adaptive general FEM/software for open/challenging problems in turbulence/fsi Research and software development of Unicorn/FEniCS, international team Johan Jansson - Asst. Prof. (BCAM+KTH) Cristina Pocci - Post doc (BCAM) Josu Izarra - MSc+Intern (BCAM) Sebastian Banzas Baro - MSc+Intern (BCAM) Johan Hoffman - Professor Aurelien Larcher - Post doc Bärbel Janssen - Post doc Rodrigo Vilela de Abreu - Phd student Cem Degirmenci - Phd student Jeannette Spühler - Phd student Kaspar Müller - Phd student + many collaborators bcamath.org/en/research/lines/cfdct ctl.csc.kth.se fenicsproject.org BCAM (Bilbao) KTH (Stockholm) I m recruiting 1 Post-doc (BCAM) and 1-2 PhD students (BCAM+KTH)

29 Summary/Conclusions Overall summary: Our research is a synthesis between Scientific Computing (Mathematics) and HPC (Computer Science) - HPSC Automated HPSC - automated discretization, error control, modeling (turbulence, FSI), from mathematical notation of PDE as input allows researchers to work on abstract level for quick development and formal correctness Advanced grand challenge applications using HPC resources with optimal strong scaling up to at least 5000 cores: full aircraft simulation with novel results (adaptivity, stall), vocal folds in the EUNISON project. Direct FEM enables prediction of turbulent flow cheaply, order of magnitude coarser meshes than RANS/LES and capturing more phenomena (stall, time-resolved). Automated error control: representation adaptivity directly using the error representation - no need for additional estimates, can be fully automated. Good a priori convergence to exact error indicator, good benchmark results.

30 G2 validation: basic benchmarks NACA0012 Re = 10 6 [Jansson/Hoffman/Jansson, 2012] Cube Re = [Hoffman/Jansson/Vilela, CMAME, 2011] Circular cylinder Re = 3900 [Hoffman, IJNMF, 2009] Sphere Re = [Hoffman, JFM, 2006] Square cylinder Re = [Hoffman, SISC, 2005] Surface mounted cube Re = [Hoffman, SISC 2005]

31 Adaptive mesh refinement - adjoint velocity Goal quantity: drag and lift for right half of airplane

32 Adaptive mesh refinement - adjoint velocity Goal quantity: drag and lift for right half of airplane. Left (non-target) side Right (target) side

33 Adaptive mesh refinement - adjoint velocity Residual, recall: M(ê) = ( R(Û), ˆφ)

34 Aerodynamic forces α = 22.4 Lift within 4% of exp., Drag within 13% of exp. Use 1536 cores on Lindgren supercomputer, ca. 300k core hours total Would like a few more adaptive iterations C_l C_d degrees angle of attack Unicorn adaptive mesh refinement experiment No. mesh points Unicorn adaptive mesh refinement experiment No. mesh points

35 Pressure distributions 8 6 cp for HiLiftPW-2 case 2b alpha= eta= geometry (scaled+translated) cp num cp exp geometry exp (scaled+translated) 4 cp x coordinate (m)

36 Pressure distributions 8 6 cp for HiLiftPW-2 case 2b alpha= eta= geometry (scaled+translated) cp num cp exp geometry exp (scaled+translated) 4 cp x coordinate (m)

37 Automated HPSC The stationary incompressible Navier-Stokes equations: { R(û) = (u )u + p ν u = 0 u = 0 û = (u, p) Weak form of stationary incompressible Navier-Stokes: r(û, ˆv) = ((u )u + p, v) + (ν u, v) + ( u, q) = 0 ˆv = (v, q) (Test function)

38 EUNISON project - research driver EUNISON EU project: Extensive UNIfied-domain SimulatiON of the human voice - aim to simulate human voice by fundamental continuum mechanics - started in March 2013, 5 universities in the EU

39 Flight simulation: other group s results EXA results from HiLiftPW-2 workshop for full aircraft: We capture stall phenomenon and uses/develops adaptivity.

40 FEniCS-HPC assembly algorithm A = 0 for all elements K T Ω A += A K The element matrix A K is defined by A K ij = a K ( ˆφ i, φ j ) for all local basis functions ˆφ i and φ j on K Cell based decomposition Dual graph based decomposition Overlap represented as ghosted entities Low data dependency during FE assembly

41 Turbulence modeling Reynolds averaged Navier-Stokes equations (RANS): Compute statistical ensemble average of NSE solutions Result in Reynolds stresses that have to be modeled (turbulent eddy viscosity) closure problem. Large eddy simulation (LES): Compute only largest scales of the flow, model subgrid scales by applying filter Filter width typically connected to mesh size Direct FEM (our approach): No RANS/LES filtering/averaging. No explicit turbulence model/subgrid model (parameter-free), turbulent dissipation only from numerical stabilization.