STCE. An (more) effective Discrete Adjoint Model for OpenFOAM

Transcription

1 An (more) effective Discrete Adjoint Model for OpenFOAM Markus Towara, Uwe Naumann Software and Tools for Computational Engineering Science RWTH Aachen University EuroAD 2013, Oxford, 10. December 2013

2 A. Sen (aboutflow Project, discrete OpenFOAM) A. Daztouri (aboutflow Project, discrete ACE+) M. Foerster (fortissifoam Project, OpenFOAM in the cloud) me (discrete OpenFOAM) RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

3 Outline OpenFOAM Introduction Discrete OpenFOAM Improvements Verification and Results Work in Progress - PISO Foam RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

4 Outline OpenFOAM Introduction Discrete OpenFOAM Improvements Verification and Results Work in Progress - PISO Foam RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

5 OpenFOAM Open Field Operation and Manipulation Open-Source (GPLv3) CFD solver developed by OpenCFD Ltd., currently at version 2.2.x > 1 Mio. Lines of Code includes tools for meshing, pre-, post-processing rising adoption in industry and academia due to lack of licence costs well suited for parallel architectures RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

6 CFD Basics Navier Stokes equations for incompressible steady flow: v v = ν 2 v 1 ρ p v = 0 momentum conservation mass conservation Or in three Dimensions: vx vx vx vx + vy + vz = p x y z x vy vy vy vx + vy + vz = p x y z y vz vz vz vx + vy + vz = p x y z z u x + v + w = 0 y z + ν 2 vx x 2 + ν 2 vy x 2 + ν 2 vz x vx y vy y vz y vx z vy z vz z 2 RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

7 CFD Basics (Decoupled) Solution of the partial differential equations (SIMPLE-Algorithm): discretize / linearize momentum conservation equations solve momentum equations for velocity v, assume pressure p as known obtained velocity field fulfills momentum equation but not mass conservation equation as the pressure field was guessed and not correct discretize mass conservation equation use mass conservation equation to correct pressure field use new pressure field to correct velocity field Loop... RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

8 Topology Optimization no optimization added material reconstruction Add penalty term α to Navier-Stokes equation 1 : (v ) v = ν 2 v p αv 1 C. Othmer: A continuous adjoint formulation for the computation of topological and surface sensitivities of ducted flows. Intern. J. f. Num. Meth. in Fluids. p , RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

9 How to find appropriate α Define Cost Function J, e.g. total pressure loss between inlet and outlet: J = p v2 n dγ Γ Calculate sensitivity of the Cost function w.r.t. parameters α i J α i =??? Calculate updated porosity field α n+1, e.g.: α n+1 i = αi n λ J n, while insuring 0 < α i < α max Loop until α converged... α n i RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

10 Outline OpenFOAM Introduction Discrete OpenFOAM Improvements Verification and Results Work in Progress - PISO Foam RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

11 Motivation for AD in Context of Shape Optimization AD promises: greater flexibility w.r.t new objectives easy adaption to new solver types / generations calculated derivatives are exact w.r.t the used discretization higher order derivatives available Problems: memory requirements how to retain parallelism? RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

12 How to implement n m, so Adjoint Mode is method of choice source code transformation operator overloading due to complexity of OpenFOAM code (templates, massively object oriented) the only suitable option operator overloading tool dco/c++ every floating point value in OpenFOAM is replaced with custom adjoint datatype dco::a1s::type needed intermediate values are stored in a structure called tape RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

13 How to implement put dco into src/openfoam include dco.hpp replace doubles with active datatype from dco OpenFOAM has own typedef for scalar floating point values just one substitution in theory we now just need to recompile OpenFOAM and are ready to go RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

14 OpenFOAM a1s mode in src/openfoam/primitives/scalar/doublescalar/doublescalar.h: replace: namespace Foam { } with: typedef double doublescalar;... #include dco.hpp namespace Foam { } typedef dco::a1s::type doublescalar;... RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

15 OpenFOAM t1s mode in src/openfoam/primitives/scalar/doublescalar/doublescalar.h: replace: namespace Foam { } with: typedef double doublescalar;... #include dco.hpp namespace Foam { } typedef dco::t1s::type doublescalar;... RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

16 in Reality some minor changes have to be made in the OpenFOAM code: unions don t support active datatypes some casts are missing and have to be done by hand some functions (pow,max,min) have to be adapted RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

17 usage of discrete OpenFOAM - t1s Black-Box tangent-linear Version of simplefoam, calculates J α i : double sens = 0; dco::t1s::set(alpha[i],1,1); for (runtime++;!runtime.end(); runtime++) {... // solve for U,p } // Sum pressure over inlet faces scaled with face area doublescalar J = gsum( p.boundaryfield() patch.magsf() ); dco::t1s::get(j,sens,1); Need to do this N-times to get full sensitivity field! RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

18 usage of discrete OpenFOAM - a1s Black-Box adjoint Version of simplefoam, calculates gradient of J: dco::a1s::static tape tape(tapesize); double sens = new double[alpha.size()]; for(int i=0; i<alpha.size(); i++) tape.register variable(alpha[i]); for (runtime++;!runtime.end(); runtime++) {... // solve for U,p } // Sum pressure over inlet faces scaled with face area doublescalar J = gsum( p.boundaryfield() patch.magsf() ); dco::a1s::set(j,1, 1); tape.interpret adjoint(); for(int i = 0; i<alpha.size(); i++) dco::a1s::get(alpha[i],sens[i], 1); RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

19 Outline OpenFOAM Introduction Discrete OpenFOAM Improvements Verification and Results Work in Progress - PISO Foam RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

20 First Results Black-Box approach severely bound by memory bandwith! (Need to allocate around 20 GB) By doing Checkpointing we can actually get faster... RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

21 How to improve? Knowledge and Profiling reveals that most of calculation time and tape memory is spent in (iterative) linear solvers Analytical insight allows us to treat the adjoints of linear solvers analytically Lemma For a Linear Equation Sytem Ax = b we can calculate the adjoints for A and b by: 2 b = A T x A T b = x Ā = b x T This gives us an additional Linear Equation System which we have to solve during the gathering of the adjoints 2 M. Giles, Collected Matrix Derivative Results for Forward and Reverse Mode Algorithmic Differentiation, Advances in Automatic Differentiation 2008 RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

22 Treatment of Linear Solvers Black-Box Continuous Linear Solver when we encounter an linear solver during the augmented forward run we can stop taping when we encounter the gap in the tape during the interpretation we have to fill in the gap by hand have to remember A and x! RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

23 Application to OpenFOAM Abstract Base Class for all Linear Solvers exists can tackle all implemented solvers at once Discretization matrices are stored in a coordinate sparse matrix format transposing matrix straightforward take care not to mess up symmetry of matrices RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

24 Outline OpenFOAM Introduction Discrete OpenFOAM Improvements Verification and Results Work in Progress - PISO Foam RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

25 Results we are seeing a nice improvement in both runtime and memory usage RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

26 Verification We calculate a reference solution with solvers GAMG for U and smoothsolver for p, both with tolerance 1e 10 After that we calculate solutions with different tolerances for the forward and adjoint linear solvers and compare to the reference solution 11k Cells in mesh Velocity Sensitivity RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

27 Verification Runtime Tape-Memory RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

28 Error RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

29 Runtime RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

30 Outline OpenFOAM Introduction Discrete OpenFOAM Improvements Verification and Results Work in Progress - PISO Foam RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

31 Work in Progress - Unsteady Adjoints Unsteady PISO solver with unsteady cost function (potentially depends on every timestep) RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

32 Work in Progress - Unsteady Adjoints RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

33 Work in Progress - Unsteady Adjoints RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

34 Summary and Outlook Fully discrete mode uses loads of memory Treatment of linear solvers greatly reduces this Its also faster Application to unsteady flow, heat transfer, turbulence in progress RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35

35 Thank you! Questions? RWTH-Aachen, Discrete OpenFOAM, EuroAD / 35