Preconditioning for large scale µfe analysis of 3D poroelasticity

Transcription

1 ECBA, Graz, July 2 5, /38 Preconditioning for large scale µfe analysis of 3D poroelasticity Peter Arbenz, Cyril Flaig, Erhan Turan ETH Zurich Workshop on Efficient Solvers in Biomedical Applications Graz, July 2 5, 2012

2 CBA, Graz, July 2 5, /38 Outline 1 Introduction 2 Mathematical model 3 Numerics 4 Numerical results 5 Poroelasticity 6 Implementation 7 More Results 8 Summary

3 ECBA, Graz, July 2 5, /38 The need for µfe analysis of bones I Osteoporosis A disease characterized by low bone mass and deterioration of bone microarchitecture (trabecular bone). High lifetime risk for a fracture caused by osteoporosis. In Switzerland, the risk for an osteoporotic fracture in women above 50 years is about 50%, for men the risk is about 20%. Since global parameters like bone density do not admit to predict the fracture risk, patients have to be treated in a more individual way.

4 ECBA, Graz, July 2 5, /38 The need for µfe analysis of bones Osteoporosis A disease characterized by low bone mass and deterioration of bone microarchitecture (trabecular bone). High lifetime risk for a fracture caused by osteoporosis. In Switzerland, the risk for an osteoporotic fracture in women above 50 years is about 50%, for men the risk is about 20%. Since global parameters like bone density do not admit to predict the fracture risk, patients have to be treated in a more individual way. Research µfe analysis improves the understanding of the importance of the structure of the trabecular bone.

5 ECBA, Graz, July 2 5, /38 The need for µfe analysis of bones II Progress New approach consists of combining 3D high-resolution CT scans of individual bones with a micro-finite element (µfe) analysis.

6 Another problem A. Wirth et al.: Mechanical competence of bone-implant systems can accurately be determined by image-based µfe analyses. Arch. Appl. Mech. 80 (2010), ECBA, Graz, July 2 5, /38

7 ECBA, Graz, July 2 5, /38 Work flow pqct: Peripheral Quantitative Computed Tomography

8 CBA, Graz, July 2 5, /38 The mathematical model Equations of linearized 3D elasticity (pure displacement formulation): Find displacement field u [H 1 (Ω)] 3 that minimizes total potential energy Ω [ µε(u) : ε(u) + λ 2 (divu)2 f t u ] dω gs t udγ, Γ N Eν with Lamé s constants λ = (1+ν)(1 2ν), µ = E 2(1+ν), volume forces f, boundary tractions g, symmetric strain tensor ε(u) := 1 2 ( u + ( u)t ). The computational domain Ω consist of identical voxels

9 CBA, Graz, July 2 5, /38 The mathematical model Equations of linearized 3D elasticity (pure displacement formulation): Find displacement field u [H 1 (Ω)] 3 that minimizes total potential energy Ω [ µε(u) : ε(u) + λ 2 (divu)2 f t u ] dω gs t udγ, Γ N Eν with Lamé s constants λ = (1+ν)(1 2ν), µ = E 2(1+ν), volume forces f, boundary tractions g, symmetric strain tensor ε(u) := 1 2 ( u + ( u)t ). The computational domain Ω consist of identical voxels

10 ECBA, Graz, July 2 5, /38 Discretization using voxel elements Finite element approximation: displacements u represented by piecewise trilinear polynomials Each voxel has 8 nodes In each node we have 3 degrees of freedom: displacements in x-, y-, z-direction In total 24 degrees of freedom per voxel strains / stresses computable by means of nodal displacements

11 ECBA, Graz, July 2 5, /38 Solving the system of equations The discretization results in a linear system of equation: Au = f A is sparse and symmetric positive definite. The fine resolution of the CT scan entails that A is HUGE. Approach to solve this linear system: preconditioned conjugate gradient (PCG) algorithm Diagonal (Jacobi) preconditioner (Rietbergen et al., 1996) Avoid the assembling of the stiffness matrix. Compute the matrix-vector multiplication in an element-by-element (EBE) fashion n el A = T e A e Te T, (1) e=1 Multigrid preconditioning (Ruge Stüben) (Adams et al., 2004) Multigrid preconditioning (SA-AMG) (Arbenz et al., 2008)

12 ECBA, Graz, July 2 5, /38 ParFE: PCG with SA-AMG preconditioning Mathematics of ParFE Smoothed aggregation AMG (Vaněk et al., 1996): Smoothed prolongator (P i ) formed from the aggregation of the matrix graph and near-nullspace. Coarser level matrix: K i+1 = Pi T K i P i Smoother: Chebyshev polynomial that is small on the upper part of the spectrum of K i Software of ParFE Trilinos ( Framework for generation of parallel objects and solvers (Heroux et al., 2000) Packages used: AztecOO/Belos, ML, Zoltan ( ParMETIS) ParFE (

13 ECBA, Graz, July 2 5, /38 ParFE: PCG with SA-AMG preconditioning Mathematics of ParFE Smoothed aggregation AMG (Vaněk et al., 1996): Smoothed prolongator (P i ) formed from the aggregation of the matrix graph and near-nullspace. Coarser level matrix: K i+1 = Pi T K i P i Smoother: Chebyshev polynomial that is small on the upper part of the spectrum of K i Software of ParFE Trilinos ( Framework for generation of parallel objects and solvers (Heroux et al., 2000) Packages used: AztecOO/Belos, ML, Zoltan ( ParMETIS) ParFE (

14 Introduction Modelling Numerics Results Poroelasticity Implementation More Results Summary Very large real bone ( dofs) Effective strains with zooms. (Image by Jean Favre, CSCS) ECBA, Graz, July 2 5, /38

15 ECBA, Graz, July 2 5, /38 PCG with matrix-free GMG preconditioning Motivation ParFE does not exploit regular structures except on finest level. To reduce overhead for information on unstructured grids: Use regular grids on all levels. Apply geometric multigrid Stay matrix-free on all levels Two approaches: Embed the bone structure in a cuboid with original regular grid extended and fill void very soft material (Margenov, 2006). Use geometric MG (Flaig et al., LNCS, 2011) The same, but only with bone data: pointer-less octree approach (Flaig et al., PARCO, 2011) Software: C++ plus MPI and a few libraries.

16 ECBA, Graz, July 2 5, /38 Some properties of the algorithm Octree structure, linearized on all levels. EBE: standard element matrices multiplied by voxel-wise bone-volume fraction Space-filling curve for numbering of elements/nodes High cache-hit rates in matvecs Used for partitioning of domain for // processing Equally sized sets of contiguous nodes perfect load balance About 15 mio. dofs per compute core with 1.33 GB memory. SA-AMG needs more than 16 more memory space

17 ECBA, Graz, July 2 5, /38 Weak scalability Weak scalability test with two meshes: c240 is encased in a cube with dofs c320 is encased in a cube with dofs Bigger grids are generated by 3D-mirroring Chebyshev smoother deg 6 Stopping criterion: r k M r 0 M 1 Biggest mesh: dofs

18 ECBA, Graz, July 2 5, /38 Weak scalability ( dofs) Smoother: Cheb. deg. 6, tol Time [s] c320 iterations c240 iterations c320 solving time c240 solving time Iterations Number of Cores Timings are measured on a Cray XT5 at CSCS. The iteration counts vary between 15 and 17. Perfect weak scalability up to 8000 cores.

19 ECBA, Graz, July 2 5, /38 Simulation of bone formation remodeling CT scan of bone specimen exposing voxel structure Bone reacts on forces it has to sustain Adapts structure to lower stresses Increases bone strength and stability Simulation of bone remodeling is a time-dependent process Sequence of slightly changing linear systems Consider only the first system Complicated geometry implies FE analysis with many degrees of freedom (dof)

20 Poroelasticity Linear Elasticity (λ + µ) ( u) + µ 2 u α p + F = 0, (2) λ, µ: Lamé constants; α Biot Willis coefficient Darcy s law k permeability, η dynamic viscosity Mass conservation α ε kk t f = k p. (3) η + S ε p t + f = S f. (4) S ε measure of released fluid volume per unit pressure in the control volume, S f is an external source or sink. ECBA, Graz, July 2 5, /38

21 CBA, Graz, July 2 5, /38 Weak formulation Ω [2µε(u):ε(v) + λ div u div v] dω α p div v dω = 0 Ω K 1 f g dω p div g dω = 0 Ω Ω α q div u dω q div f dω S ɛ p qdω = Ω Ω Ω Here, F = 0, u (H 1 (Ω)) 3, f H(div; Ω), p L 2 (Ω) (+bc) Boundary conditions: u(x, t) = u D, on Ω D, σ(x, t) n(x) = t(x, t), on Ω N, f(x, t) n(x) = 0, on Ω. Ω S f q dω

22 ECBA, Graz, July 2 5, /38 Finite element displacement/flux/pressure formulation FE Spaces: Q 1 for u, RT 0 for f, P 0 for p. Nodal, facial and elemental unknowns. The resulting system is symmetric indefinite. Alternative (u/p) formulation has fewer degrees of freedom, but denser matrices. (u/f /p) formulation: Stable. Primary variables are kept in the system: Stokes flow. Constant flux across boundaries: continuity.

23 ECBA, Graz, July 2 5, /38 Displacement/flux/pressure formulation Ω [2µε(u):ε(v) + λ div u div v] dω α p div vdω = 0 Ω K 1 f gdω p div gdω = 0 Ω Ω α q div udω q div fdω S ɛ p qdω = Ω Ω Ω Ω S f qdω A uu 0 A T pu u 0 0 A ff A T pf f = 0 A pu A pf A pp p b }{{} A

24 Introduction Modelling Numerics Results Poroelasticity Implementation More Results Summary Displacement/flux/pressure sparsity of A ECBA, Graz, July 2 5, nz = /38

25 ECBA, Graz, July 2 5, /38 General approach Use (flexible) GMRES, MINRES as solver for indefinite symmetric linear systems. Preconditioner with same block structure. Time dependency: first order implicit Euler (FD) Software: Exploit Trilinos framework, packages: Belos, ML, Amesos

26 ECBA, Graz, July 2 5, /38 Distribution of 3-component vector x u f p

27 ECBA, Graz, July 2 5, /38 Block-diagonal preconditioner Block diagonal preconditioner A uu 0 0 M 1 = 0 A ff S where M 1 not feasible. S = A pp A pu A 1 uu A T pu A pf A 1 ff AT pf Approximate S by Ŝ = A pp A pf diag (A ff ) 1 A T pf S and Ŝ are spectrally equivalent (Lipnikov, 2002) Ŝ can be generated as finite difference approximation of c 1 p + c 2 p of the pressure variables (at voxel centers).

28 ECBA, Graz, July 2 5, /38 Block-diagonal preconditioner (con t) Replace A uu by AMG V-cycle (as implemented in Trilinos ML) Replace A uu by its diagonal diag(a ff ) Replace Ŝ by AMG V-cycle A uu diag(a ff ) Ŝ A uu A ff Ŝ

29 ECBA, Graz, July 2 5, /38 Block upper-triangular preconditioner Also incorporate upper off-diagonal blocks: A uu 0 A T pu M 2 = 0 A ff A T pf 0 0 S GMRES with this preconditioner converges in 2 iteration steps. A M 1 2 = I I A pu A 1 uu A pf A 1 ff I

30 ECBA, Graz, July 2 5, /38 Block upper-triangular preconditioner Also incorporate upper off-diagonal blocks: A uu 0 A T pu M 2 = 0 A ff A T pf 0 0 S GMRES with this preconditioner converges in 2 iteration steps. M 2 not feasible. Solve diagonal block systems with preconditioners as in block-diagonal preconditioner: A uu : (PCG with) AMG preconditioner ML A ff : direct solve (no fill-in) Ŝ S: (PCG with) AMG preconditioner ML Accuracy for inner solves?

31 ECBA, Graz, July 2 5, /38 Test problem I: Benchmark problem Terzaghi s 1D Consolidation (Transient analytical) Modeled with 3D geometries using proper boundary conditions up to time steps FGMRES Solution shows stable temporal behavior. Only first few iteration steps are hard to solve Later iteration steps are much easier to solve due to good initial guess

32 Introduction Modelling Numerics Results Poroelasticity Implementation More Results Summary Test problem II: full domain mesh id b1 1 b1 2 b1 4 b1 8 b1 16 b1 32 b1 64 b1 128 ECBA, Graz, July 2 5, 2012 elements nodes faces total dof /38

33 ECBA, Graz, July 2 5, /38 Number of iterations Solve Ax = y, outer tolerance 10 5 Preconditioner A: Block diagonal, V-cycle for A uu and Ŝ Preconditioner B: Block upper-triangular, AMG-preconditioned PCG, inner tolerance 10 8 for A uu, 10 5 for Ŝ, outer iterations Prec. A Prec. B b_1 b_2 b_4 b_8 b_16 b_32 b_64 b_128 model id

34 Strong scalability b_32 A b_32 B b_64 A b_64 B linear 10 2 belos time (s) number of cores GMRES is faster with block-diagonal preconditioner than with block-triangular, although iteration count of the latter is smaller. The latter is too time-consuming. ECBA, Graz, July 2 5, /38

35 ECBA, Graz, July 2 5, /38 More preconditioners diagonal-i: block-diagonal as before. diagonal-ii: block-diagonal with PCG(10 5 ). triangular-i: block-triangular as before (PCG(10 8 )). triangular-ii: block-triangular with PCG(10 5 ). triangular-iii: block-triangular with V-cycle.

36 ECBA, Graz, July 2 5, /38 More preconditioners Comparison # cores diag-i diag-ii triang-i triang-ii triang-iii Table 2: Comparison of the preconditioners on b1 64 run time (sec.) # cores diag-i diag-ii triang-i triang-ii triang-iii Table 3: Comparison of the preconditioners on b1 64 iterations

37 Introduction Modelling Numerics Results Poroelasticity Implementation More Results Summary Test problem III: bone geometries ECBA, Graz, July 2 5, /38

38 ECBA, Graz, July 2 5, /38 Test Models mesh id elements nodes faces total dof c c c Table 4: Test meshes on a bone sample

39 ECBA, Graz, July 2 5, /38 Bone geometries Comparison preconditioner c 040 c 080 c 160 diagonal-i triangular-i triangular-ii triangular-iii Table 5: Number of outer iterations preconditioner c 040 c 080 c 160 diagonal-i triangular-i triangular-ii triangular-iii Table 6: Belos time

40 ECBA, Graz, July 2 5, /38 Summary Plain elasticity solver ParFE extendend to include poroelastic effects. Trilinos/Belos (FGMRES) is the main solver. Mixed finite element formulation is implemented into our elasticity solver ParFE PorFE. More investigations/improvements regarding preconditioners. Node-face-element-aware partitioning to be improved ParMetis.