A Parallel Implementation of the BDDC Method for Linear Elasticity

Transcription

1 A Parallel Implementation of the BDDC Method for Linear Elasticity Jakub Šístek joint work with P. Burda, M. Čertíková, J. Mandel, J. Novotný, B. Sousedík Institute of Mathematics of the AS CR, Prague Czech Technical University, Prague University of Colorado Denver MAFELAP 2009, June 9-12, 2009 INSTITUTE of MATHEMATICS Academy of Sciences Czech Republic

2 Some motivation for domain decomposition (DD) methods very large systems of algebraic equations arising from FEM difficulties with solution by conventional numerical methods direct methods slow due to the problem size iterative methods slow due to large condition number suitable preconditioner needed combination of these approaches synergy in domain decomposition (DD) methods natural way to parallelize FEM two recent methods for elliptic PDEs BDDC and FETI-DP

3 Brief overview of BDDC method Balancing Domain Decomposition by Constraints 2003 C. Dohrmann (Sandia), theory with J. Mandel (UCD) nonoverlapping primary domain decomposition method equivalent to FETI-DP [Mandel, Dohrmann, Tezaur 2005]

4 The abstract problem Variational setting u U : a(u, v) = f, v v U a (, ) symmetric positive definite form on U, is inner product on U U is finite dimensional space Matrix form u U : Au = f A symmetric positive definite matrix on U Linked together Au, v = a (u, v) u, v U

5 BDDC set-up division into subdomains selection of coarse problem nodes (also called corners) interface subdomain Ωi coarse problem nodes h H finite elements

6 Function spaces in BDDC U W c W continuous continuous at coarse no continuity problem nodes enough coarse nodes to fix floating subdomains rigid body modes captured a (, ) symmetric positive definite form on W c corresponding matrix Ãc symmetric positive definite, almost block diagonal structure, larger dimension than A operator of projection E : W c U, Range(E) = U, e.g. averaging across interfaces (arithmetic, weighted)

7 The second intermediate space in BDDC Only coarse nodes do not suffice for optimal preconditioning in 3D additional constraints on functions from W c necessary U W W c continuous add constraints only corners Examples: equivalent averages on subsets of interface (edges, faces) across interface, additional pointwise continuity constraints

8 The BDDC preconditioner Define M BDDC : r U û U M BDDC : r û = Ew, w W : a (w, z) = r, Ez, z W r - residual in an iteration of PCG û - correction to solution (preconditioned residual)

9 The coarse space in BDDC In implementation, space W may be split into independent subdomain spaces and energy-orthogonal coarse space. On each subdomain coarse degrees of freedom basis functions Ψ i prescribed values of coarse degrees of freedom, minimal energy elsewhere, [ Ai C T i C i 0 ] [ Ψi Λ i ] = A i... local subdomain stiffness matrix [ 0 I C i... matrix of constraints selects unknowns into coarse degrees of freedom Matrix of coarse problem A C assembled from local matrices A Ci = Ψ T i A i Ψ i. ].

10 The coarse space in BDDC a function from coarse space coarse basis function

11 An iteration of BDDC 1. Residual on interface r (k) = g Sû (k) S Schur complement w.r.t. interface, g condensed r.h.s. 2. Distribution of residual local problems coarse problem for i = 1,..., N N r i = Ei T r (k) r C = RCi T ΨT i Ei T r (k) 3. Correction of solution i=1 [ Ai Ci T ] [ ] [ ] ui ri = C i 0 µ i 0 A C u C = r C 4. New approximation û = N E i (Ψ i R Ci u C + u i ), û (k+1) = û (k) + û i=1

12 Frontal solver B. M. Irons, 1970 direct solver for sparse matrices arising in FEM number of flops O(n nfron 2 ), where nfron << n is the front width memory demand nfron 2 if out-of-core element-by-element approach element matrices read from file until whole line is assembled, then immediately eliminated basic scheme block 1 - free variables, block 2 - constrained (also fixed ) variables [ A11 A 12 A 21 A 22 x 2, f 1, f 2 inputs ] [ x1 x 2 ] = [ f1 x 1, Rea 2 (reaction forces) outputs f 2 ] [ + 0 Rea 2 ], (1)

13 General constraints vs. frontal solver on subdomain central idea split matrix C according to types of constraints corners as Dirichlet boundary conditions, i.e. fixed variables averages enforced by Lagrange multipliers matrix C f coarse problem construction on subdomain (index i omitted) A ff A fc C T f A cf A cc 0 C f 0 0 Ψ c f Ψ avg f I 0 λ c λ avg = I.

14 Algorithm of preconditioner setup 1. Forward step of frontal solver with corners marked as fixed variables in matrix A. 2. Find A 1 ff C f T by backward solve by frontal solver [ ] [ Aff A fc A 1 ff C T ] [ ] [ ] f C T = f 0 + ( A cf A cc 0 0 Cf A 1 ff A ) T. fc 3. Construct C f A 1 ff C f T and factorize it by LAPACK. 4. Backward solve of dual problem by LAPACK for λ from C f A 1 ff C f T λ = [ C f A 1 ff A fc I 5. Backward solve for Ψ f by frontal solver [ ] [ Aff A fc Ψ c f Ψ avg ] [ f C T = f λ I 0 0 A cf A cc [ C 6. Compute local A C = Ψ T AΨ = Ψ T T f λ Rea ]. ] [ 0 + Rea ]. ].

15 General constraints vs. frontal solver subdomain problem solution A ff A fc Cf T A cf A cc 0 C f 0 0 u f 0 µ = r 0 0. Now single right-hand side.

16 Algorithm of preconditioning action on subdomain 1. Backward step of frontal solver for A 1 ff r [ ] [ Aff A fc A 1 r ] A cf A cc 2. Backward step of LAPACK for µ C f A 1 ff ff 0 C f T = [ r 0 µ = C f A 1 ff r. ] [ 0 + Rea ]. 3. Backward step of frontal solver for u f [ ] [ ] [ Aff A fc uf C T = f µ + r 0 0 A cf A cc ] [ 0 + Rea ].

17 Implementation subdomain problems - frontal solver + LAPACK coarse problem - MUltifrontal Massively Parallel sparse direct Solver (MUMPS) mainly Fortran 77 programming language, partly Fortran 90, MPI library tested on SGI Altix 4700, CTU, Prague, CR 72 processors Intel Itanium 2, OS Linux

18 Hip joint replacement I quadratic elements, unknowns I 16 subdomains, 35 corners, 12 edges, and 35 faces I 32 subdomains, 57 corners, 12 edges, and 66 faces I 16 processors of SGI Altix 4700 Decomposition into 32 subdomains

19 Hip joint replacement von Mises stresses in improved design

20 Hip joint replacement 100 Condition number SGI Altix processors nsub = 16 nsub = condition number estimate [/] number of corners [/] Condition number for adding corners

21 Hip joint replacement Wall times for variable coarse problem SGI Altix processors nsub = 16 nsub = wall time [seconds] number of corners [/] Wall clock time for adding corners

22 Hip joint replacement, 16 subdomains coarse problem C. C.+E. C.+F. C.+E.+F. iterations cond. number est factorization (sec) pcg iter (sec) total (sec) coarse degrees of freedom: C. - Corners only adding averages to 335 corners C.+E. - Corners and averages on Edges C.+F. - Corners and averages on Faces C.+E.+F. - Corners and averages on Edges and Faces

23 Hip joint replacement, 32 subdomains coarse problem C. C.+E. C.+F. C.+E.+F. iterations cond. number est factorization (sec) pcg iter (sec) total (sec) coarse degrees of freedom: C. - Corners only adding averages to 557 corners C.+E. - Corners and averages on Edges C.+F. - Corners and averages on Faces C.+E.+F. - Corners and averages on Edges and Faces

24 Conclusion distinguish between point constraints and averages for frontal solver many matrices needed in BDDC simple side-product of frontal solver (reactions) minimal number of corners does not assure minimal solution time constraits on edges and/or faces can considerably shorten the solution time more sophisticated (adaptive) way for selection of constraints - ongoing research