Approaches to Parallel Implementation of the BDDC Method

Transcription

1 Approaches to Parallel Implementation of the BDDC Method Jakub Šístek Includes joint work with P. Burda, M. Čertíková, J. Mandel, J. Novotný, B. Sousedík. Institute of Mathematics of the AS CR, Prague Czech Technical University, Prague University of Colorado Denver September 25, 2009, Tampere University of Technology INSTITUTE of MATHEMATICS Academy of Sciences Czech Republic

2 Outline Some motivation for domain decomposition BDDC method Implementation by global matrices and generalized change of variables Implementation subdomain by subdomain with explicit coarse problem Numerical results Conclusion

3 Some motivation for domain decomposition (DD) methods very large systems of algebraic equations arising from FEM difficulties with solution by conventional numerical methods direct methods slow due to the problem size iterative methods slow due to large condition number suitable preconditioner needed combination of these approaches synergy in domain decomposition (DD) methods natural way to parallelize FEM two recent methods for elliptic PDEs BDDC and FETI-DP

4 Outline Some motivation for domain decomposition BDDC method Implementation by global matrices and generalized change of variables Implementation subdomain by subdomain with explicit coarse problem Numerical results Conclusion

5 Brief overview of BDDC method Balancing Domain Decomposition by Constraints 2003 C. Dohrmann (Sandia), theory with J. Mandel (UCD) nonoverlapping primary domain decomposition method equivalent to FETI-DP [Mandel, Dohrmann, Tezaur 2005]

6 The abstract problem Variational setting u U : a(u, v) = f, v v U a (, ) symmetric positive definite form on U, is inner product on U U is finite dimensional space Matrix form u U : Au = f A symmetric positive definite matrix on U Linked together Au, v = a (u, v) u, v U

7 BDDC set-up division into subdomains selection of coarse problem nodes (also called corners) interface subdomain Ωi coarse problem nodes h H finite elements

8 Function spaces in BDDC U W c W continuous continuous at coarse no continuity problem nodes enough coarse nodes to fix floating subdomains rigid body modes captured a (, ) symmetric positive definite form on W c corresponding matrix Ãc symmetric positive definite, almost block diagonal structure, larger dimension than A operator of projection E : W c U, Range(E) = U, e.g. averaging across interfaces (arithmetic, weighted)

9 Fictitious mesh à c can be constructed using auxiliary mesh

10 The second intermediate space in BDDC Only coarse nodes do not suffice for optimal preconditioning in 3D additional constraints on functions from W c necessary U W W c continuous add constraints only corners Examples: equivalent averages on subsets of interface (edges, faces) across interface, additional pointwise continuity constraints

11 The BDDC preconditioner Define M BDDC : r U û U M BDDC : r û = Ew, w W : a (w, z) = r, Ez, z W r - residual in an iteration of PCG û - correction to solution (preconditioned residual)

12 Outline Some motivation for domain decomposition BDDC method Implementation by global matrices and generalized change of variables Implementation subdomain by subdomain with explicit coarse problem Numerical results Conclusion

13 Enforcing additional constraints Introduce matrix G with constraints each row of G corresponds to a continuity constraint between two subdomains introduces new coupling between subdomains Example: for arithmetic averages on an edge between subdomains i and j, a row of G is g k = [ } 1 1{{ 1 1} } 1 1 {{ 1 1 } ] edge dof on Ω i edge dof on Ω j define intermediate space as { W = w W } c : Gw = 0

14 The BDDC preconditioner with averages variational form M BDDC : r û = Ew, w W : a (w, z) = r, Ez, z W matrix form à c w + G T λ = E T r Gw = 0 û = M BDDC r = Ew r - residual in an iteration of PCG û - correction to solution (preconditioned residual)

15 Change of variables Change of variables on each subdomain, such that averages appear as single node constraints. w = Tw, w = Bw, B = T 1 Matrix T invertible, contains weights of averages. Compute M BDDC r = EBw, where w is the solution to B T Ã c Bw + B T G T λ = B T E T r GBw = 0. Transformed averages may be handled as corners and further assembled [Li, Widlund 2006]. Drawback: The distinction between W c and W lost.

16 Projected change of variables Combination of projected BDDC and change of variables. Define matrix G = GB reduces to one 1 and one 1 in each row. Projection onto null(g) Construct matrix BDDC preconditioner as P = I G T (GG T ) 1 G Ã = PB T Ã c BP + t(i P) Ãw = PB T E T r û = M BDDC r = EBw

17 Outline Some motivation for domain decomposition BDDC method Implementation by global matrices and generalized change of variables Implementation subdomain by subdomain with explicit coarse problem Numerical results Conclusion

18 The coarse space in BDDC In implementation, space W may be split into independent subdomain spaces and energy-orthogonal coarse space. On each subdomain coarse degrees of freedom basis functions Ψ i prescribed values of coarse degrees of freedom, minimal energy elsewhere, [ Ai C T i C i 0 ] [ Ψi Λ i ] = A i... local subdomain stiffness matrix [ 0 I C i... matrix of constraints selects unknowns into coarse degrees of freedom Matrix of coarse problem A C assembled from local matrices A Ci = Ψ T i A i Ψ i. ].

19 The coarse space in BDDC a function from coarse space coarse basis function

20 An iteration of BDDC 1. Residual on interface r (k) = g Sû (k) S Schur complement w.r.t. interface, g condensed r.h.s. 2. Distribution of residual local problems coarse problem for i = 1,..., N N r i = Ei T r (k) r C = RCi T ΨT i Ei T r (k) 3. Correction of solution i=1 [ Ai Ci T ] [ ] [ ] ui ri = C i 0 µ i 0 A C u C = r C 4. New approximation û = N E i (Ψ i R Ci u C + u i ), û (k+1) = û (k) + û i=1

21 Outline Some motivation for domain decomposition BDDC method Implementation by global matrices and generalized change of variables Implementation subdomain by subdomain with explicit coarse problem Numerical results Conclusion

22 Parallel implementation global matrices and generalized change of variables built on top of multifrontal solver MUMPS Fortran 90 programming language, MPI library subdomain by subdomain with explicit coarse problem built on top of frontal solver (subdomains) Fortran programming language, MPI library coarse problem by multifrontal solver MUMPS Tested on SGI Altix 4700, CTU, Prague, CR 72 processors Intel Itanium 2, OS Linux.

23 Steam turbine nozzle quadratic elements, unknowns 16 subdomains by METIS, 77 corners, 14 edges, and 30 faces

24 Comparison of implementations steam turbine nozzle 16 subdomains, 16 processors of SGI Altix 4700 implementation MUMPS frontal solver constraints C C+E+F C C+E+F iterations cond. number est transforming matrix (sec) analysis (sec) factorization (sec) PCG iter (sec) total (sec) C... Corners, E... averages on Edges, F... averages on Faces

25 Hip joint replacement quadratic elements, unknowns 16 subdomains by METIS, 101 corners, 12 edges, and 35 faces

26 Comparison of implementations hip joint replacement 16 processors of SGI Altix 4700 implementation MUMPS frontal solver constraints C C+E+F C C+E+F iterations cond. number est transforming matrix (sec) analysis (sec) factorization (sec) PCG iter (sec) total (sec) C... Corners, E... averages on Edges, F... averages on Faces

27 Outline Some motivation for domain decomposition BDDC method Implementation by global matrices and generalized change of variables Implementation subdomain by subdomain with explicit coarse problem Numerical results Conclusion

28 Conclusion global matrices and generalized change of variables allows simple implementation parallelism by parallel direct solver bounded by problem size subdomain by subdomain with explicit coarse problem less straightforward (explicit coarse problem, mapping,... ) allows different handling of global coarse problem (MUMPS on coarse, multilevel extensions (ongoing research),... ) common distinguish between point constraints and averages for implementation constraits on edges and/or faces can considerably shorten the solution time averages on faces might be too expensive more sophisticated way for selection of weighted averages adaptive BDDC (ongoing research)