ETNA Kent State University

Transcription

1 Electronic Transactions on Numerical Analysis. Volume 6, pp , December 997. Copyright 997,. ISSN ETNA FAST SOLUTION OF MSC/NASTRAN SPARSE MATRIX PROBLEMS USING A MULTILEVEL APPROACH C.-A. THOLE,S.MAYER, AND A. SUPALOV Abstract. As part of the European Esprit project EUROPORT, 38 commercial an inustrial simulation coes were parallelize for istribute memory architectures. During the project, sparse matrix solvers turne out to be a major obstacle for high scalability of the parallel version of several coes. The European Commission therefore launche the PARASOL project to evelop fast parallel irect solvers an to test parallel iterative solvers on their applicability an robustness in an inustrial framework. This paper presents initial results using a special multilevel metho as preconitioner for matrices resulting from MSC/NASTRAN linear static analysis of soli structures. P-elements in MSC/NASTRAN allow the polynomial egree of the base functions to be specifie either globally or for each element. Solution epenent aaptive refinement of the p-level can be selecte. Discretisations with lower p-level can therefore be use as coarser gris for a multilevel metho. Tests have been performe using such a metho as preconitioner for a regular cube an a complicate inustrial part, which were moelle by tetraherons an hexagonal elements. Preliminary performance comparisons on small test cases (about 0000 egrees of freeom inicate that the multilevel approach is at least as fast as the currently available fastest iterative MSC/NASTRAN solver. Substantial performance improvements are expecte for full-size inustrial problems. Key wors. finite elements, multigri methos, parallel computation. AMS subject classifications. 65N30, 65N55, 65Y05.. Introuction... Parallel computing nees iterative solvers. The significant amount of work require to move large commercial application programs to parallel architectures in a portable way has hinere the acceptance of parallel technology in inustry, thereby preventing this technology from being fully utilise to increase competitiveness. Therefore, the European Commission supporte inustry in Europe through the ESPRIT initiative EUROPORT by partially funing the porting of 38 inustrially relevant coes (incluing PAM-CRASH, MSC/NASTRAN, POLYFLOW, STAR-CD, an others to parallel computers []. EUROPORT aopte the message passing paraigm for parallelization that, clearly, requires some programming effort. However, it provies the only way to obtain portability across virtually all HPC platforms incluing those with share memory an clusters of workstations. Figure. shows the performance of the parallel CFD-coe POLYFLOW using two ifferent solvers [0]. POLYFLOW is a fully implicit special purpose CFD coe for strongly non-linear problems involving rheologically complex liqui flows (plastic in inustrial processing applications. The coat hangar ie test case simulates the prouction of thin foils, a flui flow problem involving in each time step the solution of a sparse matrix problem with egrees of freeom (ofs. Figure. shows that the irect solver actually slows own, if more than 6 processors are use. An iterative solver base on omain ecomposition is even faster an more than 6 noes can be use efficiently. Receive May 7, 997. Accepte for publication October 24, 997. Communicate by D. Melson. This work was supporte by the European Commission via the Esprit Project 2060 PARASOL. GMD-SCAI.WR, Schloß Birlinghoven, D Sankt Augustin, Germany (thole@gm.e. MacNeal Schwenler GmbH, Innsbrucker Ring 5, D-862, Munich, Germany (stefan.mayer@macsch.com. GMD-SCAI.WR, Schloß Birlinghoven, D Sankt Augustin, Germany (supalov@gm.e. 246

2 C.-A. Thole, S. Mayer an A. Supalov 247 As representative for performanc results from structural analysis simulation, figure.2 shows the elapse times for the parallel sparse matrix solver use in MSC/NASTRAN. MSC/NASTRAN is a wiely use structural analysis coe an its parallel sparse matrix solver is highly optimize. The test case is a sparse matrix resulting from static analysis of a BMW-3 series car boy, which consists out of 2499 ofs mostly resulting from shell elements. For the solution of this matrix problem a spee-up of 4.9 is achieve on 8 processors. This is an excellent result for a sparse irect matrix solver, however, in general a higher scalability is esire in orer to (cost-efficiently solve much larger problems as they are use in inustry toay. Elapse time (s Direct solver Iterative solver Cray-YMP Cray-C90 Processors FIG...Performance of POLYFLOW on an IBM-SP2 for the coat-hanger-ie test case. Elapse time (s Processors FIG..2.Performance of MSC/NASTRAN on an IBM-SP2 for the BMW-3 series test case. In orer to overcome this situation the Esprit PARASOL project was launche.

3 248 Fast solution of MSC/NASTRAN sparse matrix problems.2. PARASOL - An integrate programming environment for parallel sparse matrix solvers. Consierable progress has been achieve in the evelopment of fast an robust iterative solvers for the solution of sparse matrix problems resulting from the iscretisation of partial ifferential equations. This hols in particular for omain ecomposition algorithms an hierarchical solvers like multigri methos. However, most of the commercial coes are still structure such that the sparse matrix problem is set up an afterwars solve without any aitional information about the unerlying partial ifferential equations. This limits the use of iterative solvers to CG methos using incomplete factorisations as preconitioners (an to Algebraic Multigri. The aim of PARASOL[9] is to bring together evelopers of inustrial finite element coes an experts in the area of parallel sparse matrix solvers in orer to test innovative solvers in an inustrial environment for performance an robustness. Direct an iterative parallel sparse matrix solvers will be evelope an integrate into the PARASOL library. Base on the Rutherfor-Boeing sparse matrix file format specification [5], an interface specification was evelope suite for parallel machines. This parallel interface is also able to transport geometry an other information which might be neee by iterative solvers. Base on this interface, parallel irect solvers as well as iterative sparse matrix solvers are being evelope by CERFACS, GMD, ONERA, RAL an University Bergen. The iterative solvers use omain ecomposition methos an hierarchical approaches. Inustrial simulation coes will be moifie in such a way that they support the parallel interface an will evaluate the ifferent solvers on inustrial test cases. In aition to the CFD-coe POLYFLOW, PARA- SOL involves the structural analysis coes MSC/NASTRAN an DNV SESAM as well as the metal forming coe INDEED from INPRO an ARC3D from APEX (simulation of composite rubber metal part eformations. Typical problem sizes for MSC/NASTRAN an DNV SESAM are in the orer of egrees of freeom. Metal forming an CFD coes require the solution of a sparse matrix in each time step. This restricts the problem size to about in these cases. 2. MSC/NASTRAN p-element iscretisations. The hierarchical solver evelope by GMD has been evaluate first on test cases generate by MSC/NASTRAN. MSC/NASTRAN is a commercial structural analysis coe use for the analysis an optimisation of the static an ynamic behavior of structures. This inclues a variety of proucts such as cars, airplanes, satellites, ships an builings. An overview of the technical basis of MSC/NASTRAN is given in [7]. In orer to support the user in achieving the esire accuracy with minimal effort, MSC has extene the MSC/NASTRAN element basis by p-elements. For these elements, an error estimator is provie an in several steps the orer of each base function is aapte until the esire accuracy is reache. Neighbouring elements might arrive at ifferent orer (even in ifferent irections. Soli elements, which may be use as p-elements, are the stanar soli elements (see also [7]:. HEXA: six sie soli element, 2. PENTA: five sise soli element, 3. TETRA: four sie soli element. MSC/NASTRAN supports aaptive p-elements for linear static as well as normal moes analysis an non-aaptive p-elements also for frequency response an transient analysis. For each of the p-elements or for a group, it is possible to specify an initial an maximal orer of the base functions or to set a uniform p-orer. 3. A hierarchical solver exploiting p-elements. A multigri solver requires the efinition of a coarse gri, interpolation, restriction, coarse-gri problem an the smoothing

4 C.-A. Thole, S. Mayer an A. Supalov 249 proceures. In the context of unstructure gris for finite element applications, four ifferent approaches have been propose:. Hierarchy of neste gris. A number of theoretical papers on hierarchical methos assume that a sequence of gris is alreay given. Each finer gri is a strict refinement of a coarser gri an therefore the vector space of the coarse gri base functions are a subset of the vector space of the base functions relate to the fine gri [4]. In practice, this approach can only be applie, if the finite element solver uses some kin of aaptive refinement. 2. Merging finite elements. PLTMG [2, 3], is an example in which elements are merge accoring to certain rules. In this way a non-neste sequence of gris may be generate. Ege swapping is use to further improve the coarsening. 3. Fictitious space metho. The fine gri is projecte onto an auxiliary regular gri [8]. The solution on the regular gri by multigri methos is use as a preconitioner in an iterative graient algorithm for the unstructure gri problem. 4. Algebraic Multigri. A subset of the egrees of freeom an relate matrix, interpolation an restriction are efine solely on the basis of the matrix coefficients [6]. Algebraic Multigri has been successfully applie problems resulting from a single partial ifferential equation an promising results are also reporte for matrices resulting from systems of equations [2]. The approach use for MSC-NASTRAN p-elements is base on the first strategy, using the following efinitions:. G h the iscretisation of the computational omain; 2. V p (G h vector space spanne by the base functions generate by the p-element approach; 3. V (G h vector space spanne by the linear base functions; 4. Vp (G h subspace of all f V p (G h, for which f evaluates to 0 at the noal points of G h. We have V (G h V p (G h, V (G h V p (G h an span ( V (G h Vp (G h = V p (G h. In MSC/NASTRAN, the stiffness matrix K an vectors u for p-elements are organise as follows: K = [ ] [ ] K K u, u =. K K u K an u correspon to the egrees of freeom relate to V (G h, K an u to those of V p (G h. For the solution of Ku = f, two algorithms similar to those iscusse by Axelsson [] were implemente to improve the approximation of the solution u. Metho D: u ν,0 = u ν ( u ν, = K f K u ν,0 u ν,2 = K ( f K u ν, ( u ν,2 = K f K u ν,2 u ν+ = u ν,2

5 250 Fast solution of MSC/NASTRAN sparse matrix problems Metho R(n: u ν,0 = u ν for i = to n o ( ( u ν,i =RELAX K, f K u ν,i ( u ν,i (K =RELAX, f K u ν,i u ν,n+ = K (f K u ν,n u ν,n+ = u ν,n for i = n +2 to 2n + o ( u ν+ = u ν,2n+ u ν,i (K =RELAX, f K u ν,i ( ( u ν,i =RELAX K, f K u ν,i,u ν,i,u ν,i,u ν,i,u ν,i Here, RELAX (K, f, ū performs a smoothing step on the problem Ku = f with ū as initial approximation to the solution. Pointwise Gauß-Seiel iterations have been use for the numerical experiments. Metho D-CG an metho R(n-CG enote algorithms, in which the respective proceures have been use as the preconitioner for a conjugate graient metho. Metho D correspons to a two-space metho. In this case, three large systems of equations have to be solve in each iteration step. Because this is very expensive, this metho has been use only for comparisons with metho R(n. Methos R(n an R(n-CG are calle p-solvers in the rest of this paper. FIG.4..BCELL test case. 4. Numerical results for a moel problem. Figure 4. shows the gri for the 3D BCELL test case, in which HEXA p-elements are use to approximate a cube. The four corners at the bottom are fixe, an parallel forces are applie to the top corners of the cube.

6 C.-A. Thole, S. Mayer an A. Supalov 25 TABLE 4. BCELL test cases. test case p-level size linear ofs total ofs non-zeros BCELL4.2 2fixe BCELL8.2 2fixe BCELL8.2a 2 aaptive TABLE 4.2 Reuction factors for the BCELL test cases. test case metho D R( R(2 R(3 R(4 R(5 BCELL BCELL BCELL8.2a.877 Table 4. shows the basic properties of the ifferent variants of the test case. The small ifference between the number of egrees of freeom for the cases with fixe an aaptive p-level 2 is ue to bubble functions, which are use in aition to the linear base functions for those elements, which remain linear. A factor 4 between the linear egrees of freeom an the total egrees of freeom for the BCELL8.2 case results in a reuction of about 0 of the effort for an LU ecomposition. Tables 4.2 an 4.3 show performance results for the BCELL test cases. For methos D an R(n, in Table 4.2 (as well as in the other tables in this paper, the reuction factor for one iteration of the l 2 -norm of the resiual after 00 iteration is given. In the case of CG methos (cf. Table 4.3, the number of iterations to reuce the l 2 -norm of the resiual by 6 orers of magnitue is given. The results for BCELL4.2 an BCELL8.2 inicate that the reuction factor an the number of CG iterations is inepenent of the gri size. The results show also that R(n-CG is much more efficient than R(n for this test case. (R(n nees 73 iterations for 6 igits resiual reuction. A comparison with D-CG an D shows that some coarsegri frequencies benefit from CG-acceleration. The solution of test case BCELL8.2a by R(n requires about twice as many iteration steps as BCELL8.2. A comparison with D-CG shows that in this case the smoothing is not as effective as before. The reason for this eterioration might be those elements which interface secon orer elements with first orer elements. In orer to improve the smoothing, experiments have been performe using block relaxation. 3x3 blocks an 5x5 blocks ensure that all ofs relate to one noe are treate together. However, the results in Table 4.4 show no improvements. Block relaxation involving all ofs relate to the same element might yiel some improvement. The behavior of the p-solver for tetraheral elements was evaluate with the TET P test case. For this test case, each cube of the BCELL geometry is ivie into 5 TETRA elements an a fixe orer of 2 was requeste for all elements. Table 4.5 shows that in this case the relation between the total number of ofs an the linear part is even more favourable. R(-CG requires 9 iterations like in the BCELL8.2 case. 5. Inustrial test case. In aition to the moel problems, one practical case has been consiere. The KNUCK-MOD-P uses 454 HEXA an 2 PENTA elements. Figure 6. shows the geometry of this test case. The number of R(-CG iterations is larger compare to the BCELL8.2a problem (c.f. Table 5.. This most likely results from the presence of istorte finite elements that are typically foun in iscretisations of the inustrial esigns. The methos intene for alleviation of this ifficulty are currently uner investigation.

7 252 Fast solution of MSC/NASTRAN sparse matrix problems TABLE 4.3 Number of iterations for the BCELL test cases. test case metho D-CG R(-CG R(2-CG R(3-CG R(4-CG R(5-CG BCELL BCELL BCELL8.2a TABLE 4.4 Number of iterations for the BCELL test cases using block Gauß- Seiel relaxation as smoothener with various block sizes. test case block size R(-CG R(3-CG BCELL BCELL BCELL8.2a BCELL8.2a Performance comparison. The p-solver is implemente in Fortran 90 an uses the DGSF routine from the IBM ESSL library for sparse matrix ecomposition. This solver is compare to MSC/NASTRAN (version 69 solvers on the basis of elapse time on a eicate IBM SP-2 thin2 noe with 52 Mbytes of main memory. For all solvers the ata fits into main memory an therefore swapping oes not influence the computing times. Therefore, it shoul be mentione that the reuce memory requirements of iterative solvers are not reflecte. Also, highly optimise MSC/NASTRAN solvers are being compare to a prototype implementation of the p-solver. The measurements of Table 6. show that alreay for these small test cases the prototype implementation of the p- solver outperforms all teste iterative MSC solvers as of version 69. Only in the case of the KNUCK-MOD-P test case is the MSC irect solver faster. For the p-solver, the time for the preparation an all iterations is given. The major part of the preparation time is spent in the DGSF sparse matrix ecomposition routine; for example 8 secons out of 3 for the KNUCK-MOD-P benchmark. The fact that the MSC irect solver nees only 3 secons for a problem with 4 times as many egrees of freeom shows the quality of the MSC irect solver implementation an the improvement potential of the IBM ESSL library routine. 7. Summary. The results show that a hierarchical solver can efficiently treat secon orer sparse matrix problems. There is a ifference of a factor of 4 in the effort which is neee for moel problems an realistic test cases. In any case, the p-element solvers were able to outperform actual MSC/NASTRAN iterative solvers (as of version 69 on the moels use for these tests. With respect to parallelism, the p-solver is alreay a substantial improvement. The relaxation methos are fully parallelizable. The part of the irect solver will take only a fraction of the computing time, if efficient implementations are use or the irect solver is replace by an iterative solver. Further improvements of the solver will therefore inclue avance relaxation technology an the use of iterative (multigri solvers also for the linear problem. REFERENCES [] O. AXELSSON AND I. GUSTAFSSON, Preconitioning an two-level multigri methos for arbitrary egree of approximation, Math. Comp., 40 (983, pp

8 C.-A. Thole, S. Mayer an A. Supalov 253 TABLE 4.5 Performance results for the TET P test case. test case p-level size linear ofs total ofs non-zeros R(-CG TET P 2fixe TABLE 5. Performance results for the KNUCK-MOD-P test case. test case p-level linear ofs total ofs non-zeros R(-CG KNUCK-MOD-P 2fixe [2] R. E. BANK, PLTMG: A software package for solving elliptic partial ifferential equations: Users guie 7.0, Frontiers in Applie Mathematics, Vol. 5, SIAM, Philaelphia, 994. [3] R. E.BANK AND J. XU, An algorithm for coarsening unstructure meshes, Numer. Math., 73 (996, pp. 36. [4] J. H. BRAMBLE, J. E. PASZIAK, AND J. XU, Parallel multilevel preconitioners, Math. Comp., 55 (990, pp. 22. [5] I. S. DUFF, R. G. GRIMES, AND J. G. LEWIS, The Rutherfor-Boeing Sparse Matrix Collection, Technical Report RAL-TR-97-03, Rutherfor Appleton Laboratory, Oxon OX 0QX, Englan, 997. [6] A. KRECHEL AND K. STÜBEN, Operator epenent interpolation in algebraic multigri, in Proceeings of the 5th European Multigri Conference, Stuttgart 0-04 Oct. 996, Springer Lecture Notes in Comp. Science an Engineering, to appear. [7] R. H. MACNEAL, Finite Elements: Their Design an Performance, Mechanical Engineering Series, L. Faulkner, e., Vol. 89, Marcel Dekker Inc., 994. [8] S. NEPOMNYASCHIKH, Fictitious space metho on unstructure meshes, East-West J. Numer. Math., 3 (995, pp [9] J. REID, A. SUPALOV, AND C.-A. THOLE, PARASOL interface to new parallel solvers for inustrial applications, in Proceeings of the ParCo 97, Bonn, Germany, 997, Elsevier Science Publishers, Amsteram, to appear (see also [0] K. STÜBEN, H. MIERENDORFF, C.-A. THOLE, AND O. THOMAS, EUROPORT: Parallel CFD for inustrial applications, in Parallel Computational Flui Dynamics, P. Schiano, A. Ecer, J. Periaux, an N. Satofuka, es., Elsevier Science Publishers, Amsteram, 997, pp [], Inustrial parallel computing with real coes, Parallel Comp., 22 (996, pp [2] P. VANEK, J. MANDEL, AND M. BREZINA, Algebraic multigri on unstructure meshes, UCD/CCM Report 34, University of Colorao at Denver, 994.

9 254 Fast solution of MSC/NASTRAN sparse matrix problems TABLE 6. Elapse times on a IBM SP-2 thin2 noe (52 Mbytes of memory. test case solver elapse times (sec total preparation iteration TETRA P p-solver MSC irect 26 MSC PBDJ (iterative 75 KNUCK-MOD-P p-solver MSC irect 3 MSC PBCJ (iterative 75 MSC PBDJ (iterative 93 MSC PBDC (iterative 69 FIG.6..Geometry of the KNUCK-MOD-P test case.