A Sparse Shared-Memory Multifrontal Solver in SCAD Software

Transcription

1 Proceeding of the International Multiconference on ISBN Computer Science and Information echnology, pp ISSN A Spare Shared-Memory Multifrontal Solver in SCAD Software Sergiy Fialko Cracow Univerity of echnology, Poland ul. Warzawka, 3-55 Cracow, Poland fialko@poczta.onet.pl Abtract A block-baed pare direct finite element olver for commonly ued multi-core and hared-memory multiproceor computer ha been developed. It i intended to olve linear equation et with pare ymmetrical matrice which appear in problem of tructural and olid mechanic. A tep-by-tep aembling procedure together with imultaneou elimination of fully aembled equation ditinguihe thi method from the well-known multifrontal olver, o thi approach can be interpreted a a generalization of the conventional frontal olver for an arbitrary reordering. hi olver i implemented in the commercial finite element oftware SCAD ( and it uage by numerou uer ha confirmed it efficiency and reliability. I. INRODUCION HE olver i baed on the idea of a tep-by-tep aembling of a given tructure out of eparate finite element and ubtructure obtained at the previou aembling tep, and a imultaneou elimination of the fully aembled equation. It i an evolution of the frontal olver [], and it differ from the claical multifrontal olver [][3] which i a purely algebraic olver and get the aembled global matrix in a compreed format a input data. he key ditinctive feature of the preented method are: Each node i aociated with a group of equation (uually, 6 equation per node for hell and pace frame finite element, and 3 equation per node for volumetric element in uncontrained node). hu, thi approach produce the natural aggregation of equation, in thi way improving the quality of reordering algorithm and peeding up the performance at the factorization tage. We refer to the elimination of a node of a finite element model at each elimination tep. It mean that a group of equation aociated with the given node will be eliminated. An object-oriented approach i applied. A front i a C++ cla object that encapulate eliminated node at the current elimination tep, a lit of node and a lit of equation for the current front, a pointer to the dene matrix which we denote a a frontal matrix, and o on. he whole elimination proce i performed on a equence of frontal matrice of the decreaing dimenionality. hi work wa upported by the oftware company SCAD Soft ( Key point of the method are decribed in []. he paper [5] preent an improved verion of the olver. he Choleky block factorization algorithm wa applied intead of a low-performance U factorization from [], and more efficient reordering algorithm comparing to [] were implemented. he current paper preent an extenion of thi olver onto the cla of multi-core and multiproceor hared-memory computer which are very popular today. he contemporary verion of the olver i implemented in SCAD one of the mot popular finite element oftware application for analyi and deign of building tructure in the Commonwealth of Independent State region. II. ANAYSIS SAGE A. Reordering and Symbolic Factorization Modern reordering algorithm have a heuritic nature and do not provide an exact olution of the nonzero entrie minimization problem. Moreover, we never know in advance what reordering trategy lead to the more optimal reult for a given problem. herefore everal reordering method have been developed. he mot efficient method for problem of tructural mechanic uually are the minimum degree algorithm MMD, the neted diection method ND, and the hybrid approach ND_MMD. he fat ymbolic factorization algorithm calculate the number of nonzero entrie for each method of the one lited above and then chooe the method that produce the leat number of nonzero entrie. he ue of a nodal adjacency graph intead of a graph of equation reduce the amount of data in a natural way and make the reordering algorithm and the ymbolic factorization procedure very fat: it take only a few econd even in very large problem (,000,000,000,000 equation) to check 3 reordering method and do the ymbolic factorization. B. A Proce Decriptor Data Structure he reordering method etablihe an order of the node elimination. he next tep i to define the order of the finite element aembling. he node i conidered fully aembled if all finite element, including thi very node, have been already coupled. Adding any of the remaining finite element doe not make any change in the coefficient of equation aociated with /08/$ IEEE 77

2 78 PROCEEDINGS OF HE IMCSI. VOUME 3, 008 thi node. So, all equation belonging to the fully aembled node hould be eliminated. et u conider a imple example, a quare plate with the meh x (Fig. ). Before tarting the elimination proce, the whole tructure i preented a exploded into eparate finite element. where ij i a 6x6 matrix block becaue we uppoe the deign model to have ix degree of freedom per node. he global node number (global indexe) are hown at the top. he firt block column contain the fully aembled equation. he partial factorization, covering only fully aembled equation, i then performed and the fully decompoed part of matrix i moved to a pecial buffer that contain the fully factorized global matrix. he remaining part of the matrix i an incomplete front; it will wait to be ued at the following factorization tep. ocal indexe are ued during the partial factorization. Matrix () i a frontal matrix. In the ame way the node 3, 7, 9 are eliminated, and their repective incomplete front are created. During the elimination of node all finite element have been already involved, and now we look through the lit of global indexe in the incomplete front. Node i preent in the global indexe of previou (incomplete) front, (able I), o we mut aemble incomplete front, to obtain a frontal matrix containing fully aembled equation for node. hi proce i illutrated by able I which how a proce decriptor data tructure. he number of elimination tep i equal to the number of node of the deign model. Fig. Square plate x he equence of nodal elimination produced by the reordering algorithm i:, 3, 7, 9,, 6, 8,, 5. Node i the firt node to be eliminated. hi node belong only to finite element. So, we take the finite element and node a fully aembled. he repective finite element matrix will be: , () We earch for the node to be eliminated in the lit of node of each remaining finite element and in the lit of global indexe for each of the previou front. he front from the preceding elimination tep, which contain thi eliminated node number, are the previou front. he previou front and their correponding finite element, pointed to by the lat column of able I, hould be aembled to obtain the frontal matrix for the current front. C. Frontal ree he proce decriptor data tructure allow u to create a frontal tree. We take the lat front from the bottom of able I (it i Front 9) and put it at the top of the frontal tree (Fig. ). Front 8 i the previou front for Front 9 we put it under Front 9. For Front 8, Front 7 i it previou one we put Front 7 under Front 8. And o on. We reorder the front to reduce the torage memory required for incomplete front. he new front number are hown in italic under the original front number. ABE I. A PROCESS DESCRIPOR DAA SRUCURE No. of front, elimination tep Node being eliminated it of node in the frontal matrix it of previou front it of FE fed to the aembling,,,5 3 3,6,5, 3 7 7,,5, ,8,5,6 5,,5,6, 6 6 6,8,5, 5, 7 8 8,5, 6,3 8,

3 SERGIY FIAKO: A SPARSE SHARED-MEMORY MUIFRONA SOVER IN SCAD SOFWARE 79 he frontal tree conit of (a) nodal front which have more than one previou front, (b) equential front which have only one previou front, and (c) tart front which do not have any previou front. he core memory i allocated dynamically for object of the nodal and tart front. Each equential front inherit the addre of the frontal buffer from it previou front thi help avoid the time-conuming memory allocation and copying of incomplete front. Moreover, if the equence of equential front doe not have any aembled finite element, it i poible to conolidate uch front to enlarge the block ize of fully aembled equation and improve the performance. he conolidated frontal tree i preented in Fig. 3. III. FACORIZAION OF FRONA MARIX Generally, the tructure of the frontal matrix can be very complex (Fig. ). he matrix i ymmetrical, and only the lower triangle part of it i tored in RAM. he fully aembled equation (grayed) are tored continuouly but in arbitrary part of the matrix (not necearily at the top). hi peculiarity retrict the direct application of the popular apack high-performance package (inpack, BAS) and force u to develop our own high-performance oftware for factorization in frontal matrice. Fig. 3 Conolidated frontal tree hu, a Choleky block method ha been developed, which i a generalization of the frontal elimination approach [], [], where the fully aembled equation are tored in arbitrary part of the matrix, onto the block verion. It allow u to achieve the BAS level 3 performance. Fig. A frontal tree he buffer for toring the incomplete front (BuffFront), allocated in the core memory, i virtualized. When the ize of the model exceed the capacity of the random acce memory (RAM), the buffer for the incomplete front i uploaded to hard dik. If all required previou front are in RAM during the aembling of the current nodal front, they are taken from BuffFront. Otherwie, the previou front are taken from hard dik. he BuffFront i compreed from time to time to get rid of low I/O operation during virtualization. he preented method i a ubtructure-by-ubtructure approach, becaue each front preent a collection of coupled finite element a ubtructure, and the elimination proce conit of a tep-by-tep aembling of thee ubtructure and a imultaneou elimination of fully aembled equation. A W B = A W B I 0 0 B W C W F W S B W = C I I 0 0 W W 0 0 I where W, F, W are block of fully aembled equation, A, B, C are block of partially aembled equation, which make up an incomplete front (Fig. 5) after the partial Choleky factorization, A 0 (3) B C he ign diagonal I, S, I allow one to generalize the claic Choleky factorization method onto a cla of problem with indefinite matrice. he ymbol mean that the ()

4 80 PROCEEDINGS OF HE IMCSI. VOUME 3, 008 correponding matrix block i modified during the partial factorization. he fully aembled block are moved to the buffer for the decompoed part of the matrix, and next the factorization of it i performed in the addre pace of thi buffer. Fig. he general tructure of a frontal matrix. he fully aembled equation are moved to a buffer for the decompoed matrix he incomplete front conit of ector A, B, C. he ymmetrical torage cheme i ued. he coefficient of ector A remain in their poition, ector B move up, and ector C move up and to the left by the width of the grey trip. he final tructure of the incomplete front i preented in Fig. 5. he partial block factorization i performed in everal tep: Factorize block F: F Update block W, W : S W = W S W = W = S () W W (5) Fig. 5 he tructure of the frontal matrix after the partial factorization. An empty trip appear at the bottom of matrix. Update ector A, B, C: A = A W S B = B W S C = C W S W W W Expreion () (6) are derived from () by mean of the block matrix multiplication. Fig., 5 help u to undertand the tructure of the frontal matrix. he main idea of the generalized Choleky block factorization method i imilar to the Gau block elimination approach, reorganized to ue the level 3 BAS [0]. A particular cae where a fully aembled part i at the top of the frontal matrix i preented in [5]. Matrix F ha a mall dimenionality, o there i no problem to tore all matrix multiplier () in the proceor cache and achieve a peak performance at thi tage. Stage (5) i a forward reduction performed on the package of right-hand ide it i poible to achieve a good performance here, too. he procedure that conume mot of the time i Stage (6). Each matrix multiplier W, W i divided into maller block, and the block matrix multiplication enure the level 3 BAS. IV. PARAE IMPEMENAION IN FRONA MARIX he bottleneck in the multi-core and multiproceor hared-memory computing ytem i an inufficient bandwidth of the ytem bu which lead to a weak peedup a the number of proceor grow. A typical dependence for the peedup parameter, S p = / p, where i a computing time on one proceor, p i that on p proceor, for algorithm where the performance factor q (matrixvector multiplication, non-block matrix-matrix multiplication) i preented in Fig. 6. he performance factor i q = f/m where f i the number of arithmetic operation for a given algorithm and m i the number of data tranfer between RAM and the proceor cache. A totally different dependence take place on the ame computer for the block matrix-by-matrix multiplication; here q M where M i the cache ize (fig. 7). So, it i poible to achieve an efficient peedup for computer with an inufficient bandwidth of the ytem bu by increaing the number of proceor for level 3 BAS algorithm, where (6) q M, becaue thee algorithm ue ignificantly fewer RAM cache RAM tranaction per ingle arithmetic operation than the algorithm where q. he reult preented in Fig. 6 through 8 have been obtained by pecial tet prepared by the author. herefore the main attention during the development of the parallel code i paid to the Choleky block factorization in the frontal matrix and the fork-joint parallelization technique, which i ued mainly at the tage 6 during the block matrix-by-matrix multiplication.

5 SERGIY FIAKO: A SPARSE SHARED-MEMORY MUIFRONA SOVER IN SCAD SOFWARE 8 he Microoft Viual Studio 008 environment (the C++ language for mot of the code and the C language for bottleneck fragment of the code in order to achieve a peak efficiency by compiler optimization) with the OpenMP upport i ued. All program code, including the factorization of frontal matrix, wa developed by the author. he local reordering of equation performed in the frontal matrix for the nodal and tart front allow u to reduce the amount of time-conuming operation of moving ector B, C block and thu increae the performance of method (Fig. 8). Fig. 8. Sp = / p v. number of proceor p for factorization of frontal matrice. race perfect peedup, trace local front not reordered, trace 3 reordered front. Fig. 6. Sp = / p v. number of proceor p for algorithm where the performance factor q For intance, in the example of Fig. the order of node proceing for front 7 (tab. I) mut be 5,, 8 rather than 8, 5,, becaue thi numbering enure that at each front elimination tep the fully aembled equation tay at the bottom of the matrix and no moving of incomplete front i required. ypical dependencie S p = / p v. the number of proceor p for factorization of frontal matrice for preented method are hown in Fig. 8. he total time of factorization for all frontal matrice i under conideration. An eential improvement of performance occur when the local reordering of equation in frontal matrice i performed (the dahed curve correpond to local reordering and olid curve to no local reordering). Fig. 7. Sp = / p v. number of proceor p for algorithm where the performance factor q M / We reorder the frontal node according to the equence of global reordering and then take the revere order each eliminated node occur at the end of the node lit. It enure uch a tructure of the frontal matrix that the fully aembled equation are at the bottom and no moving of incomplete front i required (only ector A i hown in the figure, ector B, C are omitted). V. NUMERICA RESUS All reult preented further below have been obtained on the computer Intel Core Quad CPU GHz, cache x: 3 KB (code), x: 3 KB (data), x:096 KB, random-acce memory DDR 800 MHz GB, operating ytem Window XP Profeional. D. et A cube with the meh 0x0x0 coniting of brick volumetric finite element i conidered. Four corner node of the bottom face are retrained. hi model comprie 68 9 node, finite element, equation; the whole model i tored in RAM. he analyi tage, which include reordering, ymbolic factorization, creation of the proce decriptor data tructure, reordering and conolidation of the frontal tree, i performed a a erial computation. he computing time i 3. he olution tage, including forward backward reduction, take. he parallel computing tage cover only the factorization of frontal matrice

6 8 PROCEEDINGS OF HE IMCSI. VOUME 3, 008 and take a predominant part of the whole time required by the numerical factorization tage (able II). he factorized matrix, a well a the buffer for incomplete front, i tored in RAM, and due to thi fact the peedup parameter on four proceor i about 3.0. It i a good reult for multi-core and hared-memory office computer. For problem of tructural mechanic olved by finite element oftware baed on the Intel Math Kernel ibrary (PARDISO olver [5]), the peedup parameter on four proceor i about.9. ABE II. IME OF NUMERICA FACORIZAION FOR CUBE 0X0X0 Number of proceor 3 Numerical factorization p, S p = / p However, the PARDISO olver work only with the core memory, o it capabilitie are retricted to problem of relatively mall dimenion. B. et A quare plate with two mehe 00x00 and 800x800 i conidered. A quadrilateral flat hell finite element i ued. he time of numerical factorization wa compared with that for the pare direct olver from ANSYS v.0. he firt problem (meh 00x00, able III) contain equation and the econd one (meh 800x800, able IV) contain i tored on hard dik, and virtualization i ued for the buffer that contain incomplete front. he calability i weak the peedup parameter on four proceor i about.9. he ize of the factorized matrix i 9 73 MB. C. et 3 he deign model of a multitory building complex i preented in Fig. 9. he number of equation i 53 67, the ize of the factorized global matrix i MB, the numerical factorization time on four proceor i 860 ec. he dik memory i ued for virtualization of the factorized matrix a well a for virtualization of the buffer where the incomplete front are tored. et A demontrate that the calability of the preented method for multi-core and multiproceor office computer i imilar to the calability of the pare direct olver baed on Intel Math Kernel ibrary when the dimenionality of the model permit to ue only the core memory. et B prove that the performance of our olver i imilar to the performance of the conventional multifrontal olver from the wellknown ANSYS oftware. In addition, tet B how that the preented olver work very efficiently with the core memory. et C illutrate that the preented method olve a really large finite element problem of tructural mechanic very efficiently. ABE III. IME OF NUMERICA FACORIZAION FOR PAE 00X00 Number of proceor Numerical factorization for ANSYS v..0, Numerical factorization for preented olver, 6 5 ABE IV. IME OF NUMERICA FACORIZAION FOR PAE 800X800 Number of proceor Numerical factorization with ANSYS v..0, failed failed failed Numerical factorization with the preented olver, For the firt model (meh 00x00), our olver tore the factorized matrix on hard dik, but the buffer for incomplete front i tored in the core memory. Due to virtualization of the factorized matrix, the calability i wore comparing to the previou model the peedup parameter on four proceor i about.9. he performance demontrated by both the pare direct olver from ANSYS v..0 and our olver i about the ame. For the econd model (meh 800x800), the olver from ANSYS v..0 failed due to inufficient RAM. Reult for our olver are preented in able IV. he factorized matrix Fig. 9. Finite element model of a multitory complex Mocow-city, node, finite element, equation

7 SERGIY FIAKO: A SPARSE SHARED-MEMORY MUIFRONA SOVER IN SCAD SOFWARE 83 VI. CONCUSION A block-baed pare direct multifrontal olver for finite element problem of tructural and olid mechanic ha been developed. hi method i oriented mainly at the uage on common hared-memory multiproceor and multi-core computer. he propoed olver ha a clear mechanical interpretation a a ubtructure-by-ubtructure method. It ue the modern reordering algorithm which effectively reduce the fill-in. he Choleky block algorithm with a generalization onto the cla of indefinite matrice i applied here to the factorization of frontal matrix. he complicated tructure of the frontal matrix retrict the direct application of the BAS high-performance package and force u to develop our own high-performance code baed on a ubdiviion of the frontal matrix into block. he OpenMP technique i applied to produce a fork-joint parallelization. he block ubdiviion and an appropriate numbering of equation in the frontal matrix, which reduce the amount of low-performance moving operation, allow u to improve the calability of the method. A comparion with the pare direct olver from ANSYS v.0 demontrate that the performance of the propoed olver i cloe to the performance of ANSYS, but the propoed olver allow u to olve eentially larger problem on the ame computer than the ANSYS olver i capable of proceing. he reliability and efficiency of the propoed olver ha been confirmed by numerou uer of the SCAD commercial oftware. REFERENCES [] B. M. Iron, A frontal olution program for finite-element analyi, International Journal for Numerical Method in Engineering,, pp. 5 3, 970. [] I. S. Duff, J. K. Reid, he multifrontal olution of indefinite pare ymmetric linear ytem, ACM ranaction on Mathematical Software, 9, pp , 983 [3] F. Dobrian, A. Pothen, Oblio: a pare direct olver library for erial and parallel computation, echnical Report decribing the OBIO oftware library, 000. [] S. Yu. Fialko, Stre-Strain Analyi of hin-walled Shell with Maive Rib, Int. App. Mech., 0, N, pp. 3 39, 00. [5] S. Yu. Fialko, A block pare direct multifrontal olver in SCAD oftware, in Proc. of the CMM-005 Computer Method in Mechanic June -, 005, Czetochowa, Poland, pp [6] A. George, J. W.-H. iu, he Evolution of the Minimum Degree Ordering Algorithm, SIAM Rev., 3, March, pp. -9, 989. [7] A. George, J. W.-H. iu, Computer olution of pare poitive definite ytem, New Jerey : Prentice-Hall, Inc. Englewood Cliff, 98, ch. 8. [8] C. Ahcraft, J. W.-H. iu, Robut Ordering of Spare Matrice Uing Multiection, echnical Report CS 96-0, Department of Computer Science, York Univerity, Ontario, Canada, 996. [9] A. George, J. W.-H. iu, Computer olution of pare poitive definite ytem, New Jerey : Prentice-Hall, Inc. Englewood Cliff, 98, ch. 5. [0] J. W. Demmel, Applied Numerical inear Algebra. SIAM, Philadellphia, 997, ch.. [] A. Kumbhar, K. Chakravarthy, R. Kehavamurthy, G. V. Rao, Utilization of Parallel Solver ibrarie to olve Structural and Fluid problem, White paper by Crane Software, oftware/product/amo-na/eng/37366.htm. [] N. I. M. Gould, Y. Hu, J. A. Scott, A numerical evaluation of pare direct olver for the olution of large pare, ymmetric linear ytem of equation, echnical report RA-R , Rutherford Appleton aboratory, 005. [3] O. Schenk, K. Gartner, On fat factorization pivoting method for pare ymmetric indefinite ytem, echnical Report CS-00-00, Department of Computer Science, Univerity of Bael, Switzerland, 00. [] O. Schenk, K. Gartner, Solving unymmetric pare ytem of linear equation with PARDISO, Journal of Future Generation Computer Sytem, 0(3), 75 87, 00. [5] O. Schenk, K. Gartner, W. Fichtner. Efficient pare U factorization with left-right looking trategy on hared memory multiproceor, BI, 0(), 58 76, 000.