A Highly Scalable Parallel Boundary Element Method for Capacitance Extraction

Transcription

1 A Highly Scalable Parallel Bounary Element Metho for Capacitance Extraction The MIT Faculty has mae this article openly available. Please share how this access benefits you. Your story matters. Citation As Publishe Publisher Hsiao, Yu-Chung, Luca Daniel. "A highly scalable parallel bounary element metho for capacitance extraction." 48th ACM/EDAC/IEEE Design Automation Conference (DAC ), June -,, San Diego, California, USA. p Association for Computing Machinery Version Author's final manuscript Accesse Fri Dec 7 :6:7 EST 8 Citable Link Terms of Use Creative Commons Attribution-Noncommercial-Share Alike Detaile Terms

2 A Highly Scalable Parallel Bounary Element Metho for Capacitance Extraction Yu-Chung Hsiao Massachusetts Institute of Technology Luca Daniel Massachusetts Institute of Technology ABSTRACT Traitional parallel bounary element methos suffer from low parallel efficiency an poor scalability ue to the long system solving time bottleneck. In this paper, we emonstrate how to avoi such a bottleneck by using an instantiable basis function approach. In our emonstrate examples, we achieve 9% parallel efficiency an scalability both in share memory an istribute memory parallel systems. Categories an Subject Descriptors J.6 [Computer-aie engineering]: Computer-aie esign (CAD) General Terms Algorithms. Keywors Parallel computing, Bounary element metho, Capacitance extraction, Fiel solver.. INTRODUCTION Stanar bounary element methos usually involve two steps: the system setup step, in which a linear system is constructe, an the system solving step, respectively. The time complexity for these two steps are O(N )ano(n )if irect methos are use. Such complexity is etermine by the bounary element metho formulation, irrespective of the choice of basis functions. Among various types of basis functions, the most prevalent choice is piecewise constant basis functions, because of the availability of close-form expressions for both collocation an Galerkin s integrations. Piecewise constant basis functions are also wiely applicable for representing any general physical charge istributions. This work is supporte by Mentor Graphics, Avance Micro Devices (AMD), the Global Research Collaboration (GRC), an the Interconnect Focus Center (IFC). Permission to make igital or har copies of all or part of this work for personal or classroom use is grante without fee provie that copies are not mae or istribute for profit or commercial avantage an that copies bear this notice an the full citation on the first page. To copy otherwise, to republish, to post on servers or to reistribute to lists, requires prior specific permission an/or a fee. DAC, June -,, San Diego, California, USA. Copyright ACM ACM //6...$.. However, using such type of basis functions inevitably results in a large linear system, which requires extremely long system solving time. In orer to solve large linear systems efficiently in singlecore computation environments, several acceleration methos base on piecewise constant basis functions were propose uring the last couple of ecaes, such as fast multipole expansion [4] an pre-correcte FFT [6]. Instea of using irect Gaussian elimination, these methos use Krylov subspace iterative methos, an exploit specialize algorithms to approximate matrix-vector proucts, reucing both time an memory complexity from O(N )too(nlog N). Such acceleration algorithms are, however, not efficiently parallelizable. An efficient parallel algorithm requires an embarrassingly parallelizable structure, which means low ata epenency, minimize memory accesses, an small amount of ata transfer. In Krylov subspace iterative methos, ue to the use of piecewise constant basis functions, the resiual vectors are large an nee to be transfere between parallel computing noes or accesse by ifferent threas in share memory for each iteration. Besies, although stanar matrix-vector prouct operations are embarrassingly parallelizable, their approximate forms aopte in the aforementione acceleration algorithms introuce large ata epenency when reusing partial calculations in orer to achieve acceleration. Some previous works, such as [] an [7], showe that the parallel performance of such matrixvector approximations is quite limite: the parallel speeup saturates very quickly, an the efficiency rops to 6% or even less at aroun only eight parallel computing noes. All these observations give rise to the nee for using a more compact solution representation than piecewise constant basis functions. When a much smaller number of basis functions, N, is use to represent the solution, irect solving methos are consiere efficient. This is because irect methos o not suffer from initialization overheas from which the matrix-vector approximation methos usually suffer. In aition, real solving performance heavily epens on whether the memory hierarchy is utilize efficiently, for instance, by consiering the machine-specific cache size in orer to reuce the cache miss rate. In practice, irect solving methos can be compute much more efficiently if the unerlying basic linear algebra operations are optimize with the consieration of harware esign. Various mature linear algebra libraries, for instance, ATLAS, GotoBLAS, an other harware venor optimize routines, provie more than one orer of magnitue faster performance than the implementation without harware consieration. The aforementione matrix-vector approximation

3 x 6 6 x. Source wire. 4 x 6 Target wire Source wire h 4 x 7 charge ensity (C/m ) 4 reflecte arch shape arch shape flat shape b(h) extension a(h) length ingrowing h length Target wire position (m) x Figure : A pair of simple crossing wires. Figure : Extracte flat an arch shapes. methos, which utilize non-stanar matrix operations, are not compatible with an o not benefit from such highly optimize linear algebra libraries. However, using a compact solution representation is not always avantageous. Its compactness usually results from using complex basis function shapes, which complicates the numerical integrations in the system setup step. Fortunately, when N is small, the linear system can be store in the matrix form, an the system setup step consists of simply filling in the matrix. Each entry of such matrix can be compute inepenently in each parallel computing noe without ata epenency an communication overheas. Therefore, highly efficient an scalable parallelization can be achieve. In this paper, we use instantiable basis functions [] as the compact solution representation for the capacitance extraction problem. Using such basis functions in a single-core environment woul result in negligible system solving time but relatively long system setup time, which is an excellent scenario for efficient parallel computing. The backgroun of bounary element methos for the capacitance extraction problem an instantiable basis functions are summarize in Section. Section iscusses a parallel matrix filling strategy that minimizes ata epenency an memory accesses. In Section 4, we evelope four generic acceleration techniques for computing integrations, whose importance is usually overlooke when piecewise constant basis functions are use. When more complex basis functions are use an the system setup time ominates, the acceleration of computing integrations can irectly improve the overall performance. Section escribes our C++ implmentation in share-memory an istribute-memory systems through OpenMP an MPI, respectively. In Section 6, their performance is presente. Our conclusions are mae in Section 7.. BACKGROUND. The BEM for capacitance extraction Given an n-conuctor interconnect geometry embee in a uniform ielectric meium, the capacitance matrix can be obtaine by solving the integral equation of the electric static problem ρ(r ) S 4πε r r s = φ(r) () for charge istribution ρ(r ). In equation (), r an r R are position vectors in D space, φ(r) :R R is the electric potential, ρ(r ):R R is the unknown charge istribution, S is the conuctor surfaces, ε is the ielectric constant of the meium, an the operator computes the Eucliean istance of its argument. By expressing ρ(r )asa linear combination of basis functions ψ j for j =,,...,N, an applying the stanar Galerkin s testing, one can transform equation () into [ N ] ψ i(r)ψ j(r ) j= s i s 4πε r r s s ρ j = ψ i(r)φ(r)s, () j s i where s i an s j are the supports of ψ i an ψ j,respectively. Equation () can be further represente as a system of linear equations Pρ =Φ, () where ρ =[ρ,ρ,...,ρ N] T, P is an N N system matrix consisting of the brackete factors for all i, j in equation (), an Φ is an N n matrix. The k-th column of Φ is the righthan sie of equation () with φ(r) = {, if ψ i is on the k-th conuctor, otherwise. After equation () is solve, the capacitance matrix C can be foun by computing the expression C =Φ T ρ.. Instantiable basis functions Unlike piecewise constant or other mathematically efine basis functions, instantiable basis functions [], are the collection of the funamental shapes extracte from elementary problems, such as a pair of crossing wires shown in Figure. The curve shown in Figure is the sie view of the charge istribution on the top face of the bottom wire inuce by the top wire. This charge istribution can be further ecompose into a constant flat shape an two arch shapes A p(u). The subscript p is a vector of parameters. The vector p contains wire separation h in Figure an other geometric parameters, epening on the require capacitance accuracy. These shapes are then extene into D flat an arch templates by aing a perpenicular coorinate with constant values, i.e., T F (u, v) =ant Ap (u, v) =A p(u), where v is in the irection into the paper. The full set of instantiable basis functions consists of two parts: inuce basis functions an face basis functions. Face basis functions are by efault place on each rectangular surface of a conuctor an have a normalize value of. Inuce basis functions are use to capture the electrostatic interactions between wires. They are instantiate by assembling flat an arch templates with proper parameter vectors p an place in the neighborhoo of wire intersections. Instantiable basis functions are constructe uner the assumption that all conuctors are embee in a uniform ielectric meium an in Manhattan geometry.

4 . PARALLELIZATION By using instantiable basis functions, more than 9% of the total runtime is spent on the system setup step. In the following, we will introuce a balance workloa ivision scheme with minimize ata communications for constructing the system matrix P. For the system solving step, we will resort to the stanar irect metho implemente in multithreae linear algebra libraries. A single instantiable basis function consists of one or multiple templates, ψ i (r) = ψ i,ī(r). ī Therefore, for N instantiable basis functions constructe from a given capacitance extraction problem, a matrix entry of P R N N can be expresse as the summation of the integrals involving every template pair: P i,j = ψ i,ī(r)ψ j, j(r) s ī j i,ī s j, j 4πɛ r r s s. (4) Each ψ k, k is either an arch template T Ap (u, v) oraconstant value of if it is a face basis function or a flat template. When both ψ i,ī an ψ j, j are constant, which is the most common situation, the corresponing integral egenerates to the Galerkin s integration of piecewise constant basis functions. The computation becomes much more balance if each integral is compute base on templates rather than basis functions. Accoringly, we collect all templates from each basis function an relabel them from to M. Amatrix P R M M can be constructe such that T i(r)t j(r) P ij = s i s j 4πε r r s s, () an then conense into P by combining the rows an the columns which are relate to the same basis function. Figure shows an example that a matrix P is constructe by four basis functions (N =4)inwhichonlyψ consists of two templates (M = ). The mapping between ψ i,ī an T i is efine as in the following orer set: {ψ,,ψ,,ψ,,ψ,,ψ 4,} = {T,T,T,T 4,T }. In practice, M is usually. to times greater than N, or equivalently, P can be 9 times larger than P. In orer to minimize memory access, we avoi allocating P but construct P irectly. Figure shows the numbers of entries of P that are summe to a single entry of P by ifferent color levels. Because P is symmetric, only those off-iagonal entries of P which are combine to the iagonal of P contribute their values twice, for instance P i=,j=4 in Figure. This statement is generally true irrespective of the number of templates inclue in a basis function. In orer to achieve high parallel efficiency, it is sufficient to construct system matrix P through equation () an ivie work accoring to the number of entries in P. Although the cost of calculating each P ij is influence by template types an spatial orientations, in practice, such work ivision is sufficiently balance as we will see in Section 6. The resulte parallelization algorithm is constructe by using an inepenent inex k which iterates the upper triangular matrix of P from to M(M +)/. The inex k is converte into (i, j) for P an then mappe to (i,j ) for P. For a parallel system of D computing noes, the full Figure : Matrix P conenses into P. Algorithm Parallelization prototype for system setup Require: D parallel noes inexe from =tod; Allocate an initialize P =; Construct an array l such that l i = i ; K M(M +)/; Partition {,,...,K } into K,K,...,K D s.t. K = K = = K D = K/D an K D = K (D ) K ; for each parallel noe {,...,D} o for all k K o j ( + +8k)/ ; i k j (j +)/; p Galerkin s integration of T i an T j ; if i j an l i = l j then P li,l j P li,l j + p; else P li,l j P li,l j + p; en if en for en for return P ; range of k is equally ivie into D partitions, an each noe is in charge of calculating the corresponing entries of P in each partition. This parallelize system setup algorithm is summarize in Algorithm. 4. INTEGRATION ACCELERATION TECHNIQUES After the system setup work is ispatche to each parallel computing noe, the integral in the square bracket of equation () is compute in a sequential environment. Although we mainly aress the electrostatic problem in this work, the techniques we evelop in this section are also applicable to other bounary element methos, as long as their corresponing Green s functions ecay with istance. In the following, we will escribe the acceleration techniques for computing the integral P i,j = yi xi y j x j y i x i y j x j T i(x, y)t j(x,y ) 4πε δx + δy + δz s s, (6) in which δu = u u for u {x, y, z}, an s an s are xy an x y, respectively. The support of T i(x, y) is [x i,x i] [y i,y i]. The support of T j(x,y ) is efine similarly. 4. General consierations In instantiable basis functions, T i an T j are efine to have at most D shape variation. When T i(x, y) =T i(x)

5 an T j(x,y )=T j(y ), we can rearrange the expression in equation (6) into xi [ y j yi ] x T i(x) T j(y j x y/(4πε) ) y x. x i y j y i x δx + δy + δz j (7) The inner brackete D integration in equation (7) is ientical to the collocation integration with piecewise constant basis functions, which has a close-form expression an can be compute efficiently. The remaining outer D integration is then compute numerically by Gaussian quarature. The analytical expressions are also available for D or 4D when one or both of T i an T j are flat templates. However, the computation cost grows rapily as the integration imension increases. For instance, a D expression involves only 8 terms whereas more than terms are present in a 4D expression. In terms of runtime, the former is roughly times faster than the latter. A useful strategy to lower the chance of invoking high imensional expressions is to utilize the ecay property of the Green s function with istance. When two templates are separate far enough, the function behaviors of the higher an the lower imensional expressions are numerically inistinguishable. Therefore, we can use the lower imensional expression to approximate the higher imensional integral. The istance above which we can start to perform such approximation (approximation istance in short) can be parameterize as a function of template parameters. Aitional approximation levels can also be inserte between integration imensions by using quarature points to further reuce the chance of invoking higher imensional expressions. 4. Acceleration techniques Theintegralin(7)canbeecomposeintotwoparts: the outer Gaussian quarature an the inner close-form expressions. Evaluating such close-form expressions is the bottleneck of overall performance. In this section, we propose four ifferent acceleration techniques that can provie an extra speeup in aition to parallelization. 4.. Direct tabulation Instea of evaluating terms in the 4D analytical expression, we can tabulate the expression as a 6-parameter table. Such tabulation is feasible because the range for each parameter we nee to tabulate is limite by the approximation istance. Beyon the approximation istance, we can use the lower imensional expressions which can also be tabulate. The target tabulation function is F efinite = y x y x y x y x 4πε r r x y xy. (8) One of the attractive features of this metho is its very manageable error control. However, its performance is limite by the expensive six-imensional linear interpolation. 4.. Tabulation of inefinite integrals We can reuce the tabulation parameters from six to three by tabulating the inefinite integral without substituting the upper an lower limits of each imension. Equation (8) can be rewritten as an inefinite integral form: F inefinite (x x,y y,z) x y x y x y x y = F efinite. (9) The error control of this formula nees careful investigation of function behavior. 4.. Tabulation of expensive subroutines When the close-form expressions are evaluate, most of the computation time is spent on calling expensive elementary functions, such as atan an log. Tabulating these single parameter functions is memory efficient even with high accuracy requirement an using zero-orer hol. The IEEE-74 floating-point representation is especially useful for tabulating the logarithmic function log ({mantissa} {exp.} )={exp.} +log ({mantissa}), in which only log ({mantissa}) nees to be tabulate []. In our experiment, tabulating the first 4 bits of mantissa is sufficient to achieve an error below % for the 4D analytical expression. This approach is times faster than the builtin log function on a Xeon. GHz machine. A similar tabulation for atan provies a 4 times speeup compare to its built-in version. This metho is attractive for its easy implementation an error control Rational fitting It can be seen from equation (9) that the analytical expression of an integral is ill-conitione. Several most significant igits of the inefinite integrals are cancele out when subtracting the lower limit substitution from the upper limit substitution. Instea, we can aopt a multivariable rational function of egree (n, m) as a more efficient an numerically stable expression fn (w) f(w) = f, () D(w) where w R k is an input vector, an f N (w),f D(w) R k R are polynomial functions efine as f N (w) = β N,αw α, () an f D(w) = α n, α α m, α β D,α w α in which α an α are k-imensional multi-inices. Using such rational function form is especially suitable for those Green s functions which ecay with istance. To fin the best coefficients β N an β D,itissufficienttoreformulate equation () into an optimization problem an apply n training samples f(w i),i=,,...,n: minimize β N,α,β D,α subject to, n f(w i)f D(w i) f N (w i) i= α m, α β D,α =. () This optimization problem can be solve by using STINS []. The constraint in equation () reuces the egrees of freeom of β N,α an β D,α by in orer to normalize the rational function. The error control of this approach relies on the choice of training samples.

6 ^ ^ D W ^ W ^ W W D D ' t / t ' t t / ' Z t t / t ' t / t W D D Figure 4: Flowchart for the system setup step in a share-memory system. Figure 6: Flowchart for the system setup step in a istribute-memory system. W nal functions an the size of the inefinite integral table are greatly increase to account for the case when equation () is evaluate near the bounary of the integral region [x,x ] [y,y ]. We chose the metho of tabulating expensive subroutines in our implementation. Figure : Partitions of system matrix P in a twonoe istribute-memory system. 4. Performance comparison an iscussion To compare the performance of the four integration techniques in Section 4., we focus on a simplifie D analytical expression f D(r) = y x y x r r x y. () The performance for each metho is liste in Table. All these results are teste on a Xeon. GHz machine implemente in C++ single precision floating points with % error tolerance. Table : Performance comparison of ifferent integration acceleration techniques for equation () Techniques Time/speeup Memory. Original analytical expr. 8 ns/.. Direct tabulation 6 ns/.6. MB. Tabulation of inef. int. 4 ns/.6. MB. Tabulation of exp. routines 8 ns/.. MB 4. Rational fitting 4 ns/.4 It shoul be note that this performance assessment is only vali for instantiable basis functions. This is because instantiable basis functions allow templates to overlap each other, which is generally not the case for other types of basis functions. As a result, the minimum egree of ratio-. IMPLEMENTATION We implemente Algorithm in both share-memory an istribute-memory systems through OpenMP an MPI, respectively.. In a share-memory system In this case, both template efinitions from the input file an the output system matrix P are store in share memory. D aitional threas are generate after P is allocate. The threa retrieves template efinitions from share memory, computes the entries of P in the partition K within its private memory, an then as the result to the corresponing entries of P. Multithreas are restore into a single main threa after every K i is compute. This process is iagramme in Figure 4.. In a istribute-memory system In a istribute-memory system of D noes, the main process = computes the entries of P in the partition K an stores the results in P as in a sequential environment. For, the process hols its own copy of template efinitions, calculates entries of P in K iniviually, stores the results in a separate partial matrix P K of P, an finally sen the resultant P K to the main process. Accoring to equation (4), ajacent partial matrices may share a common column in P. Therefore, P K is an N N matrix where N = l jfirst l jlast + an j first an j last are the column inices for P of the first an the last elements in K,respectively. When the main process receives P K, the partial matrix P K is shifte to the l jfirst -th column an ae to matrix P. Figure shows the case when the system matrix P in Figure is execute in a istribute-memory system of two noes. The general case is epicte in the flowchart in Figure 6. In our implementation, the istribute memory behavior is simulate by the operating system through MPI on a -processor--core machine.

7 x 7 9 x x x x Efficiency (%) This work in OpenMP This work in MPI Parallel fast multipole [7] Parallel pre correcte FFT [] Number of Processors Figure 7: Transistor interconnect an 4 4 bus examples. Table : Performance improvement by using the integration acceleration techniques. FASTCAP Instantiable basis func. Impr. [4] w/o accel. w/ accel. Setup time 94. ms.7 ms 86% Total time 4 ms 97.8 ms 4.4 ms 76% Memory 4 MB 8 KB. MB 6. EXAMPLES We use an inustry provie transistor interconnect structure in Figure 7 to emonstrate the effectiveness of the integration acceleration techniques. Our acceleration technique evelope in Section 4 improves the system setup time by 86% an the overall computation in a single-threae environment for this example is 6. times faster than FAST- CAP [4] with a times smaller memory requirement on a Xeon. GHz machine for the same.8% error. This accuracy is compare with a finely iscretize FASTCAP reference solution which is obtaine by refining the iscretization by % for each iteration until the solutions from the last two iterations are within.% ifference. This result is summarizeintable. We use another 4 4 bus structure in Figure 7 to show the parallel efficiency for ifferent numbers of noes in Figure 8. our solver achieves 9% parallel efficiency by using our share-memory implementation on a Xeon. GHz 4- core machine an 89% by using our istribute-memory implementation on a Xeon.6 GHz -processor--core machine, whereas the parallel efficiencies of the parallel precorrecte FFT [] an of the parallel fast multipole expansion metho [7] rop significantly to 4% an 6% at 8 cores, respectively. These efficiencies are the best available values foramuchsmaller bus example from their original papers with meium iscretization. The speeups an efficiencies of our solvers are liste in Table. 7. CONCLUSIONS In this paper, we emonstrate that the parallelization bottleneck in traitional bounary element methos can be avoie by using instantiable basis functions. Our examples showe that our solver is 6 faster than FASTCAP in a single-threa environment. Furthermore, our approach achieve about 9% parallel efficiency with parallel computing noes through MPI implementation. We plan to release our solver on public omain as open source. Figure 8: Parallel scalability for 4 4 buses. Table : Performance for 4 4 buses in Figure 7. # of Share-memory system Dist.-memory system use with 4 noes with noes noes Time Speeup Eff. Time Speeup Eff. 4. s. % 44. s. %.7 s.86 9%.7 s.94 97% 4. s.6 9%. s.6 9% s 7. 9% 4.9 s % 8. REFERENCES [] N. R. Aluru, V. B. Nakarni, an J. White. A parallel precorrecte fft base capacitance extraction program for signal integrity analysis. In Proceeings of the r annual Design Automation Conference, DAC 96, pages 6 66, New York, NY, USA, 996. ACM. [] B. Bon, Z. Mahmoo, Y. Li, R. Sreojevic, A. Megretski, V. Stojanovi, Y. Avniel, an L. Daniel. Compact moeling of nonlinear analog circuits using system ientification via semiefinite programming an incremental stability certification. Computer-Aie Design of Integrate Circuits an Systems, IEEE Transactions on, 9(8):49 6, aug.. [] Y.-C. Hsiao, T. El-Moselhy, an L. Daniel. Efficient capacitance solver for interconnect base on template-instantiate basis functions. In Electrical Performance of Electronic Packaging an Systems, 9. EPEPS 9. IEEE 8th Conference on, pages 79 8, Oct. 9. [4] K. Nabors an J. White. Fastcap: a multipole accelerate - capacitance extraction program. Computer-Aie Design of Integrate Circuits an Systems, IEEE Transactions on, ():447 9, 99. [] G. F. N. M. O. Vinyals. Revisiting a basic function on current cpus: A fast logarithm implementation with ajustable accuracy. International Computer Science Institute, 7. [6] J. Phillips an J. White. A precorrecte-fft metho for electrostatic analysis of complicate - structures. Computer-Aie Design of Integrate Circuits an Systems, IEEE Transactions on, 6():9 7, 997. [7] Y. Yuan an P. Banerjee. A parallel implementation of a fast multipole-base - capacitance extraction program on istribute memory multicomputers. Journal of Parallel an Distribute Computing, 6():7 774,.