Improving Performance of Sparse Matrix-Vector Multiplication

Transcription

1 Improving Performance of Sparse Matrix-Vector Multiplication Ali Pınar Michael T. Heath Department of Computer Science an Center of Simulation of Avance Rockets University of Illinois at Urbana-Champaign Abstract Sparse matrix-vector multiplication (SpMxV) is one of the most important computational kernels in scientific computing. It often suffers from poor cache utilization an extra loa operations because of memory inirections use to exploit sparsity. We propose alternative ata structures, along with reorering algorithms to increase effectiveness of these ata structures, to reuce the number of memory inirections. Toleo propose hanling the x blocks of a matrix separately, oing only one inirection for each block. We propose packing all contiguous nonzeros into a block to reuce the number of memory inirections further. This reuces memory inirections per block to one for the cost of an extra array in storage an a loop uring SpMxV. We also propose an algorithm to permute the nonzeros of the matrix into contiguous locations. We state this problem as the traveling salesperson problem an use associate heuristics. Experiments verify the effectiveness of our techniques. Introuction One of the most important computational kernels in scientific computing is multiplying a sparse matrix by a vector. Many algorithms use sparse matrix-vector multiplication (SpMxV) in their inner loop (iterative solvers for systems of linear equations being ust one example). The repeate execution of this operation potentially amortizes the cost of a preprocessing phase, which might lea to computational savings in subsequent executions. The importance of the SpMxV operation has mae this challenging problem the subect of numerous research efforts. Data structures use for sparse matrices usually have two components: (i) an array that stores all the floating-point entries of the matrix, (ii) arrays that store the sparsity structure of the matrix, i.e., pointers to the locations of the floating-point entries in the matrix. To exploit the sparsity of the matrix the use of pointers is unavoiable but often limits the memory system performance. One reason for this is that pointers usually lea to poor cache utilization, since they lack spatial locality. The number of cache misses for the right- an/or left-han-sie vectors can ramatically increase if the sparse matrix has an irregular sparsity structure. Another important factor is that memory inirections (pointers) require extra loa operations. In sparse matrix operations, the number of floating-point operations per loa operation is lower than that of ense matrix operations, limiting overall performance. Research supporte by the Center for Simulation of Avance Rockets, fune by the U.S. Department of Energy through the University of California uner subcontract number

2 * % We propose alternative ata structures, as well as reorering algorithms to increase the effectiveness of those ata structures, to reuce the number of memory inirections in SpMxV. Toleo [7] propose ientifying blocks of a matrix an writing the matrix as the sum of two matrices, the first of which contains all the blocks an the secon contains the remaining entries. Thus, the number of memory inirections is reuce to only one for each block. In an alternative scheme, we pack all the nonzeros in contiguous locations into a block to reuce further the number of memory inirections, because only one memory inirections is require for all the nonzeros in a block. However, we then nee to know how many nonzeros are in each block, which requires an extra array an an extra loop uring SpMxV. This work concentrates on the latter problem. We also propose reorering the matrices to increase the effectiveness of our ata structures. The obective of reorering is to permute the nonzeros of the matrix into contiguous locations, as much as possible, to enlarge the ense blocks. We can show that this problem is NP-Complete, an thus heuristics must be use for a practical solution. We propose a graph moel to reuce the problem to the well-stuie traveling salesperson problem, an thus we can use heuristics esigne for that problem. We verify experimentally the effectiveness of the propose ata structures an algorithms. Our new ata structures an reorering techniques prouce improvements of up to 33% an improvements of % on average. The remainer of this paper is organize as follows. Section iscusses the shortcomings of current sparse matrix ata structures an proposes new alternatives. Section 3 states a new matrix reorering algorithm to permute the nonzeros of the matrix into contiguous locations. Experimental results are presente in Section 4, an finally we conclue with Section 5. Sparse Matrix Data Structures The most wiely use sparse matrix storage scheme is Compresse Row Storage (CRS). As its name implies, this scheme stores the sparse matrix as a sequence of compresse rows. Three arrays are employe:, an. The nonzeros of the sparse matrix are compresse into an array! in a rowwise "#$ &% (' manner. The column inex of each nonzero entry is store in the array, i.e., is the column % (' )& inex of the nonzero entry store in!. Finally, Figure presents a SpMxV algorithm using compresse row storage. stores the inex of the first nonzero of each row. +* for to, o - % ('.*0/ % (' for - % '.* - % ('74! to )3% 54 '6 '8:9%< 5% '=' o Figure : SpMxV algorithm in compresse row storage

3 FFFFE 4 FFFFE FFFFE 9 Sparse matrix-vector multiplication (SpMxV) algorithms ten to suffer from poor memory performance. One reason for this is ineffective cache utilization. Temporal locality is limite to right- an left-han-sie )3 vectors an array. No temporal locality is present in arrays! "#$ an. In CRS, there is goo temporal locality in the left-han-sie vector -, but the access pattern for the right-han-sie vector can be quite irregular, causing excessive cache misses. Reorering the matrix to reuce cache misses was propose by Das. et al. [3], who suggeste reucing the banwith of the matrix. Temam an Jalby [6] analyze the number of cache misses as a function of the banwith for various cache parameters. urgess an iles [] experimente with various orering algorithms an foun that reorering the matrix improves performance compare with ranom orering, but they i not etect a notable sensitivity to the particular orering metho use. Toleo [7] stuie reorering, along with other techniques, an reporte that Reverse Cuthill-Mcee orering yiele slightly better performance, but the ifferences were not significant. Another problem with SpMxV is that the ratio of loa operations is higher than with ense matrix operations. One extra loa operation is require to fin the column inex of each nonzero entry for each multiply-a operation. The innermost statement in the algorithm in Figure requires three loa operations for two floating-point operations. This not only increases the total number of loa instructions, but also can cause the loa units to be a bottleneck, especially in recent architectures as iscusse in [7]. The following two sections present ata structures that can ecrease the number of memory inirections uring SpMxV operation.. Fixe-Size locking In this approach, the matrix is written as sum of several matrices, some of which contain the ense blocks of the matrix with a prespecifie size. For instance, given a matrix, we can ecompose it into two matrices?>$@ an ">A>, such that CC?>$@?>A>, where ">$@ contains the? D ense blocks of the matrix an?>a> contains the remainer. An example is illustrate in Figure. A simple greey algorithm is sufficient to 9 9 HJI III = HJI III + 9 =?>$@ + ">A> HJI III Figure : Fixe size blocking with? blocks? L MN extract the maximum number of blocks in a matrix, where. However, the problem is more ifficult for O blocks for OQP. Exploiting ense blocks can reuce the number of loa operations, as well as the total memory requirement, because only one inex per block is require. Moreover, the entry of the right-han-sie vector 9 can

4 FFFFFE be use multiple times as oppose to once in the conventional scheme after a loa operation. A similar problem has been stuie in the context of vector processors [], but those efforts concentrate on fining fewer blocks of larger size, whereas we are intereste in much smaller blocks.. locke Compresse Row Storage In this section we propose a new sparse matrix storage scheme esigne to reuce the number of loa operations. The iea of this scheme is to exploit the nonzeros in contiguous locations by packing them. Unlike fixe-size blocking, the blocks will have variable lengths. This enables longer nonzero strings to be packe into a block. As in fixe-size blocking, if we know the column inex of the first nonzero in a block, then we will also know the column inices of all its other nonzeros. In other wors, only one memory inirection (extra loa operation) is require for each block. 3 This storage scheme requires an array RTS (of length the number of blocks) in aition to the other three arrays use in CRS: a floating-point array! (of length the number of nonzeros) to store the nonzero values, an array (of length the number of blocks) to store the column number of the first nonzero for )& each block, an an array (of length the number of rows) to point to the position where the blocks 3 of each row start. RTS stores the location of the first nonzero of each block in array!. We refer to this storage scheme as blocke compresse row storage (CRS). Figure 3 presents an example of CRS, an the SpMxV operation using this storage is presente in Figure 4. U VXWY W[Z\W / / / W / XW^]XW / XẀ _aw / / / / Ẁ ]XW / / b W / / ]XW HJI IIII = c VXW=e W=fZ\W=e W=J XW=J]XWgJ XWhA_W=e iw=j]7w= b W=f]XWk = c J XA_aJ XJ]XJ XJVl m3 = c J XA_aJVX b Jnl 3 RTS = c A_aJVXfZ\JoXe i e e ae p]l Figure 3: Example of blocke compresse row storage This storage scheme reuces extra loa operations but requires an extra loop uring the SpMxV operation an thus suffers aitional loop overhea. If the sizes of the blocks are small, the overhea ue to the extra loop will ominate the gain ue to ecrease loa operations. Thus, the effectiveness of this storage scheme epens irectly on the sizes of the blocks in the matrix. The total volume of storage has a similar traeoff. Two numbers (one in "#$ an one in RTS store for each block in the matrix, as oppose to one number for each nonzero. Thus, if the blocks are large enough, the total storage size can be significantly reuce. 3 Reorering to Enlarge Dense locks The previous section escribe our ata structures to exploit ense blocks of a matrix. The effectiveness of these ata structures epen irectly on the availability of ense blocks. In this section we escribe 3 ) are

5 /, / * *, ' % O ' > +* for to, o - % ('.*0/ % (' )3% 54 for to '6 q srtsuvw*x"5% +*y/ 3% 3% 4 for O RTS to RTS '&6 - % '.* - % ('74 '89z%! q srtsuva4{ ' +*yw4 o o Figure 4: SpMxV with blocke compresse row storage reorering algorithms to enlarge ense blocks of a matrix. First, we will formally efine the problem an propose a graph moel to reuce the problem to the traveling salesperson problem (TSP). Then we will iscuss briefly heuristics we use for solving TSP. 3. Problem Formulation Our obective in reorering the matrix is to increase the sizes of the ense blocks in a row, i.e., to align the nonzeros in a row in consecutive locations as much as possible. This requires reorering of columns an is not affecte by orering of rows. Thus, rows of the matrix can be reorere without isturbing the aligne nonzeros within a row (e.g., the same reorering can be applie to rows to preserve symmetry). These techniques can also be use to align nonzeros within a column by interchanging the roles of columns an rows. A formal escription of the problem follows. iven an, satisfying r ~< matrix }c r ~< ƒ an > r\9 St ˆ cac r\~, i.e.,, fin an orering of columns to maximize the number of c Š MŒ:M e M L Š r ~ /Žr\ r ~<fƒ / fw pairs In the Appenix, we show that the problem is NP-Complete by using reuction from Hamiltonian Path problem. Thus, we must resort to heuristics for a practical solution. Such an orering problem is naturally close to TSP. However, we are looking for a path, not a tour (cycle) of vertices. Although this is a slightly ifferent version of the problem, heuristics esigne to fin a tour can still be use or aapte to fin a path of vertices. Thus, our strategy will be first to efine a graph moel that reuces our matrix reorering problem to the well-stuie TSP an use heuristics alreay esigne for TSP to reorer our matrices. For the sake of presentation, we will refer to the maximization version of TSP, i.e., iven a graph } c an a weighting function on its eges, fin a tour X> W W W aš š œ >J to maximize

6 FFFFFE, > FFFFFE ž / Ÿ ] _ HJI 9 IIII ž / _ ] 9 Figure 5: Ege weight computation. raph shows ege weights of matrix on left-han sie. Matrix after reorering is presente on right. H I IIII Notice that the weight of the ege c 7š )š ~ š +š c ~ ~hƒ fw is not inclue, since we nee a path, not a tour. Since we are trying to reorer columns of the matrix, the graph that reuces the matrix reorering problem to TSP has vertices representing the columns of the matrix. The weight of an ege between two vertices is efine as the number of rows where both respective columns have a nonzero, i.e., c Ž c Š MŒ)M Š r ~ / rt r ~ ª An alternative efinition, which also gives the basics of an algorithm, is as follows. Let «w T c q ~< matrix of zeros an ones with the same nonzero structure as, i.e., q ~< x r ~< ~< Let Ž««. The weight function can be efine by the matrix c. Figure 5 / illustrates an example of ege weight computation. In this example the weight of the c e p ege is equal to because columns 0 an share a nonzero only in the first row, whereas the weight of the c A_\ ege is equal to because columns an 4 share nonzeros in the first an secon rows. / W / as c Notice that although a weight (possibly zero) is assigne to any pair of vertices in the graph (any orering is a feasible solution for the matrix reorering problem), is expecte to be sparse, which must be exploite for the sake of efficiency (or even existence) of a practical solution. If the vertices an are in consecutive locations in the TSP solution, then the respective columns will be in consecutive locations in the orere matrix, an c nonzeros will be in contiguous locations. The larger the weights of the eges in the tour, the greater the number of nonzeros in contiguous locations will be. In general, fining a tour with maximum weight in this graph correspons to fining an orering with maximum number of contiguous nonzeros. To be more precise, the number of blocks (as in CRS scheme) in the reorere matrix is equal to the number of nonzeros in the matrix minus the total weight of the TSP tour, since each unit of weight correspons to locating a nonzero aacent to another one, thus ecreasing the eª be a

7 V n number of blocks by one. Consequently, a tour with maximum weight escribes an orering with minimum number of blocks. In the example of Figure 5, the matrix has 3 nonzeros. TSP solution use to reorer the matrix is /± ± ± _ ± ] matrix is e] 6 n, the total weight of which is The number of blocks in the reorere 3. Heuristics for TSP As in many other combinatorial optimization problems, heuristics for TSP can be classifie into two groups: constructive an improvement. Constructive heuristics irectly construct a solution for the problem, whereas improvement heuristics start with a given solution an try to improve that solution by searching the neighborhoo. A comprehensive iscussion of TSP solution methos can be foun in [4, 5]. In this work, we aopte heuristics from the literature instea of esigning our own. Here we iscuss briefly the heuristics use in our experiments. Since matrix reorering is propose as a preprocessing step, we focuse on computationally efficient heuristics. The simplest constructive heuristic is to use the initial orering of the matrix. Matrices often have some natural orering that can be use as an initial solution. We also use a vertex insertion proceure. The process starts with an initial path of one ranom vertex, an vertices are inserte one by one to the path until all are inclue in the path. At each step, a vertex that is not in the current path is ranomly chosen an inserte into the path, so that the sum of weights of eges to its successor an preecessor is maximum. The final constructive heuristic we use is to start with a ranom vertex as the initial path an procee by inserting vertices at the en of the path. At each step, the vertex that is connecte to the last vertex of the current path with the heaviest ege weight is inserte at the en of the path. Improvement heuristics have the same flavor as constructive heuristics. One of the heuristics we use epens on vertex insertion. ut this time a vertex is first remove from the current path, unlike the case in constructive heuristics, an then reinserte to maximize the total weight of the tour. We also use ege or path reinsertion proceures, which are similar to vertex insertion. In these heuristics, eges (paths of two vertices) or longer paths are reinserte instea of vertices (name Or-opt proceure in [4]). 4 Experimental Results We implemente the new ata structures an reorering algorithms in C an compare their performances with that of conventional compresse row storage (CRS) scheme. We experimente with a collection of matrices selecte from the Harwell-oeing sparse-matrix collection. All experiments were performe on a Sun Enterprise In the first set of experiments, we z " investigate the effectiveness of our orering techniques. Figure 6 presents our experimental results with blocke matrices after Reverse Cuthill-Mcee (RCM) orering,

8 using the initial orering, an TSP orering (escribe in Section 3). TSP solutions are generate by starting with the initial orering an then using a vertex insertion proceure to improve the initial solution. In Figure 6: Effectiveness of orering Figure 6, SpMxV times are normalize with respect to SpMxV times with the conventional CRS scheme. ² blocking after TSP orering always improves performance an is always superior to the other two orering methos. The ifference becomes very significant for w89 an bcsstk8. On average, TSP orering reuces the runtime of a SpMxV operation to 79% of that of the conventional scheme, whereas RCM orering an using the initial orering are limite to only, 95% an 9%, respectively. Figure 7 presents our results with the two new ata structures. Again, SpMxV times are normalize with respect to SpMxV times with the conventional CRS scheme. Fixe-size blocking is superior to CRS for most matrices an on average. CRS outperforms fixe-size blocking for four matrices: bcsstk8, cavity6, bcsstk30, an bcsstk3. We also experimente with minor moifications of the ata structures. For instance, using? ] blocks instea of " ³ increases the performance by 3% on average, but further increasing the block size oes not help. Using blocke compresse row storage only for blocks of size greater than an using the conventional scheme for the remainer increase the performance by 8%, compare with blocke compresse row storage. We are in the process of tuning our ata structures for maximum performance. The results show that significant savings can be achieve by exploiting the ense blocks in a matrix. Reorering the matrices yiels notable improvement in performance, offsetting the cost of the preprocessing phase, which is often amortize over repeate SpMxV operations with the same matrix.

9 5 Conclusions an Future Work Figure 7: Comparison of ata structures We have presente new ata structures, along with a matrix reorering algorithm to increase the effectiveness of those ata structures, esigne to ecrease the number of memory inirections uring sparse matrix-vector multiplication. The ata structures exploit ense blocks nonzeros in contiguous locations of the matrix. We also propose a reorering algorithm to enlarge the ense blocks of a matrix. Experiments verify the effectiveness of propose techniques, with observe performance improvement of up to 33% an an average improvement of %. Currently we are working on improving the performance of the propose techniques. We are also investigating alternative ways of exploiting the ensity in the sparse matrices by esigning aitional new ata structures. Finally, we plan to experiment with various architectures to observe the performance of these techniques with a variety of architectural parameters. References [] R. C. Agarwal, F.. ustavson, an M. Zubair, A high performance algorithm using pre-processing for sparse matrix vector multiplication, Proceeings of Supercomputing 9, pp [] D. A. urgess an M.. iles, Renumbering unstructure gris to improve the performance of coes on hierarchical memory machines, Tech. Rep. 95/06, Numerical Analysis roup, Oxfor University Computing laboratory, May 995. [3] R. Das, D. J. Mavriplis, J. Saltz, S. upta, an R. Ponnusamy, The esign an implementation of a parallel unstructure Euler solver using software primitives, AIAA Journal, No: 3, pp , 994. [4] J. Perttunen, The traveling salesman problem: the significance of initial solutions in composite heuristics, Vaasan orkeakoulun ulkaisua, no. 39, 989.

10 c / / [5]. Reinelt The traveling salesman: computational solutions for TSP applications, Lecture Notes in Computer Science, Vol: 840, Springer-Verlag, 994. [6] O. Temam an W. Jalby, Characterizing the behavior of sparse algorithms on caches, Proceeings of Supercomputing 9, 99. [7] S. Toleo, Improving Memory-System Performance of Sparse Matrix-Vector Multiplication, Proceeings of the 8th SIAM Conference on Parallel Processing for Scientific Computing, March 997. Appenix NP-Completeness of Problem In this section, we prove that orering a sparse matrix to increase sizes of ense blocks is NP-Complete by using reuction from the Hamiltonian Path problem. First we state a ecision version of the problem: iven an, r ~< matrix ^c, ecie if there exists an orering of columns, where the number of pairs satisfying r ~< iven a simple graph c r ~# ƒ an > is greater than or equal to a given boun µ. (no loops, no parallel eges), construct an matrix. In this matrix each row represents an ege an each column represents a vertex in the graph. Let the th column represent the th vertex. The nonzero structure matrix rj~ is efine such that there are nonzeros at an r J¹ ~ for each ege c ¹. To increase the size of ense blocks, we have to bring the columns of aacent vertices to consecutive locations in the reorere matrix. Notice that two aacent columns can share nonzeros in at most one row, 6 because there are no parallel eges. There can be at most º ² blocks of size after reorering, achieve when the vertices of consequent columns share an ege in the graph, which efines a Hamiltonian path in the graph. It is also easy to see the problem is in NP, since a given solution can be verifie in polynomial time. So we can conclue that the problem is NP-Complete.