mahines. HBSP enhanes the appliability of the BSP model by inorporating parameters that reet the relative speeds of the heterogeneous omputing omponen

Transcription

1 The Heterogeneous Bulk Synhronous Parallel Model Tiani L. Williams and Rebea J. Parsons Shool of Computer Siene University of Central Florida Orlando, FL Abstrat. Trends in parallel omputing indiate that heterogeneous parallel omputing will be one of the most widespread platforms for omputation-intensive appliations. A heterogeneous omputing environment oers onsiderably more omputational power at a lower ost than a parallel omputer. We propose the Heterogeneous Bulk Synhronous Parallel (HBSP) model, whih is based on the BSP model of parallel omputation, as a framework for developing appliations for heterogeneous parallel environments. HBSP enhanes the appliability of the BSP model by inorporating parameters that reet the relative speeds of the heterogeneous omputing omponents. Moreover, we demonstrate the utility of the model by developing parallel algorithms for heterogeneous systems. 1 Introdution Parallel omputers have made an impat on the performane of large-sale sienti and engineering appliations suh asweather foreasting, earthquake predition, and seismi data analysis. However, speial-purpose massively parallel mahines have proven to be expensive to build, diult to use, and have lagged in performane by taking insuient advantage of improving tehnologies. Heterogeneous omputing [8, 14] is a ost-eetive approah that avoids these disadvantages. A heterogeneous omputing environment an represent a diverse suite of arhiteture types suh as Pentium PCs, shared-memory multiproessors, and high-performane workstations. Unlike parallel omputing, suh an approah will leverage tehnologies that have demonstrated sustained suess, inluding: omputer networks; miroproessor tehnology; and shared-memory platforms. We propose a framework for the development of parallel appliations for heterogeneous platforms. Our model is alled Heterogeneous Bulk Synhronous Parallel (HBSP), whih is an extension to the BSP model of parallel omputation [17]. BSP provides guidane on designing appliations for good performane on homogeneous parallel mahines. Experiments [5] indiate that the model also aurately predits parallel program performane on a wide range of parallel

2 mahines. HBSP enhanes the appliability of the BSP model by inorporating parameters that reet the relative speeds of the heterogeneous omputing omponents. Our starting point for the development of algorithms for HBSP are eient BSP or HCGM [10, 11] appliations. Speially, we develop three HBSP algorithms prex sums, matrix multipliation, and randomized sample sort that distribute the omputational load aording to proessor speed without sariing performane. In fat, the ost model indiates that wall lok performane is inreased in many ases. Furthermore, these algorithms an exeute unhanged on both heterogeneous and homogeneous platforms. The rest of the paper proeeds as follows. Setion 2 reviews related work. Setion 3 desribes the HBSP model. Setion 4 presents a sampling of algorithms for HBSP. Conluding remarks and future diretions are given in Setion 5. 2 Related Work The theoretial foundations of the BSP model were presented in a series of papers by Valiant [15, 16, 17, 18, 19], whih desribe the model, how BSP omputers an be programmed either in diret mode or in automati mode (PRAM simulations), and how to onstrut eient BSP omputers. Other work presents theoretial results, empirial results, or experimental parameterization of BSP programs [1, 2, 3, 4, 5, 21]. Many alternative models of parallel omputation have been proposed in the literature a good survey on this topi are papers by Maggs, Matheson, and Tarjan [9] and Skilliorn and Talia [13]. Several models exist to support heterogeneous parallel omputation. However, they are either primarily of theoretial interest or are basially languages/runtime systems without a solid theoretial foundation. For an overview of these approahes, we refer the reader to the surveys by Siegel et al. [12] and Weems et al. [20]. One notable exeption is the the Heterogeneous Coarse-Grained Multiomputer (HCGM) model, developed by Morin [10, 11]. HBSP and HCGM are similar in struture and philosophy. The main dierene is that HGCM is not intended to be an aurate preditor of exeution times whereas HBSP attempts to provide the developer with preditable algorithmi performane. 3 Heterogeneous BSP The Heterogeneous Bulk Synhronous Parallel (HBSP) model is a generalization of the BSP model [17] of parallel omputation. The BSP model is a useful guide for parallel system development. However, it is inappropriate for heterogeneous parallel systems sine it assumes all omponents have equal omputation and ommuniation abilities. The goal of HBSP is to provide a framework that makes parallel omputing a viable option for heterogeneous systems. HBSP enhanes the appliability of BSP by inorporating parameters that reet the relative speeds of the heterogeneous omputing omponents. An HBSP omputer is haraterized by the following parameters:

3 { the number of proessor-memory omponents p labeled P 0 ; :::; P p,1 ; { the gap g j for j 2 [0::p, 1], a bandwidth indiator that reets the speed with whih proessor j an injet pakets into the network; { the lateny L, whih is the minimum duration of a superstep, and whih reets the lateny to send a paket through the network as well as the overhead to perform a barrier synhronization; { proessor parameters j for j 2 [0::p, 1], whih indiates the speed of proessor j relative to the slowest proessor, and { the total speed of the heterogeneous onguration = P p,1 i=0 i. For notational onveniene, P f (P s ) represents the fastest (slowest) proessor. The ommuniation time and the omputation speed of the fastest (slowest) proessor are g f (g s ) and f ( s ), respetively. We assume that s is normalized to 1. If i = j, then P i is j times faster than P s. Computation onsists of a sequene of supersteps. During a superstep, eah proessor performs asynhronously some ombination of loal omputation, message transmissions, and message arrivals. A message sent in one superstep is guaranteed to be available to the destination proessor at the beginning of the next superstep. Eah superstep is followed by a global synhronization of all the proessors. Exeution time of an HBSP omputer is as follows. Eah proessor, P j, an perform w i;j units of work in w i;j j time units during superstep i. Let w i = max( w i;j j ) represent the largest amount of loal omputation performed by any proessor during superstep i. Let h i;j be the largest number of pakets sent or reeived by proessor j in superstep i. Thus, the exeution time of superstep i is: w i + max fg j h i;j g + L (1) j2[0::p,1] The overall exeution time is the sum of the superstep exeution times. The HBSP model leverages existing BSP researh. The more omplex ost model does not hange the basi programming methodology, whih relies on the superstep onept. Furthermore, when j = 1 and g j = g k, where 0 j; k < p, HBSP is equivalent to BSP. 4 HBSP Algorithms This setion provides a sampling of appliations for the HBSP model based on those proposed by Morin for the HCGM model [10, 11]. Our algorithms, whih inlude prex sums, matrix multipliation, and randomized sample sort, illustrate the power and elegane of the HBSP model. In eah of the appliations, the input size is partitioned aording to a proessor's speed. If i is the speed of proessor P i, then P i holds i n input elements. When disussing the performane of the algorithms, we will often make use of a oarse-grained assumption, p n, i.e., the size of the problem is signiantly larger than the number of proessors. Our interpretation of \signiantly larger" is p n p.

4 4.1 Prex Sums Given a sequene of n numbers fx 0 ;x 1 ; :::; x n,1 g, it is required to ompute their prex sums s j = x 0 + x 1 + ::: + x j, for all j, 0 j n, 1. Under HBSP, eah proessor loally omputes its prex sums and sends the total sum to P f. Next, P f omputes the prex sums of this sequene and sends the (i, 1)st element of the prex to P i. Lastly, P i adds this value to eah element of the prex sums omputed in the rst step to obtain the prex sums of the overall result. The prex sums algorithm is shown below. 1. Eah proessor loally omputes the prex sums of its i n input elements. 2. Eah proessor, P i, sends the total sum of its input elements to P f. 3. P f omputes the prex sums of the p elements reeived in Step For 1 i p, 1;P f sends the (i, 1)st element omputed in Step 3 to P i. 5. Eah proessor omputes its nal portion of the prex sums by adding the value reeived in Step 4 to eah of the values omputed in Step 1. Analysis. In Step 1 and Step 5, eah proessor P i does O( i n)work and this an be done in O( n ) time. Steps 2 and 4 require a ommuniation time of maxfg s 1;g f pg. Step 3 takes O( p f ) omputation time. Sine f p and p n p, O( p f ) O( n ). Thus, the algorithm takes time O( n )+2 maxfg s 1;g f pg +3L: (2) If g s pg f, the ommuniation time is 2pg f, otherwise it's 2g s. 4.2 Matrix Multipliation Matrix multipliation is perhaps one of the most ommon operations used in large-sale sienti omputing. Given two n n matries A and B, we dene the matrix C = A B as C i;j = P n,1 k=0 A i;k B k;j : We assume that matrix A is partitioned among the proessors so that eah proessor, P i, holds i n rows of A and n p olumns of B. At the ompletion of the omputation, P i will hold i n rows of C. We denote the parts of A, B, and C held by P i as A i ;B i ; and C i, respetively. The matrix multipliation algorithm onsists of irulating the olumns of B among the proessors. When P i reeives olumn j of B, it an ompute olumn j of C i. One P i has seen all olumns of B, it will have omputed all of C i. The matrix multipliation algorithm is given below. 1. repeat p times. 2. P i omputes C i = A i B i. 3. P i sends B i to P (i+1)mod p. 4. end repeat

5 Analysis Step 3 requires P i to perform O( i n n p n) = O( n3 i p ) amount of work. Over p rounds, the total omputation time is O( n3 ). During Step 4, eah proessor sends and reeives n p olumns of matrix B. Therefore, the total time of HBSP matrix multipliation is O( n3 )+g sn 2 + pl: (3) 4.3 Randomized Sample Sort One approah for parallel sorting that is suitable for heterogeneous omputing is randomized sample sort. It is based on the seletion of a set of p, 1 \splitters" from a set of input keys. In partiular, we seek splitters that will divide the input keys into approximately equal-sized bukets. The standard approah is to randomly selet pr sample keys from the input set, where r is alled the oversampling ratio. The keys are sorted and the keys with ranks r; 2r; :::; (p, 1)r are seleted as splitters. By hoosing a large enough oversampling ratio, it an be shown with high probability that no buket will ontain many more keys than the average [7]. One proessors gain knowledge of the splitters, their keys are partitioned into the appropriate buket. Afterwards, proessor i loally sorts all the keys in buket i. When adapting this algorithm to the HBSP model, we hange the way in whih the splitters are hosen. To balane the work aording to the proessor speeds 0 ; :::; p,1, it is neessary that O( i n)keys fall between s i and s i+1. This leads to the following algorithm. 1. Eah proessor randomly selets a set of r sample keys from its i n input keys. 2. Eah proessor, P i, sends its sample keys to P f. 3. P f sorts the pr sample keys. Denote these keys by sample 0 ; :::; sample pr,1 where sample i is the sample key with rank i in the sorted order. P f denes p, 1 splitters, s 0 ; :::; s p,2, where s i = sample P i j d( j=0 )pre. 4. P f broadasts the p, 1 splitters to eah of the proessors. 5. All keys assigned to the ith buket are sent to the ith proessor. 6. Eah proessor sorts its buket. Analysis In Step 1, eah proessor performs O(r) O(n) amount ofwork. This requires O( n s ) time. Step 2 requires a ommuniation time of maxfg s r;g f prg. To sort the pr sample keys, P f does O(pr lg pr) O(n lg n) amount of work. This an be done in O( n f lg n) time. Broadasting the p, 1 splitters requires maxfg s (p,1);g f p(p,1)g ommuniation time. Sine eah proessor is expeted to reeive approximately i n keys [11], Step 5 uses O( n ) omputation time and maxfg i i ng ommuniation time, where i 2 [0::p, 1]. One eah proessor

6 reeives their keys, sorting them requires O( n lg n) time. Thus, the total time is n n o i O lg n + X(r +(p, 1)) + max g i f i2[0::p,1] n +4L; where (4) 8 pg >< f if g s pg f X = g >: s otherwise. 5 Conlusions and Future Diretions The HBSP model provides a framework for the development of parallel appliations for heterogeneous platforms. HBSP enhanes the appliability of BSP by inorporating parameters that reet the relative speeds of the heterogeneous omputing omponents. Although the HBSP model is somewhat more omplex than BSP, it aptures the most important aspets of heterogeneous systems. Existing BSP and HCGM algorithms provide the foundation for the HBSP algorithms presented here. These algorithms suggest that improved performane under HBSP results from utilizing the proessor speeds of the underlying system. However, experimental evidene is needed to orroborate this laim. We plan to extend this work in several diretions. First, a library based on BSPlib (a small, standardized library of BSP funtions) [6] will provide the foundation for HBSP programming. Experiments will be onduted to test the eetiveness of the model on a network of heterogeneous workstations. These experiments will test the preditability, salability, and eieny of appliations written under HBSP. Currently, the HBSP model only addresses a heterogeneous olletion of uniproessor mahines. We are investigating variants to the model to address multiproessor systems. In onlusion, the goal of HBSP is to oer a framework that makes parallel omputing a viable option for a wide range of tasks. We seek to demonstrate that it an provide a simple programming approah, portable and eient appliation ode, preditable exeution, and salable performane. Referenes [1] R. H. Bisseling. Sparse matrix omputations on bulk synhronous parallel omputers. In Proeedings of the International Conferene on Industrial and Applied Mathematis, Hamburg, July [2] R. H. Bisseling and W. F. MColl. Sienti omputing on bulk synhronous parallel arhitetures. In B. Pehrson and I. Simon, editors, Proeedings of the 13th IFIP World Computer Congress, volume 1, pages 509{514. Elsevier, [3] A. V. Gerbessiotis and C. J. Siniolakis. Deterministi sorting and randomized mean nding on the BSP model. In Eighth Annual ACM Symposium on Parallel Algorithms and Arhitetures, pages 223{232, June [4] A. V. Gerbessiotis and L. G. Valiant. Diret bulk-synhronous parallel algorithms. Journal of Parallel and Distributed Computing, 22(2):251{267, August 1994.

7 [5] M. W. Goudreau, K. Lang, S. Rao, T. Suel, and T. Tsantilas. Towards eieny and portability: Programming with the BSP model. In Eighth Annual ACM Symposium on Parallel Algorithms and Arhitetures, pages 1{12, June [6] J. M. D. Hill, B. MColl, D. C. Stefanesu, M. W. Goudreau, K. Lang, S. B. Rao, T. Suel, T. Tsantilas, and R. Bisseling. BSPlib: The BSP programming library. Parallel Computing, 24(14):1947{1980, [7] J. Huang and Y. Chow. Parallel sorting and data partitioning by sampling. In IEEE Computer Soiety's Seventh International Computer Software & Appliations Conferene (COMPSAC'83), pages 627{631, November [8] A. Khokhar, V. Prasanna, M. Shaaban, and C. Wang. Heterogeneous omputing: Challenges and opportunities. Computer, 26(6):18{27, June [9] B. M. Maggs, L. R. Matheson, and R. E. Tarjan. Models of parallel omputation: A survey and synthesis. In Proeedings of the 28th Hawaii International Conferene on System Sienes, volume 2, pages 61{70. IEEE Press, January [10] P. Morin. Coarse-grained parallel omputing on heterogeneous systems. In Proeedings of the 1998 ACM Symposium on Applied Computing, pages 629{634, [11] P. Morin. Two topis in applied algorithmis. Master's thesis, Carleton University, [12] H. J. Siegel, H. G. Dietz, and J. K. Antonio. Software support for heterogeneous omputing. In A. B. Tuker, editor, The Computer Siene and Engineering Handbook, pages CRC Press, [13] D. B. Skilliorn and D. Talia. Models and languages for parallel omputation. ACM Computing Surveys, 30(2):123{169, June [14] L. Smarr and C. E. Catlett. Metaomputing. Communiations of the ACM, 35(6):45{52, June [15] L. G. Valiant. Optimally universal parallel omputers. Philosophial Transations of the Royal Soiety of London, A 326:373{376, [16] L. G. Valiant. Bulk-synhronous parallel omputers. In M. Reeve and S. E. Zenith, editors, Parallel Proessing and Artiial Intelligene, pages 15{22. John Wiley & Sons, Chihester, [17] L. G. Valiant. A bridging model for parallel omputation. Communiations of the ACM, 33(8):103{111, [18] L. G. Valiant. General purpose parallel arhitetures. In J. van Leeuwen, editor, Handbook of Theoretial Computer Siene, volume A: Algorithms and Complexity, hapter 18, pages 943{971. MIT Press, Cambridge, MA, [19] L. G. Valiant. Why BSP omputers? In Proeedings of the 7th International Parallel Proessing Symposium, pages 2{5. IEEE Press, April [20] C. C. Weems, G. E. Weaver, and S. G. Dropsho. Linguisti support for heterogeneous parallel proessing: A survey and an approah. In Proeedings of the Heterogeneous Computing Workshop, pages 81{88, [21] T. L. Williams and M. W. Goudreau. An experimental evaluation of BSP sorting algorithms. In Proeedings of the 10th IASTED International Conferene on Parallel and Distributed Computing Systems, pages 115{118, Otober 1998.