Speed-up of Parallel Processing of Divisible Loads on k-dimensional Meshes and Tori

Transcription

1 The Computer Journal, 46(6, c British Computer Society 2003; all rights reserved Speed-up of Parallel Processing of Divisible Loads on k-dimensional Meshes Tori KEQIN LI Department of Computer Science, State University of New York, New Paltz, NY 12561, USA lik@newpaltzedu A divisible load distribution algorithm on k-dimensional meshes tori is proposed analyzed It is found that by using our algorithm, the speed-up of parallel processing of a divisible load on k-dimensional meshes tori is bounded from above by a quantity independent of network size, due to communication overhead limited network connectivity In particular, it is shown that for k-dimensional meshes tori, as the network size becomes large, the asymptotic speed-up of processing divisible loads with corner initial processors is approximately β 1 1/2k,whereβ is the ratio of the time for computing a unit load to the time for communicating a unit load It is also proved that by choosing interior initial processors, an asymptotic speed-up of 2 k β 1 1/2k can be achieved Received 29 October 2002; revised 13 February INTRODUCTION A divisible load has the property that it can be arbitrarily divided into small load fractions which are assigned to parallel processors for processing Example applications include large-scale data file processing, signal image processing, scientific numerical computing, finiteelement engineering computations, database multimedia applications, many real-time computing problems such as target identification, searching data processing in distributed sensor networks [1] The problem of divisible load distribution on parallel distributed computing systems with static interconnection networks was first proposed by Cheng Robertazzi in 1988 [2] Since then, divisible load distribution, scheduling processing have been investigated by a number of researchers for the bus [3, 4], linear array [5], tree [3, 6, 7], two-dimensional mesh [8], two-dimensional toroidal mesh [9], three-dimensional mesh [10], hypercube [11] partitionable [12, 13] networks Other studies can be found in [14, 15, 16, 17] 1 The well-known Amdahl s Law [18] states that if fraction f of a computation is sequential cannot be parallelized at all, the speed-up is bounded from above by 1/f no matter how many processors are used For a divisible load, there is no inherently sequential part, that is f 0 However, this does not imply that unbounded speed-up can be achieved The reason is that Amdahl s Law has no restriction on a parallel system, where processors can communicate with each other without cost When a divisible load is processed on a multicomputer with a static interconnection 1 The reader is also referred to the Website tom/dlthtml for more references in this field network, there is communication overhead for distributing load among the processors Also, the network topology, that determines the speed at which a divisible load is distributed over a network, has a strong impact on performance (ie parallel processing time speed-up In this paper, we propose a divisible load distribution algorithm on k-dimensional meshes tori analyze the parallel time speed-up of the algorithm We derive a recurrence relation so that the ultimate parallel processing time asymptotic speed-up can be easily calculated for k-dimensional meshes tori It is found that by using our algorithm, the speed-up of parallel processing of a divisible load on k-dimensional meshes tori is bounded from above by a quantity independent of network size, due to communication overhead limited network connectivity In particular, it is shown that for k-dimensional meshes tori, as the network size becomes large, the asymptotic speed-up of processing divisible loads with corner initial processors is approximately β 1 1/2k,where β is the ratio of the time for computing a unit load to the time for communicating a unit load We also prove that by choosing interior initial processors, an asymptotic speed-up of 2 k β 1 1/2k can be achieved [19] The significance of our research is three-fold First, these results include the earlier results for linear arrays two-dimensional meshes in [13] as special cases Second, this paper provides a unified treatment for divisible load distribution on k-dimensional meshes tori for all k 1 Third, divisible load distribution on k-dimensional meshes tori with k>3has never been addressed before our work gives an initial investigation on these networks

2 626 K LI 2 THE MODEL We consider parallel processing of a divisible load on a multicomputer system with N processors P 1,P 2,,P N connected by a static interconnection network Each processor P i has n i neighbors It is assumed that P i has n i separate ports for communication with each of the n i neighbors; ie processor P i can send messages to all its n i neighbors simultaneously Once a processor sends a load fraction to a neighbor, it can proceed with other computation communication activities This provides the capability to overlap computation communication enhances the system performance However, a neighbor (receiver must wait until a load fraction arrives before it starts to process the load fraction It is this waiting time that limits the overall system performance Let T cm be the time to transmit a unit load along a link The time to send a load to a neighbor is proportional to the size of the load, with a negligible communication startup time Let be the time to process a unit load on a processor Again, the computation time is proportional to the size of a load We use β /T cm to denote the computation granularity, which is a parameter indicating the nature of a parallel computation a parallel architecture A large (small β gives a small (large communication overhead A computation intensive load has a large β a communication intensive load has a small β An infinite β implies that the communication cost is negligible 3 THE LOAD DISTRIBUTION ALGORITHM A k-dimensional mesh M k can be specified by k 1 positive integers N 1,N 2,,N k,wheren r 2 is the size in the rth dimension of the mesh, 1 r k M k has a set of N N 1 N 2 N k processors, P k {P j1,j 2,,j k 1 j r N r, 1 r k} Each processor P j1,j 2,,j k has neighbors P j1,,j r ±1,,j k if they exist Assume that there is a load x initially on processor P N1,N 2,,N k, called the initial processor The load is to be distributed over all the N processors of a k- dimensional mesh for parallel processing We now describe our algorithm A N1,N 2,,N k for processing divisible loads on a k-dimensional mesh M k of size N N 1 N 2 N k For notational convenience, a single processor is treated as a zero-dimensional mesh M 0 with a one-processor set P 0 Our algorithm A N1,N 2,,N k for processing a divisible load x on a k-dimensional mesh M k works as follows (A1 When N 1, the single processor processes load x by itself (A2 In general, when N > 1, the initial processor P N1,N 2,,N k sends a faction α of the load x to processor P N1,N 2,,N k 1 The remaining load (1 αx is processed by the (k 1-dimensional mesh M k 1 of size N 1 N 2 N k 1 with the set of processors P k 1 {P j1,j 2,,j k 1,N k 1 j r N r, 1 r k 1} by using the load distribution algorithm A N1,N 2,,N k 1 (A3 If N k > 2, processor P N1,N 2,,N k 1 is regarded as the initial processor of the k-dimensional mesh M k of size N 1 N 2 (N k 1 with the set of processors P k {P j 1,j 2,,j k 1 j r N r, 1 r k 1, 1 j k N k 1} Upon receiving the load αx by processor P N1,N 2,,N k 1, thek-dimensional mesh M k processes the load αx by using the load distribution algorithm A N1,N 2,,N k 1 (A4 If N k 2, processor P N1,N 2,,N k 1 is regarded as the initial processor of the (k 1-dimensional mesh M k 1 of size N 1 N 2 N k 1 with the set of processors P k 1 {P j 1,j 2,,j k 1,N k 1 1 j r N r, 1 r k 1} Upon receiving the load αx by processor P N1,N 2,,N k 1, the (k 1-dimensional mesh M k 1 processes the load αx by using the load distribution algorithm A N1,N 2,,N k 1 A k-dimensional torus is similar to a k-dimensional mesh except that each processor P j1,j 2,,j k has neighbors P j1,,(j r ±1 mod N r,,j k Since a k-dimensional torus contains a k-dimensional mesh as its subnetwork, algorithm A N1,N 2,,N k is also applicable for load distribution on k-dimensional tori 4 PARALLEL TIME AND SPEED-UP Let T N1,N 2,,N k denote the parallel time for processing one unit of load on a k-dimensional mesh M k of size N N 1 N 2 N k by using the load distribution algorithm A N1,N 2,,N k Since both computation communication times are linearly proportional to the amount of load, the time for processing x units of load on a k-dimensional mesh M k of size N N 1 N 2 N k is xt N1,N 2,,N k for all x 0 The speed-up S N1,N 2,,N k is defined as the ratio of the sequential processing time to the parallel processing time, namely T 1 S N1,N 2,,N k T N1,N 2,,N k T N1,N 2,,N k We are particularly interested in T,,, S,,, lim N 1,N 2,,N k T N 1,N 2,,N k lim S N 1,N 2,,N k, N 1,N 2,,N k

3 SPEED-UP OF PARALLEL PROCESSING OF DIVISIBLE LOADS 627 ie the ultimate parallel processing time the asymptotic speed-up when the size of a k-dimensional mesh goes to infinity We use T S to represent T,,, S,,, respectively, where there are k s We now prove the main result of the paper THEOREM 41 For k-dimensional meshes, we have T 1 ( 4 T cm + Tcm 2 T 1 ( 4T (k 1 T cm + Tcm 2, k 2 Furthermore, we have S β 1 1/2k for large β all k 1 Proof The value T N1,N 2,,N k can be obtained recursively as follows First, by (A1, we have T 1 In general, when N > 1, processor P N1,N 2,,N k can proceed without waiting after sending the load fraction α in (A2 Hence, the time spent by M k 1 is (1 αt N1,N 2,,N k 1 In(A3,it takes αt cm time for the load fraction α to reach processor P N1,N 2,,N k 1 Then M k requires αt N 1,N 2,,N k 1 time to process the load fraction α To minimize the parallel processing time T N1,N 2,,N k, we need to make sure that both M k 1 M k spend the same amount of time, ie This implies that T N1,N 2,,N k (1 αt N1,N 2,,N k 1 α(t cm + T N1,N 2,,N k 1 α T N1,N 2,,N k 1 T N1,N 2,,N k 1 + T N1,N 2,,N k 1 + T cm Hence, T N1,N 2,,N k satisfies the following recurrence relation, T 1 (1 T N1,N 2,,N k T N 1,N 2,,N k 1 (T N1,N 2,,N k 1 + T cm, T N1,N 2,,N k 1 + T N1,N 2,,N k 1 + T cm N>1 (2 Taking the limit on both sides of Equation (2, we obtain T (T + T cm, k 1 T + + T cm T T (k 1 (T + T cm, k 2 T + T (k 1 + T cm That is, T 2 + T cmt T cm 0, k 1 T 2 + T cmt T (k 1 T cm 0, k 2 Solving these quadratic equations, we get T T with k 2inthetheorem As for asymptotic speed-up, we prove by induction on k 1thatS β 1 1/2k for large β When k 1, we note that S T 2 4 T cm + T 2 cm T cm 4β β for large β Whenk 2, we have S T 2 4T (k 1 T cm + T 2 cm T cm 4(T(k 1 /T cm By the induction hypothesis, ie S (k 1 /T cm T (k 1 T (k 1 /T cm β β 1 1/2k 1, T (k 1 /T cm we know that T (k 1 /T cm β 1/2k 1 for large β Therefore, S β 1 1/2k, (3 4β 1/2k for large β This proves the theorem The following corollaries are immediate consequences of Theorem 41 for two- three-dimensional meshes COROLLARY 41 For two-dimensional meshes, we have T, 1 ( 4T T cm + Tcm 2 ( 1 2 Furthermore, S, 2T cm 4 T cm + T 2 cm T 2 cm T cm β 3/4, T, 2 4β for large β COROLLARY 42 For three-dimensional meshes, we have T,, 1 ( 4T, T cm + Tcm 2 1 2T cm 2T cm 4 T cm +Tcm 2 2 T cm 2 T cm 2 T cm

4 628 K LI Furthermore, S,, T,, 2 2 β 7/8 4β for large β It is clear that Theorem 41 also holds for k-dimensional tori COROLLARY 43 For k-dimensional tori, we have T 1 ( 4 T cm + Tcm 2 T 1 ( 4T (k 1 T cm + Tcm 2, k 2 Furthermore, we have S β 1 1/2k for large β all k 1 In Table 1, we demonstrate numerical values of asymptotic speed-up S For each pair of β k, we give three values of S The first value is the exact value of S calculated by using Theorem 41 The second third values are estimations of S in Equation (3 These estimations are more accurate for small k large β than for large k small β 5 PERFORMANCE IMPROVEMENT Improved speed-up can be achieved by placing the initial load on an interior processor instead of corner boundary processors A submesh M k of a k-dimensional mesh M k of size N N 1 N 2 N k contains processors P k {P j 1,j 2,,j k a r j r b r, 1 r k}, where a r <b r for all 1 r k b r a r +1isthesizein the rth dimension of the submesh A processor P j1,j 2,,j k is called a boundary processor of M k in dimension r if j r a r or j r b r, a corner processor of M k if j r a r or j r b r for all 1 r k A processor P j1,j 2,,j k is called an interior processor of M k in dimension r if a r <j r <b r We say that a k-dimensional mesh M k is split at s r in dimension r if M k is divided into two disjoint submeshes M k containing processors P k {P j 1,j 2,,j k 1 j r N r,r r, 1 j r s r }, M k containing processors P k {P j 1,j 2,,j k 1 j r N r,r r, s r + 1 j r N r } If a processor P j1,j 2,,j k is an interior processor of a k-dimensional mesh M k in dimensions r 1,r 2,,r m, it will eventually become a corner processor of a submesh by splitting M k for m times Let T (m N 1,N 2,,N k denote the parallel time for processing one unit of load on a k-dimensional mesh M k of size N N 1 N 2 N k when the initial processor is an interior processor in m of the k dimensions, where 0 m k T (0 N 1,N 2,,N k is actually T N1,N 2,,N k Define T,,, (m lim T (m N 1,N 2,,N k N 1,N 2,,N k S,,, (m lim N 1,N 2,,N k S(m N 1,N 2,,N k, with abbreviation T (m S(m THEOREM 51 For k-dimensional meshes tori, we have T (m T /2 m S (m 2m β 1 1/2k for all 0 m k When the initial processor is an interior processor in all the k dimensions, the asymptotic speed-up of 2 k β 1 1/2k can be achieved Proof When m 1, the initial processor P N1,,j r,,n k is an interior processor in one dimension r The initial processor sends a fraction α of load x to one of its neighbors in dimension r, say,p N1,,j r +1,,N k The initial processor the selected neighbor are initial processors of two separate k-dimensional meshes M k M k obtained by splitting M k at j r in the rth dimension These two submeshes process the loads αx (1 αx respectively by using algorithm A N1,N 2,,N k It is clear that T (1 (1 αt α(t + T cm, which yields T (1 T (T + T cm T (β 1/2k + 1 2T + T cm 1/2k + 1 T 2 S (1 /T (1 1 1/2k for large β When m>1, the initial processor is an interior processor in dimensions r 1,r 2,,r m Thek-dimensional mesh M k is first split at j r1 in dimension r 1 The two resulted submeshes are further split in dimension r 2, the four resulted submeshes are further split in dimension r 3 so on It is not difficult to see that T (m T (m (1 αt (m 1 T (m 1 T (m (m 1 α(t + T cm (m 1 (T 2T (m 1 + T cm + T cm T (m 1 /2forall1 m k Hence, T (m T /2 m,s (m /T (m 2m ( /T 2 m β 1 1/2k In Table 2, we demonstrate numerical values of asymptotic speed-up S (m where k 3 It can be seen that the doubling effect S (m 2S(m 1 is stronger for small m large β than for large m small β

5 SPEED-UP OF PARALLEL PROCESSING OF DIVISIBLE LOADS 629 TABLE 1 Numerical values of asymptotic speed-up S β k 6 NOTES ON RELATED WORK Performance limits to parallel processing of divisible loads on static interconnection networks have been observed previously Asymptotic performance analyses for linear arrays were conducted in [20] The special cases of Theorems where k 1 for linear arrays are essentially similar to those in [21], the special cases of Theorems where k 2 for twodimensional meshes were obtained in [12] An infinite twodimensional mesh with the initial processor in the center was considered in [8] However, our study deals with finite meshes It was shown in [10] that the speed-up of O(β can be achieved in three-dimensional meshes The result is

6 630 K LI TABLE 2 Numerical values of asymptotic speed-up S (m (k 3 β obtained by adopting the circuit-switched routing technique which assumes that communication times are independent of the distances among processors Note that part (A4 of algorithm A N1,N 2,,N k is not related to the analysis in this paper The reason is that, in this paper, we increase the network size N by fixing k increasing the sizes of all the dimensions It is also possible to increase N by fixing N 1,N 2,,N k increasing the number of dimensions k For instance, when all the N r sarefixed at 2 N increases as k increases, we get hypercubes For hypercubes, we use the (k 1-dimensional mesh M k 1 in (A2 the (k 1-dimensional mesh M k 1 in (A4 to process the load fractions (1 αx αx, respectively Therefore, the analysis of parallel time speed-up for processing divisible loads on hypercubes follows a different direction [13] 7 CONCLUDING REMARKS We have proposed a divisible load distribution algorithm on k-dimensional meshes tori analyzed the parallel time speed-up of the algorithm We have shown that by using our algorithm on k-dimensional meshes tori, as the network size becomes large, the asymptotic speed-up of processing divisible loads with corner initial processors is approximately β 1 1/2k We have also proved that by choosing interior initial processors, the asymptotic speed-up of 2 k β 1 1/2k can be achieved The improved speed-up for large k is due to the increased network connectivity which yields a faster speed for load distribution Our work includes earlier results of linear arrays two-dimensional meshes as special cases, provides a unified treatment for divisible load distribution on k-dimensional meshes tori for all k 1 gives an initial investigation of divisible load distribution on k-dimensional meshes tori with k>3 m ACKNOWLEDGEMENTS The author wishes to express his gratitude to two anonymous reviewers for their criticism comments This material is based upon work supported by the US National Science Foundation under Grant No CCR Any opinions, findings conclusions or recommendations expressed in this material are those of the author do not necessarily reflect the views of the National Science Foundation REFERENCES [1] Bharadwaj, V, Ghose, D, Mani, V Robertazzi, T G (1996 Scheduling Divisible Loads in Parallel Distributed Systems IEEE Computer Society Press, Los Alamitos, CA [2] Cheng, Y C Robertazzi, T G (1988 Distributed computation with communication delays IEEE Trans Aerospace Electron Syst, 24, [3] Bataineh, S, Hsiung, T-Y Robertazzi, T G (1994 Closed form solutions for bus tree networks of processors load sharing a divisible job IEEE Trans Computers, 43, [4] Sohn, J Robertazzi, T G (1996 Optimal divisible job load sharing for bus networks IEEE Trans Aerospace Electron Syst, 32, [5] Mani, V Ghose, D (1994 Distributed computation in linear networks: closed-form solutions IEEE Trans Aerospace Electron Syst, 30, [6] Barlas, G D (1998 Collection-aware optimum sequencing of operations closed-form solutions for the distribution of a divisible load on arbitrary processor trees IEEE Trans Parallel Distributed Syst, 9, [7] Cheng, Y C Robertazzi, T G (1990 Distributed computation for a tree network with communication delays IEEE Trans Aerospace Electron Syst, 26, [8] Błażewicz, J Drozdowski, M (1996 The performance limits of a two-dimensional network of load sharing processors Found Comput Decision Sci, 21, 3 15 [9] Błażewicz, J, Drozdowski, M, Guinard, F Trystram, D (1999 Scheduling a divisible task in a two-dimensional toroidal mesh Discrete Appl Math, 94, [10] Drozdowski, M Głazek, W (1999 Scheduling divisible loads in a three-dimensional mesh of processors Parallel Computing, 25, [11] Błażewicz, J Drozdowski, M (1995 Scheduling divisible jobs on hypercubes Parallel Computing, 21, [12] Li, K (1998 Managing divisible load on partitionable networks In Schaeffer, J (ed, High Performance Computing Systems Applications, pp Kluwer Academic Publishers, Boston, MA [13] Li, K (2003 Parallel processing of divisible loads on partitionable static interconnection networks Cluster Computing (Special Issue on Divisible Load Scheduling, 6, [14] Błażewicz, J Drozdowski, M (1997 Distributed processing of divisible jobs with communication startup costs Discrete Appl Math, 76, [15] Błażewicz, J, Drozdowski, M Markiewicz, M (1999 Divisible task scheduling concept verification Parallel Computing, 25, 87 98

7 SPEED-UP OF PARALLEL PROCESSING OF DIVISIBLE LOADS 631 [16] Ko, K (2000 Scheduling Data Intensive Parallel Processing in Distributed Networked Environments PhD dissertation Department of Electrical Computer Engineering, State University of New York, Stony Brook, New York [17] Sohn, J, Robertazzi, T G Luryi, S (1998 Optimizing computing costs using divisible load analysis IEEE Trans Parallel Distrib Syst, 9, [18] Amdahl, G M (1967 Validity of the single processor approach to achieving large scale computing capabilities In Proc AFIPS Spring Joint Computer Conf, Vol 30, pp CSREA Press [19] Li, K (2002 Speedup of parallel processing of divisible loads on k-dimensional meshes tori In Proc Int Conf on Parallel Distributed Processing Techniques Applications, Las Vegas, Nevada, June 24 27, pp [20] Ghose, D Mani, V (1994 Distributed computation with communication delays: asymptotic performance analysis J Parallel Distrib Comput, 23, [21] Bataineh, S Robertazzi, T G (1992 Ultimate performance limits for networks of load sharing processors In Proc Conf Information Sciences Systems, pp Princeton University Press, Princeton, NJ