Semi-Distributed Load Balancing for Massively Parallel Multicomputer Systems

Transcription

1 Syracue Univerity SUFAC lectrical ngineering and Computer Science echnical eport College of ngineering and Computer Science Semi-Ditributed Load Balancing for aively Parallel ulticomputer Sytem hfaq Ahmad Syracue Univerity, School of Computer and nformation Science Arif Ghafoor Follow thi and additional work at: Part of the Computer Science Common ecommended Citation Ahmad, hfaq and Ghafoor, Arif, "Semi-Ditributed Load Balancing for aively Parallel ulticomputer Sytem" (1991). lectrical ngineering and Computer Science echnical eport hi eport i brought to you for free and open acce by the College of ngineering and Computer Science at SUFAC. t ha been accepted for incluion in lectrical ngineering and Computer Science echnical eport by an authorized adminitrator of SUFAC. For more information, pleae contact urface@yr.edu.

2 SU-CS-91-3 Semi-Ditributed Load Balancing for aively Parallel ulticomputer Sytem hfaq Alunad and Arif Ghafoor Augut1991 School of Computer and nformation Science Syracue Univerity Suite 4-//6, Center for Science and echnology Syracue, ew York

3 Semi-Ditributed Load Balancing for aively Parallel ulticomputer Sytem lhfaq Ahmad School of Computer and nformation Science, Syracue Univerity, Syracue, Y iahmad@top.ci.yr.edu Arif Ghafoor School of lectrical ngineering., Purdue Univerity, Wet Lafayette, ghafoor@dynamo. ecn.purdue. edu

4 Abtract hi paper preent a emi-ditributed approach, for load balancing in large parallel and ditributed ytem, which i different from the conventional centralized and fully ditributed approache. he propoed trategy ue a tw<rlevel hierarchical control by partitioning the interconnection tructure of a ditributed or multiproceor ytem into independent ymmetric region (phere) centered at ome control point. he central point, called cheduler, optimally chedule tak within their phere and maintain tate information with low overhead. We conider interconnection tructure belonging to a number of familie of ditance tranitive graph for evaluation, and uing their algebraic characteritic, how that identification of phere and their cheduling point i, in general, an Pcomplete problem. An efficient olution for thi problem i preented by making an excluive ue of a combinatorial tructure known a the Hadamard atrix. Pedormance of the propoed trategy ha been evaluated and compared with an efficient fully ditributed trategy, through an extenive imulation tudy. n addition to yielding high pedormance in term of repone time and better reource utilization, the propoed trategy incur le overhead in term of control meage. t i alo hown to be le enitive to the communication delay of the underlying network. Key Word :nterconnection etwork, Load Balancing, ulticomputer Sytem, etwork Partitioning, Parallel Proceor, Pedormance valuation, ak Scheduling. 1. ntroduction A a reult of evolutionary advancement in computation and communication technology, one can foreee future upercomputer coniting of thouand of proceor [13]. With the advent of proceor-memory chip and high peed channel, it i now poible to build large and dene ytem and exploit maive parallelim to olve complex cientific and engineering problem. One cla of thee higwy parallel ytem i multicomputer, which comprie a large number of computing node interconnected by a meage paing network. ulticomputer have become very popular during the lat decade and more than hundred type of uch ytem are currently in ue. he firt and econd generation of thee ytem have witneed ytem with a many a 64 node and 256 node, and the third generation i projected toward ytem compriing more than one thouand node [2]. n addition to providing enhanced availability, calability, and reource haring, thee maively parallel ytem can theoretically multiply the computational power of a ingle proceor by a large factor. he key advantage of thee ytem, however, i that they allow concurrent execution of workload characterized by computation unit known a procee or tak, which can be independent program or partitioned module of a ingle program. From a pedormance perpective, tak repone time, throughput and reource utilization are critical meaure that need to be optimized while keeping the control overhead within a reaonable value. When deigning a meage paing multicomputer y- -1-

5 tern, the problem ofload balancing on the computing node of the ytem become an important iue. he problem become more challenging, in a ytem coniting of hundred or thouand of node due to the overhead reulting from collection of tate information, communication delay, aturation effect, high probability of node failure etc. nefficient cheduling can lead to a load imbalance on variou node which can ignificantly increae the repone time of tak cheduled at heavily loaded node. Dynamic load balancing ha been conidered the inevitable olution for thi problem becaue the time dependent fluctuation in the load pattern acro the ytem need to be balanced dynamically [4], [5], [8], [9], [14], [18]. [25], [29], [35], [37]. For large-cale multicomputer ytem, we propoe a new trategy for tak cheduling and load balancing. Our approach, which i emi-ditributed in nature, i baed on partitioning of the interconnection network into independent and ymmetric phere. ach phere comprie a group of node and ha a central control point, which we call a cheduler. Accordingly, a load balancing algorithm and an information building algorithm are preented that are executed only by the cheduler. he work load ubmitted to the ytem, which i characterized by arrival of tak, i optimally balanced within and among thee phere. Similarly, an information building algorithm i employed by each cheduler to accumulate tate information of the local phere, a well a remote phere. Both the cheduling and information building algorithm are imple and incur low overhead. he number of cheduler which need to execute the cheduling and information building algorithm i relatively mall, reulting in a low control overhead. At the ame time the cheduler are ufficiently enough to effectively manage the load within their phere. We how that, in general, an optimal determination (we decribe thi determination in ection 4.1) of the center or cheduler i an P -complete problem. However, for a cla of interconnection tructure, known a ditance tranitive graph (D) [24], which poe a remarkable partitioning property, we propoe an efficient emi-ditributed deign baed on a combinatorial tructure known a the Hadamard atrix. hrough an extenive imulation tudy, we how that for large-cale ytem the propoed trategy yield better performance in term of repone time and reource utilization a compared to an efficient fully ditributed load balancing trategy a well a no load balancing. he overhead due to exchange of control meage and the impact of communication delay on repone time i alo evaluated. he propoed emi-ditributed trategy i applicable to both large parallel and ditributed ytem. For example, a hypercube topology can be extended beyond a parallel proceing environment by auming that the virtual communication network topology of a ditributed ytem i iomorphic to the hypercube provided the number of node in the ytem i Z'. An example of uch a ytem i the virtual machine looely ynchronou communication ytem (VLSCS) employing the CrOS operating ytem in which independent node can communicate via node-addreed meage [2]. For virtual topology, if the number of node in the ytem i not equal to Z', virtual node can be added to complete the topology [23]. he reult can alo be extended to a number of of D graph with a different number of node. he ret of the paper i organized a follow. n the next ection we preent a brief overview of exiting tak cheduling and load balancing trategie and dicu their limitation for large-cale -2-

6 ytem. n ection 3, we give an algebraic characterization of ditance tranitive interconnection network. n ection 4, we tate the problem of contructing the et of phere and their center. n the ame ection, the ue of Hadamard atrice for network partitioning and for the election of cheduler i dicued. he ytem model for load balancing uing the propoed emi-ditributed trategy i preented in ection 5. Simulation reult and comparion are given in ection 6. Section 7 conclude thi paper. 2. elated Work and otivation for a ew Approach he tak cheduling problem ha been extenively reported in the literature. Generally, tak cheduling technique can be claified into two categorie. n the firt category, an application compriing a tak or a et of tak with a priori knowledge about their characteritic, i cheduled to the ytem node before run time. hi type of cheduling problem i better decribed a the aignment or mapping problem [6]. he aignment can be done in a number of way uing heuritic, graph theoretic model [6], clutering [7], integer programming [36], or by many other optimization technique, depending upon the nature of the application and the target ytem. hee type of technique have alo been termed a tatic cheduling technique [4]. he econd cla of tak cheduling, which take into account the notion of time, i ued to aign tak to proceor by conidering the current tate of the ytem. he tate information concerning current load on individual node and the availability of reource i time dependent. hee type of trategie, do not aume a priori knowledge about the tak and are known a dynamic cheduling trategie. An eential property of a dynamic trategy i to balance the load acro the ytem by tranferring load from heavily loaded node to lightly loaded node. ot of the exiting dynamic load balancing technique employ centralized [11], [33] or fully ditributed model [4], [8], [9], [12], [14], [25], [28], [29], [35], [37], [38]. n a centralized model, a dedicated node gather the global information about the tate of the ytem and aign tak to individual node. On the other hand, in a fully ditributed model each node execute a cheduling algorithm by exchanging tate information with other node. any variant of fully ditributed load balancing trategie [36], alo known a adaptive load haring, exit employing a variety of policie for exchanging information and cheduling dicipline. ot of the tudie have hown that imple heuritic [14], [17], [35], [ 43] yield good performance. One claification [42] ha egregated dynamic load balancing into erver initiated and receiver initiated clae. hi claification depend upon the load tranfer requet which can be initiated by an overloaded node or under-loaded node. any fully ditributed algorithm belong to either of the two clae. For example the bidding algorithm [35], [38] belong to the ender-initiated category while the drafting algorithm [34] belong to the erver-initiated. t ha been hown [15] that the ender-initiated algorithm perform better under low to medium loading condition while receiver initiated algorithm perform better at high load provided the tak tranfer delay i not very high. A number of load balancing algorithm are compared in [44] uing a imulation model that take into account the trace of actual job tatitic. A hierarchical cheme ha alo been propoed in [ 41] but it ha many drawback uch a higher overhead and underutilization of the ytem reource. ot of thee cheme are effi- -3-

7 cient and yield a near optimal performance for mall ytem ranging from a few node to a few ten of node [44]. A performance model uing imulation, queuing and tatitical analyi ha been propoed in [1] to determine and analyze the performance of ender-initiated fully ditributed trategie. A little attention ha been paid to deigning load balancing trategie for large ytem coniting of thouand of node. For thee ytem, centralized and fully ditributed trategie are not very uitable due to the following reaon: While uing fully ditributed cheme, optimal cheduling deciion are difficult to make [4], [9]. hi i becaue a rapidly changing environment, with arrival and departure from individual node, caue a great degree of randomne and unpredictability in the ytem tate. Since, each node in the ytem make autonomou deciion, no node can enviion the precie tate of the whole ytem. hu, a node ha to make either a probabilitic deciion or make ome kind of gue about the ytem tate. he econd problem with fully ditributed model i that communication delay can tum a correct cheduling deciion into a wrong choice. For intance, a tak after going through communication et up time, queueing and tranmiion delay over multiple link may find that the detination node that wa originally conceived of a the bet choice ha become highly loaded due to arrival from ome other node [37]. hee cenario reult in occaional wrong deciion and the ytem can enter into the tate of intability or tak thrahing [8] a phenomenon when tak keep on migrating without getting executed at any node. Fully ditributed algorithm ue mall amount of information about the tate of the ytem. Since gathering large amount of tate information may decreae the accuracy, it become more appropriate to collect mall amount of information with greater accuracy [9]. Small ytem can yield good performance with limited information [14] but thi may not be true for large ytem. f the mot heavily and the mot lightly loaded ection of a large network are a far apart, a fully ditributed algorithm with limited amount of information may have to be executed for a number of time to balance the load among thee ection. educed amount of information reult in a maller range of cheduling option and hence a narrow cope of ending a tak to a uitable node. he control overhead alo depend on the ytem load and can be high at heavy loading condition. A a reult, a load balancing policy may perform wore than no load balancing cae [16]. Depite the fact that fully ditributed algorithm incur le overhead due to meage exchange, thi overhead linearly increae with the ytem ize [5], [44]. hi may reult in a proportional increae in the average repone time and extra communication traffic which can impede regular communication activity. With a few exception [17], mot reearcher have ignored the overhead of the cheduling algorithm which can delay the proceing of a tak. hi overhead i alo a linearly increaing function of the ytem ize, ince each node execute the ame algorithm. Saturation effect in large ditributed ytem i a widely cited phenomenon [17], [29] which deteriorate the performance a the ytem grow in ize. An incremental increae in ytem reource may -4-

8 reult in a decreae in throughput [8]. t i known that, with ditributed algorithm, the repone time doe not improve with an increae in ytem ize beyond a few ten of node [44]. Centralized algorithm do have the potential of yielding optimal performance [11], [34], [44] but require accumulation of global information which can become a formidable tak [37]. he torage requirement for maintaining the tate information alo become prohibitively high with a centralized model of a large ytem [5]. For a large ytem coniting of hundred or thouand node, the central cheduler can become a bottleneck and hence can lower the throughput [5]. Centralized model are alo highly vulnerable to failure. he failure of any oftware or hardware component of the central cheduler can top the operation of the whole ytem. t ha been hown [ 44), that neither fully ditributed nor centralized trategie alway yield optimal performance. t i, therefore, expected that there exit a trade-off between centralized and fully ditributed cheduling mechanim. Some recent tudie have alo treed the need to build large ytem with hierarchical architecture [1), [41). he aim ofthi paper i to preent a emi-ditributed trategy exploiting the advantage of both centralized and fully ditributed model. he propoed trategy i a two level load balancing trategy. At the firt level, load i balanced among different phere of the ytem thu providing a "ditributed environment" among phere. At the econd level, load balancing i carried out within individual phere where the cheduler of each phere act a a centralized controller for it own phere. he deign of uch a trategy involve the following tep: Formulating a network partitioning trategy for creating ymmetric phere, dentifying the control node (cheduler) for controlling their individual phere, Deigning an algorithm for performing optimal tak cheduling and load balancing within a phere a well a among phere and Developing efficient mean for collecting tate information at inter phere and intra phere level that hould reult in a mall meage traffic. n order to meet thee deign objective, we firt decribe ome of the network topologie and their algebraic characteritic which can be ued for building large ytem. 3. etwork opologie for Large Sytem and their Algebraic Characteritic n thi ection, we decribe ome of the network topologie belonging to the claical infinite familie of Ditance ranitive (D) graph which can ued to build large parallel and ditributed ytem [24]. We alo introduce the notion of phere of locality and preent ome definition and terminology which are ued to characterize thee topologie and their property of ymmetry. he reaon for analyzing D graph that many of the exiting and previouly propoed interconnection network, including the Hypercube topology, are indeed ditance-tranitive. We how that ditance-tranitivity i a highly deirable property ince thee graph are hown to be node-ymmetric which help in deigning parallel and ditributed ytem with emi-ditributed control. We focu on a cla of D graph which are governed by two algebraic tructure known a the Hamming [22] and the Johnon Aociation Scheme [24]. he graph belonging to thee cheme include the Hamming graph (the hyper- -5-

9 cube) and it derivative and the Sphere (Johnon) graph. n order to define the Dtopologie, we need the following definition. An interconnection network i repreented by an undirected graph, A = < U, >where U repreent the et of node and i the et of edge (communication link) joining the node. Let, 1, 2,..., ( -1) denote the et of node in the ytem where each node i repreented by a binary code. he degree of each node, denoted by n, repreent the number of edge incident on it. he degree i aumed to be contant. A path in A i a equence of connected node and the length of the hortet path between node i and j i called the graphical ditance and i repreented a L;j. Definition: Let a be a binary codeword. he Hamming weight w(a) of a i equal to the number of l' in a. Definition: he Hamming Ditance, Hxy, between two binary codeword, x = ( xhx2,... xn ) andy = ( YhY2 Yn ) of ome length n, i defined a Hxy = l[ilx; = y;, 1, i n) 1 where x, Y [, 1). n other word, Hamming ditance i the number of different bit in codeword. Definition: Let k = ax[lijivi,j,o i,j, -1]. k i called the diameter of the network A. Definition: Given a et of node C, it graphical covering radiu r in the graph A i defined a: r = ax;eu(injec(lij)) Definition: Let G(A) be the automorphim group of A. A i aid to be ditance-tranitive, if for each quartet of vertice u, v, x, y, A, uch that Lu,v = Lx,y, there i ome automorphim gin G(A) atifying g(u) = x and g(v) = y. Definition: A ditance-regular graph i a weaker condition of ditance-tranitive graph. A graph of diameter k i ditance-regular if V(i,j,l) [O... n] 3, V(x,y) U, Lxy {z U, Lxz = i, Lyz = i} = Jf;j, where y repreent the cardinality of ome et y and pij are contant whoe value are dependent on the characteritic of the graph. Ditance-regularity i an important property in term of decribing phere of locality which in turn define the range of a cheduler and hence the cope ofload balancing. t alo affect the number of meage generated for gathering tate information. Definition: Let vf be the number of node which are at a graphical ditance i from a node x. hi number i a contant for Vx U and i called the i-th valency. t i given a vf = P3. Some D graph are decribed below. -6-

10 3.1 he binary n-cube etwork Qn he binary n-cube network (Hypercube), which we denote a Q", conit of 2" node. Here each node i repreented a a binary vector where two node with binary codeword x andy are connected if the Hamming ditance Hxy = 1. hen for every node x in Qn, vf =( 7 ) for i =, 1, 2.. n and Lxy = Hx, and k = n. Qn i a ditance-regular graph and it P;j are given a ( ) ( n-1 ) i-j+1 i+j ', which are conitent with the definition of vf. if i + j + i even if i + j + i odd ' 3.2 he Biectional etwork Bn [22] A Biectional etwork i a folded Hypercube and i generated uing all the binary codeword of lengthn with even weight. Anodexin abiectional network i connected to a neighboryifhx, = n -1 [22]. We will denote a Biectional network a Bn. he degree of a Bn network i nand it ha 2n-l node. For Bn, k = ( n - 1 )12 and Lxy = in(hxy, Hi},) where Hi}, = n- Hxy. For every node x in Bn, the valencie are given a vf =(n,.) for i =, 1, 2.. n. 3.3 he binary odd network On [21] he Odd graph belong to the family of Johnon graph and i contructed by uing binary code with contant Hamming weight. An Odd graph On ha for vertex et the binary codeword of length 2n -1 and Hamming weight n -1. 1\vo vertice in On are connected if and only if the Hamming ditance between them i 2n-2. he On graph are elected due to their higher denity than variou other interconnection network. hey have degree n, diameter k = n -1 and (~ = f ) node. he Odd graph 3 i the celebrated Peteron graph [21]. For every node x in On [21], =( ~ )( n.-1). 1 1 z for z =, n -1, 1, n- 2, 2,.. k 4. he etwork Partitioning Strategy n thi ection, we decribe the criteria for partitioning the D topologie decribed above. We how that partitioning of the interconnection network and the election of the et of node, referred a cheduler, for the purpoe of carrying out tak cheduling for their repective phere, can be modeled -7-

11 a a problem which i P-hard. Subequently, we propoe an efficient olution, for partitioning and finding the et of cheduler for thee network, baed on a combinatorial tructure called Hadamard deign. he propoed olution i "efficient" in that the ize of the elected et of cheduler i coniderably mall and it i of the order oflogarithm of the ize of the network. hi reult in a mall number of cheduling point and hence lower overhead reulting from meage exchange. We will denote thi et a C. Due to the maller number of node in et C, the torage requirement for maintaining tatu information about other member of the et Ci alo mall. At the ame time, we how that the whole network i uniformly covered and each phere i ymmetric and equal in ize. Baed on thi partitioning, we then propoe a emi-ditributed cheduling and load balancing mechanim. n the propoed cheme, each phere i aigned a cheduler which i reponible for (a) aigning tak to individual node of the phere, (b) tranferring load to other phere, if required, and (c) maintaining the load tatu of the phere and node. A cheduler i reponible for optimally aigning tak within it phere depending upon the range of the cheduler which i, in turn, determined by the ize of the phere. ak can alo migrate between phere depending upon the degree of imbalance in the load of phere. For the propoed cheme, each cheduler i located at the center of each phere. he detail of the cheduling algorithm and information maintenance cheme are decribed in ection he etwork Partitioning Problem A mentioned above, C i the deired et of cheduling node. here can be variou poible option to elect C and devie a emi-ditributed cheduling trategy baed on thi et. However, the performance of uch a trategy depend on the "graphical location" of the cheduler node (ditance between them) of C and the range of cheduling ued by thee node. he range of cheduling quantifie the graphical ditance within which a cheduler aign tak to the node of it phere. n order to characterize phere and decribe network partitioning, we need the following definition. Definition: Let the phere aigned to a nodex C be denoted by S,{x), where i i the radiu ofthi phere. he number of node in S,-(j) i the total number of node lying at graphical ditance through i, from node x. Since the number of node at the graphical ditance i i given by valency vf, i the total ize of the phere i given a S,{x)l = j! vj. t hould be noted that, in a centralized cheme, i mut be equal to the diameter (k) of the network and in a fully ditributed cheduling, uing local information among neighbor, i i equal to 1. n the former cae, C = 1 and in the latter C =, the ize of the network. Definition: A uniform et C, of center, i the maximal et of node in A, uch that the graphical ditanceamongtheecenter i at leat() and S;(x)l i contant(uniform) Vx C, where ii the covering radiu of C. We need a d -uniform et C (for ome d to be determined) with graphically ymmetric phere, in order to deign a ymmetric cheduling algorithm. he ize of C depend on the election of d. -8-

12 ntuitively, larger 6 yield maller C, but phere with larger ize. t can alo be oberved that reducing C increae the phere ize and vice vera. n addition, a number of other conideration for load balancing are given below: (1) Since a cheduler need to ditribute tak to all the node in the phere, the diameter of the phere hould be a mall a poible. (2) C hould be mall, o that the global overhead of load balancing algorithm in term of meage exchange among cheduler, maintenance of information and torage requirement i mall. (3) he ize of the phere hould be mall, firtly, becaue a cheduler (x) need to end/receive S;(x) meage and, econdly, becaue information torage and maintenance requirement within the phere increae with the increae in phere ize. We now decribe the complexity of electing a d -uniform et (C). heorem 1: For a given value of 6 > 2, (a) Finding a uniform et C in an arbitrary graph i P-hard. (b) Determining the minimum phere ize i alo P-hard. Proof: (a) For the proof, ee [39]. (b) Finding the minimum phere ize, St_x) Vx C, require u to determine the minimum value off, which i equal to the covering radiu of the et C. Since all the above topologie are repreented uing binary code, the problem of determining the et Ci equivalent to finding a ub code with the deired covering radiu in a code, ay F. For Qn. F i the complete binary code. For a biectional network, Bn, F repreent all the codeword of length n with even weight wherea for an odd graph, On. F repreent a contant weight code of weight n with length n -1. However finding the covering radiu of a ub code, ay C, in a code F i an P-hard problem [3]. Since finding the minimum phere ize require determining the covering radiu, the complexity of the whole problem will not be le than P-hard. Q..D. he above theorem provide a rather peimitic view for finding a et C for give F. However, we preent an intereting olution to elect the et C in D graph uing a combinatorial tructure called Hadamard atrix. hi matrix i decribed in the following ection. he propoed olution for On with 6 = k, and for Qn and Bn with 6 = 1/k, i optimal in the ene that the et C i minimal Hadamard atrice Definition: A Hadamard matrix i aj by j matrix with± 1 entrie, uch that = jl, where i the identity matrix and i the tranpoe of. he complementary Hadamard matrix, denoted a ~, i obtained by multiplying all the entrie of by -1. f we replace 1 by, and -1 by 1, the matrix i aid to be in -1 notation. We will refer to thi matrix a Hadamard matrix, and ue the -1 notation in the ret of thi paper. Figure 1 how a 8 x 8 Hadamard matrix and it complement, uing -1-9-

13 = UC= Figure 1. An 8 x 8 Hadarmard atrix in -1 notation and it complement notation. t i known that Hadamard matrice of order up to 428 exit. Furthermore, it ha been conjectured that a Hadamard matrix of order n exit if n i 1, 2 or a multiple of 4. Variou method of generating Hadamard atrice include Sylveter' method, Paley' contruction and the ue of Symmetric Balanced ncomplete Block Deign (SBBD) [26] he Propoed Partitioning of D etwork and dentification of Scheduler Site he et C of cheduler node i elected from the code generated by the row of Hadamard matrix and it complement AfC. he et Ci alo called Hadamard code. ote, C = 2n for Qn. Following are the main reaon for chooing Hadamard code for the et C (we might a well elect other code uch a Hamming code or BCH code, but thee code have certain limitation a decribed below). 1. A Hadamard code i a code with rate approaching zero, aymptotically where rate of a code C with length n i defined a n~ oo (log 2 1 Cl fn) [31 ]. hi reult in the ize of a Hadamard code being coniderably maller than the ize of a Hamming code in a Qn. n fact, the ize of Hadamard code i proportional to the logarithm of the ize of the network Qn. On the other hand, the rate of a Hamming code i 1, which reult in a large ize of the code and hence the et C he range of value of n for which a Hadamard code exit, coniderably exceed the range of n for which a Hamming code exit. A decribed earlier, it i conjectured that a Hadamard matrix exit for all value on n which are le than 428 and are multiple of 4 [26]. On the other hand an extended Hamming code only exit if n i a power of 2. Similarly, a BCH code exit only for limited value of n. 3. he covering radiu of C i known [26] for many value of n, which are even power of two. 4. he following theorem how that the emi-ditributed deign uing Hadamard code reult in a et C, which provide the maximal n/2-eparated matching for Qn with diameter n. -1-

14 heorem 2: Letx,y C. hen for Qn, Lxy = n/2. and Ci the maximal poible et, with n/2-eparated matching. Proof he Hamming ditance between any two row of a Hadamard matrix i n/2, that i for Qn, Lxy = Hxy. n order to prove that the cardinality of the et i the maximum poible, aume the contrary i true, and uppoe there exit ome codeword z, uch that Hzx = n/2, for all x C. A imple counting argument reveal that there mut be at leat n(n-1 )14 l' at thoe n/2 column where z ha O'. f thee 1' are ditributed among all row of, then there are at leat (n-l)-n(n-2)/[4(n-1)] row which can not be filled to obtain thi Hamming ditance. herefore the node z i at a graphical ditance le than n/2 from thee row. Q..D. Due to the above mentioned advantage, we ue Hadamard code to contruct the et C. Since, a Hadamard code exit only when n i a multiple of 4, election of the et C can be made by modifying thi untruncated code in variou way, to form the remainder of value. hee modification are decribed below. Cae a: Qn with n mod 4 = 1. For thi cae, we conider the et C obtained from Hadamard matrice and~ (in -1 notation) of ize n-1. he modified et C for the network under conideration by appending an all O' and an alll' column, to and~ repectively, at any fixed poition, ay at extreme left. Cae b: Qn with n mod 4 = 2. hi cae i treated the ame way a the Cae (a), except we conider the et C obtained from Hadamard matrice (in -1 notation) of ize n-2 and append two column and 1 to and 1 and to~. However, the all O' row in i augmented with bit rather than with bit 1. Similarly, the all l' row in ~ i augmented with bit 11 rather than with bit 1. Cae c: Qn with n mod 4 = 3. For thi cae, the et C conit of the row of the truncated matrice and~ in -1 notation. he truncated matrice (in -1 notation) are generated by dicarding the all..q row and column. A truncated Hadamard matrix (the one without alll' column) uing Symmetric Balanced ncomplete Block Deign (SBBD) [26] can be eaily generated, ince mot of the available SBBD' are cyclic by contruction. For thi purpoe, all the block (which correpond to all the element of the et able 1. Generator code for different length Length = n -1 Generator Codeword

15 'hble he emi-ditributed tructure of variou interconnection network etwork n d C r v1 v2 vj S,(x)l Q, Hypercube Q Qg Qw B, Biectional Bg Bn Odd C, beide codeword with all O' and a111') can be generated by taking n-1 cyclic hift of a ingle generator codeword. Such generator, for different value of n-1 can be found uing the o called difference et approach [26]. 'hble 1 illutrate the generator codeword for variou value of n-1. he et of cheduler node for Q 1 can be obtained by firt contructing codeword for Q 1. he generator codeword for Q 1 i he additional6 codeword are generated by taking 6left cyclic hift of thi generator. he complete et C conit of row with all O' and all l' plu the following 7 codeword and their complement that i C= {1111, 1111, 1111, 1111, 1111, 1111, 1111, 111, 111,111, 111,111,111, 111}. herefore, the et C, for any Qn, conit of matrix and it complement JJC. For Q 8, the et C can be produced by chooing the truncated matrice, which i the ame a hown earlier in Figure 1 where each row of the matrix repreent the binary addre of the 16 cheduler. he et coniting of codeword a given in Figure 1, can alo be ued to generate the et C for the Q9 network by appending an all O' and alll' column (ay at extreme left poition), of matrix and JJC, repectively, a decribed for cae (a). Alo, the ame et can be ued to generate the et C for Q 1, a decribed in the procedure of cae (b). he et C for other Qn 'can be generated by the method decribed above. Lemma 1: A truncated matrix with and without all.q ' row provide a k eparated matching for On and Bn network, repectively. Proof i obviou from theorem

16 Lemma 1 can be ued to partition an On graph into 2n -1 phere. For intance, for 6, the generator codeword for 6 i he additional to codeword generated by taking 1 left cyclic hift of thi generator, contitute the et C for 6 he et C for Biectional network Bn can be generated by taking only the matrix for Qn (without JJC matrix ). For example, in the above example of Q 7, we can take 8 row of to from the et C for B1. herefore in a Bn network the et Ci half the ize of that for Qn. he topological characteritic and emi-ditributed tructure for Q7, Q, Q9, Q1, B1, B9, Bu. 4 and 6 network are ummarized in hble which how the number of node, the degree n of each node, the ditance d between cheduler, the cardinality of the et C, the covering radiu r, valencie vf and the ize of phere S,{x), for each network. n cae the number of node i not diviible by the number of cheduler ( a i the cae for Bu), the difference in phere ize doe not exceed 1. Figure 2 how one of the 16 phere in Q. he binary code of the cheduler in thi cae i. he covering radiu r i equal to 2 and the valencie o. vi and~. 'Vx, have value 1, 8 and 28 repectively, correponding to total volume of the phere Sj(x)!equal to 37. he addree of the node for a, vi and~. Vx, can be obtained by uing the expreion given in Section Figure 2. A phere in Qnetwork with cheduler. -13-

17 A mentioned earlier, the node in one phere can alo be hared by other phere, depending upon the range of cheduling and the graphical ditance between node and the et C. he ditribution of hared node at variou ditance with varying range of cheduling (j) within the phere i given in 'hble ll, for Q. For example, with the range of cheduling equal to 2, which i alo the covering radiu in thi cae, node at ditance 1 from a cheduler are hared by only one phere wherea node at ditance 2 are hared by exactly 4 phere. On the other hand, if the range of the cheduler i 8, the ytem i equivalent to the centralized model with 16 node trying to aign tak to the ame 256 node. ncreaing the range of cheduler beyond the covering radiu caue more haring of node among phere for which greater number of meage need to be generated to keep the load information conitent for all cheduler. herefore, the optimal range of a cheduler, the one which provide maximum coverage with minimum radiu in all the cae i et to the covering radiu of the correponding Hadamard Code. 'hble m Ditribution of node hared by different phere a a function of the covering radiu f Ditance of a node from C! S. he Propoed Semi-Ditributed Load Balancing Strategy n thi ection, we preent the propoed emi-ditributed cheme. For thi purpoe, we decribe the ytem model, the load balancing algorithm and the aociated information collection and maintenance mechanim Sytem odel Figure 3 illutrate a logical view of a fully ditributed load balancing ytem [43]. he ytem conit of identical proceing node connected by a communication network. he work load A; arriving at node i conit of independent program module or tak. hee independent tak are capable of being erviced at any node (except for the tranfer cot which delay tak proceing if the tak i migrated to another node) irrepective of where they are ubmitted. n addition, each node receive migrated tak ; from other node via communication network. ak cheduled at a node -14-

18 t c Proceor 1 t u c w K 2 Proceor 2 At A2 Proceor 2 Figure 3: A logical view of fully ditributed load balancing. are entered in the execution queue which i erved on FCFS principle. he output from each node i the tak which are either locally executed, ;, or tranferred, ;, to other node. n contrat, Figure 4 illutrate the logical view of the emi-ditributed model which conit of the C phere, each compriing S,{x) node (which i ame for every phere), 'Vx C. ak are generated at all node ofthe ytem with rate A. tak per time-unit and the node route newly arrived tak to their repective cheduler (in cae a node i a hared among cheduler, one cheduler i randomly elected). Alternatively tak can be ubmitted at cheduler themelve. herefore, tak are aumed to originate at cheduler with rate A; which i roughly equal to A.jC. ak migration i and tak tranfer ; take place between a cheduler i and other cheduler. he network traffic, therefore, can be viewed at two level. At the higher level tak migration take place among different phere and at the lower level tak are tranported from a cheduler to the node within it phere. Both kind of tak movement incur communication delay which i dependent on the tak tranfer rate of the ytem. n addition, meage paing take place among cheduler for exchanging the informa- -15-

19 tion about the accumulative load of phere. A in the cae of fully ditributed trategy, each node maintain a tak queue which i erved on FCFS principle. he Scheduler, which i a proceing element along with ome memory for keeping load information, i reponible for deciding where a tak hould be aigned. he et of cheduler, elected through the Hadamard code, are aumed to be embedded on the ytem topology. n other word, the node that are deignated a cheduler perform their regular tak execution a well a they carry out the role of the cheduler. Alternatively, we can aume that thee node are augmented by pecial purpoe proceing element that execute the cheduling algorithm without interrupting the normal tak proceing at that node. c 3 u c ;(x) 2 Sphere 1 1 w 2 K 3 Sphere C Figure 4: A logical view of emi-ditributed load balancing. -16-

20 5.2. State nformation xchange he tate information maintained by a cheduler i the accumulative load of it phere which in turn i the total number of tak being erviced in that phere at that time. hi load index i adjuted every time a tak enter a phere or finihe execution. n addition, a linked lit i maintained in a non decreaing order which ort the node of phere according to their load. he load of a node i the number of tak in the execution queue of that node. he firt element of the lit point to the mot lightly loaded node of the phere. he lit i adjuted whenever a tak i cheduled at a node or a tak finihe it execution. t i poible that a node i hared by more than one cheduler. n that cae, the cheduler that aign the tak to the hared node inform other cheduler to update their linked lit and load entrie. Similarly, a node ha to inform all of it cheduler whenever it finihe a tak he Load Balancing Algorithm A mentioned earlier, at the econd level, load balancing i done optimally within the phere by cheduler. Becaue of the orted lit maintained by the cheduler, a tak i alway cheduled at a node with the lowet load. At the higher level, load balancing i achieved by exchanging tak among phere o that the cumulative load between phere i alo equalized. Whenever a cheduler receive a tak from the outide world or from another cheduler, it execute the cheduling algorithm. Aociated with the cheduling algorithm are two parameter, namely threhold-! and threhold-2 which are ued to decide tak migration. he two threhold are adjutable ytem parameter which are et depending upon a number of factor (decribed later). hrehold-! i the load of the mot lightly loaded node within the phere when the tak i not to be cheduled within the local phere. hrehold-2 i the difference between the cumulative load of the local phere and the cumulative load of the remote phere when the tak i to be migrated to a remote phere. he load balancing algorithm i executed by a cheduler at the time it receive a locally generated tak. t conit of the following tep. Step 1. Check the load value of the node pointed by the firt element of the linked lit. hi load i the mot lightly loaded node in the local phere. Step 2. f the load of the mot lightly loaded node i le than or equal to threhold-, then go to tep 3. Otherwie go to tep 6. Step 3. Schedule the tak at that node. Step 5. ncrement the accumulative load of the phere. Stop. Step 6. Check the accumulative load of other phere. Step 7. f the difference between the cumulative load of the local phere and the cumulative load of the mot lightly loaded remote phere i le than threhold-2, end the tak to that remote phere where t i executed without further migration to any other phere. f there are more than one uch phere, elect one randomly. f there i no uch phere, then go to tep 3. Stop. Step 4. Updated the linked lit by adding the new load to the original value and adut the lit accordingly. -17-

21 hrehold-1 determine whether the tak hould be cheduled in the local phere or a remote phere hould be conidered for tranferring the tak. Suppoe the load threhold i et to one. hen if an idle node i available in the phere, that node i obviouly the bet poible choice. ven if the mot lightly loaded node already contain one tak in it local queue, the probability of that node becoming idle during the time tak migrate from the cheduler to that node, i high. he cheduler conider tak migration to another phere only if the load of the mot lightly loaded node in it phere i greater than the threhold-1. hrehold-2 determine if there i ignificant difference between the accumulative load of the local phere and that of the remote phere. One of the remote phere, which meet the threhold-2 criteria, i randomly elected. he reaon for electing a phere randomly i to avoid the mot lightly loaded phere becoming a victim of tak dumping from other phere. Choice of the load threhold hould be made according to the ytem load and the tak tranfer rate. n our imulation tudy, the value for threhold-1 have been varied between 1 and 2 and threhold-2 i varied from 1 to 6. he reaon for electing two threhold i to reduce the complexity of the cheduling algorithm. he algorithm top at tep 5 if a node with load le than threhold-1 i preent in the local phere. hi alo avoid generation of unneceary meage for information collection from other cheduler, a hown in tep 6 of the algorithm. 6. Performance valuation and Comparion For the performance evaluation of the propoed trategy, we have imulated variou D network uch a Q7, Q, Q9, Qw, B7, B9 and 6. he imulation package written for thi purpoe run on an ncore ultimax. For comparion we have elected the no load balancing trategy and a fully decentralized trategy. For the no load balancing trategy, tak arrive at all node of the ytem with a uniform arrival rate and are executed on the FCFS bai, without any load balancing. n the fully ditributed trategy, the control i fully decentralized and every node execute the ame load balancing algorithm. ak can migrate between node depending upon the deciion taken by the algorithm at each individual node. When a tak arrive at a node, that node get the load tatu from it immediate neighbor. he load tatu of a node i the number of tak cheduled at that node. f the local load i le than the load of the mot lightly loaded neighbor, the tak i executed locally. Otherwie the tak i migrated to the neighbor with the lowet load. A tak i allowed to make many migration until it find a uitable node or the number of migration made by the tak exceed a predefined tranfer limit. Several variant of thi algorithm, uch a Shortet [14], [44], Contracting Within eighborhood [27] and Greedy trategy [12] have alo been reported. he baic idea behind thi algorithm i to chedule the tak at a node with minimum load. We believe that, for a fully ditributed trategy, load exchange between neighbor i both realitic and efficient - a pointed out in a comparion [27] where thi trategy i found to perform better than another fully ditributed trategy known a Gradient odel [28]. For imulation, tak arrival proce ha been modeled a a Poion proce with average arrival rate of A. tak/unit-time which i identical for all node. he execution and communication time of -18-

22 tak have been aumed to be exponentially ditributed with a mean of 1/ 1-l time-unit/tak and 1/!Jc time-unit/tak, repectively. he performance meaure elected are mean repone time of a tak and average number of control meage generated per tak. n each imulation run, 2, to 1, tak were generated depending upon the ize of the network. he teady-tate reult are preented with 95 percent confidence interval, with the ize of the interval varying up to 5 percent of the mean value of a number of independent replication. xtenive imulation have been performed to determine the impact of the following parameter: he ytem load. he channel communication rate. Size and topology of the network. Comparion with other cheme. he detail of the impact of thee parameter i given in the following ection epone ime Performance he mean repone time of a tak i the major performance criteria. 1b analyze the impact of ytem load, defined a l/l.t, on mean repone time, the ytem load i varied from.1 to.9. he parameter elected in imulation are thoe that produced the bet achievable performance for both trategie. For different network, the tak tranfer limit for the fully ditributed trategy, for intance, i equal to the diameter which produced the bet reult through imulation. he communication delay incurred during tak migration dratically effect the tak repone time and a higher value of tak tranfer rate benefit both the trategie. However, to make meaningful and fair comparion and not ignoring the impact of communication delay at all, the tak tranfer rate i elected a 2 tak/time-unit compared to ervice rate of 1 tak/time-unit. everthele, the impact of tranfer delay, with higher and lower value of tak tranfer rate, i evaluated eparately and i preented in ection 6.4. Figure 5, 6, and 7 how the average repone time curve veru varying load condition for Q 1 o, B 9 and 6, repectively. Both fully and emi-ditributed trategie yield a ignificant improvement in repone time over the no load balancing trategy at all loading condition. For Q1, coniting of 124 node, the average repone time of the propoed emi-ditributed trategy i uperior to the fully ditributed trategy, at all loading condition a hown in Figure 5. t i to be noted that for utilization ratio up to.8, the repone time curve with emi-ditributed trategy i rather mooth and the average repone time i almot equal to 1., which i in fact the average ervice time of a tak. hi implie that with the emi-ditributed trategy the load balancing i optimal and tak are erviced virtually without any queuing delay. hi i due to the fact that under low loading condition, a cheduler i uually able to find a node whoe load index i le than or equal to threhold-1 which i et to one in thi cae. For Q 1, the phere ize i 176 and the probability of finding an idle node in a phere i very high. n other word, the cheduler alway make ue of an idle node in it own phere. he only delay incurred before a tak get executed i the communication delay reulting from tak tranfer from a cheduler to -19-

23 a node or from cheduler to cheduler. At lightly higher load level, the inter-phere tak migration tart if the cheduler do not find uitable node in their local phere. At a very high load, the tak migration among phere take place more frequently and the load i balanced between heavily and lightly loaded phere of the network, in addition to the load being balanced within the phere. When load balancing among phere take place, extra delay are incurred due to migration of tak between the cheduler. n the propoed emi-ditributed cheme, all cheduler are at equal ditance from each other except for hypercube where for each cheduler, another cheduler, whoe binary addre i the complement of thi cheduler, i alo preent. Such a pair of node i called antipodal pair. herefore, with et C = 2n, each cheduler i at equal ditance from 2n -2 cheduler (all except itelf and it complement). hi reult in an inter phere load migration in a ymmetric and decentralized manner. For B 9 and 6, all cheduler are at equal ditance from each other. A hown in Figure 6 and Figure 7, the repone time curve obtained for the three trategie for B 9 and 6 exhibit imilar pattern, with the emi-ditributed trategy outperforming the fully ditributed (time unit) l A p 6_, r , o Load Balancing.A. Fully Ditributed 5 )( Semi-Ditributed ~--~~--~~~~--r--r--~~-,-~-~--r--r--~~.1 ~4 ~5 ~6.9 LOAD P OD (A./p.) Figure 5 : he mean repone time veru ytem load with three trategie for the Q1 network. -2-

24 trategy. We have alo examined the performance of both trategie for Q 7, Q 8, Q 9 and B 7, at varying load condition and reult (not hown here) indicated that the emi-ditributed trategy yield better repone time. For the emi-ditributed trategy, threhold-1 i varied from 1 to 2 wherea threhold-2 i varied from 1 to 6, depending upon the ytem load and network characteritic. A lower (higher) value of threhold-2 dictate more (le) tak migration. he optimal value of the two threhold for all the network could not be determined due to prohibitively high imulation time. However, we obetved that a lightly higher value of threhold-2, uch a 6, yield good repone time at high load and a lower value, uch a 2 and 3, work better at low and medium load. he ize of the phere and the number of phere alo affect the choice of threhold-2. A higher value of threhold-2 i ueful for a ytem with a larger phere ize and a fewer number of phere compared to a ytem with maller phere ize and higher number of phere. For intance, Q 8 and B 9 have the ame number of node but the phere ize of B9 i larger than that of Q 8 and the et C i maller for B9. A a reult, a value of 3 for threhold-2 i found ueful for Q but a value of 5 i found to perform well for B9, at ytem load (time unit) 6 l A 4 3 p 2 1 o Load Balancing 5.t. Fully Ditributed )( Semi-Ditributed WAD P OD (A/ /l) Figure 6 : he mean repone time veru ytem load with three trategie for the B9 network -21-

25 (time unit) l A o Load Balancing A Fully Ditributed X Semi-Ditributed 3 p LOAD P OD ().j p) Figure 7 : he mean repone time veru ytem load with three trategie for the 6 network equal to.9. Similarly, for Q9 and Q1, which have equal number of phere (but of different ize), threhold-2 i adjuted a 5 and 6, repectively. Almot, for all network, a value of threhold-1 equal to 1 i found a good choice, except for high load in B 7 and Q 7, where threhold-1 and threhold-2 equal to 2 and 4, repectively, performed better. However, threhold-1 equal to 1 till performed better at low loading condition. t i alo conjectured that a higher value of threhold-1 hould perform better at a low tak tranfer rate, o that le number of tak are migrated to other phere Analyi of Load Ditribution n addition to providing a fat job turn around time, a good load balancing cheme provide a better reource utilization by trying to keep all node equally buy. Standard deviation of accumulative utilization of all the node i a meaure of goodne of a load balancing trategy giving an etimate of moothne of load ditribution. Figure 8 how tandard deviation curve for the three trategie with the ame parameter a thoe for Figure 5. he curve for the no load balancing trategy preent the variation and imbalance in utilization if the work load originally aigned to the ytem i pro- -22-

26 13 A D A 11 D o Load Balancing.t. Fully Ditributed X Semi-Ditributed D v 9 F u L z A r~--r-.--.~-.--~.---~ ~.1 UUZAO AO Figure 8: he tandard deviation of utilization of all node veru ytem load for the Qto network. ceed without any load balancing. Low tandard deviation, for both load balancing trategie, indicate a more uniform ditribution of load. However the emi-ditributed trategy reult in a better load balancing a indicated by low value of tandard deviation. hi i due to the efficient cheduling algorithm executed by cheduler ince the node with identical load are handled on an equal priority bai. hi i due to the fact that the potion of a node in the linked lit i adjuted with arrivavcompletion of a tak. Conequently, the load of all the node within a phere tend to be optimally balanced. At high load, occaional pike of high loading are moothed out by ending load to other phere. High variation in utilization at low loading condition are due to the fact that no interphere tak migration take place and in the long run, ome phere may be ubject to more tak than other Performance Comparion of Different etwork n order to analyze the impact of network ize, the phere ize and the number of cheduler, on the performance of the propoed trategy, we compare the average repone time in B 7, Q7, Q, B 9, 6, Q9and Q1 network. For thee network, able V how the percentage improvement in average repone time yielded by emi and fully ditributed trategie over the no load balancing trategy. -23-

27 hree different loading condition, low, medium and high have been choen, correponding to ytem load of.6,.8 and.9 repectively. he tak tranfer rate for all thee reult ha been elected to be 2 tak/unit-time and tranfer limit for fully ditributed trategy i et to be equal to the diameter of the network. he mot intereting obervation i the increae in pedormance of emi-ditributed trategy with the increae in ytem ize. On the other hand the pedormance of fully ditributed trategy drop a the ytem ize increae. hee obervation are valid for all loading condition. hi implie that the propoed emi-ditributed trategy i more uitable for large ytem. For example, for Q 1, at high load, repone time improvement of 82.4% and 62.57% for emi and fully ditributed trategie, repectively, can be noticed. Comparing the performance of Q 8 and B 9, which have identical number of node but different number of cheduler and phere ize, we notice that B 9 outperform Q 8 at medium load. hi i due to the larger phere ize of B 9 which reult in an increaed probability of finding an idle node within the local phere. he performance of the fully ditributed trategy i lightly better than the emi-ditributed trategy for B1 network, at high load. However, the emi-ditributed trategy outperform the fully ditributed trategy for all other network. ABL V he percentage repone improvement (decreae) over no load balancing with emi and fully ditributed trategie for variou network Low (.6) edium (.8) High (.9) ~ Semi Fully Semi Fully Semi Fully k Dit. Dit. Dit. Dit. Dit. Dit. B Q Q B Q Qto

28 6.4. Senitivity o Communication Delay he repone time performance of a load balancing trategy can degrade becaue of the communication delay incurred during tak migration. oreover, high tranmiion delay not only low tak migration but alo reult in an increaed queuing delay in the communication queue. he impact of communication delay on the tak repone time i determined by the ratio of mean communication delay to mean ervice time. f thi ratio i high, load balancing doe not prove beneficial ince it average repone time may even exceed the repone time obtained with no load balancing [32]. f fully ditributed load balancing i carried out with low communication, the tate of the ytem can change by the time a tak migration complete, and the tak may have to be remigrated to another node. hi may reult in tak thrahing. a phenomenon in which tak keep on migrating between node without etting down. On the other hand, with the emi-ditributed load balancing cheme, a tak migrate only from a cheduler to a node and/or from cheduler to cheduler. However, ince both cheme are uceptible to the communication rate of the underlying network, we preent the repone time performance for varying tak tranfer rate at medium and high ytem load, for all even network. For Q 1 Q9, Q 8, and Q 7, the curve for percentage improvement in average repone time, over the no load balancing trategy, veru varying tak tranfer rate, have been obtained with both the trategie. hee curve for low and high load are hown in Figure 9(a) and Figure 9(b ), repectively. hee reult alo help u examine the performance of hypercube topology of varying ize under different loading and communication condition. ak tranfer rate are varied from 4 tak/time-unit to 3 tak/time-unit; by increaing the tak tranfer rate beyond 3 tak/time-unit, the performance improvement ha been found to be negligible. he reult indicate that even at a very low tak tranfer rate, the repone time improvement with the emi-ditributed trategy i better than the improvement gained with the fully ditributed trategy. At ytem load equal to.6, the repone time improvement yielded by the fully ditributed cheme decreae with increae in ytem ize wherea for the emi-ditributed cheme, Q and Q7 yield the bet and the wort performance, repectively. However, all network perform better with the emi-ditributed trategy. At high load, the difference in the performance of both trategie i ignificantly high. Figure 9(b) clearly indicate that the repone time improvement, with the propoed trategy, i enhanced for the larger ytem. On the other hand, with the fully ditributed load balancing, the performance decreae with increae in ytem ize. From Figure 9(a), we alo oberve that, with the exception of Q 7, the other three network perform reaonably good under the emi-ditributed cheme, even at very low tak tranfer rate. n contrat, Figure 9(b) indicate that the performance of the emi-ditributed cheme keep on increaing with the increae in tak tranfer rate wherea the performance of the fully ditributed cheme aturate if the tak tranfer rate i increaed beyond 2 tak/time-unit. Figure lo(a) and Figure lo(b) preent percentage repone time improvement at low and high load, repectively, for the two Biectional network, B 9 and B7 From Figure lo(a), we notice the propoed cheme perform conitently better for B 9 wherea it how ome dependency toward the tak -25-

29 t % p 74-~------~=:;~;~~~~j~i~~ 9 ~ - e A Semi-Ditributed Q8 Q7 ~ Semi-Ditributed Q9, ""- X Semi-Ditributed Qut ',''e Fully Ditributed Q7 ~ '\' A Fully Ditributed Q \~ Fully Ditnbuted Q9 '\ X Fully Ditributed QlO p v t p p v ASK ASF A ('hkltime-unit) Figure 9(a): he percentage improvement in repone time obtained with fully and emi-ditributed trategie for Qtn Qg Qand Q7 at low load ( A./ J. =.6). % _ ,... e Fully Ditributed Q7... -: ',..e, _... _ -+- _ _ - _ - _... _ -11- _,... A Fully Ditributed Q8..,,'... ~ ' Q,.-:,- _* _* _ ~- _ K--)(- -)- -)-- '\ Fully Ditributed 9,'.- )( _)(--* \,',,' /J( :,:x- - '. X Fully Ditributed Qto,~,, 1 1 ASK ASF A (hk/time-unit) Figure 9(b ): he percentage improvement in repone time obtained with fully and emi-ditributed trategie for QlO, Qg Q and Q1 at high load ( J..j p. =.9). -26-

30 t p r p v % ,','X ',', Semi-Ditributed B9 ~ ~---~t:::::..-:--.jt'~llt='==jl==jt=:jt==!~~... X Semi-Ditributed B1... -,)('-~ v-', '" ~--_,x--,'x' _- ----> _-:>--)(- -:>- X Fully Ditributed B )- -> Fully Ditributed B1 ASK ASF A (hk!time-unit) Figure lo(a): he percentage improvement in repone time obtained with fully and emi-ditributed trategie for B9 and B1 at low load ( A./ f.l =.6). t % 75 p 7 p 65 v 6 K',x--,X", -~-~ )(- -)(- -)(- -)- ->-- '>- -X-- ASK ASF A (hk/time-unit).. ~ Semi-Ditributed B1 ' ' ' Fully Ditributed B1./ X Fully Ditributed B9./../ Figure lo(b): he percentage improvement in repone time obtained with fully and emi-ditributed trategie for B9 and B 7 at high load ( A./ f.l =.9). -27-

31 i p % -- X Fully Ditributed -- if p v 7,,, ASK ASF A (Dik/time-unit) Figure ll(a): he percentage improvement in repone time obtained with fully and emi-ditributed trategie for 6 at low load (A./ t =.6). i % p p v 67 x ' ' )(--)(- ->- ->t- ->-- ~-- )(- ->- -i<--,,x-- ~ -- ~.-1~~-*"-74r-"l ~X Semi-Ditributed ~-r.'~ ~~--,--~,.-,.-,-,~~ _... X Fully Ditributed...- Figure ll(b) : he percentage improvement in repone time obtained with fully and emi-ditributed trategie for 6 at high load ( l/ t =.9). -28-

32 tranfer rate for the B 7 network. arlier we noticed the ame reult for the Q 1, a hown in Figure 9(a), which lead to the concluion that the maller ytem are more uceptible to tak tranfer rate, under the propoed cheme. At high load, the repone time improvement, obtained with both the trategie are almot identical for the B 7 network wherea the reult are ubtantially different for the B 9 network. hi reconfirm our concluion that larger ytem perform better under the propoed trategy. he reult hown in Figure ll(a) and Figure ll(b) for the 6 network alo reconfirm thee reult. hee reult alo indicate that the gain in performance over no load balancing i even better at high loading condition he eage Overhead he exchange of control information, which i eential for any load balancing algorithm, hould be carried out in an efficient way. n order to deign a tate information collection policy, one mut decide what type of information i required and how frequently that information hould be interchanged. n addition, the information exchange policy hould meet at leat two objective. he firt objective i to enure that the information i accurate and ufficient! enough for the proper operation of the cheduling algorithm. he econd objective hould be to limit the amount and frequency of the information interchange within an acceptable level o that the reulting overhead i not too high. n a completely decentralized policy, every node need to maintain it own view of the ytem tate uch a load tatu of neighbor. Conequently, the frequency and amount of thee meage exchange can caue extra channel traffic which can low down the information exchange activity a well a normal tak migration. n order to have an updated information, the control meage need to be handled on ome form of priority bai. f a tak make a large number of migration, all the intermediate node generate extra meage and the overhead increae proportionally. Clearly a trade-off i at work here. One need to have ufficient information available to the cheduler while reducing the reulting overhead. n addition to average tak repone time, thi meage overhead, therefore, i another performance meaure of a load balancing policy. One of the goal of the propoed trategy i to reduce thi control overhead. We define thi overhead a the average number of meage generated per tak (thi wa calculated by dividing the total number of generated meage by the total number of tak). A explained in ection 5, in the propoed trategy, thi overhead reult from two type of information, that i, the load tatu of individual node within the phere and the accumulative load of the phere. f a node i hared among more than one phere, then the cheduler aigning the tak to that node inform the cheduler of other phere to update their load entrie for that node. Upon finihing a tak, a node inform all of it cheduler. hi reult in a conitent load information about all the node and yet the number of meage generated i coniderably mall. At the phere level, a cheduler communicate with other cheduler only when it conider a tak migration. A mentioned earlier the number of cheduler i only of the order of log. herefore, the number of uch meage i alo mall. However, the number of meage alo depend on other factor uch a ytem load, the two threhold and partitioning trategy which in turn determine the ize of -29-

33 u F - Semi-Ditributed c:::j Fully Ditributed A ~-+~~~~~~~~~~~ G ULZAO AO Figure 12: he average number of meage per tak veru ytem load for B1. r u F A G - Semi-Ditributed c:::j Fully Ditributed ULZAO AO Figure 13: he average number of meage per tak veru ytem load for Q7. u F Semi-Ditributed Fully Ditributed A G UUZAOAO Figure 14: he average number of meage per tak veru ytem load for Q. -3-

34 the phere, number of phere and haring of node among phere. t i difficult to analyze the combined effect of all thee factor, however, imulation reult preented in thi ection enable u to briefly explain the impact of thee factor, individually. n Figure 12 to 18, the average number of control meage, with varying ytem load, are plotted for B1, Q1, Q, Bg, Qg, Q1 and 6 network, repectively. hee figure indicate that the overhead i low at low loading condition. hi i due to the fact, that at low load, the cheduler do not communicate with each other ince a uitable node i available within the local phere, mot of the time. Conequently, the overhead at low loading condition i only a reult of meage exchange between node and cheduler, whenever the load tatu of a hared node i updated among cheduler. A the load increae from medium to high, the probability of finding a tak within a local phere decreae and cheduler tart communicating with each other, depending upon the value of the two threhold. f the two threhold have high value, then the frequency of uch information exchange decreae and conequently the overhead decreae. However, thi can reult in a higher repone time. t hould be pointed out that the two threhold for the reult depicted in Figure 12 to Figure 18 have been adjuted not with the objective of reducing the overhead; rather thee reult preent the overhead with threhold value that produced good repone time. n contrat, the fully ditributed trategy induce high overhead which almot double the overhead incurred by the emi-ditributed trategy, except for 6 and Qto where the overhead reulting from the emi-ditributed trategy i high due to high degree of haring of node among phere. At high load, the propoed trategy till yield better performance by inducing le overhead a compared to the fully ditributed trategy. One exception i the Q7 network, whoe phere ize i mall and, a reult, the cheduler need to communication more often in order to balance load among phere. he impact of phere ize i more apparent when the meage overhead for Q and B9 i compared. From Figure 14 and Figure 15, we note that B 9 perform better than Q; at all loading condition, the overhead for the emi-ditributed trategy with Q i higher than the overhead incurred by B 9 hi i becaue the number of cheduler in B 9 i half the number of cheduler for Q and the phere ize of B 9 i larger than that of Q. hi overhead i alo mall becaue of le haring of node in different phere (in B 9 a node i hared in either 1 or 3 phere wherea in Q, a node i hared in 1 or 4 phere). Due to le number of phere and a larger phere ize, there are le inter phere migration in B 9 and the cheduler do not frequently communicate with each other. However, a hown in Figure 16 and Figure 18, when we compare the performance of 6 and Qg, which have almot equal number of node but different number of phere and hence difference ized phere, we oberve that the overhead incurred by Q9 i lightly le than the overhead incurred by o 6. he phere ize of 6 i 112 which i very large a oppoed to the phere ize of Q9 which i 46. he comparion of Q with B 9, and 6 with Qg, indicate that there i trade-off which dictate that the phere ize hould neither be very large nor very mall (the number of cheduler hould not be too mall or too large). However, when we compare the overhead produced by the two trategie, we note -31-

35 r u F A G 2 - Semi-Ditributed c:j Fully Ditnbuted ULZAO AO Figure 15:he average number of meage per tak veru ytem load for B9. - r 2 Semi-Ditributed c:j Fully Ditributed 15 u 1 F 5 A G ULZAO AO Figure 16: he average number of meage per tak veru ytem load for Q9. - r 3 Semi-Ditributed 25 c:j Fully Ditributed u 2 15 F 1 5 A G ULZAO AO JJo Figure 17: he average number of meage per tak veru ytem load for 1o. lijjo 1iJJo -32-

36 2 15 u 1 F 5 A G - Semi-Ditributed c::j Fully Ditributed ULZAO AO Figure 18:he average number of meage per tak veru ytem load for6. that the propoed trategy conitently perform better than the fully ditributed trategy. A hown in Figure 12, the concluion i valid even for B 7 where phere ize i very mall. he concluion alo hold for Q1, a hown in Figure 17, where the phere ize i very large. 7. Concluion n thi paper, we have propoed a new load balancing approach for large-cale multicomputer ytem and have preented the concept of a emi-ditributed control. he tudy wa centered around a cla of interconnection tructure which are ditance-tranitive. he ue of Hadamard matrix reult in an efficient trategy for partitioning thee ytem for load balancing. We have evaluated the performance of the propoed trategy through an extenive imulation tudy. By comparing with a fully ditributed trategy, we have hown that the propoed trategy yield a better performance in term of average repone time, reource utilization and average number of control meage generated. One of the concluion drawn from the comparative performance of Hypercube, Biectional and Odd graph i that the network topology ha a rather low effect on repone time. n other word, for the three kind of graph under tudy, the propoed trategy uing different number of phere with different ize perform equally good. hi i true for a wide range of tak tranfer rate of the communication network. We alo notice that the propoed trategy perform better a the ytem ize increae. hi wa oberved by comparing the performance of Hypercube a well a Biectional graph. he propoed partitioning trategy reult in a mall number of phere which remain of >( log ). For intance the number ofphere for Qu i 24 and remain the ame for Q12 Q13 and Q14 he next higher value of the number of phere i 32. Similar technique are applied to Biectional and Odd graph. -33-

37 eference [1] hfaq Ahmad, Arif Ghafoor and Kihan ehrotra, "Performance Prediction for Ditributed Load Balancing in ulticomputer Sytem," hchnical eport no. SU-CS-91-12, School of Computer and nformation Science, Syracue Univerity, April1991. [2] William C. Atha and Charle L. Seitz, "ulticomputer: eage-paing Concurrent Computer," Computer, Augut 1988, pp [3]. Bannai and'[ to, Algebraic Combinatoric and aociation cheme. Benjamin-Cumming (1984). [4] Amnon Barak and Amnon Shiloh," A Ditributed Load balancing Policy for a ulticomputer," Software Practice and xperience, vol. 15(9), Sept. 1985, pp [5] Katherine. Baumgartner, alph Kling and Benjamin Wah," A Global Load Balancing Strategy for a Ditributed Sytem," in Proc. of nt'l. Cont on Future rend in Ditributed Computing Sytem, Hong Kong, 1988 pp [6] Shahid H. Bokhari, "Dual Proceor Scheduling with Dynamic eaignment," ran. on Software ng. vol. S-5, July 1979, pp [7] ichola S. Bowen, Chrito ikolaou and Arif Ghafoor, "On the Aignment Problem of Arbitrary Proce Sytem to Heterogeneou Ditributed Computer Sytem," o appear in ran. on Computer. [8] aymong. Bryant and aphael A. Finkel, "A Stable Ditributed Scheduling Algorithm," in Proc. of 2nd nt'l. Cont on Ditributed Computing Sytem, 1981, pp [9] homa L. Caavant and John G. Kuhl, "Analyi of hree Dynamic Load-Balancing Strategie with Varying Global nformation equirement," in Proc. of 7-th nt'l. Cont on Ditributed Computing Sytem, 1987, pp [1] 'bny F. Chan,"Hierarchical Algorithm and Architecture for Parallel Scientific Computing," in nt'l. Conference on Supercomputing, 199, pp [11] Yaun-Chien Chow and Walter H. Kohler,"odel for Dynamic Load balancing in Homogeneou ultiple Proceor Sytem," ran. on Computer, vol. c-36, no. 6, ay, 1982, pp [12] Shyamal Chowdhury, ''he Greedy Load Sharing Algorithm," Journal of Parallel and Ditributed Computing, no. 9, ay 199, pp [13] DAPA, "Strategic Computing: ew Generation Computing technology," Defence Advance eearch Project Agency, Arlington, VA, Oct [14] Derek L. ager, dward D. Lazowka and John Zahorjan,"Adaptive Load Sharing in Homogeneou Ditributed Sytem," ran. on Software ng.,vol. S-12, ay 1986, pp [15],"A Comparion of eceiver nitiated and Sender nitiated Adaptive Load Sharing," AC Performance valuation, arch 1986, pp [16] Kemal fe and Bojan Groelj, "inimizing Control Overhead in Adaptive Load Sharing," in Proc. of 9-th nti. Cont on Ditributed Computing Sytem, 1989, pp

38 [17] Ahmed K. zzat,. Daniel Bergeron and John L. Pokoki, "ak Allocation Heuritic for Ditributed Computing Sytem, "in Proc. of 6-th nt'l. Cant on Ditributed Computing Sytem, 1986, pp [18] Donald Ferguon, Yechiam Yemini and Chrito ickolaou "icroeconomic Algorithm for Load Balancing in Ditributed Computer Sytem, " in Proc. of 8-th nt'l. Cant on Ditributed Computing Sytem, 1988, pp [19] G. C. Fox, A. Kolawa and. William, "he mplementation of a Dynamic Load Balancer, "in Proc. of SA Hypercube ultiproceor Cont. 1987, pp [2] G. Fox,. Johnon, G. Lyzenga, S. Otto, J. Salmon, and D. Walker, Solving Problem on Concurrent Proceor, Volume : General echnique and egular Problem, Prentice Hall, nglewood Cliff, ew Jerey, [21] Arif Ghafoor and heodore Bahkow, "A Study of Odd Graph a Fault-olerant nterconnection etwork," ran. on Computer, vol. 4, no. 2, February 1991, pp [22] Arif Ghafoor, heodore Bahkow and mran Ghafoor, "Biectional Fault-olerant Communication Architecture for Supercomputer Sytem," ran. on Computer, vol. 38, no. 1, October 1989, pp [23] Arif Ghafoor and P. Bruce Berra, "An fficient Communication Structure for Ditributed Commit Protocol," Jour. on Selected Area of Communication, vol. 7, no. 3, April. 1989, pp [24] Arif Ghafoor, Sohail Sheikh and Patrick Sole, "Ditance-ranitive Graph for fault-olerant ultiproceor Sytem," n Proc. of 1989 nt'l. Cant on Parallel Proceing, 1989, pp [25) Anna Ha'c and heodore J. Johnon," Senitivity Study of the Load Balancing Algorithm in a Ditributed Sytem," Journal of Parallel and Ditributed Computing, October 199, pp [26]. Hall Jr., Combinatorial heory, 2nd d., John Wiley and Son, ew York, [27] L.V. Kale, "Comparing the performance of two dynamic load ditribution method," Proceeding of nt'l. Cant on Parallel Proceing, 1988, pp [28] Frank C. H. Lin and obert. Keller, "Gradient odel: A demand Driven Load Balancing Schem," in Proc. of 6-th nt'l Cant on Ditributed Computing Sytem, 1986, pp [29]. Livny and. elman, "Load Balancing in Homogeneou Broadcat Ditributed Sytem," in Proc. of AC Computer etwork Performance Sympoium, April 1982, pp [3] A. cloughlin, 'he Complexity of Computing the Covering adiu of a Code," ran. on nform. heory, col -3, ov., 1984, pp [31] F. J. acwilliam and.j. A Sloane, he heory of "or-co"ecting Code, vol. and, ew York: orth Holland, [32]. irchandancy, D. owly and J. A Stankovic, "Analyi of ffect of Delay on Load Sharing," ran. on Computer, vol. 38. no. 11, ov. 1989, pp [33] Lionel. i and Kai Hwang, "Optimal Load Balancing in a ultiple Proceor Sytem with any Job Clae," ran. on Software ng., vol. S-11, ay 1985, pp