Analysis of Interconnection Networks in Heterogeneous Multi-Cluster Systems

Transcription

1 Analyss of Interconnecton Networks n Heterogeneous Mult-Cluster Systems Bahman Javad, Jemal H. Abawajy, Mohammad K. Akbar, Saed Nahavand Computer Engneerng and IT Department Amrkabr Unversty of Technology 44 Hafez Ave., Tehran, Iran {javad,akbar}@ce.aut.ac.r School of Eng. and Informaton Technology Deakn Unversty VIC. 37, Australa {jemal,nahavand}@deakn.edu.au Abstract The study of nterconnecton networks s mportant because the overall performance of a dstrbuted system s often crtcally hnged on the effectveness of ts nterconnecton network. In the mean tme, the heterogenety s one of the most mportant factors of such systems. Ths paper addresses the problem of nterconnecton networks performance modelng of large-scale dstrbuted systems wth emphases on heterogeneous mult-cluster computng systems. So, we present an analytcal model to predct message latency n mult-cluster systems n the presence of cluster sze heterogenety. The model s valdated through comprehensve smulaton, whch demonstrates that the proposed model exhbts a good degree of accuracy for varous system organzatons and under dfferent workng condtons.. Introducton An ncreasng trend n the hgh performance computng (HC development s towards the networked dstrbuted systems such as commodtybased cluster computng [] and grd computng [] systems. Due to advances n computatonal and communcaton technologes, t s economcally feasble to conglomerate multple clusters to development of large-scale dstrbuted systems known as mult-cluster systems. These systems are ganng momentum both n academc and commercal sectors and a wde varety of parallel applcatons are beng hosted on such systems as well [3-5]. In ths paper, we focus on the nterconnecton networks for mult-cluster computng systems. The study of nterconnecton networks s mportant because the overall performance of a dstrbuted system s often crtcally hnged on the effectveness of ts nterconnecton network. Also, the nterconnecton network desgn plays a central role n the desgn and development of mult-cluster computng systems. Smulaton has been used to nvestgate the performance of varous components of mult-cluster computng systems [3]. Instead, we focus on analytc model. Several analytcal performance models of multcomputer systems have been proposed n the lterature for dfferent nterconnecton networks and routng algorthms (e.g., [6-9]. Unfortunately, lttle attenton has been gven to the cluster computng systems. Most of the exstng researches are based on homogenous cluster systems and the evaluatons are confned to a sngle cluster [0-]. A general model based on queung networks was proposed for a sngle cluster computng n [0]. The model assumes that the processors are homogenous. Also, extensve numercal calculaton of the model renders t too complcated. Furthermore, the model cannot be used for multcluster computng systems n the presence of heterogenety. Also, a performance model for Network of Workstatons wth processor heterogenety s dscussed n [3]. The authors recently proposed an analytcal model for mult-cluster systems n the presence of processor heterogenety n [4, 5]. To ths end, we present an analytcal performance model of nterconnecton networks for mult-cluster computng systems. The model takes nto account stochastc quanttes as well as cluster szes heterogenety. The model s valdated through comprehensve smulaton, whch demonstrated that the proposed model exhbts a good degree of accuracy for varous system szes and under dfferent operatng condtons. The rest of the paper s organzed as follows. In Secton, a bref background s dscussed. In Secton 3, we gve detaled descrpton of the proposed analytcal model. We present the model valdaton experments n Secton 4. We summarze our fndngs and conclude the paper n Secton 5.

2 . Background The system under study n ths paper s a multcluster computng systems whch s made up of C clusters, each cluster s composed of N computng nodes, {0,,..., C }, each comprsng a processor wth processng power τ and ts assocated memory module as s depcted n Fg.. Also, each cluster has two communcaton networks, an Intra- Communcaton Network (ICN, whch s used for the purpose of message passng between processors, and an nter-communcaton Network (ECN, whch s used to transmt messages between clusters, management of the system, and also for the scalablty of the system. ICN ECN Cluster # (N Computng Nodes Intra-Communcaton Network (ICN Cluster #(C- (N C- Computng Nodes Fg.. Heterogeneous mult-cluster system It should be noted that, ECN can be accessed drectly by the processors of each cluster wthout gong through the ICN (see Fg.. To nterconnect ECN and ICN, a set of Concentrators/Dspatchers [0] are used, whch combne message traffc from/to one cluster to/from other cluster. As mentoned before, the nterconnecton networks n such systems are crucal n ganng a desrable speedup. However, havng a rapd network does not necessarly guarantee to obtan a good performance, due to contenton problems. The contenton problems whch adversely affect the overall performance would happen n host nodes, network lnks, and network swtches [4]. Node contenton happens when multple data packets compete to contan a receve channel of a node, but lnk contenton occurs when two or more packets share a communcaton lnk. The swtch contenton s due to unbalanced traffc flow through the swtch, whch would result n overflow of the swtch buffer. The man factors whch have mpact on contenton of an nterconnecton network and determne ts performance are Topology, Routng algorthm and Flow control mechansm. Topology defnes how the network s physcally connected together. Hgh performance computng clusters typcally utlze Clos networks, more commonly knows as Fat-Tree or Constant Bsectonal Bandwdth networks to construct large node count non-blockng swtch confguratons [, ]. In ths paper we adopted m-port n-tree [5] as a fxed arty swtches to construct the topology for each cluster system. An m-port n-tree topology conssts of N processng nodes and N sw network swtches whch can be calculated as follows: m N n n ( m Nsw (n ( In addton, each network swtch tself has m 0,,,..., m that are communcaton ports { } attached to other swtches or processng nodes. Every swtch except root swtches uses ports n the range of { 0,,,..., ( m / } to have connecton wth ts descendants or processng node, and usng ports n the range of {( m/,( m/ +,..., m } for connecton wth ts ancestors. It can be shown that the m-port n- tree s a full bsecton bandwdth topology [8], so the lnk contenton doesn t occur n such network. Routng algorthms establsh the path between the source and the destnaton of a message. The most commercal cluster networks (e.g. Myrnet, InfnBand and QsNet adopt determnstc routngs [6]. The smplest determnstc routng used n such networks s Up*/Down* routng [7] whch can be used n networks wth both source and dstrbuted routng. Of ths, we used a determnstc routng based on Up*/Down* routng whch s proposed n [8]. In ths algorthm, each message experences two phases, an ascendng phase to get to a Nearest Common Ancestor (NCA, followed by a descendng phase. Furthermore, snce ths algorthm performs a balanced traffc dstrbuton, so the swtch contenton problem wll be extngushed. Flow control manages the allocaton of resource to messages as they progress along ther route. Two most famous flow control mechansms are store-andforward and wormhole flow control whch are wdely used n the commercal swtches. Snce the most dedcated cluster network technologes are usng wormhole flow control, we adopt ths mechansm to outlne the analytcal model. In regards of exstng heterogenety n such systems and based on the dscussons of [3], we categorzed possble heterogenety n mult-cluster systems as follows: Communcaton networks rocessors computatonal power System organzatons (e.g., cluster sze It s obvous that develop an analytcal model to cover all types of heterogenety would be qute complcated.

3 Hence, n ths paper we consder the last category (.e., system organzaton as t s llustrated n the followng sectons. 3. The Analytcal Model In ths secton, we develop an analytc model for the above mentoned mult-cluster system. The proposed model s bult on the bass of the followng assumptons whch are wdely used n smlar studes [6-0]:. Nodes generate traffc ndependently of each other, and whch follows a osson process wth a mean rate of g messages per tme unt. Moreover, the arrval process at a gven channel of each network s approxmated by an ndependent osson process.. The destnaton of each request would be any node n the system wth unform dstrbuton. 3. The number of processors n each cluster s dfferent ( N and the processng power of cluster s nodes are homogenous wth the same processng power ( τ 0 τ... τc. 4. The network swtches are nput buffered and each channel s assocated wth a sngle flt buffer. 5. Message length s fxed (M flts. 6. The source queue at the njecton channel n the source node has nfnte capacty. Moreover, messages are transferred to the node once they arrve at ther destnatons. 3.. Mean Network Latency In what follows, we fnd the mean latency of each communcaton network from cluster pont of vew, S. Snce each message may cross dfferent number of lnks to reach ts destnaton, we consder the network latency of an j-lnk message as S, and j averagng over all the possble nodes destned made by a message yelds the mean network latency as: ( jn, j n (3 j S S Where j, n s the probablty of a message whch s orgnated from cluster crossng j-lnk (j-lnk n the ascendng and j-lnk n the descendng phase to reach ts destnaton n a m-port n -tree topology. As t s mentoned n assumpton, we take nto account the unform traffc pattern so, based on the m-port n -tree topology, we can defne ths probablty as follows: j m m j,,..., n N m ( m j n N jn, j 3... Channel Message Rate (4 The message flow model of the system s shown n Fg., where the path of a flt through varous communcaton networks s llustrated. As shown n the model, the processor requests wll be drected to ICN and ECN by probabltes o and o respectvely, where {0,,..., C }. The external message of cluster leaves the ECN at the end of ascendng phase and crosses through the ICN and then start the descendng phase n the ECN of the cluster v to reach ts destnaton node. Hence, t s lke that a complete journey n the ECN. Therefore, the message rate receved n each networks can be obtaned as follows: ( ( I N ( o g (5 N + N (6 ( v E o g v o g (( ( ( v N N + N N (, o v o v v g I (7 N + Nv Gven that a newly generated message n cluster makes j-lnk to reach ts destnaton wth probablty j, n, the average number of lnks that a message makes to reach ts destnaton s gven by: avg ( j, n n (8 j d j Wth substtutng of Eq.(4 n Eq.(8, the average message dstance s obtaned as, d avg m ( mn n + n m m n (9

4 - o ( ICN (m-port n-tree g o ( ECN (m-port n-tree Con./Ds. ICN (m-port n c-tree ECN (m-port nv-tree Con./Ds. Fg.. Message flow model n the system Consequently, we could derve the rate of receved messages n each channel, whch can be wrtten as: η η η I E I ( I d avg( I (0 4nN E davg( E ( 4nN I davg( I ( 4n c where n c, the number of trees n the ICN compute n such that C ( m/ c. As t s depcted n Fg., ( the probablty o has been used as the probablty of outgong requests wthn cluster. Accordng to assumpton, ths parameter s computed by the followng equaton: o C j 0, j N N j 3... Mean Channel Servce Tme g (3 In ths topology we have two types of connectons, node to swtch (or swtch to node and swtch to swtch. In the frst and the last stage, we have node to swtch and swtch to node connecton respectvely. In the mddle stages, the swtch to swtch connecton s employed. Each type of connecton has a servce tme whch s approxmated as follows: t t α + L β (4 cn net m net α + L β (5 cs sw m net where t cn and t cs represent tmes to transmt from node to swtch (or swtch to node and swtch to swtch connecton, respectvely. α net and α sw are the network and swtch latency, β net s the transmsson tme of one byte (nverse of bandwdth and Lm s the length of each flt n bytes. Our analyss begns at the last stage and contnues backward to the frst stage. The number of stage for a message wth j-lnk journey s K j. The destnaton, stage K, s always able to receve a message, so the servce tme gven to a message at the fnal stage s t cn. The servce tme at nternal stages mght be more because a channel would be dled when the channel of subsequent stage s busy. The mean amount of tme that a message wats to acqure a channel at stage k for cluster, W k, j, s gven by the product of the channel blockng probablty n stage k, B, and the mean servce tme of a channel at k, j stage k, S k, j / [5]: W S (6 k, j k, j Bk, j The value of B s determned usng a brthdeath Markov chan [5]. Solvng ths chan for the k, j steady state probabltes gves: η S (7 B k k, j k, j The mean servce tme of a channel at stage k s equal to the message transfer tme and watng tme at subsequent stages to acqure a channel, so: S k, j K ( Ws, j + Mt otherwse (8 cs s k+ Mtcn k K Accordng to ths equaton, the mean network latency S S. s equal to 0, j ( j 3.. Mean Message Latency A message orgnatng from a gven source node n cluster sees a network latency of S (gven by Eq.(3. Due to blockng stuaton that takes place n the network, the dstrbuton functon of message latency becomes general. Therefore, a channel at source node s modeled as an M/G/ queue. The mean watng tme for an M/G/ queue s gven by [9]:

5 W ρ ρ x ( + Cx ( ρ ( (9 x (0 ( σ ( x C x ( ( x where s the mean arrval rate on the network, ( x s the mean servce tme, and σ s the varance x of the servce tme dstrbuton. Snce the mnmum servce tme of a message at the frst stage s equal to Mt cn, the varance of the servce tme dstrbuton s approxmated based on a method proposed by Draper and Ghosh [8] as follows: ( σ ( x S Mt cn As a result, the mean watng tme n source queue becomes, W ( S Mtcn ( S + ( S ( S (3 At last, the mean tme for the tal flt to reach the destnaton can be wrtten by the followng equaton: n K R t + t jn, cs cn j k (4 The mean latency seen by the message, T, crossng from source node from cluster to destnaton, conssts of three parts; the mean watng tme at the source queue ( W, the mean servce tme at the frst stage ( S, and the mean tme for the tal flt to reach the destnaton ( R. Hence, T W + S + R (5 The mean message latency n the ICN from cluster pont of vew, T I, would be found by the above equaton wth substtuton of ηk ηi and ( I Mean Message Latency for Inter-Cluster Networks As mentoned before, external messages cross through both networks, ECN and ICN, to get to the destnaton n other cluster. Snce the flow control mechansm s wormhole, the latency of these networks should be calculated as a merge one. Therefore based on the Eq.(3, we can wrte, n nv nc E& I ( jl, h ( jl, h j l h ( + + S S (6 It means each external message cross (j+l-lnk through the ECN (j-lnk n the source cluster and l- lnk n the destnaton cluster v and h-lnk n the ICN to reach ts destnaton. It can be shown that the + would be, ( j, l h + (7 ( j, l h j, n l, nv h, nc In the nter-cluster networks, the number of stages for each message journey s K j+ h+ l. Based on Eqs.(6 and (7, the mean amount of tme that a message wats to acqure a channel at stage k, n the nter-cluster networks s as follows: W η ( S (8 k,( j, l + h k k,( j, l + h Where the channel rate s drven by the followng equaton: η k ηi j k < j+ h η E otherwse (9 Smlar to the ntra-cluster network, the network latency for an nter-cluster message equals to the mean servce tme of a channel at stage 0 and can be found by Eq.(8. As before, the source queue s modeled as an M/G/ queue and the same method s used to approxmate the varance of servce tme. Thus, the mean watng tme of the source queue n the ntercluster networks can be calculated as: W E& I ( ( E & I S Mtcn S E& I + ( S E & I ( E S E& I E (30 Fnally, the arthmetc average s used to compute the mean message latency n the nter-cluster networks from cluster pont of vew, as follows:

6 T C v 0, v E& I ( W E& I + SE& I + RE & I C (3 Where the mean tme for the tal flt to reach the destnaton can be obtaned as follows: R t t n nv nc K E& I ( jl, + h cs+ cn j l h k (3 The concentrator/dspatcher s workng as smple b-drectonal buffers to nterface two external networks (.e., ECN and ICN. The mean watng tme at the concentrator/dspatcher s calculated n a smlar manner to that for the source queue (Eq.(9. The servce tme of the queue would be Mt cs and there s no varance n the servce tme, snce the messages length s fxed. So, the mean watng tme are gven by followng equatons: W s I ( Mt cs ( I Mtcs (33 Also, we model the dspatch buffers n the concentrator/dspatcher as an M/G/ queue, wth the same rate of concentrate buffers. So the mean watng tme s gven smlarly by Eq.(33. The arthmetc average of sum of the two above mentoned watng tmes gves mean watng tme at the concentrator/dspatcher as follows: C (, v W d ( W s (34 C v 0, v uttng all together, we could fnd the mean message latency from cluster pont of vew (based on Fg. wth the followng equaton: ( ( ( o ( T I o ( T E& I W d + + (35 To calculate the total mean of message latency, we use a weghted arthmetc average as follows: C N 0 N (36 4. Valdaton of the Model In order to valdate the proposed model and justfy the appled approxmatons, the model was smulated. The smulator uses the same assumptons as the analyss. Messages are generated at each node accordng to osson process wth the mean nterarrval rate of g. The destnaton node s determned by usng a unform random number generator. For each smulaton experment, statstcs were gathered for a total number of 00,000 messages. Statstc gatherng was dscarded for the frst 0,000 messages to avod dstortons due to the warm-up phase. Also, there s a dran phase at the end of smulaton n whch 0,000 generated messages were not n the statstc gatherng to provde enough tme for all packets to reach ther destnaton. Extensve valdaton experments have been performed for several combnatons of clusters szes, network szes, network technologes, and message length. The general conclusons have been found to be consstent across all the cases consdered. After all, to llustrate the result of some specfc cases to show the valdty of our model, the tems whch were examned carefully are presented n Table. Moreover, the two dfferent message lengths, M3 and 64 flts wth dfferent szes, L m 56 and 5 bytes are used. The network bandwdth s 500/tme unt and network latency and swtch latency are 0.0 and 0.0 tme unt, respectvely. Table. System organzatons for valdaton N C m Node Organzatons n n n 3 [0,] [,7] [8,3] n 3 n 4 n 5 [0,7] [8,0] [,5] The results of smulaton and analyss for a system wth above mentoned parameters are depcted n Fg. 3 and Fg. 4 n whch the mean message latences are plotted aganst the offered traffc for two dfferent system organzatons. The fgures reveal that the analytcal model predcts the mean message latency wth a good degree of accuracy when the system s n the steady state regon, that s, when t has not reached the saturaton pont. However, there are dscrepances n the results provded by the model and the smulaton when the system s under heavy traffc and approaches the saturaton pont. Ths s due to the approxmatons that have been made n the analyss to ease the model development. For nstance, n ths regon the traffc on the lnks s not completely ndependent, as we assume n our analytcal model. Also, one of the most sgnfcant term n the model under heavly loaded system, s the average watng tme at the source queue and concentrators/dspatchers. The approxmaton whch s made to compute the varance of the servce tme receved by a message at a gven channel (Eq.( s a factor of the model naccuracy. Snce, the most evaluaton studes focus on network performance n the steady state regons, so we can conclude that the proposed model can be a practcal

7 evaluaton tool that can help system desgner to explore the desgn space and examne varous desgn parameters. N0, m8, M3 N0, m8, M64 Mean Message Latency A nalyss (Lm56 Smulato n A nalyss (Lm5 Smulato n Mean Message Latency Analyss (Lm56 Smulaton Analyss (Lm5 Smulaton Offered Traffc Offered Traffc Fg. 3. Mean message latency n a system wth N0, M3 and 64 N544, m4, M3 N544, m4, M64 Mean Message Latency Analyss (Lm56 Smulaton Analyss (Lm5 Smulaton Mean Message Latency Analyss (Lm56 Smulaton Analyss (Lm5 Smulaton Offered Traffc Offered Traffc Fg. 4. Mean message latency n a system wth N544, M3 and Conclusons Analytcal models play a crucal role n evaluaton of a system under varous desgn ssues. In ths paper, an analytcal model of nterconnecton networks for mult-cluster computng systems n the presence of cluster szes heterogenety s dscussed. The proposed model has been valdated wth versatle confguratons and desgn parameters. Smulaton experments have proved that the model predcts message latency wth a reasonable accuracy. For future work, we ntent to develop and extend the model to cover other categores of heterogenety and non-unform traffc pattern as well. Acknowledgments Specal thanks to Mr. D. Sgro and R. Ruge regardng the system setup for the smulatons. The help of Dr. A. Khonsar from Tehran Unversty for hs worthwhle comments s apprecated. Fnancal support from Intellgent System Research Laboratory s also greatly apprecated. 6. References [] M.Q. Xu, Effectve Meta-Computng usng LSF Mult-Cluster, In roceedngs of the IEEE Internatonal Conference on Cluster and Grd, Brsbane, Australa, May 00, pp

8 [] I. Foster, The Grd: A New Infrastructure for st Century Scence, hyscs Today, Vol.55, No., Feb. 00, pp [3] J. H. Abawajy, and S.. Dandamud. arallel Job Schedulng on Mult-Cluster Computng Systems, In roceedngs of the IEEE Internatonal Conference on Cluster Computng, Hong Kong, Dec. 003, pp [4] DAS- 00, The DAS- Supercomputer. [5] B. Boas, Storage on the Lunatc Frnge. Lawrence Lvermore Natonal Laboratory, anel at Supercomputng Conference 003, hoenx, AZ, Nov [6] H. Sarbaz-Azad, A. Khonsar, and M. Ould-Khaoua, erformance Analyss of Determnstc Routng n Wormhole k-ary n-cubes wth Vrtual Channels, Journal of Interconnecton Networks, Vol. 3, Nos.&, 00, pp [7] M. Ould-Khaoua A erformance Model for Duato's Fully-adaptve Routng Algorthm n k-ary n-cubes, IEEE Transacton on Computers, Vol.4, No., 999, pp. -8. [8] J.T. Draper and J. Ghosh, A Comprehensve Analytcal Model for Wormhole Routng n Multcomputer Systems, Journal of arallel and Dstrbuted Computng, Vol. 3, No., 994, pp [9] Y.M. Boura and C.R. Das, erformance Analyss of Bufferng Schemes n Wormhole Routers, IEEE Transactons on Computers, Vol. 46, No. 6, Jun. 997, pp [0].C. Hu and L. Klenrock, A Queung Model for Wormhole Routng wth Tmeout, In roceedngs of the 4 th Internatonal Conference on Computer Communcatons and Networks, Nevada, LV, Sep. 995, pp [] X. Du, X. Zhang, and Z. Zhu, Memory Herarchy Consderaton for Cost-Effectve Cluster Computng, IEEE Transacton on Computers, Vol.49, No.5, Sep. 000, pp [] B. Javad, S. Khorsand, and M. K. Akbar, Study of Cluster-based arallel Systems usng Analytcal Modelng and Smulaton, Lecture Notes n Computer Scence, Vol. 9, Sprnger-Verlag, 005, pp [3] A. Clemats and A. Corana, Modelng erformance of Heterogeneous arallel Computng Systems, Journal of arallel Computng, Vol.5, No.9, Sep. 999, pp [4] A.T.T. Chun and C.L. Wang. Contenton-free Complete Exchange Algorthm on Clusters, In roceedngs of the IEEE Internatonal Conference on Cluster Computng, Saxony, Germany, Nov. 000, pp [5] X. Ln, An Effcent Communcaton Scheme for Fat- Tree Topology on Infnband Networks, M.Sc Thess, Department of Informaton Engneerng and Computer Scence, Feng Cha Unversty, Tawan [6] M. Kobuch, K. Watanae, K. Kono, A. Jouraku, and H. Amano. erformance Evaluaton of Routng Algorthm n RHNET- Cluster, In roceedngs of the IEEE Internatonal Conference on Cluster Computng, Hong Kong, Dec. 003, pp [7] M. D. Schroeder et. al. Autonet: A Hgh-Speed, Self Confgurng Local Area Network Usng ont-to- ont Lnks. SRC research report 59, Dgtal Equpment Corporaton, Apr [8] B. Javad, J. H. Abawajy, and M. K. Akbar, Modelng and Analyss of Heterogeneous Loosely- Coupled Dstrbuted Systems, Techncal Report TR C06/, School of Informaton Technology, Deakn Unversty, Australa, Jan 006. [9] L. Klenrock, Queung System: Computer Applcatons, Vol., John Wly ublsher, New York [0] W. Dally and B. Towles, rncples and ractces of Interconnecton Networks, Morgan Kaufmann ublsher, San Francsco, 004. [] InfnBand Clusterng, Delverng Better rce/erformance than Ethernet, Whte aper, Mellanox Technologes Inc., Santa Clara, CA, 005. [] Buldng Scalable, Hgh erformance Cluster/Grd Networks: The Role of Ethernet, Whte aper, Force0 Networks Inc., Mlptas, CA, 004. [3] J. Dongarra and A. Lastovetsky, An Overvew of Heterogeneous Hgh erformance and Grd Computng, In Engneerng the Grd: Status and erspectve, Eds B. DMartno, J. Dongarra, A. Hose, L. Yang, and H. Zma, Amercan Scentfc ublshers, Feb [4] B. Javad, M. K. Akbar, and J. H. Abawajy, Analyss of Mult-Cluster Computng Systems wth rocessor Heterogenety, In roceedngs of the th Internatonal Computer Socety of Iran Computer Conference, Tehran, Iran, Jan. 006, pp [5] B. Javad, J. H. Abawajy, and M. K. Akabr, Analytcal Interconnecton Networks Model for Mult-Cluster Computng Systems, In 3th Internatonal Conference on Analytcal and Stochastc Modelng Technques and Applcatons, Bonn, Germany, to be appeared.