O Noblockig Folded-Clos Networks i Computer Commuicatio Eviromets Xi Yua Departmet of Computer Sciece, Florida State Uiversity, Tallahassee, FL 3306 xyua@cs.fsu.edu Abstract Folded-Clos etworks, also referred to as fat-trees, have bee widely used as itercoects i large scale high performace computig clusters. The switchig capability of such itercoects i computer commuicatio eviromets, however, is ot well uderstood. I particular, the cocept of oblockig itercoects, which is ofte used by system vedors, has oly bee studied i the telephoe commuicatio eviromet with the assumptio of a cetralized cotroller. Such oblockig etworks do ot support oblockig commuicatios i computer commuicatio eviromets where the etwork cotrol is distributed. This paper theoretically aalyzes the coditios for folded-clos etworks to achieve oblockig commuicatios i computer commuicatio eviromets with various routig schemes icludig determiistic routig ad adaptive routig, ad establishes oblockig coditios. iput stage 0 r middle stage 0 m (a) Clos(, m, r) Fig.. output stage 0 r Clos ad folded-clos etworks 0 m 0 r (b) ftree(+m, r) I. INTRODUCTION Clos etworks ad their variatios such as folded-clos (also referred to as fat-trees) have bee widely used for multiprocessor itercoects ad system area etworks. Almost all large scale commodity high performace computig clusters are itercoected with such topologies. A three-stage Clos etwork has the iput stage, the middle stage, ad the output stage. The iput stage cosists of m switches; the middle stage cosists of r r switches; ad the output stage cosists of m switches. There are r iput switches, r output switches, ad m middle switches with each of the iput ad output switches havig a lik coectig to each of the middle switches. Fig. (a) depicts a threestage Clos etwork. We will use the otio Clos(, m, r) to deote the Clos etwork with parameters, m, ad r. Notice that the liks i Clos(, m, r) are ui-directioal. A folded-clos (fat-tree) etwork is the oe-sided versio of the Clos etwork: it basically merges the correspodig iput ad output switches ito oe switch. Fig. (b) shows a folded-clos (fat-tree) etwork. The liks i a folded-clos etwork are bidirectioal. Clos(, m, r) correspods to a two level fat-tree: the lower level switches are + m + m switches while the top level switches are r r switch. We will use the otio ftree( + m, r) to deote such a fat-tree. Clos(, m, r) is logically equivalet to ftree( + m, r). The switchig capability of f tree( + m, r) (or Clos(, m, r)) is determied by the parameters, m, ad r. The study of such etworks has focused o fidig the most cost effective values for, m, ad r to achieve oblockig commuicatio, that is, the ability to establish a coectio from a arbitrary iput port to a arbitrary output port without causig cotetio. A oblockig etwork ca provide coectivity for ay permutatio commuicatio, which cosists coectios from arbitrary iput ports to arbitrary output ports with the restrictio that each iput or output port ca be used at most oce i the commuicatio. The most importat results i this area are summarized i various oblockig coditios for Clos etworks icludig strictly oblockig [], wide-sese oblockig [4], [6], ad rearrageably oblockig [3]. Although these oblockig coditios results are sigificat, all of them were obtaied with the assumptio that a cetralized cotroller is used to maage all etwork resources, which reders the results ot applicable to computer commuicatios where the etwork cotrol is distributed. Whe used i computig clusters, folded-clos based itercoects are ofte treated as the replacemet of cetral crossbar switches that support ay permutatio commuicatio with full bisectio badwidth [7]. Although may folded-clos based itercoects for clusters are oblockig (strictly, wide sese, or rearrageably) i theory, the delivered performace is far-from that of crossbar switches [5], [7]. Without the cetralized cotroller that takes all coectio requests ad maages all etwork resources, eve a strictly oblockig etwork ca still block a permutatio commuicatio. We defie a oblockig folded-clos etwork i the computer commuicatio eviromet to be oe that, with distributed cotrol, ca support ay permutatio commuicatio without etwork cotetio. Usig this defiitio, if a folded- Clos based itercoect is oblockig, it ca support ay
permutatio commuicatio with o cotetio ad ca thus achieve full bisectio badwidth for ay permutatio commuicatio: such a itercoect behaves like a crossbar switch. The uderstadig of folded-clos based itercoects i computer commuicatio eviromets is isufficiet: techiques for buildig truly oblockig folded-clos itercoects i computer commuicatio eviromets have ot bee developed; it is uclear what the cost of such itercoects will be. Through theoretical aalysis, this paper gives aswers to these importat questios. We ivestigate folded-clos etworks that are oblockig i computer commuicatio eviromets, develop techiques for buildig such etworks, ad establish oblockig coditios for folded-clos etworks with distributed cotrol uder various routig schemes icludig determiistic routig ad adaptive routig. The major results iclude the followig. With sigle-path determiistic routig, it is ot cost effective to build oblockig f tree( + m, r) with r +. Usig relatively small top level switches to build oblockig ftree( + m, r) is ot effective. With sigle-path determiistic routig, whe r +, the oblockig coditio for ftree( + m, r) is m. Usig sigle-path determiistic routig, O(N N)- port oblockig itercoects ca be costructed usig O(N) N-port switches. Commoly used multi-path determiistic routig schemes have the same oblockig coditio as sigle-path routig. With local adaptive routig where routes may adapt based o the iformatio local to each switch, whe r c, where c is a costat, there exists a fuctio f() = O( (c+) ) such that the oblockig coditio for ftree( + m, r) is m f(). Local adaptive routig allows usig a smaller umber of top level switches to achieve oblockig commuicatio i compariso to determiistic routig. These results advace the basic uderstadig of folded- Clos etworks i computer commuicatio eviromets. They ca be used i feasibility aalysis for buildig oblockig folded-clos based itercoects uder other costraits, ad directly applied to build such itercoects. The rest of the paper is structured as follows. Sectio II discusses the related work. Sectio III describes the backgroud ad otatios used i the paper. Sectio IV cosiders determiistic routig, derives the oblockig coditio, ad presets the routig scheme for oblockig ftree( + m, r) with sigle-path determiistic routig. Sectio V cosiders adaptive routig ad shows that local adaptive routig where routes may adapt based o the iformatio local to each switch improves the oblockig coditio over determiistic routig. Fially, Sectio VI cocludes the paper. II. RELATED WORK The switchig capability of Clos etworks i the cotext of switchig telephoe traffics has bee extesive studied. Various oblockig coditios have bee established. A Clos etwork is strictly oblockig if it is always possible to set up a coectig path from a idle iput port to a idle output port idepedet of the existig coectios ad the path search algorithm. Clos showed [] that Clos(, m, r) is strictly oblockig if the umber of middle stage switches m. A etwork is wide-sese oblockig whe it is always possible to set up a path from a idle iput port to a idle output by suitably choosig routes for ew coectios. The coditios for a etwork to be wide-sese oblockig deped o the routig algorithm; some results are obtaied by Bees [4] ad Yag [6]. A etwork is rearrageably oblockig if a ew coectio from a idle iput port to a idle output port ca always be established by rearragig the paths for existig coectios. Bees [3] showed that a Clos etwork is rearrageably oblockig if m. All these results assume a cetralized cotroller ad caot be directly applied to computer etworks with distributed cotrol. After Leiserso itroduced the fat-tree topology to computer etworks [0], [], the topology has become popular ad extesive research has bee performed o this topology. The topology has bee greatly exteded [3], [4]. Sice traditioal oblockig etworks are blockig i the computer commuicatio eviromet, most studies have focused o uderstadig the performace of such etworks by aalyzig the blockig probability [6], [9], [5], or developig techiques to reduce the blockig probability [], [6], [9], [5], [7]. Various routig techiques icludig radomized routig [6], [5], multi-path routig [], [7], ad adaptive routig [9], have bee proposed to miimize the blockig probability. Eve with all these improvemets, recet studies still show that the cotemporary fat-tree based itercoects offer much lower performace tha crossbar switches [5], [7]. This paper ivestigates the coditios for folded-clos etworks to be oblockig i computer commuicatio eviromets, which differs from all existig work i the area. III. BACKGROUND AND NOTATIONS This paper focuses o ftree( + m, r) s that are oblockig i computer commuicatio eviromets. A example ftree( + m, r) is show i Fig. (b). There are two layers of switches ad oe layer of leaf odes i the topology. The leaf odes are commuicatio sources ad destiatios. f tree(+m, r) has r bottom level +m-port switches ad m top level r-port switches. It supports r leaf odes. The liks i this topology are bidirectioal. The liks from leaf odes to bottom level switches ad from bottom level switches to top level switches are upliks. The liks from top level switches to bottom level switches ad from bottom level switches to leaf odes are dowliks. As show i Fig. (b), we umber the m top level switches from 0 to m, the r bottom level switches from 0 to r, ad the r leaf odes from 0 to r. Other umberig schemes will also be used i the paper. They will be itroduced before they are used. Let us deote (s, d) a source-destiatio (SD) pair with source ode s to destiatio ode d. SRC(s, d) is the switch that s is i ad DST (s, d) is the switch d is i. We say that SRC(s, d) is the source switch of (s, d) ad that (s, d)
starts from SRC(s, d). Similarly, DST (s, d) is the destiatio switch of (s, d) ad (s, d) eds at DST (s, d). We will use the phrase SD pairs from the same switch to deote SD pairs whose sources are i the same switch, SD pairs from differet switches to deote SD pairs whose sources are i differet switches, SD pairs to the same switch to deote SD pairs whose destiatios are i the same switch, ad SD pairs to differet switches to deote SD pairs whose destiatios are i differet switches. A commuicatio patter ca be represeted by a set of SD pairs. Defiitio : A permutatio commuicatio, or permutatio, is a commuicatio patter where each leaf ode ca be the source i at most oe SD pair ad the destiatio i at most oe SD pair i the commuicatio patter. Property : Let (s, d ) ad (s, d ) be two SD pairs i a permutatio. s s ad d d. This property ca be obtaied from the defiitio of permutatio. Sice ftree( + m, r) have r leaf odes, a permutatio commuicatio ca at most have r SD pairs. Whe all source odes ad all destiatio odes are used i a permutatio, there are exactly r SD pairs. A permutatio, however, does ot require all leaf odes to be used. To support the commuicatio for a SD pair, a path must be used to carry the traffics for the commuicatio. I computer commuicatio eviromets, distributed cotrol is performed by the routig algorithm, which determies the path for each packet. We cosider several widely used routig algorithms for folded-clos etworks: sigle-path determiistic routig, multipath determiistic routig, ad adaptive routig. I sigle-path determiistic routig, oe path is used to carry all traffics for each SD pair ad the path for each SD pair is pre-determied. I (traffic oblivious) multi-path determiistic routig, the traffics for the same SD pair are distributed amog multiple predetermied paths either i a determiistic or radom maer. For both sigle path determiistic routig ad multi-path determiistic routig, the paths used are idepedet of the traffic patter. I adaptive routig, differet paths ca be used for oe SD pair ad the path to be used is determied dyamically based o the traffic coditio. Some adaptive routig algorithms require routes to adapt based o the whole commuicatio patter. We call such algorithms global adaptive routig algorithms sice the whole commuicatio patter must be cosidered i order to determie a path. Global adaptive routig is equivalet to routig with a cetralized cotroller (so that the whole commuicatio patter is kow to the routig algorithm); ad the oblockig coditios for such routig schemes o fat-trees have bee established. I this paper, we focus o local adaptive routig where routes adapt oly based o the iformatio that is available locally to each switch. With a give routig algorithm, whe packets i two SD pairs i a commuicatio patter are routed through oe etwork lik, we say that the commuicatio patter causes etwork cotetio. Defiitio : A folded-clos etwork is oblockig with a routig algorithm if ay permutatio commuicatio ca be supported without etwork cotetio usig the routig algorithm o the etwork. IV. DETERMINISTIC ROUTING This sectio cosiders oblockig folded-clos etworks with determiistic routig. We will first cosider sigle-path determiistic routig ad the discuss multi-path routig. A. Sigle-path determiistic routig I sigle-path determiistic routig, a path is determiistically assiged to each SD pair. The followig lemma gives the coditio for a folded-clos etwork to be oblockig with sigle-path determiistic routig. Lemma : For ay sigle-path determiistic routig, f tree(+m, r) is oblockig if ad oly if each lik carries traffics either from oe source or to oe destiatio. Proof: We will prove the ecessary coditio by cotradictio. Let a lik L i ftree(+m, r) carries traffics from more tha oe source ad to more tha oe destiatio. There exists at least two SD pairs, (s, d ) ad (s, d ), s s ad d d, whose traffics are routed through L. By Defiitio, the commuicatio patter that cotais oly these two SD pairs ({(s, d ), (s, d )}) is a permutatio. Sice the routig is determiistic, there is cotetio o lik L for this permutatio ad thus, the etwork is ot oblockig. Hece, if ftree( + m, r) is oblockig, each lik i the etwork carries traffics either from oe source or oe destiatio. We will ow prove the sufficiet coditio by cotradictio. Assume that we have a permutatio P that ca cause etwork cotetio: there exists two SD pairs i P that are routed through oe lik. Let the two SD pairs be (s, d ) ad (s, d ) ad the lik be L. Sice P is a permutatio, we have s s ad d d (Property ). This cotradicts to the assumptio that lik L oly carries traffics either from oe source or to oe destiatio. Hece, if each lik i the etwork carries traffics either from oe source or to oe destiatio, ftree( + m, r) is oblockig. I ftree( + m, r), each lik betwee a leaf ode ad a bottom level switch oly coects to oe leaf ode: the traffics o such a lik is either to that leaf ode or from the leaf ode regardless of the routig algorithm. Such a lik does ot have cotetio for ay permutatio. The liks betwee top level switches ad bottom level switches may have cotetio. Sice a oblockig etwork must support ay permutatio, all possible SD pairs must be assiged a path by the routig algorithm. For SD pair (s, d), where s ad d are ot i the same bottom level switch, it must be routed through a top level switch ad use the liks betwee top level switches ad bottom level switches. There are r(r ) such SD pairs i ftree( + m, r) that must be routed carefully i order to achieve oblockig commuicatio. I derivig the oblockig coditio for sigle-path determiistic routig, we must determie the smallest m such that all of the r(r ) SD pairs ca be routed with each lik supportig SD pairs with either the same source or the same destiatio. We will use a subgraph of ftree(+m, r) to
aalyze the umber of SD pairs that ca be routed through oe top level switch. Fig. shows the subgraph, which cotais all lower level switches i ftree( + m, r), but oly oe top level switch. The subgraph is effectively ftree( +, r), a regular tree topology with the root havig r childre ad each bottom level switch havig leaf odes. Fig.. # # 0 r 0 (r ) r The subgraph of ftree( + m, r) (ftree( +, r)) Lemma : Cosider usig the ftree( +, r) topology to route a subset of all possible SD pairs with source ad destiatio i differet switches. If each lik ca carry traffics either from oe source or to oe destiatio, the the largest umber of SD pairs that ca be routed through the root is at most r (r ) whe r +, ad r whe r +. Proof: Let S be a largest set of SD pairs that are routed through the root switch whe all liks carry traffics either to oe destiatio or from oe source. To cout the umber of SD pairs i S, we partitio the SD pairs i S ito three types: () the SD pairs whose source switch has or more sources i the SD pairs i S, () the SD pairs whose destiatio switch has or more destiatios i the SD pairs i S, ad (3) the SD pairs whose source switch has oe source ad whose destiatio switch has oe destiatio i the SD pairs i S. Let us deote the umber of type () SD pairs i S be NUM, the umber of type () SD pairs be NUM, the umber of type (3) SD pairs be NUM 3, ad the total umber of SD pairs i S be NUM. Let the umber of switches that have or more sources i the SD pairs i S be A; the umber of switches that have or more destiatios be B. Cosider the umber of type () SD pairs. Sice there are two or more sources i each of such switches, all these sources i oe switch must commuicate with oe destiatio. Otherwise, the lik from the switch to the root will carries SD pairs from more tha oe source ad to more tha oe destiatio. Hece, oe switch ca cotribute at most such SD pairs to NUM (whe all of the leaf odes i the switch are sources). Sice there are A such switches, the umber of such SD pairs, NUM, is at most A. Similarly, NUM A. NUM B. Now, cosider the umber of type (3) SD pairs, NUM 3. Sice there are A switches that caot be source switches for type (3) SD pairs, at most r A switches ca be source switches for such SD pairs. Sice each switch ca at most have oe destiatio i type (3) SD pairs, each source ca at most commuicate to r destiatios (oe destiatio i each of the r switches other tha the source switch). Hece, NUM 3 (r A) (r ). Usig a similar logic, by excludig switches with or more destiatios, we have NUM 3 (r B) (r ). Combie these two iequatios, we obtai NUM 3 r (r ) ( A + B )(r ). Therefore, the total umber of SD pairs, NUM NUM + NUM + NUM 3 A + B + r (r ) A+B (r ) = r (r ) + A+B ( + r) Whe r +, A+B ( + r) 0 ad NUM r (r ). Whe r +, A+B ( + r) 0. Sice A r ad B r, NUM r (r ) + A+B ( + r) r (r ) + r+r ( + r) = r. Theorem : whe r +, the umber of ports supported by a oblockig ftree( + m, r) with ay sigle-path determiistic routig is o more tha ( + m). Proof: Regardless of the routig algorithm used, a total of r(r ) SD pairs must be routed through top level switches i ftree( + m, r). From Lemma ad Lemma, whe r +, each top level switch ca route at most r SD pairs i a oblockig ftree( + m, r) for ay siglepath determiistic routig scheme. Hece, there are at least r(r ) r = (r ) top level switches eeded; ad m (r ). The umber of ports supported by ftree( + m, r) is r ( r + ) (m + ). Theorem idicates that it is ot effective to build oblockig folded-clos etworks usig relatively small top level switches. Whe r +, the total umber of ports supported by a oblockig folded-clos etwork is at most twice that i its bottom level switches. Hece, oe should focus o oblockig folded-clos etworks with relatively large top level switches (r + ). Theorem : Let ftree( + m, r) be oblockig with ay sigle-path determiistic routig. Whe r +, m. Proof: Similar to the proof of Theorem, regardless of the routig algorithm used, r(r ) SD pairs must be routed through top level switches. From Lemma ad Lemma, whe r +, each top level switch ca route at most r(r ) SD pairs i a oblockig ftree( + m, r) for ay sigle-path determiistic routig scheme. Hece, there are at
least r(r ) r(r ) = top level switches eeded; ad m. Theorem gives the lower boud of the umber of top level switches eeded to make f tree( + m, r) oblockig with ay sigle-path determiistic routig. The followig theorem establishes that this lower boud ca be achieved: the m oblockig coditio is tight. Theorem 3: There exists a sigle-path determiistic routig algorithm for ftree( +, r) that supports all permutatios without etwork cotetio. I other words, ftree( +, r) is oblockig usig that routig algorithm. Proof: We will first describe the routig algorithm ad the prove ftree( +, r) is oblockig with the routig algorithm. I ftree(+, r), there are top level switches. We will umber of top level switches by (i, j), 0 i ad 0 j. There are r bottom level switches umbered from 0 to r. Each bottom level switch v, 0 v r, coects to leaf odes umbered as (v, k), 0 k. The routig algorithm routes SD pair (s = (v, i), d = (w, j)), 0 v w r ad 0 i, j, through top level switch (i, j). That is, SD pair (s = (v, i), d = (w, j)) is routed through path (v, i) v (i, j) w (w, j). Note that whe v = w, (s = (v, i), d = (v, j)) is routed through path (v, i) v (v, j). Usig this algorithm, each uplik i ftree(+, r) carries traffics from oe source. This obviously holds for the upliks from leaf odes to bottom level switches. Cosider the uplik from a arbitrary bottom level switch v to a arbitrary top level switch (i, j). There are r SD pairs o this lik: (s = (v, i), d = (0, j)), (s = (v, i), d = (, j)),..., (s = (v, i), d = (v, j)), (s = (v, i), d = (v +, j)),..., (s = (v, i), d = (r, j)). There is oly oe source (v, i) for all the SD pairs. Similarly, each dowlik i ftree( +, r) carries traffics to oe destiatio. Cosider the dowlik from a arbitrary top level switch (i, j) to a arbitrary bottom level switch v. There are r SD pairs o this lik: (s = (0, i), d = (v, j)), (s = (, i), d = (v, j)),..., (s = (v, i), d = (v, j)), (s = (v +, i), d = (v, j)),..., (s = (r, i), d = (v, j)). There is oly oe destiatio (v, j) i all the SD pairs. Fig. 3 shows the SD pairs routed through the liks betwee top level switch (i, j) ad bottom level switch v. Hece, the algorithm routes SD pairs such that each lik i ftree( +, r) carries traffics either from at most oe source or to at most oe destiatio. It follows from Lemma that the etwork is oblockig. Discussio The mai applicatio of Clos etworks is to build large (oblockig) itercoects from smaller switches. Here, we compare oblockig folded-clos etworks i computer commuicatio eviromets with traditio rearrageably oblockig etworks i their capability for buildig larger itercoects. Cosider the case whe the same sized switches are used to build the oblockig etworks. That is, r = m +. We will use the m-port -trees (F T (m, )) [] as a represetative (i, j) (0, 0) (, ) 0 v v v+ r (0, j) (v, j) (v, i) (v+, j) (r, j) (a) SD pairs i the uplik from switch v to switch (i, j) (i, j) (0, 0) (, ) 0 v v v+ r (0, i) (v, i) (v, j) (v+, i) (r, i) (b) SD pairs i the dowlik from switch (i, j) to switch v Fig. 3. SD pairs routed through liks betwee top level switch (i, j) ad bottom level switch v rearrageably oblockig folded-clos i the compariso. Our two level oblockig folded-clos etwork, ftree(+, + ), uses + + -port switches to support 3 + oblockig ports. Let N = +, our oblockig etwork uses roughly N N-port switches to support roughly N 3 oblockig ports. Traditioal F T (N, ) uses 3N N-port switches to support N ports []. Table I compares that umber of ports ad the umber of switches eeded for the two types of folded-clos etworks usig practical buildig blocks: 0-port, 30-port, ad 4- port switches. Our oblockig etworks behave like crossbar switches while F T (m, ) is ot oblockig i computer commuicatio eviromets. It is thus expected that our oblockig etworks are more expesive to costruct. Our techique allows larger oblockig folded-clos etworks to be costructed from smaller switches as show i Table I, ad is optimal for buildig such etworks with sigle-path determiistic routig. To support larger umbers of ports, the method to build -level oblockig folded-clos etworks ca be recursively applied to build more levels of oblockig folded-clos etworks. For example, to obtai a 3-level oblockig etwork, a -level oblockig etwork ca be used to replace each of either the top level switches or the bottom level switches i -level etworks. Sice our oblockig f tree( + m, r) supports all permutatios with o cotetio, it ca be show by iductio that the recursively-built larger etwork will also support all permutatios with o cotetio ad is thus buildig ftree( +, + ) F T ( +, ) block # of # of # of # of size switches ports switches ports 0-port 36 80 30 00 (4 + 4 ) 30-port 55 50 45 450 (5 + 5 ) 4-port 88 5 63 884 (6 + 6 ) TABLE I SIZE OF NONBLOCKING ftree( +, + ) AND F T ( +, )
oblockig. Oe questio is whether it is more effective to replace bottom level switches or top level switches with a fat-tree. Theorem gives the aswer to this questio: it is more beeficial to have large top level switches. Hece, whe buildig more tha two levels folded-clos etworks, oe should replace top level switches with oblockig etworks. Usig this approach, a three-level oblockig folded- Clos etwork built with + -port switches will resemble ftree( +, 3 + ), with each top level 3 + -port switch beig realized with a ftree( +, + ). This oblockig etwork has 4 +3 3 + +-port switches ad supports 4 + 3 ports. Let N = +. The threelevel oblockig folded-clos etwork uses O(N ) O(N)- port switches to obtai a O(N )-port oblockig etwork. As a compariso, F T (N, 3) uses O(N ) O(N)-port switches to obtai O(N 3 )-port fat-trees. B. Traffic oblivious multi-path determiistic routig Besides sigle-path determiistic routig, other routig schemes such as multi-path routig determiistic routig [], [7] have also bee developed for folded-clos etworks. IfiiBad allows multiple paths to be set-up betwee two edpoits [8]. I traffic oblivious multi-path determiistic routig, the packets for oe SD pair are distributed amog multiple paths either i a determiistic or radom maer, ad the routes are decided idepedet of the traffic patter. Siglepath determiistic routig is a form of multi-path determiistic routig. Although traffic oblivious multi-path routig schemes ca achieve better load balace i folded-clos etworks tha sigle-path routig [7], splittig packets from oe SD pairs to multiple paths i a determiistic or radom maer does ot improve the oblockig coditio: uder such coditios, the timig for a path to be used to route a packet i a SD pair is upredictable (depedig o the packet arrival timig ad the routig scheme); to achieve oblockig commuicatio for ay permutatio, Lemma still eeds to hold, which leads to the same boud i Theorem (m ) for traffic oblivious multi-path determiistic routig schemes. As will be show i the ext sectio, whe the routes for a SD pair ca be adapted based o the traffic patter, the oblockig coditio ca be improved. V. ADAPTIVE ROUTING As discussed earlier, global adaptive routig where routes may adapt based o the whole commuicatio patter is equivalet to routig with a cetralized cotroller; ad the oblockig coditios for such schemes have bee established. I this work, we cosider local adaptive routig where routes may adapt based o the iformatio that is available locally to each switch. By developig a local adaptive routig algorithm that uses less tha upper level switches (m < ) i ftree(+m, r) to achieve oblockig commuicatio, we formally show that local adaptive routig improves the oblockig coditio over determiistic routig, which idicates that more cost effective oblockig ftree( + m, r) ca be built with local adaptive routig. For f tree(+m, r), routig adaptivity ca oly be achieved i iput switches. Oce a packet reaches the top level switch, there is oly oe path to each destiatio ad o adaptivity is possible. Hece, we will focus o local adaptive routig algorithms where the routes adapt oly based o the local traffic patter i each iput switch. Our proposed local adaptive routig algorithm has the followig properties. For a give commuicatio patter, the algorithm assigs oe path to carry all traffics for oe SD pair. Routig adaptivity is reflected i the fact that the paths for the same SD pair i differet patters may be differet. For a give commuicatio patter, the path for a SD pair is determied based o the SD pairs i the commuicatio patter whose sources are i the same switch (adapt based o local iformatio). SD pairs from differet switches are routed idepedetly. By routig SD pairs from differet switches idepedetly, local adaptive routig algorithms with the above two properties, icludig our proposed routig algorithm, ca be realized i a distributed maer by implemetig the routig logic i each of the iput switches, which have the iformatio of all SD pairs from the switches. I the distributed implemetatio, the algorithm does ot require global iformatio to be shared amog differet switches. Each iput switch adapts based o its local traffic patter. Routig ad traffic patters i other switches do ot affect the routig decisio. I the rest of the paper, the term local adaptive routig algorithm will refer to local adaptive routig algorithms with these two properties. The followig lemma defies a class of local adaptive routig algorithms that are the buildig blocks of our proposed scheme. Lemma 3: Let (s, d ) ad (s, d ) be two arbitrary SD pairs i a commuicatio patter where d d are i the same switch. If a local adaptive routig algorithm for ftree( + m, r) guaratees to route such two SD pairs through differet top level switches, the routes for ay two SD pairs i ay permutatio whose sources are i differet iput switches will ot have etwork cotetio usig the local adaptive routig algorithm. Proof: Let us deote SRC(s, d) the source switch of SD pair (s, d) ad DST (s, d) the destiatio switch of SD pair (s, d). Let (s, d ) ad (s, d ) to be arbitrary two SD pairs i a permutatio where s ad s are i differet switches (SRC(s, d ) SRC(s, d )). We will prove that uder the assumptio i the lemma, these two SD pairs will ot have cotetio. Let (s, d ) be routed through top level switch A ad (s, d ) be routed through top level switch B (A ad B may be the same). Sice SRC(s, d ) SRC(s, d ), upliks SRC(s, d ) A ad SRC(s, d ) B are differet regardless whether A = B or ot, ad there is o cotetio i the upliks for the two SD pairs. For the dowlik A DST (s, d ) ad B DST (s, d ), there are two cases. Whe DST (s, d ) = DST (s, d ), sice (s, d ) ad (s, d ) are from oe permutatio commuicatio, d d. By the assumptio of this lemma, we have A B ad thus, the dowliks for the two SD pairs are differet
ad there is o etwork cotetio. Whe DST (s, d ) DST (s, d ), regardless whether A = B or ot, the dowliks A DST (s, d ) ad B DST (s, d ) are differet ad there is o cotetio. Lemma 3 states that for a class of local adaptive routig algorithms that guaratee to use differet top level switches to route SD pairs with differet destiatios i the same switch, SD pairs from differet switches i a permutatio will ot have cotetio. We will use the term Class DIFF to deote this class of local adaptive routig algorithms. Lemma 3 implies the followig. ) Usig a Class DIFF algorithm, SD pairs from differet switches ca be routed idepedetly: paths for SD pairs from differet switches will ot have cotetio. ) A Class DIFF algorithm that also avoids cotetio for SD pairs from the same switch will achieve oblockig commuicatio. Hece, to desig a routig algorithm that achieves oblockig commuicatio usig Class DIFF algorithms as buildig blocks, we ca focus o cotetio free routig for SD pairs from the same switch. For ay ftree( + m, r), there exists a costat c such that r c. For practical folded-clos etworks, c is a small costat. For example, i ftree( + m, ), c =. I ftree(+m, +), c = 3. I our adaptive routig algorithm, we umber the r bottom level switches with c -based digits: s c s c...s 0, 0 s i for all 0 i c ; ad we umber the r leaf odes (sources ad destiatios) with c + -based digits: s c s c...s 0 p, where s c s c...s 0 is the switch that the leaf ode is i, ad 0 p is the local ode umber withi switch s c s c...s 0. For a ode s c s c...s 0 p, we will say that p is the first digit of the ode umber, s 0 is the secod digit,..., s c is the c + -th digit of the ode umber. Our adaptive routig algorithm for ftree( + m, r) uses Class DIFF schemes ad schedules SD pairs from each switch idepedetly. SD pairs from each switch are routed i phases. I each phase, SD pairs are routed over (c + ) top level switches. We will use the term cofiguratio to deote the group of (c + ) top level switches used i oe schedulig phase. The same routig logic is applied for all cofiguratios util all SD pairs i a patter are routed. Withi each cofiguratio, we further group the (c + ) top level switches ito c + sets of switches. We will call each set of top level switches, a partitio. I each of the partitios, the top level switches are umbered from 0 to. Let s c s c...s 0 p be a geeric destiatio ode umber. I our adaptive routig algorithm, the k-th partitio ( k c + ) i the c + partitios i a cofiguratio, is used to routes traffics to destiatios with differet k- th digits. Specifically, i the first partitio, top level switch i, 0 i is oly used to carry SD pairs with destiatios whose local ode umber p = i for all bottom level switches: oly SD pairs whose destiatios have differet local ode umbers are routed through this partitio. Similarly, the secod partitio is used to route SD pairs to destiatios with differet secod digits. Specifically, i the secod partitio, top level switch i carries traffics (from ay sources) to destiatios (s c...s (s 0 = i)(p = 0)), (s c...s (s 0 = (i + )%)(p = )),..., ((s c...s (s 0 = (i + j)%)(p = j)),..., ((s c...s (s 0 = (i + )%)(p = )). Here, % is the module operatio. The routig i other c partitios is similar to that i the secod partitio. The i- th partitio, i c +, is used to route SD pairs to destiatios with differet i-th digit values (differet s i values). Specifically, i the i-th partitio, top level switch i carries traffics to destiatios (s c...(s i = i)...s 0 (p = 0)), (s c...(s i = (i + )%)...s 0 (p = )),..., ((s c...(s i = (i + j)%))...s 0 (p = j)),..., ((s c...(s i = (i + )%)...s 0 (p = )). Note that the routig applies to all SD pairs from all source switches. Lemma 4: The routig i each partitio belogs to Class DIFF. Proof: For the first partitio, ay two destiatios with ode umbers s c...s 0 (p = ) ad s c...s 0 (p = ) i the same switch s c...s 0 will be routed through two differet top level switches ad i this partitio. The routig belogs to Class DIFF by defiitio from Lemma 3. For the secod partitio, differet destiatios i the same switch are also routed through differet top level switches. For a arbitrary bottom level switch s c...s 0, SD pairs with destiatio s c...s 0 0 are routed through top level switch switch s 0, destiatio s c...s 0 through top level switch (s 0 )%,..., ad destiatio s c...s 0 j, 0 j, through top level switch (s 0 j)% i the partitio. Thus, the routig for the secod partitio is also a Class DIFF routig algorithm. Followig a similar logic for the routig i the secod partitio, the routig for the i-th partitio, i c +, is a Class DIFF routig algorithm. As discussed earlier, usig Class DIFF algorithms as buildig blocks i a routig algorithm, if oe ca guaratee that SD pairs from the same switch i ay permutatio ca be routed without cotetio, the algorithm achieves oblockig commuicatio. The followig lemma gives the coditio for a set of SD pairs to be routed through each partitio without causig cotetio. Lemma 5: Let (s, d = s c...s 0 p ), (s, d = s c...s 0 p ),..., (s k, d k = s k c...s k 0p k ) be a set of SD pairs from the same switch i a permutatio. If d, d,..., d k each has a differet first digit p j, j k, the all of the k SD pairs ca be routed through the first partitio without cotetio. If (s j i p j )%, j k, are differet, the all of the k SD pairs ca be routed through the i+-th partitio without cotetio. Proof: Sice the routig i the partitio is a Class DIFF routig algorithm as discussed earlier, from Lemma 3, routes for the set of SD pairs will have o cotetio with paths for SD pairs from other switches i each partitio. Now cosider the routes for SD pairs from the same source. Whe d, d,..., d k each has a differet first digit p j, j k, the all of the k SD pairs will be routed through differet switches p j, j k, i the first partitio. The paths for all of the SD pairs are lik-disjoit. Similarly, whe (s j i pj )%, j k, are differet, each of the k SD pairs will be routed
through a differet top level switch (s j i pj )% i the i + - th partitio: the paths for all the SD pairs are lik-disjoit. Thus, All of the k SD pairs ca be routed through the i + -th partitio without cotetio. Lemma 5 shows that each of the partitios i a cofiguratio ca be used to route a set of SD pairs that satisfy the coditios. I the worst case, at least oe SD pair from each switch ca be routed through each partitio without causig cotetio. We will call a set of SD pairs that ca be routed through a partitio without causig cotetio the set of SD pairs that ca be routed through the partitio. Usig the routig logics, our adaptive routig algorithm for oblockig f tree( + m, r), NONBLOCKINGADAPTIVE, is described i Fig. 4. Sice the routig i each partitio is a Class DIFF routig algorithm, routig SD pairs from differet switches for the same partitio will ot have cotetio ad ca be doe idepedetly. The routig algorithm routes SD pairs from each source switch idepedetly. It cosiders cofiguratios oe at a time, greedily fids the largest umber of SD pairs that ca be routed through oe of the uused partitios i each cofiguratio, ad routes the SD pairs to partitio. This process is repeated with more cofiguratios util all SD pairs are routed. After the cofiguratios for SD pairs from all switches are computed, the algorithm merges the routes for SD pairs from differet source switches (lies (4) ad (5)): the correspodig partitios i each cofiguratio for SD pairs from differet switches ca be routed through the same top level switches without cotetio from Lemma 4. I the descriptio, we assume that there are sufficiet umber of top level switches to be allocated. I the followig, we will first prove this algorithm achieves oblockig commuicatio for ay permutatio ad the give the upper boud of the umber of top level switches required for this algorithm i Theorem 5. Theorem 4: Algorithm NONBLOCKINGADAPTIVE results i oblockig commuicatio. Proof: From Lies (7) ad (8) i Fig 4, we ca see that the NONBLOCKINGADAPTIVE oly routes SD pairs that ca be routed o a partitio to the partitio. It follows that the commuicatio is oblockig for ay permutatio (Lemma 5). Aalyzig the umber of top level switches eeded by the algorithm is more challegig. Before we prove the upper boud for the umber of top level switches eeded i NON- BLOCKINGADAPTIVE, we will show that this algorithm requires less tha top level switches for ay permutatio i ftree( + m, r). For ay two SD pairs from a switch i a permutatio, the destiatios are differet: they differ i at least oe digit i the c + -based digits represetatio. The two SD pairs ca be routed through oe partitio i a cofiguratio: if the first digits of the destiatios are differet, the two SD pairs ca be routed through the first partitio; if the first digits are the same, there exists a j such that the j-th digits of the destiatios are differet, ad the two SD pairs ca be routed through the j-th partitio (s j p s are differet). Hece, the LSET foud i lie (7) i the first Algorithm NONBLOCKINADAPTIVE: Iput: A permutatio P Output: the routes for all SD pairs i P () Let P i, 0 i r, be the set of SD pairs i P from switch i; () For each P i, 0 i r do (3) x i = 0; (4) While (P i is ot empty) do (5) C i x i = ew cofiguratio; x i ++; (6) While ((P i is ot empty) ad (C i x has uused partitios)) do (7) Fid the largest subset of P i, LSET, that ca be routed o oe of the uused partitio, P ART, i C i x i ; (8) Route SD pairs i LSET o P ART ; (9) Mark P ART as used; (0) P i = P i LSET ; () Ed while () Ed while (3) Ed for (4) Let totalcof be the largest x i, 0 i r ; (5) For j=0 to totalcof do merge correspodig partitios i C i j, 0 i r ; (6) Ed for Fig. 4. A adaptive routig algorithm for oblockig ftree( + m, r) iteratio after a ew cofiguratio is allocated will have at least two SD pairs whe there are more tha oe SD pair i P i. I other iteratios, whe P i is ot empty, the LSET at least cotais oe SD pair from each switch sice each uused partitio ca at least route oe SD pair for each switch with o cotetio. Hece, the c+ partitios i oe cofiguratio ca route at least route c+ SD pairs for each source switch. Sice each switch ca have at most SD pairs i a permutatio, at most c+ cofiguratios ad c+ c+ (c + ) = c+ top level switches are eeded, which is less tha the switches required with determiistic routig. I the followig, we will show that NONBLOCKINGADAPTIVE improve the oblockig coditio asymptotically. Lemma 6: Cosider a set of umbers ecoded with c + - based digits, d c d c...d 0. For ay k differet umbers, there exists at least oe i, 0 i c, such that there are at least k (c+) umbers i the set of k differet umbers either with differet d 0 or with differet (d i d 0 )% s. Proof: Let the umber of differet values of each digit d i amog the k differet umbers be X i. If X i < k c+ for all 0 i c, the at most X 0 X... X c differet umbers ca be i the set. Hece, k X 0 X... X c < k, which caot be true. Hece, there exists at least oe i such that X i k c+. If i = 0, there exists k c+ (more tha k (c+) ) umbers with differet d 0 ad the Lemma is prove. If i 0, there existig a set of k c+ umbers with differet di s. If this case, if the
umber of differet d 0 s i the set of umbers is less tha k (c+), the umber of differet (di d 0 )% s i the set is at least k (c+). Theorem 5: Let r c, where c is a costat. Algorithm NONBLOCKINGADAPTIVE requires at most O( (c+) ) top level switches to route ay permutatio i ftree(+m, r). Proof: Cosider the umber of cofiguratios eeded by NON- BLOCKINGADAPTIVE for ay permutatio commuicatio. Each source switch will have at most SD pairs to route i a permutatio. Let s c...s 0 p be a geeric destiatio i the permutatio. From Lemma 6, amog the differet destiatios i the SD pairs, which are represeted by c + -based digits, there exists a i, 0 i c, such that there are (c+) destiatios with differet first digits or with differet (s i p)% s. This set of (c+) SD pairs with oe for each of those destiatios ca be routed through a partitio (Lemma 5). Sice the largest subset of P i that ca be routed o oe partitio is foud i lie (7) i Fig. 4, the first partitio after each ew cofiguratio is allocated will route at least (c+) SD pairs for each source switch whe P i =. Let T () be the umber of cofiguratios eeded whe each switch has SD pairs i a permutatio to be routed. We have T () T ( (c+) ) +. For > X >, (c+) > X (c+) > ( ) (c+). Hece, T () T (/) + ( ) (c+) = T (/) + ( ) (c+) ( ) (c+) ( + ( ) (c+) + ( = O( (c+) ) ) ( (c+) ) +...) Hece, the umber of top level switches eeded for the algorithm is ot more tha T () (c+) = O( (c+) ), assumig c is a costat. Algorithm NONBLOCKINGADAPTIVE asymptotically improves the oblockig coditio for ftree( + m, r) with determiistic routig. However, it may ot achieve the lower boud of m for local adaptive routig. The tight oblockig coditio for ftree( + m, r) with local adaptive routig is, thus, still ope for ivestigatio. VI. CONCLUSION We study folded-clos etworks that are oblockig i computer commuicatio eviromets ad develop techiques to costruct such etworks. We show that it is ot effective to build oblockig folded-clos with small top level switches. We prove that for ftree( + m, r) with sigle-path determiistic routig, the oblockig coditio is m whe r +. We give the sigle-path routig scheme that ca be used to build oblockig ftree( +, r), which is optimal. We further prove that usig local adaptive routig, the oblockig coditio ca be improved over determiistic routig. The results idicate that it is possible to build large oblockig folded-clos etworks i computer commuicatio eviromets usig smaller compoets. This paper leaves a ope problem to be ivestigated i the future: what is the tight oblockig coditio for f tree( + m, r) with local adaptive routig? ACKNOWLEDGMENT The author thaks Prof. Piyush Kumar for providig the proof for the boud derived from the recurrece T () T ( c+ ) +, which is used i the proof of Theorem 5. REFERENCES [] Y. Aydoga, C. B. Stukel, C. Aykaat, ad B. Abali, Adaptive Source Routig i Multistage Itercoectio Networks, the 0th Iteratioal Parallel Processig Symposium, pages 58-67, 996. [] C. Clos, A Study of No-Blockig Switchig Networks, Bell System Techical Joural, 3:406-44, 953. [3] V.E. Bees, O Rearrageable Three-Stage Coectig Networks, Bell System Techical Joural, 4(5):48-49, Sept. 96. [4] V.E. Bees, Mathematical Theory of Coectig Networks ad Telephoe Traffic, Tcademic Press, 965. [5] P. Geoffray ad t. Hoefler, Adaptive Routig Strategies for Moder High Performace Networks, the 6th IEEE Symposium o High Performace Itercoects, pages 65-7, August 008. [6] R. I. Greeberg ad C. E. Lerserso, Ramdozied Routig o Fattrees. I 6th Aual IEEE Symposium o Foudatios of Computer Sciece, pages 4-49, Oct. 985. [7] T. Hoefler, T. Scheider, ad A. Lumsdai, Multistage Switches are ot Crossbars: Effects of Static Routig i High-Performace Networks, IEEE Iteratioal Coferece o Cluster Computig, pages 6-5, 008. [8] Ifiibad T M Trade Associatio, Ifiibad T M Architecture Specificatio, Release., October 004. [9] J. Kim, W. J. Dally, ad D. Abts, Adaptive Routig i High-Radix Clos Network, ACM SC 006, November 006. [0] C.E. Leiserso, Fat-Trees: Uiversal Networks for Hardware-Efficiet Supercomputig, IEEE Trasactios o Computers, 34(0):89-90, October 985. [] C. E. Leiserso, Z. S. Abuhamdeh, D. C. Douglas, C. R. Feyma, M. N. Gamukhi, J. V. Hill, W. D. Hillis, B. C. Kuszmaul, M. A. St. Pierre, D. S. Wells, M. C. Wog-Cha, S-W. Yag, ad R. Zak, The etwork architecture of the Coectio Machie CM-5. Joural of Parallel ad Distributed Computig, 33():45 58, Mar 996. [] X. Li, Y. Chug, ad T. Huag, A Multiple LID Routig Scheme for Fat-Tree-Based Ifiibad Networks. Proceedigs of the 8th IEEE Iteratioal Parallel ad Distributed Processig Symposium (IPDPS 04), p. a, Saa Fe, NM, April 004. [3] S.R. Ohrig, M. Ibel, S.K. Das, M.J. Kumar, O Geeralized Fat Trees, Iteratioal Parallel Processig Symposium, pages 37-44, April 995. [4] F. Petrii ad M. Vaeschi, k-ary -trees: High Performace Networks for Massively Parallel Architectures, Iteratioal Parallel Processig Symposium (IPPS), pages 87-87, 997. [5] A. Sigh, Load-Balaced Routig i Itercoectio Networks, PhD thesis, Staford Uiversity, 005. [6] Y. Yag ad J. Wag, Wide-Sese Noblockig Clos Networks uder Packig Strategy, IEEE Tras. o Computers, 48(3):65-84, March 999. [7] X. Yua, W. Nieaber, Z. Dua, ad R. Melhem, Oblivious Routig i Fat-tree Based System Area Networks with Ucertai Traffic Demads, ACM SIGMETRICS, pages 337-348, Jue 007.