Oblivious Routing for Fat-Tree Based System Area Networks with Uncertain Traffic Demands

Transcription

1 Oblivious Routing for Fat-Tree Based Syste Area Networks with Uncertain Traffic Deands Xin Yuan Wickus Nienaber Zhenhai Duan Departent of Coputer Science Florida State University Tallahassee, FL 3306 Rai Melhe Departent of Coputer Science University of ittsburgh ittsburgh, A 1560 elhe@cs.pitt.edu ABSTRACT Fat-tree based syste area networks have been widely adopted in high perforance coputing clusters. In such systes, the routing is often deterinistic and the traffic deand is usually uncertain and changing. In this paper, we study routing perforance on fat-tree based syste area networks with deterinistic routing under the assuption that the traffic deand is uncertain. The perforance of a routing algorith under uncertain traffic deands is characterized by the oblivious perforance ratio that bounds the relative perforance of the routing algorith and the optial routing algorith for any given traffic deand. We consider both single path routing where the traffic between each source-destination pair follows one path, and ultipath routing where ultiple paths can be used for the traffic between a source-destination pair. We derive lower bounds of the oblivious perforance ratio of any single path routing schee for fat-tree topologies and develop single path oblivious routing schees that achieve the optial oblivious perforance ratio for coonly used fat-tree topologies. These oblivious routing schees provide the best perforance guarantees aong all single path routing algoriths under uncertain traffic deands. For ulti-path routing, we show that it is possible to obtain a schee that is optial for any traffic deand (an oblivious perforance ratio of 1) on the fat-tree topology. These results quantitatively deonstrate that single path routing cannot guarantee high routing perforance while ulti-path routing is very effective in balancing network loads on the fat-tree topology. Categories and Subject Descriptors C..1 [Coputer Systes Organization]: Coputer- Counication Networks Network Architecture and Design General Ters erforance erission to ake digital or hard copies of all or part of this work for personal or classroo use is granted without fee provided that copies are not ade or distributed for profit or coercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific perission and/or a fee. SIGMETRICS 07, June 1 16, 007, San Diego, California, USA. Copyright 007 ACM /07/ $5.00. Keywords Oblivious routing, fat-tree, syste area networks 1. INTRODUCTION The fat-tree topology has any properties that ake it attractive for large scale interconnects and syste area networks [8, 9]. Most iportantly the bisection bandwidth of the fat-tree topology scales linearly with the network size. The topology is also inherently highly resilient with a large nuber of redundant paths between two processing nodes. The fat-tree topology is very popular for building ediu and large syste area networks [6, 1]. In particular, it has been widely adopted by high perforance coputing (HC) clusters that eploy the off-the-shelf high speed syste area networking technology, such as Myrinet [13] and Infiniband [7]. The fat-tree topology is used in any of the top 500 fastest supercoputers listed in the June 006 release [16]. Although the fat-tree topology provides rich connectivity, having a fat-tree topology alone does not guarantee high network perforance: the routing echanis also plays a crucial role. Historically, adaptive routing, which dynaically builds the path for a packet based on the network condition, has been used with the fat-tree topology to achieve load balance in the network [9]. However, the routing in the current ajor syste area networking technology such as Infiniband and Myrinet is deterinistic [7, 13]. For a fattree based syste area network with deterinistic routing, it is iportant to eploy an efficient load balance routing schee in order to fully exploit the rich connectivity provided by the fat-tree topology. Traditional load balance routing schees usually optiize the network usage for a given traffic deand. Such deand specific schees ay not be effective for syste area networks where the traffic deand is usually uncertain and changing. Consider, for exaple, the traffic deands in a large HC cluster. Since any users share such a syste and can run any different applications, the traffic deand depends both on how the processing nodes are allocated to different applications and on the counication requireent within each application. Hence, an ideal routing schee should provide load balancing across all possible traffic patterns. This requireent otivates us to study deand-oblivious load balance routing schees, which have recently been shown to proise excellent perforance guarantees with changing and uncertain traffic deands in the Internet environent [1,, 18]. In this paper, we investigate routing perforance on fat-

2 tree based syste area networks with deterinistic routing under the assuption that the traffic deand is uncertain and changing. For a given traffic deand that can be represented by a traffic atrix, the perforance of a routing schee is easured by the axiu link load etric. The perforance of a routing algorith under uncertain traffic deands is characterized by the oblivious perforance ratio [1]. The foral definition of the oblivious perforance ratio will be introduced in the next section. Inforally, a routing algorith, r, with an oblivious perforance ratio of x eans that for any traffic deand, the perforance (axiu link load) of r on the deand is at ost x ties that of the optial routing algorith for this deand. An oblivious perforance ratio of 1 eans that the routing algorith is optial for all traffic deands. Syste area networks, including Infiniband and Myrinet, typically support single path routing and soe for of ultipath routing. It is well known that single path routing is siple, but ay not be as effective as ulti-path routing in balancing network loads. On the other hand, ulti-path routing introduces coplications such as packet reordering that the network syste ust handle. However, the perforance difference between single path routing and ultipath routing on the fat-tree topology is not well understood. Without a clear understanding of the perforance difference, it is difficult to ake a wise decision about whether a syste should use single path routing for its siplicity or ulti-path routing for its perforance. This paper resolves this proble: it provides a concrete quantitative coparison between the perforance of single path routing and ulti-path routing on the fat-tree topology. This study focuses on fat-tree topologies fored with - port switches, where is a paraeter that is restricted to be a ultiple of. Although the results are obtained for this type of fat-trees, the results, as well as our analyzing techniques, can be easily extended to other types of fat-tree topologies. The ajor conclusions in this paper include the following. For a 3-level fat-tree, we prove that the oblivious perforance ratio of any single path routing algorith is at least p. For a 4-level fat-tree, we prove that the oblivious perforance ratio of any single path routing algorith is at least. For a fat-tree of height H, H > 4, we show that the oblivious perforance ratio of any single path routing algorith is at least ( H ) 3. These lower bounds indicate that single path routing cannot guarantee high routing perforance on the fat-tree topology. For exaple, for any single path routing algorith on a 4-level fat-tree fored by 16-port switches, there always exists a traffic deand such that this routing algorith is 16 = 8 ties worse than the optial algorith for that traffic deand. We show that the lower bounds are tight for 3-level and 4-level fat-trees by developing optial single path oblivious routing schees that achieve these bounds. These algoriths provide the best perforance guarantees aong all single path routing algoriths under uncertain traffic deands. It ust be noted that practical fat-tree topologies are usually no ore than 4 levels: depending on the nuber of ports in the switches foring the fat-tree, a 4-level fat-tree can easily support ore than ten thousands processing nodes. Hence, the single path routing schees developed in this paper are sufficient for ost practical fat-tree based networks. For ulti-path routing, we show that it is possible to obtain a schee that is optial for any traffic deand (an oblivious perforance ratio of 1) on the fat-tree topology. This suggests that ultipath routing is uch ore effective than single path routing in providing the worst case perforance guarantees on the fat-tree topology. The rest of the paper is organized as follows. In Section, we forally define routing and the etrics for evaluating routing schees and specify the fat-tree topology. In Section 3, we study the single path oblivious routing schees for the fat-tree topology. In Section 4, we present the results for ulti-path routing. Section 5 reports the results of our perforance study of the proposed algoriths and other routing algoriths designed for the fat-tree topology. Section 6 describes the related work. Finally, Section 7 concludes the paper.. BACKGROUND.1 Routing and its perforance etrics Let the syste have N processing nodes, nubered fro 0 to N 1. The traffic deand is described by an N N Traffic Matrix, T M. Each entry t i,j in T M, 0 i N 1, 0 i N 1, denotes the aount of traffic fro node i to node j. Let A be a set, A denotes the size of the set. The definitions of routing and the perforance etrics in this paper are odeled after [1]. A routing specifies how the traffic of each Source-Destination (SD) pair is routed across the network. We consider two types of routing schees: single path routing where only one path can be used for each SD pair, and ulti-path routing where ultiple paths can be used. In ulti-path routing, each path for an SD pair routes a fraction of the traffic for the SD pair. The ulti-path routing can be characterized by a set of paths M i,j = {Mi,j, 1 Mi,j,..., M M i,j } for each SD pair (i, j), and the fraction of the traffic routed through each path f i,j = {fi,j k k Mi,j = 1,,..., M i,j }. k=1 fi,j k = 1. Hence, a ulti-path routing, r, is specified by a set of paths M i,j and a vector representing the fraction of the traffic routed through each path f i,j for each SD pair (i, j), 0 i N 1 and 0 j N 1. Let link l Mi,j, k the contribution of the traffic t i,j to link l through path Mi,j k is thus t i,j fi,j. k Notice that link l ay be in ore than one path in M i,j. In this case, ultiple paths for the sae SD pair can contribute to the traffic on link l. Single path routing is a special case of ulti-path routing where M i,j = 1 and all traffic fro node i to node j is routed through Mi,j 1 (fi,j 1 = 1). Hence, a single path routing can be specified by a path Mi,j 1 for each SD pair (i, j). A coon etric for the perforance of a routing schee with respect to a certain traffic atrix, T M, is the axiu link load. Since all links in a fat-tree network have the sae capacity, the axiu link load is equivalent to the axiu link utilization. Let Links denote the set of all links in the network. For a ulti-path routing r, the axiu link load is given by MLOAD(r, T M) = ax { l Links i,j X i,j,k such that l M k i,j t i,j f k i,j}. For a single path routing sr, the axiu link load forula is degenerated to MLOAD(sr, T M) = ax { X t i,j}. l Links i,j such that l 1 i,j

3 An optial routing for a given traffic atrix T M is a routing that iniizes the axiu link load. Forally, the optial load for a traffic atrix T M is given by O T U(T M) = in {MLOAD(r, T M)} r is a routing The perforance ratio of a given routing r on a given traffic atrix T M easures how far r is fro being optial on the traffic atrix T M. It is defined as the axiu link load of r on T M divided by the iniu possible axiu link load on T M [1]. ERF (r, T M) = MLOAD(r, T M) O T U(T M) The value for ERF (r, T M) is always at least 1. It is exactly 1 if and only if the routing is optial for T M. When a routing is optiized for a specific traffic atrix, it does not provide any guarantees for other traffic atrices. The definition of the perforance ratio follows the copetitive analysis fraework where perforance guarantees of a certain solution are provided relative to the best possible solution. The definition of perforance ratio of a routing is extended to be with respect to a set of traffic atrices [1]. Let Γ be a set of traffic atrices, The perforance ratio of a routing r on Γ is defined as ERF (r, Γ) = ax { ERF (r, T M)} T M Γ When the set Γ includes all possible traffic atrices, the perforance ratio is referred to as the oblivious perforance ratio [1]. The oblivious perforance ratio of a routing r is denoted by ERF (r). The oblivious perforance ratio is the worst perforance ratio that a routing obtains with respect to all traffic atrices. A routing with a iniu oblivious ratio is an optial oblivious routing schee and its oblivious ratio is the optial oblivious ratio of the network.. Fat-tree topology In a fat-tree network, all links are bidirectional with the sae capacity. Figure 1 copares a binary tree with a binary fat-tree topology. In the binary tree, the nuber of links (and thus the aggregate bandwidth) is reduced by half at each level fro the leaves to the root. This can cause serious congestion towards the root. The binary fat-tree topology reedies this situation by aintaining the sae bandwidth at each level of the network. (a) Binary tree (b) Binary fat tree Figure 1: Binary tree and binary fat-tree topologies The fat-tree topology shown in Figure 1 (b) is not practical for building large networks due to the large nodal degree of the root. Alternatives were proposed to approxiate such a topology using ulti-stage networks that are fored by nodes with sall nodal degrees [6, 17]. For exaple, the fat-tree in Figure 1 (b) can be approxiated by the topology in Figure. These alternatives trade the connectivity with the ipleentation siplicity. In this paper, we focus on one of such alternatives: the fat-tree topologies fored by -port switches, where is a paraeter that is restricted to a ultiple of. Let an internal node in the fat-tree topology be a node with a degree ore than 1. All internal nodes in our fat-tree topology has a degree of (so that they can be realized by -port switches). Such a topology is a inor generalization of the topology proposed in [6]. The technique we developed for this topology can easily be extended for other fat-tree variations. Figure : Approxiate the topology in Figure 1 (b) We will follow the naing convention in [6]: the fat-tree is called -port n-tree and denoted as F T (, n). The paraeter in F T (, n), which ust be a ultiple of, specifies the nodal degree of all internal nodes in the topology. The paraeter n specifies the nuber of levels of internal nodes in the topology. Thus, the height of F T (, n) is n + 1, that is, F T (, n) is an n + 1 level tree. F T (, n) is a inor generalization of the topology in [6] in that we allow to be a ultiple of while in [6], ust be a power of. In the rest of this paper, internal nodes in F T (, n) ay also be referred to as switches since each of the internal nodes is realized by a switch when the topology is constructed. Siilarly, leaf nodes ay also be referred to as processing nodes. A 4-port 3-tree, F T (4, 3), is shown in Figure 3. Notice that in practice a processing eleent ay have ultiple network adapters and have ultiple connections to the network. Fro the network point of view, however, supporting such a processing eleent is the sae as supporting ultiple processing nodes, each having one connection to the network with the cobined traffic representing the traffic for the processing eleent. Hence, we will assue that each processing node has one connection to the network. Notice also that the topology in Figure is not an -port n-tree since there are two types of switches in the topology: toplevel switches have a degree of and the lower-level switches have a degree of 4. Next, we will describe how F T (, n) is fored. More details can be found in [6]. F T (, n) is fored by connecting the root level switches to sub-fat-trees with n 1 levels of switches. We will use the notion SUBF T (, n 1) to denote the sub-fat-trees with n 1 levels of switches. SUBF T (, l) is different fro F T (, l) in that SUBF T (, l) ust provide (open ended) up links for the sub-fat-tree while F T (, l) does not have up links. SUBF T (, l) is recursively constructed as follows. When l = 1, SUBF T (, 1) contains 1 -port switch. of the ports in the switch connect to processing nodes, and ports reain open. We will call these opened ports up-link ports since they will be used to connect to the upper level switches. We denote the nuber of up-link ports in

4 0 1 3 switch link level level 0 0 processing nodes.» ¼» ¼ ½ ¾ ½ ¾ À n 1 À ( ) top level switches n 1 ( ) n 1 ( ) 1 ¹ º ¹ º 0 1 n 1 ( ) 1 a b SUBFT(, n 1) SUBFT(, n 1) sub fat trees SUBFT(, n 1) SUBFT(4, ) Figure 3: The 4-port 3-three (F T (4, 3)) SUBF T (, l) as nu(, l). nu(, 1) =. As will be shown later, nu(, l) = ( )l. The up-link ports in SUBF T (, l) are nubered fro 0 to nu(, l) 1. SUBF T (, l) is fored by having nu(, l 1) = ( )l 1 -port top level (of the sub-fat-tree) switches connecting with SUBF T (, l 1) s. Each of the top level switches uses ports to connect to all of the SUBF T (, l 1) s. Let us nuber top level switches fro 0 to nu(, l 1) 1. The up-link ports i, 0 i < nu(, l 1), in all of the SUB(, l 1) s are connected to top level switch i. The rest ports in a top level switch are up link ports of SUBF T (, l). Top level switch i provides up-link ports i to (i+1) 1 for SUBF T (, l). Figure 4 (a) shows SUBF T (, 1) and (b) shows the structure of SUBF T (, l). Clearly, nu(, l) = nu(, l 1) = ( )l. Hence, each SUBF T (, l) has ( )l up-link ports and connects to ( )l processing nodes. 01 ( ) 1 (a) SUBFT(, 1) 0 1 A B A B C D C D ) * ) * E F E F +, +, G H G H I J I J l ( ) 1 01 ( ) 1 01 ( ) l 1 ( ) switches l 1 l 1 l 1 ( ) 1! "! " SUBFT(, l 1) SUBFT(, l 1) SUBFT(, l 1) / sub fat trees (b) SUBFT(, l) Figure 4: SUBF T (, 1) and SUBF T (, n) F T (, n) is fored by having nu(, n 1) = ( )n 1 root level switches connecting with SUBF T (, n 1) s. Let us nuber top level switches fro 0 to ( )n 1 1. The up-link port i, 0 i < nu(, n 1), in all of the SUB(, n 1) s is connected to top level switch i. Each of the ports in the root level switch connects to one SUBF T (, n 1). The structure of F T (, n) is shown in Figure 5. Note that F T (, n) does not need to aintain any open ended link and hence, each of the ports in a root level switch can connect to a sub-fat-tree. F T (, n) supports ( )n 1 processing nodes. It has n levels of switches. The root level contains nu(, n 1) = ( )n 1 switches and each of the other n 1 layers has ( )n 1 switches. Hence, the total nuber of switches in F T (, n) is (n 1) ( )n 1. Figure 3 shows the coplete topology of F T (4, 3) that has ( 3 1) ( 4 )3 1 = 0 switches and 4 ( 4 )3 1 = 16 Figure 5: The structure of F T (, n) F T (, n) has n levels of switches, which are nubered fro 0 to n 1 (root level switches being level 0 switches). Siilarly, we will classify the links according to the levels. For 0 i < n 1, the links connecting level i switches with level i + 1 switches are called level i links. The links connecting level n 1 switches with the processing nodes are level n 1 links. All links in F T (, n) are bi-directional links: with an up channel for counication fro a lower level switch to an upper level switch, and a down channel for counication fro an upper level switch to a lower level switch. We will use the ter level i up link to denote an up channel fro a level i+1 switch to a level i switch, and level i down link to denote a down channel fro a level i switch to a level i+1 switch. A path between two processing nodes in F T (, n) has two phases: the first phase contains only up channels and the second phase contains only down channels. Fro the definition of F T (, n), one can easily derive the following properties. roperty 1: F T (, n) contains SUBF T (, n 1) s, SUBF T (, n ) s,..., ( )n SUBF T (, 1) s. Level 0 (root level) switches do not belong to any sub-fattrees. Each level 1 switch is in a SUBF T (, n 1) each level switch is in a SUBF T (, n 1) and a SUBF T (, n ) and so on. A level i switch, 1 i n 1, is in a SUBF T (, n 1), a SUBF T (, n ),..., and a SUBF T (, n i). In F T (, n), we will call switches in levels 0, 1,..., and n i 1 the upper level switches for SUBF T (, i). The upper level switches for SUBF T (, i) provide connectivity aong all SUBF T (, i) s. roperty : Through upper level switches for SUBF T (, i), 1 i n 1, an up-link port in a SUBF T (, i) is only connected with the up-link ports with the sae port nuber in other SUBF T (, i) s. More specifically, up-link port j, 0 j nu(, i) 1 in one SUBF T (, i) is only connected with the up-link port j of other SUBF T (, i) s (but not other ports) through upper level switches for SUBF T (, i). It is clear that this property is true for SUBF T (, n 1). The property for general SUBF T (, i), 1 i < n 1, can be forally proven by induction on i (with base case i = n 1) and by exaining how the top level switches in SUBF T (, i) s are connected. roperty 3: Let SUBF T (, i) be the sallest sub-fattrees in F T (, n) that contains two processing nodes a and b, there exist ( )i 1 different shortest paths fro a to b. If such a sub-tree does not exist, there are ( )n 1 different shortest paths fro a to b. In this case, a and b are in different top level sub-fat-trees (SUBF T (, n 1) s). Figure 3 shows an exaple. Fro node a to node b in F T (4, 3), there are ( )n 1 = = 4 shortest paths. In both cases in roperty 3, the nuber of shortest paths between

5 any two nodes can be represented as ( )x with the value of x, 0 x n 1, depending on the positions of the source and the destination. roperty 4: In F T (, n), let there exist ( )x different shortest paths fro processing node s to processing node d. Each of the level n 1 i up/down links that carry traffic fro s to d is used by ( )x i shortest paths, 0 i x. This property is intuitive. For exaple, level n 1 links are the links connecting processing nodes. Hence, all paths fro the processing node connected by a level n 1 link ust use the link. This is the case when i = 0: all ( )x shortest paths use the link. For the next level (i = 1), a source will have choices (the fan-out fro the first switch) to go to another node (when the path uses such a link). Thus, each of such links will be used by ( )x / = ( )x 1 shortest paths. The cases for links in other levels are siilar. Consider the 4 paths fro node s to node d in Figure 3, all 4 paths use the level up/down links (the link connecting the processing node), paths use each of the level 1 up/down links that carries traffic fro a to b, and 1 path uses each of the level 0 up/down links carrying traffic fro a to b. roperty 5: In F T (, n), a level i, 0 i n 1, up link carries traffic fro at ost ( )n 1 i source nodes. A level i down link carries traffic to at ost ( )n 1 i destination nodes. This property is also intuitive. For exaple, when i = n 1, a level i = n 1 link directly connects to a processing nodes. So such a link carries traffic to/fro at ost ( )n 1 (n 1) = 1 node. When i = n, the link connects to a level n 1 switch and such a link carries traffic to/fro the ( )n 1 (n ) = nodes directly connected to that switch. 3. SINGLE ATH OBLIVIOUS ROUTING 3.1 Lower bounds of oblivious perforance ratio for single path routing In this section, we derive the lower bounds of oblivious perforance ratio for single path routing on F T (, n). The following concepts will be used in the derivation of the lower bounds. Let A = {(s 1, d 1), (s, d ),...} be a set of SD pairs. Definition 1: The set of SD pairs, A = {(s 1, d 1), (s, d ),...}, is said to be node disjoint if for any (s i, d i) A and (s j, d j) A, i j, s i s j and d i d j. Basically, in a node disjoint set of SD pairs, each source (in the source-destination pair) appears in the set as a source exactly once and each destination appears in the set as a destination exactly once. It ust be noted that a node ay appear as a source and as a destination in a node disjoint set. For exaple, {(1, ), (1, 3)} is not a node disjoint set while {(1, ), (3, 1)} is. Definition : For a given set of SD pairs A, a set of SD pairs B is said to be a node disjoint subset of A when the following conditions are et: (1) B A and () B is a node disjoint set. Definition 3: For a given set of SD pairs A, a set of SD pairs B is said to be a largest node disjoint subset of A when the following two conditions are et: (1) B is a node disjoint subset of A and () let C be a node disjoint subset of A, B C. Let L(A) be the size of a largest node disjoint subset of A. Let Ss A = {(s, x) (s, x) A} be the set of SD pairs in A with source node s and Dd A = {(x, d) (x, d) A} be the set of SD pairs in A with destination node d. SRC(A) = {s (s, d) A} is the set of source nodes in A and DST (A) = {d (s, d) A} is the set of destination nodes in A. We denote LS(A) the largest nuber of SD pairs in A either with the sae source or with the sae destination. Forally, LS(A) = ax{ ax s SRC(A) Ss A, ax Dd A }. d DST (A) For any node i, Si A and Di A are at ost LS(A). Consider for exaple A = {(1, ), (1, 3), (, 1), (, 4), (3, 1)}. The set {(1, ), (, 1)} is a node disjoint subset of A, but not a largest node disjoint subset. Both {(1, ), (, 4), (3, 1)} and {(1, 3), (, 4), (3, 1)} are largest node disjoint subsets of A. Hence, L(A) = 3. SRC(A) = {1,, 3} and DST (A) = {1,, 3, 4}. S1 A = {(1, ), (1, 3)} S A = {(, 1), (, 4)} and S3 A = {(3, 1)}. D1 A = {(, 1), (3, 1)} D A = {(1, )} D3 A = {(1, 3)} and D4 A = {(, 4)}. Hence, LS(A) =. The following leas give soe properties of these concepts. Lea 1: Let A be a set of SD pairs, SRC(A) L(A) and DST (A) L(A). roof: Straight-forward fro the largest node disjoint subset definition. Lea : Let A and B be two sets of SD pairs, L(A) + L(B) L(A S B). roof: Let C be a largest node disjoint subset of A S B. C = L(A S B). Each eleent in C ust either be in A, or in B (or in both A and B). Let C A = {(s, d) (s, d) C A} and C B = {(s, d) (s, d) C B}. We have C A + C B C = L(A S B). Since C A is a node disjoint subset of A and C B is a node disjoint subset of B, by definition, L(A) C A and L(B) C B. Hence, L(A) + L(B) L(A S B). Lea 3: Let A be a set of SD pairs. If there is a source node s such that Ss A > L(A), then L(A Ss A ) = L(A) 1. roof: Since Ss A has only one source node, L(Ss A ) = 1. Fro Lea, we have L(A Ss A ) + L(Ss A ) L((A Ss A ) S Ss A ) = L(A). Hence, L(A Ss A ) L(A) 1. Next, we will prove L(A Ss A ) L(A) 1 by contradiction. Let B = {(s 1, d 1), (s, d ),..., (s k, d k )} be a largest node disjoint subset of A Ss A. Assue that B = k = L(A Ss A ) > L(A) 1. Since A Ss A is a subset of A, k L(A). Hence, k ust be exactly equal to L(A). Since Ss A > L(A) = k, there exists at least one (s, d) Ss A such that d d i, 1 i k. Hence, the set C = B S {(s, d)} is node disjoint and C = L(A) + 1. Since C is a node disjoint subset of A, C L(A). This is the contradiction. Hence, L(A Sa A ) = L(A) 1. Lea 3a: Let A be a set of SD pairs. If there is a destination node d such that Dd A > L(A), then L(A Dd A ) = L(A) 1. Lea 4: Let A be a set of SD pairs. If there exist k source nodes s i, 1 i k, such that Ss A i > L(A), and l destination nodes a j, 1 j l, such that Dd A j > L(A), then L(A S k i=1 SA s i S l j=1 DA d j ) = L(A) k l. roof: The conclusion in this lea is obtained by repeatedly applying Lea 3 and Lea 3a.. Lea 5: Let A be a set of SD pairs. A L(A) LS(A). roof: See Appendix. Besides using the concepts introduced in the above definitions and leas, we will use a topology, called extended -layer fat-tree, in the derivation of the lower bounds for oblivious perforance ratio on F T (, n). The extended -

6 layer fat-tree, denoted as EF T (, k), contains two levels of switches. The top level contains k-port switches. The botto level contains k -port switches. Half of the ports in the botto level switches are used to connect processing nodes and the other half connecting top level switches. There is a link between each top level switch and each botto layer switch. The structure of EF T (, k) is siilar to F T (, ), which is shown in Figure 6. The difference is that F T (, ) uses the sae kind of switches in both levels while EF T (, k) has ore flexibility: the switches in the top level can be different fro the switches in the botto level. The F T (, ) topology is the sae as EF T (, ). We will also use a sub-graph of EF T (, k), which we call SEF T (, k). SEF T (, k) contains all lower level switches and processing nodes in EF T (, k), but only one root level switch. Figure 7 shows the SEF T (, k) topology, which is basically a regular tree topology with the root having k children and each level 1 switch having children. In this figure, we separate the two directional channels. (0, 0) (1, 0) (/ 1, 0) / (0, 1) (1, 1) ( 1, 0) 6 (0, 0) (0, / 1) (1, 0) (1, / 1) ( 1, 0) ( 1, / 1) Figure 6: F T (, ) topology W W X X a a b b sw(0) R sw(1) c Y d Zc (0, 0) (0, / 1) (1, 0) (1, / 1) (k 1, 0) (k 1, / 1) : < : < O O Q R Q R sw(k 1) Figure 7: SEF T (, k) topology Lea 6: Let the processing nodes in EF T (, k) be nubered fro 0 to N 1. Let A = {(s 1, d 1),..., (s A, d A )} be a set of node disjoint SD pairs (for 1 i A, s i {0,..., N 1} and d i {0,..., N 1}). When A k, the SD pairs in A can be routed in EF T (, k) with A link disjoint paths. roof: In EF T (, k), each of the top level switches has a link with each of the botto level switches. Since A k, we can assign a different top level switch for each SD pair (s i, d i) A. For each (s i, d i), if s i and d i are in the sae switch, there is only one path between s i and d i (fro s i to the switch connecting both s i and d i, and then to d i). Since A is node disjoint, these links are not used by other paths for other SD pairs. If s i and d i are not in the sae switch, the path for (s i, d i) is: fro s i to the botto level switch connecting s i to the top level switch assigned to (s i, d i) to the botto level switch connecting d i to d i. This way, all the SD pairs in A are routed with link disjoint paths. Lea 7: Let sr be a single path routing on EF T (, k). Assue that under routing sr, there exists a link l that carries traffic for a set A of node disjoint SD pairs, A k, Then, ERF (sr) A. roof: To show that ERF (sr) A, we ust show that MLOAD(sr,T M) O T U(T M) there exists a traffic atrix T M such that A. Consider a traffic atrix T M where t i,j = 1 for all (i, j) A and all other entries are 0 (no other traffic). Fro Lea 6, there exists a routing schee sr that routes the SD pairs in A using link disjoint paths. Hence, MLOAD(sr, T M) = 1 and O T U(T M) 1. Since using routing sr, the load on link l is A and MLOAD(sr, T M) A. Hence, ERF (sr) MLOAD(sr, T M) OUT U(T M) A 1 = A. For a single path routing r, let us define the axiu disjoint size on link l, ds(r, l), to be the size of the largest node disjoint subset of the set of SD pairs routed on l. The axiu disjoint size of routing r, ds(r), is defined as ds(r) = ax l Links ds(r, l). Notice that in EF T (, k) and SEF T (, k), a level 1 link directly connected to a processing node. Fro Lea 1, the axiu disjoint size on such a link is at ost 1. Lea 8: Consider using the SEF T (, k) topology to route a subset of all possible SD pairs. If the largest of the axiu disjoint sizes of all links is at ost X, the nuber of SD pairs routed through the root is at ost k(k 1)X when X. k roof: In SEF T (, k), at ost k(k 1)( ) SD pairs can be routed through the root. The lea is always true when X. Let (s, d) be a SD pair. The pair ust be routed through the root only when nodes s and d are connected to different switches. We will call the root switch in SEF T (, k) switch R and the k level 1 switches sw(0), sw(1),..., sw(k 1) as shown in Figure 7. Let S be a largest set of SD pairs that are routed through the root when the largest of the axiu disjoint sizes of all links is at ost X. Let S i,j, 0 i j k 1, be the set of SD pairs in S with source nodes in switch sw(i) and destination nodes in switch sw(j). S = S i,j such that 0 i j k 1 Si,j. Let us denote LX src i,j = [ a such that a SRC(S i,j ) and S S i,j a >X S S i,j a Let E i,j = SRC(LXi,j src ). For the SD pairs in S i,j, E i,j is the nuber of source nodes in switch sw(i), each of which has ore than X destination nodes in switch sw(j). LXi,j src contains all such SD pairs. Siilarly, we will denote LX dst i,j = [ d such that d DST (S i,j ) and D S i,j >X d D S i,j d Let F i,j = DST (LXi,j dst ). For the SD pairs in S i,j, F i,j is the nuber of destination nodes in switch sw(j), each of which has ore than X source nodes in switch sw(i). LXi,j dst contains all such SD pairs. All SD pairs in S i,j ust pass through links sw(i) R and R sw(j). First, let us consider link sw(i) R. Let S all SD pairs with source nodes in sw(i) be All i R = j i Si,j. All SD pairs in Alli R ust go through link sw(i) R. Hence, L(All i R) X. Fro Lea 4, L(All i R S x i LXsrc i,x ) X x i Ei,x. Since Si,j LXi,j src All i R S x i LXsrc i,x, we have L(S i,j LXi,j src ) L(All i R S x i LXsrc i,x ) X x i Ei,x. Hence, applying Lea 4,

7 L(S i,j LXi,j src LXi,j dst ) X X E i,x F i,j. x i Using the siilar logic, by considering link R sw(j), we can obtain L(S i,j LX src i,j LX dst i,j ) X E i,j X x j F x,j. Cobining these two in-equations, we obtain L(S i,j LXi,j src LXi,j dst ) X ( x i Ei,x+Fi,j+Ei,j+ x j Fx,j)/. Each source or destination node in S i,j LXi,j src LXi,j dst can have no ore than X SD pairs in the set (otherwise, these SD pairs would be included in either LXi,j src Hence, LS(S i,j LXi,j src LXi,j dst ) X. 5, S i,j LXi,j src LS(S i,j LX src x j S k 1 k 1 k 1 i,j LX dst i,j L(S i,j LX src i,j LX dst i,j ) (X ( x i Fx,j)/) X. Hence, 9 Sj i j i (Si,j LXsrc i,j Si,j LXsrc i,j LX dst i,j ) LX dst i,j Ei,x + Fi,j or LXi,j dst ). Fro Lea LXi,j dst ) Ei,x +Fi,j +Ei,j + j i X (X ( x i +E i,j + x j Fx,j)/) k(k 1)X kx k 1 j i Ei,j kx k 1 j i Fi,j Since each switch connects to src processing nodes, LX i,j E i,j and LXdst i,j F i,j. Hence, S = S k 1 S j i Si,j S k 1 S k 1 Sj i Sj i (Si,j LXsrc Sj i LXsrc k 1 + S k 1 k(k 1)X kx + k 1 ((Si,j LXsrc i,j LXi,j dst ) S LXi,j src i,j LXi,j dst ) i,j + S k 1 Sj i LXdst i,j j i Ei,j + = k(k 1)X ( kx )( k 1 j i Ei,j kx k 1 j i Fi,j k 1 j i Ei,j + k 1 j i Fi,j) S LX dst i,j ) j i Fi,j When X, kx. Thus, S k(k 1)X. k Let us denote the axiu nuber of SD pairs routed through SEF T (, k) when the largest of the axiu disjoint sizes of the links in SEF T (, k) is X by T(X). Obviously, when X > Y and T (Y ) is a subset of all SD pairs that can be routed, T (X) > T (Y ) regardless of the relation aong X,, and k. Lea 8 states that when X, T (X) k(k 1)X. Hence, when X <, T (X) < k k T ( ) k(k 1)( k k ). Lea 9: Let r be a single path routing algorith on EF T (, k). If k, ERF (r) p. roof: Regardless of the single path routing algorith r used, k(k 1)( ) SD pairs ust be routed through top level switches. Since there are top level switches in EF T (, k), at least one top level switch ust carry k(k 1)( ) = k(k 1) SD pairs. Consider the SEF T (, k) fored by that particular top level switch (with k(k 1) SD pairs passing through) with all level 1 switches and all processing nodes. Let the axiu disjoint size of the links connecting to this switch be X. Under the assuption k, if X < k = p, T (X) < k(k 1)( k ) k(k 1). Since there are k(k 1) SD pairs ust be routed through the switch (T (X) = k(k 1) ), X < p cannot be true. Thus, X p. Since k p, fro Lea 7, ERF (r) p. The following three leas are the ain results in this section. Lea 10: Let r be a single path routing algorith for F T (, ) ( ), ERF (r) p. roof: F T (, ) is equivalent to EF T (, ). Since in F T (, ), > and p. Fro Lea 9, ERF (r). Lea 11: Let r be a single path routing algorith for F T (, 3), ERF (r). roof: F T (, 3) is coposed by top level switches (-port) with SUBF T (, ) s. Let us consider the axiu disjoint sizes on the links that connect the SUBF T (, ) s with the root level switches (Level 0 links). By treating each SUBF T (, ) as one ( ) -port switch, F T (, 3) is approxiated as EF T (( ), ). Following the proof of Lea 9, since p ( ), the largest of the axi- q ( u disjoint sizes of the level 0 links is at least ) =. Fro Lea 7, ERF (r) =. Lea 1: Let r be a single path routing algorith for F T (, n), ERF (r) ( n 1 ) 3. roof: Let us consider the axiu disjoint sizes on links connecting the to up-link ports of SUBF T (, i) s, 1 i n 1, in F T (, n). Fro roperty 1 and roperty of F T (, n), the connectivity in F T (, n) can be partitioned into two levels (with respective to such links): the lower level connectivity provided by SUBF T (, i) s and the upper level connectivity provided by the upper level switches for SUBF T (, i) s. The connectivity in SUBF T (, i) can be approxiated as a ( )i -port switch and the upper level switches the connects the up-link ports with the sae port nuber in each of the SUBF T (, i) (roperty ), which approxiates a ( )n 1 i -port switch. Consider the case when i = (n 1), the topology can be approxiated by 3 EF T (( (n 1) ) 3, ( n 1 ) 3 ). Since ( ) n 1 3 r ( (n 1) ) 3, following the proof in Lea 9, the largest of the axiu disjoint sizes on such links is at least s ( (n 1) ) 3 = ( ) n 1 3. Fro Lea 7, ERF (r) ( n 1 ) 3. Note that the tree height of F T (, n) is n+1: F T (, ) is a 3-level fat-tree F T (, 3) is a 4-level fat-tree and F T (, n) is n + 1-level. This sub-section establishes that the lower bounds of the oblivious perforance ratio for any single path routing is p for a 3-level fat-tree, for a 4-level fattree, and ( H ) 3 for an H-level fat-tree, H > 4. Notice also that while our results are obtained for -port n-trees (F T (, n)), the techniques can be easily extended to other types of tree (e.g. the ones with root level switches being different fro switches in other levels such as the fat-tree shown in Figure ). 3. Optial single path oblivious routing for F T (, ) and F T (, 3) Most practical fat-tree topologies have no ore than three levels of switches. This is because it is coon to use 3- port or 48-port switches to construct large fat-tree topolo-

8 gies. Using 3-port switches, F T (3, ) supports up to 51 processing nodes while F T (3, 3) support up to 819 processing nodes. With 48-port switches, F T (48, 3) can support = 7648 processing nodes. Hence, optial oblivious routing schees for F T (, ) and F T (, 3) bear ost practical significance. Moreover, the developent of these algoriths also bears theoretical significance by aking the lower bounds on the oblivious perforance ratio for F T (, ) and F T (, 3) (Lea 10 and Lea 11) tight bounds. Let N be the nuber of processing nodes. Let the traffic atrix be T M with entries t i,j, 0 i N 1 and 0 j N 1, specifying the aount of traffic fro node i to node j. The total traffic sent fro node i is j ti,j and the total traffic received by node i is j tj,i. Since there is only one link connecting each processing node to the network, such traffic ust be carried on that link regardless of the routing schee. Hence, for any routing schee (single path or ulti-path) the load on the link (which has two directions) connecting to node i is ax{ j ti,j, j tj,i}. We define the base load of a traffic atrix T M as baseload(t M) = ax 0 i N 1 {ax{x j t i,j, X j t j,i}}. The iniu axiu link load on the fat-tree topology using any routing schee, single path or ulti-path, is at least baseload(t M) for any traffic atrix T M. In other words, O T U(T M) baseload(t M). Our optial single path oblivious routing schees are based on the following Lea. Lea 13: If a single path routing schee r routes SD pairs such that the SD pairs in each of the links in F T (, n) are either fro at ost X sources or towards at ost X destinations, then ERF (r) X. roof: As discussed earlier, for any traffic deand T M, on F T (, n), O T U(T M) baseload(t M). Since each link carries traffic either fro at ost X sources or towards X destinations, the load of the link is no ore than X X baseload(t M) baseload(t M), hence, ERF (r, T M) = baseload(t M) X. Since this applies for any traffic deand T M, ERF (r) X. Optial oblivious routing for F T (, ) As defined in Section, F T (, ) contains 3 switches and supports processing nodes. In this topology, switches are in the level 0 and switches are in level 1. There is one link fro each switch in level 1 to each of the switch in level 0. In order to describe the oblivious routing algorith, we will give a non-recursive description of F T (, ). The top level switch are labeled switches (0, 0), (1, 0),..., ( 1, 0). The level 1 switches are labeled switches (0, 1), (1, 1),..., ( 1, 1). Each level 1 switch (i, 1), 0 i 1, is connected with processing nodes nubered as (i, 0), (i, 1),..., (i, 1). There is a link between switch (i, 0), 0 i 1 and switch (j, 1), 0 j 1. For 0 i 1, there is a link between processing node (i, x), 0 x 1, and switch (i, 1). Figure 6 depicts the F T (, ) topology as well as the switch and processing node labeling. Lea 10 states that any single path routing r, ERF (r) p p for F T (, ). To ease exposition, let us assue that is an integer. The cases when p is not an integer can be handled with soe inor odifications. We give the algorith that deals with the cases when p is not an integer in Figure 8. Following Lea 13, the optial oblivious routing algorith schedules the SD pairs such that the traffic in each up link fro a botto level switch to a top level switch has exactly p sources while each down fro a top level switch to a botto level switch has exactly p destinations. Note that in F T (, ), each of the level 1 links carries traffic either to 1 node or fro 1 node (the node that the link is connected to) and we do not have to consider such links in the design of the optial oblivious routing algorith. Let Z = p the. The routing algorith partitions = Z processing nodes in each botto level switch into Z groups, each group having Z nodes. More specifically, the processing nodes connected to switch (i, 1), 0 i 1, are partitioned into Z = p groups: group j, 0 j Z 1, includes nodes (i, j Z), (i, j Z + 1),..., (i, j Z + Z 1). The SD pairs are scheduled as follows. The up-link (i, 1) (j, 0), 0 i 1 and 0 j 1, carries SD pairs with sources nodes in group j/z in switch (i, 1) and destination nodes in group j od Z in all other switches. More specifically, The up-link (i, 1) (0, 0) carries traffic fro group 0 processing nodes (in switch (i, 1)) to group 0 processing nodes in other switches (i, 1) (1, 0) carries traffic fro group 0 processing nodes to group 1 processing nodes in other switches... (i, 1) (Z 1, 0) carries traffic fro group 0 processing nodes to group Z 1 processing nodes in other switches (i, 1) (Z, 0) carries traffic fro group 1 processing nodes to group 0 processing nodes in other switches... (i, 1) (Z 1, 0) carries traffic fro group 1 processing nodes to group Z 1 processing nodes in other switches... (i, 1) ((Z 1)Z, 0) carries traffic fro group Z 1 processing nodes to group 0 processing nodes in other switches... (i, 1) (Z 1, 0) carries traffic fro group Z 1 processing nodes to group Z 1 processing nodes in other switches. The traffic in the down link (j, 0) (i, 1), 0 i 1 and 0 j 1, which is fixed once the traffic in the up-link is decided, carries traffic with source nodes in group j/z in all other switches than switch (i, 1) to destination nodes in group j od Z in switch (i, 1). Hence, each of the up-links carries traffic fro exactly Z source nodes and each of the down links carries traffic to exactly Z destination nodes. The detailed routing algorith, called OSRM, is shown in Figure 8. When p is an integer, the algorith works exactly as just described. When p is not an integer, the algorith partitions the sources attached with each of the level 1 switch into Z s = p groups and the destinations into Z d = p groups. It then uses the sae logic as the cases when p is an integer to schedule the SD pairs. Theore 1: When p is an integer, ERF (OSRM) = p. roof: As discuss earlier, using OSRM, each link carries traffic either fro p sources or to p destinations. Fro Lea 13, ERF (OSRM) p. Fro Lea 10, ERF (OSRM) p. Hence, ERF (OSRM) = p and OSRM is an optial oblivious routing algorith for

9 Algorith OSRM: Route fro node (s 0, s 1) to node (d 0, d 1) Let = Let Z s =, Z d = Let N s = Z s, N d = Z d if (s 0 == d 0) use route: node(s 0, s 1) switch(s 0, 1) node(d 0, d 1) if (s 0! = d 0) use route: node(s 0, s 1) switch(s 0, 1) switch(s 1/N s N d + d 1/N d, 0) switch(d 0, 1) node(d 0, d 1) Figure 8: Optial oblivious routing for F T (, ) F T (, ) when p is an integer. Optial oblivious routing for F T (, 3) We will now consider F T (, 3). Fro Section, F T (, 3) contains three levels of switches, with the top level having nu(, ) = switches and each of the other levels having switches ( SUBF T (, ) s, each SUBF T (, ) having switches at each level). We label the switches by ((i 0, i 1), level): the top level switches are labeled as ((i 0, i 1), 0), 0 i 0 1 and 0 i1 1 the level 1 switches are labeled as ((i 0, i 1), 1), 0 i 0 1, and 0 i 1 1 the level switches are labeled as ((i 0, i 1), ), 0 i 0 1 and 0 i 1 1. Notice that in the switch labeling, for levels 1 and, i 0 identifies the coluns corresponding to the i 0-th SUBF T (, ) and i 1 identifies the colun corresponding to the i 1-th SUBF T (, 1) within the i 0-th SUBF T (, ). A F T (, 3) has processing nodes, which are labeled as (p 0, p 1, p ), 0 p 0 1, 0 p 1 1, and 0 p 1. A processing node (p0, p1, p) is attached to switch ((p 0, p 1), ), 0 p 0 1, 0 p 1 1, and 0 p 1. A level switch ((i0, i1), ), 0 i0 1 and 0 i 1 1, has a link to each of the level 1 switches ((i 0, X), 1), 0 X 1. A level 1 switch ((i0, i1), 1), 0 i 0 1 and 0 i 1 1, has a link to each of the level 0 switches ((i 1, X), 0), 0 X 1. Fro Lea 11, we can see that for any single path routing algorith r on F T (, 3), ERF (r). Like in the F T (, ) case, our optial routing algorith ensures that the SD pairs in each link are either fro at ost sources or towards at ost destinations. Fro roperty 5 in Section., each level 1 or level link in F T (, 3) carries traffic either fro no ore than sources or to no ore than destinations. Hence, routing within each SUBF T (, ) does not affect the perforance oblivious ratio. Hence, we only need to focus on level 0 links (the links connecting layer 0 and layer 1 switches). The idea is siilar to that in OSRM: the routing algorith ensures that each up link out of the sub-fat-tree SUBF T (, ) carries traffic fro sources and each down link to a SUBF T (, ) carries traffic to destinations. Basically, we can treat each SUBF T (, ) as if it is a ( ) -port switch that connects to ( ) processing nodes and has ( ) up-links. The routing algorith partitions the ( ) = Z processing nodes in a SUBF T (, ) into Z = groups, each group having Z = nodes. Node (p0, p1, p) is in group p of the p0-th SUBF T (, ). Algorith OSRM3: Route fro node (s 0, s 1, s ) to (d 0, d 1, d ): if (s 0 == d 0 and s 1 == d 1) /* within one SUBF T (, )*/ /* routing won t affect the oblivious ratio */ Use route: node(s 0, s 1, s ) switch((s 0, s 1), ) node(d 0, d 1, d ) else if (s 0 == d 0) /* within one SUBF T (, ) */ /* routing won t affect the oblivious ratio */ Use route: node(s 0, s 1, s ) switch((s 0, s 1), ) switch((s 0, s ), 1) switch((s 0, d 1), ) node(d 0, d 1, d ) else /* ust be careful about links to/fro level 0 switches */ Use route: node(s 0, s 1, s ) switch((s 0, s 1), ) switch((s 0, s ), 1) switch((s, d ), 0) switch((d 0, s ), 1) switch((d 0, d 1), ) node(d 0, d 1, d ) Figure 9: Optial oblivious single routing for F T (, 3) The routing for links between SUBF T (, ) and the top level switch is siilar to that for links between level 1 switches to level 0 switches in F T (, ): the up-link ((i 0, 0), 1) ((0, 0), 0) carries traffic fro group 0 processing nodes (in the i 0-th SUBF T (, ))to group 0 processing nodes in other SUBF T (, ) s ((i 0, 0), 1) ((0, 1), 0) carries traffic fro group 0 processing nodes to group 1 processing nodes in other SUBF T (, ) s... ((i 0, 0), 1) ((0, Z 1), 0) carries traffic fro group 0 processing nodes to group Z 1 processing nodes in other SUBF T (, ) s ((i 0, 1), 1) ((1, 0), 0) carries traffic fro group 1 processing nodes to group 0 processing nodes in other SUBF T (, ) s... ((i 0, 1), 1) ((1, Z 1), 0) carries traffic fro group 1 processing nodes to group Z 1 processing nodes in other SUBF T (, ) s... ((i 0, Z 1), 1) ((Z 1, 0), 0) carries traffic fro group Z 1 processing nodes to group 0 processing nodes in other SUBF T (, ) s... ((i 0, Z 1), 1) ((Z 1, Z 1), 0) carries traffic fro group Z 1 processing nodes to group Z 1 processing nodes in other SUBF T (, ) s. This way, each up-link only carries SD pairs with exactly Z = sources. Siilarly, each down link only carries SD pairs with exactly Z destinations. The detailed routing algorith, called OSRM3, is shown in Figure 9 Theore : ERF (OSRM3) = and OSRM3 is an optial oblivious routing algorith for F T (, 3). roof: Fro above discussion, using OSRM3, the SD pairs in each link have either at ost source nodes or at ost destination nodes. Fro Lea 13, ERF (OSRM3). Fro Lea 11, an perforance oblivious ratio of is the low bound for any single path routing schee on F T (, 3). Hence, OSRM3 is an optial oblivious routing algorith for F T (, 3). 4. MULTI-ATH OBLIVIOUS ROUTING In the previous section, it is shown that any single path routing would have at best a ( n 1 ) 3 oblivious perforance ratio on F T (, n). This indicates that single path