Traffic Pattern-based Adaptive Routing for Intra-group Communication in Dragonfly Networks

Transcription

1 Traffic Pattern-based Adaptive Routing for Intra-group Communication in Dragonfly Networks Peyman Faizian, Md Safayat Raman, Md Atiqul Molla, Xin Yuan Department of Computer Science Florida State University Tallaassee, FL Scott Pakin, Mike Lang Computer, Computational, and Stat. Sci. Div. Los Alamos National Laboratory Los Alamos, NM Abstract Te Cray Cascade arcitecture uses Dragonfly as its interconnect topology and employs a globally adaptive routing sceme called UGAL. UGAL directs traffic based on link loads but may make inappropriate adaptive routing decisions in various situations, wic degrades its performance. In tis work, we propose to improve UGAL by incorporating a traffic patternbased adaptation mecanism for intra-group communication in Dragonfly. Te idea is to explicitly use te link usage statistics tat are collected in performance counters to infer te traffic pattern, and to take te inferred traffic pattern plus link loads into consideration wen making adaptive routing decisions. Our performance evaluation results on a diverse set of traffic conditions indicate tat by incorporating te traffic pattern-based adaptation mecanism, our sceme is more effective in making adaptive routing decisions and acieves lower latency under low load and iger trougput under ig load tan te existing UGAL in many situations. I. INTRODUCTION Te Dragonfly topology features a cost-effective interconnect design and provides ig aggregate bandwidt for a diverse set of traffic patterns [1]. It as been deployed in te Cray Cascade arcitecture and is te interconnect topology in a number of current and near-term supercomputers. One unique caracteristic of te Dragonfly network is tat te routing performance is very sensitive to traffic pattern. To acieve ig performance, different routing scemes must be used for different traffic patterns [1]. For example, minimal routing () sould be used for uniform traffic wile Valiant Load Balance routing () sould be used for oter traffic patterns. To unify te two routing algoritms in one system, Universal Globally Adaptive Load-balanced routing (UGAL) [1] adapts te routing decision between and pats based on te link load information derived from queue lengt. Te teoretical UGAL wit perfect global link state information (), wic cannot be implemented in practice, performs similarly as for uniform traffic and as for adversarial traffic. Various scemes tat approximate te teoretical ave been developed [2]. An adaptive routing sceme tat makes routing decisions based on link loads fundamentally optimizes for network load balancing by distributing te traffic suc tat link loads on different links are similar. However, load balancing alone is insufficient for acieving ig performance in Dragonfly. can acieve load balancing for almost any traffic pattern, yet its performance for random uniform traffic is significantly lower tan. An adaptive routing sceme suc as UGAL tat makes routing decisions based solely on link/pat loads as inerent limitations as te traffic pattern is not directly considered. Recent studies ave sown tat UGAL makes inappropriate routing decisions in various situations, wic degrades its performance and tat te problem is more severe wit imprecise global link state information [3], [4]. In tis work, we propose to enance UGAL by explicitly incorporating a traffic pattern-based adaptation mecanism. Tis work focuses on intra-group communication; but te tecniques can be extended for inter-group communication. Te proposed sceme is motivated by several observations. First, Dragonfly is developed for HPC systems; and HPC applications often ave repetitive communication patterns tat can easily be identified at different levels [5], [6]. Second, modern routers for HPC systems maintain an extensive number of performance counters. Te statistics collected in tese counters can be used to infer traffic pattern. Finally, UGAL works well under many traffic conditions, but not all [3], [4]. Incorporating te traffic-pattern information in making routing decisions can alleviate te issues wit UGAL for te traffic conditions in wic it does not perform well. Our adaptive routing sceme maintains counters for local traffic in eac router, wic gives additional information about te current traffic pattern. Based on te inferred local and global traffic condition from te counters, our routing sceme identifies nine different situations in wic different biases toward and pats are appropriate. In tis way, te proposed sceme adapts traffic based on te current traffic pattern in addition to link loads. We performed extensive simulation wit a diverse set of traffic conditions. Te simulation results indicate tat (1) under low load, te proposed sceme performs better tan te teoretical UGAL wit perfect global link state information () in terms of packet latency; and (2) under ig load, te proposed sceme performs similar to and better tan a practical UGAL implementation in terms of aggregate bandwidt. Tus, te proposed sceme is a robust and igly effective adaptive routing sceme. Te rest of te paper is structured as follows. Section II describes te Dragonfly variation adopted in te Cray Cascade arcitecture and its routing scemes. Section III presents our

2 proposed routing algoritm. Section IV reports te results of te performance study. Te related work is presented in Section V. Finally, Section VI concludes te paper. II. BACKGROUND A. Cray Cascade Dragonfly topology Te Cray Cascade arcitecture employs Dragonfly as its interconnect topology. Te arcitecture as a fixed structure for eac group, but allows variable numbers of groups to form a system. Eac group is fully connected wit all oters using optical cables. Te intra-group topology is well-defined, but te total number of groups and te inter-group bandwidt (number of links between groups) are installation-specific. As tis paper is concerned only wit intra-group communication in te Cascade arcitecture, we will describe te Cascade intra-group topology. Details about te Dragonfly and Cascade arcitectures are provided elsewere [1], [7]. Every group in Cascade is formed by a pair of cabinets. Eac cabinet ouses tree cassis. Eac cassis contains 16 blades. Eac blade connects a single router and four processing elements. So in total, eac group contains 6 cassis, eac ave 16 blades wit a total of 96 routers. Te Cascade system uses Aries routers, wic are an application specific integrated circuit (ASIC) developed by Cray. Eac cassis backplane provides all-to-all connections among sixteen Aries routers. Eac router is also connected to five oter routers in te remaining five cassis witin te same group using electrical cables. Te inter-cassis connections are made wit corresponding slots. For example, a router in slot 1 in one cassis will ave direct links to te five slot 1 routers in te oter five cassis witin te same group. Eac inter-cassis link is equivalent to tree intra-cassis links in terms of bandwidt. An Aries router as a total of 48 ports: 8 ports for local processing nodes, 15 ports connecting to 15 routers in te same cassis, 15 ports to 5 routers in te same slot but different cassis, and 10 ports to oter groups. For intra-group communication, eac Aries router can ave 18 traffic end-points (8 for te local processing nodes and 10 for te ports connecting to oter groups). Figure 1 sows te intra-group topology of a Cascade group. Logically, a Cascade group consists of a 6 16 mes wit fully connected x and y dimensions. Eac pair in te same row is connected by one link wile eac pair in te same column is connected by tree links. B. Cascade routing In Cascade, packets are routed along eiter a minimal or a non-minimal pat. Witin a group, te minimal routing () is te sortest pat routing: a minimal pat eiter as 1 op wen te source and te destination are in te same column or row, or 2 ops oterwise. In Figure 1, te minimal pat from router S to router D as two ops. Te non-minimal routing is te Valiant s Load Balance routing (), wic spreads non-uniform traffic evenly over te set of available links in te system. Non-minimal routes (or routes) in a group can be considered as using to find a pat from Fig. 1. Cray Cascade intra-group topology te source to a randomly selected intermediate router and ten from te intermediate router to te destination. For example, in Figure 1, a pat from S to D can go troug intermediate router R, resulting in a 4-op pat. Cascade supports packet-by-packet adaptive routing wit te Universal Globally Adaptive Load-balanced (UGAL) routing sceme. Te routing pipeline selects up to four possible routes at random, two minimal and two non-minimal routes and decides te pat for te packet based on estimated link loads wic are computed using a combination of downstream link load, estimated far-end, and near-end link load. UGAL selects te pat wit te ligtest load. Some details about Cascade UGAL are undisclosed, suc as ow frequently te downstream load information is propagated and ow exactly te loads are estimated and computed. In tis work, we consider two versions of UGAL: UGAL wit perfect global information () and UGAL wit local information () [1]. is not a practical routing sceme since it assumes tat te precise load on eac pat in te entire network is known, but it represents te performance upper bound tat any UGAL-type routing sceme can acieve. uses te queue lengt on te local router to approximate te load on te pat, and can be implemented in practice. and coose between and VAL routes for eac packet, based on te pat load information available at te source router. selects te pat wit te smallest accumulated queue lengt on all links along te pat. considers te queue lengt (Q) in te local router and te op counts (H) of te pats and evaluates te following inequality: Q min H min Q vlb H vlb + T (1) in wic Q min is te queue lengt on te local cannel for te pat; H min is te op count for te pat; Q vlb is te queue lengt of te non-minimal pat; H vlb is te op count of te non-minimal pat; and T is an offset constant tat can be tuned to decide ow muc te pat selection sould be biased toward pats (wit a large value of T giving preference to pats). If te inequality olds, UGAL decides tat te

3 minimal pat is less congested tan te non-minimal pat and sould be selected to route te packet. Tese link load based adaptive routing mecanisms cannot reac te observed performance of under uniform random traffic and under worst-case adversarial traffic because tey try to adapt to te network congestion only based on available link load information. As revealed in recent studies [3], [4], tese adaptive routing scemes suffer from te following limitations. Fluctuations in queue lengts due to temporary loadimbalance can result in suboptimal routing decisions made by te adaptive routing sceme, even wen te overall traffic pattern remains uncanged. Longer queue lengts can occur for multiple reasons, including ig load or te adversarial nature of te traffic. But tese routing scemes treat all tese situations alike by coosing te least loaded pat, wic is not always an optimal coice. Te performance of UGAL ( or ) depends on te congestion offset (te T in Equation 1), wic needs to be tuned empirically based on te traffic pattern. Fixing its value represents a trade-off: te routing will favor eiter minimal pats, wic would degrade te routing performance for adversarial traffic, or nonminimal pats, wic would result in low performance for uniform traffic. Tere does not exist a single value for T tat can acieve ig performance under all traffic conditions. III. TRAFFIC PATTERN-BASED ADAPTIVE ROUTING Our sceme is built upon te idea tat assuming a UGALlike routing sceme tat selects between and pats, eac router can observe te traffic tat passes troug te router and infer useful traffic pattern information to make better routing decisions. Wit a UGAL-like routing sceme, even wen a router is not on te minimal pat of a communication, it can still observe te communication, as te router can be selected as an intermediate router wen a non-minimal pat is used. Note tat te proposed adaptive routing algoritm is muc like UGAL except tat it uses traffic-pattern information to make better routing decisions tan conventional UGAL. To obtain te traffic-pattern information, our sceme maintains a number of counters and infers te local and global pattern information from te counters. Te Cray Aries provides an extensive set of performance counters so maintaining te counters needed by our sceme is not an issue on tat platform. By explicitly inferring and using te traffic pattern information, custom adaptive routing mecanisms can be tailored for te observed traffic patterns. Our approac improves UGAL as described in Section II-B by using local and global traffic-pattern information to differentiate nine cases for setting te offset value (T ) in UGAL or to use or were needed. In oter words, te adaptive routing mecanisms used in our sceme range from pure routing to wit different biases to pats to pure routing. Eac routing mecanism is selected based on inferred-traffic pattern information. In te following text, we first introduce te counters used in our routing sceme. We ten discuss ow te traffic pattern is inferred. Finally, we present our traffic pattern-based adaptive-routing algoritm. A. Traffic counters In eac router, our sceme maintains te following counters, wic count te number of packets during a window of time (e.g., te last cycles): DestC i : Tese counters record te number of packets sent to te router i from local compute nodes during te window period. Port tr i : For eac port i in te router, Port tr i records te number of troug packets (packets originated from oter routers) tat use tat port. Using tese counters, useful local and global traffic-pattern caracteristics can be derived. To support intra-group communication in Cascade, eac router must ave 96 DestC i counters and 30 Port tr i, wic is not significant relative to te number of performance counters provided by te Aries router. If te tecnique is to be extended to inter-group routing, some aggregation sceme must be used to reduce ardware overead. One natural option is to maintain a counter for eac destination group instead of for eac destination router. B. Quantifying local traffic pattern Local traffic consists of packets generated from te endpoints attaced to te router. From earlier studies of Dragonfly, it is known tat te traffic is benign wen it spreads evenly among all possible destination routers (uniform traffic), and adversarial wen te traffic concentrates on a small number of destination routers. To make effective routing decisions, it is important to know ow adversarial or ow uniform te local traffic pattern is. Suc information can be obtained wit te DestC i counter. Te basic idea is tat if te local traffic is uniform, te values in DestC i are similar and relatively small. If te local traffic is adversarial, te values in DestC i will exibit a small number of spikes (implying traffic is concentrated on a small number of destination routers) wose values can be very large, depending on te injection rate. In our sceme, we quantify local traffic to eac destination switc as benign or adversarial by examining DestC i. We first consider an experiment wit te Booksim simulation infrastructure [8]. In tis experiment we simulate Cascade s 96 intra-group routers wit eac router connecting to 18 traffic endpoints. Te routing algoritm is. Te observation window size is cycles. Te traffic pattern is eiter uniform random (UR) or an adversarial sift pattern (ADV) in wic all traffic from endpoints on router i is sent to endpoints on router i+1. We observe te counter values (DestC 1 ) in router 0. Te results are sown in Table I. Te injection rate is te number of packets generated per traffic endpoint per cycle. As can be seen from te table, for uniform traffic, te number of packets sent to eac destination is muc smaller tan te number of packets sent to te same destination under adversarial traffic and same injection rate. Eac Cascade

4 group as 96 routers. Wit uniform traffic, eac router sould receive 1/95 of te total local traffic. On te oter and, for te worst case adversarial traffic, all of te local traffic are sent to one destination router. As suc, differentiating tese two types of traffic to eac destination can be done by examining DestC i, were is te window size. We call tis value localimpact. TABLE I THE COUNTER VALUES FOR UNIFORM AND ADVERSARIAL TRAFFIC (=) Injection rate Pattern DestC 1 localimpact UR ADV 1. 4 UR ADV UR 8 6 ADV Assuming tat at most one packet can be generated from a processing node per cycle, te value for localimpact can range from 0 to p, were p is te number of processing nodes attaced to eac router. As can be seen from Table I, te values for localimpact differ significantly for te benign traffic (UR) and te adversarial traffic (ADV). For UR traffic wit 0.9 injection rate (very eavy load), te localimpact value is only 6 wile for ADV traffic wit injection rate (not eavy load), te localimpact value is 1.. Hence, localimpact is a good indicator for te local traffic pattern. In our sceme, we classify te local traffic to eac destination router into tree types, benign, mixed, and adversarial using two tresold values low l and ig l as follows. Te local traffic to destination router i is deemed benign wen localimpact = DestC i < low l, adversarial wen localimpact > ig l, and mixed wen low l localimpact ig l. In our experiments, we set low l = 0.8 and ig l = 4. Tese tresolds sould be tuned in an operational system. C. Quantifying global traffic pattern Global traffic consists of packets generated by endpoints connected to oter routers. For te global traffic pattern, te most important information for making routing decisions is ow te global traffic would affect te local router. Te impact of global traffic on a router can be quantified by te amount of troug traffic on eac port of te current router, wic is te Port tr i counter. Table II sows te average troug traffic on eac port of a router under uniform random (UR) and adversarial traffic (ADV) wit different loads. Te simulation setting is te same as tat used in Table I. As can be seen from te table, tere are distinguisable differences between te troug traffic counts for uniform and adversarial traffic. In particular, very large Port tr i (e.g. > 30) is only observed for adversarial traffic wit eavy loads. Our sceme uses Port tr i to quantify te global traffic on a particular port (port i) of a router and we call tis value globalimpact. Te sceme also classifies te global traffic on te port to be benign, mixed, and adversarial using two TABLE II THE AVERAGE Port tr i VALUES FOR UNIFORM AND ADVERSARIAL TRAFFIC (=) Injection rate Pattern Port tr i globalimpact UR ADV UR ADV UR 2 1 ADV tresold values low g and ig g. Te global traffic on port i is deemed benign wen globalimpact = Port tr i < low g, adversarial wen globalimpact > ig g, and mixed wen low g globalimpact ig g. In our experiments, we set low g = 3 and ig l = 0.6. Tese parameters can be tuned for a particular system. As discussed earlier, an adversarial global traffic pattern as defined implies a mostly adversarial global traffic pattern wit eavy load. On te oter and, benign or mixed global-traffic patterns can be interpreted as eiter uniform (wit a iger load) or adversarial global traffic patterns (wit a lower load). In te next sub-section, we will discuss ow te local traffic pattern information can be used along tis global traffic information to improve routing effectiveness. D. Te routing algoritm Let p be te packet to be routed. We will denote src(p) as te source router for te packet, dst(p) as te destination router, P(p) as te local output port in te source router for te pat. Let loc(p) be te localimpact factor at src(p) for p: loc(p) = DestC dst(p). Let glo(p) be te globalimpact factor at src(p) for p, wic is te globalimpact for te local output port for te pat: glo(p)= Port tr P(p). Te routing decision is made at src(p) tat as all of te needed information. For eac packet p, te source router will decide te pat based on te current traffic pattern and te link loads. Te current traffic condition is represented by loc(p) and glo(p). Based on values of loc(p) and glo(p), te routing algoritm operates in nine operating regions, eac wit a different routing mecanism. Te routing mecanisms used in our algoritm range from te pure routing, different routing scemes wit different offset values T, and te pure routing. is an effective adaptive routing algoritm wose performance is largely influenced by te offset value. Wit a large offset value, te routing favors minimal pats [1]. By using a range of offset values, progressively decreasing (from +64 to 64), progressively biases towards te pats. We will use te notion UGAL(T) to denote UGAL- L wit offset T, were T may be positive or negative. By using te traffic pattern information derived from te counters, te proposed routing sceme can identify different operating regions and tune te offset value accordingly to elp perform better. Te routing used in our sceme, wic

5 we call pat-lengt oblivious, is sligtly different from te conventional. As discussed in Section II-B, in te conventional routing, to select a non-minimal pat, load estimation multiplied by te op count of te pat is used as te metric to decide te pat load. As suc, te traditional routing favors sorter pats since sorter pats ave smaller op counts. In our sceme, is only used for te extreme adversarial condition. In suc a condition, preferring sorter non-minimal pats does not perform as well as pat-lengt oblivious were only load estimation is used (witout multiplying op count) to decide te pat load. In making a routing decision, te routing algoritm considers up to four pats: two pats and two pats. It determines te less loaded pat and te less loaded pat, ten selects between te two pats according to te traffic condition, as sown in Table III. Consider for example, te tird row and tird column were bot loc(p) and glo(p) are benign: tis identifies largely uniform traffic or very low load adversarial traffic wen acieves ig performance. In tis case, our algoritm eiter selects or use UGAL- L wit a very large offset (64) tat strongly favors te pat. Te mecanism to differentiate tese two routing scemes in te region is to ceck weter te combined local and global traffic exceeds a tresold. Specifically, wen loc(p)+glo(p)<tr, is used; oterwise UGAL(64) is used. Similar mecanism is also used for te case wen loc(p) is benign and glo(p) is mixed. Wen bot local traffic and global traffic are adversarial, pure routing is used. From te table, te following can also be observed. First, for te same local traffic (log(p)), as te global traffic becomes more adversarial, te algoritm biases more towards a non-minimal pat wit a decreasing offset value for. Second, for te same global traffic (glo(p)), as te local traffic becomes more adversarial, te algoritm biases more towards a nonminimal pat. TABLE III LOCAL AND GLOBAL TRAFFIC PATTERNS AND THE CORRESPONDING ROUTING MECHANISM FOR PACKET p loc(p) benign mixed adv. benign /UGAL(64) UGAL(-4) UGAL(-48) glo(p) mixed /UGAL(64) UGAL(-20) UGAL(-64) adv. UGAL(48) UGAL(-) Te effectiveness of te proposed sceme can be furter refined by using smaller categories of local and global traffic and tailoring te offset value for eac category. However, as will be sown in te next section, our algoritm is already more effective, sometimes significantly, tan te traditional UGAL- L sceme. Note tat like, our sceme only uses local information to make routing decisions. Tis demonstrates tat using traffic pattern to filter situations and facilitate different adaptive mecanisms is an effective approac for improving routing performance. A. Simulation metodology IV. PERFORMANCE STUDY Te proposed sceme as been implemented in Book- Sim [8], an interconnection network simulator, wic is used to evaluate te routing algoritms in tis study. Te experiments were designed to investigate te intra-group performance of different routing scemes on a single group of te Cray Cascade arcitecture [7]. Te network includes 96 routers, eac aving 18 traffic end points as described in Section II. Te simulation configurations are similar to Jiang et al. s [2]. We assumed single-flit packets and a 2x speedup for router crossbar over network links. Te latency of eac network link is set to 10 cycles. To avoid deadlocks, we used 4 VCs wit a 32-entry buffer size. For eac data point, te network was warmed up for 10,000 cycles, and network statistics were collected for,000 cycles. B. Routing scemes We compare te performance of 5 routing scemes under various traffic patterns. Tis includes,, UGAL- L (UGAL wit local information), (te teoretical UGAL wit global information), and (our proposed traffic pattern-based routing). In te windows size for te counters is 128 cycles. All adaptive routing decisions are made at te source router after a packet is injected into te network. C. Traffic patterns Four different families of traffic patterns are used in te evaluation. To examine te extreme case performance, we used a uniform-random traffic pattern (UR) and an adversarial sift traffic pattern (ADV). Wit UR, te probability of sending a packet to eac destination is equal, wereas wit ADV, eac node connected to a given router i sends all of its traffic to nodes connected to router i+1. In addition, two families of mixed traffic patterns are considered by combining UR and ADV traffic patterns at processing node or router levels. In node-level combined traffic patterns, eac packet is sent as eiter UR traffic or ADV traffic wit a certain probability. Tus, te traffic from eac router contains bot UR and ADV components. We will use te notation NLC URADV(UR%, ADV%) to represent te node-level combined traffic patterns, were UR% is te percentage of UR traffic and ADV% is te percentage of ADV traffic. For example, in NLC URADV(20,), eac node sends UR traffic wit 20% probability and ADV traffic wit % probability. Te oter mixed traffic pattern is te router-level combined traffic pattern. In tis case, all processing nodes connected to a specific router ave te same traffic pattern (eiter UR or ADV). However, nodes at different routers may ave different traffic patterns. We will use te notation RLC URADV(UR%, ADV%) to represent te router-level combined traffic patterns. For example, in RLC URADV(20,), 20% of te routers generate UR traffic and % of te routers generate ADV traffic.

6 Fig. 2. Uniform Random traffic (UR) Fig. 3. Adversarial sift traffic (ADV) D. Results 1) Uniform random traffic: Figure 2 sows te latencytrougput results for UR traffic. As expected, acieves te best trougput and te lowest latency among all evaluated routing scemes due to te load-balanced nature of te UR traffic pattern. acieves approximately alf of te minimal routing trougput and wit a iger latency because wit, eac packet uses twice te network resources as. acieves 92% of trougput under UR, albeit at a iger latency, wic stems from te fact tat it routes 10 38% of te traffic non-minimally (38% under low load). Tis is due to te fluctuation of te queue lengt even wit te UR traffic, wic Won et al. also observed [4]. Even wit te perfect global information, in comparison to, still as noticeable iger latency at low load and acieves 98% of trougput. By incorporating trafficpattern based adaptation mecanism, minimizes te inappropriate decisions to route non-minimally by distinguising UR traffic pattern early on. For all traffic injection rates, routes no more tan 1% of te traffic troug pats and acieves almost identical trougput and latency as. 2) Adversarial traffic: Results for ADV traffic are sown in Figure 3. fails to balance te load for tis traffic, and, as a result, te network saturates at a very low injection rate. In contrast, distributes te traffic evenly among all intermediate nodes and acieves % of te teoretically acievable trougput. starts by sending 28% of te traffic troug minimal pats under low load and gradually decreases tis amount to as low as 1% at te saturation point. reaces rougly 10% iger trougput tan UGAL- L by exploiting te global information and keeps sending about 12% of te traffic minimally even at ig loads. uses minimal routing under very low load wic explains very low latency at 0.05 injection rate. Ten as te load increases, localimpact and globalimpact increase as well. Terefore, uses smaller offsets wile using to balance te adversarial load and eventually switces to our pat-oblivious at very ig load. Tis results in a lower latency for at ig load in compare to and, as well as a iger trougput tan and traditional, wic is mainly due to te adoption of pat oblivious. 3) Node-level combined traffic: Figures 4, 5, and 6 sow te results for tree types of node-level combined traffic: NLC URADV(,), NLC URADV(20,), and NLC URADV(,20), respectively. Note tat NLC URADV(20,) means tat te traffic is 20% uniform and % adversarial. Tese simulation studies are designed to test te performance of te proposed routing sceme under mixed and more complex traffic patterns. As can be seen from te figures, in all cases, outperforms in terms of trougput and latency. Also acieves up to 25% decrease in latency in compare to under low loads in all different cases. In Figure 6, under low load (injection rate < ), routes at least 99% of te traffic using minimal pats, in contrast to, wic sends less tan 71% using minimal pats. Tis explains te lower latency. As te load increases and minimal pats get congested, swiftly sifts towards using wit smaller offset values. Tis results in a 4% increase in trougput and 9% improvement in latency in comparison to. Figures 5 and 6 also verify tat correctly distinguises te local and global traffic patterns and cooses te appropriate pats to route te traffic accordingly. Tis creates a noticeable gain in terms of latency, even relative to (up to 25% improvement). also acieves about 98% of te trougput performance of. 4) Router-level combined traffic: For all traffic patterns presented so far, every router in te network as te same beavior. Wit router-level combined traffic (RLC), traffic from different routers may ave different caracteristics. Te network is logically partitioned into two interleaving parts wit eac part aving a different traffic pattern. Tis creates an imbalance between locally generated traffic and troug traffic observed at eac router, wic elps us to understand if our routing sceme can andle cases were different parts of te network beave differently. Figures 7, 8, and 9 sow te results for tree types of router-level combined traffic: RLC URADV(,),

7 Fig. 4. NLC URADV(, ) Fig. 6. NLC URADV(, 20) Fig. 5. NLC URADV(20, ) Fig. 7. RLC URADV(, ) RLC URADV(20,), and RLC URADV(,20), respectively. For RLC, a router will see te local traffic pattern (e.g. UR) to be different from te global traffic pattern (e.g. ADV). Under suc a condition, is able to make appropriate routing decisions by correctly identifying te local and global traffic impacts as sown in all tree figures. As sown in Figure 8, except wen injection rate = 0.05, global traffic is known to be mixed or adversarial. Terefore, te routing decision can be taken wit regards to te local traffic impact. If te generated traffic is uniform (20% of te routers), a large offset value will be used to prefer minimal pats, wereas for oter routers wic are generating adversarial traffic, a very small offset value is selected. Regardless of te traffic combination, always performs better tan (and very close to ) by identifying te traffic pattern at eac router and routing traffic accordingly. Tis demonstrates tat traffic pattern-based routing can be effective even wen local traffic is very different from global traffic. V. RELATED WORK Since te time wen te Dragonfly interconnection network was first introduced, it as been clear tat a globally adaptive routing sceme is needed to overcome te traffic pattern dependent sortcomings of minimal and non-minimal routing scemes. In te seminal work by Kim et al [1], te autors proposed selecting a random intermediate group to route nonminimally in order to load-balance adversarial traffic patterns over global cannels. Jiang proposes several adaptive routing euristics tat approximate UGAL wit global link state information [2]. Garcia et al. [9] were te first to address local congestion inside Dragonfly groups and proposed allowing non-minimal routing on bot intra- and inter-group communication in teir OFAR routing sceme. OFAR-CM [10] proposes trottling packet injection at local node as well as routing troug an escaping subnetwork to mitigate congestion on OFAR routing at te cost of additional ops. Opportunistic Local Misrouting (OLM) proposed in [11] allows non-minimal routing on bot local and global levels of te Dragonfly ierarcy and te routing decision may be updated at any op. Kim et al. [4] analyzed te effect of far-end congestion in Dragonfly network. Te autors sowed tat because of link latency, te load estimation at te current router is inaccurate, wic leads to errors in adaptive routing decisions. Tey proposed a istory-window based approac wic keeps track of te number of in-fligt packets for a given window of time, and uses tat information to eliminate te problem. Te

8 Fig. 8. RLC URADV(20, ) Fig. 9. RLC URADV(, 20) paper also sows tat te local queue lengt at any router can fluctuate very rapidly depending on te network traffic, so te lengt of te queue of a specific cannel at a particular moment may not accurately represent te load. Fuentes et al., [3] observed tat routing scemes based on queue lengt information suffer from a number of limitations, including slow response time and frequent oscillation of coice between minimal and non-minimal pats. As an alternative, te autors propose contention based routing sceme were eac output port is equipped wit a contention level counter tat is used in lieu of queue occupancy to make routing decisions. We believe tat not taking traffic pattern into consideration is a fundamental issue wit all link load-based adaptive routing scemes, and propose to address te issue by explicitly estimating te current traffic pattern. VI. CONCLUSION Existing adaptive routing scemes for te Dragonfly topology make adaptive decisions based on link loads and do not consider te traffic pattern on te network tat plays an important role in effective routing in tis topology. In tis work, we proposed a traffic-pattern-based adaptive routing sceme and sowed tat by incorporating te traffic-pattern based adaptive sceme, a more effective adaptive routing sceme for intragroup communication in Dragonfly can be obtained. Since te Dragonfly inter-group connectivity bears significant similarity wit its intra-group connectivity, an effective routing sceme for intra-group communication can naturally be extended for inter-group communication. We are developing pattern-based adaptive routing scemes for inter-group communication. ACKNOWLEDGEMENTS Tis material is based upon work supported by te U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Researc, under Award Numbers and DE-SC REFERENCES [1] Jon Kim, Wiliam J Dally, Steve Scott, and Dennis Abts. Tecnology- Driven, Higly-Scalable Dragonfly Topology. In ACM SIGARCH Computer Arcitecture News, volume 36, pages IEEE Computer Society, [2] Nan Jiang, Jon Kim, and William J. Dally. Indirect adaptive routing on large scale interconnection networks. SIGARCH Comput. Arcit. News, 37(3): , June [3] P. Fuentes, E. Vallejo, M. Garcia, R. Beivide, G. Rodrguez, C. Minkenberg, and M. Valero. Contention-based nonminimal adaptive routing in ig-radix networks. In Parallel and Distributed Processing Symposium (IPDPS), 2015 IEEE International, pages , May [4] J. Won, G. Kim, J. Kim, T. Jiang, M. Parker, and S. Scott. Overcoming far-end congestion in large-scale networks. In Hig Performance Computer Arcitecture (HPCA), 2015 IEEE 21st International Symposium on, pages , Feb [5] Gregory Jonson, Darren J. Kerbyson, and Mike Lang. Optimization of InfiniBand for scientific applications. In Proceedings of te 2008 IEEE International Symposium on Parallel and Distributed Processing, pages 1 8. IEEE Computer Society Press, [6] Xin Yuan, R. Melem, and R. Gupta. Compiled communication for alloptical TDM networks. In Supercomputing, Proceedings of te 1996 ACM/IEEE Conference on, pages 25 25, [7] Greg Faanes, Abdulla Bataine, Duncan Rowet, Edwin Froese, Bob Alverson, Tim Jonson, Joe Kopnick, Mike Higgins, James Reinard, et al. Cray Cascade: A Scalable HPC System Based on a Dragonfly network. In Proceedings of te International Conference on Hig Performance Computing, Networking, Storage and Analysis, page 103. IEEE Computer Society Press, [8] Nan Jiang, Daniel U. Becker, George Micelogiannakis, James D. Balfour, Brian Towles, David E. Saw, Jon Kim, and William J. Dally. A detailed and flexible cycle-accurate network-on-cip simulator. In ISPASS, pages IEEE Computer Society, [9] M. Garcia, E. Vallejo, R. Beivide, M. Odriozola, C. Camarero, M. Valero, G. Rodrguez, J. Labarta, and C. Minkenberg. On-tefly adaptive routing in ig-radix ierarcical networks. In Parallel Processing (ICPP), st International Conference on, pages , Sept [10] M. Garcia, E. Vallejo, R. Beivide, M. Valero, and G. Rodrguez. OFAR- CM: Efficient Dragonfly networks wit simple congestion management. In Hig-Performance Interconnects (HOTI), 2013 IEEE 21st Annual Symposium on, pages 55 62, Aug [11] M. Garcia, E. Vallejo, R. Beivide, M. Odriozola, and M. Valero. Efficient routing mecanisms for Dragonfly networks. In Parallel Processing (ICPP), nd International Conference on, pages , Oct 2013.