Expeditus: Congestion-Aware Load Balancing in Clos Data Center Networks

Transcription

1 Expeditus: Congestion-Aware Load Balancing in Clos Data Center Networks Peng Wang, Hong Xu, Zhixiong Niu, Dongsu Han, Yongqiang Xiong ACM SoCC 2016, Oct 5-7, Santa Clara

2 Motivation Datacenter networks rely on multipathing to provide large bisection bandwidth for diverse set of apps (web search, big data analytics, etc) Efficient load balancing is needed to improve network utilization and app performance 2

3 Motivation Current industry standard: ECMP load balancing Hash flow s 5-tuple to select outgoing link Stateless and oblivious to link utilization Agnostic to congestion s 4 s 5 ECMP blindly put a flow on a congested path 1Gbps s 1 s 2 s 3 3

4 Motivation State-of-the-art load balancing: CONGA (SIGCOMM 14) Track end-to-end path-wise congestion information Collect information using data packet piggybacking s 4 s 5 Dst Leaf S2 S3 path 1 2 Congestion-to-Leaf s 1 s 2 s 3 Src Leaf S1 S2 path 1 2 Congestion-from-Leaf 4

5 Motivation State-of-the-art load balancing: CONGA (SIGCOMM 14) Track end-to-end path-wise congestion information Collect information using data packet piggybacking s 4 s 5 Dst Leaf S2 S3 path 1 2 Congestion-to-Leaf s 1 s 2 s 3 Src Leaf S1 S2 path Congestion-from-Leaf 5

6 Motivation State-of-the-art load balancing: CONGA (SIGCOMM 14) Track end-to-end path-wise congestion information Collect information using data packet piggybacking s 4 s 5 Dst Leaf S2 S3 path 1 2 Congestion-to-Leaf s 1 s 2 s 3 Src Leaf S1 S2 path Congestion-from-Leaf 6

7 Motivation State-of-the-art load balancing: CONGA (SIGCOMM 14) Track end-to-end path-wise congestion information Collect information using data packet piggybacking s 4 s 5 Dst Leaf S2 S3 path 1 2 Congestion-to-Leaf s 1 s 2 s 3 Src Leaf S1 S2 path Congestion-from-Leaf 7

8 Motivation State-of-the-art load balancing: CONGA (SIGCOMM 14) Track end-to-end path-wise congestion information Collect information using data packet piggybacking s 4 s 5 Dst Leaf S2 S3 path Congestion-to-Leaf s 1 s 2 s 3 Src Leaf S1 S2 path Congestion-from-Leaf 8

9 Motivation Scalability issue of CONGA Designed for 2-tier network, not scalable for 3-tier network (much more paths) Simple per-path feedback needs many concurrent flows to cover all paths in a large-scale network 9

10 Challenges Massive congestion information on all paths In a typical 3-tier Clos network, hundreds of paths exist between any pair of ToR ; A ToR switch can communicate with thousands of other core plane 1 core plane 4 core 1 m 1 40G links m m=12 s 4 m aggregation n = 4 f p 4 p=96 ToR 1 2 r 1 2 r t p r r=48 pod 1 pod p Facebook datacenter fabric network topology 10

11 Challenges Massive congestion information on all paths In a typical 3-tier Clos network, hundreds of paths exist between any pair of ToR ; A ToR switch can communicate with thousands of other core plane 1 core plane 4 core 1 m 1 40G links m m=12 s 4 m Each ToR switch need to track millions of n paths = 4 aggregation f p for all 4 p=96 destination ToRs! ToR 1 2 r 1 2 r t p r r=48 pod 1 pod p Facebook datacenter fabric network topology 11

12 Expeditus 12

13 Our Idea Aggregate congestion information in two stages ToR-aggregation core 1 m 1 m aggregation ToR 1 r src 1 r dst 13

14 Our Idea Aggregate congestion information in two stages ToR-aggregation Aggregation-core core 1 m 1 m aggregation ToR 1 r src 1 r dst 14

15 Our Idea Aggregate congestion information in two stages ToR-aggregation Aggregation-core core 1 m 1 m aggregation ToR 1 r src 1 r dst 15

16 Overview One-hop congestion information collection Local congestion monitoring for all uplinks at each ToR and aggregation switch Two-stage path selection Select path tier-by-tier in 3-tier Clos network 16

17 Information collection Switches locally monitor egress and ingress link utilization for all uplinks 1 m 1 m core aggregation 1 r 1 r ToR 17

18 Overview One-hop congestion information collection Do local congestion monitoring for all uplinks at each ToR and aggregation switch Two-stage path selection Select path tier-by-tier in 3-tier Clos network 18

19 Stage #1 Start when the first packet (Exp-request) of flow reaches source ToR switch. 1 m 1 m core aggregation 1 r 1 r ToR S Y N source ToR t 1 1 dest ToR t 3 1 Exp-request

20 Stage #1 Source ToR switch inserts egress link utilisation into Exprequest packet 1 m 1 m core aggregation 1 r 1 r ToR S Y N source ToR t 1 1 dest ToR t 3 1 Exp-request

21 Stage #1 Source ToR switch inserts egress link utilisation into Exprequest packet Exp-request S source ToR 1 Y m 1 egress info m N core aggregation 1 r 1 r ToR source ToR t 1 1 dest ToR t

22 Stage #1 Destination ToR switch uses the maximum load of two hops as effective congestion, and chooses aggregation switch ID with the minimum effective congestion Exp-request 1 m S source ToR Y egress info N 1 m core aggregation 1 r 1 r ToR source ToR t 1 1 dest ToR t 3 1 S Y N source ToR egress info + dest ToR best aggr ID ingress info (say 4)

23 Stage #1 Destination ToR switch uses the maximum load of two hops as effective congestion, and chooses aggregation switch ID with the minimum effective congestion Exp-request 1 m S source ToR Y egress info N 1 m core aggregation 1 r 1 r ToR source ToR t 1 1 dest ToR t 3 1 S Y N source ToR egress info + dest ToR best aggr ID ingress info (say 4)

24 Stage #1 Destination ToR switch uses the maximum load of two hops as effective congestion, and chooses aggregation switch ID with the minimum effective congestion Exp-request 1 m S source ToR Y egress info N 1 m core aggregation 1 r 1 r ToR source ToR t 1 1 dest ToR t 3 1 S Y N source ToR egress info + dest ToR best aggr ID ingress info (say 4)

25 Stage #1 Destination ToR switch uses the maximum load of two hops as effective congestion, and chooses aggregation switch ID with the minimum effective congestion Exp-request 1 m S source ToR Y egress info N 1 m core aggregation 1 r 1 r ToR source ToR t 1 1 dest ToR t 3 1 S Y N source ToR egress info + dest ToR best aggr ID ingress info (say 4)

26 Stage #1 Destination ToR switch generates Exp-response packet and send it to the selected aggregation switch 1 m 1 m core a 3 4 chosen dest aggr 1 r 1 r Exp-response source ToR t 1 1 dest ToR t 3 1 Generate Exp-response: cp Exp-request s header, reverse src and dst addr 26

27 Stage #2 Chosen aggregation switch insert ingress link utilisation into Exp-response packet Exp-response dest aggr ingress info 1 m 1 m core a 3 4 chosen dest aggr 1 r source ToR t 1 1 dest ToR t r ToR 27

28 Stage #2 Similarly, source aggregation switch chooses core switch ID with the minimum effective congestion It further records selected core switch ID as path selection result Flow ID Egress Port 0xF0A3 1 Path Selection a 1 4 c 4 1 chosen core 1 m 1 m core a 4 4 chosen dest aggr Exp-response 1 dest aggr r + ingress info source aggr source ToR t 1 egress info 1 dest ToR t best core ID r (say 1) ToR 28

29 Stage #2 Source ToR switch records ingress port of Exp-response packet as path selection result Two-stage path selection is done! chosen core Exp-response c m 1 m core dest aggr ingress info a 1 4 a 4 4 chosen dest aggr Flow ID Egress Port 0xF0A3 4 1 r source ToR t 1 1 dest ToR t r ToR Path Selection t

30 Implementation - Click FromDevice (eth0) FromDevice (eth1) FromDevice (eth2) FromDevice (eth3) DRE DRE Prototype Expeditus in Click software router LookupIPRoute Output 0 EXPRoute Queue 0 Queue 1 Queue 2 Queue 3 DRE DRE Overhead for the new Click modules DRE (measure link utilization) ~ 151 ns/packet EXPRoute (two-stage path selection) ~ 473 ns/packet ToDevice (eth0) ToDevice (eth1) ToDevice (eth2) ToDevice (eth3) Packet processing pipeline for 4-pod fat-tree 30

31 Evaluation Schemes compared Expeditus, ECMP (baseline), CONGA (flowlet-level granularity), CONGA-Flow (flow-level granularity) Clairvoyant: ideal scheme that uses complete global congestion information to load balance flows (impractical in reality) Realistic traffic workloads: web search and data mining Performance metric: flow completion time (FCT) 31

32 Testbed Experiments Small-scale 3-tier Clos network with 4:1 oversubscription on real testbed 1Gbps NIC at & hosts 32

33 Testbed Results 99%ile tail for small flows (< 100KB) Web search Avg for large flows (>1MB) Web search 33

34 Large-scale Simulations NS-3 network simulator Topology 12-pod 10G fat-tree, 36 equal-cost paths, 864 hosts 10G Leaf-spine fabric with 128 hosts (8 leaf, 8 spine ) 34

35 Performance in 3-tier Clos 12-pod fat-tree with network oversubscription 2:1 at ToR tier 14%~32% 19%~29% Avg for all flows Web search Data Mining Although Expeditus misses some paths, it is still able to closely track Clairvoyant 35

36 Impact of link failure Reduction by Clairvoyant and Expeditus over ECMP (web search workload, load 0.5) Aggr-core link failure ToR-aggr link failure Expeditus still provides moderate performance gains in asymmetric scenarios

37 Comparison with CONGA Leaf-spine with network oversubscription 2:1 at leaf tier Avg for all flows There may not be enough concurrent flows to cover all paths at all times, which makes CONGA performs worse than Expeditus

38 Conclusion Expeditus: a novel data plane congestion-aware load balancing protocol for Clos data centre networks Includes simple local information collection and two-stage path selection mechanism Advances state-of-the-art by enabling congestion-aware load balancing in general 3-tier Clos topologies 38

39 Thank you! Peng Wang City University of Hong Kong 39

40 Backup slides 40

41 Motivation Local congestion-aware schemes Each switch balances load based on local congestion information Blind to congestion in downstream link 100Mbps s 4 s 5 1Gbps s 1 s 2 s 3 41

42 Per-pathlet Congestion Pathlet: two-hop segments between ToR and core Each ToR switch maintains congestion information at the granularity of pathlets Still require many concurrent flows to cover all flawless 1 m 1 m core aggregation 1 r 1 r ToR source ToR t 1 1 dest ToR t

43 Structural Property 1 m 1 m core aggregation 1 r 1 r ToR Pathwise connectivity: Aggregation switch of ID i only connects to aggregation of the same ID i in other pods 43