Circuit Switching Under the Radar with REACToR

Transcription

1 Circuit Switching Under the Radar with REACToR He Liu, Feng Lu, Alex Forencich, Rishi Kapoor Malveeka Tewari, Geoffrey M. Voelker George Papen, Alex C. Snoeren, George Porter 1

2 2

3 How to build 100G datacenter networks? 3

4 Bandwidth Required Datacenters Traffic Is Skewed 100G Destinations 4

5 Bandwidth Required 10G Fat-Tree 100G 10G Destinations 5

6 Bandwidth Required 100G Fat-Tree 100G Full Bi-sectional Bandwidth But Expensive cables, transceivers, Destinations 6

7 Bandwidth Required [SIGCOMM 2010] Helios, c-through: Hotspot Circuits 100G Single optical circuit Use mirrors Reconfigures in 10ms 10G 10G Electrical Fat-tree Destinations 7

8 Bandwidth Required [NSDI 2012] OSA: More Circuits 100G Several optical circuits Destinations 8

9 Bandwidth Required [SIGCOMM 2013] Mordia: Fast Circuit Switching 100G Reconfigures in 15 microseconds. Enables time sharing a circuit. Destinations 9

10 Bandwidth Required Limitation: Still Circuit Switching 100G Good with a few big flows A B A B t Destinations 10

11 Bandwidth Required Limitation: Still Circuit Switching 100G 50G Destinations 11

12 Bandwidth Required Limitation: Inefficient with Small Flows 100G Low link utilization 50G A B C D E F A B C D E F Reconfiguring takes time t Destinations 12

13 Bandwidth Required Our Approach: REACToR 100G 100G Circuit Switching 100G Hybrid of Circuit and Packet A B A B t 10G C D E F C D E F C D E F C D E F C D E t 10G Packet Switching Destinations 13

14 Start with a Pre-existing 10G Network 10G Electrical Packet Network 10G links ToR Box 10G links End hosts 14

15 Connect via REACToR 10G Electrical Packet Network 10G links REACToR 100G links End hosts 15

16 Connect Circuits 10G Electrical Packet Network 100G Optical Circuit Network 10G links 100G links REACToR 100G links End hosts 16

17 Hybrid Network 10G links 100G links REACToR Small Flows Big Flows 100G links End hosts 17

18 Challenge: Two Different Networks Electrical Packet Low bandwidth Buffers all the way Tx at any time Optical Circuit High bandwidth Bufferless TDMA Tx only when circuit connects 18

19 Design Requirements Hybrid scheduling: classify traffic into circuits or packets Buffer packets at source hosts until circuit is available Have sources transmit when the circuit is connected Rate control to prevent downlink overload 19

20 Bandwidth Required The Hybrid Scheduling Problem Centralized TDMA Circuit Schedule Free-running Packet Network Destinations Collect traffic demand from all hosts TDMA schedule the big flows on the circuit path Schedule the rest on the packet path An oracle predicts the demand and builds the schedules. 20

21 End Host: Classify and Buffer Packets Circuit Queues B A Packet Queue C D E F Classify packets and map into different hardware queues Based on the schedule Packet path: one hardware queue for all destinations Can transmit at any time, but at 10G Circuit path: one hardware queue for each destination Can only transmit when the particular circuit is connected Buffer the packets in end-host memory 21

22 Packet Transmission Packet path: Rate limit to 10G Circuit path: Transmit only when the circuit is connected Circuit to Host A B REACToR PFC: Pause Unpause A A B NIC Packet path End Host REACToR pulls packets from the circuit queue in real-time Use PFC frames to selectively unpause queues Circuit Queues A B Packet Queue C D E F 22

23 Rate Control Problem: downlink merging 100G + 10G to 100G 100G Circuit What To do? 100G to end host 10G Packet 23

24 Rate Control Problem: downlink merging 100G + 10G to 100G 100G Circuit What To do? 100G to end host 10G Packet Our approach: Rate limit the circuit path at the source to avoid overloading 90G Circuit 100G to end host 10G Packet 24

25 Implementation 25

26 REACToR 26

27 10G/1G Prototype Circuit Switch Mordia OCS Circuit Control EPS Fulcrum Monaco 1G 10G Schedule Control Computer REACToR Virtex 6 FPGA H H H H Schedule REACToR Virtex 6 FPGA H H H H 10G Demand Linux servers (Intel G NICs) 27

28 Timing Parameters End-to-end reconfiguration time: 30 μs Schedule reconfigures every 1500 us Example: 7 flows TDMA, 86% duty cycle 1500μs μs t 28

29 Evaluation Experiment 1: Supporting TCP The performance on working with stock network stack Experiment 2: React to demand changes The dynamics on handling changes and mispredictions Experiment 3: Demonstrate the benefit of using hybrid The performance gain on handling skewed demand 29

30 Experiment 1: Supporting TCP Each host receives 7 TCP flows from all other hosts Hybrid schedule: data packets via OCS, ACKs via EPS 7 flows TDMA, fair sharing the link Check if TCP works with high throughput 30

31 TCP Throughput TDMA invisible to TCP 31

32 Experiment 2: React to Demand Changes From: Intra-rack Traffic To: Inter-rack Traffic Rack 1 Rack 2 Rack 1 Rack 2 Use pktgen to impose precise and sudden traffic pattern change. See if REACToR can react in time. 32

33 Rack 1 Rack 2 React to Demand Changes 3-host round robin demand change 4-host round robin React fast and robust to demand changes Tx with Packet 33

34 Bandwidth Bandwidth Experiment 3: Demonstrating Hybrid Simulated 64 hosts with demand of different skewness Big benefit from a small electrical packet switch Skewed Very Skewed 100G 100G Destinations Destinations 34

35 Bandwidth Bandwidth Experiment 3: Demonstrating Hybrid Simulated 64 hosts with demand of different skewness Big benefit from a small electrical packet switch Skewed Very Skewed 100G 100G 99G 50G 20 flows, 50G 20 flows, 1G Destinations Destinations 35

36 Link Utilized Optical Circuit Switching Not Enough 100% 100G Electrical Fat-tree 70% 50% Fraction of bandwidth the big flow uses 99% 36

37 Link Utilized Optical Circuit Switching Not Enough 100% 100G Electrical Fat-tree 85% Optical Circuit 75% 70% 50% Fraction of bandwidth the big flow uses 99% 37

38 Link Utilized Optical Circuit Switching Not Enough 100% Cost of switching 100G Electrical Fat-tree 85% Optical Circuit 75% Lost due to Inefficient circuit usage 70% 50% Fraction of bandwidth the big flow uses 99% 38

39 Link Utilized Optical Circuit Switching Not Enough 100% 100G Electrical Fat-tree Datacenter typically works in this region 85% Optical Circuit 75% 70% 50% Fraction of bandwidth the big flow uses 99% 39

40 Link Utilized Hybrid Switching with REACToR 100% 90% 100G Electrical Fat-tree REACToR Hybrid 85% Optical Circuit 75% 70% 50% Fraction of bandwidth the big flow uses 99% 40

41 Link Utilized Hybrid Switching with REACToR 90% 100% 100G Electrical Fat-tree 90% REACToR Hybrid Degrades Gradually 85% Optical Circuit 75% 70% 50% Fraction of bandwidth the big flow uses 99% 41

42 Link Utilized Hybrid Switching with REACToR 100% 90% 85% 90% 100G Electrical Fat-tree Optical Circuit REACToR Hybrid Performance akin to 100G Fat-tree 75% Degrades Gradually 70% 50% Fraction of bandwidth the big flow uses 99% 42

43 Conclusion REACToR + = 100G Optical Circuit Switching 10G Electrical Packet Switching 100G Electrical Packet Switching For datacenter workloads At a lower cost 43

44 Thank you! 44

45 DCTCP: Datacenter Workload [SIGCOMM 2010] 45

46 Cost of Transceivers Cost of 10G Transceivers Cost: $500 per pair Power: 1Watt per pair (100G costs even more) 3-Level Fat-tree: 27.6k hosts Transceivers per host: 46

47 Scheduling Problem: matrix decomposition Similar to BvN, but must consider reconfiguration penalty NP-complete problem Goal: schedule all the big flows (90% of the demand) Greedy approach: e.g. islip Suboptimal Naïve BvN: Fragmented by small elements and residuals A good algorithm should: Prioritize the big flows Perform full matrix decomposition (like BvN) Minimize number of reconfigurations at the same time 47