F10: A Fault- Tolerant Engineered Network

Transcription

1 F10: A Fault- Tolerant Engineered Network Vincent Liu, Daniel Halperin, Arvind Krishnamurthy, Thomas Anderson University of Washington

2 Today s Data Centers *From Al- Fares et al. SIGCOMM 08 Today s data centers are built using muli- rooted trees Commodity switches for cost, bisecion bandwidth, and resilience to failures 2

3 FatTree Example: PortLand Heartbeats to detect failures Centralized controller installs updated routes Exploits path redundancy 3

4 Unsolved Issues with FatTrees Slow Detec/on Commodity switches fail oyen Not always sure they failed (gray/parial failures) Slow Recovery Failure recovery is not local Topology does not support local reroutes Subop/mal Flow Assignment Failures result in an unbalanced tree Loses load balancing properies 4

5 F10 Co- design of topology, rouing protocols and failure detector Novel topology that enables local, fast recovery Cascading protocols for opimal recovery Fine- grained failure detector for fast detecion Same # of switches/links as FatTrees 5

6 Outline MoIvaIon & Approach Topology: AB FatTree Cascaded Failover Protocols Failure DetecIon EvaluaIon Conclusion 6

7 Why is FatTree Recovery Slow? dst src Lots of redundancy on the upward path Immediately restore connecivity at the point of failure 7

8 Why is FatTree Recovery Slow? dst No redundancy on the way down AlternaIves are many hops away src No direct path Has alternate path 8

9 Type A Subtree ConsecuIve Parents x y 9

10 Type B Subtree Strided Parents x y 10

11 AB FatTree 11

12 AlternaIves in AB FatTrees dst More nodes have alternaive, direct paths One hop away from node with an alternaive src No direct path Has alternate path 12

13 Cascaded Failover Protocols μs ms s A local rerouing mechanism Immediate restoraion A pushback noificaion scheme Restore direct paths An epoch- based centralized scheduler globally re- opimizes traffic 13

14 Local RerouIng u dst Route to a sibling in an opposite- type subtree Immediate, local rerouing around the failure 14

15 Local RerouIng MulIple Failures u dst Resilient to muliple failures, refer to paper Increased load and path dilaion 15

16 Pushback NoIficaIon u DetecIng switch broadcasts noificaion Restores direct paths, but not finished yet No direct path Has alternate path 16

17 Centralized Scheduler Related to exising work (Hedera, MicroTE) Gather traffic matrices Place long- lived flows based on their size Place shorter flows with weighted ECMP 17

18 Outline MoIvaIon & Approach Topology: AB FatTree Cascaded Failover Protocols Failure DetecIon EvaluaIon Conclusion 18

19 Why are Today s Detectors Slow? Based on loss of muliple heartbeats Detector is separated from failure Slow because: CongesIon Gray failures Don t want to waste too many resources 19

20 F10 Failure Detector Look at the link itself Send traffic to physical neighbors when idle Monitor incoming bit transiions and packets Stop sending and reroute the very next packet Can be fast because rerouing is cheap 20

21 Outline MoIvaIon & Approach Topology: AB FatTree Cascaded Failover Protocols Failure DetecIon EvaluaIon Conclusion 21

22 EvaluaIon 1. Can F10 reroute quickly? 2. Can F10 avoid congesion loss that results from failures? 3. How much does this effect applicaion performance? 22

23 Methodology Testbed Emulab w/ Click implementaion Used smaller packets to account for slower speed Packet- level simulator 24- port 10GbE switches, 3 levels Traffic model from Benson et al. IMC 2010 Failure model from Gill et al. SIGCOMM 2011 Validated using testbed 23

24 F10 Can Reroute Quickly Congestion Window time (ms) Without Failure With Failure F10 can recover from failures in under a millisecond Much less Ime than a TCP Imeout 24

25 F10 Can Avoid CongesIon Loss CDF over Time Intervals Normalized Congestion Congestion Loss Loss F10 PortLand PortLand has 7.6x the congesion loss of F10 under realisic traffic and failure condiions 25

26 F10 Improves App Performance 1 Speedup of a MapReduce computation CDF over trials Job completion Job completion time with time PortLand/F10, with PortLand/F10, i.e., Speedup Median speedup is 1.3x 26

27 Conclusion F10 is a co- design of topology, rouing protocols, and failure detector: AB FatTrees to allow local recovery and increase path diversity Pushback and global re- opimizaion restore congesion- free operaion Significant benefit to applicaion performance on typical workloads and failure condiions Thanks! 27