NetPilot: Automating Datacenter Network Failure Mitigation

Transcription

1 NetPilot: Automating Datacenter Network Failure Mitigation Xin Wu, Daniel Turner, Chao-Chih Chen, David A. Maltz, Xiaowei Yang, Lihua Yuan, Ming Zhang

2 Failures are Common and Harmful Network failures are common 10,000+ switches 2

3 Failures are Common and Harmful Network failures are common Failures cause long down times 3

4 Failures are Common and Harmful Six-month failure logs of production datacenters Network failures are common 25% of failures take 13+ hours to repair Failures cause long down times Time from detection to repair (minutes) 4

5 Failures are Common and Harmful Failures are common due to VERY large datacenters Failures cause long down times Long failure duration large revenue loss 5

6 Failures are Common and Harmful Failures are common due to VERY large datacenters Failures cause long down times Long failure duration large revenue loss 6

7 How to Shorten Failure Recovery Time?

8 Previous Work Conventional failure recovery takes 3 steps Detection Diagnosis Repair passive ping active 8

9 Previous Work Conventional failure recovery takes 3 steps Detection Diagnosis Repair Failure localization/diagnosis [M. K. Aguilera, SOSP 03] [M. Y. Chen, NSDI 04] [R.R Kompella, NSDI 05] [P.Bahl, SIGCOMM 07] [S. Kandula, SIGCOMM 09] 9

10 Automating Failure Diagnosis is Challenging Root causes are deep in network stack Diagnosis involves multiple parties 10

11 Category Failure types Diagnosis & Repair Software 21% Link layer loop Find and fix 19% Imbalance overload bugs 2% Hardware 18% FCS error Replace cable 13% Unstable power Repair power 5% Unknown 23% Switch stops forwarding N/A 9% Configuration 38% Imbalance overload 7% Lost configuration 5% High CPU utilization 2% Errors on multiple switches Update configuration 1. Root causes are deep in the network stack % 32% Errors on one switch 6% Six-month failure logs from several production DCNs 11

12 Category Failure types Diagnosis & Repair Software 21% Link layer loop Find and fix 19% Imbalance overload bugs 2% Hardware 18% FCS error Replace cable 13% Unstable power Repair power 5% Unknown 23% Switch stops forwarding N/A 9% Configuration 38% Imbalance overload 7% Lost configuration 5% High CPU utilization 2% Errors on multiple switches 2. Diagnosis involves multiple parties Update configuration 1. Root causes are deep in the network stack % 32% Errors on one switch 6% Six-month failure logs from several production DCNs 12

13 Category Failure types Diagnosis & Repair Software 21% Link layer loop Find and fix 19% Imbalance overload bugs 2% Hardware 18% FCS error Replace cable 13% Unstable power Repair power 5% 2. Diagnosis involves Failure Diagnosis Imbalance multiple overload parties Requires 7% Unknown 23% Switch stops forwarding N/A 9% Configuration 38% Lost configuration 5% Human Intervention! High CPU utilization 2% Errors on multiple switches Update configuration 1. Root causes are deep in the network stack % 32% Errors on one switch 6% Six-month failure logs from several production DCNs 13

14 Can we do something other than failure diagnosis?

15 NetPilot: Mitigating rather than Diagnosing Failures Mitigate failure symptoms ASAP, at the cost of reduced capacity Detection Diagnosis Repair 15

16 16

17 NetPilot Benefits Short recovery time Small network disruption Low operation cost Automated Detection Diagnosis Repair Mitigation 17

18 Failure Mitigation is Effective Most failures can be mitigated by simple actions Mitigation is feasible due to redundancy 18

19 Category Failure types Mitigation Repair % Software 21% Hardware 18% Unknown 23% Configurati on 38% Link layer loop Deactivate port Find and fix Imbalancetriggered Restart switch bugs overload 19% FCS error Deactivate port Replace cable 13% Unstable power Deactivate switch Repair power 5% Switch stops forwarding Imbalancetriggered overload 2% Restart switch N/A 9% Restart switch 7% Lost configuration Restart switch 5% High CPU utilization Errors on multiple switches Errors on single switch Restart switch 2% n/a Update configuration 32% Deactivate switch 6% 19

20 Category Failure types Mitigation Repair % Software 21% Hardware 18% Unknown 23% Configurati on 38% Link layer loop Deactivate port Find and fix Imbalancetriggered Restart switch bugs overload 19% FCS error Deactivate port Replace cable 13% Unstable power Deactivate switch Repair power 5% Switch stops forwarding Imbalancetriggered overload 2% Restart switch N/A 9% Restart switch 7% Lost configuration Restart switch 5% High CPU utilization Errors on multiple switches Errors on single switch Restart switch 2% n/a Update configuration 32% Deactivate switch 6% 20

21 Category Failure types Mitigation Repair % Software 21% Hardware 18% Unknown 23% Configurati on 38% Link layer loop Deactivate port Find and fix Imbalancetriggered Restart switch bugs overload 19% FCS error Deactivate port Replace cable 13% Unstable power Deactivate switch Repair power 5% 68% of failures can be Switch stops forwarding Imbalancetriggered overload 2% Restart switch N/A 9% mitigated by simple actions Restart switch 7% Lost configuration Restart switch 5% High CPU utilization Errors on multiple switches Errors on single switch Restart switch 2% n/a Update configuration 32% Deactivate switch 6% 21

22 22

23 23

24 Outline Automating failure diagnosis is challenging Failure mitigation is effective How to automate mitigation? NetPilot evaluations Conclusion 24

25 A Strawman NetPilot: Trial-and-error Network failure Localization Roll back if necessary Execute an action No Failure mitigated? Yes End 25

26 NetPilot: Challenges & Solutions Network failure Localization 1. Blind trial-and-error Roll back if necessary Execute an action takes a long time No Failure mitigated? Yes End 26

27 NetPilot: Challenges & Solutions Network failure Localization Roll back if necessary Execute an action 1. Blind trial-and-error takes a long time Failure specific localization No Failure mitigated? Yes End 27

28 NetPilot: Challenges & Solutions Network failure Localization Roll back if necessary Estimate impact Execute an action 2. Partition/overload network Impact estimation No Failure mitigated? Yes End 28

29 29

30 30

31 NetPilot: Challenges & Solutions Network failure Localization Roll back if necessary Estimate impact Rank actions Execute an action 3. Different actions have different side-effects Rank actions based on impact No Failure mitigated? Yes End 31

32 Failure Specific Localization Limited # of failure types Domain knowledge improves accuracy Failure types 1. Link layer loop 2. Imbalance-triggered overload 3. FCS error 4. Unstable power 5. Switch stops forwarding 6. Imbalance-triggered overload 7. Lost configuration 8. High CPU utilization 9. Errors on multiple switches 10. Errors on single switch 32

33 Example: Frame Check Sequence (FCS) Errors 13% of all the failures Cut-through switching Forward frames before checksums are verified Increase application latency 33

34 Localizing FCS Errors error frames seen on L frames corrupted by L frames corrupted by other links & traverse L x L : link corruption rate # of variables = # of equations = # of links Corrupted links: x L > 0 34

35 NetPilot Overview Network failure Localization Estimate impact Roll back if necessary Rank actions Execute an action No Failure mitigated? Yes End 35

36 Impact Metrics Derived from Service Level Agreement (SLA) Availability: online_server_ratio Packet loss: total_lost_pkt latency: max_link_utilization Small link utilization small (queuing) delay Total_lost_pkt & max_link_utilization derived from utilization of individual links 36

37 Estimating Link Utilization Action Traffic Topology Impact Estimator Link utilization # of flows >> redundant paths Traffic evenly distributed under ECMP Estimate the load contributed by each flow on each link Sum up the loads to compute utilization 37

38 Link Utilization Estimation is Highly Accurate 1-month traffic from a 8000-server network Log socket events on each server Ground truth: SNMP counters 38

39 NetPilot Overview Network failure Localization Roll back if necessary Estimate impact Rank actions Execute an action Choose the action with the least impact No Failure mitigated? Yes End 39

40 Outline Automating failure diagnosis is challenging Failure mitigation is effective How to automate mitigation? Localization impact estimation ranking NetPilot evaluations Mitigating load imbalance Mitigating FCS errors Mitigating overload Conclusion 40

41 Load Imbalance Agg a stops receiving traffic Localize to 4 suspects core a core b Agg a Agg b 41

42 42

43 43

44 44

45 45

46 46

47 47

48 Fast FCS Error Mitigation Human operator: after 11 trials in 3.5 hours, 2 out of 28 ports are deactivated NetPilot: deactivates 2 links in 1 trial within 15 minutes 48

49 Fast FCS Error Mitigation 3.5 hours 15 minutes Human operator: after 11 trials in 3.5 hours, 2 out of 28 ports are deactivated NetPilot: deactivates 2 links in 1 trial within 15 minutes 49

50 50

51 51

52 Mitigating Link Overload Mitigate overload by deactivating healthy links Many candidate links in production networks Choose the link(s) with the least impact core 1 core 2 core 1 core 2 core 1 core 2 agg agg agg 3 lost

53 Action Ranking Lowers Link Utilization Replay 97 overload incidents due to link failures 53

54 Conclusion Mitigation reduces failure recovery time Simple actions are effective Made possible by redundancy NetPilot: automating failure mitigation Recovery time: hour minutes Several mitigation scenarios deployed in Bing 54

55 Thank You! NetPilot: Detection Automated Diagnosis Repair Mitigation 55

56 56

57 NetPilot Shortens Recovery Time Time from detection to mitigation 6 months, many production datacenters NetPilot mitigate 3 types of failures all with in 30 minutes Operators work around 50% failures in 2 HOURS 57