Failures in Distributed Systems

Transcription

1 CPSC 426/526 Failures in Distributed Systems Ennan Zhai Computer Science Department Yale University

2 Recall: Lec-14 In lec-14, we learned: - Difference between privacy and anonymity - Anonymous communications - Case study: Dissent [OSDI 12]

3 Lecture Roadmap Failures in Distributed Systems Root Causes How to handle failures Case Study: Catastrophic Failures

4 Failures in Distributed Systems What is the meaning of failure: - Your system does not function correctly - Your system implementation does not meet your spec - What is your intuition when you say some code has bugs?

5 Why we care about failures? Data Center Outages Generate Big Losses Downtime in a data center can cost an average of $740,357 per incident -- cloud outage cost report in 2016.

6 Failures in Distributed Systems Different aspects of system failures: - Reliability: Operation & job failures, data loss & corruption - Performance: Traffic overloaded, connection delay - Availability: Node and cluster downtime - Consistency: Permanent inconsistent replicas - Scalability: Function incorrectly in large-scale clusters

7 Failures in Distributed Systems Different aspects of system failures: - Reliability: Operation & job failures, data loss & corruption - Performance: Traffic overloaded, connection delay - Availability: Node and cluster downtime - Consistency: Permanent inconsistent replicas - Scalability: Function incorrectly in large-scale clusters

8 Reliability & Availability Correlated failures resulting from EBS due to bugs in one EBS server Summary of the October 22, 2012 AWS Service Event in the US-East Region We d like to share more about the service event that occurred on Monday, October 22nd in the US-East Region. We have now completed the analysis of the events that affected AWS customers, and we want to describe what happened, our understanding of how customers were affected, and what we are doing to prevent a similar issue from occurring in the future. The Primary Event and the Impact to Amazon Elastic Block Store (EBS) and Amazon Elastic Compute Cloud (EC2)

9 Reliability & Availability Elastic Compute Cloud (EC2) Elastic Block Store (EBS)

10 Reliability & Availability Elastic Block Store (EBS)

11 Reliability & Availability VM1 VM2 VM1 VM3 VM1 VM2 VM3 VM Elastic Block Store (EBS)

12 Netflix VM1 VM2 VM1 VM3 VM1 VM2 VM3 VM EBS Cluster1 EBS Cluster2

13 Netflix VM1 VM2 VM1 VM3 VM1 VM2 VM3 VM EBS Cluster1 EBS Cluster2

14 Netflix VM1 VM2 VM1 VM3 VM1 VM2 VM3 VM EBS Cluster1 EBS Cluster2

15 Netflix VM1 VM2 VM1 VM3 VM1 VM2 VM3 VM EBS Cluster1 EBS Cluster2

16 Global Outages

17 A more tricky example

18

19 Video App Cloud Provider A Cloud Provider B

20 Video App Cloud Provider A Cloud Provider B Third-party infrastructure components

21 Video App Cloud Provider A Cloud Provider B Third-party infrastructure components ISP Router A ISP Router B ISP Router C

22 Video App Cloud Provider A Cloud Provider B Third-party infrastructure components ISP Router A ISP Router B ISP Router C Power Source

23 Video App Cloud Provider A Cloud Provider B Third-party infrastructure components ISP Router A ISP Router B ISP Router C Power Source

24 Video App Cloud Provider A Cloud Provider B Third-party infrastructure components ISP Router A ISP Router B ISP Router C Power Source

25 Video App Cloud Provider A Cloud Provider B Cloud providers do not usually share Third-party infrastructure components information about their dependencies ISP Router A ISP Router B ISP Router C Power Source

26 Failures in Distributed Systems Different aspects of system failures: - Reliability: Operation & job failures, data loss & corruption - Performance: Traffic overloaded, connection delay - Availability: Node and cluster downtime - Consistency: Permanent inconsistent replicas - Scalability: Function incorrectly in large-scale clusters

27 Failures in Distributed Systems Different aspects of system failures: - Reliability: Operation & job failures, data loss & corruption - Performance: Traffic overloaded, connection delay - Availability: Node and cluster downtime - Consistency: Permanent inconsistent replicas - Scalability: Function incorrectly in large-scale clusters

28 Consistency Failures ZooKeeper (synchronization service) Issue # Nodes A, B, C start (w/ latex txid: 10) 2. B becomes leader 3. B crashes 4. C becomes leader 5. C commits new txid-value pair (11, X) 6. A crashes, before committing the new txid C loses quorum and C crashes 8. A and B are back online after C crashes 9. A becomes leader 10. A's commits new txid-value pair (11, Y) 11. C is back online after A's new tx commit 12. C announce to B (11, X) 13. B replies diff starting with tx Inconsistency: A has (11, Y), C has (11, X) PERMANENT INCONSISTENT REPLICA

29 Failures in Distributed Systems Different aspects of system failures: - Reliability: Operation & job failures, data loss & corruption - Performance: Traffic overloaded, connection delay - Availability: Node and cluster downtime - Consistency: Permanent inconsistent replicas - Scalability: Function incorrectly in large-scale clusters

30 Scalability Failures Large data In HBase Tens of minutes Insufficient lookup opera;on R1 R R2 R100K R3

31 Lecture Roadmap Failures in Distributed Systems Root Causes How to handle failures Case Study: Catastrophic Failures

32 Root Causes Customer Environment 25% 9% 31% Misconfiguration Hardware Bugs 15% 20% 5 Storage Systems, 2011

33 Root Causes Customer Environment 25% 9% 31% Misconfiguration Hardware Bugs 15% 20% More Cloud Systems 5 Storage Systems, 2011

34 Root-cause types: Logic Logic (29%) Many domain-specific issues 48

35 Root-cause types: Error handling Logic Error handling (18%) 49

36 Example

37 Root-cause types: Op/miza/on Logic Error handling Op#miza#on (15%) 55

38 Root-cause types: Configura3on Logic Error handling Op3miza3on Configura)on (14%) 56

39 Example Configuration File Specify parameters MySQL

40 Example

41 Example

42 Root-cause types: Race Race (12%) < 50% local concurrency bugs Buggy thread interleaving Tons of work > 50% distributed concurrency bugs Reordering of messages, crashes, Cmeouts More work is needed SAMC [OSDI 14] 69

43 Root-cause types: Hang Hang (4%) Classical deadlock Un-served jobs, stalled opera8ons, Root causes? How to detect them? 70

44 Root-cause types: Space Space (4%) Big data + leak = Big leak Clean-up opera:ons must be flawless. 72

45 Root-case types: Load Load (4%) Happen when systems face high request load Relates to QoS and admission control 73

46 Lecture Roadmap Failures in Distributed Systems Root Causes How to handle failures Case Study: Catastrophic Failures

47 Failure Handling in Distributed Systems

48 Failure Handling in Distributed Systems Service initialization Changing network paths Upgrading software components Outage Service Runtime

49 Failure Handling in Distributed Systems Post-Failure Troubleshooting Failure Prevention 1. Diagnosis tools 2. Accountability 3. Provenance Service initialization Changing network paths Upgrading software components Outage Service Runtime

50 Failure Handling in Distributed Systems Post-Failure Troubleshooting Failure Prevention Failure Tolerance 1. Diagnosis tools 2. Accountability 3. Provenance Service initialization Changing network paths Upgrading software components Outage Service Runtime

51 Lecture Roadmap Failures in Distributed Systems Root Causes How to handle failures Case Study: Catastrophic Failures

52 Next Lecture In the lec-16, I will learn: - Misconfigurations in distributed systems - How important they are - How to handle misconfiguration