How Netflix Leverages Multiple Regions to Increase Availability: Isthmus and Active-Active Case Study

Transcription

1 How Netflix Leverages Multiple Regions to Increase Availability: Isthmus and Active-Active Case Study Ruslan Meshenberg November 13, Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

2 Failure

3 Assumptions Scale Hardware Will Fail Slowly Changing Large Scale Slowly Changing Small Scale Everything Works Telcos Enterprise IT Everything Is Broken Web Scale Startups Rapid Change Large Scale Rapid Change Small Scale Software Will Fail Speed

4 Incidents Impact and Mitigation Public relations media impact High customer service calls Affects AB test results PR X Incidents CS XX Incidents Metrics Impact Feature Disable XXX Incidents No Impact Fast Retry or Automated Failover XXXX Incidents Y incidents mitigated by Active Active, game day practicing YY incidents mitigated by better tools and practices YYY incidents mitigated by better data tagging

5 Does an Instance Fail? It can, plan for it Bad code / configuration pushes Latent issues Hardware failure Test with Chaos Monkey

6 Does a Zone Fail? Rarely, but happened before Routing issues DC-specific issues App-specific issues within a zone Test with Chaos Gorilla

7 Does a Region Fail? Full region unlikely, very rare Individual Services can fail region-wide Most likely, a region-wide configuration issue Test with Chaos Kong

8 Everything Fails Eventually The good news is you can do something about it Keep your services running by embracing isolation and redundancy

9 Cloud Native A New Engineering Challenge Construct a highly agile and highly available service from ephemeral and assumed broken components

10 Isolation Changes in one region should not affect others Regional outage should not affect others Network partitioning between regions should not affect functionality / operations

11 Redundancy Make more than one (of pretty much everything) Specifically, distribute services across Availability Zones and regions

12 History: X-mas Eve 2012 Netflix multi-hour outage US-East1 regional Elastic Load Balancing issue...data was deleted by a maintenance process that was inadvertently run against the production ELB state data The ELB issue affected less than 7% of running ELBs and prevented other ELBs from scaling.

13 Isthmus Normal Operation US-West-2 ELB US-East ELB ELB Zuul Infrastructure* Zone A Zone B Zone C Tunnel

14 Isthmus Failover US-West-2 ELB US-East ELB ELB Zuul Infrastructure* Zone A Zone B Zone C Tunnel

15 Isthmus

16 Zuul Overview Elastic Load Balancing Elastic Load Balancing Elastic Load Balancing

17 Zuul Details

18 Denominator Abstracting the DNS Layer Amazon Route 53 UltraDNS DynECT DNS Denominator Regional Load Balancers ELBs Zone A Cassandra Replicas Zone B Cassandra Replicas Zone C Cassandra Replicas Zone A Zone B Zone C

19 Isthmus Only for Elastic Load Balancing Failures Other services may fail region-wide Not worthwhile to develop one-offs for each one

20 Active-Active Full Regional Resiliency Regional Load Balancers Regional Load Balancers Zone A Zone B Zone C Zone A Zone B Zone C

21 Active-Active Failover Regional Load Balancers Regional Load Balancers Zone A Zone B Zone C Zone A Zone B Zone C

22 Active-Active Architecture

23 Separating the Data Eventual Consistency 2 4 region Cassandra clusters Eventual consistency!= hopeful consistency

24 Highly Available NoSQL Storage A highly scalable, available, and durable deployment pattern based on Apache Cassandra

25 Benchmarking Global Cassandra Write intensive test of cross-region replication capacity 16 x hi1.4xlarge SSD nodes per zone = 96 total 192 TB of SSD in six locations up and running Cassandra in 20 minutes Test 1 Million Reads Load after 500 ms CL.ONE with No Data Loss US-West-2 Region - Oregon Validation Load Test 1 Million Writes Load CL.ONE (Wait for One Replica to ack) US-East-1 Region - Virginia Zone A Zone B Zone C Zone A Zone B Zone C Interzone Traffic Interregion Traffic Up to 9Gbits/s, 83ms 18 TB backups from S3

26 Propagating EVCache Invalidations 4. Calls Writer with Key, Write Time, TTL & Value after checking if this is the latest event for the key in the current batch. Goes cross-region through ELB over HTTPS Drainer Writer EVCache Replication Service 7. Return keys that were successful Drainer Writer EVCache Replication Service 3. Read from SQS in batches EVCache Replication Metadata 2. Write events to SQS and EVCACHE_REGION_REPLICA- TION EVCache Replication Metadata 5. Checks write time to ensure this is the latest operation for the key US-WEST-2 6. Deletes the value for the key EVCAC EVCAC HE HE EVCACHE 8. Delete keys from SQS that were successful EVCACH EVCACH E E EVCACHE 1. Set data in EVCACHE US-EAST-1 SQS EVCache Client App Server

27 Archaius Region-isolated Configuration

28 Running Isthmus and Active-Active

29 Multiregional Monkeys Detect failure to deploy Differences in configuration Resource differences

30 Multiregional Monitoring and Alerting Per region metrics Global aggregation Anomaly detection

31 Failover and Fallback DNS (denominator) changes For fallback, ensure data consistency Some challenges Cold cache Autoscaling Automate, automate, automate

32 Validating the Whole Thing Works

33 Dev-Ops in N Regions Best practices: avoiding peak times for deployment Early problem detection / rollbacks Automated canaries / continuous delivery

34 Hyperscale Architecture Amazon Route 53 UltraDNS DynECT DNS DNS Automation Regional Load Balancers Regional Load Balancers Zone A Zone B Zone C Zone A Zone B Zone C

35 Does It Work?

36 Building Blocks Available on Netflix Github site

37 Topic Session # When What an Enterprise Can Learn from Netflix, a Cloud-native Company ENT203 Thursday, Nov 14, 4:15 PM - 5:15 PM Maximizing Audience Engagement in Media Delivery MED303 Thursday, Nov 14, 4:15 PM - 5:15 PM Scaling your Analytics with Amazon Elastic MapReduce BDT301 Thursday, Nov 14, 4:15 PM - 5:15 PM Automated Media Workflows in the Cloud MED304 Thursday, Nov 14, 5:30 PM - 6:30 PM Deft Data at Netflix: Using Amazon S3 and Amazon Elastic MapReduce for Monitoring at Gigascale BDT302 Thursday, Nov 14, 5:30 PM - 6:30 PM Encryption and Key Management in AWS SEC304 Friday, Nov 15, 9:00 AM - 10:00 AM Your Linux AMI: Optimization and Performance CPN302 Friday, Nov 15, 11:30 AM - 12:30 PM

38 Takeaways Embrace isolation and redundancy for availability NetflixOSS helps everyone to become cloud native

39 We are sincerely eager to hear your feedback on this presentation and on re:invent. Please fill out an evaluation form when you have a chance.