Microservices at Netflix Scale. First Principles, Tradeoffs, Lessons Learned Ruslan

Transcription

1 Microservices at Netflix Scale First Principles, Tradeoffs, Lessons Learned Ruslan

2 Microservices: all benefits, no costs?

3 Netflix is the world s leading Internet television network with over 81 million members in over 190 countries enjoying more than 125 million hours of TV shows and movies per day, including original series, documentaries and feature films.

4 Ruslan Meshenberg Director, Platform Engineering Runtime Systems Container Runtime Persistence and Databases Real Time Data Infrastructure

5 Netflix runs on microservices

6 Netflix journey to microservices

7 Our journey took 7 years

8 Data Center - Monolith RDBMS

9 August 2008

10 First Principles

11 Buy vs. Build Use or contribute to OSS technologies first Only build what you have to

12 Services should be stateless* Must not rely on sticky sessions Prove by Chaos testing *Except the Persistence / Caching layers

13 Scale out vs. scale up If you keep scaling up, you ll hit a limit Horizontal scaling gives you a longer runway

14 Redundancy and Isolation For Resiliency Make more than one of anything Isolate the blast radius for any given failure

15 Automate destructive testing Simian Army Started with Chaos Monkey

16 First Principles In Action

17 Stateless services Service A Service B Service B Service B Service B Service B

18 Verify stateless

19 Data from RDBMS to Cassandra NoSQL at scale Open Source Multi-Regional Multi-directional Available Partition Tolerance Tunable Consistency*

20 Multi-Regional Replication Zone B 500ms Zone B Zone A Zone B Zone A Zone B Client Client Zone A Local Quorum (Typical) Zone C Zone A Zone C Zone C Zone C Region A Bi-directional Nightly compare & repair Region B

21 Last, but not least - Billing

22 Microservices Benefits

23 Our Priorities 3. Efficiency 1. Innovation 2. Reliability

24 Innovation: tight coupling doesn t work Develop Team A Team B Team C Test Release

25 Innovation: Loose coupling Team A Team B Team C Develop, Test, Deploy, Support Develop, Test, Deploy, Support Develop, Test, Deploy, Support

26 Support Architect Run Design End-end ownership Deploy Develop Test Review

27 End-end ownership + velocity Architect Design Develop Review Test Deploy Run Support Architect Design Develop Review Test Deploy Run Support Architect Design Develop Review Test Deploy Run Support Architect Design Develop Review Test Deploy Run Support Architect Design Develop Review Test Deploy Run Support Architect Design Develop Review Test Deploy Run Support Architect Design Develop Review Test Deploy Run Support Architect Design Develop Review Test Deploy Run Support Architect Design Develop Review Test Deploy Run Support Architect Design Develop Review Test Deploy Run Support Architect Design Develop Review Test Deploy Run Support Architect Design Develop Review Test Deploy Run Support

28 Separation of concerns UI Feature A Feature B Feature C Personalization Feature D A/B Test E Leverage Mid-tier A/B Test F Feature H Infrastructure Availability Scalability Security

29 Microservices Costs

30 Microservices Is an org change! Org changes are hard!

31 Evolving the organization

32 Central infrastructure investment

33 Migration doesn t happen overnight Living in the hybrid world Supporting 2 tech stacks Double the maintenance Multi-master data replication

34 Microservices - Lessons Learned

35 IPC is crucial for loose coupling Common language between the services Establishes the contract of interaction

36 Caching to protect DBs Client Application Request Cache Client Library EVCache Client Service Client S S S S 1. Read from Cache 2. On cache miss call service 3. Service calls DB and responds 4. Service updates the cache... DB DB DB DB

37 Operational visibility matters If you can t see it, you can t improve it

38 Will your Telemetry scale? Observe Orient Act Decide

39 Edge Middle Tier & Platform Zuul EVCache ELB API Cassandra Playback

40 Reliability Matters We strive for 4 9 s of availability That leaves only 52 minutes of downtime per YEAR Netflix outages lead to

41 Disappointment

42 Outrage

43 Withdrawal

44 Humor

45 Cascading failures 99% availability 99% availability 99% availability 99% 500 = %

46 Microservice failure FIT Fault-Injection Test Framework

47 Regional fail-over x x

48 Regional fail-over

49 A word on containers Containers change the level of encapsulation from VM to process Containers can help deliver great developer experience To run containers in production at scale

50 Requires something like this: Cassandra Zookeeper Docker Docker Registry Docker Registry Registry S3 Titus UI Titus UI Titus UI Titus Rhea Rhea API Titus Master Job Management & Scheduler Fenzo Mesos Master Titus Agent container container container container container container docker zfs docker metrics agent Titus executor logging agent VPC networking driver CI/CD EC2 Autocaling API mesos agent AWS container metadata proxy Integration Amazon VM s 50

51 Microservices - Resources

52

53

54

55

56

57

58 Wrap up

59 Microservices bring great value to development velocity, availability and other dimensions

60 Microservices at scale require organizational change and centralized infrastructure investment

61 Be aware of your situation and what works for you

62 Questions? Ruslan