Microservice Layout in Netflix

Transcription

1

2

3 Microservice Layout in Netflix

4

5

6

7

8 Polyglot Persistence Powering Microservices Roopa Tangirala Engineering Manager Netflix

9 Agenda 5 Use Cases Challenges Current Approach Takeaway

10

11

12

13 AWS S3 CDE

14 Search, Analyze and visualize in near real time Distributed in-memory caching solution based on memcached Distributed NOSQL database to handle large datasets providing high availability. Distributed dynamo layer for different storage engines and protocols supporting Redis, memcached, RocksDB TitanDB is scalable graph database optimized for storing and querying graph datasets.

15 Eu-west-1 Ireland Us-west-2 Oregon Us-east-1 North Virginia

16 CDN URL

17

18 Requirements - CDN URL High availability Very low latency reads/writes (less than 1ms) High Throughput per node

19 GUESS?

20 Distributed In Memory Very low Latency responses

21 Playback Error

22 PLAYBACK CONTEXT (Tracks + Track Urls) NETFLIX OPENCONNECT CDN URLS

23 COUNTRY USER PREFERENCES DEVICE PLAYBACK CONTEXT TITLE METADATA NETWORK

24 Requirements - Playback Error Quick Incident Resolution Interactive Dashboards Near realtime Search Ad Hoc Queries

25 GUESS?

26 Powerful Search & Analytics Interactive Dashboards

27 Interactive Exploration

28 Top N queries

29 Incident To Resolution Time 2+ Hours Under 10 Minutes

30 Viewing History

31

32 Requirements - Viewing History Time series dataset Support high writes Cross region replication Large dataset

33 Growth of Viewing History 33

34 GUESS?

35 Multi-datacenter, multi-directional replication Highly availability and scalability

36 Data Model 36

37 New Data Model 37

38 Digital Asset Management Person PNG Audio Track Display Set B Artwork Webp Display Set A Movie Character JPG Trailer Montage Fr Dub Video Track Video Seg TEXT Track Fr Sub Es Sub

39 Requirements - DAM One backend plane for all asset metadata Storage of relationships/connected data Searchable

40 GUESS?

41 Distributed GraphDB Support for various storage backends

42 Distributed Delayed Queues

43 Requirements - Delayed Queues Distributed Highly concurrent At-least-once delivery semantics Delayed queue Priorities within the shard

44 GUESS?

45 Pluggable datastore supporting Redis Multi-datacenter replication

46 Data Model For each queue three set of Redis data structures are maintained: 1. A Sorted Set containing queued elements by score. 2. A Hash set that contains message payload, with key as message ID. 3. A Sorted Set containing messages consumed by client but yet to be acknowledged. Un-ack set.

47

48

49 CDE

50

51

52 Current Approach

53 Subject Matter Expert

54

55 CDE Service Empowering CDE to provide datastores as a service

56 CDE Service Thresholds/SLAs Cluster metadata Self Service Contact information Maintenance windows

57 Architecture Alert S C H E D U L E R Mantis Atlas MONITORING Datastore Maintenance Remediation Alert if needed Cluster Metadata CDE SERVICE R E M E D I A T I O N PAGE CDE Cluster Metadata/ Advisor

58 SLA cass_xyz cde

59 10 6 cas_xyz abc@netf lix.com xyz@netfl ix.com xyz@netfl ix.com cass_test abc@netf lix.com cass_abc 4

60

61

62 Machine learning Automic (UC4)

63 Pattern in Disk usage

64

65 Cde Channel

66

67

68

69

70 Common Approach CRON System Job Job Runner Job Runner Job Runner Runner

71 Streaming micro-services Source - input, handles backpressure Stage - business logic Sink - output, handles backpressure Source Source Mantis Job Stage Stage Sink Source Sink Stage Sink Source Stage Sink

72 Real Time Dash (Macro View) cluster_3 cluster_6

73 Real Time Dash (Cluster View) 1

74 Takeaway

75