Making Non-Distributed Databases, Distributed. Ioannis Papapanagiotou, PhD Shailesh Birari

Size: px

Start display at page:

Download "Making Non-Distributed Databases, Distributed. Ioannis Papapanagiotou, PhD Shailesh Birari"

Roxanne Norris
5 years ago
Views:

1 Making Non-Distributed Databases, Distributed Ioannis Papapanagiotou, PhD Shailesh Birari

2 Dynomite Ecosystem Dynomite - Proxy layer Dyno - Client Dynomite-manager - Ecosystem orchestrator Dynomite-explorer - UI

3 Problems & Observations Needed a data store: o Scalable & highly available o High throughput, low latency o Netflix use case is active-active Master-slave storage engines: o o o Do not support bi-directional replication Cannot withstand a Monkey attack Cannot easily perform maintenance

4 What is Dynomite? A framework that makes non-distributed data stores, distributed. Can be used with many key-value storage engines Features: highly available, automatic failover, node warmup, tunable consistency, backups/restores

5 Netflix Running around 2.5 years in PROD 70 clusters ~1000 nodes used by internal microservices Microservices based on Java, Python, NodeJS

6 Pluggable Storage Engines Layer on top of a nondistributed key value data store Peer-peer, Shared Nothing Auto-Sharding Multi-datacenter Linear scale Replication Gossiping RESP RESP

7 Topology Each rack contains one copy of data, partitioned across multiple nodes in that rack Multiple Racks == Higher Availability (HA)

8 Replication A client can connect to any node on the Dynomite cluster when sending requests. o o If node owns the data, data are written in local data-store and asynchronously replicated. If node does not own the data node acts as a coordinator and sends the data in the same rack & replicates to other nodes in other racks and DC.

9 Dyno Client - Java API Connection Pooling Load Balancing Effective failover Pipelining Scatter/Gather Metrics, e.g. Netflix Insights

10 Dyno Load Balancing Dyno client employs token aware load balancing. Dyno client is aware of the cluster topology of Dynomite within the region, can write to specific node using consistent hashing.

11 Dyno Failover Dyno will route requests to different racks in failure scenarios.

12 Dynomite on the Cloud RESP

13 Moving across engines Rack A Rack B

14 Dynomite-manager: Warm up 1. Dynomite-manager identifies which node has the same token in the same DC 2. Leverage master/slave replication 3. Checks for peer syncing a. difference between master and slave offset 4. Once master and slave are in sync, Dynomite is set to allow write only 5. Dynomite is set back to normal state 6. Checks for health of the node - Done!

15 Dynomite-Explorer (UI) Node.js web app with a Polymer-based user-interface Support Redis rich data types Avoid operations that can negatively impact Redis server performance Extended for Dynomite awareness Allow extension of the server to integrate with the Netflix ecosystem

16 Dynomite-Explorer

17 Roadmap Data reconciliation & repair v2 Optimizations of RocksDB configuration Optimizing backups through SST Others.

18 More information Netflix OSS: Chat:

21 Dynomite: S3 backups/restores Why? o o How? o o Disaster recovery Data corruption Storage dumps data on the instance drive Dynomite-manager sends data to S3 buckets Data per node are not large so no need for incrementals. Use case: o o clusters that use Dynomite as a storage layer Not enabled in clusters that have short TTL or use Dynomite as a cache

22 Dynomite-manager Token management for multi-region deployments Support AWS environment Automated security group update in multi-region environment Monitoring of Dynomite and the underlying storage engine Node cold bootstrap (warm up) S3 backups and restores REST API

23 Performance Setup Instance Type: Dynomite: i3.2xlarge with NVMe NDBench: m2.2xls (typical of an Replication factor: 3 Deployed Dynomite in 3 zones in us-east-1 Every zone had the same number of servers Demo app used simple workloads key/value pairs Redis: GET and SET Payload Size: 1024 Bytes 80%/20% reads over writes

24 Throughput

25 Latencies

26 Consistency DC_ONE o Reads and writes are propagated synchronously only to the node in local rack and asynchronously replicated to other racks and data centers DC_QUORUM o Reads and writes are propagated synchronously to quorum number of nodes in the local data center and asynchronously to the rest. The DC_QUORUM configuration writes to the number of nodes that make up a quorum. A quorum is calculated, and then rounded down to a whole number. If all responses are different the first response that the coordinator received is returned. DC_SAFE_QUORUM o Similarly to DC_QUORUM, but the operation succeeds only if the read/write succeeded on a quorum number of nodes and the data checksum matches. If the quorum has not been achieved then an error response is generated by Dynomite.

27 Deploying Dynomite in PROD Unit testing in Github Building EC2 AMI in experimental Pipelines for performance analysis Promotion to candidate Beta Testing Promotion to release

28 Reconciliation Reconciliation is based timestamps (newest wins) and is performed by a Spark cluster Jenkins job to avoid clock skewness

29 Reconciliation: Design Principles We would prefer to alleviate the processing load of performing the reconciliation from each node in the cluster and off load it to a high performance computation in memory cluster based on Spark.

30 Reconciliation: Architecture Forcing Redis (or any other storage engine) to dump data to the disk Encrypted communication between Dynomite and Spark cluster Chunking the data - retry in case of a failure. Bandwidth Throttler

Machine Learning meets Databases. Ioannis Papapanagiotou Cloud Database Engineering

Machine Learning meets Databases Ioannis Papapanagiotou Cloud Database Engineering Create Personalized Recommendations for discoveries of engaging video content that maximizes member joy. Personalize Everything