Multi-Tenancy & Isolation. Bogdan Munteanu - Dropbox

Size: px

Start display at page:

Download "Multi-Tenancy & Isolation. Bogdan Munteanu - Dropbox"

Horatio Hall
5 years ago
Views:

1 Multi-Tenancy & Isolation Bogdan Munteanu - Dropbox

2 Overview What is Edgestore? Workloads & API Multi-tenancy & Isolation Lessons Learned

3 What is Edgestore Distributed Metadata Store built on top of MySQL Highly Available, Scalable, Durable Abstract away sharding and caching Reduce operational burden Flexible schemas Multi-Region Setup

4 Architecture

5 Architecture cont d 2048 Shards 8 Shards per Engine (and MySQL cluster) 1 Master - 2 Slaves (semi-sync) Multi-region setup

6 MYSQL EDGESTORE Team User Id Company Size 1 Expedia NatGeo Intuit Spotify 600 Id Name Type 1 jondoe@ Jon Free 2 jenny@ Jenny Pro Edgedata Edge Schema type Gid Id Data Photo Team Entity 10:1? Company:Expedia; Name:SF.jpg; Size:5000 Size:64 Company:NatGeo; Photo Team Entity 20:1? Name:Hawaii; Size:500Size:64 Photo Team Entity 30:7? Photo Team Entity 35:3? User 15:1 User 20:2 Name:Tahoe.jpg; Company:Intuit; Size:2000 Size:128 Company:Spotity; Name:Office.jpg; Size:1024 Size:600 jondoe@, Name:Jon; Type:Free jenny@; Name:Jenny; Type:Pro

7 Shard the table Schema Id Data Schema Id Data Schema Id Data Team 10:1 Company:Expedia; Size:5000 Team 20:4 Company:NatGeo; Size:500 Team 30:1 User 40:2 Company:Intuit; Size:2000 Name:Jon; Type:Free Team 50:2 User 60:1 Company:Spotity; Size:600 Name:Jenny; Type:Pro Shard 1 Shard 2 Shard n

8 Restricted API Create/Update/Delete single and batch Compare and Set semantics Reads: Read(Id, ) List(Id, *) Count(Id, *) List(Id, condition=[equals, prefix, range]) ReadLog(Id) ListLog(Id, *) Acquire Read/Write Lock Commit/Rollback Strong consistency semantics

9 Workloads 10 million QPS 600k Writes / second 9.4mil Reads / second 90% of Reads are cache hits 1.5 million QPS to Engine fleet

10 Workloads cont d Batch Size 1 to Some read requests can return 1 row Some can return rows Rows can be between a few bytes to several MB 500+ unique Schemas

11 Engine Proto -> SQL Query Query Result -> Proto Connection Pooling Control / Reduce load to MySQL

12 Workloads cont d High QPS Write / Read Large / expensive requests: Write - large transactions Read - large number of rows, or large rows Multi-Read / Multi-Write

13 Single Request - 1 token Engine Request Handler Resource Pool

14 Batch (parallel) Request - n tokens Engine Request Handler Id1 goroutine Id2 goroutine Id3 goroutine Resource Pool

15 Batch (sequential) Request - n tokens Engine Request Handler Id 1 - Id 10 Id 11 - Id 20 Id 21 - Id 30 Resource Pool

16 More Isolation breakdowns Type of Traffic: Live traffic: Front Ends - user traffic, sync related traffic Offline traffic: Scripts / Async processing / Offline processing Type of Request: Write (Insert, Delete, Update, Create Ids, Aquire Read/ Write Locks) Read (Single read, multi read, list, count, listlog)

17 Layer Resource Pools Engine Write Live Resource Pool Request Handler Read Live Resource Pool Write Offline Resource Pool Read Offline Resource Pool

18 Breakdown by tenant What is a tenant? Source Machine Tag (e.g. front-end) Source ServiceName (e.g. FileSync) Source Schema (e.g. Team) Source Handler (e.g. Thumbnail generator) Source Script (e.g. backfill-albums)

19 Examples frontend:www:teamevent async-worker:async_task_wrapper:contacts service.py:user event taskrunner-nodequota:update_team_usage.py:user

20 CPU Memory Engine Network Storage Disk IO Mysql: Threads connected Mysql: Semi-sync Mysql: CPU / Disk IO Mysql: Threads running

21 Resources QPS is not a good metric, as requests vary considerably # Connections used (mapping to token resource pool) connections used * time 200 connections total pool = 200 * 60 = connection seconds / min: 1 connection per second for 1 min = 60 connection seconds / min 60 connections for 1 second = 60 connection seconds / min

22 Write Live - 1 minute snapshot Tenant ConnSec Used Connections Errors frontend:rpc:user 20 % 5 0 frontend:www:fileid 3 % 90 0 taskrunner: growth: team_quota 0,5 % User 1 % 1 0 Total 24,5 % 100 0

23 Percentage :00 10:01 10:02 10:03 10:04 10:05 Time

24 Percentage :00 10:01 10:02 10:03 10:04 10:05 Time

25 Percentage :00 10:01 10:02 10:03 10:04 10:05 Time

26 Throttle mechanism Auto-throttle heuristics based on history of resource usage per tenant No predefined quota Steady state usage by tenant varies wildly 0.001% - 20% Triggering event -> find bad tenant -> decide how much to throttle them -> throttle bad tenant Disabled the auto-throttling mechanism We have learned a lot

27 Timer Start Acquire Read Commit Write Lock Conn Engine

28 Resources Used Time -> Execution Time Bytes In/Out

29 Write Live - 1 minute snapshot Tenant Used Execution MB Read Conns Errors frontend:rpc:user 20 % 1 % frontend:www:file Id 3 % 3 % taskrunner: growth: team_quota 0,5 % 0,5 % User 1 % 0,5 % Total 24,5 % 5 %

30 Layer: write_live, NumTenants: 360 Throttle Controls: State: steady, TokensPrimaryPool: 300, TokensThrottledPool: 0 Throttled Tenants: [] Period 1: Used Idle Execution Conns Errors Size(MB) Tenants 6.48% 60.15% 2.58% Aggregated stats Top 5 Sources sorted by Used: 0.79% 94.11% 0.36% offline:blu 0.76% 93.68% 0.20% frontend:rpc:userentity 0.45% 19.92% 0.08% cape-sfj:cape_dispatcher:cursorentity 0.42% 52.50% 0.07% filejournal:fj_server_bin:fileid 0.36% 93.80% 0.02% frontend:www:activityentity Layer: write_live, NumTenants: 360 Throttle Controls: State: steady, TokensPrimaryPool: 300, TokensThrottledPool: 0 Throttled Tenants: [] Period 2: Used Idle Execution Conns Errors Size(MB) Tenants 100% 60.15% 52.58% Aggregated stats Top 5 Sources sorted by Used: 93.79% 0.11% 50.36% offline:blu 0.76% 93.68% 0.20% frontend:rpc:userentity 0.45% 19.92% 0.08% cape-sfj:cape_dispatcher:cursorentity 0.42% 52.50% 0.07% filejournal:fj_server_bin:fileid 0.36% 93.80% 0.02% frontend:www:activityentity

31 edgestore_throttle tenant=offline:blu tokens=30 host=abc-de-fg layer=write_live Layer: write_live, NumTenants: 360 Throttle Controls: State: throttled, TokensPrimaryPool: 270, TokensThrottledPool: 30 Throttled Tenants: [offline:blu ] Period 3: Used Idle Execution Conns Errors Size(MB) Tenants 16.20% 60.15% 7.58% Aggregated stats Top 5 Sources sorted by Used: 10.79% 0.11% 5.36% offline:blu 0.76% 93.68% 0.20% frontend:rpc:userentity 0.45% 19.92% 0.08% cape-sfj:cape_dispatcher:cursorentity 0.42% 52.50% 0.07% filejournal:fj_server_bin:fileid 0.36% 93.80% 0.02% frontend:www:activityentity

32 Impact Reduce MTTR Availability event: 1. Detection 2. Investigation 3. Containment 4. Short term fix 5. Long term fix

33 Findings Expensive queries Abusable APIs Query optimizer Inconsistencies Insufficient documentation Bugs Perf optimization

34 Lessons Learnedto isolate the error and limit blast radius 1 deployment to rule them all works There is such a thing as automating too soon Silently throttling is bad Throttling should be a temporary state Not having pre-defined quotas works Auto-throttle heuristics Manual Throttle using a throttle tool Query / Throttle / Unthrottle Aggregate tool - queries and filters all engines while investigating, root causing and fixing the underlying problem. There was a time when we shut down scripts manually not knowing who was causing the problem found issues with API, bugs, poorly documented client, best practices Throttle mechanism Future work (in progress) Multiple Isolation breakdowns (by user, by table, by tenant, by request type (Read/Write), by traffic type (Live vs Offline)

35 What s next Control Plane brain continuously query all Engines automatically throttle tenants when system is degraded detecting trends Per logical micros shard (and per Id) granularity for throttling

36 Credits Zviad Metreveli Rati Gelashvili Robert Verkuil Alex Degtiar Jonathan Lee

MySQL Performance Optimization and Troubleshooting with PMM. Peter Zaitsev, CEO, Percona

MySQL Performance Optimization and Troubleshooting with PMM Peter Zaitsev, CEO, Percona In the Presentation Practical approach to deal with some of the common MySQL Issues 2 Assumptions You re looking