One System To Fit Them All:

Transcription

1 One System To Fit Them All: Shared MySQL Hosting At Facebook Andrew Regner Production Engineer MySQL Infrastructure

2

3 Data Everyone has data to persist Also have: ZippyDB, ODS, Scuba, HBase, Scribe, RocksDB, TAO MySQL is the most mature

4 XDB "Anything" Database Larger and/or order use cases of MySQL will have their own *db XDB is supposed to be the answer for everyone else

5 Our History c Started with "CDB" allocating resources manually and with little logic behind it. 1.5 years ago, another MySQL Infra engineer was at Percona Live to talk about what we were just starting to do with XDB In the last few years, we have had the same amount of Infra engineers working on this, but 10x the company around us.

6 Things that store data video encoding (queue) data warehouse (metadata) Hundreds of Teams job scheduling server management internal tools (tasks, wiki) hack-a-thon toys Thousands of Shards visitor sign-in system backup systems (more than i can count) qualitative analysis of search results machine learning models

7 Terminology

8 server instance instance shard shard shard shard shard shard shard shard shard shard shard shard shard shard

9

10

11

12 Master replication replication Slave Slave

13 Replica Set tasks burger pong_1 feed videos tags wiki replication replication tasks burger pong_1 feed videos tags wiki tasks burger pong_1 feed videos tags wiki

14 replication replication

15

16 Philosophy of FB Infrastructure Move Fast Enable engineers to do what they need, when they need it. They understand the importance and scope of their product the best.

17 Story Time

18 Robert's New Feature xdb.profile_events

19 Robert's New Feature xdb.profile_events

20 Robert's New Feature xdb.profile_events Lands a configuration change to cause all records in his database to be re-processed. Everything is still in test mode, so not worried about the impact to anything.

21 Connected Running Lock time xdb.profile_events (master)

22 Robert's New Feature xdb.profile_events His system limits concurrency to 30 at once. Didn't realize that the script opens a new connection for each of 100 queries per execution. :-(

23 Robert's New Feature xdb.profile_events Query comments to know where the jobs is Use internal UI's to kill the job, than page Robert

24 Remember All The Tickets xdb.ticket_processing

25 Remembering All The Tickets xdb.ticket_processing Every time something changes with a Help Center support ticket, some metadata is created. Have to hold onto parts of it for legal reasons.

26 alarm 1 day ago xdb.ticket_processing

27 Remembering All The Tickets xdb.ticket_processing Guidance on maximum ideal shard size Use cases vary too much to enforce

28 Remembering All The Tickets Forgetting some things Tool that an intern / now full-time employee wrote to delete large amounts of data based arbitrary SQL in chunks with range queries. OSC on a larger host to reclaim disk space, than replace all the instances.

29 Remembering All The Tickets What we learned Existing automation is very good at hiding this Put shard sizes in front of the user ASAP Proactively reach out to owners of the top 5% by shard size Automatically notify owners when their growth looks "dangerous"

30 When All The Backups Restarted xdb.db_backup.{1..200}

31 When all the backups restarted xdb.db_backup Metadata about our MySQL Backup infrastructure is stored in a MySQL shard. When all the backup agent processes restart at the wrong time of day, they all try to (re-)start their backups, updating these shards.

32 When all the backups restarted The cobbler's son has the worst shoes Small updates to lots of shards is great on the master Slaves can't keep up :-(

33 Load All The Data xdb.app_installs

34 Load All The Data xdb.app_installs One day, we shut down a data center region (intentionally) The day after, things get turned back on. Someone's data processing pipeline needs to catch up...

35 Load All The Data xdb.app_installs Backlog of "LOAD DATA INFILE..." queries blow up all the slaves in an instant :-(

36 The nicest bad neighbor ever xdb.scheduler / xdb.looks_like

37 The nicest bad neighbor ever xdb.scheduler / xdb.looks_like Shared pool of general purpose XDB shards A bunch of little shards on a few replica sets for a queue workload A large shard on the same replica set for archival

38 Running Threads History List Length xdb.scheduler / xdb.looks_like

39 Current Tools

40 dba replblame finding the cause of lag =============================================================== Instance: xdb0123.prn1:3307 Report UUID: 91a5d53e-2fcf-49e df7fda31d1 Time started: :56:11 Length: 30s Reason: dbstatus disabled instance for lag =============================================================== Total sampled queries: 2753 myservice_data (2750): 2019 LOAD DATA INFILE? REPLACE INTO TABLE `all_the_data` FIELDS TERMINATED BY? ENCLOSED BY '?\\?\n? 420 LOAD DATA INFILE? IGNORE INTO TABLE `some_more_data` FIELDS TERMINATED BY? ENCLOSED BY '?\\?\n? 127 UPDATE `sig_tw_jobs` t SET t.status =? WHERE t.shard_id =? AND t.handle =? AND t.status =? 10 UPDATE `sig_model_snapshot` s, `sig_model` m SET s.removed =?, m.active_snapshot_id =? WHERE s.model_snapshot_id =? AND m.model_...70 more bytes 2 UPDATE `sigrid_model_snapshot` SET model_output =? WHERE model_snapshot_id =?

41 xdb task tell someone there's a problem $ xdb task xdb.fb_learning_mysql --template size assigned_to= tags=[u'dba', u'xdb', u'oncall', u'xdb_enforcement', u'disk_space'] title=xdb xdb.learning_mysql exceeding allowed disk space desc=an xdb that you own ( xdb.learning_mysql ) has exceeded disk space limits. Please cleanup some data immediately (see ). Instance sizes: Table sizes: Table sizes (information_schema): Schema Table Size(GB) learning_mysql channels learning_mysql workflow_runs learning_mysql operator_plans learning_mysql retention learning_mysql operator_runs learning_mysql job_instance_status learning_mysql job_instance_status_updates learning_mysql operator_run_outputs

42 DB Portal

43 DB Portal xdb.myservice_data

44 Looking Forward

45 Some random thoughts from the roadmap Automatic tasks / pages for shard owners Share our monitoring subscriptions Automatic detection (& killing) of bad queries Stricter enforcement of quotas / capacity

46 One System To Fit Them All: Shared MySQL Hosting At Facebook Andrew Regner Production Engineer MySQL Infrastructure