Continuous MySQL Restores Divij Rajkumar

Transcription

1

2 Continuous MySQL Restores Divij Rajkumar Production Engineer, MySQL Infrastructure, Facebook

3 Continuous Restores Why? Verify backup integrity Haven t tested your backups? You don t have them Understand resource requirements for restores Time, server capacity, network capacity Test restore orchestration Don t panic during a disaster event

4 Backups

5 Backups Everything, every day

6 mysq l shar d shar d shar d shar d shar d shar d server mysq l shar d shar d shar d shar d shar d shar d Shard 86 User 1 User 2.. User 1000

7 Backups Everything, every day Tens of thousands of servers, many, many shards Stored in HDFS 3 types: Full dumps Differential backups Binary Logs

8 Backups Full dumps mysqldump --single-transaction --set-gtid-purged=commented, stored in metadata github.com/facebook/mysql-5.6 Per shard, not per instance Shards aren t pinned to an instance Easier to locate

9 Backups Full dumps Logical, not physical Easy single-table restore No indexes backed up, reduced size Easy to debug Better compression Every 5 days

10 Backups Differential backups Diff between 2 full dumps 2 files generated - Rows deleted, Rows inserted Based off of existing dump in HDFS Take full dump -> compute diff -> upload diff

11 Backups Differential backups

12 Backups Differential backups Diff between 2 full dumps 2 files generated - Rows deleted, Rows inserted Based off of existing dump in HDFS Take full dump -> compute diff -> upload diff Possible because of logical full dumps

13 Backups Binary Logs Raw binlogs from mysql Record Previous-GTIDs in metadata Using mysqlbinlog: --start-position=4 --stop-position=121 Continuously uploaded Binlog Server to interface

14 Backups Everything, every day Tens of thousands of servers, many, many shards Stored in HDFS 3 types: Full dumps Differential backups Binary Logs

15 Restore Infrastructure

16 OR C Warchief

17 ORC - Peons

18 Restore Infrastructure - Peons Python process server Exposes Thrift interface Manages MySQL Instance Multiple peons per host Work on restore jobs 1 database == 1 job peo n mysq l restored db restored db restored db restored db peo n mysq l restored db restored db restored db

19 Restore Jobs State Machine 1. SELECT Select backup to restore 2. DOWNLOAD Download backup to disk 3. LOAD Load tables in parallel 4. VERIFY Verify loaded full/diff dump 5. REPLAY Replay binlogs + FAILED states

20 Restore Jobs mysqldump index Custom index added to mysqldump output Locate tables using per-table byte offsets Preserved across diff backups

21 Restore Jobs SELECT Pre-defined search range per job 1 day == restore most recent backup only Use metadata to ignore bad backups Only full/diff backups selected here

22 Restore Jobs DOWNLOAD For full dumps: Download mysqldump from HDFS to disk For diff backups: Stream diffs + full dump to perform 3-way merge Store recreated mysqldump to disk Single mysqldump output at the end

23 Restore Jobs LOAD Split backup into per-table streams using index Load tables in parallel

24 Restore Jobs VERIFY Perform sanity checks on loaded data Possible verification methods: Compare checksums Compare number of tables loaded Best verification: replay binlogs

25 Restore Jobs REPLAY Replay X seconds/minutes/hours worth of binlogs Enables point in time restores Verify loaded data can be written to

26 Restore Jobs State Machine LOAD TABLE LOAD TABLE SELECT DOWNLOAD LOAD TABLE VERIFY REPLAY LOAD TABLE LOAD TABLE

27 Restore Jobs Pipeline Allow processing many jobs concurrently Peon has 1 handler per job state Limit per-state concurrency

28 Restore Jobs Pipeline SELEC DOWNLOA LOA LOAD VERIF REPLA SLOT T D D TABLE Y Y 1 SLOT 2 SLOT 3 SLOT 4 SLOT 5

29 Restore Jobs Pipeline SELEC DOWNLOA LOA LOAD VERIF REPLA SLOT T D D TABLE Y Y db1963 db165 db982 db7 t1 db142 db SLOT db1000 db3794 db1340 db7 db7 t2 db651 2 SLOT db982 3 SLOT t3 db982 4 SLOT t9 db7 t5 5

30 Restore Jobs Pipeline Allow processing many jobs concurrently Peon has 1 handler per job state LOAD -> LOAD + LOAD_TABLE Limit per-state concurrency Slot capacity can be tuned

31 Restore Jobs Selecting Binlogs Full/diff backups can be taken from slaves Binlogs always taken from master Need to uniquely identify transactions GTIDs!

32 Restore Jobs Selecting Binlogs Use GTIDs purged from mysqldump: --set-gtid-purged=commented Use GTIDs purged from mysqlbinlog: --start-position=4 --stop-position=121 These are GTIDs purged, not GTIDs contained

33 Restore Jobs Selecting Binlogs GTIDs purged from LOAD state (dump_gtids): GTIDs purged in binlogs (binlog_gtids): Contains Contains Contains 775-? First binlog to replay First superset

34 Restore Jobs Replaying Binlogs Filter binlog events with mysqlbinlog: --database --skip-gtids --skip-empty-trans github.com/facebook/mysql-5.6 Last transaction to replay: --stop-datetime

35 Restore Job Lifecycle SELEC DOWNLOA LOA LOAD VERIF REPLA SLOT T D D db100 TABLE Y Y db100 1 SLOT t3 t2 t1 2 SLOT 3 SLOT 4 SLOT 5

36 CRT

37 OR C Warchief

38 CRT Continuous Restore Tier Create restore jobs for shards Periodically poll backup metadata to find backups Thrift call to Warchief to issue restore job Monitor restore progress Peons expose job and health stats via Thrift Detailed stats written to Scuba

39 CRT Continuous Restore Tier Failure handling Retry on transient errors Mark unstable Notify oncall Orchestration component Resolve dependency graph Request restores in order of dependency

40 On-demand Restores

41 oops, I accidentally ran UPDATE without WHERE

42 oops, I thought I was on my dev database

43 oops

44 OR C Warchief

45 Discover Edge Cases

46 War Stories

47 War Stories Collation Change COLLATE=latin1_bin Required for MyRocks Schema change deployed Most diff backup restores fail overnight Thanks CRT Monitoring!

48 200x increase in DOWNLOAD_FAILED!

49 War Stories Collation Change DifferentialBackup used default collation Sorting order broken in diff files Consistent sorting is required for 3-way merge 3-way merges failed for affected backups Wouldn t have been caught without continuous restores DifferentialBackup fixed to understand table collation

50 War Stories RBR Binlogs Row-based replication deployed Lots of REPLAY failures overnight The database used for the current transaction has changed since BEGIN. This is not supported! 2 common patterns: Shards belonged to same database tier Failure only happened after master promotion

51 War Stories RBR Binlogs SBR binlogs: Each BEGIN contains session database RBR binlogs: On master, each BEGIN contains session database On slave, BEGIN has no session database

52 War Stories RBR Binlogs With mysqlbinlog --skip-gtids --skip-empty-trans Expected: session databases don t change across events Binlogs not rotated after master promotion For some part of binlog, instance is a slave For some part of binlog, session database is empty For some part of binlog, session databases don t match The database used for the current transaction has changed since BEGIN. This is not supported!

53 War Stories RBR Binlogs Possible solutions: Rotate binlogs during promotion Fix MySQL server behaviour Patch mysqlbinlog???

54 Thank s! Divij Rajkumar

55