How Facebook Got Consistency with MySQL in the Cloud Sam Dunster

Transcription

1

2 How Facebook Got Consistency with MySQL in the Cloud Sam Dunster Production Engineer

3

4

5 Consistency

6 Replication

7 Replication for High Availability

8 Facebook Replicaset Region A Slave Slave Region B Region C

9 Facebook Replicaset 10s of milliseconds Region A 100s of milliseconds Slave Slave Region B Region C

10 Types of replication

11 Asynchronous Replication Slave

12 Asynchronous Replication Slave

13 Asynchronous Replication Slave Commit

14 Asynchronous Replication Slave RelayLog Commit

15 Asynchronous Replication Slave RelayLog Commit Commit

16 Live Promotion Slave Commit

17 Live Promotion Slave Commit

18 Live Promotion Commit Slave MASTER_POS_WAIT()

19 Live Promotion Slave

20 Dead Promotion Slave Commit

21 Dead Promotion Slave Commit

22 Dead Promotion Slave Commit SQL_THREAD WAIT

23 Synchronous Replication

24 Synchronous Replication Slave

25 Synchronous Replication Slave Prepare

26 Synchronous Replication Slave Prepare Commit

27 Synchronous Replication Slave Commit Commit

28 Live Promotion Slave

29 Live Promotion

30 Dead Promotion

31 Dead Promotion

32 Synchronous Constraints A few micro seconds A few micro seconds A few micro seconds

33 Semi-Synchronous Replication

34 Semi-Synchronous Replication Slave

35 Semi-Synchronous Replication Slave Binlogs

36 Semi-Synchronous Replication Binlogs Slave RelayLog

37 Semi-Synchronous Replication Binlogs Slave RelayLog

38 Semi-Synchronous Replication Binlogs Slave RelayLog Commit

39 Semi-Synchronous Constraints A few micro seconds A few micro seconds Semi Sync Slave Semi Sync Slave

40 Semi-Synchronous mysqlbinlog

41 Semi-Synchronous Constraints A few micro seconds Milliseconds OK! Logtailer A few micro seconds Logtailer Slave

42 Dead Failover Logtailer Slave

43 Dead Failover File size Slave Read Log_Pos

44 Dead Failover mysqbinlog --start-position= Slave 10000

45 Lossless semi-sync

46 Lossless Semi-Sync => Commit; Binlog Prepare => No-op InnoDB Prepare => Written to InnoDB for recovery Binlog Commit => Written to binlog InnoDB Commit => Visible from other clients <= OK;

47 Lossless Semi-Sync => Commit; Binlog Prepare => No-op InnoDB Prepare => Written to InnoDB for recovery Binlog Commit => Written to binlog InnoDB Commit => Visible from other clients <= OK; => Commit; Binlog Prepare InnoDB Prepare Binlog Commit InnoDB Commit Wait for Semi-Sync Ack <= OK;

48 Lossless Semi-Sync => Commit; Binlog Prepare => No-op InnoDB Prepare => Written to InnoDB for recovery Binlog Commit => Written to binlog InnoDB Commit => Visible from other clients <= OK; => Commit; Binlog Prepare InnoDB Prepare Binlog Commit InnoDB Commit Crash! Wait for Semi-Sync Ack <= OK;

49 Lossless Semi-Sync (MySQL 5.7) => Commit; Binlog Prepare => No-op InnoDB Prepare => Written to InnoDB for recovery Binlog Commit => Written to binlog InnoDB Commit => Visible from other clients <= OK; => Commit; Binlog Prepare InnoDB Prepare Binlog Commit InnoDB Commit Wait for Semi-Sync Ack <= OK; Crash! => Commit; InnoDB Prepare Binlog Prepare Binlog Commit Wait for Semi-Sync Ack InnoDB Commit <= OK;

50 Rollout

51 Sum of downtime Deployment date

52 mysqlbinlog + semi-sync patches

53 Binlog Server

54 Semi-sync- Replication Semi-sync Binlog Writer/Acker mysqlbinlog Local Disk Binlog Binlog Binlog log.index bin bin bin

55 Binlog Server Log-tailer Semi-sync- Replication Semi-sync Binlog Writer/Acker Binlog Reader Promotion Catchup Slave Local Disk Binlog Binlog Binlog log.index bin bin bin

56 Logtailer Slave

57 binarylogs Logtailer Slave mysqbinlog --start-position=10000

58 Logtailer CHANGE MASTER TO Slave

59 Binlog Server ++

60 Lagged slaves?

61 Lagged slaves? Error 1236

62 Replicaset slave status: Replication Greatly Lagged Slave Error Msg: has purged the required binary logs

63 Replicaset Binlog Reader/Sender Binlog Locator Binlog Server (1) change master to Binlog Server; start slave; Greatly Lagged Slave

64 Replicaset Binlog Locator DB (2) Locate the list of binlog paths to send based on the slave s GTID set HDFS file path hdfs://***.binlog-1.gz hdfs://***.binlog-2.gz hdfs://***.binlog-3.gz hdfs://***.binlog-4.gz Prev GTID set UUID:1-20 UUID:1-50 UUID:1-70 UUID:1-90 Binlog Reader/Sender Binlog Locator Binlog Server

65 Replicaset HDFS Cluster (3) Read the binlogs on HDFS and prepare binlog packet streams Binlog Reader/Sender Binlog Locator Binlog Server

66 Replicaset Binlog Reader/Sender (4) Sending binlog packet by packet Binlog Locator Greatly Lagged Slave Binlog Server

67 HDFS mode binlog_server> show master status\g *************************** 1. row *************************** File: hdfs://******.binary-logs-xxxxxx.xxxxxx.gz Position: 4 Executed_Gtid_Set: 6c597fb0-d3a4-4aab-ba a75727ed: , 765a6781-d b-8091-e6adeac313ee: Log-tailer mode binlog_server>show master status\g *************************** 1. row *************************** File: binary-logs Position: Executed_Gtid_Set: 49f5e0ca-80d be83-d1aeb5e973bc: , d9d1-49f1-b2bf-0ffb5e603b2d:

68 binlog_server> show slave status\g *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event _Host: HOSTNAME _Port: PORT Connect_Retry: 0 _Log_File: binary-logs-xxxxxx Read Log_Pos: Binlog_File: binary-logs-xxxxxx Binlog_Pos: Last_IO_Errno: 0 _Server_Id: Executed_Gtid_Set: ea4a5e01-b3e a25e-88d06db8d1a5: , b29a87bd-d60b ab8-90d7b720f169: Mysql_Replicaset: REPLICA_SET_NAME Replicaset_Tier_Version: VERSION_NUM Semisync_Slave: Yes

69 There's plenty more to Binlog Server Search for "Binlog Server at Facebook"

70 Distributed systems are really hard

71 DBAs don't scale as well as MySQL does

72 Lossless Semi-Sync => Commit; Binlog Prepare => No-op InnoDB Prepare => Written to InnoDB for recovery Binlog Commit => Written to binlog InnoDB Commit => Visible from other clients <= OK; => Commit; Binlog Prepare InnoDB Prepare Binlog Commit InnoDB Commit Wait for Semi-Sync Ack <= OK; Crash! => Commit; InnoDB Prepare Binlog Prepare Binlog Commit Wait for Semi-Sync Ack InnoDB Commit <= OK;

73 Lossless Semi-Sync => Commit; Binlog Prepare => No-op InnoDB Prepare => Written to InnoDB for recovery Binlog Commit => Written to binlog InnoDB Commit => Visible from other clients <= OK; => Commit; Binlog Prepare InnoDB Prepare Binlog Commit InnoDB Commit Wait for Semi-Sync Ack <= OK; Crash! => Commit; InnoDB Prepare Binlog Prepare Binlog Commit Wait for Semi-Sync Ack InnoDB Commit <= OK;

74 status semi-sync thread async thread transaction 1 acked

75 status semi-sync thread async thread transaction 1 acked transaction 2 prepare

76 status semi-sync thread async thread transaction 1 acked transaction 2 waiting for ack

77 status semi-sync thread async thread transaction 1 acked transaction 2 acked

78 status semi-sync thread async thread transaction 1 acked transaction 2 acked transaction 3 prepare

79 status semi-sync thread async thread transaction 1 acked transaction 2 acked transaction 3 waiting for ack

80 status semi-sync thread async thread transaction 1 acked transaction 2 acked transaction 3 acked

81 Flappy/partially-isolated master

82 MySQL Automation Client Slave Logtailer Logtailer

83 MySQL Automation Client Slave Logtailer Logtailer

84 MySQL Automation Client Slave Logtailer Logtailer

85 Logtailer failures

86 MySQL Automation Client Slave Logtailer Logtailer

87 MySQL Automation Client Slave Logtailer Logtailer

88 MySQL Automation Client Slave Logtailer Logtailer

89 MySQL Automation Client Slave??? Logtailer Logtailer

90 MySQL Automation!!! Client Slave Logtailer Logtailer

91 MySQL Automation!!! Client (blocked) Logtailer Logtailer

92 MySQL Automation Client Logtailer Logtailer

93 MySQL Automation Client Logtailer Logtailer

94 MySQL Automation lbu010 Client lbu030 Slave Logtailer Logtailer

95 MySQL Automation lbu010 Client (blocked) lbu030 Slave Logtailer Logtailer

96 MySQL Automation Client (blocked) Logtailer Logtailer

97 These situations were very rare

98 Mostly open source mysqlbinlog Lossless semi-sync Lots of other patches

99 Sam Dunster