MySQL Replication: Latest Developments

Transcription

1 MySQL Replication: Latest Developments Luís Soares Principal Software Engineer, MySQL Replication Technologies Lead 1

2 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 2

3 Program Agenda 3

4 Program Agenda Background Latest Developments in Replication GA Latest Developments in Replication Plugins Latest Developments in Replication Core Roadmap Summary 4

5 1 Background 5

6 MySQL Database Replication: Overview App INSERT... Replication stream/log Persistent replication stream/log buffer Threaded applier binary log relay log binary log Server A INSERT... Comm. Framework INSERT... Server B INSERT... Capture statements or data changes. Send, Receive, ACK, NACK, Heartbeating,... Receiver Meta-data Update Applier Meta-data Update 6

7 MySQL Database Replication: Some Notes The Binary Log Logical log that records the changes on the master. Statement or Row based format (may be intermixed). One of the reasons that made MySQL (Replication) so popular. Enables seamless integration with other systems. GTID BEGIN E1 E2... COMMIT GTID BEGIN E1 E2... COMMIT Layout of the Binary Log (i.e., the replication stream). 7

8 MySQL Database Replication: Some Notes Coordination Between Servers transactions A B Since 3.23 asynchronous (native) A transactions acks B Since 5.5 semi-synchronous (plugin) A B C 5.7-labs transactions, membership, coordination group replication (plugin) The Corner Stone of MySQL InnoDB Cluster 8

9 MySQL Database Replication: Use Cases Collection of Common Use Cases Common Use Cases 1. Read Scale-out 2. Redundancy 3. Dedicated Replicas 4. Disaster Recovery 5. Data Integration A B C D write clients read clients A B 2. A B C 5. C Crash A A B C B Image from A B C 9

10 MySQL Database Replication: Use Cases Highly Available Replication Group Group Replication Plugin A B C D E Automated failure detection. Split-brain protection. Conflict detection and resolution. Automatic Coordination. 10

11 ReplicaSet 1 MySQL Group Replication: The Enabler MySQL InnoDB Cluster MySQL Group Replication: Replication. Automated HA procedures. Cluster-wide automated coordination. Core of MySQL InnoDB Cluster! MySQL Router: Transparent connection failover. MySQL Shell: One-stop tool to manage the cluster. MySQL Shell Application MySQL Connector MySQL Router M M HA M Application MySQL Connector MySQL Router MySQL Group Replication + Tooling = MySQL InnoDB Cluster (Full Stack HA) 11

12 1 Background 2 Latest Developments in Replication GA 12

13 Timeline Server GA Releases and Major Replication Enhancements MySQL MySQL MySQL MySQL replication added to MySQL statement-based replication Lets Take a Small Step Back First! MySQL row-based replication MySQL semi-sync replication MySQL crash-safe replication metadata global transaction Identifiers multi-threaded applier binary log group commit MySQL large semi-sync enhancements 2nd generation of multi-threaded applier multi-source replication group replication server core changes XA support in replication 13

14 Timeline How MySQL Replication Has Evolved Over the Years! MySQL MySQL MySQL MySQL replication added to MySQL statement-based replication MySQL row-based replication MySQL semi-sync replication MySQL crash-safe replication metadata global transaction Identifiers multi-threaded applier binary log group commit MySQL large semi-sync enhancements 2nd generation of multi-threaded applier multi-source replication group replication server core changes XA support in replication 14

15 Interesting Notes Overall MySQL Replication is very popular and has a huge deployment base. 15+ years of production ready releases. Simple to understand, deploy and use. Ubiquitous. Recurring comment: Our business runs on MySQL Replication. MySQL Replication has seen an accelerated development over the years. Increased quality, increased feature set, increased reliability! 15

16 Interesting Stats About MySQL replication worklogs pushed to MySQL related to MySQL Replication Core 11 were ground work for MySQL Group Replication. 8 replication related contributions merged into MySQL 5.7 (and counting). Some contributions even though rejected, served as inspiration for our work. Countless technical discussions and feedback received from our users. You are an active part of this too! 16

17 Timeline Server GA Releases and Major Replication Enhancements Great! MySQL Lets check out some MySQL of the MySQL crash-safe replication metadata MySQL global transaction Identifiers nice Replication multi-threaded features applier binary log group commit MySQL in MySQL semi-sync replication 5.7. MySQL replication added to MySQL statement-based replication MySQL row-based replication MySQL large semi-sync enhancements 2nd generation of multi-threaded applier multi-source replication group replication server core changes XA support in replication 17

18 Online Reconfiguration of Global Transaction Identifiers File name and offset based replication positioning S2 S1 S3 S2 S1 S3 S2 S1 S3 S2 S1 S3 GTID based replication positioning S4 S4 S4 S4 SET = OFF_PERMISSIVE; SET = ON_PERMISSIVE; SET = ON; Details: 18

19 Online Reconfiguration of Global Transaction Identifiers File name and offset based replication positioning Reads S1 and writes always S1 allowed. S1 No need to synchronize servers, ever. No need to restart servers. S2 S3 S2 S3 S2 No need to change replication topology. S3 S2 Easy to roll back and forth and to switch between modes. S4 S4 S4 S1 S3 S4 GTID based replication positioning SET = OFF_PERMISSIVE; SET = ON_PERMISSIVE; SET = ON; Details: 19

20 Improved Replication Monitoring mysql> select * from performance_schema.replication_applier_status_by_worker\g *************************** 1. row *************************** CHANNEL_NAME: WORKER_ID: 1 THREAD_ID: 35 SERVICE_STATE: ON LAST_SEEN_TRANSACTION: 4ba0eb86-63c3-11e4-92ba-28b2bd168d07:2368 LAST_ERROR_NUMBER: 0 LAST_ERROR_MESSAGE: LAST_ERROR_TIMESTAMP: :00:00 *************************** 2. row *************************** CHANNEL_NAME: WORKER_ID: 2 THREAD_ID: 36 SERVICE_STATE: ON LAST_SEEN_TRANSACTION: 4ba0eb86-63c3-11e4-92ba-28b2bd168d07:2367 LAST_ERROR_NUMBER: 0 LAST_ERROR_MESSAGE: LAST_ERROR_TIMESTAMP: :00:00 2 rows in set (0,00 sec) 20

21 Improved Applier Throughput Locking-based Parallelism Fast and Scalable Multi-threaded Replication Applier Slave apply time (Sysbench RW on the master with 96 threads = 100%) 250% 200% 150% 100% 50% 0% 10X Single Threaded Apply 8 Threads Applier 24 Threads Applier 48 Threads Applier 96 Threads Applier (statement-based replication, durable settings, SSD storage) 21

22 Improved Applier Throughput Locking-based Parallelism Concurrent transactions, which have not blocked each other on the master, are marked as non-contending in the binary log. Non-contending transactions are applied in parallel on the other end. trx1 Commit procedure begins trx2 time Commit intervals for trx1 and trx2 overlap, thence they have not blocked each other during execution. Consequently, they can be applied in parallel. Commit procedure ends. Locks are released. 22

23 Faster Semi-sync Replication ACK Receiver Thread Sender thread does not wait for ACKs anymore, it is just responsible to send binary logs. ACK receiver thread is responsible to receive ACKs. ACK receiver thread starts and stops when semi-sync is activated and deactivated respectively Before After master trx1 trx2 ACK ACK master trx1 trx2 ACK ACK slave trx1 trx2 time slave trx1 trx2 time 23

24 Faster Semi-sync: Durability over the Network Orange bar s percentage shows the throughput gain when compared to the corresponding blue bar. Blue and stacked bars show throughput ratio computed against the best blue bar. 24

25 Loss-less Semi-sync Replication T1: INSERT INTO t1 VALUES (1000) T2: SELECT * FROM t1; empty set Master execute prepare binlog execute commit Slave relay log ACK Time mysql> SET rpl_semi_sync_master_wait_point= [AFTER_SYNC AFTER_COMMIT] 25

26 Semi-sync Replication Wait for Multiple ACKs Master does not commit transaction until it receives N ACKs from N slaves. Dynamically settable: mysql> SET GLOBAL rpl_semi_sync_master_wait_for_slave_count= N; Master Slave 1 Slave 2 Slave 3 T1: COMMIT ACK relay log relay log relay log ACK ACK T1: COMMIT succeeds Time mysql> SET GLOBAL rpl_semi_sync_master_wait_for_slave_count= 2; 26

27 Global Transaction Identifiers History in a System Table Use Cases Clients Clients S1 S1 S2 GITD GITD Potential New Master Auto positioning in the stream Skips automatically executed transactions Persists GTID history in the Binary Log Cannot become a Master Auto positioning in the stream Skips automatically executed transactions Persists GTID history in a System Table S2 GITD 27

28 Multi-Source Replication M M M M Slave can have more than one master. S The need for gathering data in a central server: Integrated backup; Complex queries for analytics purposes; Data HUB for inter-cluster replication. 28

29 Smaller, yet interesting, enhancements! Multi-threaded applier is able to retry failed transactions. Option to make the Multi-threaded applier preserve commit order. SSL options for mysqlbinlog tool. Rewrite DB rules for the mysqlbinlog tool. Function to wait for transactions to be applied, regardless of the replication stream they come from. Options to track global transaction identifiers in the return packet of the mysql protocol. Useful for tracking GTID session state and enabler for session consistency. Support for XA transactions when the binary log is enabled. Change in the defaults. E.g., binlog_format=row and sync_binlog=1. Options to fine tune the binary log group commit procedure. 29

30 Timeline Server GA Releases and Major Replication Enhancements The road to MySQL 5.7 gave us features... MySQL MySQL and infrastructure crash-safe replication to metadata take MySQL MySQL global transaction Identifiers multi-threaded applier MySQL MySQL Replication binary log and group commit HA to replication added to statement-based replication MySQL semi-sync replication MySQL large semi-sync enhancements 2nd generation of multi-threaded applier the next level! MySQL row-based replication multi-source replication group replication server core changes XA support in replication 30

31 Features and Infrastructure Notes Better replication stream handling with Global Transaction Identifiers. Global Transaction Identifiers easier roll out. Better and more scalable replication monitoring. Instrumentation and performance schema tables for replication enable different topologies and generic monitoring. Faster applier threads make replication more efficient and faster. Enabler for different parallelization schemes. Flexible deployments enable more advanced setups. Replication slave can now replicate from multiple sources at once. Infrastructure became generic: paved the way for other replication schemes (e.g., Group Replication) Best part is... It does not end here. Plenty of interesting work to do and interesting features to deliver! ;) 31

32 1 Background 2 Latest Developments in Replication GA 3 Latest Developments in Replication Plugins 32

33 MySQL Group Replication Plugin (GR plugin) A B C D E Replication Group Multi-master update everywhere with row level conflict detection. Automatic group membership management and failure detection. No need for server fail-over. Automatic reconfiguration. No single point of failure. Shared-nothing state machine replication. InnoDB compliant. Off-the-shelf hardware friendly. 33

34 Timeline Server GA and GR Releases Oh! Wait... MySQL MySQL MySQL crash-safe replication metadata Lets take another MySQL global transaction Identifiers multi-threaded applier binary log group commit small step back! MySQL semi-sync replication MySQL MySQL replication added to MySQL statement-based replication MySQL row-based replication large semi-sync enhancements 2nd generation of multi-threaded applier multi-source replication group replication server core changes XA support in replication 34

35 Timeline Server GA and GR Releases MySQL GR labs Hello world! GR labs version handling GR labs support for corosync 2.x GR labs auto-inc fields handling recovery enhancements GR labs multi-platform support Paxos-based consensus GR labs bug fixes MySQL lifecycle interfaces P_S tables for GR Server side changes GR labs (RC) multi-threaded applier support single primary mode GR labs (Beta) performance enhancements replication stream compression SSL support IP whitelisting read-only mode error log enhancements split brain handling 35

36 Comments Feature rich new replication plugin based on proven distributed systems algorithms (Paxos). Compression, multi-platform, dynamic membership, distributed agreement, quorum based message passing, SSL, IP whitelisting, integrated with the server,... Frequent releases on Labs. Enabled community involvement early in the design and implementation. Increasing interest on the technology. Large changes done to server core, replication framework and plugin. Suggests really fast development pace. 36

37 Comments Modular design and implementation modernized also the server core. Latest release presents two features/enhancements: MTS support (servers now are able to apply the group replication stream in parallel). Single Primary Mode (with automatic leader election on primary failure and groupwide information propagation). For those that do not want to deploy GR in multi-master mode. 37

38 Timeline Server GA and GR Releases Alright! MySQL MySQL MySQL crash-safe replication metadata Lets have a MySQL global look transaction Identifiers multi-threaded applier binary log group commit under the hood! MySQL semi-sync replication MySQL MySQL replication added to MySQL statement-based replication MySQL row-based replication large semi-sync enhancements 2nd generation of multi-threaded applier multi-source replication group replication server core changes XA support in replication 38

39 MySQL Group Replication Plugin Architecture General Architecture MySQL Server APIs: Capture / Apply / Lifecycle Capture Applier Recovery Replication Protocol Logics Group Communication System API Group Communication Engine (Paxos) Group Built on top of proven technology! Shares many pieces of async Replication. Multi-Master approach to replication. Built on reusable components! Layered implementation approach. Interface driven development. Decoupled from the server core. The plugin listens to server events. Capture procedure of async replication. Communication decoupled from the rest. 39

40 MySQL Group Replication Plugin Architecture Server and Plugin APIs MySQL Server APIs: Capture / Apply / Lifecycle Capture Applier Recovery Replication Protocol Logics Group Communication System API Group Communication Engine (Paxos) Group Server to Plugin: Lifecycle events: server start/stop, ready for connections, before/after commit,... Some semi-sync interfaces reused. New interfaces created. Internals are hidden from the plugin. Plugin to Server: Communicate fate of a transaction. Communicate GTID, Recovery Actions. Engage the relay log. 40

41 MySQL Group Replication Plugin Architecture Plugin Core MySQL Server APIs: Capture / Apply / Lifecycle Capture Applier Recovery Replication Protocol Logics Group Communication System API Group Communication Engine (Paxos) Maintains distributed execution context. Conflict detection. Distributed recovery: Handle membership changes; Donate state if needed; Collect state if needed. Inject transactions into the relay log. Decide the fate of on-going transactions. Group 41

42 MySQL Group Replication Plugin Architecture Generic Communication API MySQL Server APIs: Capture / Apply / Lifecycle Capture Applier Recovery Replication Protocol Logics Group Communication System API Group Communication Engine (Paxos) Abstracts the communication engine. Maps the interface to a specific communication engine. Group 42

43 MySQL Group Replication Plugin Architecture Group Communication Engine MySQL Server APIs: Capture / Apply / Lifecycle Capture Applier Recovery Replication Protocol Logics Group Communication System API Group Communication Engine (Paxos) Optimized Paxos implementation. Distributed agreement between servers. Totally ordered message delivery. Safe message delivery. Closed group communication. View synchrony. Group 43

44 Timeline Server GA and GR Releases Very well! MySQL MySQL MySQL crash-safe replication metadata MySQL global transaction Identifiers multi-threaded applier binary log group commit and how does it scale! MySQL semi-sync replication MySQL Lets check how it performs MySQL replication added to MySQL statement-based replication MySQL row-based replication large semi-sync enhancements 2nd generation of multi-threaded applier multi-source replication group replication server core changes XA support in replication 44

45 THROUGHPUT (TPS) Performance Group Replication Throughput (as perceived by the client application) Single-master Sustained Single-master Peak Multi-master Sustained Multi-master Peak Single-server (MySQL ) NUMBER OF SERVERS IN THE GROUP. Replication Performance blogs at: Peak Throughput The number of transactions that writers can propagate to the group (per second). Sustained Throughput The number of transactions that can be propagated to the group without increasing the replication lag on any node (per second). Servers 9 Dual Xeon E v3 Enterprise SSD Storage 10Gbps Ethernet Network Client 1 Dual Xeon E v3 10Gbps Ethernet Network Sysbench 0.5 RW workload 45

46 Performance On a sustained throughput: Multi-master performance degrades gracefully while going from a group with 2 servers to a group with 9 servers. Single primary performance degrades marginally when growing the group size. On a peak throughput: Multi-master exhibits 1.8X speedup when compared to the single server. Read load is balanced across the servers in the group. Write load is lower since execution is balanced across the group, whereas in single primary mode the primary becomes a bottleneck. With a single primary there is no lag on the other members. 46

47 Timeline Server GA and GR Releases MySQL MySQL MySQL MySQL replication added to MySQL statement-based replication The fine print? Here it is... MySQL row-based replication MySQL semi-sync replication MySQL crash-safe replication metadata global transaction Identifiers multi-threaded applier binary log group commit MySQL large semi-sync enhancements 2nd generation of multi-threaded applier multi-source replication group replication server core changes XA support in replication 47

48 Requirements (by design) Requires InnoDB storage engine. Requires a primary key on every table. Requires global transaction identifiers turned on. Requires binary log turned on. Requires binary log row format. Optimistic execution: transactions may abort on COMMIT. Maximum 9 servers in the group. Prohibited Serializable (on multi-master) Cascading Foreign Keys (on multi-master) Transaction savepoints Binary log events checksum Heads Up Concurrent DDL (on multi-master) 48

49 1 Background 2 Latest Developments in Replication GA 3 Latest Developments in Replication Plugins 4 Latest Developments in Replication Core 49

50 Timeline Server GA and GR Releases Last but not least, MySQL came out, and together with MySQL MySQL MySQL crash-safe replication metadata MySQL global transaction Identifiers multi-threaded applier MySQL binary log group commit replication features. added to MySQL Lets have a look. statement-based replication MySQL semi-sync replication MySQL it some interesting replication MySQL row-based replication large semi-sync enhancements 2nd generation of multi-threaded applier multi-source replication group replication server core changes XA support in replication 50

51 Cleaning up and Incremental Improvements START SLAVE supports other clauses than just SQL_BEFORE_GTIDS when multi-threaded is active. mysql> SET GLOBAL slave_parallel_workers=4; Query OK, 0 rows affected (0,00 sec) mysql> START SLAVE UNTIL MASTER_LOG_FILE="master-bin ", MASTER_LOG_POS=4; Query OK, 0 rows affected (0,00 sec) Removed deprecated binary log events from the codebase. 51

52 Stages to track Row-based replication progress. Three new performance schema stages added: Applying batch of row changes (write) Applying batch of row changes (update) Applying batch of row changes (delete) Progress for each stage is reported on the performance schema table EVENTS_STAGES_CURRENT. mysql> SELECT THREAD_ID, EVENT_NAME, WORK_COMPLETED, WORK_ESTIMATED FROM events_stages_current WHERE EVENT_NAME LIKE '%Applying batch of row%'\g *************************** 1. row *************************** THREAD_ID: 36 EVENT_NAME: stage/sql/applying batch of row changes (write) WORK_COMPLETED: 98 WORK_ESTIMATED: row in set (0,00 sec) 52

53 Provision to set GTID_PURGED. The user is now able to set GTID_PURGED without having to have GTID_EXECUTED empty. SET GLOBAL GTID_PURGED= + <gtid_set> Adds this range to the existing GTID_PURGED. SET GLOBAL GTID_PURGED= <gtid_set> Sets GTID_PURGED to the gtid_set provided. Noteworthy: gtid_set must be a super set of the existing GTID_PURGED do not forget GTIDs; gtid_set must not intersect the set of gtids in GTID_EXECUTED that have not yet been purged same GTID cannot be purged and not purged at the same time. 53

54 Provision to set GTID_PURGED. mysql> SELECT *************************** 1. row *************************** 1 row in set (0,00 sec) mysql> SET GLOBAL GTID_PURGED=+' :1'; Query OK, 0 rows affected (0,00 sec) mysql> *************************** 1. row :1 1 row in set (0,00 sec) mysql> SET GLOBAL GTID_PURGED=' :1-10'; Query OK, 0 rows affected (0,00 sec) mysql> *************************** 1. row : row in set (0,00 sec) 54

55 1 Background 2 Latest Developments in Replication GA 3 Latest Developments in Replication Plugins 4 Latest Developments in Replication Core 5 Roadmap 55

56 Timeline Server GA and GR Releases What now? MySQL MySQL MySQL crash-safe replication metadata Well, there is more MySQL global transaction to Identifiers come. multi-threaded applier MySQL binary log group commit replication We added are to MySQL living exciting times! statement-based replication MySQL semi-sync replication MySQL MySQL row-based replication large semi-sync enhancements 2nd generation of multi-threaded applier multi-source replication group replication server core changes XA support in replication 56

57 ReplicaSet 1 ReplicaSet 2 ReplicaSet 3 The End Goal: MySQL InnoDB Clusters Application MySQL Connector Application MySQL Connector Application MySQL Connector Application MySQL Connector MySQL Router MySQL Router MySQL Router MySQL Router M HA M HA M HA MySQL Shell M M M M M M S1 S2 S3 S4 S S1 S2 S3 S4 S S1 S2 S3 S4 57

58 The Road Ahead MySQL Document Store Relational & Document Model S1 Read Scale-Out Async Replication + Auto Failover S3 Timeline MySQL HA Out-of-the-box HA S2 Write Scale-Out Sharding S4 58

59 Specifically on the Replication Infrastructure! Production-ready MySQL Group Replication Then, continue to extend Group Replication and Group Communication functionality. MySQL Replication Usability Instrument even more replication and extend replication P_S tables. Simpler administrative commands, more online and distributed operations. MySQL Replication Performance Continue to improve replication stream multi-threaded (slave) applier. Continue to improve replication stream pipeline performance and efficiency. 59

60 1 Background 2 Latest Developments in Replication GA 3 Latest Developments in Replication Plugins 4 Latest Developments in Replication Core Roadmap 5 Summary 6 60

61 Summary Replication feature set in MySQL 5.7 shows many appealing features and infrastructure to build more advanced solutions based on MySQL technology. Group Replication plugin is the next big thing in MySQL Replication horizon. High Availability, Multi-master, Conflict detection Performance, usability, flexibility, all in one package. MySQL already shows some interesting replication features, such as more observability and flexibility. The vision and roadmap are sound and clearly show where MySQL Replication is heading. 61

62 Where to go from here? Packages Reference Documentation Blogs from the Engineers (news, quirks, technical information and much more) Technical documentation about Group Replication (by the engineers): 62

63