MySQL Multi-Source Replication

Transcription

1 MySQL Multi-Source Replication Max Bubenick - max.bubenick@percona.com Technical Operations Manager Wagner Bianchi - wagner.bianchi@percona.com Principal Technical Services Engineer

2 This is gonna be a multi-source replication walkthrough on MariaDB and Percona Server

3 enables a replication slave to receive transactions from multiple sources/masters simultaneously; aggregate data from multiple servers - Data Mart/Data Warehouse; merge table shards - no auto_increment conflict control; can be carefully used connecting many location on one geo-positioned cluster; centralize all data for backup propose.

4

5

6 No gtid_mode variable to turn on GTID Binary logs has positions and GTIDs CHANGE MASTER TO CHANGE MASTER 'foo' TO CHANGE MASTER TO MASTER_USE_GTID=CURRENT_POS SLAVE_POS Accessed using SHOW ALL SLAVES STATUS; SET works well; set show status like 'Slave_running'; set show status like 'Slave_running'; -- default slave connection -- named connection name

7 box03 box02 On the multi-source slave side: box04 box01 Multi-Source Slave Use SHOW ALL SLAVES STATUS\G Use Use STOP/START ALL SLAVES Use STOP/START SLAVE 'box02' MariaDB [(none)]> pager egrep "Connection Gtid" PAGER set to 'egrep "Connection Seconds"' MariaDB [(none)]> show all slaves status\g Connection_name: box02 Gtid_IO_Pos: ,3-3-1 Gtid_Slave_Pos: ,3-3-1 Connection_name: box03 Gtid_IO_Pos: ,3-3-1 Gtid_Slave_Pos: ,3-3-1 Connection_name: box04 Gtid_IO_Pos: ,3-3-1 Gtid_Slave_Pos: , rows in set (0.00 sec)

8 #: box01 - multi-source slave [client] port=3306 socket=/var/lib/mysql/mysql.sock [mysqld] user=mysql port=3306 socket=/var/lib/mysql/mysql.sock basedir=/usr datadir=/var/lib/mysql read_only=1 #: repl vars server_id=1 report_host=box01 report_port=3306 report_user=repl log_bin=mysql-bin log_bin_index=mysql.index log_slave_updates=true binlog_format=row #: verify checksum on master master_verify_checksum=1 #: gtid vars gtid_domain_id=1 gtid_ignore_duplicates=on gtid_strict_mode=1 #: msr slave parallel mode * box02.slave_parallel_mode=optimistic box03.slave_parallel_mode=optimistic box04.slave_parallel_mode=optimistic #: other slave variables slave_parallel_threads=16 slave_domain_parallel_threads=4 slave_parallel_max_queued=512m slave_net_timeout=15 slave_sql_verify_checksum=1 slave_compressed_protocol=1 #: binary log group commit behavior #binlog_commit_wait_usec= #binlog_commit_wait_count=20

9 gtid_domain_id=3 gtid_domain_id=2 gtid_domain_id=4 #: box01 MariaDB [(none)]> select *************** 1. row *************************** row in set (0.00 sec) #: box02 MariaDB [(none)]> *************** 1. row 2 1 row in set (0.00 sec) #: box03 MariaDB [(none)]> *************** 1. row 3 1 row in set (0.00 sec) box01 Multi-Source Slave gtid_domain_id=1 #: box04 MariaDB [(none)]> *************** 1. row 4 1 row in set (0.00 sec)

10 Make sure the replication user is set on all the servers; Make sure all the servers has unique server_id and gtid_domain_id; #: Connection name with box02 MariaDB [(none)]> change master 'box02' to master_host=' ', master_user='repl', master_use_gtid=current_pos; #: Connection name with box03 MariaDB [(none)]> change master 'box03' to master_host=' ', master_user='repl', master_use_gtid=current_pos; #: Connection name with box04 MariaDB [(none)]> change master 'box04' to master_host=' ', master_user='repl', master_use_gtid=current_pos;

11 # start all slaves MariaDB [(none)]> start all slaves; Query OK, 0 rows affected, 3 warnings (0.02 sec) MariaDB [(none)]> show warnings; Level Code Message Note 1937 SLAVE 'box04' started Note 1937 SLAVE 'box03' started Note 1937 SLAVE 'box02' started rows in set (0.00 sec) MariaDB [(none)]> stop all slaves; Query OK, 0 rows affected, 3 warnings (0.02 sec)

12 #: setting the MariaDB [(none)]> set Query OK, 0 rows affected (0.00 sec) #: showing some box02 Connection name's status replication variables MariaDB [(none)]> pager egrep "Slave_IO_State Using_Gtid Gtid_IO_Pos" PAGER set to 'egrep "Slave_IO_State Using_Gtid Gtid_IO_Pos"' MariaDB [(none)]> show slave status\g Slave_IO_State: Waiting for master to send event Using_Gtid: Current_Pos Gtid_IO_Pos: ,2-2-1, row in set (0.00 sec) #: starting and stopping just box02, as set MariaDB [(none)]> stop slave; Query OK, 0 rows affected (0.01 sec) MariaDB [(none)]> start slave; Query OK, 0 rows affected (0.02 sec)

13 #: relay logs - one group -rw-rw mysql mysql -rw-rw mysql mysql -rw-rw mysql mysql -rw-rw mysql mysql -rw-rw mysql mysql -rw-rw mysql mysql -rw-rw mysql mysql -rw-rw mysql mysql -rw-rw mysql mysql for each set connection name - host+relay-bin+connection_name Mar 23 00:32 maria01-relay-bin-box Mar 23 00:32 maria01-relay-bin-box Mar 23 00:32 maria01-relay-bin-box02.index Mar 23 00:32 maria01-relay-bin-box Mar 23 00:32 maria01-relay-bin-box Mar 23 00:32 maria01-relay-bin-box03.index Mar 23 00:32 maria01-relay-bin-box Mar 23 00:32 maria01-relay-bin-box Mar 23 00:32 maria01-relay-bin-box04.index #: master.info - one per set connection name - master-connection_name.info -rw-rw mysql mysql 153 Mar 23 00:32 master-box02.info -rw-rw mysql mysql 153 Mar 23 00:32 master-box03.info -rw-rw mysql mysql 153 Mar 23 00:32 master-box04.info #: multi-master.info file, listing all set connections names -rw-rw mysql mysql 18 Feb 15 11:00 multi-master.info [root@maria01 mysql]# cat multi-master.info #: don't edit this file :) box02 box03 box04

14 #: check gtid_current and gtid_slave pos variables MariaDB [mysql]> select *************************** 1. row *************************** ,2-2-1, ,2-2-1, row in set (0.00 sec) #: check current clave current and slave pos from mysql.gtid_slave_pos table MariaDB [mysql]> select * from mysql.gtid_slave_pos; domain_id sub_id server_id seq_no rows in set (0.00 sec)

15 #: checking global variable MariaDB [(none)]> SELECT *************************** 1. row *************************** , , , help keeping binlogs identical across multiple servers 1 row in set (0.00 sec) CHANGE MASTER 'foo' TO

16 TO CHANGE MASTER CHANGE MASTER 'foo' TO MASTER_USE_GTID=SLAVE_POS; #: checking global variable MariaDB [(none)]> SELECT *************************** 1. row *************************** , , , row in set (0.00 sec) #: checking global (can't be set per Connection Name) MariaDB [(none)]> SET GLOBAL GTID_SLAVE_POS='1-1-66, , , '; Query OK, 0 rows affected, 3 warnings (0.01 sec) #: SHOULD BE DONE AFTER STOPPING REPLICATION CONNECTION NAMES...

17 #: comparing both and MariaDB [(none)]> SELECT *************************** 1. row *************************** , , , ADDITIONAL , , , row in set (0.00 sec) #: setting - TRANSACTIONS CANNOT BE REPLAYED MariaDB [(none)]> SET GLOBAL GTID_SLAVE_POS='1-1-66, , , '; ERROR 1947 (HY000): Specified GTID conflicts with the binary log which contains a more recent GTID If MASTER_GTID_POS=CURRENT_POS is used, the binlog position will override the new value #: command above was embraced by START/STOP ALL SLAVES.

18 slave_parallel_threads sets the number of threads will take care about the updates on relay logs based on their timestamps;

19 #: making a multi-source slave multi-threaded MariaDB [(none)]> stop all slaves; Query OK, 0 rows affected, 3 warnings (0.00 sec) MariaDB [(none)]> set global slave_parallel_threads=12; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> set global slave_domain_parallel_threads=4; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> start all slaves; Query OK, 0 rows affected, 3 warnings (0.03 sec) MariaDB [(none)]> *************************** 1. row optimistic # will retry transaction in case of parallelism 12 # total of threads available for slave to execute relay 4 # minimum # of thread used for a domain_id all time 1 row in set (0.00 sec)

20 #: making a multi-source slave multi-threaded MariaDB [(none)]> SELECT ID,TIME,STATE,USER FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER='system user'; ID TIME STATE USER Waiting for master to send event system user Slave has read all relay log; waiting for the slave I/O thread t system user Waiting for master to send event system user Slave has read all relay log; waiting for the slave I/O thread t system user Slave has read all relay log; waiting for the slave I/O thread t system user Waiting for master to send event system user Waiting for work from SQL thread system user Waiting for work from SQL thread system user 10 0 Update_rows_log_event::ha_update_row(-1) system user 9 0 Unlocking tables system user Waiting for work from SQL thread system user Waiting for work from SQL thread system user 6 0 Update_rows_log_event::ha_update_row(-1) system user 5 0 Update_rows_log_event::ha_update_row(-1) system user 4 0 Update_rows_log_event::ha_update_row(-1) system user 3 0 Update_rows_log_event::ha_update_row(-1) system user rows in set (0.07 sec)

21 binlog_commit_wait_usec= binlog_commit_wait_count=20 #: binary logs for group commit mysql]# mysqlbinlog mysql-bin vvvv egrep "cid=353579" # :37:27 server id 2 end_log_pos GTID cid= # :37:27 server id 2 end_log_pos GTID cid= # :37:27 server id 2 end_log_pos GTID cid= # :37:27 server id 2 end_log_pos GTID cid= # :37:27 server id 2 end_log_pos GTID cid= # :37:27 server id 2 end_log_pos GTID cid= trans trans trans trans trans trans MariaDB [(none)]> show global status where variable_name in ('Binlog_commits','Binlog_group_commits'); Variable_name Value Binlog_commits the bigger the difference between the two variables Binlog_group_commits 5523 the bigger the apparent group commit efficiency rows in set (0.02 sec)

22

23 One can restart both threads or just SQL_THREAD; #: adding new schema to box02 box02> set sql_log_bin=0; create database if not exists box02_new; set sql_log_bin=1; Query OK, 0 rows affected (0.00 sec) Query OK, 1 row affected (0.01 sec) Query OK, 0 rows affected (0.00 sec) #: adding filters on multi-source slave box01> stop slave 'box02'; set global box02.replicate_ignore_db='box02_new'; start slave 'box02'; Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) #: remove filter box01> stop slave; set global box02.replicate_ignore_db=''; start slave; Query OK, 0 rows affected (0.01 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.11 sec)

24 sql_slave_skip_counter If parallel replication is enabled: MariaDB [box02]> set default_master_connection='box02'; set global sql_slave_skip_counter=1; Query OK, 0 rows affected (0.00 sec) ERROR 1966 (HY000): When using parallel replication and GTID with multiple replication can not be used. Instead, explicitly can be used to skip to after a given GTID position. Stop all slaves, turn off the parallel replication and skip_counter; Set a new value MariaDB [box02]> stop all slaves; , , '; Query OK, 0 rows affected, 3 warnings (0.00 sec) Query OK, 0 rows affected (0.01 sec) MariaDB [box02]> start all slaves; Query OK, 0 rows affected, 3 warnings (0.01 sec)

25

26 gtid_mode=on PERFORMANCE_SCHEMA SHOW SLAVE STATUS SHOW SLAVE STATUS FOR CHANNEL mysql> pager egrep "Slave_IO Channel" PAGER set to 'egrep "Slave_IO Channel"' mysql> show slave status\g Slave_IO_State: Waiting for master to send event Slave_IO_Running: Yes Channel_Name: box02 Slave_IO_State: Waiting for master to send event Slave_IO_Running: Yes Channel_Name: box03 Slave_IO_State: Waiting for master to send event Slave_IO_Running: Yes Channel_Name: box04 3 rows in set (0.00 sec)

27 box03 box02 box04 box01 Multi-Source Slave #: multi-source slave configuration file [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock user=mysql report_host=box01 report_port=3306 report_user=repl #: gtid configuration server_id=1 gtid_mode=on log_bin=mysql-bin log_bin_index=mysql-bin.index log_slave_updates=true enforce_gtid_consistency #: binary logs (GROUP COMMIT) binlog_group_commit_sync_delay= binlog_group_commit_sync_no_delay_cont=20 #binlog_order_commits=0 # increase parallelism #: repositories for crash-safe master_info_repository=table relay_log_info_repository=table #: slave configs slave_parallel_workers=4 slave_parallel_type='logical_clock' slave_compressed_protocol=1 slave_pending_jobs_size_max=256m

28 #: Replication Channel for box02 box01> change master to master_host=' ', master_user='repl', master_auto_position=1 for channel 'box02'; #: Replication Channel for box03 box01> change master to master_host=' ', master_user='repl', master_auto_position=1 for channel 'box03'; #: Replication Channel for box04 box01> change master to master_host=' ', master_user='repl', master_auto_position=1 for channel 'box04';

29 #: relay logs - one group -rw-r mysql mysql -rw-r mysql mysql -rw-r mysql mysql -rw-r mysql mysql -rw-r mysql mysql -rw-r mysql mysql -rw-r mysql mysql -rw-r mysql mysql -rw-r mysql mysql for each set connection name Mar 27 00:51 percona01-relay-bin-box Mar 27 00:51 percona01-relay-bin-box Mar 27 00:51 percona01-relay-bin-box02.index Mar 27 00:51 percona01-relay-bin-box Mar 27 00:51 percona01-relay-bin-box Mar 27 00:51 percona01-relay-bin-box03.index Mar 27 00:51 percona01-relay-bin-box Mar 27 00:51 percona01-relay-bin-box Mar 27 00:51 percona01-relay-bin-box04.index #:No master.info file as it needs to be configured with crash-safe, repos as TABLE mysql> show variables where variable_name in ('master_info_repository','relay_log_info_repository'); Variable_name Value master_info_repository TABLE relay_log_info_repository TABLE rows in set (0.00 sec)

30 #: stopping mysql> stop Query OK, 0 mysql> show all slaves slave; rows affected (0.06 sec) slave status\g Slave_IO_Running: No Slave_SQL_Running: No [...snip...] 3 rows in set (0.00 sec) #: stopping just one specific slave mysql> stop slave for channel 'box02'; Query OK, 0 rows affected (0.01 sec) mysql> show slave status for channel 'box02'\g Slave_IO_Running: No Slave_SQL_Running: No Slave_SQL_Running_State: 1 row in set (0.00 sec)

31 #: starting all slaves mysql> start slave; Query OK, 0 rows affected (0.06 sec) mysql> show slave status\g Slave_IO_Running: Yes Slave_SQL_Running: Yes [...snip...] 3 rows in set (0.00 sec) #: starting just one specific slave mysql> start slave for channel 'box02'; Query OK, 0 rows affected (0.01 sec) mysql> show slave status for channel 'box02'\g Slave_IO_Running: No Slave_SQL_Running: No Slave_SQL_Running_State: 1 row in set (0.00 sec)

32 #: show all slaves status mysql> show slave status\g Channel_Name: box02 Channel_Name: box03 Channel_Name: box04 3 rows in set (0.00 sec) #: show slave status for a specific slave mysql> show slave status for channel 'box02'\g Slave_IO_State: Waiting for master to send event Retrieved_Gtid_Set: 61be13a1-d574-11e5-83c fb806: Executed_Gtid_Set: 4bd77dee-d572-11e5-b09f fb806: Channel_Name: box02 1 row in set (0.00 sec)

33 #: PERFORMANCE_SCHEMA replication tables mysql> show tables from performance_schema like 'replication%'; Tables_in_performance_schema (replication%) replication_applier_configuration replication_applier_status replication_applier_status_by_coordinator replication_applier_status_by_worker replication_connection_configuration replication_connection_status replication_group_member_stats replication_group_members rows in set (0.00 sec)

34 #: PERFORMANCE_SCHEMA replication tables mysql> select channel_name,service_state,last_heartbeat_timestamp -> from performance_schema.replication_connection_status\g *************************** 1. row *************************** channel_name: box02 service_state: ON last_heartbeat_timestamp: :22:39 *************************** 2. row *************************** channel_name: box03 service_state: ON last_heartbeat_timestamp: :22:30 *************************** 3. row *************************** channel_name: box04 service_state: ON last_heartbeat_timestamp: :22:31 3 rows in set (0.00 sec)

35 mysql> select LOGICAL_CLOCK row in set (0.00 sec) #: PERFORMANCE_SCHEMA, checking coordinators state (multi-threaded slaves) mysql> select channel_name, service_state -> from performance_schema.replication_applier_status_by_coordinator\g *************************** 1. row *************************** channel_name: box02 service_state: ON *************************** 2. row *************************** channel_name: box03 service_state: ON *************************** 3. row *************************** channel_name: box04 service_state: ON 3 rows in set (0.00 sec)

36 #: PERFORMANCE_SCHEMA, checking coordinators state (multi-threaded slaves) mysql> select channel_name,thread_id,service_state,last_seen_transaction -> from performance_schema.replication_applier_status_by_worker -> where last_seen_transaction<>''\g *************************** 1. row *************************** channel_name: box03 thread_id: 71 service_state: ON last_seen_transaction: ab d573-11e5-bc fb806:2536 *************************** 2. row *************************** channel_name: box03 thread_id: 72 service_state: ON last_seen_transaction: ab d573-11e5-bc fb806:2537 *************************** 3. row *************************** channel_name: box03 thread_id: 73 service_state: ON last_seen_transaction: ab d573-11e5-bc fb806:

37 CHANGE REPLICATION FILTER #: adding new schema to box02 box02> set sql_log_bin=0; create database if not exists box02_new; set sql_log_bin=1; Query OK, 0 rows affected (0.00 sec) Query OK, 1 row affected (0.01 sec) Query OK, 0 rows affected (0.00 sec) #: adding filters on multi-source slave mysql> stop slave; change replication filter replicate_ignore_db=(box02_new); start slave; Query OK, 0 rows affected (0.01 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.11 sec) #: remove filter mysql> stop slave; change Query OK, 0 rows affected Query OK, 0 rows affected Query OK, 0 rows affected replication filter replicate_ignore_db=(); start slave; (0.01 sec) (0.00 sec) (0.11 sec)

38 REPLICATE_DO_DB REPLICATE_IGNORE_DB REPLICATE_DO_TABLE REPLICATE_IGNORE_TABLE REPLICATE_WILD_DO_TABLE REPLICATE_WILD_IGNORE_TABLE REPLICATE_REWRITE_DB

39 REPLICATE_REWRITE_DB #: rewriting updates on db A to db B box01> stop slave; change replication filter replicate_rewrite_db=((box02,box03)); start slave; Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.07 sec) #: writing some data to A to be routed to B box02> insert into box02.t1 set i=10; Query OK, 1 row affected (1.01 sec) #: checking data on A box01> select * from box02.t1; Empty set (0.00 sec) #: checking data on B box01> select * from box03.t1\g *************************** 1. row *************************** i: 10 1 row in set (0.00 sec) #: remove filter box01> stop slave; change Query OK, 0 rows affected Query OK, 0 rows affected Query OK, 0 rows affected replication filter replicate_rewrite_db=(); start slave; (0.01 sec) (0.00 sec) (0.12 sec)

40 box01> SELECT -> FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER='system user'; ID USER STATE TIME INFO box01 32 system user Waiting for an event from Coordinator 721 NULL box01 33 system user Waiting for an event from Coordinator 721 NULL box01 34 system user Waiting for an event from Coordinator 721 NULL box01 35 system user Waiting for an event from Coordinator 721 NULL box01 36 system user Waiting for an event from Coordinator 721 NULL box01 25 system user Waiting for an event from Coordinator 721 NULL box01 26 system user Waiting for an event from Coordinator 721 NULL box01 27 system user Waiting for master to send event 721 NULL [...snip...] rows in set (0.00 sec)

41 UUID() mysql> select * from performance_schema.replication_applier_status_by_worker\g *************************** 1. row *************************** CHANNEL_NAME: box02 WORKER_ID: 1 THREAD_ID: NULL SERVICE_STATE: OFF LAST_SEEN_TRANSACTION: fa11b361-d572-11e5-b63e fb806:66 LAST_ERROR_NUMBER: 1062 LAST_ERROR_MESSAGE: Worker 0 failed executing transaction fa11b361-d572-11e5-b63e' fb806:66' at master log mysql-bin , end_log_pos 793; Could not execute Write_rows event on table box02.t1; Duplicate entry '1' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log mysql-bin , end_log_pos 793 LAST_ERROR_TIMESTAMP: :18:14 mysql> stop commit; set Query OK, 0 Query OK, 0 Query OK, 0 Query OK, 0 slave for channel 'box02'; set gtid_next=' fa11b361-d572-11e5-b63e fb806:66'; begin; gtid_next=automatic; start slave for channel 'box02'; rows affected (0.00 sec) rows affected (1.01 sec) rows affected (0.00 sec) rows affected (0.02 sec)

42 Feature Multi-Source Slave Creation Parallel Threads Replication Filters Skip Replication Errors Possible number of sources/masters MariaDB MySQL 5.7 CHANGE MASTER 'name' TO...; CHANGE MASTER TO FOR CHANNEL 'name'; slave_parallel_mode=optimistic slave_parallel_threads=16 # ALL slave_domain_parallel_threads=4 slave_parallel_max_queued=512m slave_parallel_workers=4 # per RC slave_parallel_type='logical_clock' slave_pending_jobs_size_max=256m set global box02. replicate_ignore_db='foo'; SET GLOBAL GTID_SLAVE_POS='1-1-66'; SET GLOBAL slave_exec_mode='idempotent'; SET GLOBAL sql_slave_skip_counter=1; You can for now only have 64 masters CHANGE REPLICATION FILTER... SET GLOBAL slave_exec_mode='idempotent'; SET GTID_NEXT='UUID:TRX_ID' SET GLOBAL sql_slave_skip_counter=1; start slave for channel 'xxxx'; (NO GTID) 256 replication channels for any combination of hostname and port

43