Setting up Multi-Source Replication in MariaDB 10.0

Transcription

1 Setting up Multi-Source Replication in MariaDB 10.0 November 3, 2014 Derek Downey MySQL Principal Consultant

2 Who am I? Web Developer and Sysadmin background MySQL DBA for 10+ years MySQL Principal Consultant at Pythian

3 About Pythian 10,000 Pythian currently manages more than 10,000 systems. 300 Pythian currently employs more than 300 people in 25 countries worldwide Pythian was founded in leading brands trust us to keep their systems fast, reliable, and secure Elite DBA & SysAdmin workforce: 7 Oracle ACEs, 2 Oracle ACE Directors, 5 Microsoft MVPs, 1 Cloudera Champion of Big Data More than any other company, regardless of head count Oracle, Microsoft, MySQL, Hadoop, Cassandra, MongoDB, and more. Infrastructure, Cloud, DevOps, and application expertise Big data practice includes architects, R&D, data scientists, and operations capabilities Zero lock-in, utility billing model, easily blended into existing teams.

4

5

6 Overview What is multi-source replication? Why would we use multi-source replication? How multi-source replication is implemented in MariaDB Introduction to Vagrant / AWS environment Set up mirrored environment Work through some issues, conflicts and failures Set up a fan-in environment Limitations, issues, and missing features Q&A

7 Hands-on access Vagrant installation requires VirtualBox and Vagrant 1.5+ git:

8 What is Multi-source Replication

9 Multi-source Replication

10 Why Multi-source Replication

11 MySQL Topology Fragile Doesn t scale Stand Alone But pity anyone who falls and has no one to help them up - Ecclesiastes 4:10

12 Concept #1 - binary log Contains DDL/DML executed (successfully?) on master Required for replication Point-in-time recovery

13 MySQL Topology Slave roles Failover Read scaling Master/Slave

14 Concept #2 - asynchronous Changes on master don t wait for acknowledgement from slave Slave lag: unexpected results from read slaves At best: semi-synchronous (5.5+)

15 MySQL Topology Can affect performance of master Complexity of failover Master/Slave

16 MySQL Topology Master/Slave

17 MySQL Topology Active Master/Passive Master

18 MySQL Topology Active Master/Passive Master

19 MySQL Topology Active Master/Active Master

20 MySQL Topology Disaster Recovery

21 MySQL Topology Mirrored

22 Concept #3 MySQL slaves can only have a single master

23 MySQL Topology Ring

24 MySQL Topology Shards

25 MySQL Topology Fan-in

26 MariaDB Multi-source Replication Implementation

27 MariaDB Multi-Source Replication Still requires binary log Still asynchronous Works with either binlog coordinates or MariaDB GTID Removes single master constraint No conflict resolution Replication filters can be different per connection Relay log and info files are generated per connection

28 MariaDB New Syntax CHANGE MASTER [ connec2on_name ]... FLUSH RELAY LOGS ['connec4on_name'] RESET SLAVE ['connec2on_name'] SHOW RELAYLOG ['connec4on_name'] EVENTS SHOW SLAVE ['connec2on_name'] STATUS SHOW ALL SLAVES STATUS START SLAVE ['connec2on_name'...]] START ALL SLAVES STOP SLAVE ['connec2on_name'] STOP ALL SLAVES...

29 MariaDB Variables default_master_connection: connection_name to use if an explicit name is not provided is dynamic log-basename: use to avoid missing files (error, binlog, relaylog, pid, socket, etc) replication filters: prefix connection: main_connection.replicate_do_db if no prefix, variable is used for all connections that don t have variable set.

30 MariaDB New Files multi-master-info-file: contains all master connections in use master--(connection).info: Current master position for the connection that has been applied in the slave relay-log-(connection).xxxxx: relay logs for a given connection relay-log-index-(connection).index: All relay logs for a given connection relay-log-(connection).info: Current master position for the relay log (not applied)

31 MariaDB New Status Variables Com_start_all_slaves Com_start_slave: replaces Com_slave_start Com_stop_slave: replaces Com_slave_stop Com_stop_all_slaves

32 GTID Replication 3 numbers separated by dashes: eg first number: domain id second number: server id third number: sequence number; increment per event Domain IDs: must be unique on multi-masters to guarantee binlog order of each stream slaves keep track of binlog position for each domainid

33 GTID Replication Can switch from normal coordinates to GTID after slave first connects to a master that tracks GTIDs MASTER_USE_GTID: slave_pos: last replicated gtid (for the domain) current_pos: includes events in master binlog no: not using gtid

34 GTID: slave_pos vs current_pos slave_pos: SELECT last replicated gtid Use if changes on slave should not affect GTID replication position current_pos: SELECT includes events in binlog not replicated; server was previously a master no need to know previous role of server

35 Vagrant Instances Hands-on

36 Logging In Vagrant # star4ng/stopping instances $ cd pluk- mariadb $ vagrant up $ vagrant ssh pluk $ sudo su - pluk

37 Environment Overview mysqld_mul2: 4 instances config file: /etc/my.cnf datadir: /opt/mysql/(instance) master1: port 3306 slave1: port 3307 master2: port 3308 slave2: port 3309

38 Interacting with MariaDB # star4ng/stopping instances $ sudo service mysqld_mul4 {start stop restart report} [1-4] # connec4on bash aliases $ mysql_master1 $ mysql_slave1 $ mysql_master2 $ mysql_slave2

39 Setting up a Mirrored Environment Hands-on

40 MySQL Topology Mirrored

41 Script Repository pluk$ cd /home/pluk/scripts/mirrored

42 slave1 - slave of master1 pluk$ cat 1_slave1.sql pluk$ mysql_slave1 (root@slave1.sock) [(none)]> CHANGE MASTER 'master1' TO MASTER_HOST=' ', - - can t use unix socket MASTER_PORT = 3306, - - master1 MASTER_USER = 'repl', MASTER_PASSWORD = 'mul4_repl', MASTER_USE_GTID = current_pos; (root@slave1.sock) [(none)]>start SLAVE 'master1';

43 slave1 - slave of master1 (root@slave1.sock) [(none)]>start SLAVE 'master1'; (root@slave1.sock) [(none)]>show SLAVE STATUS \G Empty set (0.00 sec)

44 slave1 - default_master_connection (root@slave1.sock) row in set (0.00 sec) (root@slave1.sock) Query OK, 0 rows affected (0.00 sec) (root@slave1.sock) [(none)]>show SLAVE STATUS \G

45 slave1 - multiple files (root@slave1.sock) [(none)]>\! ls - l /opt/mysql/slave1/*info - rw- rw mysql mysql 110 Oct 22 00:42 /opt/mysql/slave1/master- master1.info - rw- rw mysql mysql 8 Oct 22 00:26 /opt/mysql/slave1/mul4- master.info - rw- rw mysql mysql 61 Oct 22 00:42 /opt/mysql/slave1/relay- log- master1.inf

46 master1 - slave of slave1 pluk$ cat 2_master1_s1.sql pluk$ mysql_master1 (root@master1.sock) [(none)]> CHANGE MASTER 'slave1' TO MASTER_HOST=' ', MASTER_PORT = 3307, - - slave1 MASTER_USER = 'repl', MASTER_PASSWORD = 'mul4_repl', MASTER_USE_GTID = current_pos; (root@master1.sock) [(none)]>start SLAVE 'slave1'; (root@master1.sock) [(none)]>show SLAVE 'slave1' STATUS \G

47 slave2 - slave of master2 pluk$ cat 3_slave2.sql pluk$ mysql_slave2 (root@slave2.sock) [(none)]> CHANGE MASTER 'master2' TO MASTER_HOST=' ', MASTER_PORT = 3308, - - master2 MASTER_USER = 'repl', MASTER_PASSWORD = 'mul4_repl', MASTER_USE_GTID = current_pos; (root@slave2.sock) [(none)]>start SLAVE 'master2'; (root@slave2.sock) [(none)]>show SLAVE 'master2' STATUS \G

48 master2 - slave of slave2 pluk$ cat 4_master2_s2.sql pluk$ mysql_master2 (root@master2.sock) [(none)]> CHANGE MASTER 'slave2' TO MASTER_HOST=' ', MASTER_PORT = 3309, - - slave2 MASTER_USER = 'repl', MASTER_PASSWORD = 'mul4_repl', MASTER_USE_GTID = current_pos; (root@master2.sock) [(none)]>start SLAVE 'slave2'; (root@master2.sock) [(none)]>show SLAVE 'slave2' STATUS \G

49 MySQL Topology Pairs

50 MySQL Topology Mirrored

51 master2 - slave of master1 pluk$ cat 5_master2_m1.sql pluk$ mysql_master2 (root@master2.sock) [(none)]> CHANGE MASTER 'master1' TO MASTER_HOST=' ', MASTER_PORT = 3306, - - master1 MASTER_USER = 'repl', MASTER_PASSWORD = 'mul4_repl', MASTER_USE_GTID = current_pos; (root@master2.sock) [(none)]>start SLAVE 'master1'; (root@master2.sock) [(none)]>show SLAVE 'master1' STATUS \G

52 master1 - slave of master2 pluk$ cat 6_master1_m2.sql pluk$ mysql_master1 (root@master1.sock) [(none)]> CHANGE MASTER 'master2' TO MASTER_HOST=' ', MASTER_PORT = 3308, - - master2 MASTER_USER = 'repl', MASTER_PASSWORD = 'mul4_repl', MASTER_USE_GTID = current_pos; (root@master1.sock) [(none)]>start SLAVE 'master2'; (root@master1.sock) [(none)]>show SLAVE 'master2' STATUS \G

53 MySQL Topology Mirrored

54 Handling Failures Hands-on

55 MySQL Topology Mirrored

56 Multi-master Conflict STOP SLAVE; SET GTID_NEXT="7d72f9b e2-a3d ef5:5"; BEGIN; COMMIT; SET GTID_NEXT="AUTOMATIC"; START SLAVE; [...] Retrieved_Gtid_Set: 7d72f9b e2-a3d ef5:1-5 Executed_Gtid_Set: 7d72f9b e2-a3d ef5:1-5 Source:

57 Multi-master Conflict SET %connection% ; SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; START SLAVE;

58 master1 failure pluk$ sudo service mysqld_mul4 stop 1 pluk$ sudo service mysqld_mul4 report (root@master2.sock) [(none)]> master1 (root@master2.sock) [(none)]> STOP SLAVE; (root@master2.sock) [(none)]> CHANGE MASTER TO MASTER_PORT=3307; (root@master2.sock) [(none)]> START SLAVE; (root@slave1.sock) [(none)]> master1 (root@slave1.sock) [(none)]> STOP SLAVE; (root@slave1.sock) [(none)]> CHANGE MASTER TO MASTER_PORT=3308; (root@slave1.sock) [(none)]> START SLAVE;

59 Setting up a Fan-in Environment Hands-on

60 MySQL Topology Fan-in

61 Fan-in Reset environment pluk$ cd /home/pluk/scripts/fan- in pluk$ bash reset_environment.sh

62 slave1 - slave of master1 (root@slave1.sock) [(none)]>change MASTER 'master1' TO MASTER_HOST=' ', MASTER_PORT = 3306, - - master1 MASTER_USER = 'repl', MASTER_PASSWORD = 'mul4_repl', MASTER_USE_GTID = current_pos;

63 slave1 - slave of master2 (root@slave1.sock) [(none)]> CHANGE MASTER 'master2' TO MASTER_HOST=' ', MASTER_PORT = 3308, - - master2 MASTER_USER = 'repl', MASTER_PASSWORD = 'mul4_repl', MASTER_USE_GTID = current_pos;

64 Limitations and Missing Features

65 Limitations Currently only 64 masters slave_net_timeout affects all connections, so doesn t check against slave_heartbeat_period replication/standard-replication/multi-source-replication/ #limitations

66 Importable Tablespaces Ability to importing innodb file_per_table tablespaces from a different server Useful for extremely large datasets where logical restore would take too long Downside of feature: doesn t support partitions. (root@slave1.sock) [foo]>alter TABLE e DISCARD TABLESPACE; ERROR 1031 (HY000): Storage engine InnoDB of the table `foo`.`e` doesn't have this op4on

67 Semi-sync Replication Master waits for acknowledgement of receipt from a semi-sync slave before returning to client Gain: reduced chance of lost transactions Lose: performance Currently slated 10.2:

68 Complexity Monitoring multiple connections Connection naming for failover purposes Careful with duplicated events; gtid_ignore_duplicates=1 Skipping events won t progress gtid sequence on slaves

69 Resources MariaDB multi-source docs: replication/standard-replication/multi-source-replication MariaDB GTID docs: replication/standard-replication/global-transaction-id MariaDB bug database: MySQL multi-source worklog (for MySQL 5.7):

70 Resources Kristian Nielsen blogs on replication streams and GTID

71 Q&A twitter: