MariaDB Enterprise Cluster. MariaDB Training

Size: px

Start display at page:

Download "MariaDB Enterprise Cluster. MariaDB Training"

Nickolas Gilbert
6 years ago
Views:

1 MariaDB Enterprise Cluster MariaDB Training

2 Introduction MariaDB Enterprise Cluster Introduction State Transfer Schema Changes Load Balancing Architecture Installation Caveats Multi-Master Conflicts Back-Ups Recovery Advanced Topics Conclusion

and Venture Capital About 100 Employees Several

3 Introducing MariaDB Ab Founders from MySQL the Company and Community Funded by Founders, Employees, and Venture Capital About 100 Employees Several former MySQL Employees and Community Members in over 14 Countries 3

4 Personal Introductions Instructor Name and Background Participants Name and Company MariaDB Experience How You Use MariaDB Needs Related to Course Topics 4

5 Class Schedule & Personal Concerns Starting and Ending Times Planned Breaks On- Site Location of Rest Rooms Smoking Areas Snacks and Drinks LVC Classes Chat with Everyone 5

6 Course Outline Overview Architecture Load Balancing Advanced Topics Installation State Transfer Caveats Multi-Master Conflicts Schema Changes Back-Ups Recovery 6

7 Architecture MariaDB Enterprise Cluster Introduction State Transfer Schema Changes Load Balancing Architecture Installation Caveats Multi-Master Conflicts Back-Ups Recovery Advanced Topics Conclusion

8 Advantages of Galera Parallel Slave Applying Practically No Slave Lag Instant Trivial Failover Automatic Node Provisioning Works Well in WAN 8

9 MariaDB Replication Server Centric Style Asynchronous Replication No Conflict Detection Multi-Master Replication Master Server Slave Server If Node C Crashes, Does Cluster Survive? If Node B Crashes and Clients Switch to C, How does Node B Rejoin? Which Node has Data X? How do you Back-Up Cluster? Node A Node C Node B 9

10 Galera Approach Data Centric Style Data Doesn t Belong to a Node Nodes Belong to Data Data is Synchronized among Two or More Servers DataSet Server 1 Server 2 Server 3 Server N 10

11 Galera Approach Galera Nodes are Anonymous All are Equal Galera Cluster is One Large Distributed Master A DataSet Needs an Identifier DataSet Identier is a Cluster Identifier 00295a79-9c48-11e2-bdf0-9a916cbb9294 DataSet Cluster 11

12 Global Transaction Identifier (GTID) DataSet plus Sequence of Atomic Changes equals GTID 00295a79-9c48-11e2-bdf0-9a916cbb9294:64201 DataSet 00295a79-9c48-11e2-bdf0-9a916cbb9294:

13 Global Transaction Identifier (GTID) Initial DataSet 00295a79-9c48-11e2-bdf0-9a916cbb9294:0 First Change and Transaction 00295a79-9c48-11e2-bdf0-9a916cbb9294:1 Undefined GTID :-1 13

14 Global Transaction Identifier (GTID) MySQL 5.6 GTID e-7c1e-11e2-a6e ef5:12345 (server identifier : transaction processed by server) MariaDB 10 GTID (domain - server identifier - data change in asynchronous cluster) Galera GTID 00295a79-9c48-11e2-bdf0-9a916cbb9294:64201 (data & cluster Identifier : data change in cluster) 14

15 Global Transaction Identifier (GTID) Visible in MySQL e-7c1e-11e2-a6e ef5: e-7c1e-11e2-a6e ef5: e-7c1e-11e2-a6e ef5:12347 New Master Promoted f4e3bf7a-a91f-11e2-4e02-3f8dbcffaed8:1 f4e3bf7a-a91f-11e2-4e02-3f8dbcffaed8:2 f4e3bf7a-a91f-11e2-4e02-3f8dbcffaed8:3 15

16 Global Transaction Identifier (GTID) Visible in MariaDB New Master Promoted

17 Global Transaction Identifier (GTID) Visible in Galera 00295a79-9c48-11e2-bdf0-9a916cbb9294: a79-9c48-11e2-bdf0-9a916cbb9294: a79-9c48-11e2-bdf0-9a916cbb9294:64203 New Master Promoted 00295a79-9c48-11e2-bdf0-9a916cbb9294: a79-9c48-11e2-bdf0-9a916cbb9294: a79-9c48-11e2-bdf0-9a916cbb9294:

18 Master or Slave Not a Node Role or Function A Relation Between a Node and a Client client1 client2 slave1 slave2 master1 slave2 Cluster slave1 master2 master for client1 slave1 slave2 slave1 slave2 master for client1 18

19 Cluster Address wsrep_cluster_address Documentation on Galera Cluster Addresses: 19

20 Cluster Address gcomm:// handshake

21 Cluster Address

22 Cluster Address

23 Cluster Address wsrep_cluster_address = gcomm://node1,node2 Try to Connect to Members (node1, node2) Can only Join a Running Cluster

24 Node Synchronization (State Transfer)

25 Node Synchronization (State Transfer) joined synced synced desync undefined synced 25

26 Node Synchronization (State Transfer) New Node Cluster Old Node UNDEFINED Verify Node Synched Donor Found Wait Be a Donor SYNCHED JOINER DONOR Transfer Missing Data (Private Channel) JOINED JOINED catch-up catch-up SYNCHED SYNCHED 26

27 Avoiding Split Brain Distinguishing Server Crash from Network Failure in Shared Nothing Architecture Decision Algorithm Used to Avoid Spilt Brain Absolute Majority Needed in Galera Uneven Number of Nodes Safer 27

28 Primary Component Primary 28

29 Primary Component Continues Working Primary Non-Primary 29

30 Primary Component Continues Working Tries to Reconnect Primary Non-Primary 30

31 Primary Component Primary 31

32 Split Brain Non-Primary Split Brain Possible with Even Number of Nodes Non-Primary 32

33 Synchronous Penalties Galera Copies Data Buffer to All Cluster Members on COMMIT from Client (~1 RTT added latency) Connection throughput equals 1/RTT trx/sec Total throughput equals 1/RTT trx/sec #connections A Given Row Can't be Modified More Than 1/RTT times a second Round Trip Time (RTT) is Length of Time for a Signal to be sent and Receipt of Acknowledgement 33

34 Slave Lag in Galera Client Master Node START TRANSACTION Slave Node PROCESS COMMIT Replicate Write Set Certify Acknowledge OK COMMIT APPLY SELECT (stale data) COMMIT 34

35 Nuances Uses Only InnoDB Primary Keys are Necessary Commit Latency Transaction Size Limited to 2GB Unlimited in Future Versions DEADLOCK on COMMIT AUTO_INCREMENT Handled Differently Cluster Performance Limited to Slowest Node No Cluster-Wide Read Locks No XA Transactions 35

36 Installation MariaDB Enterprise Cluster Introduction State Transfer Schema Changes Load Balancing Architecture Installation Caveats Multi-Master Conflicts Back-Ups Recovery Advanced Topics Conclusion

37 Installing from a Repository MariaDB Repository Configuration Tool ( Available for all Major Distributions Downloading MariaDB & Galera: 37

38 Installing MariaDB Galera Cluster Install Extra Packages for Enterprise Linux $ yum install epel-release Install Galera Enabled MariaDB Server $ yum install MariaDB-server Install Network Copying Tool $ yum install socat 38

39 Additional Utilities XtraBackup Useful for SST Not Supported in MariaDB 10.2 $ yum install xtrabackup Percona Toolkit $ yum install percona-toolkit XtraBackup Repo: 39

40 Linux Configuration SELinux Disable Enforce Set Server to SELinux Mode Set Start Timeout for systemd (CentOS 7+) cat /selinux/enforce echo 0 > /selinux/enforce SELINUX=permissive Excerpt from /etc/selinux/config [Service] TimeoutStartSec="30 min" systemctl daemon-reload Executed from Command-Line Start Timeout Settings: SELinux Configuration: 40

41 Linux Firewall Ports Used Port 3306 Client Connections to Nodes Port 4567 Replication Protocol Port 4568 Incremental State Transfers (IST) Port 4444 Snapshot State Transfer (SST; socat) iptables -L /etc/init.d/iptables stop chkconfig iptables off Executed from Command-Line IST & SST Methods may have Additional Connectivity Requirements mysqldump requires mysql client connections (port 3306) between Nodes Rsync and XtraBackup use Netcat (nc) between Nodes 41

42 MariaDB Configuration [mysqld] max_connections=1024 binlog_format=row innodb_buffer_pool_size=200m innodb_log_file_size=100m innodb_flush_log_at_trx_commit=2 default-storage-engine=innodb innodb_autoinc_lock_mode=2 Only InnoDB Tables will be Replicated --enforce-storage-engine=innodb Documentation on Enforcing Storage Engine Setting: Documentation on InnoDB Flush Log at Transaction Commit Setting: 42

43 MariaDB Configuration Galera Settings wsrep_provider=/usr/lib64/galera/libgalera_smm.so wsrep_cluster_name="our_cluster" wsrep_node_name=node1 wsrep_cluster_address="gcomm:// , , " wsrep_node_address= wsrep_slave_threads=1 wsrep_sst_method=rsync wsrep_sst_auth=galera:galera wsrep_on=on 43

44 Start First Node Start galera1 as First Node Use SHOW STATUS and SHOW VARIABLES to List Galera Options systemctl start mariadb galera_new_cluster Executed from Command-Line on First Node SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'; SHOW GLOBAL VARIABLES LIKE 'wsrep%'; Executed from mysql Client on First Node mysql -u root -p -e / "SHOW VARIABLES LIKE 'wsrep_provider_options" Executed from Command-Line on First Node 44

45 Configure Galera wsrep_* Variables Configure MariaDB Replication Behavior wsrep_provider_options is a Collection of Variables that Control Galera Replication Plugin Behavior Galera System Variables: Galera Status Variables: 45

46 Default Enterprise Cluster Related Values base_host = base_port = 4567 cert.log_conflicts = no evs.causal_keepalive_period = PT1S evs.debug_log_mask = 0x1 evs.inactive_check_period = PT0.5S evs.inactive_timeout = PT15S evs.info_log_mask = 0 evs.install_timeout = PT15S evs.join_retrans_period = PT1S evs.keepalive_period = PT1S evs.max_install_timeouts = 1 evs.send_window = 4 evs.stats_report_period = PT1M evs.suspect_timeout = PT5S evs.use_aggregate = true evs.user_send_window = 2 evs.version = 0 evs.view_forget_timeout = PT5M Galera System Variables: Galera Status Variables: gcache.dir = /var/lib/mysql/ gcache.keep_pages_size = 0 gcache.mem_size = 0 gcache.name = /var/lib/mysql// galera.cache gcache.page_size = 128M gcache.size = 128M gcs.fc_debug = 0 gcs.fc_factor = 1 gcs.fc_limit = 16 gcs.fc_master_slave = NO gcs.max_packet_size = gcs.max_throttle = 0.25 gcs.recv_q_hard_limit = gcs.recv_q_soft_limit = 0.25 gcs.sync_donor = NO gmcast.listen_addr = tcp:// :4567 gmcast.mcast_addr = gmcast.mcast_ttl = 1 gmcast.peer_timeout = PT3S gmcast.time_wait = PT5S gmcast.version = 0 gmcast.segment = 0 ist.recv_addr = pc.checksum = true pc.ignore_quorum = false pc.ignore_sb = false pc.linger = PT20S pc.npvo = false pc.version = 0 pc.weight = 1 protonet.backend = asio protonet.version = 0 replicator.causal_read_timeout = PT30S replicator.commit_order = 3 46

47 Start Other Nodes Second and Third Nodes must be Provided an Address for Connecting to the Cluster Use wsrep_cluster_address with IP Address of Node and at Least One Other Node wsrep_cluster_address = gcomm:// , , Excerpt from Configuration File of Each Node systemctl start mariadb Executed from Command-Line on Second and Third Node 47

48 Cluster Address wsrep_cluster_address Write All Nodes Planned for Cluster in Address String wsrep_cluster_address=gcomm:// , ,... Don't Leave wsrep_cluster_address=gcomm:// in my.cnf The wsrep_urls Parameter for mysqld_safe is Deprecated Galea Cluster Address: 48

49 MariaDB Configuration [mysqld] logbin binlog_format=row wsrep_on=on wsrep_provider=/usr/lib64/galera/libgalera_smm.so wsrep_cluster_name="our_cluster" wsrep_node_name=node2 wsrep_cluster_address="gcomm:// , , " wsrep_node_address= wsrep_slave_threads=1 wsrep_sst_method=rsync wsrep_sst_receive_address= :3306 wsrep_sst_auth=galera:rover123 49

50 Start Other Nodes Join the Second and Third Nodes to Cluster Node 2 systemctl start mariadb Executed from Command-Line on Second Node SHOW GLOBAL STATUS LIKE 'wsrep%'; Executed from mysql Client on Second Node Node 3 systemctl start mariadb Executed from Command-Line on Third Node SHOW GLOBAL STATUS LIKE 'wsrep%'; Executed from mysql Client on Third Node 50

51 Initial Cluster Startup All Nodes Should Be Running Now and Consistent Test Replication 1 CREATE TABLE test.table1 (col1 INT UNSIGNED KEY); INSERT INTO test.table1 VALUES (1),(2),(3); Executed from mysql Client on Node 1 2 SELECT * FROM test.table1; INSERT INTO table1 VALUES (4); Executed from mysql Client on Node 2 3 SELECT * FROM table1; Executed from mysql Client on Node 3 51

52 Sysbench Benchmarking Tool Install RPM Packages To Build a Database for Testing: sysbench [options] prepare To Run a Test on Database: sysbench [options] run $ wget $ rpm -ivh Create a Database $ sysbench --test=sysbench_tests/db/common.lua --mysql-host=node1 \ --mysql-user=test --mysql-db=sbtest --oltp-table-size= prepare $ sysbench --test=sysbench_tests/db/oltp.lua --mysql-host=node1 \ --mysql- user=test --mysql-db=sbtest --oltp-table-size= \ --report-interval=1 --max-requests=0 --tx-rate=10 run grep tps Set Up SysBench: Project Page: 52

53 State Transfers MariaDB Enterprise Cluster Introduction State Transfer Schema Changes Load Balancing Architecture Installation Caveats Multi-Master Conflicts Back-Ups Recovery Advanced Topics Conclusion

54 Node Recovery Procedure A Node Determines it's not in Synch Communicates with Other Nodes to Recover Determines its Current Cluster State to Determine Transfer Method Required Documentation on Node Recovery: 54

55 Recovery Methods Incremental State Transfer (IST) Partial Recovery (Delta's) of Write Sets Snapshot State Transfer (SST) Full Recovery of Data Error Log shows Method Used and Outcome Check Error Logs on Joiner and Donor Nodes 55

56 Incremental State Transfer Incremental State Transfer is very Effective gcache.size Parameter Defines Size of Write Sets kept for IST Gcache is mmap File Available Disk Space is Upper Limit for Size Allocation Used if Joining Node had a Previous State Later than Start of Cluster Only Possible after Graceful Shutdown Changes are Tracked by gcache galera.cache on File System and in Memory Documentation on IST: 56

57 Galera Saved State grastate.dat is Replication State File Residing in Database System Data Directory Included in Snapshot uuid is Universal Unique Identifier seqno is Sequence Number of Last Transaction Value Greater than 0 Indicates Proper Shutdown Value -1 means No Known Sequence Position (Server Crashed or Running) # GALERA saved state version: 2.1 uuid: 82e240bb e ebd7b6 seqno: -1 57

58 Snapshot State Transfer Used to Send Full Database State Set wsrep_sst_method to Dictate Method to Use mysqldump rsync (default) xtrabackup xtrabackup-v2 Recommended New Joining Nodes use SST Documentation on SST: 58

59 Forced SST Expediting Full Recovery Force SST by Deleting all Files in datadir 1 systemctl stop mariadb rm -rf /var/lib/mysql/* Stop All Nodes and Synchronize from galera1 2 Executed from Command-Line on Node 2 systemctl stop mariadb cd /var/lib/mysql rm -rf * systemctl start mariadb Executed from Command-Line on Node 3 3 systemctl start mariadb tail -f mysqld.err Executed from Command-Line on Node 2 59

60 XtraBackup SST Alternate Full Recovery Full Recovery without Downtime on Donor Requires Installing and Appropriate Credentials Add datadir to Configuration File 1 2 yum install xtrabackup Executed from Command-Line on First Node GRANT PROCESS, RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'xtrabackup'@localhost IDENTIFIED BY 'galera123'; Executed from mysql Client on Node 1 3 [server] datadir = /var/lib/mysql Excerpt from my.cnf on Node 1 XtraBackup SST: 60

61 Configuring XtraBackup SST Stop Second Node and Edit its MariaDB Configuration File Configure XtraBackup SST and Rejoin Node to Cluster 1 2 systemctl stop mariadb Executed from Command-Line on Second Node [server] wsrep_sst_method = xtrabackup-v2 wsrep_sst_auth = galera:galera Excerpt from my.cnf on Node 2 3 rm -rf /var/lib/mysql/* systemctl start mariadb Executed from Command-Line on Second Node 61

62 Verify IST Works Stop Second Node and Rejoin it to Cluster Check Error Log for IST Messages Regularly Verify SST Works 1 systemctl stop mariadb Executed from Command-Line on Second Node 2 CREATE TABLE test.testeroo (col1 INT); Executed from mysql Client on Node 1 3 systemctl start mariadb Executed from Command-Line on Second Node 4 journalctl -xe Executed from Command-Line on Second Node 62

63 Caveats MariaDB Enterprise Cluster Introduction State Transfer Schema Changes Load Balancing Architecture Installation Caveats Multi-Master Conflicts Back-Ups Recovery Advanced Topics Conclusion

64 Caveats Consistent Reads Primary Key is Required Dealing with AUTO_INCREMENT Detecting Slow Nodes Parallel Replication MyISAM Replication XA Transactions 64

65 Slave Lag in Galera Client Master Node Slave Node START TRANSACTION PROCESS COMMIT Replicate changeset Certification Acknowledgement OK COMMIT APPLY SELECT (waiting due to wsrep_sync_wait) COMMIT 65

66 Consistent Reads Settings Set wsrep_sync_wait to 1 Every Read (e.g., SELECT) Waits for Slave Queue to be Applied Fully Set Timeout for Maximum Causal Read Wait replicator.causal_read_keepalive Read Locks only on Local Node 66

67 Spotting Causality Failure Try Work Load to Spot Causality Failure Configure Read Causality and Test Again for i in { }; do echo -n -e "\nround $i "; mysql -ugalera -pgalera -h0 -P11001 test -e"insert INTO t VALUES($i)"; ret=$(mysql --batch -ugalera -pgalera -h0 -P11002 test -e"select * FROM t WHERE i=$i"); if [ "$ret" = "" ]; then echo -n "FAILED" fi done galera2-mariadb> SET SESSION wsrep_sync_wait = 1; 67

68 Dealing with AUTO_INCREMENT Galera Provides Automatic Control of AUTO_INCREMENT Galera will adjust auto_increment_increment and auto_increment_offset variables so Inserts in Separate Nodes will Interleave Increments AUTO_INCREMENT by Number of Nodes wsrep_auto_increment_control = ON Excerpt from my.cnf File Galera & AUTO_INCREMENT: 68

69 Controlling AUTO_INCREMENT Start First and Second Node Create AUTO_INCREMENT Table and Insert Rows through All Nodes Start Third Node and Insert More Rows 1 CREATE TABLE a (i INT PRIMARY KEY AUTO_INCREMENT, n VARCHAR(10)); INSERT INTO a (n) VALUES ('n1'),('n1'),('n1'); SHOW VARIABLES LIKE 'auto_increment%'; Executed from mysql Client on Node 1 2 INSERT INTO a (n) VALUES ('n2'),('n2'),('n2'); SHOW VARIABLES LIKE 'auto_increment%'; SELECT * FROM a; 3 SHOW VARIABLES LIKE 'auto_increment%'; Executed from mysql Client on Node 1 4 SHOW VARIABLES LIKE 'auto_increment%'; INSERT INTO a (n) VALUES ('n3'),('n3'),('n3'); SELECT * FROM a; Executed from mysql Client on Node 3 Executed from mysql Client on Node 2 69

70 Controlling AUTO_INCREMENT Disable AUTO_INCREMENT Control in galera2 Stop and Start galera3 1 SET GLOBAL wsrep_auto_increment_control=0; Executed from mysql Client on Node 2 3 SHOW VARIABLES LIKE 'auto_increment%'; 2 systemctl stop mariadb Executed from Command-Line on Node 3 4 systemctl start mariadb Executed from Command-Line on Node 3 Execute on Both Node 1 and 2 5 SHOW VARIABLES LIKE 'auto_increment%'; 70

71 Detecting a Slow Node Nodes should have Equal Capacity Use Equivalent Hardware Galera Implements Flow Control to Protect Cluster from Slave Queue Growth wsrep_flow_control_sent wsrep_flow_control_recvd wsrep_flow_control_paused wsrep_local_recv_queue Number of Times Node Asked for Flow Control Number of Times Node Received Flow Control Stop Signal Fraction of Time Node had to Pause for Flow Control Length of Slave Queue Slow Nodes in Galera: 71

72 Slave Queue Tuning Nodes should have Equal Capacity Use Equivalent Hardware Galera Implements Flow Control to Protect Cluster from Slave Queue Growth A Value Greater than 0.02 for wsrep_flow_control_paused indicates a Performance Problem in Cluster Managing Galera Flow: 72

73 Slave Queue Tuning wsrep_slave_threads Number of Parallel Threads Applying to Slave Queue Slave Thread Count = 1 to 3 x Logical CPU Cores gcs.fc_limit High Limit for Flow Control (default 16) Flow Control Stops when Reached 5 x wsrep_slave_threads gcs.fc_factor Factor times Limit is Lower Limit (default 1; recommended 0.8) Flow Control Resumes when Slave Queue Reduced to this Level 73

74 Flow Control Configure Flow Control Limit Low on Second Node Push Heavy Load against the First Node and Check Status on Second SET GLOBAL wsrep_provider_options = 'gcs.fc_limit=2'; Executed from mysql Client on Node 2 sysbench --test=/usr/share/doc/sysbench/tests/db/oltp.lua \ --mysql-user=root --mysql-password=fido123 \ --mysql-db=sbtest --oltp-table-size=25000 \ --report-interval=5 --max-requests=0 --num-threads=10 run Executed from Command-Line on First Node 74

75 Parallel Replication Any Number of Slave Applier Threads can be Started Slave Control Assigns Write Sets for Slave Appliers Slave Thread MariaDB Slave Thread Slave Control is on ROW Level Applying Only is Parallel Commit Order is Dictated Slave Control Slave Queue 75

76 True Parallel Applying Every Application Benefits Works on Row Level Not on Database or Table SET GLOBAL wsrep_slave_threads = n; Set to Number of Slaves that makes Sense Monitor wsrep_cert_deps_distance Maximum about 4 times Number of CPU Cores 76

77 True Parallel Applying Start Heavy Load on First Node Monitor CPU on Second Node with top Configure More Slave Threads sysbench --test=/usr/share/doc/sysbench/tests/db/oltp.lua \ --mysql- user=root --mysql-password=fido123 \ --mysql-db=sbtest --oltp-table-size=25000 \ --report-interval=5 --max-requests=0 --num-threads=10 run Executed from Command-Line on First Node SET GLOBAL wsrep_slave_threads = 2; SHOW PROCESSLIST; SET GLOBAL wsrep_slave_threads = 4; SHOW PROCESSLIST; Executed from mysql Client on Node 2 77

78 Multi-Master Conflicts MariaDB Enterprise Cluster Introduction State Transfer Schema Changes Load Balancing Architecture Installation Caveats Multi-Master Conflicts Back-Ups Recovery Advanced Topics Conclusion

79 Multi-Master Conflicts Galera Uses Optimistic Concurrency Control When Two Transactions Modify same Row on Different Nodes at Same Time, One Transaction must Abort Victim Transaction will get Deadlock Error (1213) upon COMMIT Application should Retry Deadlocked Transactions Not All Applications have Retrying Logic Built-In 79

80 Retry Transactions Galera can Retry autocommit Transaction for Client Application within MariaDB Server MariaB will Not Return Deadlock Error will Retry Silently Transaction Set wsrep_retry_autocommit to Number of Retries before Returning Deadlock Error Retrying Applies Only to autocommit Transactions since Not Safe for Multi-Statement Transactions 80

81 Diagnostics Set wsrep_log_conflicts to send Information on each Cluster Conflict to MariaDB Error Log Use Cert.log_conflicts to Log Conflicting Transaction Database Hot-Spots Rows to which Many Transactions want to Write Simultaneously Patterns like Queue or ID Allocation can be Hot-Spots Monitor Status Variables wsrep_local_bf_aborts wsrep_local_cert_failures Set wsrep_debug so All Conflicts will be Logged 81

82 Diagnose Conflicts Check if Application Logic can be Changed to get Deadlock Exception and Apply Retrying Logic in Application Try Setting wsrep_retry_autocommit Limit Number of Master Nodes or Change to Master-Slave Model Treat Writes Only to Hot-Spot Table as Master-Slave, if can Limit Access to Table SET GLOBAL wsrep_log_conflicts=1; SET GLOBAL wsrep_provider_options = 'cert.log_conflicts=1'; Executed from mysql Client on Node 1 tail -f /var/lib/mysql/node1.err Executed from Command-Line on First Node SHOW LOCAL STATUS LIKE 'wsrep_local%'; Executed from mysql Client on Node 1 82

83 Create Ad-Hoc Conflict Any Command after Conflict will Result in Deadlock Error 1 CREATE TABLE c (i INT PRIMARY KEY, j INT); INSERT INTO c VALUES (1,0),(2,0); BEGIN; UPDATE c SET j=1 WHERE i=1; Executed from mysql Client on Node COMMIT; BEGIN; UPDATE c SET j=1 WHERE i=1; Executed from mysql Client on Node 1 UPDATE c SET j=2 WHERE i=1; Executed from mysql Client on Node 2 2 UPDATE c SET j=2 WHERE i=1; Executed from mysql Client on Node 2 5 SELECT * FROM c; Executed from mysql Client on Node 1 83

84 Schema Changes MariaDB Enterprise Cluster Introduction State Transfer Schema Changes Load Balancing Architecture Installation Caveats Multi-Master Conflicts Back-Ups Recovery Advanced Topics Conclusion

85 Schema Upgrades DDL is Non-Transactional Bad for Replication Galera has Two Methods for DDL Total Order Isolation (TOI) Rolling Schema Upgrade (RSU) Use wsrep_osu_method to choose Option pt-online-schema-change provides Lockless Schema Upgrade Use TOI Mode Careful of Foreign Keys 85

86 Total Order Isolation DDL is Replicated Up-Front Each Node gets the DDL Statement and Must Process DDL at Same Slot in Transaction Stream Galera Isolates and Locks Entire Cluster for Duration of DDL Processing Documentation on Isolation Levels: 86

87 Rolling Schema Upgrade DDL is Not Replicated Galera will Remove Node from Replication for Duration of DDL Processing When done with DDL, Node will get Missed Transactions (e.g., IST) DBA should Roll RSU Operation over All Nodes Requires Backward Compatible Schema Changes Use Only under Certain Conditions Planned SQL is Not Conflicting SQL will Not Generate Inconsistency 87

88 Schema Upgrade Strategy Best Practices Plan Upgrades Try to make Backwards Compatible Rehearse Upgrades Determine DDL Execution Time Use RSU if Possible ALTER TABLE to Create New AUTO_INCREMENT Column will Cause Problems Every Node has Different AUTO_INCREMENT and Offset Settings 88

89 Load Testing a Schema Upgrade Start Moderate sysbench Load sysbench --test=/usr/share/doc/sysbench/tests/db/oltp.lua \ --mysql-user=root --mysql-password=fido1123 \ --mysql-db=sbtest --oltp-table-size=25000 \ --report-interval=5 --max-requests=0 --tx-rate=10 run Issue Some DDL under TOI 1 2 ALTER TABLE sbtest ADD COLUMN (m int UNSIGNED KEY); Executed from mysql Client on Node 1 ALTER TABLE sbtest ADD COLUMN (n int UNSIGNED); 3 CREATE TABLE m (i int); Executed from mysql Client on Node 1 4 CREATE TABLE n (i int); Executed from mysql Client on Node 2 Executed from mysql Client on Node 2 89

90 Test Schema Upgrade Issue some DDL under RSU 1 2 SET GLOBAL wsrep_osu_method=rsu; ALTER TABLE sbtest DROP COLUMN m; Executed from mysql Client on Node 1 SHOW CREATE TABLE sbtest; SET GLOBAL wsrep_osu_method=rsu; ALTER TABLE sbtest DROP COLUMN n; Executed from mysql Client on Node DROP TABLE m; Executed from mysql Client on Node 1 DROP TABLE n; Executed from mysql Client on Node 2 90

91 Back-Ups with Galera MariaDB Enterprise Cluster Introduction State Transfer Schema Changes Load Balancing Architecture Installation Caveats Multi-Master Conflicts Back-Ups Recovery Advanced Topics Conclusion

92 Galera for Back-Ups All Galera Nodes are Continuously Up-to-Date Best Practices Dedicate a Reference Node for Back-ups Assign Global Transaction Identifier with Back-up Possible Methods Desychronize a Node for Back-up XtraBackup 92

93 Back-Ups with Global Transaction Identifier Global Transaction Identifier (GTID) marks Position in Cluster Transaction Stream Backup with Known GTID makes Utilizing IST Possible when Joining New Nodes Recovering a Node Provisioning New Nodes Use XtraBackup's wrapper innobackupex with -- galera-info 93

94 Back-Up by Desynchronizing a Node Load Balancing Isolate Back-Up Node Server 1 Server 2 Server 3 Galera Replication 94

95 Back-Up by Desynchronizing a Node Desynchronize a Node from Group (Enable wsrep_desync) Load Balancing Server 1 Server 2 Server 3 Desynchronize Galera Replication 95

96 Back-Up by Desynchronizing a Node Load Balancing FLUSH TABLES WITH READ LOCK; Block Changes Server 1 Server 2 Server 3 Galera Replication 96

97 Back-Up by Desynchronizing a Node Read GTID from Status and Assign to Back- Up (wsrep_cluster_uuid, wsrep_last_committed) Load Balancing Back-Up Server 1 Server 2 Server 3 Galera Replication 97

98 Performing Back-Up After Desynchronizing Node Make Logical Backup mysqldump mydumper Physical Back-Up Copy with cp LVM Snapshot 98

99 Back-Up by Desynchronizing a Node Load Balancing UNLOCK TABLES; Replicate Changes Server 1 Server 2 Server 3 Galera Replication 99

100 Back-Up by Desynchronizing a Node Disable wsrep_desync Load Balancing Server 1 Server 2 Server 3 Galera Replication 100

101 XtraBackup Hot Back-Up Method Can be Used Any Time Simple and Efficient Use --galera-info Option to get Global Transaction Identifier Logged into Separate Galera Information File 101

102 Recovery MariaDB Enterprise Cluster Introduction State Transfer Schema Changes Load Balancing Architecture Installation Caveats Multi-Master Conflicts Back-Ups Recovery Advanced Topics Conclusion

103 Recovering from Downtime Scheduled Maintenance Clean Shut-Downs Unexpected Cluster Failures Power Outage Network Outage Out of Synch Nodes 103

104 Recovering from Clean Shutdown Cluster Start Order is Important Always in Reverse Order symtemctl stop mariadb Start First the Node with Latest Writes or Changes If Start Instead another Bootstrap Node, Force SST for Subsequent Nodes Execute on All Three Nodes SST Initiated Unless All Nodes are in Same State (GTID) Shutdown Cleanly All Nodes Cluster can Shrink to One Node if Node Shutdowns are Graceful 104

105 Recovery Galera GTID (seqno) is Written in Persistent InnoDB Storage after each Commit Galera will use InnoDB GTID if seqno in grastate is Undefined (-1) seqno is Undefined in Case of Crash Persistent GTID can be Queried with galera_recovery_information Old Non systemd Installations Use --wsrep_recover Use InnoDB GTID if Unsure which Node was Last Updated 105

106 Split Brain Shutdown Third Node and Kill Second Node Make One Node the Primary Component (PC) Never have More than One Primary Component 1 2 Node 1 changes to Non-Primary State to Avoid Split Brain systemdtl stop mariadb Executed from Command-Line on Node 3 killall -9 mysqld Executed from Command-Line on Node SET GLOBAL wsrep_provider_options = 'pc.bootstrap=1'; Executed from mysql Client on Node 1 systemdtl start mariadb Executed from Command-Line on Node 2 systemdtl start mariadb Executed from Command-Line on Node 3 106

107 Split Brain Scenarios Even Number of Nodes Two Data Center Configurations Two-Node Configuration Weighted Nodes with One-Node Majority ( ) SET GLOBAL wsrep_provider_options = 'pc.ignore_sb=1'; SET GLOBAL wsrep_provider_options = 'pc.weight=3'; Executed from mysql Client on Node 1 Caution pc.ignore_sb is Dangerous 107

108 Two-Node Cluster and Split Brain wsrep_provider_options="pc.ignore_sb" Clients VIP 108

109 Load Balancing MariaDB Enterprise Cluster Introduction State Transfer Schema Changes Load Balancing Architecture Installation Caveats Multi-Master Conflicts Back-Ups Recovery Advanced Topics Conclusion

110 Load Balancing Tools Use glbd Daemon for Load Balancing through a Daemon process Use libglb to Load Balance in Application Process Need to Set Two Environment Variables to use libglb LD_PRELOAD=<path-to-libglb> GLB_OPTIONS='options string' MaxScale is an Intelligent Proxy that $ LD_PRELOAD=src/.libs/libglb.so \ GLB_OPTIONS="--random " \ mysql -u root p fido123 -h P 3306 Documentation on Galera Load Balancer: Project Page: 110

111 Advanced Galera Topics MariaDB Enterprise Cluster Introduction State Transfer Schema Changes Load Balancing Architecture Installation Caveats Multi-Master Conflicts Back-Ups Recovery Advanced Topics Conclusion

112 Weighted Quorum Better Control on Primary & Non- Primary State in Case of Failure For Link Failure, MAIN remains Primary server MAIN1: pc.weight=1 server MAIN2: pc.weight=1 server DR1: pc.weight=0 server DR2: pc.weight=0 Manual Intervention to make DR Primary Documentation on Weighted Quorum: 112

113 Multicasting Value of Multi-Casting WAN vs. LAN Recommended to set Multicast before Bootstrapping Cluster Problems may occur if Migrating from Unicast to Multicast $> ip route add /4 dev eth0 [server] wsrep_provider_options="gmcast.mcast_addr= " wsrep_cluster_address="gcomm:// " 113

114 Multiple Galera Cluster (Asynchronous) Multi-Site Replication is Possible with Galera Additional Caution Required All Tables must have Primary Keys Manual Management of AUTO_INCREMENT [server] wsrep_auto_increment_control=0 auto_increment_increment = 16 auto_increment_offset = 1,2,3,4,5,6,7,8,9( ) server_id = 1234 Use Same server_id for All 114

115 WAN Replication Link Failures happens More Frequently No Need to Arbitrate Always Increase the Timeouts Writes Stall for Longer Time while Link is Down Monitor the Network wsrep_provider_options = "evs.keepalive_period = PT3S; evs.suspect_timeout = PT30S; evs.inactive_timeout = PT1M; evs.install_timeout = PT1M" Defaults: evs.keepalive_period = PT1S evs.suspect_timeout = PT5S evs.inactive_timeout = PT15S evs.install_timeout = PT15S 115

116 Cluster Notifications Cluster can Trigger Notifications Notification can be Sent to MaxScale Script API for Notification Syntax wsrep_notify_cmd Defines the Script to Handle Notifications Load Balancer Configuration Monitoring 116

117 Cluster Notifications Notification Script (/usr/bin/ wsrep_notify.sh) # Command to Call when Node Status or Cluster Membership Changes. # Will be Passed All or Some of Following Options: # --status - New Status of Node # --uuid - UUID of Cluster # --primary - Whether Component is Primary ("yes" or "no") # --members - Comma-Separated List of Members # --index - Index of Node in List echo "$@" >> /tmp/wsrep_notifications.log 117

118 Cluster Notifications Configure the Second Node to Call the Notification Handler Cause Cluster State Changes 1 2 [server] wsrep_notify_cmd = /usr/bin/wsrep_notify.sh Excerpt from Configuration File on Node 2 tail -f /tmp/wsrep_notifications.log Executed from Command-Line on Node 2 3 systemctl stop mariadb Executed from Command-Line on Node 1 118

119 Master & Slave Settings Use if Writing only to One Master Set wsrep_sync_wait to 1 SET GLOBAL VARIABLE wsrep_provider_options = "gcs.fc_limit = 256; gcs.fc_factor = 0.99; gcs.fc_master_slave = yes"; Can Release Flow Control before Pausing Replication Defaults gcs.fc_limit = 16 gcs.fc_factor = 0.5 fc_master_slave = NO 119

120 Conclusions MariaDB Enterprise Cluster Introduction State Transfer Schema Changes Load Balancing Architecture Installation Caveats Multi-Master Conflicts Back-Ups Recovery Advanced Topics Conclusion

121 On-Line Resources Documentation Knowledge Base Open-Source Community Post Questions on Forums Ask Questions on IRC FreeNode MariaDB Documentation and Knowledge Base: 121

122 Other Training Courses Public In-Person Courses Private In-Person Courses Live Virtual Training Courses Self-Paced On-Line Courses Training Courses: Upcoming Courses: 122

123 Thanks for Participating MariaDB Enterprise Cluster Introduction State Transfer Schema Changes Load Balancing Architecture Installation Caveats Multi-Master Conflicts Back-Ups Recovery Advanced Topics Conclusion

Migrating to XtraDB Cluster 2014 Edition

Migrating to XtraDB Cluster 2014 Edition Jay Janssen Managing Consultant Overview of XtraDB Cluster Percona Server + Galera Cluster of Innodb nodes Readable and Writable Virtually Synchronous All data