Migrating to XtraDB Cluster 2014 Edition

Transcription

1 Migrating to XtraDB Cluster 2014 Edition Jay Janssen Managing Consultant

2 Overview of XtraDB Cluster Percona Server + Galera Cluster of Innodb nodes Readable and Writable Virtually Synchronous All data on all nodes Established Cluster Synchronizes new nodes Donor node contributes to Joiner SST = Full Backup IST = Incremental Handles node failures / Splitbrains

3 XtraDB Cluster FAQ Standard Replication works Not a write scalable cluster Every node has identical data LAN/WAN architectures MyISAM not recommended More sensitive to all nodes Flow control Writes integrated w/ replication Application / cluster connections unmanaged

4 Characteristics of Galera Replication When a client receives a successful commit: The transaction is committed locally All other nodes have the transaction The transaction will commit on all the other nodes Conflicts are possible, but prevented and pushed to the application to handle (Certification) By way of deadlock errors Node communication is efficient (~1 RTT) Delivery is enough to ensure a node knows enough to apply a transaction or drop it No other communication about a transaction is necessary

5 Application Considerations

6 Why does the Application care about replication? Conflicts mean deadlock errors (multi-node writing only) Commits will take longer than they do on a single server replication and local certification must happen on commit A given row can t be modified > 1 / RTT Workload dependent!

7 Actual Workloads Small transactions DDLs Tables Large Trx expose serialization points and can stall writes By default DDLs block replication and apply simultaneously on all nodes Beware long running DDLs on the cluster All Innodb, All defined Primary keys Percona offers PXC workload analysis Based off Percona Server enhanced slow query log and pt-query-digest analysis

8 Application to Cluster Connects For writes: Best practice to start: any single node Multi-reader ok if hotspots not an issue For Reads: All/most nodes load balanced to your choosing Replication lag still possible! Possible to instruct a connection for causal reads on a session basis Be sure that nodes are functioning members of the cluster! 3306 connection is not enough

9 Node failure and Split-brain

10 Node failure and Split brain Node failure handled via timeout means writes block until failure is identified and dealt with Quorum vote establishes if remaining node(s) can continue Primary component Node(s) without quorum reject queries Non-primary component Can Auto-rejoin later

11 Node counts >= 3 nodes in the cluster for proper failover Galera arbitrator for the odd node Having 2 data nodes can cause issues when one of them needs a state transfer At least 3 data nodes are recommended Galera replication performance decreases as the node count increases (except with multi-cast) Largest prod cluster I ve seen: ~6-8 nodes

12 Multi-DC and auto-failover Two DCs must be weighted to prefer one Good for Primary/DR setups 3 DCs are required for proper auto-failover that is colo agnostic Node counts in DCs are more dependent on HA needs in DC WAN segments Optimize cross-wan replication Smarter donor selection

13 Operational Considerations

14 What gear to use Writes are as fast as the slowest node Flow control can be tuned independently on nodes Standard MySQL + Innodb choices VMs work fine, just beware of over provisioning Cluster very sensitive to VM stalls

15 Example Configuration [mysqld] datadir binlog_format = /var/lib/mysql = ROW innodb_buffer_pool_size = 128M innodb_log_file_size = 64M innodb_locks_unsafe_for_binlog = 1 innodb_autoinc_lock_mode = 2 wsrep_cluster_name wsrep_cluster_address wsrep_node_name wsrep_node_address wsrep_provider wsrep_provider_options wsrep_sst_method wsrep_sst_auth = our_cluster = gcomm://node1,node2,node3 = percona1 = node1 = /usr/lib64/libgalera_smm.so = gcache.size=2g; gcs.fc_limit=512 = xtrabackup = backupuser:password

16 Cluster Communication Standard MySQL port for client traffic port: 3306 Cluster replication (gcomm) port: 4567 Supports multicast, SSL. Unicast by default State Snapshot Transfer (SST) port: 4444 Can block donor Incremental State Transfer (IST) port: 4568 Like SST, but gives only missing trxns to Joiner All ports can be changed All communication types can be run on different networks

17 Managing individual nodes A node without wsrep config acts just like a standard MySQL server A node with cluster options is mostly managed with standard start / stop operations wsrep_cluster_address helps the node find an existing cluster member this setting does not define the cluster The first node in a cluster must be bootstrapped Data on that node becomes cluster data

18 Maintaining Nodes Rolling restarts for standard maintenance Schema changes Default: TOI -- block nodes and do the DDL on all RSU -- Rolling schema upgrade desync do the DDL on one node only Roll the change across all nodes separately Must be a backwards compatible change pt-online-schema-change Non-primary cluster segments can be manually recovered Just be sure there isn t another Primary component out there!

19 Monitoring SHOW GLOBAL STATUS like wsrep% ; Cluster integrity - verify consistency across all cluster nodes Cluster UUID, config version, # of active nodes, Primary state Node status Local node status, Flow control active, etc.

20 Realtime Galera status root# myq_status wsrep mycluster / node2 (idx: 2) / Galera 3.7(r7f44a18) Wsrep Cluster Node Repl Queue Ops Bytes Conflct Gcache Window Flow time P cnf # Stat Laten Up Dn Up Dn Up Dn lcf bfa ist idx dst appl comm p_ms $ myq_status wsrep 12:40:53 P 6 2 Sync 1.1ms K :40:54 P 6 2 Sync 1.1ms K :40:55 P 6 2 Sync 1.1ms K :40:56 P 7 3 Sync 2.1ms K :40:57 P 7 3 Dono 2.1ms K :41:21 P 7 3 Dono 5.0ms K :41:22 P 7 3 Dono N/A K :41:24 P 7 3 Dono N/A :41:25 P 7 3 Sync 2.5ms K :41:26 P 7 3 Sync 2.5ms K :41:27 P 7 3 Sync 2.5ms K :41:28 P 7 3 Sync 2.5ms K :42:14 P 7 3 Sync 2.6ms K :42:15 P 7 3 Sync 2.6ms K :42:16 P 7 3 Sync 2.6ms K :42:17 P 7 3 Sync 2.6ms :42:18 P 7 3 Sync 2.6ms :42:19 P 7 3 Sync 3.0ms * 12:42:20 P 7 3 Sync 3.0ms :42:21 P 7 3 Sync 3.0ms

21

22 Application to Cluster HA Tools for PXC Considerations Proper health checking Respond properly to Donor and Non-primary nodes Load balancing as needed Some choices Virtual IPs (and keepalived or Pacemaker) HAproxy or hardware load balancers Application side library (mysqlnd-ms) Scalearc (L7 MySQL proxy, commercial)

23 Converting Standalone MySQL to XtraDB Cluster

24 Method 1 - Single server Migrating a single server Stop MySQL replace the packages add Galera (wsrep_*) settings start MySQL (bootstrap) This sets up a 1 node cluster Node data is baseline for cluster, all new nodes synced Galera is not destructive to the datadir on the first node SST will clear datadir for nodes joining a cluster Easiest from Percona Server Use at least two nodes to test latency

25 Method 2 - Master/Slave Immediate to Cluster Confident all nodes are in sync With downtime Stop all writes, wait for replication RESET SLAVE, stop MySQL Upgrade software, wsrep settings Bootstrap first node - initial cluster Start the others with wsrep_sst_method=skip The slaves will join the cluster, skipping SST Change wsrep_sst_method later!

26 Method 3 - Slave Gradually to the Cluster New cluster from one slave Node replicates from old master log-slave-updates on this node Test like any other slave Move more slaves to cluster Real SST more testing Cut writes over to the cluster Move old master to the cluster

27 Method 3B - Dual-master

28 PXC Migration Recommendations Evaluate your workload for PXC compatibility Test carefully: Regression and load Be prepared for operational differences HA testing before production Stay with single writer to start Avoid major MySQL version upgrades Dual-master your cluster with the old master for rollback

29 Resources Percona XtraDB Cluster homepage Galera Documentation PXC tutorial (self-guided or at a conference) Our blogs on PXC: Google Groups codership-team percona-discussion