2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 1 Replication Tips and Tricks Dr. Mats Kindahl Lead Developer, Replication mats@sun.com mysqlmusings.blogspot.com Dr. Lars Thalmann Development Manager, Replication and Backup lars.thalmann@sun.com larsthalmann.blogspot.com Conference and Expo April 23 th, 2009
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 2 Tricks and Tips L Some old, some new Some easy, some more advanced Meant to be food for thoughts
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 3 Slave stopped now what? Check position it stopped on SHOW SLAVE STATUS Next event to transfer from master: Master_Log_File, Read_Master_Log_Pos Next event to apply: With respect to master log: Relay_Master_Log_File, Exec_Master_Log_Pos With respect to relay log: Relay_Log_File, Relay_Log_Pos Use mysqlbinlog to read the contents mysqlbinlog master-log.000001 mysqlbinlog relay-log.000001 Investigate the problem, and possibly: Remove rows in database; START SLAVE SET SQL_SLAVE_SKIP_COUNTER=1; START SLAVE M
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 4 Examining binlog mysqlbinlog --hexdump master-bin.000001 Statement-based INSERT1/2: Query event header L $ mysqlbinlog --hexdump master-bin.000001 # at 235 #060420 20:16:02 server id 1 end_log_pos 351 # Position Timestamp Type Master ID # 000000eb e2 cf 47 44 02 01 00 00 00 # Size Master Pos Flags # 74 00 00 00 5f 01 00 00 10 00
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 5 Examining binlog mysqlbinlog --hexdump master-bin.000001 Statement-based INSERT 2/2: Query event data # 000000fe 02 00 00 00 00 00 00 00 # 04 00 00 1a 00 00 00 40... # 0000010e 00 00......std # 0000011e 04 08......test.INSE # 0000012e 52 54... RT.INTO.t1.VALUE # 0000013e 53 20... S...A...B...X # 0000014e 27 2c......Y...X...X. # 0000015e 29. # Query thread_id=2 exec_time=0 error_code=0 SET TIMESTAMP=1145556962; INSERT INTO t1 VALUES ('A','B'), ('X','Y'), ('X','X'); L
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 6 Examining binlog mysqlbinlog --verbose master-bin.000001 Reconstructing INSERT from row-based events L $ mysqlbinlog --verbose master-bin.000001 BINLOG ' qznvsrmbaaaakqaaaaycaaaaabaaaaaaaaaabhrlc3qaanqxaaedaae= qznvsrcbaaaajwaaac0caaaqabaaaaaaaaeaaf/+awaaap4eaaaa '/*!*/; ### INSERT INTO test.t1 ### SET ### @1=3 ### INSERT INTO test.t1 ### SET ### @1=4
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 7 Replication to get online backup Replicate to a slave server So that all data is there Stop replication To stop changes to the slave Do backup on the slave It does not matter if this is blocking since the master is not blocked M This technique is used by the Time Machine Backup
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 8 L Extra slave to initiate new slaves Extra slave New slave
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 9 Using mysqldump to inititialize slave M Dump data from a server master and restore it on another server slave : mysqldump --host=master uuser -master-data --lock-all-tables --databases test mysql --host=slave uuser This locks the database!
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 10 Using mysqldump to inititialize slave M To avoid locking the entire table: mysqldump --host=master uuser -single-transaction - master-data --databases test mysql --host=slave uuser Note: only works for tables that can handle consistent snapshots, such as InnoDB
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 11 Using replication to repopulate a table M Write a script that, for each table t1: SELECT * INTO OUTFILE t1.txt FROM t1; DROP TABLE IF EXISTS t1; Works with statementbased replication CREATE TABLE t1 ; LOAD DATA INFILE t1.txt INTO TABLE t1; A version using INSERT-SELECT (works for row-based): CREATE TEMORARY TABLE t1_tmp LIKE t1; INSERT INTO t1_tmp SELECT * FROM t1; Works with rowbased replication DROP TABLE IF EXISTS t1; CREATE TABLE t1 SELECT * FROM t1_tmp; For row-based replication, the temporary table is not transferred to the slave, so this will only send the data that is necessary to bootstrap the table.
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 12 L Point-in-time recovery using mysqlbinlog For point in time recovery, you can take the replication log and apply it into the. mysqlbinlog \ --read-from-remote-server \ --host=master -uroot --position=192 \ --to-datetime= 2009-04-11 12:36:56 \ master-bin.00002[2-4] mysql -uroot --host=slave
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 13 M Proxy to do multi-master Proxy Proxy writes a new binary log
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 14 Use a default engine for a server CREATE TABLE ( uid INT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(64) ) ENGINE=INNODB; SET GLOBAL STORAGE_ENGINE = INNODB; CREATE TABLE ( uid INT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(64) ); Not replicated, so remember to set default engine on slaves!
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 15 M Split replication stream on table replication do table=east_coast replication do table=west_coast Relay Slave ALTER TABLE xxx ENGINE = BLACKHOLE SET GLOBAL STORAGE_ENGINE = BLACKHOLE Relay Slave
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 16 L Cluster multi-source replication Cluster Cluster Cluster Let masters update different data
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 17 L Multi-channel replication with fail-over Cluster Cluster
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 18 M Slave having time-sharing masters Clocked switching 1. Stop slave 2. Save position 3. Change master 4. Start slave
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 19 Stop the slave and save position using sh(1) socket="mysql-test/var/tmp/$1.sock" mysql_exec () { # Read commands from stdin client/mysql --vertical --socket="$1" --batch \ --skip-column-names --batch user=root } stop_slave () { echo STOP SLAVE mysql_exec "$1"; } fetch_pos () { echo SHOW SLAVE STATUS mysql_exec "$1" grep '\<Master_\(Host\ Port\ Log_File\)\ \<Read_Master_Log_Pos' cut -f2 -d: } stop_slave "$socket" echo `fetch_pos "$socket"` >$1.savepos M
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 20 Restore the slave position and start slave using sh(1)... start_slave () { echo START SLAVE mysql_exec "$1"; } M change_to_pos () { mysql_exec "$1" <<EOF # Pass command using here-is CHANGE MASTER TO MASTER_HOST='$2', MASTER_PORT=$3, MASTER_LOG_FILE='$4', MASTER_LOG_POS=$5 EOF } name cat $1.savepos { read host port file pos change_to_pos $socket "$host" "$port" "$file" "$pos" start_slave "$socket" }
cnt=1 2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 21 Rotate between several slaves using sh(1) while true; do save_pos_and_stop_slave mysqld.$cnt cnt=`expr $cnt % 5 + 1` restore_pos_and_start_slave mysqld.$cnt } M names are of the form mysqld.x savepos files are of the form mysqld.x.savepos
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 22 M Encrypted replication CHANGE MASTER TO MASTER_SSL = 1, MASTER_SSL_CA = 'cacert.pem' MASTER_SSL_CERT = mycert.pem MASTER_SSL_KEY = mykey.pem'
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 23 Using current database to filter In STATEMENT format, you can set current db to filter statements: If --binlog-ignore-db=db1 is used on server, then the following INSERT is not logged: USE foo; INSERT into bar.t1 VALUES(1); If --replicate-ignore-db=db1 is used on slave, then the following INSERT is not applied: USE foo; INSERT into bar.t1 VALUES(1); M In ROW format, you can not use this trick, since row-based always use the actual database: The following INSERT is logged and applied if db1 is not filtered: USE foo; INSERT into db1.t1 VALUES(1);
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 24 M More columns on slave than master You can add a column on the slave: Master: CREATE TABLE t1 (a INT, b INT); Slave: ALTER TABLE t1 ADD ts TIMESTAMP; Extra columns have to have default values! Master: INSERT INTO t1(a,b) VALUES (10,20); Row-based replication need extra column last
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 25 M Less columns on slave than master In row-based replication, you can remove columns on the slave: Master: CREATE TABLE t1 (a INT, b INT, comments TEXT); Slave: ALTER TABLE t1 DROP comments; Only possible to remove columns from the end Master: INSERT INTO t1 VALUES (1,2, Do not store this on slave );
New in 6.0/5.4 2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 26 Replication Heartbeat Master Replication I/O thread Slave SE1 SE2 Binlog Relay Binlog SE1 SE2 Binlog Storage Engines
New in 6.0/5.4 2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 27 Replication Heartbeat Master Slave Automatic checking of connection status No more relay log rotates when the master is idle Detection of master/slave disconnect configurable in millisecs CHANGE MASTER SET master_heartbeat_period= val; SHOW STATUS like 'slave_heartbeat period' SHOW STATUS like 'slave_received_heartbeats'
New in 6.0/5.4 2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 28 Ignoring s in Circular Replication A B D C Circular replication If server A is removed from the circle, server B can be set to terminate A's events in the new circle B> CHANGE MASTER TO MASTER_HOST=C... IGNORE_SERVER_IDS=(A)
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 29 L Replication database test c1 c2 c3 1 4 2 5 3 6 copy rpl_db c1 c2 c3 1 4 2 5 3 6 master replication c1 c2 c3 rpl_db 1 4 2 5 3 6 Slave
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 30 L 1. Two options to only replicate this part: 1. On master: Replication database [mysqld]... binlog-do-db=rpl_db Note that this filters the binlog on master On slave: [mysqld]... replicate-do-db=rpl_db This filters the replication execution on the slave Note that you need row-based for this to work In statement-based replication, the source tables are need to execute the statement.
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 31 L Replication database CREATE DATABASE rpl_db; USE test; CREATE TABLE t1 (a INT, b INT, c INT); USE rpl_db; CREATE TABLE t1 (a INT, b INT, c INT); USE test; # Not replicated INSERT INTO t1 VALUES (1,2,3), (2,5,9); USE rpl_db; # Replicated INSERT INTO t1 VALUES (3,4,5), (4,6,11);
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 32 Less columns on slave than master L a b c a c trigger master rpl_db replication a c slave rpl_db
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 33 L Less columns on slave than master Master: CREATE DATABASE rpl_db; USE test; CREATE TABLE t1 (a INT, b BLOB, c INT); CREATE TRIGGER tr_t1 AFTER INSERT ON test.t1 FOR EACH ROW INSERT INTO rpl_db.t1_v(a,c) VALUES(NEW.a,NEW.c); USE rpl_db; CREATE TABLE t1_v (a INT, c INT); Use like this: USE test; SET @blob = REPEAT('beef',100); INSERT INTO t1 VALUES (1,@blob,3), (2,@blob,9);
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 34 M Only replicate some rows to slave a b c 1 4 2 5 3 6 trigger rpl_db a b c 4 5 6 master replication a b c 4 5 6 Slave rpl_db
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 35 M Only replicate some rows to master # Just replicate rows that has odd numbers in first column. USE rpl_db; CREATE TABLE t1_h (a INT, b BLOB, c INT); --delimiter // CREATE TRIGGER slice_t1_horiz AFTER INSERT ON test.t1 FOR EACH ROW BEGIN IF NEW.a MOD 2 = 1 THEN INSERT INTO rpl_db.t1_h VALUES (NEW.a, NEW.b, NEW.c); END IF; END// --delimiter ;
Provides: 2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 36 Execution of replication events on slave is to be N seconds behind master User interface: Feature Preview Time-delayed replication http://forge.mysql.com/wiki/replicationfeatures/delayedreplication CHANGE MASTER TO MASTER_DELAY= N N is a non-negative less than MAX_ULONG number of seconds to wait by the slave. (Attempts to set higher is rejected with error.) Controlling of a delayed setup is via SHOW SLAVE STATUS Seconds_behind_master Kudos: Kay Röpke, original patch; Sven, modifications L
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 37 Scriptable Replication High-level structure http://forge.mysql.com/wiki/replicationfeatures/scriptablereplication Client Master Slave Dump I/O SQL function befo local q = eve slave:query(q end function befo local q = eve slave:query(q end Binary Log function befo local q = eve slave:query(q end Extension modules Relay Log function befo local q = eve slave:query(q end
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 38 The oracle algorithm Paul Tuckfield s (Google/YouTube) module(..., package.seeall); require luasql.mysql pattern = { ["UPDATE%s+(%w+).*%s(WHERE.*)"] = "SELECT * FROM %1 %2", ["DELETE%s+FROM%s+(%w+).*%s(WHERE.*)"] = "SELECT * FROM %1 %2", } env = luasql.mysql() con = env:connect("test", root,, localhost, mysql.port) function before_write(event) local line = event.query if not line then return end for pat,repl in pairs(pattern) do local str = string.gsub(line, pat, repl) if str then con:execute(str); break; end end end
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 39 Semi-synchronous Replication Original patch: Mark Callaghan and Wei Li, Google Adoptions: Zhenxing He, Sun Microsystems Application Logging/ Replication Semi-Sync Replicator Semi-Sync Receiver Relay Log/ Applier Application Master Replication L R R A Slave Ack SE1 SE2 Binlog Relay Binlog SE1 SE2 Storage Engines Storage Engines
On master: New in 6.0/5.4 2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 40 Install Plug-ins and Play INSTALL PLUGIN 'rpl_semi_sync_master' SONAME 'libsemisync_master.so'; SET rpl_semi_sync_master_enabled=1; SET rpl_semi_sync_master_timeout=1000; On slave: milliseconds INSTALL PLUGIN 'rpl_semi_sync_slave' SONAME 'libsemisync_slave.so'; SET rpl_semi_sync_slave_enabled=1; START SLAVE;
2009-04-23 Lars Thalmann & Mats Kindahl Replication Tricks and Tips AB 2007-9 www.mysql.com 41 Look into the DevZone L Replication DevZone http://dev.mysql.com/replication Dr. Mats Kindahl Lead Developer, Replication Technology mats@sun.com Dr. Lars Thalmann Development Manager, Replication and Backup Technology lars.thalmann@sun.com