The Hazards of Multi-writing in a Dual-Master Setup

Transcription

1 The Hazards of Multi-writing in a Dual-Master Setup Jay Janssen MySQL Consulting Lead November 15th, 2012

2 Explaining the Problem

3 Rules of the Replication Road A given MySQL instance: Can be both a master and slave Will have zero or one masters Must be uniquely identified in replication via server-id Master? MySQL Instance No other topological restrictions No Guarantee of replication lag Semi-sync: kind of (more later) CHANGE MASTER is painful! Slave(s)? Slave(s)? Slave(s)? Slave(s)?

4 Dual Masters Useful for Switch/Fail-over w/o CHANGE MASTER Two mysql servers, each is: Master of the other Slave of the other Server-id prevents cycles Writes on A replicate to B Then replicate back to A Then are filtered by server-id Master A Master B

5 How a Write Works on a Single Server Innodb rows getting changed are exclusively locked until the end of the transaction or statement if autocommit=1 MyISAM entire table is exclusively locked for the statement No other client can claim the exclusive lock until the first client releases it Sounds obvious, right?

6 Writes between Master and Slave The client: obtains lock(s) on the master does the write finishes, releasing the lock The write replicates to the slave The slave SQL thread Master 1. Client locks and writes on master. Then lock(s) released. 2. Replication to slave obtains lock(s) on the slave does the write finishes, releasing the lock Slave 3. SQL thread locks and writes on slave. Then lock(s) released.

7 Writes on a Dual-master Write only locks on the master client connects to The other master is lock-less during the write Nothing prevents the same lock(s) from being acquired simultaneously on the other master This violates the Isolation attribute of ACID compliance Master A Master B Client 1 Locks row 100 Client 2 Locks row 100

8 The Practical Bit

9 Why does that matter? (part 1) 1. percona1 mysql> create table t ( 2. i int unsigned not null auto_increment primary key, 3. j varchar(32) 4. ); 5. percona1 mysql> set autocommit=0; 6. percona1 mysql> begin; 7. percona1 mysql> insert into t (j) values ('percona1'); 9. percona2 mysql> set autocommit=0; 10. percona2 mysql> insert into t (j) values ('percona2'); 11. percona2 mysql> commit; 13. percona1 mysql> commit; percona1 mysql> show slave status\g Slave_SQL_Running: No Last_Errno: 1062 Last_Error: Error 'Duplicate entry '1' for key 'PRIMARY'' on query. Default database: 'test'. Query: 'insert into t (j) values ('percona2')'

10 Why does that matter? (part 2) 1. percona1 mysql> create table t( 2. i int unsigned not null auto_increment primary key, 3. username varchar(32) not null, 4. unique key( `username` ) 5. ); 6. percona1 mysql> insert into t (username) values ('user1'), ('user2'); 8. percona1 mysql> set autocommit=0; 9. percona1 mysql> update t set username='fred1234' where i=1; 10. Query OK, 1 row affected (0.01 sec) 12. percona2 mysql> set autocommit=0; 13. percona2 mysql> update t set username='fred1234' where i=2; 14. percona2 mysql> commit; 16. percona1 mysql> commit; percona1 mysql> show slave status\g Slave_SQL_Running: No Last_Errno: 1062 Last_Error: Error 'Duplicate entry 'fred1234' for key 'username'' on query. Default database: 'test'. Query: 'update t set username='fred1234' where i=2'

11 Why does that matter (part 3) 1. percona1 mysql> create table t ( 2. i int unsigned not null auto_increment primary key, 3. j varchar(32) 4. ); 5. percona1 mysql> insert into t (j) values ('test1'); 7. percona1 mysql> set autocommit=0; 8. percona1 mysql> update t set j='percona1' where i=1; 10. percona2 mysql> set autocommit=0; 11. percona2 mysql> update t set j='percona2' where i=1; 12. percona2 mysql> commit; 14. percona1 mysql> commit; percona1 mysql> select * from t; i j percona percona2 mysql> select * from t; i j percona

12 Intermingled Writes Simultaneous writes to multiple masters each master gets a different order Time Master A A1 A2 B1 A3 B2 A4 B3 B4 A5 B5 Master B B1 A1 A2 B2 B3 B4 A3 A4 A5 B5

13 Why does that matter (part 4) 1. percona1 mysql> create table t ( 2. i int unsigned not null auto_increment primary key, 3. j varchar(32) 4. ); 5. percona1 mysql> insert into t (j) values ('x'), ('x'), ('x'); 6. percona1 mysql> set autocommit=0; 7. percona1 mysql> update t set j='y'; 9. percona2 mysql> set autocommit=0; 10. percona2 mysql> insert into t (j) values ('z'); 11. percona2 mysql> commit; 13. percona1 mysql> commit; percona1 mysql> select * from t; i j y 2 y 3 y 4 z percona2 mysql> select * from t; i j y 2 y 3 y 4 y

14 Why is this a hard problem to solve? Isolation is an assumption most developers make CAP Theorem Consistency Availability Partition Tolerance Multi-writing takes Availability + Partition Tolerance Over Consistency

15 When Multi-writing can work Database entity isolation An entity is only modified by either: One client at a time, globally Only on one master at a time Relational data means data that is interrelated Perfect data isolation tends not to be realistic RDBMS + non-relational data

16 Wait, why shouldn t I...? Write to both masters for availability? Increase write scalability by writing to both masters? Put all my mysql servers in a big ring topology? Just use auto_increment_offset to solve the multiwriting problem?

17 Solutions

18 Semi-Sync Replication Designated slaves guaranteed to receive on commit Still no guarantee when the transaction is applied Multi-writer issues still a problem here Can help remove gaps in a dual-master failover

19 Single-writer management systems Multi-Master Replication Manager (MMM) Master High Availability Manager (MHA) Percona Replication Manager (PRM) Tungsten Replicator/Cluster Global Transaction IDs in MySQL 5.6/5.7

20 Percona XtraDB Cluster / Galera Synchronous replication (not standard MySQL repl) Write anywhere, read anywhere Still doesn t acquire locks on all nodes handles locking conflicts in a special way

21 Summary No distributed system behaves just like the standalone equivalent There are always tradeoffs Single-writer is usually the most application-friendly method to handle writes Multi-writing is usually (but not always) the opposite

22 Questions?