G a l e r a C l u s t e r Schema Upgrades

Transcription

1 G a l e r a C l u s t e r Schema Upgrades Seppo Jaakola Codership

2 Agenda Galera Cluster Overview DDL vs DML Demo of DDL Replication in Galera Cluster Rolling Schema Upgrade (RSU) Total Order Isolation (TOI) Non Blocking Operations (NBO) Pt-online-schema-change InnoDB ONLINE DDL Summary 2

3 Galera Cluster Clients Synchronous Replication Multi-Master Read & Writes to Any Node Automatic Node provisioning MariaDB REP API MariaDB REP API MariaDB REP API No Lost Transactions No Slave Lag Scalability Works in LAN / WAN / Cloud Galera Replication Plugin 3

4 DML vs DDL Transactional SQL All DML on InnoDB ACID Can be rolled back NON Transactional SQL DDL, DCL..., DML on non transactional SE Not possible to rollback Requires up-front locking Galera uses optimistic concurrency control, and depends on possibility to rollback a conflicting operation Only transactional SQL can be replicated through Galera transactional replication For others (DDL, DCL...), we either have to skip replication or use upfront locking 4

5 Schema Upgrade Schema Upgrade Transactional DML Transactional DML Transactional DML Daase State Transactional State Transactional State Transactional State 5

6 DDL Challenge DDL execution may take long time, and due to strict commit order requirement, replication may pause In synchronous cluster, paused replication will mean hanging cluster 6

7 Strict Commit Order Transactions in replication ### 101 = DML = DDL 7

8 Strict Commit Order applying committing

9 Strict Commit Order applying committing

10 Strict Commit Order applying committing

11 DML vs DDL Notes on some DDL: TRUNCATE is DDL! Is fast to execute, but nevertheless has some impact OPTIMIZE, REPAIR, ANALYZE Table admin commands are now replicated CREATE / DROP INDEX Hold MDL on affected le, and can stall the replication 11

12 Schema Upgrades in Galera Cluster

13 Schema Upgrades in Galera Galera Cluster 3 has two inbuilt methods for DDL replication: TOI Total Order Isolation RSU Rolling Schema Upgrade Galera 4 has optimized TOI method, to be non blocking in replication level. This DDL replication method is called Non Blocking DDL (NBO) The DDL replication method of choice is declared by variable: wsrep_osu_method = TOI RSU NBO 13

14 Demo of TOI, RSU and NBO

15 Total Order Isolation

16 Total Order Isolation - TOI ALTER TABLE... Node A Node B Node C Galera Replication 16

17 Total Order Isolation - TOI ALTER TABLE... Node A Node B Node C Galera Replication ALTER 17

18 Total Order Isolation - TOI ALTER TABLE... Node A Node B Node C ALTER ALTER ALTER Galera Replication 18

19 Total Order Isolation - TOI ALTER TABLE... Node A Node B Node C Galera Replication 19

20 Rolling Schema Upgrade

21 Non-Blocking DDL - RSU Node A Node B Node C Galera Replication 21

22 Non-Blocking DDL - RSU ALTER TABLE... High priority DDL Not replicated Node A Node B Node C desync Galera Replication 22

23 Non-Blocking DDL - RSU ALTER TABLE... Node A Node B Node C desync Galera Replication 23

24 Non-Blocking DDL - RSU Read only Node A Node B Node C Galera Replication 24

25 Non-Blocking DDL - RSU ALTER TABLE... Read only Node A Node B Node C Galera Replication 25

26 Non-Blocking DDL - RSU Read only Read only Node A Node B Node C Galera Replication 26

27 Non-Blocking DDL - RSU ALTER TABLE... Node A Node B Node C Galera Replication 27

28 Non-Blocking DDL - RSU Node A Node B Node C Galera Replication 28

29 RSU Requires schema backwards compatibility MySQL ROW replication compatibility must be honored Binlogs will deviate, however: After full RSU round, binlog event counts should be matching RSU events are logged with different GTID group 29

30 Schema Backward Compatibility

31 Schema Backward Compatibility App v1 App Upgrade App v2 MySQL Schema v1 Schema Upgrade MySQL Schema v2 MySQL Schema v2 New application version may use the new schema features 31

32 Application Backward Compatibility App App v1 Upgrade App v2 App v2 MySQL Schema v1 MySQL Schema v1 Schema Upgrade MySQL Schema v2 New application version must be able to use the old schema 32

33 Backwards Compatibility New/old application should be able to use both old and new schema Application should be backwards compatible ROW replication between old and new schema should be possible Schema change should be backwards compatible 33

34 App Backwards Compatibility New/old application should be able to use both old and new schema New application can have compatibility mode to detect the version of underlying daase If old app cannot use new schema, the old app must connect to one node only, which will be the last to upgrade Dropping les or columns can be a problem But drops can be done delayed: e.g. in v2 -> v3 upgrade, obsolete v1 elements can be dropped as neither v2 or v3 app will use them any more Creating unique indexes or new constraints may cause problems for compatibility 34

35 ROW Replication Compatibility MySQL guarantees ROW replication event compatibility with some limitations Newer MySQL versions tolerate more variation between source and target les, check out this page for latest status: Source and target can have different number of columns But columns must be in same order New columns in the end, and must have default values Some data type conversions are also supported 35

36 ROW Replication Compatibility ALTER TABLE points ADD COLUMN (z int default 0) x y x y z points points 36

37 ROW Replication Compatibility Insert into points(x,y) values (5,7) x y 5 7 x y z Replication points points 37

38 ROW Replication Compatibility ALTER TABLE points DROP COLUMN y x y 5 7 x z 5 0 points points 38

39 ROW Replication Compatibility Insert into points(x,y) values (12,199) x y x z Replication points points 39

40 Non Blocking DDL

41 Non-Blocking DDL Current TOI replication blocks whole cluster for the duration of DDL statement processing Galera Cluster 4.0 optimizes DDL replication (TOI) to lock only the affected le 41

42 Non-Blocking DDL - NBO ALTER TABLE t1 Node A Node B 42

43 DDL Start Phase ALTER TABLE t1 Node A Node B ALTER t1 seqno 43

44 Non-Blocking DDL - NBO ALTER TABLE t1 Node A Node B ALTER t1 ALTER t1 seqno 44

45 Non-Blocking DDL - NBO ALTER TABLE t1 UPDATE t2 Node A Node B ALTER t1 ALTER t1 45

46 Non-Blocking DDL - NBO ALTER TABLE t1 UPDATE t1 Node A Node B ALTER t1 ALTER t1 46

47 Non-Blocking DDL - NBO UPDATE t1 UPDATE t1 ALTER TABLE t1 Node A Node B ALTER t1 ALTER t1 47

48 Non-Blocking DDL - NBO UPDATE t1 UPDATE t1 ALTER TABLE t1 UPDATE t3 Node A Node B ALTER t1 ALTER t1 48

49 DDL End Phase UPDATE t1 UPDATE t1 ALTER TABLE t1 Node A Node B DDL end protocol ALTER t1 ALTER t1 seqno 49

50 Non-Blocking DDL - NBO UPDATE t4 UPDATE t1 UPDATE t1 Node A Node B 50

51 Non Blocking DDL Replication sets no locks on DDL But, affected le is locked in all cluster nodes in MySQL terms Online DDL in MySQL 5.6 makes some part of DDL completely non-blocking Online DDL will be supported case by case Other les are accessible to everybody 51

52 Pt-online-schema-change InnoDB ONLINE DDL

53 Pt-online-schema-change A perl script to modify le structure in backround 53

54 Pt-online-schema-change 1. Create new le with same structure 2. alter new le 3. Install triggers to copy changes in old le to new le Insert trigger Update trigger emp Delete trigger _emp_new 54

55 Pt-online-schema-change 4. copy data from old le to new le in small chunks emp _emp_new 55

56 Pt-online-schema-change Client transactions on old le will be reflected to the new le by triggers Update emp Update trigger emp _emp_new 56

57 Pt-online-schema-change 5. Rename new le to replace old le emp _emp_new emp 57

58 Pt-online-schema-change Is supported by Galera Cluster DDL Statements must be run by TOI Data copy transactions are replicated by Galera Triggers fire in master node and trigger data changes in new le are replicated by Galera Some limitations, e.g for trigger & foreign key support 58

59 Pt-osc in Galera Cluster Pt-osc. OLD NEW CREATE TABLE... ALTER TABLE... CREATE TRIGGER... TOI TOI TOI TOI TOI 59

60 Pt-osc in Galera Cluster Pt-osc. OLD NEW Chunk copying 60

61 Pt-osc in Galera Cluster Pt-osc. OLD NEW RENAME TABLE... OLD NEW 61

62 InnoDB ONLINE DDL InnoDB enhancement to process certain DDL in non blocking way Logically does the same thing as pt-onlineschema-change But operates internally inside InnoDB ONLINE DDL support in Galera Cluster requires deeper integration, and 4.0 does not yet support online DDL 62

63 InnoDB ONLINE DDL in Replication ALTER TABLE..., LOCK=NONE. le le ALTER 63

64 InnoDB ONLINE DDL in Replication ALTER TABLE..., LOCK=NONE. ALTER ALTER le le 64

65 InnoDB ONLINE DDL in Replication ALTER TABLE..., LOCK=NONE. Update le Update le ALTER ALTER le le 65

66 Summary DDL is interruptive operation Plan your schema upgrades up-front Pick best strategy, TOI, NBO (in galera 4), RSU, pt-osc For RSU, pay attention to ROW replication compatibility Test your schema upgrade strategy 66