MySQL High Availability with Percona XtraDB Cluster 5.7 Hands on tutorial!

Transcription

1 MySQL High Availability with Percona XtraDB Cluster 5.7 Hands on tutorial! 1

2 Introducing pxc-strict-mode 2 Peter Zaitsev pz@percona.com Alok Pathak alok.pathak@percona.com Krunal Bauskar krunal.bauskar@percona.com

3 Table of Contents 3 Introduction to PXC Setting up Environment Bootstrapping State Transfers Certi cation Errors Flow Control What's new with PXC-5.7 Introducing pxc_strict_mode Exploring PXC through PFS Securing PXC Load Balancers Percona Monitoring and Management (PMM)

4 Introduction to PXC 4 True Multi-master replication Synchronous replication Data consistency Multi-threaded Slave Transparent to Applications No complex failover Automatic Node Provisioning

5 Introduction to PXC 5

6 Are you ready? Setting Up The Environment 6

7 Setting Up The Environment 7 Fetch a USB Stick, copy the pxc57.ova virtualbox image Open Virtualbox, File -> Import Appliance -> Browse the pxc57.ova le

8 Setting Up The Environment 8 Click Next and Import

9 Setting Up The Environment 9 Start pxc[1-3] virtual machines one by one

10 Setting Up The Environment 10 Versions Serverversion: logPerconaXtraDBCluster(GPL) ProxySQLversion1.2.3

11 Attention Hands On! 11 When you see in the right bottom, there is an exercise that you should do!

12 Testing The Environment 12 ssh/putty to: pxc1: pxc2: pxc3: rootpassword is perconapassword Verify ssh between nodes Open 2 ssh sessions to every node

13 Bootstrapping A Cluster 13 Bootstrapping a node gives it permission to form a new cluster The bootstrapped node is the source of truth for all nodes going forward Bootstrapping should NOT happen automatically without a system with split-brain protection that can coordinate it. Usually this is done manually

14 Bootstrapping PXC 14 #servicemysqlbootstrap-pxc #/etc/init.d/mysqlbootstrap-pxc #/etc/init.d/mysqlstart--wsrep-new-cluster or with systemdenvironments like Centos 7: This sets empty gcomm:// address which starts the node as the rst of the cluster. Never use it to join an existing cluster. Today we are using 64-bit Centos 7.

15 Bootstrapping PXC 15 The node you bootstrapped.. Verify it with: pxc1mysql>showglobalstatuslike'wsrep%'; Variable_name Value wsrep_local_state 4 wsrep_local_state_comment... Synced wsrep_cluster_size 1 wsrep_cluster_status wsrep_connected Primary ON... wsrep_ready ON rowsinset(0.00sec)

16 State Transfers Recap On IST/SST 16 SST: State Snapshot Transfer Snapshot the whole database and transfer, using: IST: Incremental State Transfer Only transfer missing transactions One node of the cluster is DONOR

17 State Snapshot Transfers (SST) State Snapshot Transfer Methods Methods Type Speed BlocksDonor mysqldump Logical Slow Blocks rsync Physical Fastest Blocks xtrabackup-v2 Physical Fast Briefly The recommended SST method is xtrabackup-v2 because it locks donor for least amount of time. wsrep_sst_method=xtrabackup-v2 wsrep_sst_auth=sst:s3cret wsrep_sst_donor=<donor_list>

18 Adding new nodes to the Cluster 18 pxc2#systemctlstartmysql pxc3#systemctlstartmysql More veri cation pxc3mysql>showglobalstatuslike'wsrep%'; Variable_name Value wsrep_local_state wsrep_local_state_comment 4 Synced... wsrep_cluster_size... 3 wsrep_cluster_status Primary wsrep_connected... ON rowsinset(0.00sec)

19 myq_status 19 Monitor Galera metrics using myq_status #myq_statuswsrep mycluster/pxc1(idx:0)/galera3.17(r447d194) Wsrep Cluster NodeRepl Queue Ops Bytes Conflct Gcache Wind timepcnf #StatLaten Up Dn Up Dn Up Dn lcf bfa ist idxdstap 06:31:14P 3 3Sync2.5ms 06:31:15P 3 3Sync2.6ms K K k k :31:16P 3 3Sync2.5ms 06:31:17P 3 3Sync2.5ms K K k k P=Primary:Thenodeispartofaprimarycomponent(N=Non-primary) cnf=clusterconfigid:thisincrementswhenevertheclusterstatechanges NodeStat-Syncisnormal.Otherpossiblestates:Dono,Init ReplLatency:Averagetimeforatransactiontoreplicateacrossthecluster Queue:Replicationqueues(Upisoutbound,Dnisinbound).Dnqueue->applyqueue Ops:Replicationevents.UpOpsisdatathatwaswrittenonthisnodebyaclient Bytes:Replicationsize(sameasOps,butinBytes). Conflct:lcfandbfaarereplicationconflicts

20 Bootstrapping PXC With Stop/Start Of MySQL 20 When you need to start a new cluster from scratch, you decide which node to start with and you bootstrap it #systemctlstartmysql@bootstrap That node becomes the cluster source of truth (SSTs for all new nodes)

21 Bootstrapping PXC Without Restarting MySQL 21 When a cluster is already partitioned and you want to bring it up again. 1 or more nodes need to be in Non-Primary state. Choose the node that is newest and can be enabled (to work with application) To bootstrap online: mysql>setglobalwsrep_provider_options="pc.bootstrap=true"; be sure there is NO OTHER PRIMARY partition or there will be a split brain!!

22 Bootstrapping PXC Without Restarting MySQL 22 Use Case: Only 1 of the 3 nodes is available and the other 2 nodes crashed, causing node 1 to go Non-Primary. In Multi Datacenter environments: DC1 has 2 nodes, DC2 has 1 node, If DC1 dies, the single node in DC2 will go Non-Primary. To activate secondary DC, a bootstrap is necessary

23 Recover Cleanly Shutdown Cluster 23 Run the application (run_app.shproxysql) on pxc1 One by one, stop mysql on all 3 nodes pxc1#systemctlstopmysql@bootstrap pxc2#systemctlstopmysql pxc3#systemctlstopmysql How to restart Galera Cluster?

24 Recover Cleanly Shutdown Cluster 24 To restart an entire Galera Cluster, complete the following steps: Identify the node with the highest seqno. Start the most advanced node as the rst node of the cluster. Start the rest of the node as usual. How to identify most advanced node?

25 Recover Cleanly Shutdown Cluster 25 To restart an entire Galera Cluster, complete the following steps: Identify the node with the highest seqno. Start the most advanced node as the rst node of the cluster. Start the rest of the node as usual. How to identify most advanced node? Solution #cat/var/lib/mysql/grastate.dat #GALERAsavedstate version:2.1 uuid: e5a514d5-80f2-11e ea69a6a66f6e seqno: cert_index: Bootstrap node with highest seqno and start other nodes.

26 What if node crashed? 26 On a crashed node grastate.dat will look like this: cat/var/lib/mysql/grastate.dat #GALERAsavedstate version:2.1 uuid: e5a514d5-80f2-11e ea69a6a66f6e seqno: -1 cert_index: How can you know the seqno?

27 What if node crashed? 27 On a crashed node grastate.dat will look like this: cat/var/lib/mysql/grastate.dat #GALERAsavedstate version:2.1 uuid: e5a514d5-80f2-11e ea69a6a66f6e seqno: -1 cert_index: How can you know the seqno? Solution #mysqld_safe--wsrep-recover Loggingto'/var/lib/mysql/error.log'. Startingmysqlddaemonwithdatabasesfrom/var/lib/mysql WSREP:Runningpositionrecoverywith--log_error='/var/lib/mysql/wsrep_recovery.Uh8R WSREP:Recoveredpositione5a514d5-80f2-11e ea69a6a66f6e: mysqldfrompidfile/var/lib/mysql/pxc1.pidended

28 When putting in production unprepared... Certification Errors 28

29 Certification 29 What it does: Determine if writeset can be applied. Based on unapplied earlier transactions on master Such con icts must come from other nodes Happens on every node, individually Deterministic Results are not reported to other nodes in the cluster, as every node does certi cation and is a determinstic process. Pass: enter apply queue (commit success on master) Fail: drop transaction (or return deadlock on master) Serialized by GTID Cost based on # of keys or # of rows

30 Certification 30

31 Certification 31

32 Certification 32

33 Certification 33

34 Certification 34

35 Certification 35

36 Certification 36

37 Conflict Detection 37 Local Certi cation Failure (lcf) Transaction fails certi cation Post-replication Deadlock/Transaction Rollback Status Counter: wsrep_local_cert_failures Brute Force Abort (bfa) (Most Common) Deadlock/Transaction rolled back by applier threads Pre-commit Transaction Rollback Status Counter: wsrep_local_bf_aborts

38 Conflict Deadlock/Rollback 38 Note: Transaction Rollback can be gotten on any statement, including SELECT and COMMIT Example: pxc1mysql>commit; ERROR1213(40001):WSREPdetecteddeadlock/conflictandabortedthe transaction.tryrestartingthetransaction

39 Multi writer Conflict Types Brute Force Abort (bfa) 39 Transaction rolled back by applier threads Pre-commit Transaction Rollback can be gotten on any statement, including SELECT and COMMIT Status Counter: wsrep_local_bf_aborts

40 Brute Force Abort (bfa) 40

41 Brute Force Abort (bfa) 41

42 Brute Force Abort (bfa) 42

43 Brute Force Abort (bfa) 43

44 Brute Force Abort (bfa) 44

45 Brute Force Abort (bfa) 45

46 Brute Force Abort (bfa) 46

47 Multi writer Conflict Types Local Certification Failure (lcf) 47 Transaction fails certi cation Post-replication Deadlock on commit Status Counter: wsrep_local_cert_failures

48 Local Certification Failure (lcf) 48

49 Local Certification Failure (lcf) 49

50 Local Certification Failure (lcf) 50

51 Local Certification Failure (lcf) 51

52 Local Certification Failure (lcf) 52

53 Local Certification Failure (lcf) 53

54 Local Certification Failure (lcf) 54

55 Local Certification Failure (lcf) 55

56 Local Certification Failure (lcf) 56

57 Local Certification Failure (lcf) 57

58 Local Certification Failure (lcf) 58

59 Local Certification Failure (lcf) 59

60 Reproducing Conflict 60 On pxc1, create test table: pxc1mysql>createtabletest.deadlocks( iintunsignednotnullprimarykey, jvarchar(32), tdatetime ); pxc1mysql>insertintotest.deadlocksvalues(1,null,null); Run myq_status on pxc1: #myq_statuswsrep mycluster/pxc1(idx:1)/galera3.17(r447d194) Wsrep Cluster NodeRepl Queue Ops Bytes Conflct Gcache Wind timepcnf #StatLaten Up Dn Up Dn Up Dn lcf bfa ist idxdstap 06:47:51P 5 3Sync2.6ms K k :47:52P 5 3Sync2.6ms K8.0b k

61 Reproducing Conflict 61 On pxc1: pxc1mysql>begin; pxc1mysql>updatetest.deadlockssetj='pxc1',t=now()wherei=1; Before commit, go to pxc3: pxc3mysql>begin; pxc3mysql>updatetest.deadlockssetj='pxc3',t=now()wherei=1; pxc3mysql>commit; Now commit the transaction on pxc1: pxc1mysql>commit; pxc1mysql>select*fromtest.deadlocks;

62 Reproducing Conflict 62 It fails: pxc1mysql>commit; ERROR1213(40001):WSREPdetecteddeadlock/conflictandabortedthe transaction.tryrestartingthetransaction

63 Reproducing Conflict 63 It fails: pxc1mysql>commit; ERROR1213(40001):WSREPdetecteddeadlock/conflictandabortedthe transaction.tryrestartingthetransaction Which commit succeeded? Is this a lcf or a bfa? How would you diagnose this error?

64 Reproducing Conflict 64 Which commit succeeded? PXC3, rst one that got in cluster. Is this a lcf or a bfa? BFA How would you diagnose this error? #myq_statuswsrep mycluster/pxc1(idx:1)/galera3.17(r447d194) Wsrep Cluster NodeRepl Queue Ops Bytes Conflct Gcache Window timepcnf #StatLaten Up Dn Up Dn Up Dn lcf bfaist idxdstapp 06:48:10P 5 3Sync2.4ms K k :48:11P 5 3Sync2.4ms K0.7K k :48:12P 5 3Sync2.4ms K8.0b k showglobalstatuslike'wsrep_local_bf%'; showglobalstatuslike'wsrep_local_cert%'; Variable_name Value wsrep_local_bf_aborts 1 wsrep_local_cert_failures

65 Reproducing Conflict Log Conflicts pxc1mysql>setglobalwsrep_log_conflicts=on; 65 **PriorityTRANSACTION: TRANSACTION839659,ACTIVE0secstartingindexread mysqltablesinuse1,locked1 1lockstruct(s),heapsize1136,0rowlock(s) MySQLthreadid2,OSthreadhandle ,queryid648862Systemlock **VictimTRANSACTION: TRANSACTION838894,ACTIVE18sec 2lockstruct(s),heapsize1136,1rowlock(s),undologentries1 MySQLthreadid26212,OSthreadhandle ,queryid640599localhostro **WAITINGFORTHISLOCKTOBEGRANTED: RECORDLOCKSspaceid26pageno5nbits144indexPRIMARYoftablesbtest.sbtest T06:57: Z2[Note]WSREP:clusterconflictduetohighpriorityabo T06:57: Z2[Note]WSREP:Winningthread: THD:2,mode:applier,state:executing,conflict:noconflict,seqno: SQL:(null) T06:57: Z2[Note]WSREP:Victimthread: THD:26212,mode:local,state:idle,conflict:noconflict,seqno:-1 SQL:(null)

66 Reproducing Conflicts Summary 66 Con icts are of a concern when determining if PXC is a t for the application: Long running transactions increase chance of con Heavy write workload on multiple nodes Large transactions increase chance of con icts icts These issues can usually be resolved by writing to 1 node only.

67 Hey! Wait for me! Flow Control 67

68 Flow Control 68 Avoiding nodes drift behind (slave lag?) Any node in the cluster can ask the other nodes to pause writes if it lags behind too much. Caused by wsrep_local_recv_queue exceeding a node s gcs.fc_limit Can cause all writes on all nodes in the entire cluster to be stalled.

69 Flow Control 69

70 Flow Control 70

71 Flow Control 71

72 Flow Control 72

73 Flow Control 73

74 Flow Control 74

75 Flow Control 75

76 Flow Control 76

77 Flow Control 77

78 Flow Control 78

79 Flow Control 79

80 Flow Control 80

81 Flow Control 81

82 Flow Control 82

83 Flow Control 83

84 Flow Control 84

85 Flow Control 85

86 Flow Control 86

87 Flow Control 87

88 Flow Control 88

89 Flow Control Status Counters 89 wsrep_ ow_control_paused_ns: nanoseconds since starts of node did the cluster get stalled. wsrep_ ow_control_recv: Amount of ow control messages received from other nodes wsrep_ ow_control_sent: Amount of ow control messages sent to other nodes

90 Flow Control Observing Flow Control 90 Run the application (run_app.sh) Run myq_statuswsrepon all nodes. LOCKTABLEsbtest.sbtest1on pxc3 and observe its effect on the cluster.

91 Flow Control Observing Flow Control 91 Run the application (run_app.sh) Run myq_statuswsrepon all nodes. LOCKTABLEsbtest.sbtest1on pxc3 and observe its effect on the cluster. pxc1 run_app.sh all nodes: myq_statuswsrep pxc3mysql>setglobalpxc_strict_mode='disabled'; pxc3mysql>locktablessbtest.sbtest1write; waituntilflowcontrolkicksin... pxc3mysql>unlocktables; pxc3mysql>setglobalpxc_strict_mode='enforcing';

92 Flow Control 92 Any node in the cluster can ask the other nodes to pause writes if it lags behind too much. A node can lag behind more before sending ow control messages, controlled via gcs.fc_limit. This can be controlled per node. Is there another alternative?

93 Flow Control DESYNC mode 93 It's possible to let a node going behind the ow control limit. This can be performed by setting wsrep_desync=on

94 Flow Control DESYNC mode 94 It's possible to let a node going behind the ow control limit. This can be performed by setting wsrep_desync=on pxc3mysql>setglobalwsrep_desync=on;

95 What's new in PXC 5.7? 95 Save your workload from experimental features: Introducing pxc-strict-mode. Extended support for PFS. Now monitor Galera and wsrep instrument through PFS. Support for encrypted tablespaces in Multi-Master Topology. PXC can wire encrypted tablespace to new booting node. Proxy-SQL compatible PXC PXC is ProxySQL compatible with new single step con guration.

96 What's new in PXC 5.7? 96 PMM enabled monitoring for PXC Effectively monitor PXC using PMM. MySQL/PS and galera-3.17 compatibility Bug xes, Improved logging and lot more. Simpli ed packaging for PXC PXC packages now takes care of galera installation too. Support to use latest Xtrabackup with enhanced security checks. Feel power of improved and latest XB with PXC.

97 Why pxc_strict_mode? PXC/Galera limitations: 97 Supports only Transactional Storage Engine. Explicit locks don t t in multi-master topology. Need of primary key for certi cation data generation. XA-transaction are not supported.

98 Understanding pxc_strict_mode 98 ENFORCING (DEFAULT/RECOMMENDED) Use of experimental features raises error. (during startup server refuse to start and runtime operation is blocked. error is logged) DISABLED PXC-5.6 compatible. Allows experimental feature. No error. No warning PERMISSIVE Use of experimental features results in a warning at startup and runtime. Server continues to accept the setting and operate. MASTER Same as ENFORCING for all experimental features except explicttable-locking validation checks are not performed under this mode.

99 Understanding pxc_strict_mode 99 pxc_strict_mode is local to given node and if user plan to toggle it, it should be done on all the nodes for cluster consistency and correctness. Toggling pxc_strict_mode from less strict mode to more strict mode For example: DISABLED -> ENFORCING... need to ensure ENFORCING characteristics are met. Like wsrep_replicate_myisam=off binlog_format=row log_output=file/none tx_isolation=serializable.

100 pxc_strict_mode blocks 100 Semantics blocked: DML and DDL operations (except CREATE/DROP) are not permitted on non-transactional Storage Engine. Table can be converted from non-transactional SE to transactional SE using ALTER. Trying to enable MyISAM replication is blocked. binlog-format has to be ROW.

101 pxc_strict_mode blocks 101 Semantics blocked: DML to tables without primary-key is not allowed. log-output has to be directed to FILE or DISABLED (NONE). Explicit TABLE locking feature (LOCK table, GET_LOCK, FLUSH TABLE WITH READ LOCK, Setting SERIALIZABLE transaction level) is blocked. auto-increment mode has to be INTERLEAVED. Combining Schema and DML changes in single statement like CTAS is not permitted.

102 pxc_strict_mode hands on 102 Let's try some simple command with pxc-strict-mode ENFORCING pxc3mysql>createtablet1(iint,primarykeypk(i))engine=myisam; pxc3mysql>insertintot1values(1); ERROR1105(HY000):Percona-XtraDB-ClusterprohibitsuseofDMLcommand onatable(test.t1)thatresidesinnon-transactionalstorageengine withpxc_strict_mode=enforcingormaster pxc3mysql>setwsrep_replicate_myisam=1; ERROR1105(HY000):Percona-XtraDB-ClusterprohibitsuseofMyISAMtable replicationfeaturewithpxc_strict_mode=enforcingormaster pxc3mysql>altertablet1engine=innodb; pxc3mysql>insertintot1values(1); pxc3mysql>locktablet1read; ERROR1105(HY000):Percona-XtraDB-Clusterprohibitsuseof LOCKTABLE/FLUSHTABLE<table>WITHREADLOCKwithpxc_strict_mode=ENFORCING pxc3mysql>droptablet1;

103 Let's have a small test now? 103 pxc3mysql>setglobalpxc_strict_mode='disabled'; pxc3mysql>createtablemyisam(iint,primarykeypk(i))engine=myisam; pxc3mysql>setglobalwsrep_replicate_myisam=1; pxc3mysql>insertintomyisamvalues(1),(2),(3); pxc3mysql>altertablemyisamaddcolumnjint; pxc3mysql>updatemyisamsetj=100; pxc3mysql>select*frommyisam; pxc3mysql>droptablemyisam; pxc3mysql>setglobalpxc_strict_mode='enforcing'; WillitPASS?FAIL?WARNING?

104 Restore pxc_strict_mode 104 Remember to restore the mode back to ENFORCING before we proceed with next set of experiment. pxc3mysql>setglobalwsrep_replicate_myisam=0; pxc3mysql>setglobalpxc_strict_mode='enforcing';

105 Monitoring PXC through Performance Schema Traditional method to monitor PXC or MySQL through log le is time consuming and may need special tool even to detect occurrence of an event. Performance Schema is effective way and has become de facto standard for monitoring different elements of MySQL. Till 5.6 PXC had limited support for performance_schema where-in only wsrep related instruments were exposed through performance schema that too in limited fashion. Starting PXC-5.7 we have taken big-step further enabling monitoring of galera instruments and other wsrep-instruments as part of performance schema. 105

106 Monitoring PXC through Performance Schema Instruments that are monitored: THREADS: applier, rollback, service_thd, gcomm conn, receiver, sst/ist threads, etc LOCK/COND_VARIABLES: from wsrep and galera library. FILE: record-set le, ring-buffer le (default gcache), gcache-page le. STAGES: Different stage threads are passing through. With this information, user should able to track some of the most important instruments that can help get some insight as to where server is really spending time or out-of-path ow like rollback of transactions. 106

107 Experimenting P_S threads 107 Let's see what all different threads are active: pxc3mysql>selectthread_id,namefromperformance_schema.threads wherenamelike'%galera%'ornamelike'%wsrep%'; thread/galera/thread_galera_service_thd 4 thread/galera/thread_galera_gcommconn 5 thread/galera/thread_galera_receiver 6 thread/sql/thread_wsrep_applier 7 thread/sql/thread_wsrep_rollbacker rowsinset(0.03sec) pxc3mysql>setglobalwsrep_slave_threads=10; pxc3mysql>selectcount(*)fromperformance_schema.threads wherenamelike'%thread_wsrep_applier%'; pxc3mysql>setglobalwsrep_slave_threads=1; pxc3mysql>selectcount(*)fromperformance_schema.threads wherenamelike'%thread_wsrep_applier%'; pxc1#/root/bin/run_app.sh[stopappafter2seconds] pxc3mysql>selectcount(*)fromperformance_schema.threads wherenamelike'%thread_wsrep_applier%';

108 Experimenting P_S files 108 Let's now checkout les pxc3mysql>select*fromperformance_schema.file_instances wherefile_namelike '%galera%'; FILE_NAME EVENT_NAME OPEN_COU /var/lib/mysql/galera.cache wait/io/file/galera/file_galera_ringbuffer rowinset(0.02sec) pxc3mysql>select*fromperformance_schema.file_instances wherefile_namelike'%gcache%'; FILE_NAME EVENT_NAME /var/lib/mysql/dn1/gcache.page wait/io/file/galera/file_galera_gcache_page

109 Experimenting P_S files 109 Some other importants points related to les: Galera uses mmap for most of these les and so we avoid updating read/write stats for le available through le_summary_by_instance/ le_summary_by_event_name.* Only dynamically changing caching write-set are cached. grastate/gvwstate les are not tracked.

110 Experimenting P_S mutex/cond_var 110 Enable instruments: pxc3mysql>useperformance_schema; pxc3mysql>updatesetup_consumerssetenabled='yes' wherenamelike'%events_waits_%'; pxc3mysql>updatesetup_consumerssetenabled='yes' wherenamelike'%events_stages_%'; pxc3mysql>updatesetup_instrumentssetenabled='yes',timed='yes' wherenamelike'%wsrep%'ornamelike'%galera%'; pxc3mysql>usetest;createtablet(iint,primarykeypk(i))engine=innodb; pxc3mysql>begin;insertintotvalues(1); pxc2mysql>usetest;begin;insertintotvalues(1);commit; pxc3mysql>commit; pxc3mysql>selectthread_id,event_name,timer_waitfrom performance_schema.events_waits_historywhereevent_namelike'%rollback%'; pxc3mysql>selectthread_id,event_name,timer_waitfrom performance_schema.events_waits_currentwhereevent_namelike'%rollback%'; pxc3mysql>selectthread_id,event_id,event_name,timer_waitfrom performance_schema.events_waits_historywhereevent_namelike'%monitor%'; (Checktheapplyandcommitmonitorbeinginvokedbyapplierthread)

111 Experimenting P_S stages 111 Findout precisely what the threads are doing pxc3mysql>select*fromperformance_schema.events_stages_current; [Whatarethreadcurrentlyexecuting.] pxc3mysql>select*fromperformance_schema.events_stages_history; [Whatactionwasreplicatedearlier.UPDATE,INSERT,DELETE. Wasitfinallycommitted] stage/wsrep/wsrep:aborteractive wsrep stage/wsrep/wsrep:applyingwriteset wsrep stage/wsrep/wsrep:writingrows 167 stage/wsrep/wsrep:applieridle log_e wsrep stage/wsrep/wsrep:committing wsrep pxc3mysql>useperformance_schema; pxc3mysql>updatesetup_consumerssetenabled='no' wherenamelike'%events_waits_%'; pxc3mysql>updatesetup_consumerssetenabled='no' wherenamelike'%events_stages_%'; pxc3mysql>updatesetup_instrumentssetenabled='no',timed='no' wherenamelike'%wsrep%'ornamelike'%galera%';

112 Securing PXC 112 There are 2 main components to secure. Traf c (SST/IST inter-node traf c). [5.6 feature] Tablespace. [5.7 feature] Securing inter-node traf c is acheieved using SSL and support is present since PXC-5.6. For SST using xtrabackup-v2 or mysqldump can help achieve this. Recommended approach is xtrabackup due to added exibility in locking database while servicing SST. For IST galera in-build SSL support ensure that replication channel is secured.

113 Securing PXC 113 PXC-5.7 extended support for MySQL introduced encrypted tablespace. This means using XB/mysqldump these encrpyted tablespaces will be moved to new booting node and re-encrpyted with new node con guration making them auto-available to end-user in secured fashion. This help completes the loop of ensuring everything is secured in PXC. --early-plugin-load [keyring_file_data]

114 Spread the load Load Balancers 114

115 Load Balancers 115 With PXC a Load Balancer is commonly used: Lot's of choice HAProxy (most-common) ProxySQL MariaDB MaxScale ScaleArc (proprietary) MySQL Router

116 Load Balancers ProxySQL 116 High-performance Proxy for MySQL and forks Query routing and read/write split On-the- y rewrite of queries Caching reads outside the database server Supports failover High Availability Runtime recon guration

117 Load Balancers ProxySQL 117 All backends are grouped into hostgroups Hostgroups have logical functionalities

118 Load Balancers ProxySQL 118 All backends are grouped into hostgroups Hostgroups have logical functionalities HG10 : Write masters

119 Load Balancers ProxySQL 119 All backends are grouped into hostgroups Hostgroups have logical functionalities HG10 : Write masters HF11 : Read Slaves

120 Load Balancers ProxySQL 120 All backends are grouped into hostgroups Hostgroups have logical functionalities HG10 : Write masters HF11 : Read Slaves Read/Write Split

121 Load Balancers ProxySQL Scheduler 121 Scheduler is a cron-like solution embedded directly inside ProxySQL Can run any external scripts Support for Percona XtraDB Cluster

122 Load Balancers ProxySQL Admin 122 Tool for con guring Percona XtraDB Cluster nodes with ProxySQL --enableoption to automatically con gure a PXC into ProxySQL Add Percona XtraDB Cluster node into the ProxySQL database Add the monitoring scripts into the ProxySQL scheduler table Create two new users, one user for monitoring, other one for communicating with the cluster --disableoption to remove PXC con guration from ProxySQL

123 Load Balancers ProxySQL 123

124 Load Balancers ProxySQL Admin mysql--host= user=admin--password=admin--port= YourMySQLconnectionidis20795 Serverversion:5.5.30(ProxySQLAdminModule) 124 ProxySQL Admin(proxysql-admin) to con gure PXC nodes into ProxySQL. /bin/proxysql-admin--proxysql-user=admin--proxysql-password=admin\ --proxysql-port=6032--proxysql-host= \ --cluster-user=test--cluster-password=test--cluster-port=3306\ --cluster-host= \ --galera-check-interval=1000--enable--mode='loadbal'--adduser

125 Load Balancers ProxySQL Setup pxc1mysql>selecthostgroup_id,hostname,port,statusfrommysql_servers; hostgroup_id hostname port status pxc ONLINE 10 pxc ONLINE 10 pxc ONLINE rowsinset(0.00sec) 125 pxc1mysql>selectusername,password,active,default_hostgroupfrommysql_users; username password active default_hostgroup test *94BDCEBE19083CE2A1F rowinset(0.01sec)

126 Load Balancers ProxySQL Setup Usage:/bin/proxysql_galera_checker<hostgroup_idwrite>[hostgroup_idread] [numberwriters][writersarereaders0 1}[log_file] 126 pxc1mysql>selectid,active,interval_ms,filename,arg1,arg2, arg3,arg4,arg5fromscheduler\g **************************1.row*************************** id:10 active:1 interval_ms:1000 filename:/bin/proxysql_galera_checker arg1:10 arg2:10 arg3:3 arg4:1 arg5:/var/lib/proxysql/proxysql_galera_check.log **************************2.row*************************** id:11 active:1 interval_ms:5000 filename:/bin/proxysql_node_monitor arg1:10 arg2:/var/lib/proxysql/proxysql_node_monitor.log...

127 Load Balancers ProxySQL Load balancing 127 ProxySQL is con gured to listen on port 6033for applications whiletrue; do sleep1; done pxc pxc pxc

128 Load Balancers ProxySQL R/W Split 128

129 Load Balancers ProxySQL Hostgroups 129 All backends are grouped into hostgroups Serverversion:5.5.30(ProxySQLAdminModule)... pxc1mysql>selecthostgroup_id,hostname,port,status,weightfrommysql_servers; hostgroup_id hostname port status weight pxc ONLINE pxc ONLINE pxc ONLINE pxc ONLINE pxc ONLINE pxc ONLINE rowsinset(0.00sec)

130 Load Balancers ProxySQL R/W split pxc1mysql>selectactive,match_pattern,destination_hostgroup,apply frommysql_query_rules; active match_pattern destination_hostgroup apply ^SELECT.* ^SELECT.*FORUPDATE rowsinset(0.00sec) 130 Application User: pxc1mysql>selectusername,password,active,default_hostgroupfrommysql_users; username password active default_hostgroup test *94BDCEBE19083CE2A1F959FD rowinset(0.00sec)

131 Load Balancers ProxySQL Scheduler Usage:/bin/proxysql_galera_checker [hostgroup_idwrite][hostgroup_idread][numberwriters] [writersarereaders0 1][log_file] 131 pxc1mysql>selectid,active,interval_ms,filename, arg1,arg2,arg3, arg4,arg5fromscheduler\g **************************1.row*************************** id:10 active:1 interval_ms:3000 filename:/bin/proxysql_galera_checker arg1:10 arg2:11 arg3:1 arg4:0 arg5:/var/lib/proxysql/proxysql_galera_check.log 1rowinset(0.00sec)

132 Load Balancers ProxySQL Scheduler pxc1mysql> LOADMYSQLSERVERSTORUNTIME; SAVEMYSQLSERVERSTODISK; LOADMYSQLUSERSTORUNTIME; SAVEMYSQLUSERSTODISK; LOADMYSQLQUERYRULESTORUNTIME; SAVEMYSQLQUERYRULESTODISK; LOADSCHEDULERTORUNTIME; SAVESCHEDULERTODISK; LOADMYSQLVARIABLESTORUNTIME; SAVEMYSQLVARIABLESTODISK; 132

133 Load Balancers ProxySQL R/W split pxc1mysql>selecthostgroup_id,hostname,port,status,weightfrommysql_servers; hostgroup_id hostname port status weight pxc ONLINE pxc OFFLINE_SOFT pxc OFFLINE_SOFT pxc OFFLINE_SOFT pxc ONLINE pxc ONLINE rowsinset(0.00sec) 133

134 Load Balancers ProxySQL R/W split 134 ProxySQL is con gured to listen on port 6033for applications whiletrue; do sleep1; done pxc pxc

135 Load Balancers ProxySQL R/W split 135 Let's verify how it balance our writes query: whiletrue; do mysql-hpxc1-p6033-utest-ptest\ orderbyrand()limit1forupdate;"2>/dev/null; sleep1; done id k pxc id k pxc

136 MySQL High Availability with Percona XtraDB Cluster 5.7 Q&A 136

137 137 Remember to rate our talk on the mobile app! Peter Zaitsev Alok Pathak Krunal Bauskar