Percona Live Europe 2016 Use ProxySQL to Improve Your MySQL High Availability Solution Marco Tusa Manager Consulting Amsterdam, Netherlands October 3

Transcription

1 Percona Live Europe 2016 Use ProxySQL to Improve Your MySQL High Availability Solution Marco Tusa Manager Consulting Amsterdam, Netherlands October 3 5, 2016

2 Percona Live 2017 Use ProxySQL to Improve Your MySQL High Availability Solution Marco tusa Santa Clara, California April 24th 27th, 2017

3 About me Marco The Grinch Open source enthusiast 3

4 Agenda Brief ProxySQL introduction How it interact Master-slave How interact with PXC Extending with scheduler ProxySQL HA Time cost 4

5 What is HA % (36 days) MySQL Replication % (8 hours) Linux Heartbeat with DRBD % (8 hours) RHCS with Shared Storage (Active/Passive) % (52 minutes) MHA with at least 3 nodes % (52 minutes) Linux Heartbeat with DRBD and Replication % (52 minutes) Linux Heartbeat with Shared Storage and Replication % (26 minutes) Multi Master (galera - percona cluster) 3 node minimum % (5 minutes) MySQL Cluster 5

6 What is HA 6

7 What is HA 7

8 Who will save the Day? 8

9 Who will save the Day? 9

10 Objectively What we have is: MySQL Group Replication 10

11 The telephone game 11

12 All talking 12

13 Black box: 13

14 Use the Right Tool for the Job 14

15 What we need 15

16 Solution should be MANAGEBLE FAST SCALABLE FREE 16

17 My Motto and do the right thing! 17

18 ProxySQL 18

19 ProxySQL 19

20 Understand the basics - what it does... Multiple Writers Query re-write Write / Read (Master/Slave) Sharding & Geographic Distribution Traffic Load dispatch (aka Mirroring) 20

21 Understand the basics how this is done? Define Users Servers group by function Host Groups Rules define what, where and how Query Rules Special Grouping for Replication Replication Host Group 21

22 Understand the basics HostGroup (HG) & Replication HG 22

23 Understand the basics Query rules 23

24 Understand the basics Scheduler 24

25 Basic scenario HG: 10 U: app_rw HG: 11 U: app_r HG: 12 U: ETL_R 25

26 Basic scenario - Users insert into mysql_users (username,password,active,default_hostgroup,default_schema) values ('app_rw','test',1,10,'test_schema'); insert into mysql_users (username,password,active,default_hostgroup,default_schema) values ('app_r','test',1,11,'test_schema'); insert into mysql_users (username,password,active,default_hostgroup,default_schema) values ('etl_r','test',1,12, etl_schema'); LOAD MYSQL USERS TO RUNTIME;SAVE MYSQL USERS TO DISK; 26

27 Basic scenario Servers / HG INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES (' ',10,3306,1000); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES (' ',11,3306,10000,2); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES (' ',11,3306,10000,2); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES (' ',11,3306,1,2); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES (' ',12,3306,10); INSERT INTO mysql_replication_hostgroups VALUES (10,11); LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK; 27

28 Basic scenario Query rules insert into mysql_query_rules (username,destination_hostgroup,active,retries,match_digest,apply) values( app_rw',10,1,3,'^select.*for UPDATE',1); insert into mysql_query_rules (username,destination_hostgroup,active,retries,match_digest,apply) values( app_rw',11,1,3,'^select ',1); insert into mysql_query_rules (username,destination_hostgroup,active) values( app_r',11,1); insert into mysql_query_rules (username,destination_hostgroup,active) values('etl_r',12,1); LOAD MYSQL QUERY RULES TO RUNTIME;SAVE MYSQL QUERY RULES TO DISK; 28

29 Basic scenario with ProxySQL Writes Reads Reads 29

30 ProxySQL Replication HG danger! READ_ONLY = 0 WRITE HG READ_ONLY = 1 READ HG READ_ONLY=0 READ_ONLY=0 READ_ONLY=1 READ_ONLY=1 HG 10 & HG 11 HG 10 HG & HG:11 HG 11 30

31 ProxySQL + MHA (or any controller) Writes Reads Reads 31

32 ProxySQL + MHA (or any controller) No movable part like: VIP DNS (Route 53 or similar) No Crazy scripts to manage other proxy (ie HAProxy) 32

33 What timing with MHA? MHA discovery time (ping x 3 times) -> ~9 sec MHA recovery (binlog) variable time MHA Slave shift + Master READ-ONLY < 2 sec ProxySQL will react as soon as READ_ONLY=0 33

34 What is needed with MHA? Nothing all defaults master_ip_failover_script=master_ip_failover master_ip_online_change_script=master_ip_online_change Test it: masterha_master_switch --master_state=alive --conf=/etc/mha.cnf -- orig_master_is_new_slave --interactive=0 -- running_updates_limit=0 34

35 Consistent distributed behavior for HA? Given no VIP or centralize element. 35

36 Integration with PXC\Galera 36

37 Integration with PXC\Galera 37

38 Integration with PXC\Galera 38

39 Integration with PXC\Galera read_only wsrep_status wsrep_rejectqueries wsrep_donorrejectqueries wsrep_connected wsrep_desinccount wsrep_ready wsrep_provider wsrep_segment Number of nodes in by segment Retry loop Online Offline_soft Offline_hard Shunned But we also have Hostgroups Replication HG 39

40 Integration with PXC\Galera 40

41 Integration with PXC\Galera HG: 10 HG: 10 HG: 11 HG: 12 HG: 11 HG: 12 41

42 Integration with PXC\Galera Multiple writer vs Single writer Multi writer based on weight and Hostgroups Need Replication HG and READ_ONLY Failover is done without the need of performing actions on the nodes Failover is done modifying the READ_ONLY value Single WRITER node isolation not guarantee Single WRITER node isolation is guarantee 42

43 Integration with PXC\Galera - Users insert into mysql_users (username,password,active,default_hostgroup,default_schema) values ('app_rw','test',1,10,'test_schema'); insert into mysql_users (username,password,active,default_hostgroup,default_schema) values ('app_r','test',1,11,'test_schema'); insert into mysql_users (username,password,active,default_hostgroup,default_schema) values ( appdr_r','test',1,12, etl_schema'); LOAD MYSQL USERS TO RUNTIME;SAVE MYSQL USERS TO DISK; 43

44 Integration with PXC\Galera QR insert into mysql_query_rules (username,destination_hostgroup,active,retries,match_digest,apply) values( app_rw',10,1,3,'^select.*for UPDATE',1); insert into mysql_query_rules (username,destination_hostgroup,active,retries,match_digest,apply) values( app_rw',11,1,3,'^select ',1); insert into mysql_query_rules (username,destination_hostgroup,active) values( app_r',11,1); insert into mysql_query_rules (username,destination_hostgroup,active) values( appdr_r',12,1); LOAD MYSQL QUERY RULES TO RUNTIME;SAVE MYSQL QUERY RULES TO DISK; 44

45 Integration with PXC\Galera HG MW INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES (' ',10,3306,50000); ; INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES (' ',11,3306,1); ; INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES (' ',12,3306,100); ; LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK; 45

46 Integration with PXC\Galera HG SW INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES (' ',10,3306,100); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES (' ',11,3306,1); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES (' ',11,3306,100); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES (' ',11,3306,100); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES (' ',12,3306,100); ; INSERT INTO mysql_replication_hostgroups VALUES (10,11); LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK; 46

47 Integration with PXC ProxySQL is configured Now what? Example script Full guide: 47

48 Integration with PXC Set the script: INSERT INTO scheduler (id,active,interval_ms,filename,arg1) values (10,0,2000,"/var/lib/proxysql/galera_check.pl","- u=admin -p=admin -h= H=500:W,501:R - P= execution_time=1 --retry_down=2 --retry_up=1 --main_segment=1 --debug=0 -- log=/var/lib/proxysql/galeralog"); LOAD SCHEDULER TO RUNTIME;SAVE SCHEDULER TO DISK; update scheduler set active=1 where id=10; LOAD SCHEDULER TO RUNTIME; 48

49 Integration with PXC MW hostgroup_id hostname status weight comment ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

50 Integration with PXC MW hostgroup_id hostname status weight comment OFFLINE_SOFT _W_11_R_retry_up=0;10_W_11_R_retry_down=0; ONLINE ONLINE OFFLINE_SOFT ONLINE ONLINE ONLINE ONLINE

51 Integration with PXC MW Variable_name Value wsrep_sst_donor_rejects_queries ON ProxySQL Think all is ok Application writes will fail 51

52 Integration with PXC MW hostgroup_id hostname status weight comment ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

53 Integration with PXC MW WRITE Fail-over is automatically perform by ProxySQL following the weight Until an active node exists the HG 53

54 Integration with PXC SW WRITE Fail-over will not happen Until READ_ONLY variable is modified on a reader HG node. WRITER if DONOR will never be OFFLINE_SOFT 54

55 Integration with PXC\Galera SW manual 55

56 Integration with PXC\Galera SW Auto 56

57 Integration with PXC - Timing Change state to: OFFLINE_SOFT = ~2 sec HostGroup 9000 = ~2 sec ProxySQL Shunned = ~1.2 sec (5 fails in 1sec) Failover (auto) = ~3 sec 57

58 Avoid SPOF 58

59 Avoid SPOF using Keepalived 59

60 Avoid SPOF using Keepalived 60

61 Cost in time 61

62 Cost in time 62

63 So what % (36 days) MySQL Replication % (8 hours) Linux Heartbeat with DRBD % (8 hours) RHCS with Shared Storage (Active/Passive) % (52 minutes) MHA with at least 3 nodes % (52 minutes) Linux Heartbeat with DRBD and Replication % (52 minutes) Linux Heartbeat with Shared Storage and Replication % (26 minutes) Multi Master (galera - percona cluster) 3 node minimum % (5 minutes) MySQL Cluster 63

64 Cost in $$$ 64

65 Brief summary Improve service flexibility with r/w split; Allow simple way to manage access to specific servers by usage (like DW-ETL); Implement sharding; Allow Query revrite/query fix/blocking without need to modify code; Improve failover time; Reduce master downtime impact on reads; Not SPOF (as vip) 65

66 Integration with MHA 66

67 67

68 68

69 Contacts To contact Me To follow me Consulting = No mission refused! 69