Geographically Dispersed Percona XtraDB Cluster Deployment. Marco (the Grinch) Tusa September 2017 Dublin

Transcription

1 Geographically Dispersed Percona XtraDB Cluster Deployment Marco (the Grinch) Tusa September 2017 Dublin

2 About me Marco The Grinch Open source enthusiast Percona consulting Team Leader 2

3 Agenda What is PXC When nodes interacts Let us clarify, geo dispersed- What to keep in mind then How to measure latency correctly Use the right way (sync/async) Use help like replication manager 3

4 What is PXC/Galera? (Virtually) Synchronous Replication: True multi-master Web traffic No slave lag Balancer No master-slave failover or VIP Multi-threaded applayers Automatic node provisioning Elastic scale (in out) Geographic distributed (with segments) Mix with Async replication Galera 4

5 What PXC/Galera is NOT? Not Write-scalable solution Not great for a high amount of parallel, small requests Not great for working with Foreign Keys Not good for sharding Data (each node has the entire dataset) Web traffic Balancer Galera 5

6 What is a Node Standard MySQL Replication Galera MySQL Replication Node Master Slave 9cba28fa-a8be-11e4-8f41-9f963e1dbf4f Slave Node Node 6

7 Segments A segment is a logical grouping of nodes. Replication between Segment is optimized (writeset - some level of communication) Traffic and messaging is reduced In case of SST, the donor is chosen by proximity 7

8 More nodes more problems Use a two phase commit, or distributed locking with capacity formula: m = n x o x t (where messages/sec = number of nodes due to process o number of operation with t transaction throughput) 8

9 When nodes interacts Keepalive and checks for cluster health Writeset on writer commit Certification results Ack on local apply Flow Control IST/SST 9

10 Let us clarify, geo dispersed 1 Geo dispersed or multi-site, cluster is a cluster configuration used to help ensure high system and application availability in the event of site disaster. In this configuration, servers are separated geographically and the physical storage (quorum diskor) DATA is synchronously replicated between sites. ( 10

11 Let us clarify, geo dispersed 2 For some environments, latency is the sole focus of performance. As an example of latency, shows a network transfer, such as an HTTP GET request, with the time split into latency and data transfer components. 11

12 Geo dispersed Geo dispersed is determinate by the latency existing between nodes NOT by the geographic location itself. 12

13 How to measure latency correctly 1 wsrep_evs_repl_latency (It measures latency from the time point when a message is sent out to the time point when a message is received.) wsrep_replicated/wsrep_replicated netperf 13

14 How to measure latency correctly 2 14

15 How to measure latency correctly 2 PING 14

16 How to measure latency correctly 2 PING 14

17 How to measure latency correctly 2 Why? 14

18 Brief digression Ref: 15

19 Brief digression Ref: 15

20 Brief digression Ref: 15

21 Brief digression PING use ICMP (Internet Control Message Protocol) NOT TCP over IP Default data size is 56 bytes plus header (8 bytes) 16

22 Brief digression PING use ICMP (Internet Control Message Protocol) NOT TCP over IP Default data size is 56 bytes plus header (8 bytes) 16

23 Brief digression PING use ICMP (Internet Control Message Protocol) NOT TCP over IP Default data size is 56 bytes plus header (8 bytes) ping -M do -s c

24 Brief digression PING use ICMP (Internet Control Message Protocol) NOT TCP over IP Default data size is 56 bytes plus header (8 bytes) ping -M do -s c

25 Brief digression PING use ICMP (Internet Control Message Protocol) NOT TCP over IP Default data size is 56 bytes plus header (8 bytes) Not good enough! 16

26 Brief digression 17

27 Brief digression TCP means Transmission Control Protocol and as the name says, it is design to control the data transmission happening between source and destination. TCP implementations use the IP protocol encapsulation for the transmission of the data: 17

28 Brief digression TCP means Transmission Control Protocol and as the name says, it is design to control the data transmission happening between source and destination. TCP implementations use the IP protocol encapsulation for the transmission of the data: 17

29 Brief digression TCP means Transmission Control Protocol and as the name says, it is design to control the data transmission happening between source and destination. TCP implementations use the IP protocol encapsulation for the transmission of the data: Looks the same thing than before right? 17

30 Brief digression TCP means Transmission Control Protocol and as the name says, it is design to control the data transmission happening between source and destination. TCP implementations use the IP protocol encapsulation for the transmission of the data: Looks the same thing than before right? 17 WRONG!

31 Brief digression A TCP implementation has several characteristics that make sense to summarize: 18

32 Brief digression A TCP implementation has several characteristics that make sense to summarize: Is stream oriented 18

33 Brief digression A TCP implementation has several characteristics that make sense to summarize: Is stream oriented Establish a connection 18

34 Brief digression A TCP implementation has several characteristics that make sense to summarize: Is stream oriented Establish a connection Monitor the data transfer 18

35 Brief digression A TCP implementation has several characteristics that make sense to summarize: Is stream oriented Establish a connection Monitor the data transfer Buffered transmission 18

36 Brief digression A TCP implementation has several characteristics that make sense to summarize: Is stream oriented Establish a connection Monitor the data transfer Buffered transmission Unstructured stream 18

37 Brief digression A TCP implementation has several characteristics that make sense to summarize: Is stream oriented Establish a connection Monitor the data transfer Buffered transmission Unstructured stream Full-duplex connection 18

38 Brief digression A TCP implementation has several characteristics that make sense to summarize: Is stream oriented Establish a connection Monitor the data transfer Buffered transmission Unstructured stream Full-duplex connection Stream as a sequence of octet split in segments 18

39 Brief digression TCP dispatcher use Dynamic Slide Window 19

40 Brief digression TCP dispatcher use Dynamic Slide Window 19

41 Brief digression TCP dispatcher use Dynamic Slide Window Dispatcher manages three pointers associated to each connection: The first pointer indicate the start of the sliding window The second pointer indicates the higher octet that can be dispatchtet. The third pointer indicates the window limit 19

42 How to measure latency correctly 3 Back to us 20

43 How to measure latency correctly 3 Back to us Check for gateway MTU cut ping -M do -s c

44 How to measure latency correctly 3 Back to us Check for gateway MTU cut ping -M do -s c Consider sent and received messages (IE Wsrep replicated bytes & Wsrep received bytes) 20

45 How to measure latency correctly 3 Back to us Check for gateway MTU cut ping -M do -s c Consider sent and received messages (IE Wsrep replicated bytes & Wsrep received bytes) Check Kernel settings for: Buffering Congestion control Frame utilization 20

46 How to measure latency correctly 3 Back to us Check for gateway MTU cut ping -M do -s c Consider sent and received messages (IE Wsrep replicated bytes & Wsrep received bytes) Check Kernel settings for: Buffering Congestion control Frame utilization Test with netperf (IE) netperf -H t TCP_RR -v 2 -l b 2 -r 250K -R 1M -s 250K,10M -S 10K,256K 20

47 How to measure latency correctly 3 Back to us Check for gateway MTU cut ping -M do -s c Consider sent and received messages (IE Wsrep replicated bytes & Wsrep received bytes) Check Kernel settings for: Buffering Congestion control Frame utilization Test with netperf (IE) netperf -H t TCP_RR -v 2 -l b 2 -r 250K -R 1M -s 250K,10M -S 10K,256K Check the wsrep_evs_repl_latency value in SHOW GLOBAL STATUS like wsrep% ; 20

48 What is the right limit? Depends by the usage Balance incoming write/s consistence reads 21

49 Last chance for (virtually) Synchronous Wan settings: 22

50 Last chance for (virtually) Synchronous Wan settings: evs.inactive_check_period = PT30S; evs.inactive_timeout = PT1M; evs.suspect_timeout = PT40S; evs.stats_report_period = PT3M; evs.join_retrans_period=pt0.5s! don t use PING 22

51 Last chance for (virtually) Synchronous Wan settings: evs.inactive_check_period = PT30S; evs.inactive_timeout = PT1M; evs.suspect_timeout = PT40S; evs.stats_report_period = PT3M; evs.join_retrans_period=pt0.5s! don t use PING Master_Slave not a very good option though wsrep_provider_options = "gcs.fc_limit = 256; gcs.fc_factor = 0.99; gcs.fc_master_slave = YES" 22

52 Async replication kicks in We can use almost the same models we used 23

53 Async replication kicks in We can use almost the same models we used Challenge is to shift from one Master-Node to a new one Or from a slave to another 23

54 Async replication ways Standard binlog position using XID and wsrep_last_committed Variable_name Value wsrep_last_committed Binlog # at 544 # :26:56 server id 3306 end_log_pos 575 CRC32 0x3ae1edcd Xid = Simple to install/setup Nightmare to maintain 24

55 Async replication ways Using GTID All nodes on a cluster will have the same GTID Move from a slave from one node to another can be automated. Existing: Yves Trudeau solution : Single link DC1->DC2 Multiple Link DC1 -> DC2 ->DC3 25

56 Conclusions Plan carefully you network and DC-DC connectivity Keep the number of nodes inside a PXC cluster to minimum Test properly (not ping) the latency on the network Use PXC/Galera replication between geo-distributed only if it is safe Do not esitate to shift to Async replication Use existing solutions to help you manage async replication between PXCs 26

57 27

58 Q&A 28

59 Contacts To contact Me To follow me Consulting = No mission refused! 29