Introduction. As such horrible solutions like the following are still implemented.

Transcription

1 Introduction Nowadays most companies use geographic distributed platforms to better serve ir customers. It is quite common to see companies with datacenters in North America, Europe and Asia; each site hosting databases and shared data. In some cases data is simply spread around for better accessibility; in or cases it is localized and different on each geographic location; in most cases a mix two. Useless to say that most solutions were implemented using MySQL, and asynchronous replication. MySQL Asynchronous replication had being most flexible solution, but at same time also most unreliable given poor performance, lack certification and possible data drift. In this scenario use alternative solutions, like MySQL Synchronous (galera) replication had being a serious challenge. This because nodes interactions was so intense and dense, that poor network performance between locations was preventing system to work properly; or to put it in anor way, it was possible only when exceptional network performances were present. As such horrible solutions like following are still implemented. I assume re is no need to explain how multi-circular solutions are a source trouble, and 1 / 11

2 how y seems to work, until you realize your data is screw. So question is, what is status MySQL Synchronous replication and re is any possibility to successfully implement it in place Asynchronous replication? The honest trustable answer is it depends. There are few factors that may allow or prevent usage MySQL Synchronous replication, but before describe m, let us review what had happened in development galera 3.x that had significantly changed scenario. The first obvious step is to validate network link, to do so I suggest to follow method describe in my previous article ( Effective way to check network connection ). Once you had certify that, next step is to design correctly cluster, assigning different geographic areas to logical grouping with segment feature Galera provide, to know more about segments (geographic-replication-with-mysql-and-galera ) Assuming two scenarios, one is for simple Disaster Recovery, while second for data distribution. The first one be locate on same continent, like Italy and France or Canada and USA. While second can be distributed like Italy, Canada and USA. We can come up with a schema for this two solutions that looks like: 2 / 11

3 OR 3 / 11

4 This looks easy, but I can tell you just right now that while solution distributed on 3 geographic areas is going to work, first one have issue in case crash. To understand why, and to correctly design segments you need to understand anor important concept in Galera, which is quorum calculation. Generally, you hear or read saying that Galera cluster should be deploy using an odd number nodes to correctly manage quorum calculation. This is not really true and not always needed if you understand how it works, and calculate it correctly. Galera Cluster supports a weighted quorum, where each node can be assigned a weight in 0 to 255 range, with which it participate in quorum calculations. The formula for calculation is: 4 / 11

5 In short weight sum previous view, excluding node that had left gently, divided by two must be less than sum weight current view. Wait... what is a view? A view is logical grouping nodes composing cluster. WSREP: view(view_id(prim,28b4b776,78) memb { 28b4b776,1 79cc1886, e,2 f218f33d,2 } joined {} left {} partitioned { b9aabaa5,1 <--- node is shutting down}) This is a view with ID 78 containing a group nodes that is PRIMARY Component, and having one node shutting down gently. The View information is kept inside galera and on modification node(s) presence or access it is updated, ID is incremented and new view is compared with previous following formula describe above. I think is quite clear, as such let see why I said that first solution not correctly work (by default), or more correctly not work as you may expect. 5 / 11

6 So we have 6 nodes distributed cross 2 geo site each with different segment identifier. If network between two sites have issues and cluster cannot communicate whole cluster become NON-Primary: As you can see if ONE two segment become non reachable, or not have enough quorum to remain PRIMARY given 3 is not greater n 6/2 (3). This is obviously scenario in which all weight is set as default to 1. This is also why it is recommended to use ODD nodes. 6 / 11

7 Just for clarity, see what happened if ONE node goes down and THEN network crashes. As you can see here Final view has quorum, and in that case site in segment 1 be able to stay up as PRIMARY, given 3 is greater n 5/2. 7 / 11

8 Anyhow back to our production DR site how this can be set? The first one is to decide that one two side always win, like say production: In this scenario Segment 1 always win, and to promote DR to PRIMARY you must do it manually. That work, but may be is not what we expect if we choose this solution for DR purposes. The or option is to use a trick and add a witness like arbitrator GARBD. 8 / 11

9 I don t like use GARBD, never had, in Codership documentation: If one datacenter fails or loses WAN connection, node that sees arbitrator, and by extension sees clients, continues operation. And Even though Galera Arbitrator does not store data, it must see all replication traffic. Placing Galera Arbitrator in a location with poor network connectivity to rest cluster may lead to poor cluster performance. This means that if you use GARBD you in any case have all cost traffic but not benefit a real node. If this is not clear enough I show you a simple case in which it may be more an issue n a solution. In this scenario we use GARBD and see what happen 9 / 11

10 We segments have not be able but to communicate, point is but we are may able have tothare itsee on both side, as such as witness think to be good one. In short this is call aka GARBD, each split-brain,m nightmare any DBA and SA. As geographically such simple distributed but real cluster solution and when add AT Galera acrash 7 also for DR, is to think at as atwo segment calculate only use to anor, quorum aquorum, third including properly segment and WriteSets. in work case asusing two man-in--middle segments node, temporary passing that messages allow toitifconnect. from cluster one tonot So PRIMARY cluster in case without be real able crash issue. to survive On ONE and or distribute hand segment, inleast case data. ors be one able tounable keep network going link, Conclusion Use asynchronous replication to cover geographic distribution, may still be an option when network or specific data access mode prevent it. But use MySQ/Galera may be help you a lot in keep your data consistency under control and to manage HA more efficiently. As such whatever need you may have (DR or distributed writes) use three different segments and sites, no matter if only for DR. This improve robustness your solution. 10 / 11

11 MySQL/Galera is not only a good solution to have a geographical write distributed solution, but is also a robust solution in case crash one network link. In that case cluster continue to work, but it may be in degraded state, given third segment have to forward data to or two nodes. About that I have not yet perform extensive tests, but I and post additional information. 11 / 11