Percona XtraDB Cluster

Transcription

1 Percona XtraDB Cluster Field Guide Issue No. 1 By Jay Janssen, Jervin Real and Peter Zaitsev Copyright Percona LLC

2 Percona XtraDB Cluster: Field Guide Issue No. 1 This is the first is a series of short Percona ebooks containing useful tips, examples and best practices for enterprise users of Percona XtraDB Cluster. If there is a topic you would like us to include in our next Issue, please let us know at info@percona.com or give is a call at Table of Contents Chapter 1: Useful MySQL 5.6 features you get for free in PXC 5.6 Chapter 2: Follow these basics when migrating to Percona XtraDB Cluster Chapter 3: Setting up a simple cluster Chapter 4: Galera Flow Control in PXC Chapter 5: Investigating MySQL Replication Latency Copyright Percona LLC

3 Percona XtraDB Cluster: Chapter 1 Useful MySQL 5.6 features you get for free in PXC 5.6 Useful MySQL 5.6 features you get for free in PXC 5.6 I get a lot of questions about Percona XtraDB Cluster 5.6 (PXC 5.6), specifically about whether such and such MySQL 5.6 Community Edition feature is in Percona XtraDB Cluster 5.6. The short answer is: yes, all features in community MySQL 5.6 are in Percona Server 5.6 and, in turn, are in PXC 5.6. Whether or not the new feature is useful in 5.6 really depends on how useful it is in general with Galera. I thought it would be useful to highlight a few features and try to show them working: Innodb Fulltext Indexes By Jay Janssen Yes, FTS works in Innodb in MySQL 5.6, so why wouldn t it work in Percona XtraDB Cluster 5.6? To test this I used the Sakila database, which contains a single table with FULLTEXT. In the sakila-schema.sql file, it is still designated a MyISAM table: I edited that file to change MyISAM to Innodb, loaded the schema and data into my 3 node cluster: and it works seamlessly: 3

4 Percona XtraDB Cluster: Chapter 1 Useful MySQL 5.6 features you get for free in PXC 5.6 Sure enough, I can run this query on any node and it works fine: There might be a few caveats and differences from how FTS works in Innodb vs MyISAM, but it is there. Minimal replication images Galera relies heavily on RBR events, but until 5.6 those were entire row copies, even if you only changed a single column in the table. In 5.6 you can change this to send only the updated data using the variable binlog_row_image=minimal. Using a simple sysbench update test for 1 minute, I can determine the baseline size of the replicated data: This results in 62.3 MB of data replicated in this test. 4

5 Percona XtraDB Cluster: Chapter 1 Useful MySQL 5.6 features you get for free in PXC 5.6 If I set binlog_row_image=minimal on all nodes and do a rolling restart, I can see how this changes: This yields a mere 13.4MB, that s 80% smaller, quite a savings! This benefit, of course, fully depends on the types of workloads you are doing. Durable Memcache Cluster It turns out this feature does not work properly with Galera, see below for an explanation: 5.6 introduces an Memcached interface for Innodb. This means any standard memcache client can talk to our PXC nodes with the memcache protocol and the data is: To set this up, we need to simply load the innodb_memcache schema from the example and restart the daemon to get a listening memcached port: 5

6 Percona XtraDB Cluster: Chapter 1 Useful MySQL 5.6 features you get for free in PXC 5.6 This all appears to work and I can fetch the sample AA row from all the nodes with the memcached interface: However, if I try to update a row, it does not seem to replicate (even if I set innodb_api_enable_binlog): So unfortunately the memcached plugin must use some backdoor to Innodb that Galera is unaware of. I ve filed a bug on the issue, but it s not clear if there will be an easy solution or if a whole lot of code will be necessary to make this work properly. In the short-term, however, you can at least read data from all nodes with the memcached plugin as long as data is only written using the standard SQL interface. 6

7 Percona XtraDB Cluster: Chapter 1 Useful MySQL 5.6 features you get for free in PXC 5.6 Async replication GTID Integration Async GTIDs were introduced in 5.6 in order to make CHANGE MASTER easier. You have always been able to use async replication from any cluster node, but now with this new GTID support, it is much easier to failover to another node in the cluster as a new master. If we take one node out of our cluster to be a slave and enable GTID binary logging on the other two by adding these settings: If I generate some writes on the cluster, I can see GTIDs are working: Notice that we re at GTID 1505 on both nodes, even though the binary log position happens to be different. I set up my slave to replicate from node1 (.70.2): 7

8 Percona XtraDB Cluster: Chapter 1 Useful MySQL 5.6 features you get for free in PXC 5.6 And it s all caught up. If put some load on the cluster, I can easily change to node2 as my master without needing to stop writes: So this seems to work pretty well. It does turns out there is a bit of a bug, but it s actually with Xtrabackup currently the binary logs are not copied in Xtrabackup SST and this can cause GTID inconsistencies within nodes in the cluster. I would expect this to get fixed relatively quickly. Conclusion MySQL 5.6 introduces a lot of new interesting features that are even more compelling in the PXC/Galera world. If you want to experiment for yourself, I pushed the Vagrant environment I used to Github at: 8

9 Percona XtraDB Cluster: Chapter 2 Follow these basics when migrating to Percona XtraDB Cluster Follow these basics when migrating to Percona XtraDB Cluster log_slave_updates is REQUIRED You need to have log_slave_updates enabled on the cluster node acting as async slave for replicated events from the async master to be applied to the other nodes, that is if you have more than one PXC node. This is because before Galera can create writesets for the replicated events, binlog events must be generated for the transactions first. Under normal async replication, an event will not be written to the slave s binary unless log_slave_updates is enabled, this is similar to Percona XtraDB Cluster in that if you want an async event replicated to the whole cluster you have to have the same enabled.. MyISAM in PXC May Lead to Inconsistencies and May Not Even Work! MyISAM tables are supported within Percona XtraDB Cluster, however, MyISAM has only basic support, primarily because the storage engine is non-transactional and so PXC cannot guarantee that the data will remain consistent within the cluster. Also, at the time of this writing, from async stream, MyISAM is not being replicated at all which I reported on this bug. This would be a showstopper for anyone who wants to, but still have MyISAM tables. You can still try by filtering MyISAM tables though if you can leave them behind. Lastly, once that bug above is fixed, and you still have MyISAM tables you wish to keep running under PXC, wsrep_replicate_myisam allows you to do so. However, if you can, you should consider moving to InnoDB altogether. There are very few reasons to stay with MyISAM nowadays i.e. if you have FULLTEXT you simply cannot replace in short term. Control Your Auto-Incrementing Columns By Jervin Real Percona XtraDB Cluster (PXC) is hot right now and some users jump right in without enough testing. Consequently, they re more likely to either suffer failure or issues that prevent them from moving forward. If you are thinking of migrating your workload to Percona XtraDB Cluster, make sure to go through these basics. PXC/Galera controls auto-incrementing values internally within the cluster, this is to avoid collisions when INSERTs are happening on not only a single node. However, this may work differently when replicating from an async master, for example like the one described on these two bugs. Galera use writesets to replicate cluster events to the other nodes, in essence these are RBR events, plus a few additional structures used for certification. Having said that, it would be good if your async master can use ROW based binlog format as well to achieve better consistency, if you have an async master <= 5.0 though, you can workaround this by turning off wsrep_auto_increment_control from the Percona XtraDB Cluster nodes as workaround. Note that with the latter, make sure to not forget turning the feature back on when you switch to the new cluster especially if you are planning to write on multiple nodes. 9

10 Percona XtraDB Cluster: Chapter 2 Follow these basics when migrating to Percona XtraDB Cluster Have PRIMARY KEYS If you still have tables without PRIMARY KEYs, then its time to make one for them. Galera does not work well with those and even if there is basic support when wsrep_certify_nonpk is enabled, you can still hit issues like when automatic creation of primary keys for use during certification becomes non-deterministic. Although the previous bug has been fixed on latest release ( ), table without PK imposes an additional overhead, and because cluster performance is somewhat dependent on the slowest node this overhead can easily become visible on the whole cluster and your async replication being affected. Be Prepared for some Latency PXC can take workloads, however not just any workload it shines with small transactions but not with big ones. If you are consistently running overnight reporting jobs and getting them through the replication stream expect some replication lag. This is because because synchronous replication inside PXC has an additional overhead, this means the SQL_THREAD will not be able to execute events as fast, on top of that, other factors affecting async replication like if your workload is CPU or IO bound. Peter wrote some good details about it here. 10

11 Percona XtraDB Cluster: Chapter 3 Setting up a simple cluster Setting up a simple cluster By Jay Janssen Percona XtraDB Cluster (PXC) is different enough from async replication that it can be a bit of a puzzle how to do things the Galera way. This post will attempt to illustrate the basics of setting up 2 node Percona XtraDB Cluster 5.6 cluster from scratch. Requirements Two servers (could be VMs) that can talk to each other. I m using CentOS for this post. Here s a dirtsimple Vagrant setup: to make this easy (on Virtualbox). These servers are talking over the /24 internal network for our example. 11

12 Percona XtraDB Cluster: Chapter 3 Setting up a simple cluster Install the software These steps should be repeated on both nodes: 12

13 Percona XtraDB Cluster: Chapter 3 Setting up a simple cluster 13

14 Percona XtraDB Cluster: Chapter 3 Setting up a simple cluster 14

15 Percona XtraDB Cluster: Chapter 3 Setting up a simple cluster 15

16 Percona XtraDB Cluster: Chapter 3 Setting up a simple cluster Disable IPtables and SElinux It is possible to run Percona XtraDB Cluster 5.6 with these enabled, but for simplicity here we just disable them (on both nodes!): Configure the cluster nodes Create a my.cnf file on each node and put this into it: Note that the wsrep_node_address should be the proper address on each node. We only need this because in this environment we are not using the default NIC. Bootstrap node1 Bootstrapping is simply starting up the first node in the cluster. Any data on this node is taken as the source of truth for the other nodes. 16

17 Percona XtraDB Cluster: Chapter 3 Setting up a simple cluster We can see the cluster is Primary, the size is 1, and our local state is Synced. This is a one node cluster! 17

18 Percona XtraDB Cluster: Chapter 3 Setting up a simple cluster Prep for SST SST is how new nodes (post-bootstrap) get a copy of data when joining the cluster. It is in essence (and reality) a full backup. We specified Xtrabackup as our backup and a username/password (sst:secret). We need to setup a GRANT on node1 so we can run Percona XtraBackup against it to SST node2: This GRANT should not be necessary to re-issue more than once if you are adding more nodes to the cluster. Start node2 Assuming you ve installed the software and my.cnf on node2, then it should be ready to start up: If we check the status of the cluster again: (see next page) 18

19 Percona XtraDB Cluster: Chapter 3 Setting up a simple cluster We can see that there are now 2 nodes in the cluster! 19

20 Percona XtraDB Cluster: Chapter 3 Setting up a simple cluster The network connection is established over the default Galera port of 4567: Summary In these steps we: Installed Percona XtraDB Cluster 5.6 server package and dependencies Did the bare-minimum configuration to get it started Bootstrapped the first node Prepared for SST Started the second node (SST was copied by netcat over port 4444) Confirmed both nodes were in the cluster The setup can certainly be more involved in this, but this gives a simple illustration at what it takes to get things rolling. 20

21 Percona XtraDB Cluster: Chapter 4 Galera Flow Control in PXC Galera Flow Control in Percona XtraDB Cluster By Jay Janssen What is flow control? One of the things that people don t often expect when switching to Galera is existence of a replication feedback mechanism, unlike anything you find in standard async MySQL replication. It is my belief that the lack of understanding of this system, or even that it exists, leads to unnecessary frustration with Galera and cluster stalls that are preventable. This feedback, called flow control, allows any node in the cluster to instruct the group when it needs replication to pause and when it is ready for replication to continue. This prevents any node in the synchronous replication group from getting too far behind the others in applying replication. This may sound counter-intuitive at first: how would synchronous replication get behind? As I ve mentioned before, Galera s replication is synchronous to the point of ensuring transactions are copied to all nodes and global ordering is established, but apply and commit is asynchronous on all but the node the transaction is run on. It s important to realize that Galera prevents conflicts to such transactions that have been certified but not yet applied, so multi-node writing will not lead to inconsistencies, but that is beyond the scope of this ebook. Tuning flow control Flow control is triggered when a Synced node exceeds a specific threshold relative to the size of the receive queue (visible via the wsrep_local_recv_queue global status variable). Donor/Desynced nodes do not apply flow control, though they may enter states where the recv_queue grows substantially. Therefore care should be taken for applications to avoid using Donor/Desynced nodes, particularly when using a blocking SST method like rsync or mysqldump. gcs.fc_limit This setting controls when flow control engages. Simply speaking, if the wsrep_local_recv_queue exceeds this size on a given node, a pausing flow control message will be sent. However, it s a bit trickier than that, because of fc_master_slave (see next page). The fc_limit defaults to 16 transactions. This effectively means that this is as far as a given node can be behind committing transactions from the cluster. 21

22 Percona XtraDB Cluster: Chapter 4 Galera Flow Control in PXC gcs.fc_master_slave The fc_limit is modified dynamically if you have fc_master_slave disabled (which it is by default). This mode actually adjusts the fc_limit dynamically based on the number of nodes in the cluster. The more nodes in the cluster, the larger the calculated fc_limit becomes. The theory behind this is that the larger the cluster gets (and presumably busier with more writes coming from more nodes), the more leeway each node will get to be a bit further behind applying. If you only write to a single node in Percona XtraDB Cluster 5.6, then it is recommended you disable this feature by setting fc_master_slave=yes. Despite its name, this setting really does no more than to change if the fc_limit is dynamically resized or not. It contains no other magic that helps single node writing in PXC to perform better. gcs.fc_factor If fc_limit controls when flow control is enabled, then fc_factor addresses when it is released. The factor is a number between 0.0 and 1.0, which is multiplied by the current fc_limit (adjusted by the above calculation if fc_master_slave=no). This yields the number of transactions the recv queue must fall BELOW before another flow control message is sent by the node giving the cluster permission to continue replication. This setting traditionally defaulted to 0.5, meaning the queue had to fall below 50% of the fc_limit before replication was resumed. A large fc_limit in this case might mean a long wait before flow control gets relaxed again. However, this was recently modified to a default of 1.0 to allow replication to resume as soon as possible. An example configuration tuning flow control in a master/slave cluster might be: Working with flow control What happens during flow control Simply speaking: flow control makes replication stop, and therefore makes writes (which are synchronous) stop, on all nodes until flow control is relaxed. In normal operation we would expect that a large receive queue might be the result of some brief performance issue on a given node, or perhaps the effect of some large transaction briefly stalling an applier thread. However, it is possible to halt queue applying on any node by simply by running FLUSH TABLES WITH READ LOCK, or perhaps by LOCK TABLE, in which case flow control will kick in just as soon as the fc_limit is exceeded. Therefore, care must be taken that your application or some other maintenance operation (like a backup) doesn t inadvertently cause flow control on your cluster. 22

23 Percona XtraDB Cluster: Chapter 4 Galera Flow Control in PXC The cost of increasing the fc_limit Keeping the fc_limit small has two three purposes: 1. It limits the amount of delay any node in the cluster might have applying cluster transactions. Therefore, it keeps reads more up to date without needing to use wsrep_causal_reads. 2. It minimizes the expense of certification by keeping the window between new transactions being committed and the oldest unapplied transaction small. The larger the queue is, the more costly certification gets. EDIT: actually the cost of certification depends only the size of the transactions, which translates into number of unique key lookups into the certification index, which is a hash table. A small fc_limit does however keep the certification index smaller in memory. 3. It keeps the certification interval small, which minimizes replication conflicts on a cluster where writes happen on all nodes. On a master/slave cluster, therefore, it s reasonable to increase the fc_limit because the only lagging nodes will be the slaves with no writes coming from them. However, with multi-node writing, larger queues will make certification more expensive replication conflicts more likely and therefore time-consuming to the application. How to tell if flow control is happening and where it is coming from There are two global status variables you can check to see what flow control is happening: wsrep_flow_control_paused the fraction of time (out of 1.0) since the last SHOW GLOBAL STATUS that flow control is effect, regardless of which node caused it. Generally speaking, anything above 0.0 is to be avoided. wsrep_flow_control_sent the number of flow control messages sent by the local node to the cluster. This can be used to discover which node is causing flow control. I would strongly recommend monitoring and graphing wsrep_flow_control_sent so you can tell if and when flow control is happening and what node (or nodes) are causing it. Using myq_gadgets, I can easily see flow control if I execute a FLUSH TABLES WITH READ LOCK on node3: (See next page) 23

24 Percona XtraDB Cluster: Chapter 4 Galera Flow Control in PXC Notice node3's queue fills up, it sends 1 flow control message (to pause) and then Flow control is in a pause state 100% of the time. We can tell flow control came from this node because Flow snt shows a message sent as soon as flow control is engaged. Flow control and State transfer donation Donor nodes should not cause flow control because they are moved from the Synced to the Donor/Desynced state. Donors in that state will continue to apply replication as they are permitted, but will build up a large replication queue without flow control if they are blocked by the underlying SST method, i.e., by FLUSH TABLES WITH READ LOCK. 24

25 Percona XtraDB Cluster: Chapter 5 Investigating MySQL Replication Latency Investigating MySQL Replication Latency By Peter Zaitsev I was curious to check how Percona XtraDB Cluster behaves when it comes to MySQL replication latency or better yet, call it data propagation latency. It was interesting to see whenever I can get stale data reads from other cluster nodes after write performed to some specific node. To test it I wrote quite a simple script (you can find it in the end of the post) which connects to one node in the cluster, performs an update and then immediately does the read from second node. If the data has been already propagated good, if not we ll continue to retry reads until it finally propagates, and then measure the latency. This is used to see whenever application can see any stale reads. My setup is 3 Percona XtraDB Cluster nodes talking through dedicated 1Gbit cluster network (DPE1, DPE2, DPE3) and I m running the test from 4th server (SMT2) so it is pretty realistic setup from typical data center latency point of view though the server hardware is not the most recent. First lets look at baseline when cluster has no load but running the script doing writes to DPE1 and immediately reading from DPE2... These results tell me 2 things. First Replication by default in Percona XtraDB Cluster is Asynchronous from Data Propagation Standpoint it takes time (though short one in this case) for changes committed on the one node to become visible to the other. Second it is actually doing quite well with less than 1% of tests able to see any inconsistency and the delay being less than 1ms in average with rather stable results. But we do not setup clusters to be idle right? So lets to another test, now running the Sysbench load on DPE1. With concurrency of 32 this corresponds to pretty significant load. Results become as follows: As expected we can observe inconsistency much more frequently in almost 40% though the average observed delay remains just few milliseconds, which is something most applications would not even notice. Now if we run sysbench on DPE2 (the load on the node which we re reading from): 25

26 Percona XtraDB Cluster: Chapter 5 Investigating MySQL Replication Latency We can observe the effect in similar amount of cases but delay is higher in this case both in average and the Maximum one. This tells me from data propagation standpoint the cluster is more sensitive to the load on the nodes which receive the data, not the ones where writes are done. Lets remember though what Sysbench OLTP has only rather small portion of writes. What if we look at workloads which consists 100% of Writes. We can do it with Sysbench, for example. Running this load on DPE1 I m getting: Surprise! results are actually better than if we put mixed load as we can observe any delay only in about 11%. However if we run the same side load on DPE2 we get: Which is the worse result out there with over 50% samples produced inconsistent data and average delay for those over 5ms and outliers going to half a second. From these results I read the side load on the node TO which updates are being propagated causes largest delay. At this point I remembered there is one more test I can run. What is if I put side load on DPE3 server, from which I m not touching from the test at all? No surprise here as DPE3 is not being directly read or written to the load on it should cause minimal delays to data propagation from DPE1 to DPE2. The propagation latency we ve observed in the test so far is quite good but it is not synchronous replication behavior we still can t treat the cluster as if it were single server from generic application. Right. Default configuration for Percona XtraDB Cluster at this point is to replicate data asynchronously, but still guaranty there is no conflicts and data inconsistency then updates are done on multiple nodes. There is an option you can enable to get fully synchronous replication behavior: 26

27 Percona XtraDB Cluster: Chapter 5 Investigating MySQL Replication Latency When this option is enabled the cluster will wait for the data to be actually replicated (committed) before serving the read. The great thing is wsrep_causal_reads is session variables so you can mix different applications on the same cluster some requiring better data consistency guarantees, other being OK with a little bit stale data but looking for the best performance possible. So far so good. We can make cluster to handle significant load with small transactions and still have very respectful data propagation delay or we can enable wsrep_causal_reads=1 option and get full data consistency. But what happens if we have some larger transactions? To test this I have created the copy of sbtest table and will run a long update while running my test to see how the latency is impacted: Running this query on the DPE1 box I m getting following result: So the propagation delay was pretty good until this given query had to be replicated, in which case we could observe the replication delay for over 45 seconds which is quite nasty. Note though delay was for less period than it takes to execute the query on the master. This is because application of the changes on the master in parallel and updates to the sbtest table and sbtest2 table can be done in parallel (even changes to the same table can) but the certification process is serial as well as sending write set to the other nodes, and it must be taking some 45 seconds to send the write set and perform certification. If we run the same query on DPE2 the interesting thing happens. The script does not show any data propagation delays but it visibly stalls, as I guess because the UPDATE statement issued to DPE1 is blocked for some time. To check this idea I decided to use the sysbench script with very simple point update queries to see if we get any significant stalls. We can see quite respectful performance with longest request taking some 300ms so no stalls. Lets do the run again now running the same update statement on the different cluster node: 27

28 Percona XtraDB Cluster: Chapter 5 Investigating MySQL Replication Latency As we see there is a stall in update for 50+ second, again while certification is happening. So certification does not only delay data propagation but can stall updates done to the different tables on the different nodes. Summary: Percona XtraDB Cluster performs very well when it comes to small transactions offering very small propagation delay and an option of synchronous replication all together. However when it comes to large transactions you can get in a lot of trouble with major stalls both in terms of data propagation and in terms of writes. The system I did test on is pretty old and I would expect modern systems can run certification several times faster still taking tens of seconds for what I would consider medium size transaction modifying 1 million of rows is rather long time. So make sure to have a good understanding how large transactions your application has and how longs stalls it can handle. 28

29 About the authors Jay Janssen: Percona principal consultant Jay joined Percona in 2011 after 7 years at Yahoo working in a variety of fields including High Availability architectures, MySQL training, tool building, global server load balancing, multi-datacenter environments, operationalization, and monitoring. He holds a B.S. of Computer Science from Rochester Institute of Technology. Jervin Real: Percona support engineer When you come to Percona for consulting and support, chances are he'll be greeting you first. His primary role is to make sure customer issues are handled efficiently and professionally. Jervin joined Percona in May Peter Zaitsev: Percona founder and CEO Peter managed the High Performance Group within MySQL until 2006, when he founded Percona. Peter has a Master's Degree in Computer Science and is an expert in database kernels, computer hardware, and application scaling. Copyright Percona LLC

30 Powered by TCPDF ( About Percona Percona has made MySQL faster and more reliable for over 2,000 consulting and support customers worldwide since Percona provides enterprise-grade MySQL support, Consulting, Training, Remote DBA, and Server Development services to companies such as Cisco Systems, Alcatel-Lucent, Groupon, and the BBC. Percona's founders authored the definitive book High Performance MySQL from O'Reilly Press and the widely read MySQL Performance Blog. Percona also develops software for MySQL users, including Percona Server, Percona XtraBackup, Percona XtraDB Cluster, and Percona Toolkit. The popular Percona Live conferences draw attendees and acclaimed speakers from around the world. For more information, visit Copyright Percona LLC