Choosing a MySQL HA Solution Today Choosing the best solution among a myriad of options
Questions...Questions...Questions??? How to zero in on the right solution
You can t hit a target if you don t have one! Many in upper management don t really know what they want and then just end up saying they want it all! Your job is to determine what is most essential component for success. 4 No solution is a One-Size- Fits-All; rather, the best you can hope for is One-Size- Fits-Most.
What is the problem you are trying to solve? A very important question!
Redundancy vs. Scaling vs. High Availability These are not necessarily all the same! Redundancy Scaling Need multiple copies of data in event of a disaster Need to increase read and/or write throughput 6 High Availability Need to minimize outage duration
CAP Theorem Choose any two of the following: Consistency All nodes see the same data at the same time Availability Every request receives a response about whether it succeeded or not CAP Theorem Consistency Availability Partition Tolerance 7 Partition Tolerance The system continues to operate despite arbitrary partitioning due to network failures
Guaranteeing Consistency The problem is that although MySQL replication is great, it alone does not guarantee consistency across all nodes. There is always the potential that data is out of sync since transactions can be lost during failover and other reasons. 8 Galera-based clusters such as PXC are certification-based to prevent this!
Can I afford to lose data? Depends on the Application Apps should check status codes on transactions to be sure they were committed. Many do not! Lost transactions During failover, simple replication schemes have the possibility of losing data 9 Inconsistent nodes Without conflict detection and resolution, it is unavoidable Run pt-table-checksum often to check for inconsistent data across replication nodes Use a Galera-based Distributed Cluster, such as PXC with certification processes
Avoiding Single Point of Failure What is watching your system? Or is anything standing ready to intervene in a failure? For replication, take a look at MHA and MySQL Orchestrator. Both are great tools to perform failover of a Replica. There are others. 10 For PXC, failover is typically much faster, but it is not the perfect solution in every case.
Can I afford lost transactions? Many MySQL DBAs worry about setting innodb_flush_log_at_trx_commit to 1 for ACID compliance and sync_binlog but then use replication with no consistency checks! Is this logically consistent? PXC maintains consistency through certification 11
Conflict Detection & Resolution 12 Galera s Certification Process Transaction continues on a node as normal until it reaches COMMIT stage Changes are collected into a writeset Writeset is sent to all nodes for certification PKs are used to determine if the writeset can be applied If certification fails, the writeset is dropped and the transaction is rolled back. If it succeeds, the transaction commits and the writesets are applied to all of the nodes. All nodes will reach the same decision on every transaction and is thus deterministic.
Do I want Failover or a Distributed System? Failover Pitfalls Failover systems have a monitor which detects failed nodes and moves services elsewhere if available Failover takes time! Distributed Systems to the Rescue Distributed Systems minimize failover time 13
Automatic or Manual Failover? Advantage of Manual Failover The primary advantage to failing over manually is that a human usually can make a better decision as to whether failover is necessary. Systems rarely get it perfect, but they can be close! Advantage of Automatic Failover More Nines due to minimized outages No need to wait on a DBA to perform 14
How Fast Does Failover Have to Occur? Replication / MHA / MMM Depends upon how long it takes for pending Replica transactions to complete before failover can occur Typically around 30 seconds DRBD Typically between 15 and 30 seconds PXC / MySQL Cluster VERY fast failover. Typically less than 1 second depending upon Load Balancer 15
How Many 9 s Do You Really Need? Every manager always says As many as I can get. That sounds great, but the reality is that tradeoffs are required! Many applications can tolerate a few minutes of downtime with minimal impact. 16
17
Do I need to scale reads and/or writes? Scaling Reads Most solutions offer ability to read from multiple nodes or replicas MHA, PXC, MySQL Cluster, and others are well suited for this Scaling Writes Many people wrongly try to scale writes by writing to multiple nodes in PXC leading to conflicts Others try it with Master-Master Replication which is also problematic Possibly the best solution in this regard is MySQL Cluster 18
What about provisioning new nodes? Replication Largely, this is a manual process MySQL Utilities makes this easier than ever Distributed Clusters PXC and MySQL Cluster make this much easier PXC uses state transfer (either SST or IST) to automate the process for cluster nodes 19
The Rule of Threes With PXC, try to have three of everything If you span a data center, have 3 data centers If your nodes are on a switch, try to have 3 switches PXC really needs at least three nodes in the cluster. An odd number is preferred for voting reasons. Forget about trying to keep a cluster alive during failure with only two data centers. You are better off making one a DR site. Forget about custom weighting to try to get by on two data centers. The 51% rule will get you anyway! 20
How many data centers do I have? What if I only have 1 data center? You can gain protection against a single failed node or more, depending on cluster size What if I have 2 data centers? You should probably be considering the second data center as a DR solution What about 3 or more? Most robust solution when using Galerabased clusters such as PXC 21
How do I plan for Disaster Recovery? Make sure the DR node(s) can handle the traffic, if even at minimized performance level Replicating from a PXC Cluster to a DR site Asyncronous Replication from PXC to a single node Asyncronous Replication from PXC to a replication topology Asyncronous Replication from PXC to another PXC cluster 22
What storage engine(s) do I need? MHA Not storage engine dependent. Works with all storage engines PXC Requires InnoDB. Support for MyISAM is experimental and should not be used in Production MySQL Cluster Requires NDB Storage Engine Build Optimize Fix Manage 23
Load Balancer Options HAProxy Open-source software solution Cannot split reads and writes. If that is a requirement, the app will need to do it! F5 BigIP Typical hardware solution MaxScale Can do read/write splitting Elastic Load Balancer (ELB) Amazon solution 24
What happens if the cluster reboots? A power outage in a single data center could lead to issues PXC can be configured to auto bootstrap May not always work when all nodes lose power simultaneously. While server is running, the grastate.dat file shows -1 for seqno Surviving a Reboot Helpful if nodes are shutdown by a System Administrator for a reboot or other such process Normal shutdown sets seqno properly 25
Do I need to be able to read after writing? Asynchronous Replication does not guarantee consistent views of data across nodes PXC offers Causal Reads Replica will wait for the event to be applied before processing additional queries, guaranteeing a consistent read state across nodes. 26
What if I do a lot of data loading? In the recent past, it was conventional wisdom to use replication in such scenarios over PXC. MTS does help if data is distributed over multiple schemas but is not a fit for all situations. PXC is now a viable option since we discovered a bug in Galera which did not properly split large transactions. 27
Have I taken precautions against split brain? Split Brain occurs when a cluster has its nodes divided from one another, most often due to network blip, and nodes form two or more new and independent (and thus divergent) clusters PXC is configured to go into a nonprimary state and refuse to take traffic 28 A newer setting with PXC will allow for dirty reads for non-primary nodes
Does my app require high concurrency? Newer approaches to replication allow for parallel threads (PXC has had this from the beginning.), such as Multi- Thread Slaves (MTS) MTS Allows a replica to have multiple SQL threads all with their own relay logs Enable GTID to make backups via Percona XTRABackup safer due to not being able to trust SHOW SLAVE STATUS to get relay log position 29
Am I limited on RAM? Some Distributed solutions such as MySQL Cluster require a lot of RAM, even with file-based tables. Be sure to plan appropriately. PXC works much more like a stand-alone node 30
31 How stable is my network? Networks are never really 100% reliable. Some Network Problems are due to outside factors such as system resource contention (especially on virtual machines) Network problems cause inappropriate failover issues. Use LAN segments with PXC to minimize network traffic across WAN
Making the right choice depends upon... Knowing what you really need! Knowing your options. Knowing your constraints! Understanding the pros/cons of each solution Setting expectations properly! 32
Q&A Your chance to ask questions!