Choosing a MySQL HA Solution. Ernie Souhrada, Senior Consultant Webinar Presentation 05 June 2013

Transcription

1 Choosing a MySQL HA Solution Ernie Souhrada, Senior Consultant Webinar Presentation 05 June 2013

2 Agenda Know Thy Presenter Why Are We Here? High Availability? HA! So Many Options... So Little Time Escaping Choice Paralysis Square Pegs, Square Holes Square Pegs, Round Holes Pegs? What Pegs?

3 Know Thy Presenter Joined Percona in April 2012 Mathematics / Political Science academic Coming up on 20 years in IT, 15 with MySQL Skier, Psytrancer, Technological Generalist Specialization is for insects

4 Why Are We Here? The great existential question There is no perfect solution. Beware the snake-oil peddlers. It's all about what works for you. Think. Ask questions. Experiment.

5 High Availability is Easy?! Availability is usually measured in percentages that indicate the amount of downtime per year. 98% availability ~ 1 week per year 99% ~ 3.65 days 99.9% ~ 8.75 hours The holy grail % ~ 5 minutes But what does it mean for a system to be considered available? How do we check? When is an UP server not really UP?

6 HA! Think again. PING ping statistics packets transmitted, 1 received, 0% packet loss, time 37851ms rtt min/avg/max/mdev = / / /0.000 ms time HEAD head OK real 0m15.539s SELECT COUNT(*) FROM mysql.user; MySQL Server has gone away.

7 A-HA, Now IT Makes Sense Thinking too simplistically about availability can be misleading. Downtime vs. scheduled downtime System responsiveness The system is greater than the sum of its parts. Have a meaningful SLA or don't have one at all.

8 MySQL HA: So Many Options... Traditional (async) MySQL replication Master-Slave or Master-Master Manual Failover Yes, this can be an HA solution. External manager frameworks / applications An alphabet soup of options: PRM, MHA, MMM, VIPs, keepalived, Pacemaker, Heartbeat

9 So Little Time. Non-traditional MySQL replication MySQL Semi-sync replication (MySQL 5.5+) Tungsten Replicator Galera (Percona XtraDB Cluster, MariaDB/Galera) MySQL Cluster (NDB) Non-MySQL replication Shared storage DRBD, Lustre, NFS, SANs, Other Esoterica Clustrix, Xeround, etc.

10 Escaping Choice Paralysis Choosing a solution doesn't have to be painful. Avoid tunnel vision. Avoid buzzwords and the flavor of the week. What works for Google probably isn't right for you. Consider three perspectives (often related) Business Philosophical Technological

11 Satisfying the Suits Your CEOs may not know FHA from MHA, but they have a vested interest in the outcome. Cost of downtime The Lamborghini Factor It's not just financial Risk analysis Budgetary constraints Tolerance for lost transactions Maturity of the proposed solution Supportability

12 Placating the Philosophers Automated versus Manual Failover GitHub's well-publicized issues from The debate isn't likely to end anytime soon. Some people have very strong opinions here. No automated solution is guaranteed to do the right thing every time, but manual failover may involve a longer outage if a human can't be reached in a timely fashion. Need to balance the ideals with what's realistic. A 24x7 NOC is great for manual failover. Tolerance for data loss / drift Maturity of the proposed solution

13 Talking Technically MySQL Feature Set Usage Application-Related Issues Performance Scalability Failover / Recovery Time Operational Complexity CAP Theorem In-house knowledge / skills

14 Square Pegs, Square Holes In light of these perspectives, where do our previously-discussed HA options fit? Traditional MySQL Replication PRO: It's cheap and well-understood. Both automated and manual failover options are available. Read-scaling is easy, and it works with any storage engine. CON: It's MySQL replication. Data drift is obscenely easy. Active-active replication topologies are brittle, and without some serious voodoo, one slave can still only have one master.

15 A Round of Hand-Waving I started out as a physicist Master-Slave w/manual Failover Simple slave promotion. Not too many moving parts unless you have multiple slaves that need to be re-homed. Master-Master (1 active) w/manual Failover Change DNS, move a VIP, build failover awareness into your application, etc. Still reasonably simple as long as writes go to one place. Additional slaves can complicate matters; binary log coordinates differ between the master servers.

16 Hand-Waving, Part Deux M-S or M-M with automated failover Keepalived: VIP management HAProxy: L4 traffic director Pacemaker or Hearbeat: VIP management PRM (Percona Replication Manager): Automated solution for master promotion and slave re-homing. Can be used manually, too, but not really designed for such. Generally works very well, but because it's a Pacemaker resource agent, it's subject to the whims of the Pacemaker developers, which has been an issue recently with CentOS 6.4. Does not make any guarantees about node consistency after a failover, but can be paired with semi-sync replication.

17 Hand-Waving, Part The Third M-S / M-M with automated failover, continued MMM Multi-Master Replication Manager Agent-based system. Unreliable agent communication Not sure if it's even still actively being developed. Don't use. MHA Master High Availability for MySQL Tries very hard to ensure data consistency when promoting a new slave into the master role. Can be dropped into an existing MySQL topology without extensive reconfiguration. The preferred choice of Percona's Remote DBA team. MySQL Utilities New tools from Oracle designed to work with MySQL 5.6 and GTID-based replication. Have yet to see this in the wild.

18 Use Cases, Contraindications I Traditional replication is the most generic HA solution out there. If it runs fine on a single MySQL server, there's almost certainly a way to make it work reasonably well with replication. But... Pacemaker/Corosync solutions can have trouble with highlatency networks. None of the automated solutions handle extremely high load very well. MHA requires SSH connectivity; Pacemaker/Corosync solutions can lose messages and trigger spurious failovers. Under the hood, it's still replication. Periodic data consistency checks (think pt-table-checksum) should be de rigueur. Oracle's new MySQL Utilities are MySQL 5.6+ only.

19 Square Pegs, Round Holes Non-traditional MySQL replication Semi-sync replication (MySQL 5.5+) Master doesn't return success to the client until a slave has acknowledged receipt of the event (or the semi-sync timeout occurs). Sounded like a good idea in theory, but never saw wide adoption as far as I'm aware. Still requires something like PRM, MHA, or other external management framework to actually effect a failover.

20 Square Pegs, Round Holes II Tungsten Replicator (from Continuent) Comes in FOSS and commercial flavors. Java-based binary log processing and relaying framework. Complex replication topologies are possible, including multi-master and replication between different DB platforms. This is cool, if it works as advertised. Haven't seen it or worked with it in the wild, so I'm hesitant to make any judgement calls.

21 All Your Square Pegs are in RAM MySQL Cluster (NDB) Until recently, ran entirely in memory, but now disk-based tables are available (with some limitations). Handles sharding and data redistribution automatically as nodes are added. If your application is of the type that NDB was designed for (lots of small, simple writes, simple key-value lookups that don't require JOINs), it will likely outperform any other cluster/multi-machine solution in the MySQL ecosphere. But... Setup, tuning and configuration is extremely complicated; NDB-specific knowledge is required.

22 Synchronous Pegs Galera-based Solutions Percona XtraDB Cluster (PXC), MariaDB + Galera Synchronous replication Can support true multi-master writing, reads are served up locally. InnoDB-only (MyISAM support is experimental) In the CAP theorem, Galera's focus is on consistency over availability. Tremendous potential here; seems to solve a lot of the most common gripes about MySQL replication, but still in the process of maturing.

23 Use Cases, Contraindications II Semi-sync replication Can be paired with MHA or PRM. Probably won't work very well if all the slaves are in remote datacenters. Tungsten Doesn't appear to work with PXC. NDB Cluster Extremely fast for a specific class of applications (small writes, key-value lookups without JOINs) Can achieve % uptime if configured properly. Setup, tuning, and troubleshooting are complex.

24 Use Cases, Contraindications III PXC / MariaDB+Galera Good for InnoDB where all the tables have PKs. FK handling has had some issues recently. Can be used over the WAN if the application or the end users can handle the increased latency at COMMIT time, but this can also be a deal-breaker. A PXC cluster can automatically repair itself when a node drops out / returns; likewise, it can automatically expand the cluster if a new node joins.

25 Pegs? What Pegs? Replication outside of MySQL DRBD / Shared(SAN) Storage (typically used with Pacemaker or some other management framework) DRBD is mature and focused on data integrity, but there's roughly a 20% disk performance penalty right off the top. DRBD only really works with a two-server pair, and the second server sits idle. DRBD failover requires a MySQL crash recovery when it spins up on the other side. Many enterprise-level SANs have built-in facilities for snapshots and replication independent of the database.

26 There is No Peg Other Esoterica Clustrix: Completely separate database product that's MySQL-protocol compatible on the wire. Can't say anything else about it, but I suspect that it's geared for a few specific use cases. Xeround: Apparently no longer in business.

27 One Last Round of Use Cases I probably wouldn't entertain Clustrix unless it solved a specific need that I couldn't meet some other way. I just don't know enough about it. For small to medium installations where data integrity is paramount and the write load isn't likely to exceed the capacity of a single machine, DRBD is a very good choice. Bare metal servers with directly-connectable NIC ports are a must.

28 Parting Thoughts It bears repeating: there is no perfect solution, there is only what fits best for you. Plan and test thoroughly now, or cry later. If you want another nine, be prepared to add another zero. Any Questions?

29 Join us in Portland for Percona MySQL University on Monday, 17 June Registration is FREE, but space is limited. Visit for additional information.