Reliable Crash Detection and Failover with Orchestrator

Transcription

1 1 Reliable Crash Detection and Failover with Orchestrator Shlomi Noach, PerconaLive 2016 " How people build software

2 Agenda Orchestrator Topologies, crash scenarios Crash detection methods Promotion complexity Limbo states, split brain Flapping & acknowledgement Visibility & control Configuration vs. State based analysis & recovery State of the orchestra 2

3 Orchestrator MySQL replication topology manager github.com/outbrain/orchestrator Free & open source 3

4 4

5 Simple replication What could possibly go wrong? 5

6 Crash detection 6

7 Observe/monitor " How do you observe your database availability? 7

8 Monitor master only " Common: ping, check :3306, issue SELECT 1 8

9 Monitor master only And if response is bad? " - is this a false positive? - try again - and again? - How many times until you re sure? How much time have you lost? 9

10 Orchestrator s observation $ $ $ Continuously probes your MySQL servers - Figuring out who replicates from who - Building the topology tree - Understands replication rules - At time of crash, knows what set setup should have been $ 10

11 Observe entire topology Holistic approach, used by Orchestrator " MySQL monitoring calls for MySQL specific solution - Monitor master and replicas (issue queries) - Check replicas status - Make an analysis based on result from all servers involved. 11

12 Multi layered/multi DC replication " How do you check an intermediate master (IM) availability? 12

13 Multi layered/multi DC replication Holistic approach, used by Orchestrator " Monitoring the IM and its replicas give the bigger picture - you may actually not care about the IM s availability as long as its replicas are happy 13

14 Dead intermediate master Orchestrator s analysis IM unreachable, its replicas are reachable, and are all in agreement their master is unreachable. 14

15 Dead master Orchestrator s analysis Master unreachable, its replicas are, and are all in agreement their master is unreachable. 15

16 Dead master & some replicas Orchestrator s analysis Master unreachable, some of its replicas are, and are all in agreement their master is unreachable. Other replicas are unreachable. 16

17 Locked master Orchestrator s analysis (pending) Master is reachable, but does not execute writes. - all replicas are in agreement that master is reachable - no replica is making progress can be handled as a failed master case 17

18 Recovery & promotion constraints You ve made the decision to promote a new master Which one? Are all options valid? Is the current state what you think the current state is? 18

19 Promotion constraints most up to date less up to date You wish to promote the most up to date replica, otherwise you give up on any replica that is more advanced delayed 24 hours 19

20 Promotion constraints log_slave_updates log_slave_updates You must not promote a replica that has no binary logs, or without log_slave_updates No binary logs 20

21 Promotion constraints DC1 DC1 DC1 You prefer to promote a replica from same DC as failed master DC2 21

22 Promotion constraints SBR SBR SBR You must not promote Row Based Replication server on top of Statement Based Replication RBR 22

23 Promotion constraints Promoting 5.7 means losing 5.6 (replication not forward compatible) So Perhaps worth losing the 5.7 server?

24 Promotion constraints But if most of your servers are 5.7, and 5.7 turns to be most up to date, better promote 5.7 and drop the 5.6 Orchestrator handles this logic and prioritizes promotion candidates by overall count and state of replicas

25 Promotion constraints, real life most up to date, DC2 DC1 less up to date, DC1 Orchestrator can promote one, non-ideal replica, have the rest of the replicas converge, and then refactor again, promoting an ideal server no binary logs, DC1 25

26 Ways to avoid promotion constraints mess Make sure first replication tier is consistent, Have variety on 2nd tier

27 Ways to avoid promotion constraints mess 5.6, semi-sync Use semi-sync on designated servers. They will be most up-to-date upon failure

28 Ways to avoid promotion constraints mess %%%% %%%% %%%% 5.6 %%%% %%%% %%%% Solve the problem by aligning relay logs on the replicas upon master failure. %%%% %%%% %%%% 5.7 That s what MHA does Work In Progress: Orchestrator to support this Will require passwordless SSH 28

29 Limbos Master failed; one replica lost along. Recovery went well. What happens when master is back alive? 29

30 Limbos OHAI VIP is mine Also, good for traffic What will promoted master say? What will lost replica say? What will lost master say? Give me traffic 30

31 Solving limbos Master_host: //old.master.com Read only Orchestrator forcibly breaks replication on lost replica RESET SLAVE ALL or forced detach master on promoted replica read_only=1 on old master, if possible iptables on old master Can t find coordinates 31

32 DC split brain You re dead DC2 I can t hear you You re dead " They re " DC1 They re dead dead 32

33 33

34 Flapping & rolling failovers The master is diagnosed as being dead A new master is promoted Turns out some app client is killing it Rolling failover What does happen to a dead master that comes back alive? 34

35 Flapping & rolling failovers Orchestrator sets a minimal interval between two automated failovers First one is automated; an immediate one following gets blocked A human acknowledging the first failover implicitly resets. Good to go for next automated failover. And a human can always command a failover. 35

36 Flapping & rolling failovers Orchestrator marks a failed master as downtimed Even if said server is back in the game (human intervention), this particular server will not be failed over in the duration of the downtime. A human can terminate the downtime 36

37 Recap: how orchestrator performs master failover Detection: everyone agrees the master is dead Is this incident muted? Has this cluster just recently recovered from another failure without ack? 37

38 Recap: how orchestrator performs master failover Pick most up to date replica which will also make for least lost servers (the two are not necessarily the same) most up to date 38

39 Recap: how orchestrator performs master failover Refactor topology Oh wait, actually, now that everything s connected, is there a better server to promote? Go for it, refactor again Mark old master as downtimed Detach promoted master from old master 39

40 Recap: how orchestrator performs master failover Invoke external hooks Orchestrator does not use nor imply a specific service discovery technique Your own app/scripts to change VIP/ CNAME/Zk entries/proxy/whatever 40

41 Visibility & control Flapping and rolling failovers are avoided by having memory of past/recent events Orchestrator audits: Detection Recoveries Refactoring operations (alas without context) Owners, reasons, internal operations To audit table; to orchestrator log; to syslog Audit log available via API 41

42 Visibility & control Control via: Web interface Web API Command line interface Hubot.orc sup > No incidents which require a failover to report..orc recover failed.server.com.orc ack failed-cluster.orc relocate this.replica below that.one.orc graceful-takeover my-cluster 42

43 Configuration vs. State based recoveries In configuration based recoveries: You designate specific roles to specific servers i.e. this server will have to be promoted or these are the relevant servers, these are not $ You must then match your operations to those dictated rules. Any change you make (provision, deprovision, relocate, ) must be reflected in configuration Implies chef/puppet deploy; reload of services 43

44 Configuration vs. State based recoveries In state based recoveries: You trust the tooling to make the best of a situation % Basically do whatever a human would do You still want to have roles for your servers chef/puppet may still be involved But those can be added/removed dynamically, and the tooling adapts to change of state 44

45 Orchestrator s detection reliability There is no n-nines number Orchestrator has proven to be very accurate, in production environments Depending on both orchestrator & MySQL configuration, detection may take ~5-10 seconds 45

46 Orchestrator HA & HTTP PROXY LAYER Orchestrator services " " " " Leader MYSQL PROXY LAYER Orchestrator is highly available Supports multiple services competing for leadership Requires highly available backend database. Supports master-master setup, and guarantees it to be collision free Backend DB 46

47 Recent developments Binary log indexing: makes for Pseudo-GTID matching within 1s-2s. Reduced recovery time Planned master takeover, forced master takeover Smarter promotion rules Fuzzy names (it s the simple stuff that makes life happier) SSL (Square contributions) Better master-master support Replication structure analysis MIT license 47

48 What s on the roadmap? Ongoing, intended Relay log alignment Semi-sync (currently via contributions) Likely Failure detection consensus / leadership handover Maybe orchestrator-agent xtrabackup Always Reliability, performance, simplification 48

49 What s on the roadmap? GitHub commitment to Orchestrator We use it, we will make it better Currently merging changes upstream GitHub will become upstream Better documentation, tutorials, sample public AMI World domination Open and grateful for Contributions Please discuss via Issues beforehand 49

50 Orchestrator/related talks Choosing a MySQL HA solution today Michael Patrick (Percona) Tuesday 19, 5:15pm Orchestrator at Square John Cesario, Grier Johnson, Brian Ip (Square) Thursday 21, 3:00pm 50

51 GitHub talks Tutorial: MySQL GTID Implementation, Maintenance, and Best Practices Gillian Gunson (GitHub), Brian Cain (Dropbox), Mark Filipi (SurveyMonkey), Monday 18, 9:30am Growing MySQL at GitHub Tom Krouper, Jonah Berquist Wednesday 20, 1:00pm Rookie DBA Mistakes: How I Screwed Up So You Don't Have To Gillian Gunson Thursday 21, 12:50pm Co-speaking: Dirty Little Secrets Jonah Berquist, Shlomi Noach Thursday 21, 3:00pm 51

52 Thank you Questions? github.com/shlomi-noach " How people build software