If Only I Could Find My Databases-Service Discovery with SmartStack and MySQL. Susanne Lehmann, Yelp

Size: px

Start display at page:

Download "If Only I Could Find My Databases-Service Discovery with SmartStack and MySQL. Susanne Lehmann, Yelp"

Abraham Gilbert
5 years ago
Views:

1 If Only I Could Find My Databases-Service Discovery with SmartStack and MySQL Susanne Lehmann, Yelp susanne@yelp.com

2 Me I ve been a DBA for 16 years I ve been working with all kinds of relational and NoSQL databases At Yelp, I work with 3 other DBAs to support ~400 engineers We support currently MySQL and PostgreSQL *My Slides will be online

3 Yelp s Mission: Connecting people with great local businesses.

4 I am a dba! Why should I care about Service Discovery?

5 What Is Service Discovery? When having a cluster of machines running services that need to be found and need to find each other In our case they need to find the databases

6 What Is Service Discovery? Typically, you could read locations from a config file When hardware was relatively static, this config file had to be updated only occasionally Database servers come and go more often Systems now are based on SOA/Microservices

7 What Is Service Discovery? Typically, you could read locations from a config file When hardware was relatively static, this config file had to be updated only occasionally Database servers come and go more often Systems now are based on SOA/Microservices Dynamically discover the dbs and communicate with them

8 Service Discovery at Yelp

9 DB Environment at Yelp MySQL 5.6 Many logical shards, no horizontal sharding (yet) Hundreds of MySQL db servers We now have hundreds of services

10 DB Environment at Yelp Past: DNS plus YAML files

11 Environment at Yelp Hierarchy of grouping machines together /-----topology_env-----\ machine---habitat-+-region---superregion-+-ecosystem---runtimeenv \-datacenter

12 DB Environment at Yelp One yaml file per habitat Hundreds of services on one MySQL cluster One entry per service Taking one db host out of the pool Editing 100 of entries in ~10 files

13 DB Environment at Yelp editing yaml files sucks

14 BIG Overall Goal?!?!?

15 Peak Capacity Provisioned capacity

16 Peak Capacity Oh dear

17 Matching Capacity

18 Goals Auto scaling Easier manual scaling Faster failover from one datacenter to another First stab into auto-healing databases

19 Because Who Doesn t Want to Have DBAs who have time to automate more things More $$$ to spend on cool offsites, watching Hacker movies

20 We Wanted Instead Something dynamic and very scalable With very good introspection End-users shouldn t be impacted by the change Easy to shift traffic from one Amazon region to another Easy to mark a host as to be in/out of the pool Something open source Not create a SPOF

21 There Is No Silver Bullet VIPs Conf.d in combination with etcd DNSs Consul (Serf, Hashicorp)

22 SmartStack Service discovery tool by Airbnb It is decoupled, dynamic and eventual consistent One possible solution, the one for us: Simple We used it already successfully in most of our infrastructure Within MySQL we used it to bootstrap hosts

23 How to Locate a Service? There are 2 sides of the problem, of locating services: Service registration Service discovery

24 How Does SmartStack Work It consists of 2 separate applications Nerve for service registration Synapse for service discovery (ZooKeeper for the central registry)

25 Nerve Nerve is responsible for registering services in ZooKeeper Applications expose a health endpoint, that Nerve continuously monitors Provided the service is healthy, it will be registered in ZooKeeper The Service itself does not interact with ZooKeeper

26 Nerve

27 Synapse Synapse is responsible for the service discovery part Gets push notification from ZooKeeper for currently registered services It (usually) re-configures a locally running HAProxy instance

28 Synapse

29 HAProxy/JSON For MySQL we use static json files instead of HAProxy Prevents long living TCP connections going through HAProxy This helped us to choose a buddy database for our MySQL bootstrapping scripts* (*Check out Matt Ullmer s Percona talk earlier today: Bootstrapping a New Datacenter Made Easy )

30 How Does SmartStack Work? API endpoint: healthy? Provider Nerve Service Host JSON files HAProxy Zookeeper Service Synapse

31 How Does SmartStack Work With MySQL? We were using the SmartStack infrastructure already for MySQL bootstrapping Concerned about performance Our connections infrastructure was not fully ready to discover changes always right away

32 How Does SmartStack Work With MySQL? We choose to keep the yaml files Scripts which are querying json files, created by SmartStack to write topology.yaml for us It also enables us to fallback to manual editing in case of any failure of SmartStack

33 Python Scripts Data we get from the json files: cat /var/run/synapse/services/mysql_master_main.main.json [{"name":"db123","host":"10.xx.xx.xx","port":3306,"id":1,"weight":40}]

34 DB Environment At Yelp Past: DNS plus YAML files

35 Python Scripts Credentials.yaml file This file needs to be edited only once and not constantly

36 Global View In SmartStack you advertise everything on a level that you configure Most of our databases only advertised on a region level MySQL master needs to be advertised everywhere We wrote a service, to have a global view when we needed it

37 Python Script Implementation generate_topology.py --cluster: Clustername for which you want to create the topology file, e.g. primary. Defaults to all. --habitat: Habitat for which you want to create the topology file. Defaults to the habitat where it's ran. --credentials: Path to a yaml file that contains passwords. Initially took the credentials.yaml and checked for the local synapse.conf.json files It merged these files and created a new topology file Now it checks the global service

38 Implementation Milestones M1: We need to make sure that all of the databases we want to have in topology are in SmartStack (i.e. defined successfully in SmartStack repository and puppet git repository). M2: Implement the discovery service M3: Change the generate_topology.py script to be able to work with the discovery service M4: Implement integration tests such as predictably generating a topology file given Zookeeper status for the discovery service. M5: Deploy the auto-topology package via puppet M6: Use generate_topology.py instead of manually editing topology.yaml M7: Use the below described rollout plan in dev and eventually other habitats M8: Implement jenkins pipeline to automatically update topology

39 How Can We Control The System? sudo updown down all waiting 10 seconds for Synapse eventual consistency... done

40 How Does SmartStack Work? API endpoint: healthy? Provider Nerve Service Host JSON files HAProxy Zookeeper Service Synapse

41 How Does SmartStack Work With MySQL? Service Port 3306? Nerve JSON files HAProxy Zookeeper Synapse

42 How Does SmartStack Work With MySQL And Our Scripts? MY Service Host Port 3306 MySQL Nerve JSON files Service Synapse Zookeeper Deploy Host Yaml files

43 Advantages of Using SmartStack and MySQL No more topology.yaml file wrangling Less downtime So much time saving when editing those files Ops team can now take out a broken host

44 Advantages of Using SmartStack and MySQL No more topology.yaml file wrangling Less downtime So much time saving when editing those files Ops team can now take out a broken host Faster MTTR

45 Matching Capacity

46 Failure Modes Nerve is unavailable Synapse is unavailable Zookeeper is unavailable

47 Advantages of Using SmartStack and MySQL

48 Challenges of Using SmartStack and MySQL

49 Challenges of Using SmartStack and MySQL The synapse.conf.json file represents a local view You can extra_advertise a database in SmartStack Master advertised everywhere/replicas not advertised everywhere

50 Logic of the generate_topology Script

51 Challenges of Using SmartStack and MySQL

52 Elastic Scaling

53 Other Glue We Use More scripts To bootstrap databases To re- parent leaf replicas Terraform ASGs Puppet Jenkins

54 Bootstrap Scripts Install MySQL Configure my.cnf Set up SmartStack (packages, configs) Set up cronjobs Configure all OS services (e.g. ntpd, syslog-ng) Configure Sensu (Check out Matt Ullmer s Percona talk earlier today: Bootstrapping a New Datacenter Made Easy )

55 Re-parent Scripts Not on GTIDs (yet) Up to 5 level hierarchy tree in MySQL clusters In order to create the correct hierarchy we wrote some nifty scripts

56 Terraform+ASGs

57 module.mysql_aux_b_readdb.aws_autoscaling_group.mysql_replicas: Refreshing state... (ID: mysql-read-aux-uswest1bprod) State successfully pushed! The Terraform execution plan has been generated and is shown below. Resources are shown in alphabetical order for quick scanning. Green resources will be created (or destroyed and then created if an existing resource exists), yellow resources are being changed in-place, and red resources will be destroyed. Cyan entries are data sources to be read. Your plan was also saved to the path below. Call the "apply" subcommand with this plan file and Terraform will exactly execute this execution plan. Path:./tf_plan ~ module.mysql_service_a_readdb.aws_autoscaling_group.mysql_replicas desired_capacity: "3" => "2" max_size: "4" => "3" ~ module.mysql_service_b_readdb.aws_autoscaling_group.mysql_replicas desired_capacity: "3" => "2" max_size: "4" => "3" Plan: 0 to add, 2 to change, 0 to destroy.

58 What Empowers This To Do Now? Elastic scale for real Auto-healing Substantial savings $$$

59 Future Use the tools that we ve built to the max Omitting the topology files completely? Omitting the json files? Using HAProxy for master db failover?

60 Yelp DBAs Four DBAs Matt Ullmer ( Bootstrapping a New Datacenter Made Easy ) Patrick King (26 April - 1:00 PostgreSQL to MySQL: A DBA Perspective) Jenni Snyder Susanne Lehmann

61 We're Hiring!

62 Questions?

63 engineeringblog.yelp.com github.com/yelp

Mesos at Yelp: Building a production ready PaaS. Rob Johnson

Mesos at Yelp: Building a production ready PaaS Rob Johnson robj@yelp.com/@rob_johnson_ Who Am I: - Rob Johnson - Operations Team at Yelp - Spend most of my time working on PaaSTA Yelp s Mission: Connecting