Percona Live Europe Amsterdam, Netherlands October 3 5, 2016

Transcription

1 Percona Live Europe 2016 Amsterdam, Netherlands October 3 5, 2016

2 Microservices, MySQL & Service Discovery Abbas Ahmed : Principal Database rentalcars.com

3 Who am I? Principal Database Rentalcars.com (6+ years) Prior experience in Telecom, Talent & Social Media sectors. RHCE Trainer, MySQL DBA, Couchbase DBA, SQL Server DBA Maker.. Love my Raspberry PI 3

4 Service Discovery 4

5 Service Discovery A common problem with many solutions 4

6 Service Discovery A common problem with many solutions Problem: How does an application find out where to send a database or API request. 4

7 Service Discovery A common problem with many solutions Problem: How does an application find out where to send a database or API request. Usually considered in a web service, or micro service context. 4

8 Service Discovery A common problem with many solutions Problem: How does an application find out where to send a database or API request. Usually considered in a web service, or micro service context. The invoked API could also be a dynamically scalable service. 4

9 Service Discovery A common problem with many solutions Problem: How does an application find out where to send a database or API request. Usually considered in a web service, or micro service context. The invoked API could also be a dynamically scalable service. Specially relevant in large deployments with hundreds or thousands of service endpoints involved. 4

10 Service Discovery (continued) 5

11 Service Discovery (continued) Traditional approaches: 5

12 Service Discovery (continued) Traditional approaches: Static config (Really hard to change, impossible to manage for large deployments, Scheduled Puppet runs!) 5

13 Service Discovery (continued) Traditional approaches: Static config (Really hard to change, impossible to manage for large deployments, Scheduled Puppet runs!) DNS A/CNAME records ( TTL expiry, DNS zone reloads, DNS propagation delays) 5

14 Service Discovery (continued) Traditional approaches: Static config (Really hard to change, impossible to manage for large deployments, Scheduled Puppet runs!) DNS A/CNAME records ( TTL expiry, DNS zone reloads, DNS propagation delays) DNS SRV records ( Requires additional intelligence on clients) 5

15 Service Discovery (continued) Traditional approaches: Static config (Really hard to change, impossible to manage for large deployments, Scheduled Puppet runs!) DNS A/CNAME records ( TTL expiry, DNS zone reloads, DNS propagation delays) DNS SRV records ( Requires additional intelligence on clients) HAProxy/Maxscale/ProxySQL or some other load balancer to mask complexity from application servers. (No standard Management API) 5

16 Rentalcars.com Infrastructure Two Data centres More than a thousand servers Monolithic Applications Elements in Tech Stack MySQL (Large replica sets ~ 200 servers) Couchbase PHP Java Hazelcast HDFS/Spark/Impala 6

17 Evolving Infrastructure 7

18 Evolving Infrastructure A shift to Micro Services architecture, legacy applications being split into smaller, cohesive sets of micro services. 7

19 Evolving Infrastructure A shift to Micro Services architecture, legacy applications being split into smaller, cohesive sets of micro services. At the database end, smaller replica sets with focused, Micro service specific datasets. 7

20 Evolving Infrastructure A shift to Micro Services architecture, legacy applications being split into smaller, cohesive sets of micro services. At the database end, smaller replica sets with focused, Micro service specific datasets. Challenge - Lots of the above 7

21 Challenges on the MySQL layer 8

22 Challenges on the MySQL layer Dynamic scaling of MySQL replica sets 8

23 Challenges on the MySQL layer Dynamic scaling of MySQL replica sets Graceful handling of database node failures 8

24 Challenges on the MySQL layer Dynamic scaling of MySQL replica sets Graceful handling of database node failures Communicate replication topology changes to the application layer 8

25 Challenges on the MySQL layer Dynamic scaling of MySQL replica sets Graceful handling of database node failures Communicate replication topology changes to the application layer Topology awareness features lacking in MySQL connectors 8

26 Challenges on the MySQL layer Dynamic scaling of MySQL replica sets Graceful handling of database node failures Communicate replication topology changes to the application layer Topology awareness features lacking in MySQL connectors Absence of a standard topology manager. 8

27 Challenges on the MySQL layer Dynamic scaling of MySQL replica sets Graceful handling of database node failures Communicate replication topology changes to the application layer Topology awareness features lacking in MySQL connectors Absence of a standard topology manager. MySQL Fabric aware connectors not available for all of our stack (at time of project start, at least) 8

28 Possible Solutions 9

29 Possible Solutions Galera / XTRADB Cluster with MariaDB Maxscale / HAproxy (Limited by number of nodes) 9

30 Possible Solutions Galera / XTRADB Cluster with MariaDB Maxscale / HAproxy (Limited by number of nodes) Master-Slave replica set with a topology manager in place. The choices were: 9

31 Possible Solutions Galera / XTRADB Cluster with MariaDB Maxscale / HAproxy (Limited by number of nodes) Master-Slave replica set with a topology manager in place. The choices were: - Outbrain Orchestrator 9

32 Possible Solutions Galera / XTRADB Cluster with MariaDB Maxscale / HAproxy (Limited by number of nodes) Master-Slave replica set with a topology manager in place. The choices were: - Outbrain Orchestrator - MySQL Fabric 9

33 Possible Solutions Galera / XTRADB Cluster with MariaDB Maxscale / HAproxy (Limited by number of nodes) Master-Slave replica set with a topology manager in place. The choices were: - Outbrain Orchestrator - MySQL Fabric Ideally, we could have used MySQL Fabric as the topology manager with Fabric aware connectors OR MySQL router. (Wasn t production ready when we started). 9

34 Our Solution - First iteration 10

35 Our Solution - First iteration Choices Made: 10

36 Our Solution - First iteration Choices Made: Load Balancer: MariaDB Maxscale 10

37 Our Solution - First iteration Choices Made: Load Balancer: MariaDB Maxscale Topology Manager: Outbrain Orchestrator 10

38 Our Solution - First iteration Choices Made: Load Balancer: MariaDB Maxscale Topology Manager: Outbrain Orchestrator Service Registry: Hashicorp Consul with Consul templates & confd 10

39 Our Solution - First iteration Choices Made: Load Balancer: MariaDB Maxscale Topology Manager: Outbrain Orchestrator Service Registry: Hashicorp Consul with Consul templates & confd 10

40 Why Use Consul? 11

41 Why Use Consul? Easy to manage clustered service offering: 11

42 Why Use Consul? Easy to manage clustered service offering: Service Registry 11

43 Why Use Consul? Easy to manage clustered service offering: Service Registry Multi-Datacenter awareness 11

44 Why Use Consul? Easy to manage clustered service offering: Service Registry Multi-Datacenter awareness DNS based discovery service 11

45 Why Use Consul? Easy to manage clustered service offering: Service Registry Multi-Datacenter awareness DNS based discovery service Distributed Key / Value store 11

46 Why Use Consul? Easy to manage clustered service offering: Service Registry Multi-Datacenter awareness DNS based discovery service Distributed Key / Value store Service Health Checking 11

47 Maxscale - Things to consider 12

48 Maxscale - Things to consider In the event of a MySQL replication topology change MariaDB detects using its own monitoring, but adding additional nodes or taking an existing node offline requires external help. 12

49 Maxscale - Things to consider In the event of a MySQL replication topology change MariaDB detects using its own monitoring, but adding additional nodes or taking an existing node offline requires external help. To force MariaDB MaxScale to reread the configuration file, we can send a SIGHUP signal to the MariaDB MaxScale process or execute reload config in the maxadmin client. 12

50 Maxscale - Things to consider In the event of a MySQL replication topology change MariaDB detects using its own monitoring, but adding additional nodes or taking an existing node offline requires external help. To force MariaDB MaxScale to reread the configuration file, we can send a SIGHUP signal to the MariaDB MaxScale process or execute reload config in the maxadmin client. Encrypt Maxscale passwords 12

51 Maxscale - Things to consider In the event of a MySQL replication topology change MariaDB detects using its own monitoring, but adding additional nodes or taking an existing node offline requires external help. To force MariaDB MaxScale to reread the configuration file, we can send a SIGHUP signal to the MariaDB MaxScale process or execute reload config in the maxadmin client. Encrypt Maxscale passwords Application grants need to be adjusted slightly 12

52 Maxscale - Things to consider In the event of a MySQL replication topology change MariaDB detects using its own monitoring, but adding additional nodes or taking an existing node offline requires external help. To force MariaDB MaxScale to reread the configuration file, we can send a SIGHUP signal to the MariaDB MaxScale process or execute reload config in the maxadmin client. Encrypt Maxscale passwords Application grants need to be adjusted slightly GRANT SELECT on DB1.* to App_User@ app% ; Needs to become 12

53 Maxscale - Things to consider In the event of a MySQL replication topology change MariaDB detects using its own monitoring, but adding additional nodes or taking an existing node offline requires external help. To force MariaDB MaxScale to reread the configuration file, we can send a SIGHUP signal to the MariaDB MaxScale process or execute reload config in the maxadmin client. Encrypt Maxscale passwords Application grants need to be adjusted slightly GRANT SELECT on DB1.* to App_User@ app% ; Needs to become GRANT SELECT on DB1.* to App_User@ app-01.dc1.infrastructure.com ; OR 12

54 Maxscale - Things to consider In the event of a MySQL replication topology change MariaDB detects using its own monitoring, but adding additional nodes or taking an existing node offline requires external help. To force MariaDB MaxScale to reread the configuration file, we can send a SIGHUP signal to the MariaDB MaxScale process or execute reload config in the maxadmin client. Encrypt Maxscale passwords Application grants need to be adjusted slightly GRANT SELECT on DB1.* to App_User@ app% ; Needs to become GRANT SELECT on DB1.* to App_User@ app-01.dc1.infrastructure.com ; OR GRANT SELECT on DB1.* to App_User@ ; 12

55 Maxscale - Things to consider In the event of a MySQL replication topology change MariaDB detects using its own monitoring, but adding additional nodes or taking an existing node offline requires external help. To force MariaDB MaxScale to reread the configuration file, we can send a SIGHUP signal to the MariaDB MaxScale process or execute reload config in the maxadmin client. Encrypt Maxscale passwords Application grants need to be adjusted slightly GRANT SELECT on DB1.* to App_User@ app% ; Needs to become GRANT SELECT on DB1.* to App_User@ app-01.dc1.infrastructure.com ; OR GRANT SELECT on DB1.* to App_User@ ; AND remember! 12

56 Maxscale - Things to consider In the event of a MySQL replication topology change MariaDB detects using its own monitoring, but adding additional nodes or taking an existing node offline requires external help. To force MariaDB MaxScale to reread the configuration file, we can send a SIGHUP signal to the MariaDB MaxScale process or execute reload config in the maxadmin client. Encrypt Maxscale passwords Application grants need to be adjusted slightly GRANT SELECT on DB1.* to App_User@ app% ; Needs to become GRANT SELECT on DB1.* to App_User@ app-01.dc1.infrastructure.com ; OR GRANT SELECT on DB1.* to App_User@ ; AND remember! Add SHOW DATABASES to the App_User Grants. (Maxscale version 1.4.3) 12

57 maxkeys - Maxscale password encryption #Maxscale Secrets file { '/var/lib/maxscale/.secrets': } -> ensure => file, owner => 'maxscale', group => 'maxscale', mode => '0400', source => 'puppet:///modules/ms/var/lib/maxscale/secrets', require => Package['maxscale']; service { 'maxscale': ensure => 'running', enable => true; } -> 13

58 A possible architecture 14

59 Actual Architecture - Iteration 1 15

60 Architecture Iteration 1 - Zoomed In 16

61 Benefits from this arrangement 17

62 Benefits from this arrangement Resilient DB connections for the apps. 17

63 Benefits from this arrangement Resilient DB connections for the apps. Immediate application DNS endpoint changes communicated by Consul. 17

64 Benefits from this arrangement Resilient DB connections for the apps. Immediate application DNS endpoint changes communicated by Consul. Consul manages separate DNS zone for each DC. 17

65 Benefits from this arrangement Resilient DB connections for the apps. Immediate application DNS endpoint changes communicated by Consul. Consul manages separate DNS zone for each DC. If topology changes, Orchestrator informs Consul via REST API. So the applications are able to discover the correct DB endpoints. 17

66 Benefits from this arrangement Resilient DB connections for the apps. Immediate application DNS endpoint changes communicated by Consul. Consul manages separate DNS zone for each DC. If topology changes, Orchestrator informs Consul via REST API. So the applications are able to discover the correct DB endpoints. This includes: 17

67 Benefits from this arrangement Resilient DB connections for the apps. Immediate application DNS endpoint changes communicated by Consul. Consul manages separate DNS zone for each DC. If topology changes, Orchestrator informs Consul via REST API. So the applications are able to discover the correct DB endpoints. This includes: - Adding / removing MySQL nodes. 17

68 Benefits from this arrangement Resilient DB connections for the apps. Immediate application DNS endpoint changes communicated by Consul. Consul manages separate DNS zone for each DC. If topology changes, Orchestrator informs Consul via REST API. So the applications are able to discover the correct DB endpoints. This includes: - Adding / removing MySQL nodes. - Election of a new node if Master MySQL goes down. 17

69 Benefits from this arrangement Resilient DB connections for the apps. Immediate application DNS endpoint changes communicated by Consul. Consul manages separate DNS zone for each DC. If topology changes, Orchestrator informs Consul via REST API. So the applications are able to discover the correct DB endpoints. This includes: - Adding / removing MySQL nodes. - Election of a new node if Master MySQL goes down. Simplified DB access grants for applications 17

70 Benefits from this arrangement Resilient DB connections for the apps. Immediate application DNS endpoint changes communicated by Consul. Consul manages separate DNS zone for each DC. If topology changes, Orchestrator informs Consul via REST API. So the applications are able to discover the correct DB endpoints. This includes: - Adding / removing MySQL nodes. - Election of a new node if Master MySQL goes down. Simplified DB access grants for applications New DB Servers register with Consul when they come up for the first time. Then they can participate in the discovery process. 17

71 Benefits from this arrangement Resilient DB connections for the apps. Immediate application DNS endpoint changes communicated by Consul. Consul manages separate DNS zone for each DC. If topology changes, Orchestrator informs Consul via REST API. So the applications are able to discover the correct DB endpoints. This includes: - Adding / removing MySQL nodes. - Election of a new node if Master MySQL goes down. Simplified DB access grants for applications New DB Servers register with Consul when they come up for the first time. Then they can participate in the discovery process. 17

72 Generating Maxscale Config - templ file [MySQL Monitor] type=monitor module=mysqlmon servers={{ $nodes := ls "/confd/maxscale/mydb/" }}{{ join $nodes "," }} user=maxscale passwd=d48e0e975ae acd monitor_interval=10000 [CLI Listener] type=listener service=cli protocol=maxscaled port=6603 [MYDBReadLBService] type=service router=readconnroute servers={{ $nodes := ls "/confd/maxscale/mydb/" }}{{ join $nodes "," }} router_options=running user=maxscale passwd=d48e0e975ae acd {{ range gets "/confd/maxscale/mydb/*" }} {{- $data := json.value }} [{{ base.key }}] type=server address={{ $data.addr }} port=3306 protocol=mysqlbackend {{end}} 18

73 ConfD - Configuration example # confd configuration file { } "/etc/confd/conf.d/${template}.toml": ensure => file, owner => 'root', group => 'root', mode => '0644', content => template("mysvc/etc/confd/conf.d/${template}.toml.erb"), notify => Service['confd']; "/etc/confd/templates/${template}.tmpl": ensure => file, owner => 'root', group => 'root', mode => '0644', content => template( mysvc/etc/confd/templates/${template}.tmpl.erb"), notify => Service['confd']; 19

74 Putting everything together [template] src = "my-db.tmpl" dest = "/etc/maxscale.cnf" keys = [ "/confd/maxscale/mydb/", ] check_cmd = "/sbin/service maxscale configtest" reload_cmd = "/sbin/service maxscale reload" 20

75 Consul Template # Maxscale Config consul_template::watch { 'maxscale': template => 'my/etc/consul_template/mydb_maxscale.erb', destination => '/etc/maxscale.cnf', command => /bin/service maxscale reload'; } 21

76 Demo time 22

77 How could this solution evolve? Add additional server specific parameters like weightage etc. to the Consul values. Replace load balancer functionality with ProxySQL? Any suggestions from the audience? 23

78 Questions?