MySQL As A Service. Operationalizing 19 Years of Infrastructure at GoDaddy

Transcription

1 MySQL As A Service Operationalizing 19 Years of Infrastructure at GoDaddy

2 WHOAMI Nathan Northcutt Senior Software Engineer MySQL DevOps ~10 years performance engineering & distributed data services. 2

3 Presentation In 3 Pictures I m excited to be here and share our story! Sometimes the pace of business grows faster than your infrastructure Automation complex tasks can have interesting results 3

4 Agenda 1. DevOps at GoDaddy 2. Daily Actions vs. Business Value 3. Containerization 4. Non-Greenfield Initiative 5. Service Building Blocks 6. Operational Impacts 7. Questions ~ 5 Minutes ~ 15 Minutes ~ 5 Minutes 4

5 Daily Actions vs. Business Value

6 DevOps At GoDaddy CentOS 4 Percona 4.X Percona 5.0 CentOS 6 Percona 5.1 Percona 5.5 CentOS 5 Customers DevOps Infrastructure Millions of internal/external users ~6.5 million databases > 15,000 instances Global footprint is growing 8 operational staff Ticket driven, 1 week on-call rotations. Tribal knowledge, complex environments. EXTREMELY heterogeneous Application specific configurations Nearly impossible to automate. 6

7 Business Value Performance Metrics Usage Warnings Track capacity/utilization On demand create/delete Monitoring Provisioning Host patching MySQL versioning Patching / Upgrades Configuration Safe runtime settings Adjust for workloads Backup / Restore High Availability DR backups 2x daily On demand backup/restore Master/Slave topologies 99.99% availability targets 7

8 GoDaddy DBaaS Requirements System interactions should be via API, both for customers and operations. Needs to scale with the business growth. Solution MUST have clear migration for legacy infrastructure. Fault tolerance should extend across zones/data centers, limit single points of failure. System should be patched/updated on a weekly basis. 8

9 DBaaS System Design

10 Data Plane Options Run multiple instances Run multiple VMs Run multiple containers Supported OOB No isolation Complete isolation Resource inefficient Resource isolation Shared kernel 10

11 Our Container Topology System Resources Net I/O CPU RAM Docker Mount Host FS Data Overlays 1. /var/lib/mysql 2. /var/log/mysql 3. /etc/my.cnf Kernel Namespaces / cgroups Overlay FS Redirect /var/lib/dbaas/xxxx Docker Container Container Structure initd (systemctl) DBaaS Overlay (MySQL, XTraBackup, services) CentOS 7 Image Run full systemd MySQL services DBaaS services Custom CentOS 7 11

12 Container Benefits No longer a heterogeneous environment and has predictable performance characteristics. Difference between two instances is the data and resource throttles. Instances are now defined as a network route + raw data files. Host server allocation becomes a bin packing problem, instances are completely isolated. All of these map back to well known service patterns! 12

13 Provisioning Process Data Plane Capacity Service Layer 1. Request Instance 2. Locate Server Persistent Storage DR User 3. Notify Routing 4. Connect via DNS:Port Routing Layer 5. Route to current master. 13

14 Legacy Onboarding Routing Layer Data Plane Capacity Persistent Storage 1. Use DNS:Port 6. Redirect 4. Restore into DBaaS Legacy Products 2. Forward to legacy 5. Slave replication 3. DR Backup Tools User Server IP + Port New Data Plane Capacity! 14

15 Minor Patching Cycle Development Repository 4. Image moved into production repository. Production Repository 1. Daily Docker Image builds 2. Data plane uses images for new capacity requests. 3. Integration tests continually run in our development environment to identify issues. 5. Data plane uses images for new capacity requests. 15

16 Upgrades MySQL (5.6) Persistent Storage Current Route 1. Backup Master 5. Backup Master 6. Failover to Slave to master Routing Layer 3. In place upgrade (or mysqldump). MySQL (5.7) 2. Copy down master binary data. 16

17 Service Building Blocks 1. Service handles operational oriented commands to achieve user functionality. Move Instance 2. Bottom tier is simple, requires little to no coordination between operations. 3. Middle tier is built out of bottom tier, simple tasks. Some are more complex, others are still quite simple. 4. Upper tier requires coordination on multiple, prior steps to complete. Complexity Backup Master Create Slave Create Instance Restore Promote Slave Failover Route Delete Slave Delete Instance 5. Why Move? Resize, upgrades, host balancing, promote environments, optimized routes etc. *Build off previous commands to create complexity 17

18 Operational Impacts

19 Remediation Services Routing Layer Metrics Data Plane Hosts Persistent Storage Services Layer Monitor service availability and proxy health. Replace dead/unresponsive nodes. Detect network issues between components. Monitor metrics feeds for alerts and usage warnings. Move noisy neighbors. Maintain HA topologies. 19

20 Early Results Provisioning time reduced from ~2 weeks to < 15 seconds. Metrics dashboards provide easy request tracing to quickly troubleshoot issues and deploy appropriate changes without request resubmission. System automatically repaired several intentional failures related to networking, capacity exhaustion and host failure. System applies patches and minor upgrades DAILY for several months with no issues or user impacts in our development and integration environments. 20

21 Rate My Session 21

22 Questions?

23 Appendix

24 Container Anatomy Net I/O CPU RAM Docker Mount Host FS System Resources Namespaces / cgroups Overlay FS Kernel Container Main Process Additional Image Overlay(s) Docker Application Target Process Image Resources Base Image 24

25 Network Routing Proxy Cluster Master / Slave(s) 1. DNS Lookup 3. Proxy TCP Load Balancer 2. Resolve to LB 4. Replication DNS:Port Routing Services 1. Update Proxy Configuration 2. Eventual Consistency Sync Persistent Storage 25

26 DevOps Tooling HTML5 Core Services 1. D3 interactive displays for current topology and communications. 2. HTML5 displays to support mobile as well as desktop/laptop interaction. 3. Hooks into services to perform support or operations on behalf of users. 4. Dashboards for system utilization/metrics for quick investigation and resolution. Routing Services 26