Automating MySQL operations with Puppet. Kris Buytaert &

Transcription

1 Automating MySQL operations with Puppet Kris Buytaert &

2 Kris Buytaert I used to be a Dev, Then Became an Op Chief Trolling Officer and Open Source Everything is an effing DNS Problem Building Clouds since before the bookstore Some books, some papers, some blogs Evangelizing devops Organiser of #devopsdays, #cfgmgmtcamp, #loadays,. Part of the travelling geek circus

3 Julien Pivotto I am a sysadmin I develop sometimes Chief Traveling Officer and Open Source Born almost at cloud age :) Monitoring, automation, databases, all mixed

4 What's this devops thing anyhow?

5 C(L)AMS Culture (Lean) Automation Measurement Sharing Damon Edwards and John Willis Gene Kim

6 A global movement to improve the quality of software delivery leveraging Open Source experience, started in Gent in 2009

7 Because the old days: Put this Code Live, here's a tarball NOW! What dependencies? No machines available? What database? Security? High Availability? Scalability? My computer can't install this?

8 10 days into operations

9 DevSalesDBASecQAMark etingmonitoringbusops Nope DevOps didn t forget anyone Actually DevOps does not make sense without everyone Also devops!= everyone is dba/tester/dev/op... devops!= a job title devops!= a team name

10 Why automate?

11 Common Problems Many manual changes to systems Many undocumented changes Emergency Administration only Disaster Recovery site is a Disaster Time to deliver a box is to slow All boxen are different Computers don t work hard enough for us

12 More Problems How long does it take to reinstall a machine from 0 To the exact same point as before? With different Hardware? In a different cloud? What about your (customer/personal data ) Did I just redirect prod frontend to dev backend?

13 Automation is not just about scale I don t automate coz I just have X servers is no excuse You don t need zillions of apps/servers to benefit from automation It starts from one server Especially in the database world where we look for performance, fine tuning, and consistency

14 Security? Monitoring that your platform hasn't changed. Why is selinux disabled? Who added / dropped that firewall? What did this originally look like? Is this file really what Bernd meant it to be?

15 #monitoringsucks Monitoring is out of sync with reality Managed manually Can't keep up..

16 Challenges Reproducability Speed Auditing Keeping stuff in sync Monitoring Security Backup

17 Facts! Data Backup is only a part Sysadmin backup needs to be done also Manual Installations = bad Bad installations = unusable infrastructure Bad installations = unproductive users Bad installations = manual efforts Manual efforts = no time No time = no updates no patches no security Manual work = high costs No security + high costs = Bancrupcy

18 th The 10 floor test Grab a random machine (don t take a backup before) th Throw it out a 10 floor window Can you recover it in 10 minutes?

19 Imagesprawl AND Snowflakes Image Sprawl : Copy vm 3x Modify 2x Copy 21x How the Heck did we get here? SnowFlakes : Don't touch this box it might break Look how nice it is!

20 You never deploy something just once Local test experiment, Integration Platform Vagrant box / local containers Same codebase,, different environment Dev/ UAT/ Prod / DR Or your customer just forgot to renew the lease on his VPS. #toldyouso

21 What's different in the cloud? Scale Velocity Change

22 Your machines as Cattle

23 Treat your people as pets

24 Configuration Mgmt Configure 1000 nodes, Modify files, Think : Cfengine,Puppet, Chef, Salt Put configs under version control Please don't roll your own...

25 Infrastructure as Code Treat configuration automation as code Development best practices Model your infrastructure Version your cookbooks / manifests Test your cookbooks/ manifests Dev/ test /uat / prod for your infra Model your infrastructure A working service = automated ( Application Code + Infrastructure Code + Security + Monitoring ) IAC -ne scripting (or translating bash to yaml)

26 Auditability git log git blame Review, authorization

27 File monitoring

28 Other benefits of code Code can be tested You can attach it to a bugtracker You can deploy, redeploy, You can analyse it without getting access to the infra You can revert changes

29 Puppet Open Source Config Management Tool Ecosystem 10y old ; very mature Ruby, Clojure, Jruby, C++,... Large community Lots of open source modules (cose shared by other people to automate things) Puppet is also the name of the company behind it

30 Puppet in short You define the desired state of your infra Puppet looks at the current state and takes actions to get to the final state You have the report of what has been changed/errored/ It is declarative

31 How Puppet Works Puppet works by default on a server/agent model Agents fetch their catalog (= final desired state) from the master and apply that locally Then they report back to the server The catalog is computed depending on facts the machine sends to the server, user input (parameters) and modules (source code)

32 Puppet Modules Puppet modules are reusable set of Puppet manifests Contains resources, templates, files, e.g. Puppet-apache, puppet-postgresql, puppetcorosync Thousands of them are published on github and Puppet Forge

33 Roles and Profiles A profile is a puppet module that explains how to install a service A role is a Puppet manifest that includes profiles A server should get one and only one role class profile_mysql { class { '::mysql::server': } class { '::mysql::xtrabackup': } } class role::mysql_server { include profile_mysql include profile_base }

34 Defining a Service profile that : Configures service using a standard module call with hiera based parameters Configures Backup Configures logrotation Configures logshipping Exports Monitoring Needs

35 MySQL+Puppet There used to be: Puppet-percona Puppet-mariadb Puppetlabs-mysql Now community has been built around puppetlabs-mysql (4 millions download on puppet forge 215 contributors) Same story with other pieces of software (apache, ntp, postgresql, iptables...)

36 What is in the module? Mysql server config Mysql db creation Mysql users creation Root password Mysql bindings (c, java, perl, php ) Backups

37 Setting up mysql (or mariadb depending on distro default) class { '::mysql::server': }

38 You probably want this class { '::mysql::server': service_manage false, } And let your HA service taking care of that

39 Options class { '::mysql::server': override_options 'mysqld' 'server-id' 'bind-address' 'log-error' 'pid-file' 'innodb_file_per_table' }, }, } { { $::mysql_server_id, ' ', '/var/log/mysqld.log', '/var/run/mysqld/mysqld.pid', true,

40 Users mysql_user { "backup@ ": ensure 'present', password_hash mysql_password( secret ), } mysql_grant { "backup@ /*.*": ensure 'present', privileges [ 'ALL' ], table '*.*', user "backup@ ", }

41 But its Puppet like 2011 Nowadays in infra as code you split data and code Hiera : yaml as source of data Other backends than yaml available Vault Merges data from env-specific, os-specific, datacenter-specific key value stores...

42 In Hiera: --mysql::server::db: "dev": user: "dev" password: "devpass" host: " " grant: - "ALL" In Puppet: create_resources(mysql::db, hiera('mysql::server::db', {}))

43 Exported resources

44 Exported resources You can export resources from nodes And fetch them in another node Exporting: Collecting: user password dbname host grant tag } { $name: $user, $password, $dbname, $host, $grant, $tag, Mysql::Db << tag == $tag >>

45 Downsides & Workarounds Exported resources are slow, puppet needs to run on the app server to export resources to the mysql server Requires a puppetserver (with puppetdb) Workaround 1: Accept that and make your app resilient to DB not ready Workaround 2: Design your app in hiera as yaml and pick the info from there in both app server and database server

46 Replication user mysql_user { 'replica@%': ensure 'present', password_hash mysql_password('percona17'), } mysql_grant { 'replica@%/*.*': ensure 'present', privileges ['REPLICATION SLAVE'], table '*.*', user 'slave_user@%', }

47 Replication Options class { '::mysql::server': override_options 'mysqld' 'server-id' 'binlog-ignore-db' 'log-bin' 'binlog-format' }, }, } Chose if you want to run in a Puppet exec, a more Intelligent script, (MariaDB based example:) { { 1, [ mysql, performance_schema ], 'binary-log.', 'ROW', CHANGE MASTER TO master_host="$mysql_master", master_port=3306, master_user='replica', master_password='percona17', master_use_gtid=current_pos; START SLAVE;

48 Configuring HA Example using corosync/pacemaker (puppetcorosync) cs_primitive { 'mysql_vip': cs_primitive { p_mysql : primitive_class primitive_type provided_by Promotable operations { 'monitor' 'interval' 'timeout' { 'monitor' 'interval' 'timeout' 'role' ] } 'ocf', 'mysql', 'heartbeat', true, [ { } '10s', '30s' } }, { '5s', '30s', 'Master', } }, primitive_class primitive_type provided_by Parameters 'ip' 'cidr_netmask' Operations 'monitor' 'interval' 'ocf', 'IPaddr2', 'heartbeat', { ' ', '24' }, { { '10s' } }, cs_colocation { 'mysql_master': Primitives [ 'mysql_vip', 'p_mysql:master' ], }

49 Conclusion IAC solves a lot of problems Improves Security Creates Monitoring Love Creates Speed Creates Consistency But it still is code, and needs to be treated like code!

50 More of Gent, 5 & 6 february Right after Fosdem

51 Contact Kris Buytaert Julien Pivotto Further Inuits Essensteenweg 31 Brasschaat Belgium