Let Robots Manage Your Schema Without Killing All Humans. Jenni

Transcription

1 Let Robots Manage Your Schema Without Killing All Humans Jenni

2 Yelp s Mission Connecting people with great local businesses.

3 Yelp Stats As of Q M 95M 70% 32

4 Database Yelp 300+ Engineers 3 Database Engineers 100+ MySQL servers ~75 DBs/Schemas

5 Schema Change is Good New features Fix bugs Speed things up

6 Schema Yelp - History Devs run ALTER TABLE at will #yoloops 2011 Replication delay becomes a thing ALTER TABLE during "maintenance windows" Tables getting bigger More TZs means no "quiet time"

7 Schema Yelp - History pt-online-schema-change Dev makes change in dev DBA uses pt-online-schema-change in stages prod

8 History of Schema Yelp environments: 2 dev, 2 stage, 1 prod environments: 2 dev, 2 CI, 2 stage, and 1 prod environments: 3 dev, 3 CI, 4 stage, 1 prod

9 HOW DID WE LET IT COME TO THIS?!

10 SAD DEVELOPERS

11 SAD DBAs

12 Solution: Automation

13 What We Were Already Doing Right Schema conventions Rollback safety Reviewboard GrossQueryChecker Killing long queries & transactions

14 But, We Can't Kill our Queries!

15 We Can't Kill our Queries!

16 Rollback Safety De-couple schema changes from code DEFAULT NULL NOT NULL DEFAULT something Backfill with code New column if data must be transformed

17 We Already Had These Tools pt-online-schema-change Liquibase Python Jenkins

18 Intro to pt-online-schema-change

19 More on pt-online-schema-change Works pretty good out of the box Deployed w/ puppet Configs go in /etc/percona-toolkit End result: pt-online-schema-change $DSN --alter "ADD KEY (woot)"

20 # pt-online-schema-change.cnf: config file # This file is managed by Puppet. Please do not edit locally. chunk-time=0.5 critical-load=threads_running=175 max-lag=4 no-check-replication-filters no-drop-old-table set-vars=lock_wait_timeout=5 statistics # we only create the slave_lag plugin that uses # repl_delay_reporter (by way of smartstack on Ubuntu hosts) recurse=0 plugin=/etc/percona-toolkit/pt-online-schema-change-plugin.pl

21 Pt-online-schema-change - Dangers Waiting for table metadata lock This version of MySQL doesn't yet support multiple triggers with the same action time and event for one table Bugs

22 Plugins at Yelp Verify read_only Replication delay IRC notifications Replication coordinates

23 # sample plugin code here # pt-online-schema-change-plugin.pl use strict ; package pt_online_schema_change_plugin ; sub init { my ($self, %args) my $dbh = $self->{aux_cxn}->dbh; = $dbh->selectrow_array("select DATABASE()") ; if ( $self->{execute} ) { `/nail/sys/bin/nodebot dba "pt-online-schema-change starting alter in $dbname[0]: $self->{alter}"` ; } }

24 sub before_create_new_table { my ($self, %args) # connect to the database, and verify that read_only==0 my $dbh = $self->{aux_cxn}->dbh; = $dbh->selectrow_array("select \@\@read_only") ; if ( $is_read_only[0]!= 0 ) { print "\nerror: MySQL is running in read-only mode, bailing out!\n\n" ; exit 0 ; } else { return ; } }

25 Replication Delay Reporter Service Returns JSON or single number Uses app's client config

26 Why Not Have Devs Run It? Issues are rare, but serious DBAs prefer to resolve Hard to share right info

27 Liquibase Already had it in house yaml format Extension support, liquibase-percona

28 Liquibase - Files and Tables Change file - SQL Change log file - yaml Bookkeeping table in each database

29 cat pushplans/db_yelp/2015/08/ testing-liquibase.sql --liquibase formatted sql --changeset author:jsnyder failonerror:true -- new table test, this is our first attempt at a liquibase changeset CREATE TABLE `test` ( `id` int(11) NOT NULL AUTO_INCREMENT, `a` varchar(32) DEFAULT NULL, `b` int(11) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 ;

30 $ cat databasechangelog.yaml databasechangelog: - relativetochangelog:true - include: file: pushplans/db_yelp/2015/08/ testingliquibase.sql - include: file: pushplans/db_yelp/2015/08/ testingliquibase2.sql

31 CREATE TABLE `DATABASECHANGELOG` ( `ID` varchar(255) NOT NULL, `AUTHOR` varchar(255) NOT NULL, `FILENAME` varchar(255) NOT NULL, `DATEEXECUTED` datetime NOT NULL, `ORDEREXECUTED` int(11) NOT NULL, `EXECTYPE` varchar(10) NOT NULL, `MD5SUM` varchar(35) DEFAULT NULL, `DESCRIPTION` varchar(255) DEFAULT NULL, `COMMENTS` varchar(255) DEFAULT NULL, `TAG` varchar(255) DEFAULT NULL, `LIQUIBASE` varchar(20) DEFAULT NULL, `CONTEXTS` varchar(255) DEFAULT NULL, `LABELS` varchar(255) DEFAULT NULL ) ENGINE=InnoDB DEFAULT CHARSET=latin1

32 Liquibase - Drawbacks yaml syntax not fully supported liquibase-percona still adding ALTER types

33 Some Duct Tape Wrapper script for Liquibase Generate changes as yaml for Liquibase Collect information about environment Because, logging

34 Jenkins Polls repos for changes Runs jobs Notifies of the output Jenkins Job Builder

35 Jenkins - How we use it Detects new schema change(s) Knows about each environment Runs job

36

37 In Practice: Developer Follows checklist to develop SQL Runs wrapper to develop/test locally Code Review Push to stage, prod Code merged to master

38 In Practice: Automation Jenkins Runs wrapper Push to all other environments Idempotence FTW

39 Where We Are Now Not 100% automated Edge cases Manual resolution of issues Ready to use pt-kill more >:) Experimenting with Anemometer Pretty Happy

40

41 engineeringblog.yelp.com github.com/yelp