Opera&onal DBA In A Nutshell. Tom De Cooman Dimitri Vanoverbeke

Transcription

1 Opera&onal DBA In A Nutshell Tom De Cooman <tom.decooman@percona.com> Dimitri Vanoverbeke <dimitri.vanoverbeke@percona.com>

2 Introduc&on 2 Welcome! Who are we? Fans of devopsreacaons.tumblr.com What will we talk about?

3 Basic Housekeeping 3 Full Day Tutorial HandsOn! Laptop Required! Breaks Wireless SSID =?? How to ask for help Using the documentaaon

4 Beware! 4 This tutorial is not for you if You know what replicate_same_server_id is! You know all GRANTs by heart! pt-query-digest is your favorite tool! You ve been dealing with replicaaon inconsistencies for a while already! Your backup environment is the most awesome there is!

5 Opera&onal DBA In A Nutshell 5 SeTng Up Today s Environment InstallaAon Logging In MySQL Privileges DiagnosAcs TroubleshooAng Backups ReplicaAon Schema Changes ConfiguraAon OpAmizaAon How doing operaaons feels

6 Opera&onal DBA In A Nutshell 6 SeAng Up Todays Environment InstallaAon Logging In MySQL Privileges DiagnosAcs TroubleshooAng Backups ReplicaAon Schema Changes ConfiguraAon OpAmizaAon How doing operaaons feels

7 SeAng Up Our VM 7 Copy Files From USB SAck Install VirtualBox, open the vbox files, start all VMs Test ConnecAvity ssh -p 2221 root@localhost (password: vagrant) ssh -p 2222 root@localhost (password: vagrant) hdp://localhost:8080/ (you should see an apache test page) VBoxManage.exe modifyvm <machine name> --hwvirtex off

8 Opera&onal DBA In A Nutshell 8 SeTng Up Todays Environment Installa&on Logging In MySQL Privileges DiagnosAcs TroubleshooAng Backups ReplicaAon Schema Changes ConfiguraAon OpAmizaAon How doing operaaons feels

9 What versions of MySQL do you run? 9 MySQL 5.7 Community ediaon? Percona Server 5.7? MariaDB 10.1? WebScaleSQL? MySQL Cluster? MySQL 5.7 Enterprise ediaon?

10 What is MySQL community Edi&on 10 Standard Oracle community Version (previously MySQL AB, Sun) Contains a standard MySQL version with InnoDB and MyISAM as a standard The original source Packages for a large array of operaang systems including Windows and MacOS Open Source

11 What is Percona Server 11 Enhanced version of MySQL community ediaon Added plugins for authenacaaon (PAM) audiang diagnosacs Improved and Extra storage engines Improved Performance (Patches) Improved backup and management capabiliaes Hotbackup using Percona XtraBackup Monitoring using PMM or other tools Packages for Linux distribuaons

12 What is MariaDB server 12 Fork of MySQL community EdiAon Improved FuncAonaliAes Connect storage engine AddiAonal storage engines AddiAonal plugins Packages for Linux and Windows

13 Other op&ons 13 WebScaleSQL database developed by large web organisaaons based on MySQL community 5.6 No ready available packages MySQL Cluster NDB storage engine (specific use case) MySQL 5.7 Enterprise ediaon? MySQL community ediaon with addiaonal plugins AuthenAcaAon Firewall MySQL enterprise backup and monitor

14 The Percona toolkit 14 Commandline Tools System tasks Dataset management and tooling Percona Server for MySQL, MySQL, MariaDB, Percona Server for MongoDB and MongoDB Open Source Packages for RHEL and Debian based systems hdps:// percona-toolkit

15 Installa&on Of MySQL - HandsOn! 15 Install Percona Server # yum install Percona-Server-server-57 # service mysql start <ONLY ON MASTER> Verify if you can connect # mysql Verify the applicaaon: hdp://localhost:8080/my-movies/ Ping the other nodes [root@node1 ~]# mysql Welcome to the MySQL monitor. Co Your MySQL connection id is 270 Server version: Percona Se Copyright (c) 2000, 2013, Oracle Oracle is a registered trademark affiliates. Other names may be tr owners. node1 mysql>

16 Today s Applica&on - Basic IMDB Interface 16

17 Opera&onal DBA In A Nutshell 17 SeTng Up Todays Environment InstallaAon Logging In MySQL Privileges DiagnosAcs TroubleshooAng Backups ReplicaAon Schema Changes ConfiguraAon OpAmizaAon How doing operaaons feels

18 What are DML statements? 18 Data ManipulaAon Language Example: INSERT into resto_visitor values(5,'julian', highway 5,12); INSERT INTO tbl_name (col1,col2) VALUES(15,col1*2); UPDATE resto_visitor set name='evelyn',age=17 where id=103; Usage It s how you can interact with a Database. Fetch informaaon, update informaaon, remove or add informaaon as in: SELECT, UPDATE, DELETE, ADD

19 What are DDL statements? 19 Data Definition Language Example: CREATE TABLE Accounts ( Account_number Bigint(16), Account_name varchar(255), Amount Bigint(16), PRIMARY KEY (Account_number), UNIQUE(Account_name), ); Usage DDL queries define the structure on which you develop your application. Your structure will also define how the database server searches for information in a table.

20 Other query types? 20 DCL (Data control language) GRANT, REVOKE, TCL (Transaction control language) -- start a new transaction start transaction; -- Debit UPDATE sb_accounts SET balance = balance WHERE account_no = ; -- Credit UPDATE ca_accounts SET balance = balance WHERE account_no = ; -- commit changes commit; Overview:

21 Logging In - HandsOn! 21 # mysql mysql> help mysql> SHOW DATABASES; mysql> show schemas; mysql> USE imdb; mysql> SHOW TABLES; mysql> SELECT * FROM users LIMIT 1; mysql> SELECT * FROM users LIMIT 2 \G mysql> SHOW PROCESSLIST; mysql> exit

22 MySQL CLI - Basic Features - HandsOn! 22 Prompt mysql> prompt Master > PROMPT set to 'Master > Master > prompt mysql> PROMPT set to mysql> Edit mysql> edit Opens up a beautiful little editor: use sakila; select * from city limit 10;

23 MySQL CLI - Basic Features - HandsOn! 23 Tee mysql> tee /tmp/tee.log Logging to file /tmp/tee.log' mysql> select * from sakila.city limit 10; mysql> exit # cat /tmp/tee.log Formatting help help! \g \G Status mysql> status mysql Ver Distrib , for Linux (x86_64) using EditLine wrapper Connection id: Current database: sakila Current user: root@localhost SSL: Not in use

24 MySQL CLI - Basic Features - HandsOn! 24 Status mysql> status; mysql> show engine innodb status \G mysql> show global status; mysql> show processlist;

25 MySQL CLI - Basic Features - HandsOn! 25 Pager mysql> pager grep queries mysql> show engine innodb status \G 0 queries inside InnoDB, 0 queries in queue mysql> pager md5sum PAGER set to 'md5sum' mysql> select * from city limit 10; 449d5bcae6e0e5b19e a7e6-10 rows in set (0.00 sec) mysql> select city_id, city, country_id, last_update FROM city LIMIT 10 ; 449d5bcae6e0e5b19e a7e6-10 rows in set (0.00 sec) mysql> pager Default pager wasn't set, using stdout.

26 MySQL Workbench 26 Visual Tool Database Design & Modeling SQL Development Database Administration Database Migration Editions (Webpage) Community Standard Enterprise

27 MySQL Workbench 27

28 MySQL Workbench 28

29 MySQL Workbench 29

30 Opera&onal DBA In A Nutshell 30 SeTng Up Todays Environment InstallaAon Logging In MySQL Privileges DiagnosAcs TroubleshooAng Backups ReplicaAon Schema Changes ConfiguraAon OpAmizaAon How doing operaaons feels

31 MySQL Privileges 31 Users Grants mysql database Percona Toolkit Default Permissions CreaAng ApplicaAon user Asking the security team for a firewall excepaon

32 Users 32 IdenAfy users based on: user: username host: hostname/ip/network of the client that connects different host, different user, different grants use of wildcards Examples: localhost, localhost app0001, % , app.fq.dn CreaAng A User: >CREATE USER 'dim0'@'app0001'; Drop user: change CREATE into DROP

33 Grants 33 Grant the user some kind of privilege Grant... to: server, database, table, column, trigger, stored procedure, view, index Example: INSERT, SELECT, UPDATE, DELETE SQL Command: > GRANT SELECT ON db.* TO app0001 ; > GRANT INSERT ON *.* TO app0001 ; Revoking privileges: change GRANT into REVOKE

34 Table/Column Level Grants 34 Possible: > GRANT SELECT ON db.table TO app ; > GRANT SELECT (col) ON db.table to app ; Using too many columns might make authenacaaon slower Not commonly used

35 Grants 35 Complete list of grants: CREATE DROP GRANT OPTION LOCK TABLES EVENT ALTER DELETE INDEX INSERT SELECT UPDATE CREATE TEMPORARY TABLES TRIGGER CREATE VIEW SHOW VIEW ALTER ROUTINE CREATE ROUTINE EXECUTE FILE CREATE USER PROCESS PROXY RELOAD REPLICATION CLIENT REPLICATION SLAVE SHOW DATABASES SHUTDOWN SUPER ALL [PRIVILEGES] USAGE

36 Password 36 > SET PASSWORD FOR app0001 = PASSWORD( pass ); > SET PASSWORD FOR app0001 = *196BDEDE2... ; > SELECT PASSWORD('pass')\G PASSWORD('pass'): *196BDEDE2AE4F84CA44C47D54D78478C7E2BD7B7 > CREATE USER app IDENTIFIED BY PASSWORD *196BDEDE2... ; > GRANT SELECT ON db.* TO app IDENTIFIED BY pass ; Mysql >5.7.6 > SET PASSWORD FOR app0001 = pass ;

37 SHOW GRANTS 37 Show grants: manual Check the mysql.user table Check privileges for a certain combo > select user, host from mysql.user; user host root % root root ::1 plmce localhost > show grants for localhost ;

38 SHOW GRANTS 38 Show grants: The Percona Way! pt-show-grants ~]# pt-show-grants -- Grants dumped by pt-show-grants -- Dumped from server Localhost via UNIX socket, MySQL log at :02:56 -- Grants for GRANT USAGE ON *.* TO IDENTIFIED BY PASSWORD '*221D A46BA8556B182ADF26D8D360220F'; GRANT ALL PRIVILEGES ON `imdb`.* TO

39 mysql Database 39 node1 mysql> show tables; Tables_in_mysql columns_priv db event func general_log help_category help_keyword help_relation help_topic host ndb_binlog_index plugin proc procs_priv proxies_priv servers slow_log tables_priv time_zone time_zone_leap_second time_zone_name time_zone_transition time_zone_transition_type user rows in set (0.00 sec)

40 Default user/permissions 40 If using CentOS/RHEL a password is created for root automagically: /var/log/mysql/mysql.log [Note] A temporary password is generated for root@localhost: 8)13mQG5OYl Debian/Ubuntu will prompt during installaaon, if no password included, it will only listen to the local socket. hdps://

41 mysql_secure_installa&on 41 Reworked since 5.7 Password validaaon plugin included You can set a password for root accounts. You can remove root accounts that are accessible from outside the local host. You can remove anonymous-user accounts. You can remove the test database (which by default can be accessed by all users, even anonymous users), and privileges that permit anyone to access databases with names that start with test_.

42 lib]# pt-show-grants -- Grants dumped by pt-show-grants -- Dumped from server Localhost via UNIX socket, MySQL log at :18:28 -- Grants for GRANT USAGE ON *.* TO -- Grants for GRANT USAGE ON *.* TO -- Grants for GRANT USAGE ON *.* TO IDENTIFIED BY PASSWORD '*43DD FBDE5B177730FAD3405BC6DAD7'; MySQL Privileges (1) - HandsOn! 42 Install Percona Toolkit # yum install percona-toolkit # pt-show-grants show all grants on a system (in order) good for version control

43 Default Permissions Are Bad - HandsOn! 43 root@ip :~# mysql_secure_installation Securing the MySQL server deployment. Enter password for user root: VALIDATE PASSWORD PLUGIN Press y Y for Yes, any other key for No: No Using existing password for root. Change the password for root? ((Press y Y for Yes, any other key for No) : Y Remove anonymous users? (Press y Y for Yes, any other key for No) : Y Success. Remove test database and access to it? (Press y Y for Yes, any other key for No) : Y - Dropping test database... Success. - Removing privileges on test database... Success. Reloading the privilege tables will ensure that all changes made so far will take effect immediately. Reload privilege tables now? (Press y Y for Yes, any other key for No) : Y Success. All done!

44 MySQL Privileges (2) - HandsOn! 44 Now we have secure privileges # mysql -u root -p Create a user for my-movies applicaaon mysql> GRANT ALL PRIVILEGES ON imdb.* TO 'mymovies'@'localhost' IDENTIFIED BY 'password'; Change user&password: /var/www/html/my-movies/lib/config.inc.php Verify ApplicaAon: hdp://localhost:8080/my-movies Doublecheck with pt-show-grants # pt-show-grants -u root --ask-pass Create a.my.cnf file [mysql] user= password=

45 Opera&onal DBA In A Nutshell 45 SeTng Up Todays Environment InstallaAon Logging In MySQL Privileges Diagnos&cs Troubleshoo&ng Backups ReplicaAon Schema Changes ConfiguraAon OpAmizaAon How doing operaaons feels

46 Troubleshoo&ng DiagnosAcs OperaAng System MySQL Percona Toolkit Queries PMM AlerAng Troubleshooting a problem that I don t understand 46

47 GeAng ready! - HandsOn! 47 Open up New Terminal window # yum install pmm-client innotop # pmm-admin config --server server-user plmce --server-password plmce # pmm-admin add linux:metrics # pmm-admin add mysql # pmm-admin list Services: mysql:queries linux:metrics mysql:metrics Check PMM: hdp://localhost:8443 (plmce/plmce) Add load # time /root/percona/add_load.py

48 Diagnos&cs OS - HandsOn! 48 OS metrics (memory, cpu...) vmstat free mpstat top ps sar # cat /proc/meminfo

49 Diagnos&cs OS - HandsOn! 49 vmstat: Virtual Memory StaAsAcs [root@master percona]# vmstat 5 procs memory swap io---- -system cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st free [root@master ~]# free -m total used free shared buff/cache available Mem: Swap: mpstat: Processor Statistics [root@master percona]# mpstat 5 Linux el7.x86_64 (master) 04/17/2017 _x86_64_ (1 CPU) 02:09:05 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 02:09:10 PM all

50 Diagnos&cs OS - HandsOn! 50 iostat commonly used: iostat -x -m 1 avg-cpu: %user %nice %system %iowait %steal %idle Device: rrqm/s wrqm/s r/s w/s rmb/s wmb/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda dm dm dm %ual: how many % of Ame at least one request was busy await+svctm: response Ame writes and reads combined

51 Disk Subsystem Sta&s&cs - HandsOn! 51 pt-diskstats similar to iostat more detailed interacave hide inacave disks space for header / for regex? for help reads /proc/diskstats show rd/wr response Ame

52 pt-diskstats - HandsOn! 52 reads /proc/diskstats, shows wr/rd response Ame # pt-diskstats #ts device rd_s rd_avkb rd_mb_s rd_mrg rd_cnc rd_rt wr_s wr_avkb wr_mb_s wr_mrg wr_cnc wr_rt busy in_prg io_s 1.0 dm % % % dm % % %

53 Diagnos&cs MySQL - HandsOn! 53 MySQL: Errorlog: /var/lib/mysql/error.log SHOW GLOBAL STATUS mysqladmin extended-status SHOW ENGINE INNODB STATUS SHOW PROCESSLIST innotop mysqladmin mysqladmin proc stat

54 pt-mext - HandsOn! 54 # pt-mext -r -- mysqladmin -p ext -i 10 -c 3 Binlog_cache_disk_use Binlog_cache_use Bytes_received Bytes_sent Com_begin Com_delete Com_insert Com_replace Com_select Com_set_option Connections Created_tmp_disk_tables Created_tmp_files Created_tmp_tables

55 pt-mext 55 Look at current global behavior of database Query OpAmizaAon necessary? (sorang_%, handler_%, range_%, tmp_table_%) Innodb misbehaving?...

56 Enhanced Slow Log Sta&s&cs - HandsOn! 56 > SET GLOBAL slow_query_log=on; > SET GLOBAL slow_query_log_file='/var/ lib/mysql/slow.log'; > SET GLOBAL long_query_time=0; > SET GLOBAL log_slow_verbosity=full; # cat /root/percona/queries.sql mysql # tail -n 200 /var/log/mysql/slow.log # Time: :58:47 # User@Host: localhost [] # Thread_id: 10 Schema: imdb Last_errno: 0 Killed: 0 # Query_time: Lock_time: Rows_sent: 1 Rows_examined: Rows_affected: 0 Rows_read: 1 # Bytes_sent: 131 Tmp_tables: 1 Tmp_disk_tables: 1 Tmp_table_sizes: # InnoDB_trx_id: 1403 # QC_Hit: No Full_scan: Yes Full_join: No Tmp_table: Yes Tmp_table_on_disk: Yes # Filesort: Yes Filesort_on_disk: Yes Merge_passes: 5 # InnoDB_IO_r_ops: InnoDB_IO_r_bytes: # InnoDB_IO_r_wait: # InnoDB_rec_lock_wait: InnoDB_queue_wait: # InnoDB_pages_distinct: SET timestamp= ; select STRAIGHT_JOIN count(*) as c, person_id FROM cast_info FORCE INDEX(person_id) INNER JOIN

57 pt-query-digest 57 generate reports from slow query log pt-query-digest /var/lib/mysql/slow.log less general log (not so useful) tcpdump Using tcpdump

58 pt-query-digest - HandsOn! 58 slow query log pt-query-digest /var/lib/mysql/slow.log less # Profile # Rank QID Response time Calls R/Call Apdx V/M Item # ==== === =============== ===== ==== ===== ==== # % SELECT table1 table9 table2 table3 table4 # % table7 table6 table8 table SELECT table5 table6 table8 # % SELECT table13 # % SELECT table11 table12 table9 table2 table14 table15 table16 table14 table17 # % table8 table SELECT table8 table4 table14 # % ADMIN CONNECT # % SELECT table19 # % SELECT table13 # % UPDATE table20

59 pt-query-digest - HandsOn! 59 # Overall: 11 total, 8 unique, 0.30 QPS, 0.99x concurrency # Time range: T18:37:07 to T18:37:44 # Attribute total min max avg 95% stddev median # ============ ======= ======= ======= ======= ======= ======= ======= # Exec time 37s 7us 19s 3s 9s 6s 6ms # Lock time 39ms 0 30ms 4ms 4ms 8ms 495us # Rows sent # Rows examine 39.75M M 3.61M 8.86M 6.31M 0.99 # Rows affecte # Bytes sent 4.65k # Merge passes # Tmp tables # Tmp disk tbl # Tmp tbl size 16.00k k 1.45k k 0 # Query size

60 pt-query-digest - HandsOn! 60 Profile Rank Query ID Response time Calls R/Call V/M Item ==== ================== ============= ===== ======= ===== ============== 1 0x212DEB0A % SELECT imdb.cast_info imdb.titl 2 0x8C9D4708FCE85CF % SELECT imdb.movie_info MISC 0xMISC % <6 ITEMS>

61 pt-query-digest - HandsOn! 61 # Query 2: 0.22 QPS, 2.00x concurrency, ID 0x8C9D4708FCE85CF3 at byte 2567 # This item is included in the report because it matches --limit. # Scores: V/M = 0.00 # Time range: T18:37:16 to T18:37:25 # Attribute pct total min max avg 95% stddev median # ============ === ======= ======= ======= ======= ======= ======= ======= # Count 18 2 # Exec time 48 18s 9s 9s 9s 9s 147ms 9s # Lock time 6 3ms 257us 2ms 1ms 2ms 2ms 1ms # Rows sent # Rows examine M 9.30M 9.30M 9.30M 9.30M M # Rows affecte # Bytes sent k # Merge passes # Tmp tables # Tmp disk tbl # Tmp tbl size # Query size

62 pt-query-digest - HandsOn! 62 # Query_time distribution # 1us # 10us # 100us # 1ms # 10ms # 100ms # 1s ################################################################ # 10s+ # Tables # SHOW TABLE STATUS FROM `imdb` LIKE 'movie_info'\g # SHOW CREATE TABLE `imdb`.`movie_info`\g # EXPLAIN /*!50100 PARTITIONS*/ SELECT * FROM imdb.movie_info WHERE movie_id = 93221\G

63 Op&mizing This Applica&on - HandsOn! 63 The problem is that the database is not opamised, indexing is not done properly

64 Op&mizing This Applica&on - HandsOn! 64 Indexes to find the rows matching a WHERE clause quickly reduce amount of data to be examined helps avoid sorang and temp tables random IO get s turned into sequenaal IO Examples SELECT * FROM imdb.title WHERE `id` = 455; SELECT * FROM imdb.movie_info WHERE movie_id = 635; EXPLAIN! We have Some SuggesAons: # less /root/percona/billkarwin-opt.sql

65 Op&mizing This Applica&on - HandsOn! 65 Apply the changes: # cat /root/percona/billkarwin-opt.sql mysql -D imdb Have another look with pt-query-digest # cd /var/lib/mysql/; mv slow.log slow.log_old # killall -HUP mysqld # cat /root/percona/queries.sql mysql # pt-query-digest /var/lib/mysql/slow.log less

66 Aler&ng 66 Things To Alert on: Can Connect to MySQL. ReplicaAon IO Thread is OK ReplicaAon SQL Thread is OK ReplicaAon Seconds Behind Master. ConnecAons InnoDB is enabled. No tables are marked as corrupt. Not the best: tmp tables, raaos, qps Focus on what maders: usually in upper layers

67 Available Tools 67 Many SoluAons Available: MonYog Severalnines ClusterControl MySQL Enterprise Monitor CacA, Munin, Graphite, OpenTSDB, MRTG Nagios, Icinga New Relic Zabbix ZenOSS Percona Monitoring and Management

68 Monitoring and Management: PMM 68

69 Monitoring and Management: PMM 69 Percona Monitoring and Management Open Source MySQL and MongoDB Tooling Metrics Prometheus Grafana QAN: Query Analytics Orchestrator: Topology management

70 Monitoring and Management: PMM 70

71 PMM: Deploying 71 PMM Server Docker images Amazon amis Virtualbox images PMM Client packages available for RHEL/CentOS Ubuntu/Debian

72 PMM: Usage 72 PMM Landing page Add load: /root/percona/add_load.py Links to Query Analytics Metrics Monitor MySQL Replication Topology Manager

73 PMM: Usage 73 Metrics DEMO

74 PMM: Usage 74 Query Analytics DEMO

75 PMM: Handson! 75 Check out Grafana Check out QAN

76 Opera&onal DBA In A Nutshell 76 SeTng Up Todays Environment InstallaAon Logging In MySQL Privileges DiagnosAcs TroubleshooAng Backups ReplicaAon Schema Changes ConfiguraAon OpAmizaAon How doing operaaons feels

77 Backing up your Data - Challenges 77 MySQL is mula-engine Different storage engines call for different strategies MySQL can be quite busy Do we need to shut it down? How much load will we generate? How quickly can we restore?

78 Backing up your Data - Concepts 78 Impact of Backup Hot backup - does not prevent readers or writers from performing operaaons. Warm backup - readers may conanue, but write acavity must be queued unal the backup is completed. Cold backup - system is unavailable during backup window.

79 Backing up your Data - Concepts 79 Type of Backup Physical - stores data files in the naave format. Logical - stores data in a generic representaaon, such as a SQL dump.

80 A few available backup solu&ons 80 Impact (Perf) Warmth Backup Time Restore Time Cold Backup very high cold very fast fast mysqldump high warm medium slow mydumper high warm medium medium SnapshoTng high/medium hot/warm fast fast MySQL Enterprise Backup low/medium warm fast fast XtraBackup low/medium warm fast fast

81 mysqldump 81 Included by default with MySQL Logical backup soluaon - creates SQL file to reproduce data Pro s Easy to use, reliable Cons High impact, long restore Ame

82 mysqldump - HandsOn! 82 Take backup: # mysqldump --all-databases --single-transaction > /var/backup/dump.sql --all-databases for every database (including mysql). --single-transacaon is only safe when all InnoDB. This ensures a hot backup. --master-data=1 (opaonal) records the binary log coordinates. Restore as follows (don t do this): # mysql < dump.sql

83 mysqldump - HandsOn! 83 [root@node1 ~]# vmstat 5 procs memory swap io system cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st WaiAng on MySQLdump to Finish

84 Percona XtraBackup 84 Completely free and open source Physical backup soluaon - Stores data in similar format as the original. Pro s Powerful, scriptable, feature-packed Cons MulAple steps required to restore, complexity

85 Percona XtraBackup - Features 85 Non-blocking Support for MyISAM Compression ParAal Backups Throdling Incremental Backups ExporAng/ImporAng individual tables Streaming Parallel copying Compact Backups (since 2.1!) EncrypAon (since 2.1!)...

86 Percona XtraBackup - Opera&on 86 Two separate "manual" steps are required for taking and storing the backup: (1) - Backing up (2) - Preparing InnoDB files for recovery Restoring it happens in a single step: (3) - Restoring the files

87 Percona XtraBackup - Opera&on (2) 87 Performing and restoring a mixed full backup: Backing up (1): Copy datafiles while recording transacaon log changes FLUSH TABLES WITH READ LOCK; Get binlog posiaon Copy all.myd,.myi,.trg,.trn,... files Stop recording transacaon log changes UNLOCK TABLES; Preparing/Restoring: Prepare backup by applying recorded tlog changes (2) Restoring files to original locaaon (3)

88 Percona XtraBackup - HandsOn! 88 Simple full backup and restore: Install Xtrabackup # yum install percona-xtrabackup-24 (1) # innobackupex /var/backup (2) # innobackupex --apply-log /var/backup/[backupfolder]/ (3) # innobackupex --copy-back /var/backup/[backupfolder]/ # chown -R mysql:mysql /var/lib/mysql

89 Percona XtraBackup - HandsOn! 89 (1) - Full Backup [root@node1 ~]# innobackupex /var/backup/ InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy and Percona Ireland Ltd All Rights Reserved. This software is published under the GNU GENERAL PUBLIC LICENSE Version 2, June :26:20 innobackupex: Starting mysql with options: --unbuffered :26:20 innobackupex: Connected to database with mysql child process (pid= :26:26 innobackupex: Connection to database server closed IMPORTANT: Please check that the backup run completes successfully. At the end of a successful backup run innobackupex prints "completed OK!". innobackupex: Using mysql Ver Distrib , for Linux (i686) using readline 5 innobackupex: Using mysql server version Copyright (c) 2000, 2014, Oracle and/or its a All rights reserved. innobackupex: Created backup directory /var/backup/ _ :26:26 innobackupex: Starting mysql with options: --unbuffered --

90 Percona XtraBackup - HandsOn! 90 Backup Folder Structure [root@node _ ]# ls backup backup-my.cnf cacti ibdata1 imdb mysql performance_schema xtrabackup_binary xtrabackup_binlog_info xtrabackup_checkpoints xtrabackup_logfile (2) - Applying the transacaon logs [root@node _ ]# innobackupex --apply-log /var/backup/ _ / InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy and Percona Ireland Ltd All Rights Reserved. This software is published under the GNU GENERAL PUBLIC LICENSE Version 2, June IMPORTANT: Please check that the apply-log run completes successfully. At the end of a successful apply-log run innobackupex prints "completed OK!" :34:38 innobackupex: Starting ibbackup with command: xtrabackup_55 --defaults-file="/va _ /backup-my.cnf" --defaults-group="mysqld" --prepare --target-dir=/var/lib/mysq tmpdir=/tmp xtrabackup_55 version for Percona Server Linux (i686) (revision id: 521) xtrabackup: cd to /var/lib/mysql/backup/ _ xtrabackup: This target seems to be not prepared yet. xtrabackup: xtrabackup_logfile detected: size= , start_lsn=( ) xtrabackup: Temporary instance for recovery is set as followings.

91 Understanding Backup Requirements 91 Recovery Time ObjecAve - How quickly must I restore? Recovery Point ObjecAve - How much data can I lose? Risks - What failures should this data be protected against?

92 Advanced Backup Topics 92 Incremental Backups HandsOn! Delayed Slaves Using pt-slave-delay to keep a slave behind on its master by a certain amount of Ame. Good help against potenaal disastrous user mistakes Example: # pt-slave-delay --delay 1m --interval 15s

93 Advanced Backup Topics - HandsOn! 93 Incremental Backup Create a full backup # innobackupex /var/backup/ Create incremental backup # innobackupex --incremental /var/backup/inc/ --incremental-basedir=/ var/backup/[backupfolder] Incremental Restore Apply log # innobackupex --apply-log --redo-only /var/backup/[backupfolder] Apply incremental backup to the full backup # innobackupex --apply-log --redo-only /var/backup/[backupfolder] incremental-dir=/var/backup/inc/[incbackupfolder] Finish preparing the backup # innobackupex --apply-log /var/backup/[backupfolder] Restore as usual

94 Advanced Backup Topics 94 Point-In-Time recovery Making use of the binary log mysqlbinlog can be used to pipe commands into mysql Use in combinaaon with incremental backups to restore right up to the point where the disaster occurred.

95 Advanced Backup Topics 95 Checking backup binlog posiaon _ ]# cat xtrabackup_binlog_info node1-bin Checking binlog contents mysql]# mysqlbinlog --start-position= stop-position= node1-bin /*!50530 SET /*!40019 SET /*!50003 SET [...] Applying contents to mysql mysql]# mysqlbinlog --start-position= stop-position= node1-bin mysql

96 Advanced Backup Topics 96 Managing & Retaining your backups Storing offsite in case of disaster Recovery tesang RetenAon policies Monitoring Backup & Restore success rate

97 Opera&onal DBA In A Nutshell 97 SeTng Up Todays Environment InstallaAon Logging In MySQL Privileges DiagnosAcs TroubleshooAng Backups Replica&on Schema Changes ConfiguraAon OpAmizaAon How doing operaaons feels

98 MySQL Replica&on 98 ReplicaAon? Binary Logs SeTng Up ReplicaAon Commands Breaking & Fixing ReplicaAon Inconsistencies

99 What is Replica&on? 99 ReplicaAon enables data from one MySQL database server (the master) to be replicated to one or more MySQL database servers (the slaves) (Source: hdp://dev.mysql.com/doc/refman/5.6/en/replicaaon.html)

100 Replica&on 100 Happens at MySQL level, not Storage Engine Level Asynchronous! (Semi-sync available in 5.5) A server can have mulaple masters (Since MySQL 5.7) IO Thread: Fetches from master SQL Thread: Executes on slave Single Threaded ExecuAon (UnAl 5.6)

101 Example Asynchronous Replica&on 101

102 Asynchronous Replica&on how does it work? 102

103 Other types of replica&on 103 Semi-synchronous replicaaon Galera replicaaon (MySQL with Galera, PXC, MariaDB Cluster) MySQL Cluster MySQL group replicaaon MySQL InnoDB cluster

104 Semi Synchronous Replica&on (S&ll the same) 104

105 Semi Synchronous Replica&on How? 105

106 Group Replica&on 106

107 Galera (Percona XtraDB Cluster, MariaDB Cluster) 107

108 Binary Logs 108 BINLOG BINLOG BINLOG BINLOG BINLOG. BINLOG.index BINLOG.index BINLOG BINLOG BINLOG

109 Binary Logs 109 Set of files Contains all writes and schema changes IncremenAng numbers (000001,000002,000003,...) 2 Formats: Statement Based (SBR) Row Based (RBR, since MySQL 5.1) Prior to MySQL 5.7.7, statement-based logging format was the default. In MySQL and later, row-based logging format is the default. Binary files (extract with MySQLbinlog)

110 Binary Log - Formats (binlog_format) 110 Statement Based ReplicaAon (SBR): Writes statements to binary logs, slave executes the statement eg: DELETE FROM wp_opaons WHERE opaon_name = '_transient_doing_cron'

111 Binary Log - Formats (binlog_format) 111 Row Based ReplicaAon (RBR, since 5.1): Write row changes (larger binlogs) Does not need to parse/execute queries, just make the changes necessary Mixed: CombinaAon of both: defaults to SBR, use RBR when necessary

112 Looking at Binary Log Contents 112 mysqlbinlog SHOW BINLOG EVENTS

113 Example SBR 113 > SHOW GLOBAL VARIABLES LIKE 'binlog_format'; Variable_name Value binlog_format STATEMENT row in set (0.00 sec) > CREATE DATABASE replication; Query OK, 1 row affected (0.14 sec) > use replication Database changed > CREATE TABLE repl (a int) ENGINE=innodb; Query OK, 0 rows affected (0.25 sec) > INSERT INTO repl VALUES (1); Query OK, 1 row affected (0.14 sec)

114 Example SBR - mysqlbinlog # mysqlbinlog mysql-bin # at 106 # :19:13 server id 9999 end_log_pos 203 Query thread_id=11 CREATE DATABASE replication /*!*/; # at 203 # :19:32 server id 9999 end_log_pos 312 Query thread_id=11 exec_time=1 error_code=0 use replication/*!*/; SET TIMESTAMP= /*!*/; CREATE TABLE repl (a INT) ENGINE=innodb /*!*/; # at 312 # :19:55 server id 9999 end_log_pos 387 Query thread_id=11 exec_time=0 error_code=0 SET TIMESTAMP= /*!*/; BEGIN 114

115 > SHOW BINLOG EVENTS FROM 106\G *************************** 1. row *************************** Log_name: mysql-bin Example Pos: 106 SBR - SHOW BINLOG EVENTS Event_type: Query Server_id: 1 End_log_pos: 203 Info: CREATE DATABASE replication *************************** 2. row *************************** Log_name: mysql-bin Pos: 203 Event_type: Query Server_id: 1 End_log_pos: 312 Info: use `replication`; CREATE TABLE repl (a INT) ENGINE=innodb *************************** 3. row *************************** Log_name: mysql-bin Pos: 312 Event_type: Query Server_id: 1 End_log_pos: 387 Info: BEGIN *************************** 4. row *************************** 115

116 > INSERT INTO repl VALUES (1); Query OK, 1 row affected (0.14 sec) Example RBR 116 > SHOW GLOBAL VARIABLES LIKE 'binlog_format'; Variable_name Value binlog_format ROW row in set (0.00 sec) > CREATE DATABASE replication; Query OK, 1 row affected (0.14 sec) > use replication Database changed > CREATE TABLE repl (a int) ENGINE=innodb; Query OK, 0 rows affected (0.25 sec)

117 /*!*/; # at 703 # :55:02 server id 1 end_log_pos 812 Query thread_id=11 exec_time=0 error_code=0 use Example replication/*!*/; RBR - mysqlbinlog SET TIMESTAMP= /*!*/; CREATE TABLE repl (a int) ENGINE=innodb /*!*/; # at # at 937 # :55:06 server id 1 end_log_pos 937 Table_map: `replication`.`repl` mapped to number 17 # :55:06 server id 1 end_log_pos 971 Write_rows: table id 17 flags: STMT_END_F BINLOG ' SgsHTxMBAAAAMgAAAKkDAAAAABEAAAAAAAEAC3JlcGxpY2F0aW9uAARyZXBsAAEDAA E= SgsHTxcBAAAAIgAAAMsDAAAAABEAAAAAAAEAAf/+AQAAAA== '/*!*/; # at 971 # :55:06 server id 1 end_log_pos 998 Xid = 34 COMMIT/*!*/; 117

118 # mysqlbinlog mysql-bin verbose --verbose... Example RBR - # at 937 # mysqlbinlog 15:55:06 server --verbose id 1 end_log_pos 937 Table_map: `replication`.`repl` mapped to number 17 # :55:06 server id 1 end_log_pos 971 Write_rows: table id 17 flags: STMT_END_F BINLOG ' SgsHTxMBAAAAMgAAAKkDAAAAABEAAAAAAAEAC3JlcGxpY2F0aW9uAARyZXBsAAEDAA E= SgsHTxcBAAAAIgAAAMsDAAAAABEAAAAAAAEAAf/+AQAAAA== '/*!*/; ### INSERT INTO replication.repl ### SET /* INT meta=0 nullable=1 is_null=0 */ # at

119 Pos: 703 Event_type: Query Server_id: 1 nd_log_pos: Example 812RBR - SHOW BINLOG EVENTS Info: use `replication`; CREATE TABLE repl (a int) NGINE=innodb.. ************************** 4. row *************************** Log_name: mysql-bin Pos: 887 Event_type: Table_map Server_id: 1 nd_log_pos: 937 Info: table_id: 17 (replication.repl) ************************** 5. row *************************** Log_name: mysql-bin Pos: 937 Event_type: Write_rows Server_id: 1 nd_log_pos: 971 Info: table_id: 17 flags: STMT_END_F ************************** 6. row *************************** Log_name: mysql-bin Pos:

120 SeAng up Replica&on 120 Install the slave Change master/slave MySQL configuraaon Prerequisites Configure ReplicaAon Start ReplicaAon/Check Status

121 Change master/slave Configura&on - HandsOn! 121 On both the master and slave, make the following changes to /etc/my.cnf: server-id=<unique number> log-slave-updates log-bin=log-bin After making the changes, restart the master [root@node1 ~]# service mysql restart For future steps, drop the IMDB database node1 mysql> DROP DATABASE IMDB;

122 Why server-id? 122 Avoid events to be wriden more than once IdenAfier for your system to keep track of replicaaon

123 Prerequisites (1) - HandsOn! 123 Please perform the following steps on the master [root@node1 ~]# rm -rf /var/backup/* [root@node1 ~]# innobackupex /var/backup/ [root@node1 ~]# innobackupex apply-log /var/ backup/[backupfolder] [root@node1 ~]# ln -s /var/backup/[backupfolder] / var/backup/last_full

124 Prerequisites (2) - HandsOn! 124 No way to easily create slave with 1 command It s required to Create/Restore consistent backup node2# rm -rf /var/lib/mysql/* node2# scp -r root@ :/var/backup/last_full/* /var/ lib/mysql/ node2# chown -R mysql:mysql /var/lib/mysql/ node2# yum install Percona-Server-server-57 node2# service mysql start node2# nano ~/.my.cnf

125 Configure Replica&on - HandsOn! 125 On Master, add permissions: node1 mysql> GRANT REPLICATION SLAVE ON *.* TO % IDENTIFIED BY slaveme ; Get Binlog file/posiaon from backup on the slave: node2# cat /var/lib/mysql/xtrabackup_binlog_info Example output Filename log_position log-bin On Slave, configure replicaaon: node2 mysql> CHANGE MASTER TO MASTER_HOST= , MASTER_USER= repl, MASTER_PASSWORD= slaveme,master_log_file=[logfile],master_log _POS=[position];

126 Start Replica&on mysql> show master status \G *************************** 1. row *************************** File: log-bin Position: Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 1 row in set (0.00 sec) Binlog acave

127 Start Replica&on node2 mysql> show slave status\g *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: log-bin Read_Master_Log_Pos: Relay_Log_File: slave-relay-bin Relay_Log_Pos: Relay_Master_Log_File: log-bin Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: Relay_Log_Space: Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Slave running? What is it execuang? How Far Is Slave Behind?

128 Commands 128 SQL Commands AdministraAve Commands DiagnosAcs Commands Shell Commands mysqlbinlog

129 Administra&ve Commands 129 Rotate binary log: FLUSH BINARY LOGS Rotate relay log: FLUSH RELAY LOGS Remove binary logs: PURGE MASTER LOGS TO mysql-bin ; PURGE MASTER LOGS BEFORE :00:00 ; Remove all binary logs: RESET MASTER Remove slave configuraaon and files: RESET SLAVE

130 Diagnos&cs Commands 130 On Master SHOW MASTER STATUS SHOW PROCESSLIST SHOW SLAVE HOSTS SHOW BINLOG EVENTS On Slave SHOW SLAVE STATUS SHOW PROCESSLIST

131 On Master:SHOW MASTER STATUS - HandsOn! 131 Current binary log file and posiaon: mysql> show master status \G File: log-bin Position: Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 1 row in set (0.00 sec)

132 On Master: SHOW PROCESSLIST - HandsOn! 132 Find Connected Slaves using SHOW PROCESSLIST: master> SHOW PROCESSLIST\G *************************** 1. row *************************** Id: 1171 User: repl Host: :48590 db: NULL Command: Binlog Dump Time: 3720 State: Master has sent all binlog to slave; waiting for more updates Info: NULL Rows_sent: 0 Rows_examined: 0

133 On Slave: SHOW PROCESSLIST - HandsOn! 133 Slave Thread Status: mysql> show processlist \G *************************** 1. row *************************** Id: 6 User: system user Host: db: NULL Command: Connect Time: 3782 State: Waiting for master to send event Info: NULL Rows_sent: 0 Rows_examined: 0 *************************** 2. row *************************** Id: 7 User: system user Host: db: NULL Command: Connect Time: 3317 State: Slave has read all relay log; waiting for more updates Info: NULL Rows_sent: 0 Rows_examined: 0

134 Other Common Configura&on Op&ons 134 Don t start replicaaon at start: skip_slave_start Put relay log events in it s own binary log: log_slave_updates Disallow wriang on slaves: read_only AutomaAcally remove binlogs: expire_logs_days Change binlog format: binlog_format STANDARD ROW in 5.7

135 Breaking & Fixing Replica&on - HandsOn! 135 Create table on Slave First, then Master: CREATE TABLE sakila.plmce ( id int auto_increment primary key, name varchar(20) ); Check Slave Status: node2> SHOW SLAVE STATUS\G ReplicaAon broke :(

136 Breaking & Fixing Replica&on - HandsOn! 136 ReplicaAon is Broken. Here we understand what went wrong, so let s just skip this record! (ONLY DO THIS WHEN YOU ARE SURE) node2 mysql> STOP SLAVE; node2 mysql> SET GLOBAL sql_slave_skip_counter=1 node2 mysql> START SLAVE; node2 mysql> SHOW SLAVE STATUS\G Listening to someone saying that he skipped a record and now replicaaon works fine

137 Inconsistencies - HandsOn! 137 Let s delete some data from the slave: mysql> DELETE FROM sakila.film_actor LIMIT 5; On the Master: node1 mysql> GRANT ALL PRIVILEGES ON *.* to % identified by password ; # pt-table-checksum h=localhost,u=pttc,p=password --no-checkbinlog-format Amer Checksumming, let s verify: # pt-table-checksum h=localhost,u=pttc, p=password --no-check-binlog-format --replicate-check-only Almost ran an UPDATE without WHERE clause

138 Opera&onal DBA In A Nutshell 138 SeTng Up Todays Environment InstallaAon Logging In MySQL Privileges DiagnosAcs TroubleshooAng Backups ReplicaAon Schema Changes ConfiguraAon OpAmizaAon How doing operaaons feels

139 Schema Changes - Techniques 139 Why? Achieving schema perfecaon can be painful Performance opamizaaons MigraAons MySQL 5.5 vs 5.6 and 5.7 MySQL 5.6/5.7 supports online schema changes Prior to MySQL 5.6 use pt-online-schema-change

140 Online Schema Changes - HandsOn! 140 Connect to your MySQL instance Open second terminal node1> insert into sakila.actor (first_name, last_name) values ('LeT','watDim0ooootjeen'); alter table on your system if you want the changes locally # node1> ALTER TABLE sakila.actor ADD INDEX new_idx (first_name, last_name);

141 Online Schema Changes 141 In 5.5, use pt-online-schema-change if you want to replicate changes # time pt-online-schema-change \ --execute --alter \ DROP INDEX new_idx" \ D=sakila,t=actor

142 Online Schema Changes - HandsOn! 142 New in MySQL sweet hot alter tables mysql> ALTER TABLE sakila.actor DROP INDEX new_idx, LOCK=NONE; Query OK, 0 rows affected (0.02 sec) Records: 0 Duplicates: 0 Warnings: 0 However: hdps://dev.mysql.com/doc/refman/5.6/en/innodb-create-indexoverview.html#innodb-online-ddl-summary-grid

143 Opera&onal DBA In A Nutshell 143 SeTng Up Todays Environment InstallaAon Logging In MySQL Privileges DiagnosAcs TroubleshooAng Backups ReplicaAon Schema Changes Configura&on Op&miza&on How doing operaaons feels

144 Op&miza&on - Check Variables 144 Get variables mysql> show global variables like innodb_buffer_pool_size'; mysql> mysql> show global variables like innodb_% ; Set mysql> mysql> set global table_open_cache = 3000; Make the changes persistent my.cnf

145 Op&miza&on - Things Not To Op&mize 145 innodb_io_capacity, innodb_io_capacity_max: don t oversize sort_buffer_size,join_buffer_size, read_buffer_size, read_rnd_buffer_size: leave default globally innodb_thread_concurrency: Default is fine for most environments query_cache_type, query_cache_size: disable or do not oversize (+64MB) innodb_flush_method=o_direct and ext3

146 Op&miza&on - Things To Op&mize 146 innodb_log_file_size: Increase InnoDB Transaction Logs innodb_buffer_pool_size innodb_io_capacity(_max) innodb-flush-method=o_direct innodb_flush_log_at_trx_commit increases performance DECREASES reliability! (no more ACID compliant)

147 Op&miza&on - Things To Op&mize 147 Read more at mysql-5-7-performance-tuning-immediately-afterinstallation/

148 Keep in mind the following op&misa&on process 148 Low Hanging Fruit MySQL Configuration OS settings Indexes Caching Medium Level InfraStructure MySQL Version Hard Changes Major schema changes ApplicaAon architecture changes

149 Want to learn more? 149 Some good resources to help you along: Blogs Planet MySQL: planet.mysql.com MySQL Performance Blog: Recorded Webinars

150 Opera&onal DBA In A Nutshell - Q&A 150 SeAng Up Todays Environment Installa&on Logging In MySQL Privileges Diagnos&cs Troubleshoo&ng Monitoring Backups Replica&on Schema Changes Configura&on Op&miza&on How doing operaaons now feels

151 Want to learn more? 151 Some good resources to help you along: Books High Performance MySQL, 3rd EdiAon (Baron Schwartz, Peter Zaitsev, Vadim Tkachenko) EffecAve MySQL series (Ronald Bradford) Instant InnoDB (Mad Reid)