Operational DBA In a Nutshell - HandsOn Reference Guide

Transcription

1 1/12 Operational DBA In a Nutshell - HandsOn Reference Guide Contents 1 Operational DBA In a Nutshell 2 2 Installation of MySQL Setting Up Our VM Installation of MySQL Logging In MySQL Privileges MySQL Privileges (1) Default Permissions Are Bad MySQL Privileges (2) Diagnostics, Troubleshooting & Monitoring Add Some Load First Diagnostics pt-mext pt-diskstats Enhanced Slow Log Statistics pt-query-digest Trending Optimizing This Application Backups mysqldump XtraBackup Advanced Backup Topics Replication Change Master/Slave Configuration Prerequisites (1) Prerequisites (2) Configure Replication Start Replication Administrative Commands

2 1 Operational DBA In a Nutshell 2/ Diagnostics Commands Breaking & Fixing Replication Inconsistencies Schema Changes Online Schema Changes Operational DBA In a Nutshell This guide serves the purpose of allowing you to follow along and catch up in case you are unable to finish the assigned hands-on tasks during the available time. It follows the same structure as the tutorial, and includes the slide numbers and subjects for easy reference. You can also use it to repeat the exercises on the virtual machines, after this tutorial is finished. Note If you experience trouble with one of these steps, don t hesitate to ask for help at any time. 2 Installation of MySQL 2.1 Setting Up Our VM Logging in (using the password vagrant ): # ssh -p 2221 root@localhost Browsing to the test page to confirm connectivity: Installation of MySQL Download and install Percona Server from the repository: # yum install Percona-Server-server-56 # service mysql start 2.3 Logging In Log in and run some exploratory commands:

3 3 MySQL Privileges 3/12 # mysql mysql> SHOW DATABASES; mysql> USE imdb; mysql> SHOW TABLES; mysql> SELECT * FROM users LIMIT 1; mysql> SHOW PROCESSLIST; After you are finished: mysql> exit 3 MySQL Privileges 3.1 MySQL Privileges (1) Install the Percona Toolkit tools, and list up your database grants: # yum install percona-toolkit # pt-show-grants 3.2 Default Permissions Are Bad Walk through the security wizard: # mysql_secure_installation Make sure to change the password to root. 3.3 MySQL Privileges (2) Create a user for our web application: # mysql -u root -p Enter password: mysql> GRANT ALL PRIVILEGES ON imdb.* TO 'mymovies'@'localhost' IDENTIFIED BY 'password'; mysql> exit Change the configuration for our web application to match the username and password you just chose. Note You can use whichever editor you are comfortable with during this tutorial. We ll be using nano in this example.

4 4 Diagnostics, Troubleshooting Monitoring 4/12 # nano /var/www/html/my-movies/lib/config.inc.php Change the following lines: define('mysql_user', 'root'); define('mysql_password', ''); to: define('mysql_user', mymovies ); define('mysql_password', password ); Verify the application is running correctly by surfing to: # Confirm that our security has tightened by rerunning: # pt-show-grants -u root --ask-pass Edit the.my.cnf file in your homefolder for easy login during the exercises: # nano ~/.my.cnf It should contain the following lines: [client] user=root password=root 4 Diagnostics, Troubleshooting & Monitoring 4.1 Add Some Load First Open up a separate terminal, log into node1 and run the following script: # /home/percona/add_load.py 4.2 Diagnostics There are various places you can check to gain some understanding as to what is going on. Here are some examples:

5 4.3 pt-mext 5/12 # tail /var/lib/mysql/node1.err # mysqladmin extended-status In the MySQL client: mysql> SHOW GLOBAL STATUS; mysql> SHOW ENGINE INNODB STATUS; mysql> SHOW PROCESSLIST; Some tools to display OS level metrics: # vmstat # iostat # mpstat # top # ps # cat /proc/meminfo innotop: # innotop 4.3 pt-mext Using pt-mext to capture changes over time: # pt-mext -r -- mysqladmin ext -i 10 -c pt-diskstats Using pt-diskstats to get a sense of the storage activity: # pt-diskstats --devices-regex sdb1 4.5 Enhanced Slow Log Statistics Enable Slow Query Logging in MySQL: mysql> SET GLOBAL slow_query_log=on; mysql> SET GLOBAL slow_query_log_file='/var/lib/mysql/slow.log'; mysql> SET GLOBAL long_query_time=0; mysql> SET GLOBAL log_slow_verbosity=full; Look at the Slow Query Logfile:

6 4.6 pt-query-digest 6/12 # tail -n 200 /var/lib/mysql/slow.log 4.6 pt-query-digest Analyze the Slow Query Logfile with pt-query-digest: # pt-query-digest /var/lib/mysql/slow.log less Store the pt-query-digest reports in the database: # pt-query-digest --review localhost /var/lib/mysql/slow.log Using filters: # pt-query-digest --filter '$event->{fingerprint} =~ m/^select/' /var/lib/mysql/slow.log 4.7 Trending Use a browser to go to the following address (username: admin, password: plmce): Optimizing This Application Take a look at the change recommendations and apply them: # less /home/percona/.data/billkarwin-opt.sql # cat /home/percona/.data/billkarwin-opt.sql mysql -D imdb Note The above may take a while to complete. If you want to take a look at the difference: # cd /var/lib/mysql/; mv slow.log slow.log_old # killall -HUP mysqld # /home/percona/add_load.py -- in a separate terminal

7 5 Backups 7/12 # pt-query-digest /var/lib/mysql/slow.log less 5 Backups 5.1 mysqldump Take a consistent logical backup: # mysqldump --all-databases --single-transaction > /var/backup/dump.sql While the above command is executing, take a look at what's happening: # vmstat XtraBackup Install XtraBackup: # yum install percona-xtrabackup Take a backup to the specified location: # innobackupex /var/backup Apply the log files to the backup to prepare it for restoration: # innobackupex --apply-log /var/backup/[backupfolder] To restore, execute the following commands: # innobackupex --copy-back /var/backup/[backupfolder]/ # chown -R mysql:mysql /var/lib/mysql 5.3 Advanced Backup Topics Create an incremental backup based on the previously created full backup: # innobackupex --incremental /var/backup/inc/ --incremental-basedir=/var/backup/[backupfolder] Apply the incremental backups to the original full backup:

8 6 Replication 8/12 # innobackupex --apply-log --redo-only /var/backup/[backupfolder] # innobackupex --apply-log --redo-only /var/backup/[backupfolder] --incremental-basedir=/var/backup/inc/[incbackupfolder] # innobackupex --apply-log /var/backup/[backupfolder] 6 Replication 6.1 Change Master/Slave Configuration Edit the configuration files on the Master and Slave to prepare it for replication. On both the master and slave, edit /etc/my.cnf to add the following: gtid_mode=on log-slave-updates enforce-gtid-consistency After updating the configuration, restart the master: [root@node1 ~]# service mysql restart node1 mysql> DROP DATABASE IMDB; 6.2 Prerequisites (1) We need to take a full backup to populate the slave instance with data. Prepare it as follows: [root@node1 ~]# rm -rf /var/backup/* [root@node1 ~]# innobackupex /var/backup/ [root@node1 ~]# innobackupex --apply-log /var/backup/[backupfolder] [root@node1 ~]# ln -s /var/backup/[backupfolder] /var/backup/last_full 6.3 Prerequisites (2) [root@node2 ~]# rm -rf /var/lib/mysql/* [root@node2 ~]# scp -r root@ :/var/backup/last_full/* /var/lib/mysql/ [root@node2 ~]# chown -R mysql:mysql /var/lib/mysql [root@node2 ~]# yum install Percona-Server-server-56 [root@node2 ~]# service mysql start [root@node2 ~]# nano ~/.my.cnf 6.4 Configure Replication Add the necessary permissions to the master and retrieve the binlog position:

9 6.5 Start Replication 9/12 node1 mysql> GRANT REPLICATION SLAVE ON *.* TO % IDENTIFIED BY slaveme ; node1 mysql> exit On the slave, configure replication: # cat /var/lib/mysql/xtrabackup_binlog_info node2 mysql> SET GLOBAL gtid_purged= [GTID] ; node2 mysql> CHANGE MASTER TO master_host = ' ', master_user = 'repl', master_password = 'slaveme', MASTER_AUTO_POSITION=1; 6.5 Start Replication Start the slave: node2 mysql> START SLAVE; node2 mysql> SHOW SLAVE STATUS\G Edit some rows and verify the master's status: node1 mysql> DELETE FROM rental LIMIT 10; node1 mysql> SHOW MASTER STATUS\G Verify that the same GTID is being displayed on the slave: node2 mysql> SHOW SLAVE STATUS\G 6.6 Administrative Commands Some administrative commands for archiving/removing the binary logs: node1 mysql> PURGE MASTER LOGS TO 'node1-bin '; node1 mysql> PURGE MASTER LOGS BEFORE ' :00:00'; node1 mysql> RESET MASTER; Removing the Slave configuration and files from node2: node2 mysql> RESET SLAVE; 6.7 Diagnostics Commands On the Master:

10 6.8 Breaking Fixing Replication 10/12 node1 mysql> SHOW MASTER STATUS\G node1 mysql> SHOW PROCESSLIST\G node1 mysql> SHOW SLAVE HOSTS\G node1 mysql> SHOW BINLOG EVENTS\G On the Slave: node2 mysql> SHOW SLAVE STATUS\G node2 mysql> SHOW PROCESSLIST\G 6.8 Breaking & Fixing Replication Create an inconsistency by creating a table, first on the slave, then on the master: node2 mysql> CREATE TABLE sakila.pluk ( id int auto_increment primary key, name varchar(20) ); node1 mysql> CREATE TABLE sakila.pluk ( id int auto_increment primary key, name varchar(20) ); Verify replication status on the slave: node2 mysql> SHOW SLAVE STATUS\G "Fix" replication by skipping the record on the Slave. To know which record to skip, we need to look at the slave status to retrieve the correct GTID by comparing Retrieved_Gtid_Set with Executed_Gtid_Set: node2 mysql> STOP SLAVE; node2 mysql> SET GTID_NEXT="[GTIDPOSITION]"; node2 mysql> BEGIN; COMMIT; node2 mysql> SET GTID_NEXT="AUTOMATIC"; node2 mysql> START SLAVE; node2 mysql> SHOW SLAVE STATUS\G 6.9 Inconsistencies Delete some data from the slave to make it inconsistent with the master: node2 mysql> DELETE FROM sakila.rental LIMIT 10; To be able to use pt-table-checksum, create the following user on the master:

11 7 Schema Changes 11/12 node1 mysql> grant all privileges on *.* to identified by 'password'; To track down the inconsistencies on the master, use pt-table-checksum on node1: # pt-table-checksum h=localhost,u=pttc,p=password --no-check-binlog-format # pt-table-checksum h=localhost,u=pttc,p=password --no-check-binlog-format --replicate-check-only 7 Schema Changes 7.1 Online Schema Changes Insert a value into the actor table on node1 and add an index: node1 mysql> use sakila; node1 mysql> INSERT INTO sakila.actor (first_name, last_name) VALUES ('let','watdim0ooootjeen'); node1 mysql> ALTER TABLE sakila.actor ADD INDEX new_idx (first_name, last_name); Using hot alter tables in MySQL 5.6: node1 mysql> ALTER TABLE sakila.actor DROP INDEX new_idx, LOCK=NONE;