Maximizing SQL reviews with pt-query-digest

Transcription

1 PALOMINODB OPERATIONAL EXCELLENCE FOR DATABASES Maximizing SQL reviews with pt-query-digest Mark Filipi

2 What is pt-query-digest Analyzes MySQL queries from slow, general and binary log files. Processlist, tcpdump and more also available.

3 What is pt-query-digest Fingerprints each query pattern, calculating statistics for rows examine, returned, etc. Prints report containing important query information

4 Why use pt-query-digest ) Easily locate slowest queries 2) See most executed queries 3) Find bottlenecks for tuning opportunities.

5 How to use pt-query-digest pt-query-digest slow.log. > slow.txt slow.log.: % 39:5 remain slow.log.: 2% 38:5 remain slow.log.: 3% 37:26 remain...

6 Alternate usage pt-query-digest slow.log. --type binlog --type genlog --type tcpdump tcpdump -s x -nn -q -tttt -i any -c port 336 \ > mysql.tcp.txt :program:`pt-query-digest` --type tcpdump mysql.tcp.txt

7 Default output 228.2s user time, 9.9s system time, 765.3M rss, M vsz Current date: Wed Apr 24 :42:5 23 Hostname: XXXXXXXXXX Files: slow.log. Overall: 5.5M total, 9 unique, QPS,.78x concurrency Time range: :2:2 to :2:2 Attribute total min max avg 95% stddev median ============ ======= ======= ======= ======= ======= ======= ======= Exec time 6764s 46us 59s 2ms 85us 555ms 38us Lock time 263s 52ms 47us 57us 36us 44us Rows sent.35m k Rows examine 5.9G k.3k k Rows affecte 6.5k Rows read 5.89G k.2k k Bytes sent 8.37G 6.7M.6k.53k 3.7k k Tmp tables

8 Default output - cont Profile Rank Query ID Response time Calls R/Call Apdx V/M Item ==== ================== ================ ======= ======= ==== ===== ==== xe944a3e3d5bef % SELECT user_metrics 2 x925e93282e2b % SELECT devices 3 x54c99f3a94f6d5a % SELECT user_metrics 4 x4db3ebda663c % SELECT levels 5 xd3e83fa246fccb % SELECT users 6 xb5d9ee7c2c24b % SELECT devices 7 x54d33962e5247a % SELECT given_items 8 xd9bed65ddcef98b % SELECT users app_friends weekly_stats 9 x44ccdf3322b % SELECT unlock_dialogs unlock_dialog_translations x8acf8e33226bcd % SELECT users xe82376cb7e268c MISC xmisc % 2.6% UPDATE user_metrics.72 NS. <6 ITEMS>

9 Default output - cont Query :.4 QPS,.55x concurrency, ID xe944a3e3d5bef59 at byte This item is included in the report because it matches --limit. Scores: Apdex =.27 [.], V/M = 2.62 Query_time sparkline: ^_ Time range: :5:3 to 6:5:23 Attribute pct total min max avg 95% stddev median ============ === ======= ======= ======= ======= ======= ======= ======= Count 7 Exec time s s 58s 4s 57s 7s 4s Lock time 43ms 4us 2us 25us 28us 3us 23us Rows sent.66k Rows examine.66k Rows affecte Rows read 2.44k Bytes sent 3.28M.97k.98k.98k.96k..96k Tmp tables Tmp disk tbl Tmp tbl size Query size 57.45k

10 Query output Query :.4 QPS,.55x concurrency, ID xe944a3e3d5bef59 at byte This item is included in the report because it matches --limit. Scores: Apdex =.27 [.], V/M = 2.62 Query_time sparkline: ^_ Time range: :5:3 to 6:5:23 Attribute pct total min max avg 95% stddev median ============ === ======= ======= ======= ======= ======= ======= ======= Count 7 Exec time s s 58s 4s 57s 7s 4s Lock time 43ms 4us 2us 25us 28us 3us 23us Rows sent.66k Rows examine.66k Rows affecte Rows read 2.44k Bytes sent 3.28M.97k.98k.98k.96k..96k Tmp tables Tmp disk tbl Tmp tbl size Query size 57.45k

11 Query output - cont String: Databases slow_db Hosts InnoDB trxid F8696E3EF (/%), F8696E4 (/%) more Last errno Users slow_user Query_time distribution us us us ms ms ms s s+ Tables SHOW TABLE STATUS FROM `slow_db` LIKE 'user_metrics'\g SHOW CREATE TABLE `slow_db`.`user_metrics`\g EXPLAIN /*!5 PARTITIONS*/ SELECT `user_metrics`.* FROM `user_metrics` WHERE `user_metrics`.`user_id` = 2745 LIMIT \G

12 Explain output mysql> explain -> SELECT `user_metrics`.* FROM `user_metrics` WHERE `user_metrics`.`user_id` = 2745 LIMIT ; id select_type table type possible_keys key key_len ref rows Extra SIMPLE user_metrics const index_user_metrics_on_user_id index_user_metrics_on_user_id 4 const row in set (. sec)

13 Query output - 2 Query 2: 3.3 QPS, 5.93x concurrency, ID x925e93282e2b at byte This item is included in the report because it matches --limit. Scores: Apdex =.5 [.], V/M =.2 Query_time sparkline: _^_ Time range: :34:5 to 9:27:2 Attribute pct total min max avg 95% stddev median ============ === ======= ======= ======= ======= ======= ======= ======= Count 53 Exec time s 856ms 4s 2s 2s 59ms 2s Lock time 38ms 4us 4ms 3us 44us 62us 23us Rows sent Rows examine G k k k k k Rows affecte Rows read G k k k k k Bytes sent.3m k Tmp tables Tmp disk tbl Tmp tbl size Query size 88.73k

14 Query output 2 - cont String: Databases slow_db Hosts InnoDB trxid F87B2DD (/%), F87B2DD4 (/%) more Last errno Users slow_db Query_time distribution us us us ms ms ms s s+ Tables SHOW TABLE STATUS FROM `slow_db` LIKE 'devices'\g SHOW CREATE TABLE `slow_db`.`devices`\g EXPLAIN /*!5 PARTITIONS*/ SELECT `devices`.* FROM `devices` WHERE `devices`.`current_user_id` = \G

15 Explain output - 2 mysql> EXPLAIN /*!5 PARTITIONS*/ -> SELECT `devices`.* FROM `devices` WHERE `devices`.`current_user_id` = \G ***************************. row *************************** id: select_type: SIMPLE table: devices partitions: NULL type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: Extra: Using where row in set (. sec)

16 Explain output - 2 mmysql> show create table devices\g ***************************. row *************************** Table: devices Create Table: CREATE TABLE `devices` ( `id` int() NOT NULL AUTO_INCREMENT, `os` enum('ios') NOT NULL, `uuid` varchar(255) NOT NULL, `current_user_id` int() NOT NULL, `platform_id` int() DEFAULT NULL, `ios_device` varchar(255) DEFAULT NULL, `ios_version` varchar(255) DEFAULT NULL, `merged_at` datetime DEFAULT NULL, `created_at` datetime DEFAULT NULL, `updated_at` datetime DEFAULT NULL, `apn_token` varchar(255) DEFAULT NULL, `apn_token_updated_at` datetime DEFAULT NULL, PRIMARY KEY (`id`), UNIQUE KEY `index_devices_on_os_uuid` (`os`,`uuid`), KEY `index_devices_on_apn_token` (`apn_token`) ) ENGINE=InnoDB AUTO_INCREMENT=55873 DEFAULT CHARSET=utf8 row in set (. sec)

17 Query output 3 - compare Query 9: QPS,.2x concurrency, ID x44ccdf3322b46 at byte Time range: :2:2 to 2:3:4 Attribute pct total min max avg 95% stddev median ============ === ======= ======= ======= ======= ======= ======= ======= Count Exec time 2 44s 29us 3s 274us 67us 2ms 38us Lock time s 4us 52ms 48us 57us 36us 44us Rows sent Rows examine M Rows affecte Rows read M Bytes sent G.57k.57k.57k.57k.57k Tmp tables Tmp disk tbl Tmp tbl size Query size G

18 Query output 3 Query_time distribution us us us ms ms ms s s+ Tables SHOW TABLE STATUS FROM `slow_db` LIKE 'unlock_dialogs'\g SHOW CREATE TABLE `slow_db`.`unlock_dialogs`\g SHOW TABLE STATUS FROM `slow_db` LIKE 'unlock_dialog_translations'\g SHOW CREATE TABLE `slow_db`.`unlock_dialog_translations`\g EXPLAIN /*!5 PARTITIONS*/ SELECT `unlock_dialogs`.`id` AS t_r, `unlock_dialogs`.`level_id` AS t_r, `unlock_dialogs`. `games_played` AS t_r2, `unlock_dialogs`.`screen` AS t_r3, `unlock_dialogs`.`text` AS t_r4, `unlock_dialogs`.`character_id` AS t_r5, `unlock_dialogs`.`orientation` AS t_r6, `unlock_dialogs`.`sort_order` AS t_r7, `unlock_dialogs`.`created_at` AS t_r8, `unlock_dialogs`.`updated_at` AS t_r9, `unlock_dialog_translations`.`id` AS t_r, `unlock_dialog_translations`.`unlock_dialog_id` AS t_r, `unlock_dialog_translations`.`locale` AS t_r2, `unlock_dialog_translations`.`text` AS t_r3, `unlock_dialog_translations`. `created_at` AS t_r4, `unlock_dialog_translations`.`updated_at` AS t_r5 FROM `unlock_dialogs` LEFT OUTER JOIN `unlock_dialog_translations` ON `unlock_dialog_translations`.`unlock_dialog_id` = `unlock_dialogs`.`id` WHERE `unlock_dialogs`.`games_played` = 366 AND `unlock_dialog_translations`.`locale` IN ('en', 'en')\g

19 Query Explain 3 mysql> EXPLAIN... ***************************. row *************************** id: select_type: SIMPLE table: unlock_dialogs partitions: NULL type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 35 Extra: Using where *************************** 2. row *************************** id: select_type: SIMPLE table: unlock_dialog_translations partitions: NULL type: ref possible_keys: idx_dialog_locale key: idx_dialog_locale key_len: 5 ref: slow_db.unlock_dialogs.id rows: Extra: Using where 2 rows in set (. sec)

20 MySQL Variables to set --slow-query-log Logs SQL statements that run for more than long_query_time --log-queries-not-using-indexes --long-query-time - Use logrotate to keep N slow query logs

21 MySQL Variables to set Other options: log_slow_admin_statements log_slow_slave_statements (Percona) log_slow_verbosity ( query_plan / innodb / profiling ) (Percona) log_slow_filter ( full_scan / tmp_table / filesort_on_disk / etc)

22 Extended Verbosity Time: :29:54 web [ ] Thread_id: Schema: dbname_prod Last_errno: Killed: Query_time: Lock_time:.44 Rows_sent: 98 Rows_examined: 46 Rows_affected: Rows_read: Bytes_sent: 2548 Tmp_tables: Tmp_disk_tables: Tmp_table_sizes: InnoDB_trx_id: 7BE946 QC_Hit: No Full_scan: No Full_join: No Tmp_table: No Tmp_table_on_disk: No Filesort: No Filesort_on_disk: No Merge_passes: InnoDB_IO_r_ops: 9 InnoDB_IO_r_bytes: InnoDB_IO_r_wait: InnoDB_rec_lock_wait:. InnoDB_queue_wait:. InnoDB_pages_distinct: 43 SET timestamp= ; SELECT...

23 pt-query-digest variables to consider --order-by - output sorted by count --limit - default top 95%, but of queries allowed --explain - connects to mysql to execute and output EXPLAIN for each query -u, --ask-pass Caution: Explain at query execution may be different than pt-query-digest's explain

24 Limitations Memory Beware the OOM killer CPU Cycles aren't free Log collection period Non-mysql issues

25 What about RDS?

26 RDS slow log mysql> select * from mysql.slow_log limit ; start_time user_host query_time lock_time rows_sent rows_examined db insert_id server_id sql_text last_insert_id :9:28 ip-not-your-host.ec2.internal [not.your.host] ::2 :: atvi_codlive_web SELECT `ranks`.* FROM `ranks` WHERE `title` = 'mw3' AND `id` = 39 ORDER BY `ranks`.`id` ASC LIMIT

27 RDS slow log mysql> SELECT CONCAT( ' Time: ', DATE_FORMAT(start_time, '%y%m%d %H%i%s'), '\n', ' ', user_host, '\n', ' Query_time: ', TIME_TO_SEC(query_time), ' Lock_time: ', TIME_TO_SEC(lock_time), ' Rows_sent: ', rows_sent, ' Rows_examined: ', rows_examined, '\n', sql_text, ';' ) FROM mysql.slow_log limit ; CONCAT( ' Time: ', DATE_FORMAT(start_time, '%y%m%d %H%i%s'), '\n', ' User@Host: ', user_host, '\n', ' Query_time: ', TIME_TO_SEC(query_time), ' Lock_time: ', TIME_TO_SEC (lock_time), ' Rows_sent: ', rows_sent, ' Rows_examined: ', rows_examined, '\n', Time: User@Host: ip-not-your-host.ec2.internal [not.your.host] Query_time: 2 Lock_time: Rows_sent: Rows_examined: SELECT `ranks`.* FROM `ranks` WHERE `title` = 'mw3' AND `id` = 39 ORDER BY `ranks`.`id` ASC LIMIT ; row in set (. sec)

28 RDS slow log mysql -u user -p -h host.rds.amazonaws.com -D mysql -s -r -e "SELECT CONCAT( ' Time: ', DATE_FORMAT(start_time, '% y%m%d %H%i%s'), '\n', ' ', user_host, '\n', ' Query_time: ', TIME_TO_SEC(query_time), ' Lock_time: ', TIME_TO_SEC(lock_time), ' Rows_sent: ', rows_sent, ' Rows_examined: ', rows_examined, '\n', sql_text, ';' ) FROM mysql.slow_log" > /tmp/mysql.slow_log.log

29 Thanks for coming! Mark Filipi