MySQL SYS Schema Deep Dive

Transcription

1 MySQL SYS Schema Deep Dive Mark Leith Senior Software Development Manager MySQL Enterprise Tools, Oracle

2 Program Agenda Introduction to the MySQL sys schema Installation Formatting / Helper Routines Tracing Routines Summary Views

3 Program Agenda Introduction to the MySQL sys schema Installation Formatting / Helper Routines Tracing Routines Summary Views

4 You are : Using 5.1? Using 5.5? Using 5.6? Using 5.7? A Performance Schema user? A Sys Schema user?

5 Why the sys schema? Performance Schema exposes a huge amount of data Many of its tables can implement multiple different use cases, depending on the type of data The sys schema layers on top the (sometimes complex) queries in to a packaged set of easy to use views It also adds a number of functions and procedures that help format, configure the output of, and record, Performance Schema data Implements many of the common day to day DBA and Developer debugging and tuning use cases

6 Program Agenda Introduction to the MySQL sys schema Installation Formatting / Helper Routines Tracing Routines Summary Views

7 Installation from the command line $ git clone /tmp/sys $ cd /tmp/sys $ mysql -u user -p < sys_<version>.sql <version> can be 56 or 57, for 5.6 and 5.7 respectively

8 Installation from MySQL Workbench 6.1+

9 Installation on the new MySQL 5.7 GA Do nothing other than install/mysql_upgrade it is there by default! You will be able to install future upgrades from the sys repository as you wish, no restrictions there

10 Checking sys is loaded select * from sys.version; sys_version mysql_version mysql> select * -> from sys.schema_object_overview -> where db = 'sys'; db object_type count sys BASE TABLE 1 sys INDEX (BTREE) 1 sys TRIGGER 2 sys FUNCTION 21 sys PROCEDURE 26 sys VIEW

11 Program Agenda Introduction to the MySQL sys schema Installation Formatting / Helper Routines Tracing Routines Summary Views

12 Formatting / Helper Functions Make output human readable Format time appropriately Format bytes appropriately Truncate output for large width values for CLI Extract object names Check instrumentation state format_time format_bytes format_path format_statement extract_table_from_file_name extract_schema_from_file_name ps_is_account_enabled ps_is_consumer_enabled ps_is_instrument_default_enabled ps_is_instrument_default_timed ps_is_thread_instrumented 12

13 format_time() function mysql> select format_time( ) as time -> union select format_time( ) -> union select format_time( ); time ms 8.33 s 00:01:

14 format_bytes() function mysql> select format_bytes(23423) as bytes -> union select format_bytes( ) -> union select format_bytes( ) -> union select format_bytes( ); bytes KiB MiB GiB 2.09 TiB

15 Performance Schema Config Helper Procedures ps_setup_show_disabled() / ps_setup_show_enabled() ps_setup_disable_thread() / ps_setup_enable_thread() ps_setup_disable_background_threads() / ps_setup_enable_background_threads() ps_setup_disable_instrument() / ps_setup_enable_instrument() ps_setup_disable_consumer() / ps_setup_enable_consumer() ps_setup_save() / ps_setup_reload_saved() ps_setup_reset_to_default() / ps_truncate_all_tables()

16 Program Agenda Introduction to the MySQL sys schema Installation Formatting / Helper Routines Tracing Routines Summary Views

17 Statement Tracing Routines Dump all P_S data to a dot graph file to trace a thread Look in the statement history table to capture more info on a statement digest Show detailed transaction info Dump as much diagnostics info as possible (support enabler) ps_trace_thread ps_trace_statement_digest ps_thread_trx_info ps_thread_stack ps_statement_avg_latency_histogram diagnostics 17

18 ps_trace_statement_digest procedure ps_trace_statement_digest() analyses live traffic looking for certain statement digest for a period of time Captures statistics on each matching statement it finds Returns a report of the captured stats An overall summary A break down for the longest running example An EXPLAIN (if the statement is not truncated)

19 ps_trace_statement_digest procedure ps_trace_statement_digest() parameters in_digest The statement digest to analyse in_runtime How long to run analysis for in_interval How often to snapshot data in_start_fresh Whether to truncate P_S tables first in_auto_enable Whether to auto enable required config

20 ps_trace_statement_digest procedure example mysql> call ps_trace_statement_digest( 6134e9d6f25eb8e6cddf11f6938f202a, 60, 1, true, true); SUMMARY STATISTICS SUMMARY STATISTICS row in set (59.93 sec) executions exec_time lock_time rows_sent rows_examined tmp_tables full_scans s ms row in set (59.93 sec) event_name count latency stage/sql/sending data s stage/sql/update ms stage/sql/removing tmp table ms...

21 ps_trace_statement_digest procedure example LONGEST RUNNING STATEMENT LONGEST RUNNING STATEMENT row in set (59.95 sec) thread_id exec_time lock_time rows_sent rows_examined tmp_tables full_scan ms us row in set (59.95 sec) sql_text /* mem dbpool.default */ insert into `mem quan`.`normalized_statements_by_server_by_schema` (firstseen, lastseen, normalized_statement_id, `schema`, server, id) values ( , , x'808aceb88fc4b45bc47c4fe9d86c7f26', 'mem', '753c939e-1b99-11e3-b6d4-bc761a1f4f2f', x'7cfe81f98b1e3fe0895d1aa2c39b326d') ON DUPLICATE KEY UPDATE

22 ps_trace_statement_digest procedure example event_name latency stage/sql/init ms stage/sql/checking permissions 2.35 us stage/sql/opening tables us stage/sql/init us stage/sql/system lock 3.66 us stage/sql/update ms stage/sql/end 1.11 us stage/sql/query end 1.31 us stage/sql/closing tables 5.44 ms stage/sql/freeing items us rows in set (59.99 sec) id select_type table type possible_keys key key_len ref rows Extra SIMPLE NULL NULL NULL NULL NULL NULL NULL No tables used

23 sys.ps_thread_trx_info(<thread_id>) SELECT sys.ps_thread_trx_info(48) as trx_info\g *************************** 1. row *************************** trx_info: [ { "time": " us", "state": "COMMITTED", "mode": "READ WRITE", "autocommitted": "NO", "gtid": "AUTOMATIC", "isolation": "REPEATABLE READ", "statements_executed": [ { "sql_text": "INSERT INTO info VALUES (1, 'foo')", "time": " us", "schema": "trx", "rows_examined": 0, "rows_affected": 1, "rows_sent": 0, "tmp_tables": 0, "tmp_disk_tables": 0, "sort_rows": 0, "sort_merge_passes": 0 },... Pass in thread id

24 ps_trace_thread procedure ps_trace_thread() monitors a specific thread for a period Captures as much information on the thread activity as possible Returns a dot formatted file, that can graph the event hierarchy

25 ps_trace_thread procedure ps_trace_thread parameters in_thread_id The thread to analyse in_outfile The file to dump the data to (INTO OUTFILE) in_max_runtime How long to run analysis for in_interval How often to snapshot for data in_start_fresh Whether to truncate P_S tables first in_auto_setup Whether to auto enable required config in_debug Whether to also print debug info (source info)

26 ps_trace_thread procedure example mysql> call ps_trace_thread(27768, '/tmp/stack_27768.dot', 60, 0.1, true, true, true); Info Data collection starting for THREAD_ID = row in set (4.82 sec) Info Stack trace written to /tmp/stack_27768.dot row in set (60.90 sec) Convert to PDF dot -Tpdf -o /tmp/stack_27768.pdf /tmp/stack_27768.dot row in set (60.90 sec) Convert to PNG dot -Tpng -o /tmp/stack_27768.png /tmp/stack_27768.dot row in set (60.90 sec)+

27 ps_trace_thread procedure example cerberus:~ mark$ more Documents/statement_graphs/deletes.dot digraph events { graph [rankdir=lr, nodesep="0.10"]; node [label="\n"]; graph [bb="0,0,1860,17476"]; 1 [label="(0.20μ) mysys/thr_lock_threads\n", style=filled, color=lightskyblue, pos="209,18", width="3.19", height="0.50"]; 2 [label="(0.18μ) sql/thd::lock_thd_data\n", style=filled, color=lightskyblue, pos="209,61", width="3.06", height="0.50"]; 3 [label="(0.24μ) sql/hash_filo::lock\n", style=filled, color=lightskyblue, pos="209,104", width="2.42", height= 0.50"]; 4 [label="(0.31μ) sql/logger::lock_logger - read_lock\n", style=filled, color=orchid, pos="209,147", width="4.22", height="0.50"]; 5 [label="(0.08μ) sql/log::lock_log\n", style=filled, color=lightskyblue, pos="209,190", width="2.64", height="0.50"]; 6 [label="(12.97μ) wait/io/file/sql/query_log - write 49 bytes\n/users/mark/mysql/lp-mysql-trunk/mysql-test/var/mysqld.1/ mysqld.log\n", style=filled, color=red, shape=box, pos="209,235", width="5.78", height="0.56"]; 7 [label="(176.14μ) limit 1\nerrors: 0\nwarnings: 0\nlock time: 0.00μ\nrows affected: 0\nrows sent:\ 209,18", width="3.19", height="0.50"];mp disk tables: 0\nselect scan: 0\nselect full join: : 2 [label="(0.18μ) sql/thd::lock_thd_data\n", style=filled, color=lightskyblue, pos="209,61", width="3.06", height="0.50"]; 3 [label="(0.24μ) sql/hash_filo::lock\n", style=filled, color=lightskyblue, pos="209,104", width="2.42", height="0.50"]; 4 [label="(0.31μ) sql/logger::lock_logger - read_lock\n", style=filled, color=orchid, pos="209,147", width="4.22", height="0.50"]; 5 [label="(0.08μ) sql/log::lock_log\n", style=filled, color=lightskyblue, pos="209,190", width="2.64", height="0.50"];

28 ps_trace_thread procedure example MySQL 5.7 Procedures Transactions Statements Stages Waits

29

30

31

32

33

34

35 Program Agenda Introduction to the MySQL sys schema Installation Formatting / Helper Routines Tracing Routines Summary Views

36 MySQL sys views Reference set of views solving various administrator use cases Simple views, create/copy your own, sys is not locked down Build upon both performance_schema and INFORMATION_SCHEMA Both formatted and raw views are available All raw views are prefixed with x$ Raw views are there for tools to poll Formatted views are for humans and the command line

37 User / Host Summary Views High level user/host overviews Breakdowns by: IO usage Stages Statement details Following show users only user_summary user_summary_by_file_io user_summary_by_file_io_type user_summary_by_stages user_summary_by_statement_latency user_summary_by_statement_type host_summary host_summary_by_file_io host_summary_by_file_io_type host_summary_by_stages host_summary_by_statement_latency host_summary_by_statement_type 37

38 user_summary view mysql> select * from user_summary\g *************************** 1. row *************************** user: mark statements: 3072 statement_latency: 1.77 s statement_avg_latency: us table_scans: 7 file_ios: file_io_latency: ms current_connections: 1 total_connections: 1 unique_hosts: 1 MySQL 5.7 current_memory: KiB total_memory_allocated: MiB

39 user_summary_by_file_io_type view mysql> select * from user_summary_by_file_io_type; user event_name total latency max_latency mark wait/io/file/myisam/dfile ms ms mark wait/io/file/myisam/kfile ms ms mark wait/io/file/sql/dbopt us 9.17 us root wait/io/file/innodb/innodb_log_file :28: s root wait/io/file/myisam/kfile :17: s root wait/io/file/myisam/dfile :07: s root wait/io/file/sql/binlog :03: s root wait/io/file/innodb/innodb_data_file :02: ms root wait/io/file/sql/frm s ms root wait/io/file/sql/dbopt s ms

40 user_summary_by_statement_latency view mysql> select * from user_summary_by_statement_latency\g *************************** 1. row *************************** user: mark total: 176 total_latency: 1.26 s max_latency: ms lock_latency: ms rows_sent: 311 rows_examined: 307 rows_affected: 0 full_scans: 7

41 user_summary_by_statement_type view mysql> select * from user_summary_by_statement_type\g *************************** 1. row *************************** user: mark statement: commit total: total_latency: 13.33h max_latency: s lock_latency: 0 ps rows_sent: 0 rows_examined: 0 rows_affected: 0 full_scans: 0

42 IO Summary Views IO by current threads Global summaries by file and class, aggregating by bytes or latency Stream of last file IO events in raw form io_by_thread_by_latency io_global_by_file_by_bytes io_global_by_file_by_latency io_global_by_wait_by_bytes io_global_by_wait_by_latency 42

43 io_by_thread_by_latency view mysql> select * from io_by_thread_by_latency\g *************************** 1. row *************************** user: main total: 1193 total_latency: ms min_latency: ns avg_latency: us max_latency: ms thread_id: 1 processlist_id: NULL *************************** 2. row *************************** user: root@localhost total: 533 total_latency: ms min_latency: ns avg_latency: 8.41 ms max_latency: ms thread_id: 23 processlist_id: 4

44 io_global_by_file_by_latency view mysql> select * from io_global_by_file_by_latency limit 1\G *************************** 1. row *************************** total: total_latency: 00:28:53.06 count_read: 1006 read_latency: 4.51 s count_write: write_latency: s count_misc: misc_latency: 00:28:15.16

45 latest_file_io view mysql> select * from latest_file_io; thread file latency operation requested us lock NULL 6.16 us write 1.50 KiB ms sync NULL us write 905 bytes us write 1.00 KiB us read 905 bytes 6.65 us write 1.00 KiB us sync NULL us write 1.00 KiB us sync NULL us write 601 bytes 7.55 us read 601 bytes us write 1.00 KiB us sync NULL us write 601 bytes 6.47 us read 601 bytes 9.22 us write 1.50 KiB

46 Schema Analysis Views Object overview Table usage statistics Index usage statistics Following show users only schema_index_statistics schema_object_overview schema_table_statistics schema_table_statistics_with_buffer schema_tables_with_full_table_scans schema_unused_indexes 46

47 schema_table_statistics view mysql> select * from schema_table_statistics limit 1\G *************************** 1. row *************************** table_schema: mem quan table_name: normalized_statements_by_server_by_schema_data total_latency: 00:55:07.80 rows_fetched: fetch_latency: s rows_inserted: insert_latency: 00:52:18.10 rows_updated: update_latency: 00:02:28.62 rows_deleted: delete_latency: 3.98 s io_read_requests: io_read: MiB io_read_latency: 00:03:04.10 io_write_requests: io_write: GiB io_write_latency: s io_misc_requests: io_misc_latency: 00:14:06.02

48 schema_index_statistics view mysql> select * from schema_index_statistics limit 1\G *************************** 1. row *************************** table_schema: mem inventory table_name: mysqlconnectionconfiguration index_name: PRIMARY rows_selected: select_latency: 00:07:05.03 rows_inserted: 0 insert_latency: 0 ps rows_updated: update_latency: s rows_deleted: 0 delete_latency: 0 ps

49 schema_tables_with_full_table_scans view mysql> select * from schema_tables_with_full_table_scans limit 10; object_schema object_name rows_full_scanned latency mem30 instruments fsstatistics s mem30 instruments preparedstatementapidata ms mem30 instruments mysqlprocessactivity ms mem30 instruments querycachequeriesincachedata ms mem30 instruments rowaccessesdata s mem30 instruments connectionsmaxdata ms mem30 instruments innodbdatadictionarymemorydata ms mem30 instruments innodbbufferpooldata s mem30 instruments avgrowaccessesdata s mem30 instruments innodbrowdetailsdata ms

50 schema_unused_indexes view mysql> select * from schema_unused_indexes limit 10; object_schema object_name index_name mem30 bean_config plists path mem30 config group_selections name mem30 config notification_groups name mem30 config user_form_defaults FKC1AEF1F9E7EE2CFB mem30 enterprise whats_new_entries entryid mem30 events action_logs policyid mem30 events action_logs eventid mem30 events action_logs ts mem30 events advisor_scopes_advisorcategoryids FKF0CB0DB962CAB4 mem30 events advisor_scopes_advisorclassids FK650AF1BF962CAB You should ensure you have representative time frame before taking any actions!

51 Wait Analysis Views Wait summaries by class Wait details per: User Host Globally wait_classes_global_by_avg_latency wait_classes_global_by_latency waits_by_host_by_latency waits_by_user_by_latency waits_global_by_latency 51

52 waits_global_by_latency view mysql> select * from waits_global_by_latency limit 10; events total total_latency avg_latency max_latency wait/io/table/sql/handler h us 2.29 s wait/io/file/innodb/innodb_data_file h us 1.49 s wait/io/file/innodb/innodb_log_file :36: us 1.30 s wait/io/file/myisam/kfile :18: us 1.17 s wait/io/file/myisam/dfile :08: us 1.18 s wait/io/file/sql/binlog :03: us 1.42 s wait/lock/table/sql/handler :01: us ms wait/io/socket/sql/client_connection s 6.54 us ms wait/io/file/sql/frm s us ms wait/io/file/sql/dbopt s us ms InnoDB only instance, this MyISAM IO is all temporary tables, focus tuning there!

53 waits_by_user_by_latency view mysql> select * from waits_by_user_by_latency; user event total total_latency avg_latency max_latency mark wait/io/file/myisam/dfile ms us ms mark wait/io/file/myisam/kfile ms us ms mark wait/io/file/sql/dbopt us 8.64 us 9.17 us root wait/io/table/sql/handler h us 2.29 s root wait/io/file/innodb/innodb_log_file :30: us 1.30 s root wait/io/file/myisam/kfile :18: us 1.17 s root wait/io/file/myisam/dfile :07: us 1.18 s root wait/io/file/sql/binlog :03: us 1.42 s root wait/io/file/innodb/innodb_data_file :03: ms ms root wait/lock/table/sql/handler :01: us ms root is doing the most temporary table IO

54 Lock Wait Views Table Metadata Lock Waits InnoDB Row Lock Waits schema_table_lock_waits innodb_lock_waits 54

55 sys.schema_table_lock_waits mysql> select * from sys.schema_table_lock_waits\g *************************** 1. row *************************** object_schema: test object_name: t waiting_thread_id: 43 waiting_pid: 21 waiting_account: msandbox@localhost waiting_lock_type: SHARED_UPGRADABLE waiting_lock_duration: TRANSACTION waiting_query: alter table test.t add foo int waiting_query_secs: 988 waiting_query_rows_affected: 0 waiting_query_rows_examined: 0 blocking_thread_id: 42 blocking_pid: 20 blocking_account: msandbox@localhost blocking_lock_type: SHARED_NO_READ_WRITE blocking_lock_duration: TRANSACTION sql_kill_blocking_query: KILL QUERY 20 sql_kill_blocking_connection: KILL 20

56 sys.innodb_lock_waits mysql> SELECT * FROM sys.innodb_lock_waits\g *************************** 1. row *************************** wait_started: :39:20 wait_age: 00:00:07 wait_age_secs: 7 locked_table: `db1`.`t1` locked_index: PRIMARY locked_type: RECORD waiting_trx_id: waiting_trx_started: :39:15 waiting_trx_age: 00:00:12 waiting_trx_rows_locked: 0 waiting_trx_rows_modified: 0 waiting_pid: 3 waiting_query: UPDATE t1 SET val = val + 1 WHERE id = 2 waiting_lock_id: :2363:3:3 waiting_lock_mode: X blocking_trx_id: blocking_pid: 4... blocking_query: UPDATE t1 SET val blocking_lock_id: :2363:3:3 blocking_lock_mode: X blocking_trx_started: :39:11 blocking_trx_age: 00:00:16 blocking_trx_rows_locked: 1 blocking_trx_rows_modified: 1 sql_kill_blocking_query: KILL QUERY 4 sql_kill_blocking_connection: KILL 4

57 Statement Analysis Views Statement overview Find statements by: Those with errors Those with full table scans Those creating temp tables Those that cause sorting With latency in 95th %ile statement_analysis statements_with_errors_or_warnings statements_with_full_table_scans statements_with_runtimes_in_95th_p ercentile statements_with_sorting statements_with_temp_tables 57

58 statement_analysis view mysql> select * from statement_analysis limit 2\G *************************** 1. row *************************** query: INSERT INTO `mem30 quan`. `n... nduration` = IF ( VALUES (... db: mem full_scan: exec_count: 4657 err_count: 0 warn_count: 0 total_latency: 00:03:57.40 max_latency: s avg_latency: ms lock_latency: 00:03:00.93 rows_sent: 0 rows_sent_avg: 0 rows_examined: 0 rows_examined_avg: 0 rows_affected: 4794 rows_affected_avg: 1 tmp_tables: 0 tmp_disk_tables: 0 rows_sorted: 0 sort_merge_passes: 0 digest: b98bfe333642e8e4e72c d865 first_seen: :27:01 last_seen: :53:00

59 statements_with_errors_or_warnings view mysql> select * from statements_with_errors_or_warnings limit 1\G *************************** 1. row *************************** query: CREATE TEMPORARY TABLE IF NOT... _logs` ( `id` INT8 NOT NULL ) db: mem exec_count: 1725 errors: 1725 error_pct: warnings: 0 warning_pct: first_seen: :42:32 last_seen: :39:22 digest: 51fb979dbc3910a6294c7cdabd7839c3

60 statements_with_temp_tables view mysql> select * from statements_with_temp_tables limit 1\G *************************** 1. row *************************** query: SELECT * FROM ( SELECT `digest... `, `sum_no_index_used` AS... db: mysql exec_count: 68 total_latency: 2.31 s memory_tmp_tables: 476 disk_tmp_tables: 204 avg_tmp_tables_per_query: 7 tmp_tables_to_disk_pct: 43 first_seen: :26:18 last_seen: :33:00 digest: 50761c6a d8a136b9ed6

61 statements_with_full_table_scans view mysql> select * from statements_with_full_table_scans limit 1\G *************************** 1. row *************************** query: SELECT `pojotimese0_`. `id` A... ty_timeseries` `pojotimese0_` db: mem exec_count: 1 total_latency: us no_index_used_count: 1 no_good_index_used_count: 0 no_index_used_pct: 100 rows_sent: 116 rows_examined: 116 rows_sent_avg: 116 rows_examined_avg: 116 first_seen: :25:03 last_seen: :25:03 digest: f81b5ce10baee9be754cf35fd245a516

62 Miscellaneous Views Better SHOW PROCESSLIST No mutex contention Live statistics for query Consolidated metrics processlist session metrics 62

63 processlist / session views mysql> select * from session\g *************************** 1. row *************************** thd_id: 7890 conn_id: 7870 user: root@localhost db: mem command: Query state: update time: 0 current_statement: /* mem dbpool.default */ inser...<truncated> lock_latency: 0 ps rows_examined: 0 rows_sent: 0 rows_affected: 0 tmp_tables: 0 tmp_disk_tables: 0 full_scan: NO last_statement: NULL last_statement_latency: NULL last_wait: wait/io/table/sql/handler last_wait_latency: Still Waiting source: handler.cc:7274

64 sys.sessions progress estimation mysql> select * from sys.sessions\g *************************** 1. row *************************** thd_id: conn_id: user: msandbox@localhost db: test command: Query state: alter table (flush) time: 18 current_statement: alter table t1 add column g int statement_latency: s progress: lock_latency: ms rows_examined: 0 rows_sent: 0 rows_affected: 0 tmp_tables: 0 tmp_disk_tables: 0...

65 sys.metrics performance_schema.global_status mysql> select * from sys.metrics limit 199, 10; Variable_name Variable_value Type Enabled tc_log_page_waits 0 Global Status YES threads_cached 7 Global Status YES threads_connected 3 Global Status YES threads_created 10 Global Status YES threads_running 3 Global Status YES uptime Global Status YES uptime_since_flush_status Global Status YES adaptive_hash_pages_added 0 InnoDB Metrics - adaptive_hash_index NO adaptive_hash_pages_removed 0 InnoDB Metrics - adaptive_hash_index NO adaptive_hash_rows_added 0 InnoDB Metrics - adaptive_hash_index NO information_schema.innodb_metrics

66

67