Introduction to Troubleshooting Performance What Affects Query Execution? Sveta Smirnova Principal Support Engineer April, 7, 2016

Transcription

1 Introduction to Troubleshooting Performance What Affects Query Execution? Sveta Smirnova Principal Support Engineer April, 7, 2016

2 Terms of conditions The query Controlling optimizer Concurrency Hardware MySQL Options Table of Contents 2

3 Terms of conditions 3

4 Slow is relative Mind your data 1,000,000 rows 1 row Mind use case Popular website Admin interface Weekly cron job Mind location Server, used my multiple connections Dedicated for OLAP queries 4

5 Is MySQL the weak link? MySQL can run together with other applications top -b -n 1 head -n 20 top - 12:31:58 up 7 days, 16:54, 6 users, load average: 1,04... Tasks: 292 total, 1 running, 290 sleeping, 1 stopped, 0 zombie %Cpu(s): 18,6 us, 3,3 sy, 0,1 ni, 76,7 id, 1,3 wa, 0,0 hi... KiB Mem: total, used, free, buffers KiB Swap: 0 total, 0 used, 0 free cached PID USER NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND sveta S 104,1 0,6 0:24.75 mysqld sveta ,136g S 26,0 13,9 487:03.82 firefox 5

6 Is MySQL the weak link? MySQL can run together with other applications Query can run fast, but application can be slow if ($result = $mysqli->query($query)) { - Database finished its job! while ($row = $result->fetch_row()) { $handle = fopen( my_file.txt ); fputs($handle, $row); fclose($handle); }

7 Is MySQL the weak link? MySQL can run together with other applications Query can run fast, but application can be slow How to find if MySQL is a weak link? <app code> $start_time = microtime(true); $result = $mysqli->query($query); - Running query $end_time = microtime(true); print("query took ". ($end_time - $start_time). " microseconds"); 5

8 What affects performance? Query Concurrency Hardware MySQL configuration 6

9 The query 7

10 Heart of the application You communicate with database using queries Even if you use NoSQL interface They are not SQL queries, but still queries Data, which you request, matters 1,000,000,000 rows 1 row 8

11 Troubleshooting vs performance tuning Query can use bad practice SELECT * FROM many columns table Table can be defined in weird way No normalization Dozens columns like Characteristic N Troubleshooting has to live with these There can be good reasons Legacy application De-normalization to speed up SELECTs Usually with price of more locking and slow updates You name it 9

12 Query execution workflow Query sent Connection Pool: Authentication, Caches; SQL interface; Parser Optimizer Storage engines Hardware 10

13 Controlling optimizer 11

14 Pre-requisites Find queries to optimize While designing application In slow query log In Performance Schema mysql> select sql_text, (timer_end - timer_start)/ as time \ -> from events_statements_history_long \ -> where (timer_end - timer_start) > 4* \ -> and event_name= statement/sql/select \G *************************** 1. row *************************** sql_text: select count(*) from titles join employees using (emp_no) wh time: rows in set (0.01 sec) 12

15 Pre-requisites Find queries to optimize While designing application In slow query log In Performance Schema In sys schema statement analysis statements with full table scans statements with runtimes in 95th percentile statements with sorting statements with temp tables statements with errors or warnings 12

16 Pre-requisites Find queries to optimize Make sure they are sane SELECT COUNT(*) FROM huge_table LEFT JOIN other_table USING(id) \ LEFT JOIN another_table USING(id) \ LEFT JOIN one_more_table USING(id) \ LEFT JOIN and_yet_more_table USING(id) WHERE another_field = foo ; vs SELECT COUNT(*) FROM huge_table WHERE another_field = foo ; Or you really need them 12

17 MySQL Indexes B-Tree (Mostly) Fractal Tree R-Tree (Spatial) Hash (Memory SE) Engine-dependent d001 d003 d008 d009 d003****** d009****** d008****** d009****** d001****** d003****** d009****** d008****** d009****** d001****** d008****** d008****** d001****** 13

18 SHOW INDEX Controlling indexes Displays index statistics as seen by optimizer Most important filed is Cardinality Should be equal to number of distinct values ANALYZE Up to 10 % difference is OK Updates index statistics Recommended to use together with InnoDB persistent statistics 14

19 Use case: Is statistic actual? mysql> show index from sbtest1; Table Key_name Column_name Cardinality sbtest1 PRIMARY id sbtest1 k_1 k mysql> select count(distinct id), count(distinct k) from sbtest1; count(distinct id) count(distinct k) row in set (0,58 sec) 15

20 Estimates: EXPLAIN Tables, for which information is printed Length of the key Additional information Number of select Possible keys Number of examined rows mysql> explain extended select * from t1 join t2 where t1.int_key=1; id select_type table type possible_keys key key_len ref rows f... Extra SIMPLE t1 ref int_key,ik int_key 5 const NULL 1 SIMPLE t2 index NULL pk 9 NULL Using inde Product of rows here: how many rows in all tables will be accessed Using join bu Table, for which information is printed For this example estimated value is 4*6 = 24 (Block Nested % of filtered rows 2 rows in set, 1 warning (0.00 sec) Select type Key, which was actually used rows x filtered / 100 number of rows, which will be joined with another table How data is accessed Which columns were compared with the index Note (Code 1003): /* select#1 */ select test. t1. pk AS pk, test. t1. int_key AS int_key, test AS pk, test. t2. int_key AS int_key from test. t1 join test. t2 where ( test. t1. int_key Actual (optimized) query as executed by MySQL Server 16

21 EXTENDED EXPLAIN extensions mysql> explain extended select * from t1 join t2 where t1.int_key=1;... mysql> SHOW WARNINGS; Note (Code 1003): /* select#1 */ select test. t1. pk AS pk, test. t1. int_key AS int_key, test. t2. pk AS pk, test. t2. int_key AS int_key from test. t1 join test. t2 where ( test. t1. int_key = 1) PARTITIONS FORMAT=JSON 17

22 EXTENDED PARTITIONS EXPLAIN extensions mysql> explain partitions select , account_id from users -> where = and account_id=896; id select_type table partitions type possible_keys SIMPLE users p0 const idx_ _account_id row in set (0,03 sec) FORMAT=JSON 17

23 EXPLAIN extensions EXTENDED PARTITIONS FORMAT=JSON mysql> EXPLAIN FORMAT=JSON SELECT user, host FROM mysql.user\g *************************** 1. row *************************** EXPLAIN: { "used_key_parts": [ "query_block": { "Host", "select_id": 1, "User" "table": { ], "table_name": "user", "key_length": "228", "access_type": "index", "rows": 8, "key": "PRIMARY", "filtered": 100, "using_index": true } } } 17

24 Actual execution Status variables Handler % mysql> select count(*) from employees join titles using(emp_no)... *************************** 1. row *************************** count(*): row in set (3.24 sec) mysql> SHOW STATUS LIKE Handler_read_% ; Variable_name Value Handler_read_first 1 Handler_read_key Handler_read_last 0 Handler_read_next

25 Actual execution Status variables Handler % INFORMATION SCHEMA.OPTIMIZER TRACE join preparation, join optimization, join execution considered execution plans, refine plan, more PERFORMANCE SCHEMA.EVENTS STATEMENTS * PERFORMANCE SCHEMA.EVENTS STAGES * ANALYZE statement - MariaDB only! 18

26 Actual execution Status variables Handler % INFORMATION SCHEMA.OPTIMIZER TRACE PERFORMANCE SCHEMA.EVENTS STATEMENTS * events statements * and prepared statements instances: important fields CREATED TMP DISK TABLES CREATED TMP TABLES SELECT FULL JOIN SELECT RANGE CHECK SELECT SCAN SORT MERGE PASSES SORT SCAN PERFORMANCE SCHEMA.EVENTS STAGES * ANALYZE statement - MariaDB only! 18

27 Actual execution Status variables Handler % INFORMATION SCHEMA.OPTIMIZER TRACE PERFORMANCE SCHEMA.EVENTS STATEMENTS * PERFORMANCE SCHEMA.EVENTS STAGES * EVENT NAME LIKE stage/sql/%tmp% stage/sql/%lock% stage/%/waiting for% stage/sql/end stage/sql/freeing items stage/sql/sending data stage/sql/cleaning up stage/sql/closing tables ANALYZE statement - MariaDB only! 18

28 Actual execution Status variables Handler % INFORMATION SCHEMA.OPTIMIZER TRACE PERFORMANCE SCHEMA.EVENTS STATEMENTS * PERFORMANCE SCHEMA.EVENTS STAGES * ANALYZE statement - MariaDB only! Two new columns r rows - actual number of read rows r filtered - actual number of filtered rows 18

29 Concurrency 19

30 Common concurrency issues Query or transaction waits a lock, held by another one Fight for system resources Symptoms Many processes are waiting You see query in the slow log, but it runs fast if single-thread environment Query execution time varies 20

31 Lock types and transactions Lock types Levels MDL Table-level Row-level What do they lock Read locks Block writes Write locks Block reads and writes Transactions Server-level MDL locks Table locks Engine-level Table locks Row locks AUTOCOMMIT supported 21

32 Locks diagnostic Table-level PROCESSLIST: Waiting for table lock P S.TABLE HANDLES Row-level InnoDB monitors SHOW ENGINE INNODB STATUS Tables in INFORMATION SCHEMA P S.EVENTS TRANSACTIONS Option innodb print all deadlocks MDL PROCESSLIST: Waiting for metadata lock P S.METADATA LOCKS 22

33 Hardware 23

34 What affects MySQL Performance? RAM CPU IO (Disk) Ntework 24

35 Common issues with hardware Too high resource usage MySQL is swapping Disk IO is near the limit of its capacity Network bandwidth is too small More Too low resource usage Single CPU core used on 24 cores system Memory usage is low while disk IO is high due to temporary tables More 25

36 OS-level tools Tools to work with hardware Performance Schema Little support in 5.5 More in 5.6 Even more in 5.7 Engine status Limited 26

37 Tools to work with hardware For true sysadmins Consult Brendan D. Gregg s Linux Performance webpage Image License: Attribution-ShareAlike

38 Tools to work with hardware For true sysadmins Consult Brendan D. Gregg s Linux Performance webpage Necessary minimum Following slides pt-stalk For automatization To provide for Percona Support for future analysis Does not use Performance Schema! 27

39 Memory What uses memory in MySQL Buffers Temporary tables Internal structures - out of user control! OS did not show memory as freed yet? 28

40 free top vmstat Investigation Memory diagnostic before 5.7 Total size of buffers Number of temporary tables Number of parallel connections There was no way to know how exactly memory was allocated 29

41 Memory diagnostic in 5.7 mysql> select thread_id tid, user, current_allocated ca, total_allocated -> from sys.memory_by_thread_by_current_bytes; tid user ca total_allocated sql/main 2.53 GiB 2.69 GiB 150 root@ MiB MiB 146 sql/slave_sql 1.31 MiB 1.44 MiB 145 sql/slave_io 1.08 MiB 2.79 MiB... 4 innodb/io_log_thread bytes KiB 72 innodb/io_write_thread bytes 1.10 MiB rows in set (2.65 sec) 30

42 Disk diagnostics df iostat ls -l /proc/{pid OF MYSQLD}/fd InnoDB status performance schema.table io waits % sys.io % 31

43 Disk diagnostics InnoDB status mysql> show engine innodb status\g FILE I/O I/O thread 0 state: waiting for i/o request (insert buffer thread) I/O thread 1 state: waiting for i/o request (log thread) I/O thread 2 state: waiting for i/o request (read thread) I/O thread 5 state: waiting for i/o request (write thread) Pending normal aio reads: [0, 0], aio writes: [0, 0], ibuf aio reads:, log i/o s:, sync i/o s: Pending flushes (fsync) log: 0; buffer pool: OS file reads, 102 OS file writes, 17 OS fsyncs reads/s, avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s

44 Disk diagnostics InnoDB status ROW OPERATIONS queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Process ID=26676, Main thread ID= , state: sleeping Number of rows inserted 19, updated 0, deleted 0, read inserts/s, 0.00 updates/s, 0.00 deletes/s, reads/s 31

45 Disk diagnostics mysql> select OBJECT_SCHEMA, OBJECT_NAME, INDEX_NAME \ -> from table_io_waits_summary_by_index_usage where COUNT_STAR=0; OBJECT_SCHEMA OBJECT_NAME INDEX_NAME employees departments PRIMARY employees departments dept_name employees dept_emp PRIMARY employees dept_emp emp_no employees dept_emp dept_no employees dept_manager PRIMARY employees dept_manager emp_no

46 Disk diagnostics sys.io % mysql> select event_name, count_read, total_read, total_requested \ -> from io_global_by_wait_by_bytes; event_name count_read total_read total_requested innodb/innodb_data_file MiB MiB sql/binlog MiB MiB sql/io_cache MiB MiB sql/frm KiB KiB innodb/innodb_log_file KiB KiB sql/errmsg KiB KiB

47 Disk diagnostics sys.io % Other useful views host summary by file io host summary by file io type io by thread by latency io global by file by [bytes latency] - Per-file io global by file by latency io global by wait by [bytes latency] latest file io - Latest IO operations user summary by file io user summary by file io type 31

48 CPU diagnostics top %Cpu(s): 15.2 us, 2.7 sy, 1.8 ni, 79.4 id, 0.9 wa, 0.0 hi, 0.0 si, 0.0 st PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND sveta S :43.73 mysqld iostat -x avg-cpu: %user %nice %system %iowait %steal %idle ps sveta@thinkie> ps -eo pid,user,comm,pcpu,pmem,vsz grep mysqld sveta mysqld sveta mysqld

49 CPU: problem solving Low load Is your mysqld really active? Check OS limits ulimit User and global options Increase concurrency-related options innodb thread concurrency innodb read io threads innodb write io threads Max (2 X (number of CPU cores))! High load Move job to memory: increase buffers Tune queries 33

50 Network: what is worth attention? Stability Bandwidth Speed RTT ping -c 2 PING ( ) 56(84) bytes of data. 64 bytes from ( ): icmp_seq=1 ttl=54 time 64 bytes from ( ): icmp_seq=2 ttl=54 time ping statistics packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = / / /0.524 ms 34

51 Network:diagnostics log-warnings=2 Send huge file (1G or larger) tcpdump P S.accounts, hosts, users P S.host cache 35

52 P S.accounts, hosts, users Network:diagnostics mysql> select user, host, current_connections as cur, \ -> total_connections as total from accounts; user host cur total foo localhost 0 3 root localhost 1 3 NULL NULL rows in set (0.01 sec) 35

53 Network:diagnostics P S.host cache mysql> select IP, HOST, SUM_CONNECT_ERRORS, COUNT_NAMEINFO_PERMANENT_ERROR -> COUNT_PROXY_USER_ERRORS, COUNT_DEFAULT_DATABASE_ERRORS, \ -> FIRST_SEEN, LAST_SEEN from host_cache\g ************************ 1. row ************************ IP: HOST: foo.bar.com SUM_CONNECT_ERRORS: 1 COUNT_NAMEINFO_PERMANENT_ERRORS: 0 COUNT_PROXY_USER_ERRORS: 0 COUNT_DEFAULT_DATABASE_ERRORS: 1 FIRST_SEEN: :29:45 LAST_SEEN: :32:

54 MySQL Options 36

55 Types of options Not directly affecting performance Which locks to set innodb locks unsafe for binlog When to flush to disk innodb flush log at trx commit Directly affecting performance Concurrency-related innodb thread concurrency Memory buffers innodb buffer pool size 37

56 System variables and options: scope Global Control parameters, necessary for all server processes Location of server files: datadir etc. Shared buffers More Session Control connection-specific parameters MySQL option tables - direct link 38

57 Buffers: when allocated Those which control behavior of whole server Once at server startup Can start with low values, then grow to specified Connection options For every connection when connection opens Operation-specific For every operation when needed Can be allocated more than once per query 39

58 System variables: control before 5.7 SHOW [GLOBAL] STATUS Tables in Information Schema GLOBAL,SESSION VARIABLES GLOBAL,SESSION STATUS GLOBAL Since server start SESSION For operations in current session Can be reset FLUSH STATUS 40

59 System variables: control in 5.7 Performance Schema tables variabes by* user variables by* status by* Statistics grouped by Global Session Thread Account/Host/User 41

60 System variables: best practices Record currently used variables SHOW [GLOBAL] VARIABLES Make change dynamically if possible SET [GLOBAL] var name=new VAL Test in one session first Then change global variable Change configuration file after you are happy with results 42

61 When affecting option is not known Record currently used variables SHOW [GLOBAL] VARIABLES Start mysqld with option no-defaults This option must be first one! Check if problem is solved Change variable values one-by-one until you find one which leads to the problem 43

62 Summary 44

63 Summary Performance affected by 4 main factors Query Concurrency Hardware Configuration Study all of them Iterate while troubleshooting 45

64 Summary Performance affected by 4 main factors Query Optimize! Concurrency Hardware Configuration Study all of them Iterate while troubleshooting 45

65 Summary Performance affected by 4 main factors Query Concurrency Remove conflicts! Hardware Configuration Study all of them Iterate while troubleshooting 45

66 Summary Performance affected by 4 main factors Query Concurrency Hardware Find bottlenecks! Configuration Study all of them Iterate while troubleshooting 45

67 Summary Performance affected by 4 main factors Query Concurrency Hardware Configuration Mind the options! Study all of them Iterate while troubleshooting 45

68 More information MySQL Troubleshooting book High Performance MySQL book Planet MySQL Percona Data Performance Blog MySQL User Reference Manual Bug trackers EXPLAIN FORMAT=JSON is Cool! series 46

69 Percona Live Data Performance Conference April in Santa Clara, CA at the Santa Clara Convention Center Register with code WebinarPL to receive 15% off at registration MongoDB, MySQL, NoSQL, Data in the Cloud

70 Place for your questions??? 48

71 Thank you!