Troubleshooting Best Practices

Transcription

1 Troubleshooting Best Practices Monitoring the Production Database Without Killing Performance June, 27, 2018 Sveta Smirnova

2 Table of Contents Introduction: Between Desire and Reality Why Monitoring is Expensive? How to Find Golden Ratio? Turn ON by Demand Summary 2

3 Introduction: Between Desire and Reality

4 Desire When disaster happens we need 4

5 Desire When disaster happens we need As many information as possible 4

6 Desire When disaster happens we need As many information as possible For this exact case 4

7 Desire When disaster happens we need As many information as possible For this exact case Any slow down is not acceptable! 4

8 Reality Almost any monitoring is not free 5

9 Reality Almost any monitoring is not free Think about Memory Disk space CPU All of them 5

10 Testing: Internal Tools Everything is ON: general-log innodb-monitor-enable=all innodb-print-all-deadlocks=1 innodb-status-file=1 innodb-status-output=1 slow-query-log performance-schema=1 userstat=1 thread-statistics=1 6

11 Testing: Internal Tools OFF: general-log=0 innodb-monitor-disable=all innodb-print-all-deadlocks=0 innodb-status-file=0 innodb-status-output=0 slow-query-log=0 performance-schema=0 userstat=0 thread-statistics=0 7

12 Testing: Performance Schema Instruments: counters Locks Temporary tables Time spent... 8

13 Testing: Performance Schema Instruments: counters Consumers Tables contain data In memory You can tune their size 8

14 Testing: Performance Schema Instruments: counters Consumers Instruments and Consumers You can turn any of them ON or OFF 8

15 Testing: Internal Tools Percona-Server Standard OLTP SysBench test: for i in echo do echo $i LD_PRELOAD=/data/sveta/5.7.17/lib/mysql/libjemalloc.so /data/sveta/sbkk/bin/sysbench \ --test=/data/sveta/sysbench/sysbench/tests/db/oltp_prepared.lua --db-driver=mysql \ --oltp-tables-count=8 --oltp-table-size= mysql-table-engine=$engine \ --mysql-user=msandbox --mysql-password=msandbox --mysql-socket=/tmp/mysql_sandbox5717.sock \ --num-threads=$i --max-requests=0 --mysql-db=$db --max-time=300 --percentile=0 \ run >> $engine-oltp-prepared-$suffix.log done 9

16 Monitoring is OFF 10

17 Monitoring is ON 11

18 Only Performance Schema 12

19 Performance Schema: defaults, MDL and Memory 13

20 Why Monitoring is Expensive?

21 Log Files Require Disk Space General or binary log can use all free space 15

22 Log Files Require Disk Space General or binary log can use all free space Imagine write load 40GB/hour? 15

23 Log Files Require Disk Space General or binary log can use all free space Imagine write load 40GB/hour? Rotate policy 15

24 Memory Performance Schema Requires pre-allocated memory 16

25 Memory Performance Schema Requires pre-allocated memory Memory allocated 5.7: After first use 5.6: At server startup Never gets free 16

26 Memory Performance Schema Requires pre-allocated memory Memory allocated 5.7: After first use 5.6: At server startup Never gets free Size of history tables is limited 16

27 Memory Performance Schema Information Schema Some queries require too much memory MySQL bug #72322 Fixed in , , 5.7.8,

28 Memory Performance Schema Information Schema Some queries require too much memory MySQL 8.0 re-designed tables No temporary tables for each query Index support MySQL Release Notes 16

29 Memory Performance Schema Information Schema Some queries require too much memory MySQL 8.0 re-designed tables Upgrade! 16

30 CPU Usage Performance Schema Fires code instruction each time when examined function called 17

31 CPU Usage Performance Schema Fires code instruction each time when examined function called Number of calls doubled 17

32 CPU Usage Performance Schema Fires code instruction each time when examined function called Number of calls doubled Binary log sync binlog fires fsync calls 17

33 Network ProxySQL and other proxy Additional layer 18

34 Internal Locks SHOW SLAVE STATUS 5.6: SHOW SLAVE STATUS NONBLOCKING Only Percona Server! MySQL WL #6402 in

35 Internal Locks SHOW SLAVE STATUS SHOW [GLOBAL] STATUS Performance slows when called in parallel MySQL Bug #42930 Fixed in Upgrade! 19

36 Internal Locks SHOW SLAVE STATUS SHOW [GLOBAL] STATUS Information Schema 19

37 Internal Locks SHOW SLAVE STATUS SHOW [GLOBAL] STATUS Information Schema Binary log 19

38 Applications with Graphical User Interface Use same instruments STATUS Variables MySQL and operating system log files Performance Schema 20

39 Applications with Graphical User Interface Use same instruments STATUS Variables MySQL and operating system log files Performance Schema Have same limitations 20

40 How to Find Golden Ratio?

41 You Cannot Turn All Log Files OFF! 22

42 Start with Necessary Minimum MySQL error log 23

43 Start with Necessary Minimum MySQL error log Log files of operating system 23

44 Turn ON by Demand General, slow query log files 24

45 Turn ON by Demand General, slow query log files InnoDB Monitors 24

46 Turn ON by Demand General, slow query log files InnoDB Monitors Consumers in Performance Schema Performance Schema already ON 24

47 Turn ON by Demand General, slow query log files InnoDB Monitors Consumers in Performance Schema Performance Schema already ON Data collection with help of pt-stalk 24

48 Turn ON by Demand

49 Dynamically General query/audit log 26

50 Dynamically General query/audit log Slow query log 26

51 Dynamically General query/audit log Slow query log InnoDB Monitors 26

52 Dynamically General query/audit log Slow query log InnoDB Monitors Performance Schema Instruments Consumers 26

53 Restart Required Error log 27

54 Restart Required Error log Binary log 27

55 Restart Required Error log Binary log Performance Schema 27

56 Task Dependent Optimization 28

57 Queries Optimization: Built-Ins Tables in P S: statements and stages mysql> SELECT THREAD_ID TID, SUBSTR(SQL_TEXT, 1, 50) SQL_TEXT, ROWS_SENT RS, -> ROWS_EXAMINED RE,CREATED_TMP_TABLES,NO_INDEX_USED,NO_GOOD_INDEX_USED -> FROM performance_schema.events_statements_history -> WHERE NO_INDEX_USED=1 OR NO_GOOD_INDEX_USED=1\G ********************** 1. row ********************** TID: SQL_TEXT: select emp_no, first_name, last_name from employee RS: RE: CREATED_TMP_TABLES: 0 NO_INDEX_USED: 1 NO_GOOD_INDEX_USED:

58 Queries Optimization: Built-Ins Tables in P S: statements and stages Slow query log # Time: :54:55 # User@Host: sveta.ritfest.ru [ ] Id: 9841 # Schema: rit Last_errno: 0 Killed: 0 # Query_time: Lock_time: Rows_sent: Rows_examined: Rows_affected: 0 # Bytes_sent: SET timestamp= ; SELECT data FROM usertable WHERE YCSB_KEY = user ; 29

59 Query Optimization: Graphical Query Analytics in PMM 30

60 Query Optimization: Graphical Query Analytics in PMM Advice for SQL in Solarwinds Query Analyzer (QUAN) in MEM Queries in VividCortex Others 30

61 Application Optimization InnoDB Monitors 31

62 Application Optimization InnoDB Monitors Performance Schema 31

63 Application Optimization InnoDB Monitors Performance Schema Data collection on demand Manually SHOW SELECT... pt-stalk FROM INFORMATION SCHEMA 31

64 Application Optimization 31 InnoDB Monitors Performance Schema Data collection on demand Manually SHOW SELECT... FROM INFORMATION SCHEMA pt-stalk Graphical OpenSource: PMM Commercial

65 Task Dependent Optimization Queries Application Locks 32

66 Locks P S MDL tables 33

67 Locks P S MDL tables InnoDB Monitors 33

68 Locks P S MDL tables InnoDB Monitors innodb print all deadlocks 33

69 Task Dependent Optimization Queries Application Locks Crashes 34

70 Crashes Core dumps 35

71 Crashes Core dumps General query log 35

72 Crashes Core dumps General query log Audit log 35

73 Crashes Core dumps General query log Audit log Binary log Only commands, changing data SELECT can cause crash too Crash can happen before command is written! 35

74 Task Dependent Optimization Queries Application Locks Crashes Unexpected behavior 36

75 Unexpected Behavior General query log 37

76 Unexpected Behavior General query log Binary log Only for written data 37

77 Summary

78 Always ON MySQL error log Operating system log files 39

79 Turn ON on Demand P S InnoDB Monitors General, slow, binary log 40

80 External Tools Percona Toolkit PMM MySQL Enterprise Monitor Interactive Demo for SolarWinds VividCortex MONYOG 41

81 Thank You!