Effective Testing for Live Applications. March, 29, 2018 Sveta Smirnova

Transcription

1 Effective Testing for Live Applications March, 29, 2018 Sveta Smirnova

2 Table of Contents Sometimes You Have to Test on Production Wrong Data SELECT Returns Nonsense Wrong Data in the Database Performance Issues Crashes 2

3 MySQL Troubleshooting Webinars I taught to Repeat problematic queries Crash servers 3

4 MySQL Troubleshooting Webinars I taught to Repeat problematic queries Crash servers Not what you want to do in production 3

5 Not Acceptable Updating wrong data Intended performance slowdowns Crashes 4

6 Sandboxes Slower machines Smaller disks Blocking backups 5

7 Concurrency Hard to imitate Load testing tools are not perfect 6

8 Sometimes You Have to Test on Production

9 Safety First Data must stay consistent No intended crashes Tests should not make performance worse 8

10 We will Discuss Wrong data SELECT UPDATE/INSERT/DELETE DDL 9

11 We will Discuss Wrong data Performance Slow queries Locking issues Hardware-related 9

12 We will Discuss Wrong data Performance Crashes 9

13 General Workflow Measure: record what is wrong Actual and expected results Query execution time Overall application performance 10

14 General Workflow Measure: record what is wrong Actual and expected results Query execution time Overall application performance Plan changes 10

15 General Workflow Measure: record what is wrong Actual and expected results Query execution time Overall application performance Plan changes Implement them May require maintenance window 10

16 General Workflow Measure: record what is wrong Actual and expected results Query execution time Overall application performance Plan changes Implement them May require maintenance window Measure again 10

17 Challenges We need to know what causes bad behavior 11

18 Challenges We need to know what causes bad behavior It is hard to find out 11

19 Challenges We need to know what causes bad behavior It is hard to find out Issue can have many potential solutions You can find one only after many iterations 11

20 Challenges We need to know what causes bad behavior It is hard to find out Issue can have many potential solutions You can find one only after many iterations You don t want to cause worse damage 11

21 Wrong Data

22 Wrong Data SELECT Returns Nonsense

23 There is No Harm in Experimenting This is just a query 14

24 There is No Harm in Experimenting This is just a query Does not change anything 14

25 There is No Harm in Experimenting This is just a query Does not change anything Works fast 14

26 There is No Harm in Experimenting This is just a query Does not change anything Works fast No effect on others 14

27 Just Run It Use all troubleshooting techniques you know 15

28 Just Run It Use all troubleshooting techniques you know Which we discussed Troubleshooting Slow Queries 15

29 Just Run It Use all troubleshooting techniques you know Which we discussed Troubleshooting Slow Queries And not 15

30 Measure Result of SELECT EXPLAIN output Desired result Can you use sandbox for tests? 16

31 General Query Analyzing Checklist Is data in the table? 17

32 General Query Analyzing Checklist Is data in the table? Does simplified query return correct result? 17

33 General Query Analyzing Checklist Is data in the table? Does simplified query return correct result? Try removing ORDER BY GROUP BY WHERE JOIN 17

34 General Query Analyzing Checklist Is data in the table? Does simplified query return correct result? Try removing ORDER BY GROUP BY WHERE JOIN Can you use part of data for tests? 17

35 Reducing Query Example Data in the table mysql> select a from t1; a rows in set (0.00 sec) 18

36 Reducing Query Example Problematic query mysql> select distinct ceil(a) from t1; ceil(a) Here is the problem! rows in set (0.00 sec) 18

37 Reducing Query Example Wrong data type mysql> select ceil(a) a from t1; a rows in set, 4 warnings (0.00 sec) Warning (Code 1292): Truncated incorrect DECIMAL value: Warning (Code 1292): Truncated incorrect DECIMAL value: Warning (Code 1292): Truncated incorrect DECIMAL value: Warning (Code 1292): Truncated incorrect DECIMAL value:... 18

38 Reducing Query Example Solution mysql> select distinct cast(ceil(a) as signed) from t1; cast(ceil(a) as signed) rows in set (0.00 sec) 18

39 Reducing Query Example Based on MySQL Bug #

40 Challenges Large datasets Removing WHERE leads to high IO 19

41 Challenges Large datasets Removing WHERE leads to high IO Solutions Limit data by primary key access Run EXPLAIN before query Adjust in parts 19

42 Wrong Data Wrong Data in the Database

43 You Have Wrong Data in the Database Incorrect UPDATE statement 21

44 You Have Wrong Data in the Database Incorrect UPDATE statement Concurrency 21

45 You Have Wrong Data in the Database Incorrect UPDATE statement Concurrency Wrong order of master events on slave 21

46 You Already Know Problematic Query All updates are in the binary log 22

47 You Already Know Problematic Query All updates are in the binary log With RBR use mysqlbinlog --verbose May help to discover logic error 22

48 You Already Know Problematic Query All updates are in the binary log With RBR use mysqlbinlog --verbose May help to discover logic error Audit, general, application logs 22

49 How to Deal with Tons of Updates? Quickly run through them 23

50 How to Deal with Tons of Updates? Quickly run through them Treat similar queries as same one UPDATE foo SET bar=8 WHERE baz=9 UPDATE foo SET bar=9 WHERE baz=8 23

51 How to Deal with Tons of Updates? Quickly run through them Treat similar queries as same one Search for not trivial queries DELETE FROM t1, t2 USING t1, t2; 23

52 How to Deal with Tons of Updates? Quickly run through them Treat similar queries as same one Search for not trivial queries DELETE FROM t1, t2 USING t1, t2; Will work only for matching rows 23

53 How to Deal with Tons of Updates? Quickly run through them Treat similar queries as same one Search for not trivial queries Examine logs for number of affected rows Slow query log in Percona Server has it performance schema.events statements * Performance Schema for MySQL Troubleshooting 23

54 Concurrency and Conflicts Order of events 24

55 Concurrency and Conflicts Order of events Transaction isolation level 24

56 Concurrency and Conflicts Order of events Transaction isolation level Content of binary log 24

57 Concurrency and Conflicts Order of events Transaction isolation level Content of binary log Application logic Log files What do you expect from it? 24

58 Measure What was wrong DML queries What changed How result changed 25

59 Test safely update City, Country set City.Population=Country.Population+1000 where City.CountryCode=Country.Code and Continent= Oceania and Region= Polynesia ; 26

60 Test safely update City, Country set City.Population=Country.Population+1000 where City.CountryCode=Country.Code and Continent= Oceania and Region= Polynesia ; Convert to SELECT select City.Population, Country.Population, Country.Code from City, Country where City.CountryCode=Country.Code and Continent= Oceania and Region= Polynesia ; 26

61 Test safely update City, Country set City.Population=Country.Population+1000 where City.CountryCode=Country.Code and Continent= Oceania and Region= Polynesia ; Convert to SELECT Multi-statement transaction and rollback mysql> begin; Query OK, 0 rows affected (0.00 sec) mysql> update City, Country set City.Population=Country.Population > where City.CountryCode=Country.Code and Continent= Oceania and Region= Polynesia ; Query OK, 12 rows affected (0.01 sec) Rows matched: 12 Changed: 12 Warnings: 0 mysql> rollback; Query OK, 0 rows affected (0.13 sec) 26

62 Test safely update City, Country set City.Population=Country.Population+1000 where City.CountryCode=Country.Code and Continent= Oceania and Region= Polynesia ; Convert to SELECT Multi-statement transaction and rollback Experiment with slave 26

63 Test safely update City, Country set City.Population=Country.Population+1000 where City.CountryCode=Country.Code and Continent= Oceania and Region= Polynesia ; Convert to SELECT Multi-statement transaction and rollback Experiment with slave Be ready to revert changes! 26

64 Performance Issues

65 Measure Most important for Performance Issues 28

66 Measure Most important for Performance Issues Application response time How long web page loads? How long took AJAX request? How much took request to database? 28

67 Measure Most important for Performance Issues Application response time Slow query log 28

68 Measure Most important for Performance Issues Application response time Slow query log Throughout data collection pt-stalk Performance Schema Manual 28

69 What is Wrong with Hardware Metrics? They say nothing about state 29

70 What is Wrong with Hardware Metrics? They say nothing about state High CPU Is database starving? Is it running many client threads? 29

71 What is Wrong with Hardware Metrics? They say nothing about state High CPU Low CPU Is database idle? Did you limit concurrency? 29

72 What is Wrong with Hardware Metrics? They say nothing about state High CPU Low CPU High Memory Usage Too large buffers? Too many temporary tables? 29

73 What is Wrong with Hardware Metrics? They say nothing about state High CPU Low CPU High Memory Usage You got the idea 29

74 Measure in Right Time! Baseline: for normal load Makes sense to understand hardware limits 30

75 Measure in Right Time! Baseline: for normal load For troubleshooting: for real load 30

76 Test Workflow Collect data 31

77 Test Workflow Collect data Plan changes Record everything you modify Choose time 31

78 Test Workflow Collect data Plan changes Record everything you modify Choose time Make changes 31

79 Test Workflow Collect data Plan changes Record everything you modify Choose time Make changes Collect again 31

80 Test Workflow Collect data Plan changes Record everything you modify Choose time Make changes Collect again Compare 31

81 Can I get Situation Worse? Yes, you can! 32

82 Can I get Situation Worse? Yes, you can! But you are already in trouble 32

83 Can I get Situation Worse? Yes, you can! But you are already in trouble You have to do something 32

84 Safe Way to Test Understand what you change 33

85 Safe Way to Test Understand what you change Test in session 33

86 Safe Way to Test Understand what you change Test in session Change dynamic variables 33

87 Safe Way to Test Understand what you change Test in session Change dynamic variables Store changes Configuration file 8.0+: SET PERSIST 33

88 Safe Way to Test Understand what you change Test in session Change dynamic variables Store changes Configuration file 8.0+: SET PERSIST Always measure effect! 33

89 If Things Went Wrong Revert changes 34

90 If Things Went Wrong Revert changes Record what caused failure 34

91 If Things Went Wrong Revert changes Record what caused failure Easy if you changed in right time 34

92 Changing in Less Busy Time May work Follow regular workflow Compare with data, collected for similar load 35

93 Changing in Less Busy Time May work Follow regular workflow Compare with data, collected for similar load Won t work Issue happens only during high activity Number of active writing clients is greater than your disk and CPU can handle InnoDB Buffer Pool aggressively purges old data only when it cannot do it in idle time 35

94 Changing in Less Busy Time May work Follow regular workflow Compare with data, collected for similar load Won t work Issue happens only during high activity Number of active writing clients is greater than your disk and CPU can handle InnoDB Buffer Pool aggressively purges old data only when it cannot do it in idle time You may try to scale Use case for sandbox Not always work 35

95 Upgrade Performance: Issues Application runs slower than before 36

96 Upgrade Performance: Issues Application runs slower than before Reason is unknown 36

97 Upgrade Performance: Issues Application runs slower than before Reason is unknown A lot of changes Hardware Operating System MySQL version 36

98 Upgrade Performance: Best Practices Record performance stats for old version pt-stalk Slow query log for few days 37

99 Upgrade Performance: Best Practices Record performance stats for old version pt-stalk Slow query log for few days Upgrade 37

100 Upgrade Performance: Best Practices Record performance stats for old version pt-stalk Slow query log for few days Upgrade Let it run 37

101 Upgrade Performance: Best Practices Record performance stats for old version pt-stalk Slow query log for few days Upgrade Let it run Measure again 37

102 Upgrade Performance: Best Practices Record performance stats for old version pt-stalk Slow query log for few days Upgrade Let it run Measure again You will know exactly what went wrong! 37

103 Calling Support Collect data At the problematic time pt-stalk in daemon mode pt-stalk --daemonize --iterations=2 --sleep=30 --dest=/path/where/you/want/logs/stored --user=root --password=mysql-root-pass --function=status --variable=threads_running --threshold=25 38

104 Calling Support Collect data At the problematic time pt-stalk in daemon mode Identify how it affects your application 38

105 Calling Support Collect data At the problematic time pt-stalk in daemon mode Identify how it affects your application Read answers about changes you need to make 38

106 Calling Support Collect data At the problematic time pt-stalk in daemon mode Identify how it affects your application Read answers about changes you need to make Apply them incrementally 38

107 Calling Support Collect data At the problematic time pt-stalk in daemon mode Identify how it affects your application Read answers about changes you need to make Apply them incrementally Measure result 38

108 Crashes

109 You have to Make Changes You are in trouble already There is nothing to be more afraid of 40

110 What to Change? Query 41

111 What to Change? Query Scenario 41

112 What to Change? Query Scenario Options 41

113 What to Change? Query Scenario Options Hardware 41

114 Log Files Error log 42

115 Log Files Error log Core file 42

116 Log Files Error log Core file Troubleshooting MySQL Crashes Debug symbols Analyze hints 42

117 Workflow Study logs 43

118 Workflow Study logs Make changes 43

119 Workflow Study logs Make changes Repeat workload 43

120 Workflow Study logs Make changes Repeat workload Do best to imitate in sandbox! 43

121 Crash in Random Places Check if mysqld is the reason 44

122 Crash in Random Places Check if mysqld is the reason Monitor OS around crash times 44

123 Crash in Random Places Check if mysqld is the reason Monitor OS around crash times PMM or similar tool is a must Impractical to have scheduled job 44

124 System and Crashes Server state 45

125 System and Crashes System Overview: 2.7 Weeks before 45

126 System and Crashes System Overview: 2.7 Weeks before 45

127 System and Crashes System Overview: 2.7 Weeks before 45

128 Summary You must monitor your system 46

129 Summary You must monitor your system Keep historic logs, so you know what changed Slow query logs pt-stalk collections PMM 46

130 Summary You must monitor your system Keep historic logs, so you know what changed Slow query logs pt-stalk collections PMM Make changes sequentially 46

131 Summary You must monitor your system Keep historic logs, so you know what changed Slow query logs pt-stalk collections PMM Make changes sequentially Avoid to replace everything at a time 46

132 Summary You must monitor your system Keep historic logs, so you know what changed Slow query logs pt-stalk collections PMM Make changes sequentially Avoid to replace everything at a time Measure before and after the change 46

133 More Information Troubleshooting Webinars at Percona MySQL is crashing: a support engineer s point of view pmmdemo.percona.com/ 47

134 Thank you!