Manage MySQL like a devops sysadmin. Frédéric Descamps

Transcription

1 Manage MySQL like a devops sysadmin Frédéric Descamps Webinar Oct 2012

2 Who am I? Frédéric Managing MySQL since 3.23 (as far as I remember) devops believer

3 To start with... Let's use devops like slides...

4 Let's use devops like slides... 4

5 from 5

6 6

7 to 7

8 8

9 devops is a movement willing to help aligning IT with company needs (starting by aligning devs and - Gildas Le Nadan - The Devops movement, Mixit Lyon

10 devops is about communication! 10

11 devops is about communication! between dev and ops 11

12 Small and frequent successful iterations devs ops This is one of the most fundamental ideas in devops: 12 the feedback loop!

13 We can say that devops is bringing together people and practices from development and operations 13

14 I am a happy op! I am a happy dev! We can say that devops is bringing together people and practices from development and operations 14

15 devops now = being professional improving the collaboration by increasing trust and respect 15

16 /me likes! HOW can I reach that goal? 16

17 devops requires visibility! 17

18 Everybody should have a status overview Is there any problem? How are the test going? How is performance? 18

19 devops is about CAMS 19

20 CULTURE AUTOMATION 20

21 What techniques can I use? 21

22 Monitoring Infrastructure as Code Continuous Improvement 22

23 What about DBAs? 23

24 Join the team, follow the movement! 24

25 You are already between devs and ops 25

26 I am a happy dba! You are already between devs and ops 26

27 But we are still closer to the dark side of the force: sysadmins 27

28 But we are still closer to the dark side of the force: sysadmins 28

29 How can we apply devops ideas to our dba job? 29

30 1. communication 2. configuration management 3. testing 4. visibility 30

31 1. Communication Usually DEVS communicate with managers, quality managers, products managers,... 31

32 OPS communicate with support, ISP's and... their servers 32

33 DBAs don't communicate 33

34 The interaction between developers, sysadmins, dbas and all the actors should begin from the start of any projects 34

35 35

36 36

37 Discuss everything database related with developers and sysadmins: disks and raid design schema design & partitions additional columns & indexes tables repair,... 37

38 2. configuration management Can we automate DBAs job? Should we? Even if we have only one database? 38

39 39

40 40

41 A lot of things to configure binaries databases configs cron jobs monitoring grants replication 41

42 Documentation is needed and must be up to date 42

43 Needs to be reproducible quickly without missing anything! 43

44 Keep track of any changes Mind the gap 44

45 3. testing Test every changes every new package... EVERYTHING 45

46 4. visibility Expose monitoring, trending, test results and project status to EVERYONE in your company!! Everybody should know why the database is slooooow! 46

47 Use all these metrics to plan ahead all the changes! NOT ALONE BUT WITH EVERYBODY 47

48 notions practices ideas methodologies 48

49 We are DBAs we work with data, we need something more concrete! 49

50 We are DBA's we work with data, we need something more concrete! 50

51 TRUE! 51

52 communication 52

53 Use an agile method to handle your projects 53

54 For operations a good method is kanban 54

55 55

56 56

57 Use dashboards and make them available to EVERYONE 57

58 share all your dashboards with everybody in your company 58

59 configuration management 59

60 consider your infrastructure as code 60

61 Use puppet (or any other configuration management tool) to deploy everything related to your database 61

62 Save your puppet recipes into a version control system 62

63 but also save your database schema changes in that system 63

64 test 64

65 65

66 What should you test as a devops dba? 66

67 Test if you can deploy everything and if you didn't forget anything 67

68 Test also the performance! Is it faster with this change? 68

69 use Percona Toolkit 69

70 automate your tests 70

71 visibility 71

72 monitoring 72

73 Monitor your database with an alerting system (nagios). Disk/IO, CPU load, Network traffic, disk usage 73

74 But do you want to be woken up at night because replication of the machine taking backups is 60sec behind? 74

75 NO! 75

76 Then do you want to be woken up in the middle of the night because they are 1000 more selects than usual? 76

77 NO! 77

78 Your alerting should answer those questions: Is the database server running? Does it allow connections? Is replication lagging on production slaves? 78

79 trending 79

80 The most interesting facts for a DBA are the trending results! Use cacti... with Percona Monitoring Plugins They provide very good metrics 80

81 Correlate changes in the trending results with your configuration changes... everything should be tracked 81

82 others? 82

83 You can also use other devops oriented tools like : 83

84 with kanban plugin puppet-dashboard 84

85 If there is only one thing you remember from this presentation 85

86 Yes, even a schema change MUST be considered like a migration! 86

87 review it test it plan it do it 87

88 and now that we have a devdbaops team 88

89 we can start working on really interesting tasks for everyone 89

90 the smooth migration 90

91 When this goal is achieved DEVS have less pressure OPS have less pressure DBAs have less pressure 91

92 collaboration = less pressure 92

93 93

94 application present future v 1.9 v 2.0 db schema v 1.9 v

95 application present future v 1.9 v 2.0 db schema v 1.9 v

96 present application v 1.9 v 2.0 db schema v 1.9 v

97 present application v 1.9 Issue!! v 2.0 db schema v 1.9 v

98 application v 1.9 v 2.0 db schema v 1.9 v

99 application v 1.9 v 2.0 Issue!! db schema v 1.9 v

100 application v 1.9 v 2.0 db schema v 1.9 v

101 This can only be the result of 101

102 collaboration between DEVS, DBAs & OPS 102

103 demo 103

104 git puppet Percona server Jenkins VirtualBox vagrant mcollective 104

105 Change the schema (add an index for example) Commit! 105

106 Launches a new virtual machine from scratch using vagrant Install everything and populate a test db 106

107 Start some basic tests with cucumberpuppet to see if all puppet code was correctly performed No error is a success Otherwise is a fail 107

108 Then start some tests with perconaplayback and/or pt-query-digest and compares to old values Faster is a success Slower is a fail 108

109 Projects links: Photo Credit:

110 You can find all these recipes here!

111 Speaking at In London, December 3rd & 4th

112 Speaking at 40% off passes using the discount code MySQL-UK In London, December 3rd & 4th

113 @lefred