Event Console Integration , Sven Panne Check_MK Conference #4

Transcription

1 Event Console Integration , Sven Panne

2 Two Kinds of Monitoring Status-based Event-based Pushing/pulling states Pushing of messages Classic Check_MK Independent of state Event Console Both kinds are useful! 2

3 Design Considerations High frequency of incoming messages 3 Human-manageable amount of actual events Not a high-performance syslog archive

4 Architectural Overview I Basic Pipeline EC SNMP syslog spool local 4 Trap Parser syslog Parser Shell Script Rule Actions Engine Notifications

5 Architectural Overview II Persistence & GUI GUI Monitoring Core Livestatus EC History SNMP syslog spool local 5 Events Trap Parser syslog Parser Views Events History Details Shell Script Rule Actions Engine Notifications

6 Architectural Overview III There and Back Again Monitoring Core GUI Views Events History Details Livestatus EC Check History SNMP syslog spool local Event Service Events Trap Parser syslog Parser Shell Script Rule Actions Engine Notifications Logwatch 6

7 Architectural Overview IV Host Information GUI Monitoring Core History SNMP syslog spool local Hosts, DownEvents times Trap Parser syslog Parser Views Events History Details Livestatus EC Check Event Service Shell Script Rule Actions Engine Notifications Logwatch 7

8 Events are useful, even during host downtimes Still generated, but marked specially Three chunks yet Automatically archived after downtime to be completedtactical overview considers them handled Actions can be skipped 8

9 Further Improvements Finer configuration of limits More powerful rewriting Export of rule packs via MKPs 9

10

11 Background jobs , Tom Bärwinkel

12 Examples of long running processes Create Report 12 Bake Agents Host renaming

13 Problem Apache process timeouts 13

14 The usual workaround Let s just increase the timeout locate files increase timeout update 14 restart

15 Still a problem - What s the status? No user feedback 15 + Blocked processes + Apache restarts

16 Solution - separation of concerns GUI 16 configuration start jobs show progress Background jobs Common Interface do actual work send updates

17 How it looks in practice - Reports 17

18 How it looks in practice - Reports 18

19 The Background jobs overview 19

20 Special rules for critical jobs Affected jobs: bake agents rename host Only one job allowed Error page with details Require acknowledgement 20

21 Upcoming background jobs Download of agent output Service discovery SLA calculations

22

23 Time-specific check parameters , Konstantin Büttner

24 Use case Recurring events lead to conditions that would be abnormal during other times Eg. backup jobs etc. 24

25 Predictive Levels? 25

26 Solution Timespecific parameters Define parameters for time periods Generic solution works for all check plugins 26

27 How does it work? 27

28 How does it work? 28

29 29

30

31 Customizable graph layouts , Konstantin Büttner

32 Our objective in graphing Traditional graphing system are for customizing graphs But: day-to-day, we d rather just have graphs Tweaking graphs is fun, but time consuming Our objective: Make our data useful out of the box 32

33 A bit of history - PNP4Nagios PNP graph In the Olden Days Graphs were defined for each check type Templates in PHP ( ノ ಠ 益 ಠ) ノ彡 No semantics, no uniformity* No customizability** *except for checks sharing templates **unless you wrote your own templates 33

34 A bit of history - Metrics Metrics graph A new dawn 34 Most visible: New look Most important: Semantics Checks no longer have graphs...but values have semantics Automatic graphs Custom graphs

35 Now: More customizability Unified options for dashlets, views, reports (Werk #5569) 35

36 36

37 37

38 Now: More customizability Unified options for dashlets, views, reports (Werk #5569) Vertical scaling and mirroring customizable (Werk #3920) 38

39 39

40 40

41

42 Performance improvements , Konstantin Büttner

43 Why is a performance focus critical? The typical Check_MK setup keeps growing more services, more sites, more users Better scalability makes new things possible 43

44 How to make things go faster Avoid unnecessary work Optimize Make things feel faster

45 Avoid unnecessary work - Tabs A typical browser: 50 open tabs You don t use them all but they all make regular livestatus queries, though New: Only update active tabs/windows (Werk #4753) Benefit: No more wasted data transfers 45

46 Avoid unnecessary work s Graphs can make s reasonably large Multiply that by the number of recipients... What s a mail server for? (Werk #4813) Benefit: Notification system becomes more scalable 46

47 Avoid unnecessary work Sites Frequent scenario: multi-site setup For many users, only few sites are actually relevant Yet, GUI in the past contacted all sites for every user No more: Now configureable for each user (Werk #4921) 47

48 Optimize BI Large aggregations now compile faster (Werk #5142) Magic - it s the algorithm, stupid Further reworking already planned 48

49 Optimize liveproxyd Previously: # The main loop of the daemon goes here One python process GIL limits scalability Now what? 49

50 Optimize liveproxyd Now: One process per site, master process to manage Process view: More resilient, more performant through use of multiple CPUs (cf. Werk #4901) 50

51 Make things feel faster - Graphs Previously: View with many Graphs? Better wait for all the data Now: Load the view with placeholders, update graphs asynchronously 51

52 Take-home message Attention to detail pays off Continuously improving If you have lots of sites and lots of users, Werk #4921 may help you

53