Automated System Monitoring

Transcription

1 Automated System Monitoring Josh Malone Systems Administrator National Radio Astronomy Observatory Charlottesville, VA 2

2 WHAT IS AUTOMATED MONITORING? 7

3 Automated Monitoring Workflow 8

4 Monitoring Packages: Open Source Pandora FMS Opsview Core Naemon Captialware ServerStatus Core Sensu All Trademarks and Logos are property of their respective trademark or copyright holders and are used by permission or fair use for education. Neither the presenter nor the conference organizers are affiliated in any way with any companies mentioned here. 9

5 Monitoring Packages: Commercial Nagios XI Groundwork Sensaphone (IMS 4000) Statseeker PRTG network monitor CopperEgg WhatsUp Gold PRTG network monitor op5 (Naemon) All Trademarks and Logos are property of their respective trademark or copyright holders and are used by permission or fair use for education. Neither the presenter nor the conference organizers are affiliated in any way with any companies mentioned here. 10

6 What can monitoring do for you? Spot small problems before they become big ones Checklist when restoring from a power outage Learn about outages before your users do Gives you better problem reports than users Problems you might never spot otherwise Failed HDDs in RAIDs Full /var partitions Logs not rotating System temperature rising 11

7 Without Monitoring With Monitoring dhcp out of leases The Internet s down - fix it!!! dhcp server down dns server not responding ethernet switch down ISP link down / saturated 12

8 Without Monitoring With Monitoring connectivity issues ZOMG! Our web site is down! O Noes!!! web server down apache not running web server disk full server load too high 13

9 What can monitoring do for you? Capacity planning Performance data can generate graphs of utilization RAM, Disk, etc. Availability reports - CAUTION Easy to generate -- even easier to generate wrong Make sure your configurations actually catch problems Will also include problems with Nagios itself :( If you re going to quote your availability numbers (SLAs, etc.) make sure you understand what you re actually monitoring. 14

10 ENVIRONMENT MONITORING 15

11 Temperature Smoke Water Humidity Motion Door / closure Mains power Environment Monitoring 16

12 Environment Monitoring Sensaphone IMS-4000 Connect sensors to measure desired metrics IP-based Nodes can connect remote sensors Wireless sensors available Notification via POTS line and voice dialer as well as SNMP support Use my plugin w/ Nagios! 17

13 ServersCheck Temp, Humidity Wireless (2.4GHz) Environment Monitoring NetBotz Temp, humidity, smoke, water, vibration, doors, cameras 18

14 NAGIOS 19

15 Nagios Open source host / service monitoring package Nagios Ain't Gonna Insist On Sainthood Originally released in 1999 as NetSaint Available in 2 versions: Core and XI Nagios Core: Open-source, freely available Nagios XI: Commercial Free license for up to 7 hosts Available as source installer or VMware appliance 20

16 Nagios Architecture 21

17 What s a plugin? Plugins actually run the service or host checks. Each plugin monitors a different type of service Data from plugin is communicated to Nagios using a (very) simple API Plugins can also report Performance Data (perfdata) to be graphed or tracked Requires a perfdata add-on (or Nagios XI) Plugins can be written in any language Perl plugins can run using Nagios s embedded perl interpreter for increased performance 22

18 Where to Monitor a Service? Host ping TCP port 443 SSL handshake HTTP return code Page load time Page content Is server host alive? Is Apache listening? Is SSL functional? Is the page found? Does page load quickly? Is it the right page? 23

19 Custom Plugins Nagios can monitor anything you can write a script to check Simple API: just write text to stdout and exit with a value You can write plugins in ANY language you choose! bash, python, tcl, expect perl (Nagios has embedded perl interpreter for speed) C, C++ Huge collection of plugins available at: Be wary of some community plug-ins! Test first!!! 24

20 Performance Data Metrics about the state of the service Can be used to generate graphs showing trends, etc. Performance data processing requires some external add-on like PNP4Nagios 25

21 My Plugins Framework Perl Net::SNMP Plugin for APC Smart-UPS, 26

22 Agent-less vs Agent-full Checks Agent-less No agent installed on the monitored host All check plugins run on the monitoring server Service to be monitored must be network-accessible Default mode of Nagios Agent-full Must install agent on server to be monitored Check logic runs on monitored host Can access services nonnetwork services SNMP can be a powerful agent for checks Server-specific agents 27

23 USING NAGIOS 28

24 About Nagios Replacements When Nagios went commercial, the open-source community decided that it needed not one, not two, but three replacements for Nagios: Icinga and Naemon (forks of Nagios) and Shinken (a drop-in replacement). Most linux distros are now shipping one or more of these compatible replacements rather than the official Nagios Core. Not a single distro I checked is shipping Nagios 4. Either Shinken, Naemon or Icinga should work the same as the material covered here, but I have only briefly tested Icinga and have not tested Shinken or Naemon at all. 29

25 Navbar Overview Main window 30

26 The Tactical Overview Displays overview of monitored services and hosts Shows if Any services / hosts have notifications disabled Any services / hosts are flapping Active / passive checks enabled / disabled Warning / Critical / Okay breakdown 31

27 The Tactical Overview 32

28 Host summary Services View Service summary 33

29 Click on Services - Critical 34

30 Host and Service Groups Organize services or hosts into groups by function, etc. Can disable alerts, schedule downtime, etc. on whole group Can show availability report for a whole group Group services by desired reporting capability Groups get a unique URL so you can send a single link to check on a group of hosts Great for PHBs! Also great for delegated IT departments 35

31 Service Groups 36

32 Acknowledging an Outage Click on service name (or hostname) that has the problem Under Service Commands Click Acknowledge this service problem You must enter a comment about why you are acknowledging the problem (i.e., Bob is working on it ) Click Commit 37

33 Acknowledging an Outage Click Here 38

34 Acknowledging an Outage 39

35 SMS pages Configure a contact to use an -to-sms gateway Some carriers require an MMS gateway to process the From address 40

36 Add-ons to Consider PNP4Nagios - Performance data graphing NConf - Web-based configurator for Hosts, Services, etc. NagiosQL - Web-based admin tool for Nagios NDOUtils - Export data from Nagios to MySQL 41

37 THANK YOU! Previous talks available at: 42