Beyond the Ordinary. mail2eventdb. handling for the Netways Event

Transcription

1 Beyond the Ordinary mail2eventdb handling for the Netways Event

2 About me Martin Fürstenau Senior System Engineer at Oce Printing Systems, Poing lecturer adult education center IT and Unix experience over 20 years. Linux from first 1.x on, Nagios experince over 5 years Administration of mainframes, Unix/Linux, Windows, databases, datacenter automation, planning datacenters, network administration,e-learning, etc. Currently monitoring around 1500 hosts with nearly services from a single CentOS cluster. 2

3 Why the heck ?? Appliances often only send s Applications often can only send s Other monitoring applications should be integrated - easy with as a first step It is easy for developers to integrate notification - from a simple script to a highly sophisticated monster (coffee) application. is the standard for kind of low level notification 3

4 And why Netways eventdb?? Pro: First - it was the idea behind it: a single interface/plugin for events. Second - it was the idea behind it. Contra: Suboptimal documentation Due to its history (syslog handling) some misinformation/ not complete information in the docs. Some strange drawbacks in the plugin (check_eventdb.pl) 4

5 Decision Implement it as agent for Netways eventdb Some improvements (patches) for the web-frontend index.php (mainly separation of styles and logo) After having mail2eventdb ready I discovered that check_eventdb.pl had only implemented a very small subset of all the functions I expected useful. Therefore: Nearly complete rewrite of check_eventdb.pl 5

6 Design goals No temporary data should be stored on disc for analysis Textmails, MIME and Multipart-MIME should be processed Nested MIMEs should be unsupported due to the fact that it is unlikely that an alarm is part of a forwarded mail. HTML should be converted to flat text. German umlauts should be translated. A general conversion (ISO or UTF8 to 7-Bit Ascii or so) should not be implemented (too much effort). base64 coded messages (for example Microsoft MOM) should be decoded. 6

7 Why so complex? Normally, system messages will be sent as plain text But sometimes for example an application running on an AS400 or so is sending s to the admin And for example the responsible admin can add an address but cannot change the format. Or for some reasons MOM is also used but cannot contact your incident management system. However Nagios can do it. Or you have an application sending able to sent s but nothing else. In any case it can happen that you can add a receiver but you can't change anything else 7

8 Features (1) Accepts Internet mail (text), MIME and Multipart MIME messages (nested MIME not supported) Extracts sender, subject and content (text) Converts HTML in message content to text Converts base64 coded message content to text Converts german umlauts to ASCII but there is no general conversion (ISO or UTF8 to 7-Bit Ascii or so) Has an API to add filter scripts written in Perl or any other language to extract and reformat host and/or application from subject and/or content data from subject and/or content reformat/filter data and deliver the data back to the program 8

9 Features (2) Writes the reformatted/filtered data (host, errorcode, content) to the eventdb It is possible (but normally it should not be done) to modify facility and type/level in your filter Includes an option to store mails in a given directory (because you need some stored mails to develop your filter) Includes a testmode for testing filters while developing them. 9

10 How the agent works The agent receives the message from the mail program (postfix) via pipe. config files are used for test mode for production mode If necessary the message will be converted to plain text The message will be split into 3 parts: Sender Subject Content The relation between the sending host and the sender has been defined in the config file. The optional relation between the sending application and the sender has also been defined in the config file. 10

11 How the mail flows Postfix mail2eventdb Call Filter Filter EventDB Mailstore 11

12 Installation & Configuration - Basics mail2eventdb comes as a zipped tar file. It includes 3 subdirectories: conf bin filter 2 configuration files (test and production) are located in the conf directory. Sample filter skeletons are located in the filter directory 12

13 Installation & Configuration - Variables How many characters from the content will be written to the logfile in case of trouble BEWARE!! If empty the whole content will be written to logfile. $content_logsize = "250"; This is the default for the logfile. Can also be changed via commandline (switch -l). my $logfile_default = "/var/log/mail2eventdb/mail2eventdb.log"; Directory where configuration and filter are stored. my $m2edb_dir= "/etc/nagios/eventdb/mail2eventdb/"; Central configuration files my $m2edb_cfg = $m2edb_dir."conf/mail2eventdb.cfg"; my $m2edb_cfg_t = $m2edb_dir."conf/mail2eventdb_t.cfg"; The name of the directory where the filters reside my $filter_dir = $m2edb_dir."filter"; 13

14 Installation & Configuration - Variables Set the default filter. If empty no default filter will be used. my $filter_default = "default.fltr"; Set your databasename for eventdb here. my $eventdb = "eventdb"; Set your databasehost for eventdb here my $eventdb_host = "localhost"; Set your databaseuser for eventdb here my $eventdb_user = "eventdb"; Same for the database password my $eventdb_pass = "is-an-event"; 14

15 Installation & Configuration - Variables Facility used for eventdb. Not necessary to follow syslog conventions because the database is addressed directly. my $eventdb_facility = "mail2eventdb"; Default type for eventdb. Can be changed to your needs. But if you changed it modify index.php. my $eventdb_type = "mail"; 15

16 The configfiles [LogMail] # 0 means mail will not be stored (normal processing) # 1 means mail will be stored store_mail = 1 # BEWARE!! The mailstore should always contain the the absolute path. # Otherwise it may be that you have to search for your mailstore (or do you # know where your program is located when it is started from Postfix?) mail_store = /tmp/testmail # Please use ALWAYS lowercase letters for mail addresses even in the case the # address used in your mail to be analyzed uses uppercase letters. Otherwise # you will received a error message [Mail2Host] duck@localhost.something.de = fileserver1 duck@ducktales.something.de = fileserver2 root@ocalhost.something.de = mailserver1 [Mail2Appl] duck@localhost.something.de = main-erp-system 16

17 What is a filter? A filter will always be called from mail2eventdb Sender, Subject and Content are handed over Filtered values for error_host, program priority and other values including originial or modified message related to eventdb will be given back. The data given back from the filter is in an appropriate format to be inserted into the eventdb Filters are basically Perl programs. For those not able to write Perl programs there is a two step filter: The Perl program is calling a shell script (or whatever you prefer) via system call. Your script gives the data back to the Perl program. The Perl program gives the data back to mail2eventdb 17

18 Filter Names Filter names are built based on host and application names determined from the config files Example: Mailaddress belongs to host harry then the name of the file is harry.fltr If belongs also to application mainerp-system the name of the filter would be: harry_main-erp-system.fltr 18

19 Testmode You need 3 things for developing new filters Messages (Stored with store_mail = 1 in the config) mail2eventdb -t for testmode a config file for test How it is done: cat File_with_mail mail2eventdb -t The output will be displayed onto your monitor, not inserted into the database 19

20 Some words about some additions/modifications The web-frontend The web-frontend of Netways EventDB was enhanced by some small patches (included in the SVN) Some modifications are currently available separately from The logo is separated from the program code The styles have moved to a separate stylesheet This makes customizing easier. 20

21 And some words about a replacement - check_eventdb2.pl As mentioned earlier the original version was very limited. The code was cleaned up and enhanced/rewritten. New functions: - -r --regexp will treat message strings as regular expressions - -s --show shows the found messages in Nagios. Length of string can be set in $message_length and multiline output can be HTML style or Nagios 3 style. See $ml_show_msg. - --acknowledged will include acknowledged entries in the select statement - --version added to print program version - Values for priority can be added as single scalar (for example emerg) or as a comma separated list (for example emerg,alert,crit) 21

22 And some words about a replacement - check_eventdb2.pl More new functions - acting as eventhandler: In case of using check_eventdb as an event handler all three variables (--servicestate, --servicestatetype and --serviceoutput) nust be used together. Records found will be acknowledged. --servicestate Contains $SERVICESTATE$ from Nagios if started as eventhandler for acknowledging or deleting records/entries --servicestatetype Contains $SERVICESTATETYPE$ from Nagios if started as eventhandler for acknowledging records/ entries --serviceoutput Contains $SERVICEOUTPUT$ from Nagios if started as eventhandler for acknowledging or deleting records/entries. $SERVICESTATE$ MUST be OK or UP, $SERVICESTATETYPE$ MUST be hard and $SERVICEOUTPUT$ should contain the output of your "submit passive check result" together with the name of the operator (for example "problem fixed - D.Duck") 22

23 And some words about a replacement - check_eventdb2.pl More new functions: - variable matches removed. It was replaced by $rowcnt because we let not count the database. We count the rows received from the database. - --maxage Maximum age for records use for alerts - --recover The messagestring following this will cause a recovery. Wildcards are allowed. If -r is set the string will be interpreted as a regular expression. -H must been set. And the message containing the recovery string must be the latest. If there are alerts newer than the recovery string only this alerts are used. --recover_regexp Interpret recoverstring as regular expression (awk/sed style). Works only with --recover set. --recover_prio The messagestring following this will contain the priority for recovery -H must been set. --recover_prog This will contain a name of a program causing the recovery in case the recovery message is not sent by the program causing the error. -H must been set. 23

24 Where to find all the stuff? Netways EventDB: mail2evendb: check_eventdb2.pl: index.php: 24

25 And now? Please stay tuned while we play a little bit with it 25

26 The End Thanks for either your attention or not snoring Any questions - or buried by information? 26