CARMA Logging. Marc Pound Doxygen ICD at mpound/carma/loggingapi.html. 1. Introduction

Transcription

1 draft February 13, 2003 CARMA Logging Marc Pound Doxygen ICD at mpound/carma/loggingapi.html 1. Introduction CARMA Logging is based on the open source project log4cpp. 1 Log4cpp is intended to be a C++ implementation of the Java log4j package. 2 There are three main classes in log4cpp: Categories, Appenders and Layouts. These three types of components work together to enable logging of messages according to message type and level, and to control at runtime (via a configuration file) how these messages are formatted and where they are reported. Category (now called Logger in log4j) contains the methods the developer will call to push a log message through the system. These methods have names that indicate the priority level of the message passed to them, e.g. info(const string& message), crit(const string& message). Priority levels correspond to the priorities of syslog(3) as given in Table 1. The Appender class indicates the output channel for the message, e.g. syslog, file, rolling file. A Category may have more than one Appender, and Appenders may filter messages based on their priority. The Layout class is simply the output format of the message. Categories follow a named hierarchy, or logging space, that is categorized according to some developerchosen criteria. A category is an ancestor of another category if its name followed by a dot is a prefix of the descendant category name. A category is a parent of a child logger if there are no ancestors between itself and the descendant cateory. For instance, the category named carma.monitor is child of the category named carma. The nice thing is that the developer does not have to do anything to create the hierarchy other than decide where to put the dot. The Category constructor parses the name and inserts the new category in its proper place in the hierarchy. 2. Design For CARMA Logging, the default output channel will be syslog and the default Layout will be CarmaLayout, a subclass of log4cpp::layout. Under normal operation, any CARMA subsystem process can instantiate a Logger object to pass it logging messages to the syslog daemon (syslogd) on the Array Control 1 The log4cpp API is at mpound/carma/log4cpp/html/api. 2 A good description of how log4j and thus log4cpp is intended to work is at You are strongly urged to read this description, since I m not going to repeat it all here. You should also familiarize yourself with syslog(3), syslogd(8), and syslog.conf(5).

2 2 Fig. 1. CARMA Logging via syslog. The subsystem process uses an a Category (Logger) instance to pass messages to the syslog daemon on the computer where the database resides. (The RDBMS work package specifies that the RDBMS will have its own dedicated computer.) The syslog daemon will parcel out the log messages to one or more files on /var/log. A separate process (Syslog2DB) reads the log files asynchronously and stores them in the database.

3 3 Computer (see Figure 1). For development purposes, the local syslog daemon may be may be used. The rules in /etc/syslog.conf will instruct syslogd how to dispose of the messages. A separate process, Syslog2DB, asynchronously reads the syslog files in /var/log and stores them in a database. This design has a number of advantages: 1. It leverages the mature, robust Unix syslog system to handle fundamental logging. 2. Based on logs for January 2003, BIMA generates log messages at an average rate of per 24 hrs. Let s assume CARMA will triple that (about once every 3 seconds). This rate is easily handled by syslog. 3. It does not matter if the RDBMS computer is Solaris or Linux; syslog works cross-platform (I tested this). 4. It does not require development of a database logger under the log4cpp umbrella. 5. Little new C++ code is needed, and the code that is needed is straightforward. 6. A barebones logging system that stops at the syslog level can be operational in short order. Database can be implemented later. 7. Development work can proceed using the syslog facility on the developer s machine. The RDBMS machine is not required. 8. Repeated messages are decimated by the syslog facility, before they get into the database. For debug work, a Logger that writes to a non-system file on the mounted filesystem could also be used. 3. Implementation 3.1. The CARMA Logger The class carma::services::logger has two static methods that return Category instances. The Category class contains the needed methods for logging messages at various priorities (see Table 1). getsyslogger(const string& identity, const string& host, const string& logname) returns a Category reference with a syslog Appender instantiated and attached. The Category is identified by logname and the Appender is identified by identity and host. If the host string passed to getsyslogger is lexically equal (strcmp(3)) to the value returned by gethostbyname(3), the Appender will be a log4cpp::syslogappender, otherwise it will be a log4cpp::remotesyslogappender. This is transparent to the caller. The parameter logname is the hierarchical namespace for the logger that indicates the subsystem under which the current process is running, e.g., carma.monitor.

4 4 getfilelogger(const string& identity, const string& pathname, const string& logname) returns a Category reference with a file Appender instantiated and attached. The parameter identity corresponds to ident in openlog(3) and pathname corresponds to pathname in open(2). For either method call, if and only if all of the passed parameters are the same as those used in a previous call, a reference to the previous Category object is returned if that object still exists. If a previous logname is passed with different identity and host/pathname, rather than replace the Appender or add a new one, an exception will occur. The reason for this is that Appenders should be replaced or added through appropriate use of the addappender() and setadditivity() methods of Category. Examples of syslog and file loggers are shown in the Appendix. An Appender needs a Layout object that tell it how to format the output messages. The default layout, CarmaLayout, consists of a timestamp, the calling program, the logname, the priority level, the nested diagnostic context 3 and the message itself. The class log4cpp::timestamp is not used directly; it is converted to a timestamp related to the carma::util::time class. Currently, the closest frame count to the log4cpp::timestamp that was created with the message is used, but some other timestamp format is possible. (The frame count is the closest half-second since Jan 01, 2002 UT.) The nested diagnostic context is a way to distinguish similar log output from different sources. For instance, each CARMA antenna may be running an instance of a particular program. This program might have the same identity and logname regardless of where it is running. Through proper use the NDC, the exact antenna from which a log message originates may be placed within the message (see example code). In the log4cpp package, it is implemented (log4cpp::ndc) as a simple stack in which the developer pushes or pops messages. Since the NDC is a stack, the developer can push and pop at specific critical locations in the code for a more detailed diagnostic context. The exact format for the CarmaLayout will be defined in a configuration file (a feature supported by log4cpp). The configuration file can define multiple layout styles tuned to different Appenders and Categories. For instance, carma.monitor.* loggers could have a different layout than carma.correlator.* loggers. We may not use this feature, but it is nice to have available. Use of a configuration files also allows reconfiguration without recompilation Syslog configuration The /etc/syslog.conf file is the main configuration file for the syslog daemon which logs system messages on *nix systems. This file specifies rules for logging. Each line in syslog.conf contains a rule and an action for that rule. The rule is made up of a facility and priority indicator. Take, for example, the lines # RULE ACTION 3 See Patterns for Logging Diagnostic Messages by Neil Harrison.

5 5 kern.* kern.crit kern.info;kern.!err /var/adm/kernel-info The first line means all kernel messages get written to /var/adm/kernel, the second means all kernel messages of priority CRITICAL and higher get forwarded to remote host finlandia, and the third means save all kernel messages that come with priorities from INFO up to WARNING in the file /var/adm/kernel-info. In addition to the standard facilities (kern, mail, cron, user, etc), syslog.conf provides 8 facilities local0 through local7 for administrator-defined use. Thus syslog.conf is very flexible, and one can log almost anything at any level of detail. CARMA Logging may use one or more of the local3-local7 facilities (we will leave local0 through local2 for other admin purposes). In the strawman syslog.conf below, we separate out messages from major subsystems. An alternative is to write everything to carma.log and let the database sort it out. Another alternative is to split the logs up by priority level, e.g. all debug messages regardless of origin goes into one file, all infos in another, etc. # # Proposed /etc/syslog.conf entries for CARMA Logging System. # These are in addition to the default system syslog.conf # entries. # $Id: syslog.conf,v /01/13 21:59:06 mpound Exp $ $Date: 2003/01/13 21:59:06 $ # Use "-" preceding file names to omit sync ing after # every write to it. # # Everybody gets emergency messages (via wall) *.=emerg * # local3 catch-all for carma local3.!=debug local3.=debug -/var/log/carma.log /var/log/carma.debug # local4 monitor system local4.!=debug -/var/log/carma.monitor local4.debug /var/log/carma.monitor.debug # local5 correlator local5.!=debug -/var/log/carma.correlator local5.debug /var/log/carma.correlator.debug

6 6 # local6 antenna local6.!=debug -/var/log/carma.antenna local6.debug /var/log/carma.antenna.debug # local7 loberotator local7.!=debug -/var/log/carma.loberotator local7.debug /var/log/carma.loberotator.debug 3.3. Database Writer This section is unfinished until I learn more about databases. But I can state a few things. Syslog2DB should be a daemon. Syslog2DB does not need to run continuously. It can wake up a regular intervals and process any new log messages. If we decide that syslog should redirect messages according to absolute priority (e.g. all debug to one log, all warn to another, etc), then the daemon could check for critical messages more often than non-critical ones. The database table would likely be simple, with columns: Timestamp, Priority, Caller, Logname, NDC, Message. These should be sufficient for system diagnosis. Implementation details depend on which DB we use. Of course we have incanted We will use ODBC but this is not a sufficient specification. For instance, MySql and Postgres each have their own ODBC libraries/frontends. Syslog2DB could be written in Perl which is geared for text-processing. 4. Larger Issues to be Resolved 1. What exactly will the CARMA timestamp format be? Will CARMA have one and only one format? I think it makes sense to use a single format for a timestamp associated with any data storage. Applications that desire other formats can generate them on the fly with methods provided by carma::util::time. 2. What will the CARMA database vendor be? Illinois currently uses Postgres and has a lot of experience with it. The NOAO proposal system that we proposed to adopt in the long term uses MySql. 3. We need a general policy for daemons that we write. PJT and MWP suggest they all go in /etc/init.d, follow the init.d [start stop restart] scripting interface, and use the standard Unix rcn.d protocol for starting at the appropriate run level.

7 7 Table 1. Syslog priority levels. Level Meaning log4cpp::category method EMERG system is unusable emerg(const string&) ALERT action must be taken immediately alert(const string&) CRIT critical conditions crit(const string&) ERR error conditions error(const string&) WARNING warning conditions warn(const string&) NOTICE normal but significant condition notice(const string&) INFO informational info(const string&) DEBUG debug-level messages debug(const string&)

8 8 A. Example Usage A.1. Logging to Syslog /** * $Id: Log2Syslog.cc,v /02/12 22:56:32 mpound Exp $ * Example usage of CARMA logging to syslog. Log2Syslog.cc Marc Pound */ #include "CarmaLogger.h" using log4cpp; int main(int argc, char* argv[]) { // I think every program should define this. string progname = "Antenna Calibrator Control"; // Get a reference to a Category (Logger) object, assigning it the // name of the calling program, the computer on which the // syslogd is running, and a hierarchical namespace for // this Logger. The layout will already be set to CarmaLayout and // the appender additivity will be set to false. Category& mylogger = carma::services::logger.getsyslogger( progname, "rdbms.mmarray.org", "carma.antenna.control"); // Push the diagnostic context of this program on the // NDC stack. I ve just made up this config call, but // note the NDC is obtained in an antenna-independent way. string whoami = carma::util::config.getconfiguration().getantennaname(); NDC::push(whoami); // Set up the priority for the Category, in this example // the INFO priority (so attempts to log DEBUG messages // will fail, since DEBUG is lower priority than INFO.) mylogger.setpriority(priority::info);

9 9 // Log an info message. mylogger.info("information is a good thing."); // You can also use the stream methods, but be sure to // to include a Priority level (runtime error if absent). std::string message = "I ll be gone "; int value = 500; mylogger << Priority::WARN << message << value << " miles when the day is done."; // Log an emergency message. mylogger.emerg("abandon ship!"); } // Clean up and flush all appenders mylogger.shutdown(); return 0;

10 10 A.2. Logging to a File /** * $Id: Log2File.cc,v /02/12 22:56:32 mpound Exp $ * Example usage of log4cpp logging to a (local) file. Log2File.cc Marc Pound */ #include "CarmaLogger.h" using namespace std; using namespace log4cpp; int main(int argc, char* argv[]) { // I think every program should define this. string progname = "Antenna Calibrator Control"; // Get a reference to a Category (Logger) object, assigning it the // name of the calling program, the output file, and the // a hierarchical namespace for this Logger. // The layout will already be set to CarmaLayout and // the appender additivity will be set to false. Category& mylogger = carma::services::logger.getfilelogger( "Antenna Control Program", "/home/mpound/acp.log", "carma.antenna.control"); // Push the diagnostic context of this program on the // NDC stack. I ve just made up this config call, but // note the NDC is obtained in an antenna-independent way. string whoami = carma::util::config.getconfiguration().getantennaname(); NDC::push(whoami); // Allow messages of DEBUG level and above to be passed // to appenders. But set the FileAppender s threshold to // INFO. mylogger.setpriority(priority::debug); mylogger.getappender("antenna Control Program")->setThreshold(Priority::INFO);

11 11 // Here we will be tricky and add another appender to the // Category. The second appender will be an output stream // appender that will catch DEBUG and above messages, while // the FileAppender instantiated as part of mylogger catches // INFO priority and above. Ideally, we d like a log4cpp analog // of log4j s LevelRangeFilter, which would allow us to // catch ONLY debug messages in the OstreamAppender. I will // likely build this eventually. OstreamAppender* osappender = new OstreamAppender("Antenna Control Debug", &cout); osappender->setthreshold(priority::debug); // An Appender when added to Category becomes an additional output // destination unless additivity is set to false. When it is false, // the appender added to the category replaces all previously // existing appenders. Make mylogger accept multiple appenders. mylogger.setadditivity(true); // A side-effect of setting additivity to true is that the // root appender of the Category (every Category has a default // root appender) will get passed the messages. The root appender // appends to stderr, which is annoying. So we remove the // root appender. mylogger.getroot().removeallappenders(); // Instantiate a layout will tells how to format the output stream file. // In this example, we use a layout that comes predefined with // log4cpp. The log4cpp::basiclayout includes a simple timestamp. Layout* layout = new BasicLayout(); // Attach the layout object to ostream appender // and add it to mylogger. osappender->setlayout(layout); mylogger.addappender(osappender); // Log an info message. This will go only to the output file. mylogger.info("information is a good thing."); // Debug messages will go only to output stream appender. mylogger.debug("you are surrounded by twisty little passages, all alike.");

12 12 // You can also use the stream methods, but be sure to // to include a Priority level (runtime error if absent). std::string message = "I ll be gone "; int value = 500; mylogger << Priority::WARN << message << value << " miles when the day is done."; // Log an emergency message. mylogger.emerg("abandon ship!"); } // Clean up and flush all appenders mylogger.shutdown(); return 0;