Log Correlation Engine 3.6 Administration and User Guide

Transcription

1 Log Correlation Engine 3.6 Administration and User Guide May 7, 2012 (Revision 7) The newest version of this document is available at the following URL: Copyright Tenable Network Security, Inc. Tenable Network Security, Nessus and ProfessionalFeed are registered trademarks of Tenable Network Security, Inc. Tenable, the Tenable logo, the Nessus logo, and/or other Tenable products referenced herein are trademarks of Tenable Network Security, Inc., and may be registered in certain jurisdictions. All other product names, company names, marks, logos, and symbols may be the trademarks of their respective owners. Tenable Network Security, Inc Columbia Gateway Drive, Suite 100, Columbia, MD

2 Table of Contents Introduction... 5 Standards and Conventions... 5 Components of the Log Correlation Engine... 5 IDS Collection and Correlation... 6 IDS Collection Only... 7 Prerequisites... 7 Supported Operating Systems/Platforms... 7 Licenses... 7 LCE Manager... 7 SecurityCenter... 8 Secure Shell Public Keys... 8 Secure the Log Correlation Engine Server System... 8 LCE Server Installation... 8 Getting Started... 8 Installation Location... 9 Installing the Package... 9 Files and Layout...11 Base Directories...11 The admin Directory...11 The daemons Directory...11 The db Directory...11 Installing the License...11 Hostname Determination...11 Key Installation...12 Upgrading the License...12 Configuring Security Center 3 SSH Public Keys...12 Automated SSH Key Exchange...13 Configuration...13 The lce.conf File...13 Basic Options...13 Pointing to a New Plugins Directory...21 Adding Log Correlation Engine Client Information...21 Debug Mode...22 Correlation Statistics...23 Performance Monitoring (Security Center 3 only)...23 Syslog Considerations...24 Sending Syslog Messages to Other Hosts...24 syslog Compliant Messages...25 Content of Forwarded syslog Messages...25 Storing All Logs with save-all...25 Different File System...26 Utilizing logrotate...26 Multiple Plugin Matches per Log File multiple-matches...27 Quick Example...27 The rules.conf File...27 Copyright Tenable Network Security, Inc. 2

3 Syntax...28 Syslog Syntax...28 Custom Command Syntax...28 LCE Rule Filters...28 LCE Shell Command Options...29 Updating Plugins (PRM files) and TASL Scripts...31 Automatic Plugin (PRM files) and TASL Updates...31 Updating Individual PRM Files...31 Excluding PRM files...32 Excluding TASL files...32 LCE Operations...33 Starting LCE...33 Halting LCE...33 Operating the stats Daemon...33 Upgrading LCE...34 Additional Features /Alerting/Execution...34 Importing LCE Data Manually...35 Disk Usage Prediction (Security Center 3 only)...35 User Tracking...36 Save/Search Raw Logs...37 Working with SecurityCenter or LCE manager...38 Adding the LCE to SecurityCenter/LCE manager...38 Configuring Organizations...40 Analyzing Security Events...40 TASL Scripts...40 Working with the LCE Manager...40 Installing the LCE Standalone Manager Interface...40 LCE Manager Use and Operation...41 For More Information...41 About Tenable Network Security...42 Appendix 1: Sample lce.conf File...43 Appendix 2: Troubleshooting...50 Appendix 3: Manual SC4/LCE Key Exchange...51 Appendix 4: Manual SC3/LCE Key Exchange...53 Manual SSH Key Exchange...53 Download Keys to the Log Correlation Engine...54 Install the Public Key...55 Copyright Tenable Network Security, Inc. 3

4 Test the Trust Relationship...55 Appendix 5: Non-Tenable License Declarations...57 Related 3 rd Party and Open-Source Licenses...57 Copyright Tenable Network Security, Inc. 4

5 INTRODUCTION This document describes the installation, configuration and administration of Tenable Network Security s Log Correlation Engine 3.6 for both standalone and SecurityCenter deployments. Please share your comments and suggestions with us by ing them to support@tenable.com. A working knowledge of Unix command line syntax and is assumed. If the LCE is to be managed by Tenable s SecurityCenter, knowledge of SecurityCenter operation and architecture is assumed. Familiarity with system log formats from various operating systems, network devices and applications and a basic understanding of Unix is also assumed. This document is intended to be used with LCE installations using version 3.6 and higher. STANDARDS AND CONVENTIONS Throughout the documentation, filenames, daemons and executables are indicated with a courier bold font such as gunzip, httpd and /etc/passwd. Command line options and keywords are also indicated with the courier bold font. Command line options may or may not include the command line prompt and output text from the results of the command. Often, the command being run will be boldfaced to indicate what the user typed. Below is an example running of the Unix pwd command. # pwd /opt/local/lce # Important notes and considerations are highlighted with this symbol and grey text boxes. Tips, examples and best practices are highlighted with this symbol and white on blue text. COMPONENTS OF THE LOG CORRELATION ENGINE The LCE is available in either standalone version with a GUI management interface or can be managed by SecurityCenter. Any references to SecurityCenter assume SecurityCenter 4 unless otherwise noted. The Log Correlation Engine (LCE) is made up of three basic components: the client, the daemon/server component (lced) and the user interface (GUI - SecurityCenter or LCE Manager). Copyright Tenable Network Security, Inc. 5

6 Throughout this document, the LCE daemon/server component will normally be referred to as the LCE server, while the GUI component will be referred to as the LCE Manager or SecurityCenter for simplicity. The LCE client is installed on hosts to monitor and collects events to forward on to the LCE server daemon. When received by the LCE server, events are both stored as raw logs and normalized and correlated with vulnerabilities (if applicable). The LCE Manager UI makes both the raw and normalized event data available to the user for event and vulnerability analysis and mitigation. LCE Manager users work with log data from a wide variety of sources. Each organization can make queries to one or more LCE servers that contain events from a wide variety of devices including firewalls, servers, routers, honeypots, applications and many other sources. The LCE supports many types of agents including: > Windows Event Logs (collected locally or remotely via WMIC) > Windows / Unix system and application logs > Checkpoint OPSEC events > Cisco RDEP events > Cisco SDEE events > Netflow > Splunk > Sniffed TCP and UDP network traffic > Sniffed syslog messages in motion > File monitoring (Unix and Windows) LCE has many signature processing libraries to parse logs and can normalize and correlate most network IDS devices, as well as messages from SecurityCenter. The LCE supports the following IDS sources: IDS Collection and Correlation > Bro > Cisco IDS > Enterasys Dragon > HP TippingPoint > IBM Proventia (SNMP) > Juniper NetScreen IDP > McAfee IntruShield (limited correlation) > Fortinet IDS events > Tenable s Passive Vulnerability Scanner > Snort (and Snort based products) TippingPoint syslog event format must be modified to use a comma delimiter rather than a tab delimiter before it can be processed by the LCE. Copyright Tenable Network Security, Inc. 6

7 IDS Collection Only > AirMagnet > Checkpoint (Network Flight Recorder) > Portaledge > Toplayer IPS There are thousands of normalization rules that support most operating systems, firewalls, network routers, intrusion detection systems, honeypots and other network devices. The list of officially supported log sources is frequently updated on the Tenable web site. PREREQUISITES It is important to ensure that the prerequisite requirements for LCE are met before beginning installation. These requirements include: > Operating System installation > LCE license > LCE management installation (standalone or SecurityCenter) > LCE clients > Secure Shell key generation (if it is to be managed by SecurityCenter using a public/private key pair and not passwords) > A hardened CentOS/RHEL OS Supported Operating Systems/Platforms The LCE server component is available for the Red Hat ES (Enterprise Server) 3.x, 4.x and 5.x Operating Systems for 32-bit platforms (4.x and 5.x for 64-bit platforms). One or more LCE servers can be installed to operate with a single SecurityCenter. The LCE server can be installed on the SecurityCenter, but is not recommended for performance reasons. Conversely, the LCE server is meant to be installed on the LCE Manager because of the lessened performance impact due to a lack of vulnerability data. Licenses LCE servers are licensed to the specific hostname of the system it is to be installed on. There is no limit to the number of events or IPs that the LCE can be configured to monitor. However, since it is being managed from SecurityCenter, there is a hard limit to the number of SecurityCenter IP addresses. There are two different licenses available for the LCE. The first license lets you create up to 255 data silos (see Storage of Data in Silos for more information). The second license limits the number of data silos that can be created to 50. There is no difference in the software that is installed, just the number of silos that can be created. Data silos are always limited to a maximum size of 4 GB per silo. LCE Manager If the LCE is not going to be managed by SecurityCenter, the LCE Manager interface must be installed. The LCE Manager uses a different install package as SecurityCenter and shares the LCE server license key. Copyright Tenable Network Security, Inc. 7

8 SecurityCenter If the LCE is to be managed by SecurityCenter, you must first download and install SecurityCenter, which is available from the Tenable Support Portal. Please refer to the SecurityCenter documentation for more information on installation and configuration of SecurityCenter. LCE server and greater is required for SecurityCenter 4. LCE server and greater is required for SecurityCenter 4.2. Secure Shell Public Keys LCE analysis is provided to SecurityCenter through the use of command execution across a Secure Shell network session. When SecurityCenter wishes to query a LCE server, it invokes a Secure Shell session to the configured LCE server. All execution and analysis of LCE data occurs on the LCE server. Secure Shell public keys are configured such that SecurityCenter can invoke commands on the LCE server. Non system-administrator accounts are used to perform these queries. The trust relationship is only needed from SecurityCenter to the LCE server. Specific directions for enabling Secure Shell public keys between SecurityCenter and the LCE (Security Center 3.X only) are explained later in this document. Secure the Log Correlation Engine Server System It is recommended that the server operating system be locked down before installation to ensure that no unnecessary services are running. The only service that is required to support remote users is Secure Shell (SSH). While the LCE daemon is operational, it will listen on UDP port 514 for syslog messages, UDP port 162 for SNMP and on TCP port for the LCE API if LCE clients are operational. The system running the LCE can operate a syslog daemon, but the syslog daemon must not be listening on UDP port 514. When the LCE is not running, there should be nothing listening on port 514. LCE SERVER INSTALLATION If the LCE server is to be managed by SecurityCenter, you must first upgrade SecurityCenter to version 4 or greater before installing LCE Please contact Tenable Support at support@tenable.com if you have any questions regarding this. GETTING STARTED Before beginning the LCE installation, it is important to understand the high-level steps required to facilitate a successful installation. These steps are typically performed in the following order: 1. Download the LCE server RPM and confirm the integrity of the installation package by comparing the download MD5 checksum with the one listed in the product release notes. 2. Install the LCE server RPM 3. Copy the LCE license key (lce.key) to /opt/lce/daemons Copyright Tenable Network Security, Inc. 8

9 4. Start the LCE daemon (service lce start) 5. (Security Center 3 only) Perform the SSH key exchange (/opt/lce/daemons/addsecurity-center.sh) 6. Configure the LCE server daemon (/opt/lce/daemons/lce.conf and optionally rules.conf for LCE 3.4 and greater) 7. If the LCE server is to be managed by SecurityCenter, add it via the SecurityCenter web interface. If LCE is to be managed as a standalone installation, install the LCE Manager first and then add the LCE through the LCE Manager interface. The package used to install the standalone LCE Manager is the same as that used to install SecurityCenter. When the package is launched, the option to choose either SecurityCenter installation or standalone LCE is displayed. Choose the appropriate option for your installation. Refer to the paragraphs below for more detailed guidance. INSTALLATION LOCATION The installation files may be placed anywhere on the installed system. The total uncompressed file size of the LCE distribution is usually less than 2 megabytes and can be deleted after installation. INSTALLING THE PACKAGE In order to ensure consistency of audit record time stamps between the LCE and SecurityCenter, make sure that the underlying OS makes use of NTP as described in: US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/sect- Date_and_Time_Configuration-Command_Line_Configuration-Network_Time_Protocol.html If you are upgrading from a previous version of LCE, please skip this section and see the section titled Upgrading the Log Correlation Engine below. For new installations, please follow the instructions in this section. As the root user, install the LCE RPM using the following command: # rpm -i lce-3.x-es5.i386.rpm An example is shown below: # ls -l total 488 -rw-r-r-- 1 root root Sep 10 10:20 lce-3.x-es5.i386.rpm # rpm -i lce-3.x-es5.i386.rpm # tail /etc/passwd xfs:x:43:43:x Font Server:/etc/X11/fs:/sbin/nologin ntp:x:38:38::/etc/ntp:/sbin/nologin Copyright Tenable Network Security, Inc. 9

10 gdm:x:42:42::/var/gdm:/sbin/nologin desktop:x:80:80:desktop:/var/lib/menu/kde:/sbin/nologin apache:x:48:48:apache:/var/www:/sbin/nologin webalizer:x:67:67:webalizer:/var/www/usage:/sbin/nologin squid:x:23:23::/var/spool/squid:/sbin/nologin named:x:25:25:named:/var/named:/sbin/nologin lce:x:251:252:tenable LCE User:/opt/lce:/bin/bash # ls -l /opt/lce/ total rwxr-x--- 1 lce lce Apr 14 14:06 add_logs drwxr-x--- 3 lce lce 4096 Apr 14 20:00 admin -rwxr-x--- 1 lce lce Apr 14 14:06 convertdb drwxr-x--- 4 lce lce 4096 Apr 23 14:19 daemons drwxr-x--- 3 lce lce 4096 Apr 20 13:56 db -rw-r r-- 1 root root Apr 17 12:21 debug.gz -rwxr-x--- 1 lce lce 3327 Apr 14 14:05 lce_debug.pl -rwxr-x--- 1 lce lce Apr 14 14:05 lce.txt drwx lce lce 4096 Apr 25 20:33 log_archive -rwxr-x--- 1 lce lce Apr 14 14:06 print_stats -rwxr-x--- 1 lce lce 6800 Apr 14 14:06 pump_cache -rwxr-x--- 1 lce lce Apr 14 14:06 rebuild_logs -rwxr-x--- 1 lce lce 3275 Apr 14 14:05 redo-silo.sh -rwxr-x--- 1 lce lce Apr 14 14:06 search_logs -rwxr-x--- 1 lce lce Apr 14 14:06 showids -rwxr-x--- 1 lce lce 3465 Apr 14 14:05 silo-manager.sh -rwxr-x--- 1 lce lce Apr 14 14:06 tasl -rwxr-x--- 1 lce lce Apr 14 14:05 Tenable_Log_Correlation_Engine_3.x_Software_License_Agreement_v3_ pdf # ls -l /opt/lce/daemons/ total rwxr-x--- 1 lce lce 3889 Sep 15 12:33 add-security-center.sh -rw-r lce lce Sep 25 11:56 lce.conf -rwsr-xr-x 1 root root Sep 15 12:33 lced -rw-r--r-- 1 root root 5 Sep 25 12:03 lced.pid -rwxr-x--- 1 lce lce 284 Sep 15 12:33 lce-install-key.sh -rw-r lce lce 1285 Sep 25 12:01 lce.key -rwxr-x--- 1 lce lce Sep 15 12:33 lce_update_plugins.pl drwxr-x--- 2 lce lce Sep 25 11:49 plugins drwxr-x--- 2 lce lce 4096 Sep 25 11:49 plugins_archive -rwxr-x--- 1 lce lce 1660 Sep 15 12:33 rules.conf -rwxr-x--- 1 lce lce Sep 15 12:33 stats # The add-security-center.sh script indicated in the ls command above is provided for backwards compatibility with Security Center 3.X and is not used for SecurityCenter 4. The RPM will create a user and group named lce and install the LCE to the /opt/lce directory. All files will be installed with the user and group of lce except for the actual lced daemon, which is set-user-id root. This must be started as the lce user and once the daemon has bound to the syslog port (UDP port 514) it will drop privileges. If the lced Copyright Tenable Network Security, Inc. 10

11 daemon terminates abnormally for any reason, the system will automatically restart the daemon and add a warning to the LCE logs. FILES AND LAYOUT Base Directories Within the /opt/lce directory are the showids tool and three sub-directories: admin, daemons and db. The admin Directory This directory contains all of the LCE s log files. There is a subdirectory named log that contains log files. Log file names are based on the ISO format of year and month date such as log for the month of May in The daemons Directory This directory holds the actual lced binary, its configuration files (lce.conf and rules.conf) and several helper functions to update the LCE plugins and to install the Secure Shell key from the Security Center 3. The license key must also be placed in this directory and renamed to lce.key. Within this directory is the plugins directory that contains all of the libraries used by the LCE to parse events. When the LCE loads, it will load all libraries in this directory. Tenable recommends that users who wish to customize their plugins create a second directory for this. Within the lce.conf file, the plugins-directory keyword is used to specify the location of the active directory containing the LCE plugins. Information about LCE plugins and writing them is included in a separate document entitled Log Correlation Engine Log Normalization Guide. The db Directory As the LCE is operating, it keeps all of the event data in this directory. Each silo will be labeled with a lce(number).db and lce(number).raw format. For example, a running LCE server with 100 silos will have lce0.db, lce0.raw, lce1.db, lce1.raw, and so on up to lce99.db and lce99.raw. When a silo is rolled and is no longer the active silo, there will also be a cache and an index field. For example, silo #5 would have lce5-cache and lce5-index directories. These will contain the results from previous cached queries and indexing. INSTALLING THE LICENSE Hostname Determination The LCE uses the hostname (not the domain name) of the system it is being installed on. To determine the hostname needed for a LCE license, simply run the hostname tool and report what is returned. For example: # hostname badger Copyright Tenable Network Security, Inc. 11

12 # Key Installation The demo or commercial license key file for the LCE must be placed on the system running the LCE and moved to the /opt/lce/daemons directory. Once located there, it must be renamed to lce.key. It must also be readable by user and group lce. The lced process will complain at run time if the license is not valid. From the directory where the key was copied to, run the following commands: # cp tenable-lce key /opt/lce/daemons/lce.key # cd /opt/lce/daemons # chown lce:lce lce.key # chmod 640 lce.key # ls -l lce.key -rw-r lce lce 1285 May 3 15:14 lce.key # /etc/rc.d/init.d/lce start # Use a file transfer program such as secure FTP (SFTP) or secure copy (SCP) via Secure Shell to transfer the ASCII key file to the correct location (/opt/lce/daemons/). Upgrading the License It is possible to upgrade from your silo license to one with a higher capacity (e.g., 50-silo to 255-silo). A replacement license key file will be required. To upgrade your license: > Stop the LCE service. > Replace the existing /opt/lce/daemons/lce.key with the new, upgraded license key. > Edit the file /opt/lce/daemons/lce.conf to change the number of silos to an appropriate number for your configuration. > Start the LCE service. To verify what key is currently installed, use the tail command to examine the LCE s log file in /opt/lce/admin/log. If a 50-silo license is installed but the LCE is configured for more than 50 silos, the log file will indicate this as in the following example: Sep 29, 05 16:46 (Log Daemon) WARNING: lced - too many silos configured. Using licensed amount of 50 silos. Sep 29, 05 16:46 (Log Daemon) DEBUG: lced started (version 3.4.2) Sep 29, 05 16:46 (Log Daemon) DEBUG: db_currsilo_load - Loaded current silo successfully Sep 29, 05 16:46 (Log Daemon) DEBUG: lced - loaded 3178 plugins Sep 29, 05 16:46 (Log Daemon) DEBUG: lced - current silo is 0 Sep 29, 05 16:46 (Log Daemon) DEBUG: lced - number of silos is 50 CONFIGURING SECURITY CENTER 3 SSH PUBLIC KEYS This section can be skipped if LCE is not being managed by Security Center 3. Copyright Tenable Network Security, Inc. 12

13 Automated SSH Key Exchange Users running Security Center 3 require additional steps to add their LCE to the Security Center. These steps involves a scripted or manual SSH key exchange. The LCE add process for SecurityCenter 4 is considerably easier and involves adding a licensed LCE through the Resources -> Log Correlation Engines -> Add dialog in the SecurityCenter 4 web UI. This process is described in depth here. If remote root access is not available between the SecurityCenter and LCE, the key exchange portion of this process can be performed using the steps defined in the Appendix 4: Manual LCE Key Exchange portion of this document below. The SSH key exchange enables the Security Center to communicate with the LCE server and can be performed easily using the /opt/lce/daemons/add-security-center.sh script. The script will prompt for the Security Center IP address, admin username and password, similar to the following output: # sh add-security-center.sh Please enter the following information for the Security Center that will be granted access to LCE: Security Center IP: Administrator username: admin Password for admin: The Security Center at has successfully been granted SSH access to the Log Correlation Engine. # The username and password specified here is the Security Center administrative login username and password. These credentials are required for the Security Center to LCE key exchange. After running the add-security-center.sh script, you can log in to the Security Center as the admin user and add the LCE server. CONFIGURATION THE LCE.CONF FILE All LCE configurations are accomplished in the /opt/lce/daemons/lce.conf file. Any changes to this file must be performed using a text editor. For changes to go into effect, the lced process needs to be restarted. Basic Options Option Description Copyright Tenable Network Security, Inc. 13

14 LCE Configuration Options plugins-directory correlation-scriptsdirectory daemons-directory database-directory log-directory key-file pid-file die-file server-address Specifies the directory where active PRM files are stored. Specifies the directory where TASL scripts are stored. Specifies the location of the LCE ~/daemons directory and all configuration files. Do not change this setting. Specifies the location of the LCE database directory. This can be reconfigured to make use of directories elsewhere on the server that may have more disk space. Ensure that the lce user has permissions to write and read from any directories outside of /opt/lce. Specifies the location of where the ISO compliant LCE log files will be created. Path to the LCE s demo or commercial key file. When the lced process starts, it will store its process ID file in the file specified here. While the lced process is running, it will periodically check for the existence of this file. If it is found, a graceful exit will be performed. This option allows you to specify the network interfaces that lced will listen on. More than one interface may be specified on separate lines as in: server-address server-address By default, this option is commented out and lced will listen on all available network addresses. server-port silo-size number-silos This option specifies the port number that lced listens on. By default, it is set to 31300, but may be reset to another value. Specifies the maximum amount of data from matched log events that will be stored in one indexed file (silo). Uses an M extension to specify megabytes. For example, 4096M would specify the maximum size of 4 Gigabytes. Extensions of G specify gigabytes. For example, 1G would specify 1 gigabyte. By default, this is set to 1024M. Specifies the number of silos that lced will create. The maximum number of silos that can be created is 255 for a full license. If you have a 50-silo license, change this value to the appropriate value. Copyright Tenable Network Security, Inc. 14

15 forward-syslog This option allows you to specify one or more syslog servers to forward events to. If you have more than one syslog server, specify each on as separate line as follows: forward-syslog forward-syslog This option is turned off by default. securitycenter-ip additional-querymemory (Security Center 3.X only) Specifies the IP address of the Security Center. This is used so that TASL scripts can generate alerts to the Security Center without having its address hard-coded. The lce_queryd process is responsible for high-performance processing of all query requests. By default, up to about one gigabyte of memory is used. For systems with large amounts of available memory, the following option can be used to allocate additional memory for the query daemon. This will increase the number of query results that can be cached, improving response time during event analysis in SecurityCenter. The option can be specified in kilobytes, megabytes or gigabytes by appending a K, M or G to the value. This value should never be set above 1500M on 32-bit systems. save-nonmatched archive-directory enable-log-archiving disk-alert-percentage If this keyword is enabled, LCE will create a file named notmatched.txt in the database directory and fill it with log events that have not matched any LCE plugin. This is an excellent way to analyze events that may be inadvertently ignored. There is a hardcoded limit of 2GB for this option in addition to the number of events specified. The archive-directory keyword specifies a directory where all log events received by LCE will be saved. The logs are highly compressed, and are stored according to date using a unique method designed to allow historical information to be retained while using minimal disk space and allowing efficient search. This setting is disabled by default. This keyword takes no arguments and specifies whether compressed raw logs will be collected or not. If this keyword is enabled, the supporting keywords archive-directory and disk-alert-percentage must be configured. When log archiving (enable-log-archiving) is enabled, the disk-alert-percentage option generates an alert when the LCE is in danger of running out of disk space to store logs. When disk usage reaches the specified percentage of capacity, a daily TASL alert will be sent to Security Center 3. Copyright Tenable Network Security, Inc. 15

16 For SecurityCenter 4, a LCE event rule could be created using rules.conf to generate an alert via , syslog or other means. SecurityCenter 4 and greater does not receive syslog data directly. The data is received instead by the LCE server. log-client-activity save-all rollover-script save-database When this option is uncommented, client activity will be logged to the LCE database. This includes connect, disconnect and failed login events. Specifics a log file where all events (not just the ones matched with a LCE plugin) are stored. This log file does not rotate and must be managed by the logrotate process. Note that this will require significantly more disk space than just keeping the events that match plugin criteria. This option is most useful when used in conjunction with logrotate and an external storage device. Specifies the script used to rollover silos as they become full. When the maximum silo size is reached, a new silo becomes the active silo. If the maximum number of silos is reached, the oldest silo is overwritten with new data. The /opt/lce/silo-manager.sh script performs some housekeeping, compression and indexing of the collected data. When the maximum number of silos has been reached and an older silo must be overwritten for the next silo roll, the silo to be overwritten can first be saved for future use. This option specifies whether or not to save the normalized database file. If there is insufficient disk space on the silo archive device, LCE will no longer attempt to save a silo before overwriting. If this occurs, log messages will be generated warning of the event. The event alerting functionality of LCE can be leveraged to automatically notify concerned individuals (e.g., alert) when this sort of event occurs. Please reference the rules.conf section of this document for more information. save-index save-raw As in the save-database option, this option specifies whether the index files are to be saved. The save-database option must be enabled for this option to be useful. This option specifies whether the lce#.raw files are saved. These files contain the original matched log messages before Copyright Tenable Network Security, Inc. 16

17 normalization. location checksum-forward This option specifies the location to save the data for the previous three options. The location setting allows the silo to be saved to any mounted volume or other physical location. Each time the active silo changes, an MD5 checksum is generated for the previous silo. This allows the integrity of the.raw and.db data files to be checked at a later time. This option allows the MD5 checksums to be forwarded to one or more off-site locations. You may specify multiple addresses to forward checksum data on separate lines as shown: checksum-forward checksum-forward A new event is logged whenever a set of MD5 silo checksums is calculated. The event has type lce and name silo-md5- checksums. The same message is also logged to the saveall file and log archive, if enabled. auto-update-prms auto-update-tasls proxy-address proxy-user proxy-password max-memory-usage cache-results-days pre-cache-file excluded-domains-file excluded-domain-name The auto-update-prms keyword allows PRM files to be updated automatically by LCE. The auto-update-tasls keyword does the same for TASL scripts. If the value is set to yes, files will be updated on each silo roll. The /opt/lce/daemons/lce_update_plugins.pl script allows both PRM and TASL scripts to be updated from the command line. Proxy address and credentials used for both manual and automatic updates of LCE plugins. When a log message is defined in a plugin, LCE provides the option of specifying a hostname instead of an IP address for the srcip and dstip fields. In this case, LCE automatically attempts to resolve the provided hostname to an IP address using DNS. Since the same hostname is typically encountered multiple times, caching the results of lookups can greatly increase performance. These options configure DNS caching in LCE. The max-memory-usage option can go up to 360K domain names. The cache-results-days option specifies the number of days to cache a hostname-to-ip mapping before updating the result with a new lookup. Hosts listed in the pre-cache-file are looked up and stored when LCE starts. A particular hostname or all domain names with a certain extension can be excluded using the exclude-domain-name option. In this case, the matching hosts are looked up at every occurrence. For users that wish to maintain a more extensive list of domains to exclude, the excluded-domains- Copyright Tenable Network Security, Inc. 17

18 file option can be used to direct LCE to read the list of domains from a file. LCE keeps track of network users on the basis of their usernames. These options set restrictions on which usernames are considered valid. Any usernames failing to match the specified criteria are disregarded and invalid is reported as the user for the associated log entries. accept-letters: This is a yes/no option that specifies whether alpha characters [a-za-z] are allowed. accept-numbers: This is a yes/no option that specifies whether numbers [0-9]are allowed. max-username-characters: Specifies the maximum number of characters allowed in a username. additional-valid-characters: Specifies which special characters are considered valid for usernames. By default, the following characters are considered valid: > The dash character, as in - > The underscore character, as in _ > The dot character, as in. > The at sign character, as For example, the following address would be considered valid under the default criteria: b.j-smith@a_b.com Only the special characters that are specified with the additional-valid-characters setting are considered to be valid. The semicolon character, ; is not permitted in this context. excluded-users-file trusted-plugins-file accept-letters accept-numbers additional-validcharacters max-usernamecharacters display-heartbeatmessages This option allows you to specify a text file that contains a list of users whose activity you do not want tracked. This is useful for application user ids that do not actually log in. This keyword allows you to specify a text file that lists the plugins for which user tracking will be enabled. Each LCE client that communicates with the LCE periodically sends a heartbeat message, indicating that the connection is active. This option enables SecurityCenter to display information about the connection activity, hostname and IP address of the LCE client and the revision number of the Copyright Tenable Network Security, Inc. 18

19 application. tasl-report-frequency clear-tasl-log-onupdate max-tasl-queue-memory sampleable-tasls-file syslog-sensors-file IDS The Log Correlation Engine TASL processor has the ability to periodically report the performance of scripts as they are applied to incoming events. This option represents how frequently (in minutes) a report will be generated. The latest report can be found in the tasl.log file, stored in the directory specified by the log-directory keyword. In addition to the performance report, the TASL processor logs detail technical information related to scripts such as syntax and runtime errors. This data is written to a file called tasl_scripts.log in the log directory. Since errors in scripts cause log entries to be generated each time they are encountered, this log can potentially grow large. When set to yes, this option causes the log file to be reset each time the scripts are automatically updated. As a result, all log messages will be relevant to the currently installed scripts after an update. To maximize performance on multi-processor and multi-core systems, correlated TASL events are processed in parallel to receive regular incoming events. Since some TASL scripts can run for an extended period of time, the primary event processor can potentially receive many TASL-triggering events while a TASL script is still being executed. In this case, the TASL job is stored in a queue for later processing. This option defines the maximum size of this queue. On systems with extremely large volumes of data, setting the maximum queue size higher results in increased performance. If a TASL script that can be sampled is triggered while the queue is full, its callback functions will not be executed. This option allows you to specify a file listing the filenames of TASL scripts that can be ignored when the system is busy and the processing queue has reached the specified maximum amount of memory. For logs received via syslog, a sensor name can be assigned to each IP address sending data to LCE. This sensor name will be associated with all logs from the designated source, regardless of whether or not another sensor name is extracted from the log text. The Log Correlation Engine has the ability to receive IDS events from multiple sources. In addition to being normalized and stored in the log database, each event will be checked against any SecurityCenter vulnerability databases. If a host is vulnerable to attack, the event is marked as such, allowing rules to trigger on this scenario so that the information can be distributed to the affected administrators. Copyright Tenable Network Security, Inc. 19

20 For each IDS sensor, a sensor name and type must be defined as in the example below. The supported types are Snort, Bro, RealSecure, Dragon, IntruVert, IntruShield, NetScreen, NFR, Fortinet, PVS, LCE, Cisco, TippingPoint- Sensor and TippingPoint-SMS. sensor-name sensor-type client block debug block Name to be used within the SecurityCenter logs. IDS sensor type. This block of keywords specifies credential information for a LCE client. Each LCE client must be individually configured. This block of keywords specifies a debug mode that may be useful for diagnostics. Statistics Daemon Configuration Options (stats_options) event-directory log-directory syslog-alert Directory containing the LCE events.txt file. Log files are named according to date and stored in the specified directory. The stats daemon will generate syslog messages to one or more recipients. By default, a local address of is used to send messages to the lced services. However, more than one syslog server can be configured on separate lines in the following format: syslog-alert syslog-alert stddev-threshold stddev-unit-threshold iteration-threshold nhits-frequencythreshold This keyword specifies the minimum standard deviation that must occur for an event before an alert will be generated for it. The higher this number, the more statistically significant a sequence of events needs to be before an alert is raised. If an event occurs more or less than 5.0 standard deviation units, an alert will be generated. Setting this value higher will cut down on any sequence of events that occur close to the standard deviation. This keyword specifies the number of iterations (days) perevent are required before alerts will be generated. If a large amount of LCE data is already present, set this number to a low value or even to zero. The stats daemon can be started to read in all or just part of the existing LCE data. If you have NO LCE data, leave this value around 7 so the stats daemon will not alert on anything until it has 7 days of event data. If an event occurs less than 10% of the time then an alert will be generated. Even if an event may be statistically significant, that sequence of events may also occur periodically. For example, 50% of the time you are within a Copyright Tenable Network Security, Inc. 20

21 standard deviation, however, occasionally (the other 50%) you have outliers 2 and 3 standard deviations away. Those outliers may be the cause of 90% of the alerts generated in this case. Setting this value to 10, 20 or other values would only alert for hours that were both out of the allotted standard deviation, and also are event counts that have not occurred before. Network Range include-networks Make sure this range matches IP addresses that are considered internal from an event perspective. Starting with LCE 3.6.1, this range is used by a number of TASL scripts and the Stats daemon to define inbound/outbound/internal specifications for LCE events. Prior to 3.6.1, these ranges were solely used by the Stats daemon. This is different from the Directions filter on the SecurityCenter 4.2 events page, which uses the logged-in user s managed ranges to determine event direction. The following sections define your internal network range. All networks specified in the first section are included, while the exclude-networks block is used to make exceptions. exclude-networks Provides exceptions to the include-networks directive ranges specified above. Pointing to a New Plugins Directory To reconfigure the LCE to use only a sub-set of the plugins it is shipped with, create a new directory in the /opt/lce/daemons directory. For example, we can create an activeplugins directory. The plugins-directory keyword in the lce.conf file will need to point to this new directory as shown below: plugins-directory /opt/lce/active-plugins/ The desired plugins must be manually copied into this new directory and then the lced process restarted. Adding Log Correlation Engine Client Information To configure a LCE client, add the IP address it will be connecting from and a secret key it will be using to connect with. The shared secret key may contain any ASCII character except a semicolon (;) or space ( ). The LCE only supports one type of remote authentication that is based on the shared secret. Copyright Tenable Network Security, Inc. 21

22 All client designations are specified inside the lce_options section of the lce.conf file. The following example shows an LCE client connecting in from a different IP address with a different shared key: client { client-auth auth-secret-key s3kret } client { client-auth auth-secret-key myp455word } Multiple clients can be added using a variety of methods. The client directive supports a single IP address, an IP address range (e.g., or ) and the use of CIDR notation (e.g., /26). As of LCE 3.0, the client directive can be configured with an optional sensor-name field. This field defines a default sensor name to be reported for logs from a particular client that do not provide a sensor name to their associated plugin rule. An example of this designation is shown below: client { client-auth auth-secret-key tenable sensor-name TNS2010 } Debug Mode The lce.conf file can also include a variety of debug information. Information about plugins loaded, LCE client status and operation are all written to the current log file. To place the LCE into debug mode, uncomment the following section of the lce.conf file: debug { lce-api { client-auth } matches { match-success } silo { } silo-switch } match-tree { tree-matches } The lce-api keyword logs all remote client authentication attempts. Enabling this can be helpful when diagnosing remote agent problems. The matches and match-tree keywords Copyright Tenable Network Security, Inc. 22

23 detail how a message is analyzed by the existing plugins. This is a good way to diagnose if a particular plugin is not matching. Lastly, the silo debug keyword logs when a silo is rotated and indexed. Enabling these debug messages is a great way to learn how the LCE operates, however, they generate a lot of information and can create multi-gigabyte log files. If the lced daemon terminates abnormally for any reason, the system will automatically restart the daemon and add a warning to the LCE logs. Correlation Statistics Starting with LCE 3.6.1, runtime statistics pertaining to logging and correlation are collected including: > Logs/bytes per second > Number/percentage of logs matched/unmatched > Number of events correlating with vulnerabilities > Number/percentage of logs from clients, syslog, and IDS > Number of TASL alerts generated This information is logged once per hour and is written both to the application log and to the normalized database under the event name LCE-Server_Statistics (type lce ). Example Correlation Statistics Output: An average of 50 logs are being received each second. A total of 5,778 logs (521,046 bytes) have been received. 2,232 logs have been matched by plugins (38.63%). 3,546 logs did not match (61.37%). Log source breakdown: 5,774 from clients (99.93%), 2 via syslog (0.07%), 0 from IDS devices (0.00%). No log events have correlated with vulnerabilities. 2 TASL alerts have been generated. Performance Monitoring (Security Center 3 only) The LCE logs network statistics, CPU/memory load and disk space after each silo roll to the local log file and sends it to Security Center. This is done by sending two messages; one contains the information from netstat, while the second contains the memory, disk and CPU data. In order to be recognized, the messages contain a TASL header, with the IP of the LCE host as the source, the IP address of Security Center as the destination and source/destination ports 0. For example: TASL : :0 LCE-Statistics The Security Center user can view the entire contents of an LCE-Statistics message by choosing the Raw Syslog Events option. The following are examples of each message. Copyright Tenable Network Security, Inc. 23

24 Netstat information: TASL : :0 LCE-Statistics Network Statistics Udp: 4489 packets received 76 packets to unknown port received packet receive errors packets sent Memory, CPU and disk information: TASL : :0 LCE-Statistics Memory Load: / kilobytes main memory used (95.97%), with kilobytes free (4.03%) 208/ kilobytes swap memory used (0.01%), with kilobytes free (99.99%) kilobytes of data cached Disk Load: 3.5G/4.4G (45%) disk storage used on LCE filesystem (55% free) CPU load average: 0.48, 0.36, 0.25 The netstat statistics include all UDP packets, not just syslog generated traffic. The dropped packet count may significantly increase if the LCE is UDP port scanned, receives a large number of SNMP probes or any other UDP based traffic. If silos are being archived, and there are archives within the directory, LCE will send disk load information to Security Center. If a silo is set to be archived, but there are no archives within the directory, no log entry will be sent. The following are examples of the log entry: 7.68Gb/8.21Gb (93.50%) disk storage used on database file system, with 0.53Gb free (6.50%) 4.0Gb/5.0Gb (80.00%) disk storage used on silo archive file system, with 1.0Gb free (20.00%) SYSLOG CONSIDERATIONS Sending Syslog Messages to Other Hosts The LCE can be the focal point of your entire log aggregation strategy. If a Storage Area Network, syslog server or some other type of log aggregation solution is deployed in your network, the LCE can be configured to send a copy of any received message to one or more syslog servers. These messages include any message received from any client. To configure the LCE to forward these messages, simply enter multiple lines in the lce.conf file with the forward-syslog keyword. The actual syslog service is not used to Copyright Tenable Network Security, Inc. 24

25 forward the messages. All packet generation is handled by the lced process. The following is an example section of the lce.conf that forwards messages to multiple syslog servers: forward-syslog forward-syslog # save-nonmatched 200 # save-all /opt/lce/db/lce.log # multiple-matches client { client-auth auth-secret-key s3kret } syslog Compliant Messages Logs forwarded by the LCE will retain the original syslog alert level and facility, if one was present. If one was not present, the LCE assigns a log level of auth.warning. Typically, LCE clients do not send syslog compliant messages. If a LCE client were configured to monitor a log file that retained an original message s syslog alert level and facility, then this would be retained if forwarded by the LCE. This allows for a remote syslog server that is receiving events from the LCE to process the received messages and place them in specific files. Depending on the type of syslog server, it may be possible to place logs from a router into one file, operating system logs into another and so on. Content of Forwarded syslog Messages When the LCE forwards a message, it also adds any matched information to the log file as shown below: Jun 30 17:45:36 lce: [not-matched] :0 -> :0 :: <37>sshd(pam_unix)[15322]: authentication failure; logname= uid=0 euid=0 tty=nodevssh ruser= rhost= The :: characters are used to separate LCE s heading from the original message. In this case, the message would also have been sent with a syslog facility of <37> since that was the facility of the original message. Additionally, notice that the LCE tagged the example event above with a not-matched keyword. This means that the LCE did not possess a.prm file to process the log. If it did, the matched event name would be present in the same location. STORING ALL LOGS WITH SAVE-ALL Many organizations have regulatory requirements to save all of their log data for 30 days or some other length of time. It may also be part of that requirement that the data not be manipulated, normalized or otherwise processed in case it must be used in a legal proceeding. Any exculpatory evidence in the original logs must not be missing as well. The LCE s method of storing data in silos for high-speed normalization and analysis by many different administrators is not the best place to keep one central log file. The LCE has Copyright Tenable Network Security, Inc. 25

26 means to save every message, even ones that do not match a certain plugin to a central log file. This log file is saved with the save-all keyword in the lce.conf configuration file. The save-all keyword merely specifies the path and output directory of a given log file. It defaults to the location of /opt/lce/db/lce.log which is in the same directory as the silos. As the LCE daemon receives events through the API or from syslog, it will save the message into the file specified by the save-all keyword. This log file will grow very large. Maintain rotation and compression of these logs with the logrotate program that is already installed on all Unix systems supported by the LCE. Different File System Since the lce.log file will grow to extremely large sizes, it is highly recommended to place this file on a different physical file system. If the LCE server is placed on a system with two hard drives, consider creating separate partitions for both the LCE silo data and the lce.log files. If your network has use of a Storage Area Network (SAN), consider using this to store the lce.log file. Many times, these storage devices can be mounted through a network file system (NFS) or Windows file share (SMB) resource. Make sure that write permissions from the LCE server are available and there is sufficient network bandwidth to send the data, if you use a SAN. Utilizing logrotate By default, the LCE daemon will log more and more data to the lce.log file. To avoid this, use the logrotate process to manage the growth of the lce.log file and rotate it when deemed necessary. The logrotate tool can be configured to rotate the lce.log file based on size or date. To fully configure the tool, please consult the manual page so it can be configured to your desired requirements. Below is an example section of the logrotate.conf file found in the /etc directory on Red Hat systems: # save-all lce.log /opt/lce/db/ lce.log { daily rotate 7 compress postrotate /usr/bin/killall -USR1 lced cp lce.log.1 lce.`date +"%Y%m%d%H%M"`.log gzip lce.`date +"%Y%m%d*"`.log endscript } Add this data to the logrotate.conf file. The logrotate tool is already run by the cron utility to manage the rotation of the system logs such as /var/log/messages. It is the process that will periodically rename your log files with.1,.2 and.3 extensions such as messages.3. Copyright Tenable Network Security, Inc. 26

27 MULTIPLE PLUGIN MATCHES PER LOG FILE MULTIPLE-MATCHES By default, the LCE daemon will stop processing a log file as soon as one match has been made. This behavior may be overridden by adding the multiple-matches keyword in the lce.conf file. With this feature enabled, the LCE daemon will attempt to exercise the entire plugin set across every log message. This behavior is useful for extracting multiple forms of information out of a log file. For example, there may be a plugin that looks for a generic user login failure and another that looks for a login failure for user root. Without the multiple-matches keyword, only one of the plugins will match, even though both are valid. Even more so than with normal LCE operation, be sure to remove unneeded libraries with the multiple-matches keyword enabled, otherwise, the LCE s performance can be diminished. Quick Example Tenable implemented this feature when a customer was encountered who had a firewall log with NATed addresses. For each transaction, the firewall logged the external Internet address, the customer s Internet address and their internal RFC1918 address. What they wanted was the ability to type in any of the IP addresses in question to produce a report of the history. For example, a student may receive via DHCP inside a high school. The school s public IP address at the firewall may be and the student may have been attacking a web site at These public addresses were chosen at random and are in no way intended to be example organizations or potential targets. We did not want to use RFC1918 addresses as example external addresses. For any network browsing, a firewall log may have all three IP addresses. Without the multiple-match keyword, there is only one pair of IP addresses that can be matched on. However, with it enabled, two rules can be used to process the same log file and extract the specific IP addresses. In the customer s case, they decided to log external to public IP and public IP to internal IP firewall logs. They generated two LCE events for each firewall log event. However, when they added in the DHCP logs, they were able to use the IP address of a potential attacked target to get the actual internal IP address and MAC address. When someone outside of their network contacted them and complained of a spammer, worm or malicious activity, they were able to type in the IP address of the target, see which public IP was in use at the time, and then see which internal IP addresses were related. THE RULES.CONF FILE The rules.conf file, like lce.conf, is located in the /opt/lce/daemons directory and is used to configure active response operations used by the LCE daemon. LCE rules are configured to analyze LCE event content and fire if preset conditions are met. Active responses include the ability to send automatic s (send ), syslog alerts (sendsyslog) or run custom commands on the LCE system. Copyright Tenable Network Security, Inc. 27

28 Syntax send -subject "$Name" -body "$Log" -to Syslog Syntax sendsyslog message "$Log" Custom Command Syntax my_custom_firewall_reconfig_command -block $sip LCE Rule Filters The following fields are optional filters. A plus sign signifies that events matching the specified values will receive rule application, while a minus sign signifies that matching events will not. If no + filter is used, all events are matched by default for the field, unless excluded specifically with the - filter. Multiple values can be specified for any filter. Do not use spaces to precede LCE rules. If there is a space at the beginning of an option, that option will be ignored. Option Description IPs The following five formats are supported for both +IPs and - IPs: Events Sensors Types Ports Protocols Users > / > /32 > > > Considers both the primary and secondary event names. The Events field allows spaces in event names (because Nessus IDS signatures contain spaces), and thus events must only be separated by commas and not spaces. Spaces, commas or both may be used to separate entries in the other fields. Sensor that detected the LCE event LCE event type Source or destination port within the LCE event Specified by TCP, UDP, ICMP or a number Username associated with the event Copyright Tenable Network Security, Inc. 28

29 Vulnerable Ignore RateLimit yes or no Single keyword causes all events matching the rule s filters to be ignored by LCE. If an event is ignored in this manner, there will be no LCE database entry written for it, no other matching rules will fire and no TASLs filtering on the event will be executed. Ignored events are only evidenced through the save-all file and log archive. A string indicating the maximum number of event responses per time period that will be allowed. When the quantity of incoming matching logs exceeds this constraint, the remaining logs will be queued or ignored. This string follows the format: (integer) per [second, minute, hour, day, week, month, year] MaxQueue Threshold The maximum number of matching events to queue; those coming in while the queue is full will be ignored. A string indicating the minimum number of matching events that must occur in a given time period before event responses are generated. This string follows the format: (integer) in a [second, minute, hour, day, week, month, year] LCE Shell Command Options The following case sensitive variables may be included in the shell command string: Option $sip $dip $sport $dport $proto $vuln $sensor $event1 Description Source IP of event Destination IP of event Source port of event Destination port of event Protocol of event, displayed as N/A, TCP, UDP, ICMP or a number for other protocols no if the event was not correlated with a vulnerability, yes otherwise Name of sensor generating the event Primary event name Copyright Tenable Network Security, Inc. 29

30 $event2 $type $time $user $log $queued_logs Secondary event name Type name of event Time event was recorded at LCE (format: Mon MM, YYYY H:M:S) Username associated with the event Raw text of log All logs currently in the event rules queue. Use of this variable has the effect of emptying the rule s queue LCE ships with the following example rules.conf file: # Below is an example Log Correlation Engine rule. It filters on a specified set of # IP addresses, applying the rule to only those logs with attributes matching constraints # on event, sensor, IP address, and user. The threshold setting requires that a minimum of # 5 matching events occur within a five-minute window before the rule goes into effect. # After this threshold is reached, the LCE syslog utility is invoked to transmit a log at # most once per minute. If the quantity of matching incoming logs exceeds the rate limit, # up to 100 will be queued and transmitted at the specified maximum rate of 1 per minute. # # Name: Potential SSH account username/password guessing # +Events: SSH-Invalid_User, SSH-Failed_Password # +IPs: , /8, / # -IPs: # -IPs: , # +Sensors: DMZ-1, DMZ-2 # -Users: (unknown) # Command: /opt/lce/tools/send_syslog message "Possible password guessing evidence: $log" -priority 97 # Threshold: 5 in a minute # RateLimit: 1 per minute # MaxQueue: 100 # The following example rule will cause all login events from the local server to be ignored. # Ignored events will only be evidenced in the LCE save-all file or log archive. # # Name: Ignore local logins # +Types: login # +IPs: Copyright Tenable Network Security, Inc. 30

31 # ignore # The last example demonstrates a rule to all vulnerability correlations to an administrator. # # Name: vulnerability correlations # +Vulnerable: yes # Command: echo "IDS event correlated with vulnerability ($sip -> $dip) at $time: $log" /opt/lce/tools/msmtp -C /opt/lce/tools/msmtp.conf admin@yourdomain.com UPDATING PLUGINS (PRM FILES) AND TASL SCRIPTS This section describes methods for updating LCE plugins (files with a.prm extension) and TASL scripts. AUTOMATIC PLUGIN (PRM FILES) AND TASL UPDATES Plugin updates occur automatically after every silo roll unless explicitly disabled in the LCE configuration file. Tenable Network Security strongly recommends that PRM files and TASL scripts be updated automatically as part of the LCE s maintenance processes or updated directly with the lce_update_plugins.pl script located in the /opt/lce/daemons directory. The script will automatically look at each PRM and TASL that is present in the plugins directory and compare it to the one online at Tenable s web site. If there is a difference, the script will download and install any new scripts and optionally restart the LCE. Usage of the lce_update_plugins.pl script can be found on the command line. UPDATING INDIVIDUAL PRM FILES Individual PRM files can be updated using lce_update_plugins.pl located in the /opt/lce/daemons directory. This script offers several options for updating PRM files and TASL scripts, which are displayed using the -h (help) option as shown below: # /opt/lce/daemons/lce_update_plugins.pl -h LCE Plugin Updated Help Screen Usage:./lce_update_plugins.pl -OPTIONS Available options: -a (download & update all latest plugins from Tenable) -p (download & update latest PRM plugins from Tenable) -t (download & update latest TASL scripts from Tenable) -o (download & update only plugins/tasl scripts available in corresponding directories) -i (Only inspect if new plugins are available. This switch will report plugins that changed, but will not update them.) -s (silent mode) -v (verbose mode) -D (default mode - equivalent to -aov) Copyright Tenable Network Security, Inc. 31

32 Examples:./lce_update_plugins.pl -a (Update all plugins/tasl scripts)./lce_update_plugins.pl -as (Update all plugins and remain silent)./lce_update_plugins.pl -aov (Update plugins/tasl scripts available in corresponding directories and report all actions)./lce_update_plugins.pl -aiv (Check for updated versions of all installed plugins, but do not update them, and report all actions)./lce_update_plugins.pl -tiv (Check for updated versions of all installed TASL scripts, but do not update them, and report all actions) # The directories containing the PRM files and TASL scripts are specified in the lce.conf file with the plugins-directory and correlation-scripts-directory keywords. When the lce_update_plugins.pl script is manually invoked, the files contained in these plugins and correlation scripts (TASL) directories will be archived to the /opt/lce/daemons/plugins_archive directory. The backups of the files in the TASL directory will appear in the plugins_archive directory as a time stamped file such as tasls_ tar.gz, and the backups of the files in the plugins directory will appear in the plugins_archive directory as a time stamped file such as plugins_ tar.gz. For example, if the script is invoked with the -p option, only the PRM files will be updated, without affecting the TASL scripts. In this case, all files in the plugins directory will first be archived to the plugins_archive directory before downloading and updating the latest plugins. Likewise, invoking the script with the -t option will only update the TASL scripts without affecting the PRM files. In this case, all files in the correlation scripts (TASL) directory will first be archived to the plugins_archive directory before downloading and updating the latest TASL scripts. Invoking the script with the -a option will first archive all files in both the plugins and correlation scripts (TASL) directories before downloading and updating all the latest PRM files and TASL scripts. EXCLUDING PRM FILES In some cases, a user may wish to allow the global updates of PRM files, but specifically exclude some from being run. This can be facilitated by using the /opt/lce/admin/disabled-prms.txt file. The PRM files to be processed but not loaded can be specified in this file, without a pathname, separated by any amount or type of whitespace. If there is a need to customize a plugin or plugins, rename the original file before making modifications. Once done, include the name of the original plugin in the disabled-prms.txt file. EXCLUDING TASL FILES Starting with LCE 3.6.1, TASLs can be disabled selectively by adding the TASL script file name (e.g., program_accounting.tasl) to the /opt/lce/admin/disabled-tasls.txt file and then restarting the LCE daemon. This is useful for cases where a particular TASL script is not needed by an organization or where the TASL might be causing performance issues and needs to be disabled either temporarily or permanently. Copyright Tenable Network Security, Inc. 32

33 Any disabled TASLs can be re-enabled by removing them from the disabled-tasls.txt file and restarting the LCE daemon. LCE OPERATIONS At any time, the version of the lced binary can be determined by running it with the -v option, as follows: # /opt/lce/daemons/lced -v Log Correlation Engine version 3.6 # Use the following command to see how the LCE is configured during Linux startup and shutdown: # chkconfig --list lce To change how the LCE will behave during Linux startup and shutdown use the following command: # chkconfig [--level <levels>] lce <on/off/reset>) Please refer to your own Red Hat Linux documentation on how to use chkconfig in conjunction with Linux run levels to configure the LCE startup and shutdown to your requirements. STARTING LCE The RPM installation places a LCE start-up (RC) script in /etc/rc.d/init.d. Use the following command to start the LCE: # service lce start If the lced daemon terminates abnormally for any reason, the system will automatically restart the daemon and add a warning to the LCE logs. HALTING LCE Similarly, the RC script can be used to halt the LCE and gracefully exit any log analysis or log writing it is performing. Use the following command to stop the LCE server: # service lce stop OPERATING THE STATS DAEMON Although this document does not cover all aspects of the stats daemon, a separate RC script is included in the LCE RPM for starting and stopping the daemon. Use the following commands to stop and start the stats daemon: Copyright Tenable Network Security, Inc. 33

34 # service stats stop # service stats start UPGRADING LCE The LCE is upgraded simply by using the rpm command with the -U switch to force an upgrade. The LCE stops and starts the service during the upgrade process, which makes a manual stop/start unnecessary. # rpm -U lce-3.x-es5.i386.rpm ADDITIONAL FEATURES /ALERTING/EXECUTION Starting in LCE 3.4, LCE can be configured with the ability to interpret received log events based on log content and use configurable rules to generate active responses from the LCE server. These rules are configured on the LCE server in the /opt/lce/daemons/rules.conf file and can perform three primary responses: > alerting > syslog alerting > command execution Examples of practical applications include configuring rules to rate limit certain types of log events, administrators immediately when an attack is detected and send customized commands to a firewall when an inbound attack is detected and firewall reconfiguration needs to take place. Various fields within the received log alert are automatically placed in variables that may be used as parameters within the active response. For example, consider the following rules.conf configuration: Name: DMZ Login Matching-IPS: , , Event: SC4-Login Command: send -subject "$Name" -body "$Log" -to "rgula@example.com,joe@example.com" RateLimit: 5m This rule takes LCE events labeled SC4-Login to the specified IP addresses and automatically generates an alert to the specified administrator addresses. In addition, a rate limit is applied such that only one would be sent every five minutes to prevent the LCE server from overwhelming the server system. Configuration possibilities are limited only by the imagination of the LCE server administrator. Copyright Tenable Network Security, Inc. 34

35 IMPORTING LCE DATA MANUALLY LCE data can be collected both via real-time logging and manually in batch mode using the import_logs tool. These events will show up in the normalized event view along with events collected in real-time. This command-line tool allows data to be imported into the LCE that may not be available in real-time, but is still important for correlation of vulnerability data and for analysis of security posture and events. Usage: # /opt/lce/tools/import_logs <list of log files and directories to import> [-d, --disable-rules] [-a, --approximate-timestamps] [-c, -- current-time] [-o, --output-prefix <prefix>] Each item in the <list of log files and directories to import> is a file name or directory name. A directory name may or not end with a slash. For example: # /opt/lce/tools/import_logs /directory1 file1 file2 /directory2/ Directory imports are non-recursive. The following table describes the options available for import_logs: Option Description -d disable-rules Do not apply LCE event rules to imported logs. -a, --approximatetimestamps -c, --current-time -o, --output-prefix <prefix> If no timestamp can be determined for an event, assign the most recent known timestamp. Use the current system time for all imported logs rather than the timestamps contained within the event text. Use the specified prefix when naming newly generated silos. For example, the -o Snort option will generate silos with names like SnortJun Aug db.gz. The default prefix is lce. This option can aid in the process of searching for logs created by a particular import instance. The log importer tool logs its actions to /opt/lce/admin/log/importer and archives within this directory can be checked in the event that an import does not go as expected. DISK USAGE PREDICTION (SECURITY CENTER 3 ONLY) Intelligent disk space monitoring has is enabled when enable-log-archiving is set. This monitoring uses a complex algorithm to look at past disk usage trends and current disk usage to determine the point in time when the system is likely to run out of disk space. Disk usage prediction results are available in the Security Center LCE Archive Status user interface. An example screen capture is included below: Copyright Tenable Network Security, Inc. 35

36 USER TRACKING The LCE server has a feature that is designed to track users. User tracking can be applied to any event coming into the LCE server, regardless of the source of the event. Events correlated from Windows, Unix or other network devices can be monitored. When LCE encounters a log that has no username field, it will assign the username of the user most recently associated with the source IP of the incoming log, or associated with the destination IP of the log if a destination IP (dstip) is provided but a source IP (srcip) is not. If no user was previously tracked at either of the IPs, or if no IP is provided, an (unknown) entry is assigned. When a user changes IP addresses (i.e., a LCE receives a log where the user s srcip differs from the srcip in the previous log tagged with the username), the new IP address is also associated with the user. The last three IP addresses per user are stored for the user, allowing for cases where a single user logs into multiple systems at the same time. For example, the following event shows a user becoming active at a new IP address: Copyright Tenable Network Security, Inc. 36

37 Network user IP address change: user aolguy94 became active at with event login ( :0) The data used to track usernames is stored in the files usernames.txt, ip_user.dat and user_ip.dat in the LCE database directory. The.dat files are written when the LCE service is shut down gracefully. In case of a server crash, the data is automatically backed up every 10 minutes. A maximum of 65,534 unique usernames can be stored. If the maximum is reached, incoming logs with new users will have the user fields marked with the (unknown) entry. User tracking in LCE will function if the following conditions are met: > The LCE server has plugins that can match the events and pull usernames from the events. For example, plugin 3209 in os_win2k_sec.prm has the following line: log=event:windows-account_used_for_login sensor:$1 dstip:$2 type:login user:$4 The user:$4 directive tells the plugin to add the username to the available event searchable fields. As a result, searches that query this event based on the username will return results. > The plugin IDs have been added to the /opt/lce/admin/trusted_plugins.txt configuration file (one plugin ID per line). A list of the plugins provided by Tenable that include user information is found at the end of /opt/lce/daemons/plugins/prm_map.prm. > The user tracking settings have been properly configured in the lce.conf file. Please refer to the Configuration section of this document for a description of the following applicable keywords: > accept-letters > accept-numbers > additional-valid-characters > max-username-characters If these conditions are not met, usernames may still be stored in normalized events; however, they cannot be searched using the event filter username parameter. Another way to search for usernames in logs is through the raw log search feature of SecurityCenter described below. SAVE/SEARCH RAW LOGS Copyright Tenable Network Security, Inc. 37

38 A maximum of 20,000 raw log events can be retrieved per LCE server per query. This can lead to an apparent discrepancy in counts between raw log events and event counts from other views. Users have the ability to save and search all logs that have been processed by LCE. If the log archiving feature is used as a replacement for the save-all feature, overall storage requirements will actually be reduced. Archived logs are expected to compress to 3-10% of the original data size. Additionally, data retrieved using log archiving may come from multiple LCE sources compared to the single source supported by save-all. This ability is controlled by the enable-log-archiving parameter in lce.conf and is disabled by default. If disk space resources are not an issue, this parameter may be safely enabled to allow full archive searches and archive statistical analysis. Two supporting options are included in lce.conf, disk-alert-percentage and archive-directory. The directory entry for archive-directory specifies where compressed raw logs are to be stored. The disk-alert-percentage option generates a daily TASL alert if disk usage reaches 75% or higher. These options can be modified as needed and are only available when enable-log-archiving is enabled. Since the raw log data is compressed, disk usage is minimized and access time is optimized. Additionally, queries of multiple LCE servers provide a better overall view of data importance. For example, if you suspect a worm outbreak in the enterprise and know that a particular keyword was being written to the logs every time a host was infected, you could perform a strategic keyword search against all LCE servers in the enterprise to gain a better perspective of the overall range of the worm s penetration. This feature complements existing log search functionality and further extends the power of log correlation. In addition to this SecurityCenter graphical log search, users may perform raw log data searches from the LCE Server command-line. For more information and examples about this functionality, see the LCE Log Normalization Guide available through the Tenable Support Portal. WORKING WITH SECURITYCENTER OR LCE MANAGER ADDING THE LCE TO SECURITYCENTER/LCE MANAGER To add your LCE server to SecurityCenter, log into SecurityCenter/LCE manager as the admin user and click on Resources and then Log Correlation Engines. A screen similar to the one below is displayed with the currently available LCE servers. The Add button displays a dialog box with the following fields: Copyright Tenable Network Security, Inc. 38

39 Option Name Description The unique name that this LCE server will be known as. Description Descriptive text for the LCE server. IP address The IP address of the LCE server. Managed Ranges Organizations Addresses that will be monitored by the LCE. Specify an IP Address (x.x.x.x), IP Range (x.x.x.x-x.x.x.y), CIDR Block (x.x.x.x/24) or a combination of these separated by commas. The space character is not allowed within the range specification. This option is used to indicate IP ranges that will be classified as Internal ranges for classifying event direction. Select the customer that this LCE is assigned to from the drop down menu. This field does not apply to systems being managed by a LCE manager since only a single organization is supported for each LCE manager. An example of this screen is shown below: After clicking on Submit, the LCE admin credentials (usually root user) are requested to establish an authenticated session between SecurityCenter and the LCE. After the LCE server is successfully added, highlight the new LCE server to display options pertinent to that server. Copyright Tenable Network Security, Inc. 39

40 If company policy prohibits remote root login, a manual key exchange process can be used. The key file can be generated manually by the tns user on the SecurityCenter using ssh-keygen or with the SecurityCenter administrator web interface ( System -> Keys ). The key would be copied between systems by an authorized user and then installed as described in the Appendix 4: Manual SSH Key Exchange section of this document. If you are using DNS in your environment, make sure it is configured for reverse DNS resolution to facilitate query speeds. If you are not using DNS, modify the /etc/hosts file to include your SecurityCenter or LCE Manager IP and hostname. For example: SecurityCenter4.example.com SecurityCenter4 More information about SecurityCenter configuration options is available through the SecurityCenter Administration Guide available on the Tenable Support Portal. CONFIGURING ORGANIZATIONS As a SecurityCenter administrator, LCE servers can be associated with various organizations. Through the web interface, SecurityCenter can be configured such that users of specific organizations can make queries to each LCE server. This is documented in the SecurityCenter Documentation. ANALYZING SECURITY EVENTS A wide variety of LCE analysis and reporting tools are available to SecurityCenter users. These users can make use of any LCE event that intersects with their range of managed IP addresses. All analysis and reporting options are described in the SecurityCenter 4 User Guide. TASL Scripts Sending IDS events to the LCE allows the LCE to run the PRM and.prmx rules to normalize the events and TASL scripts to provide further event processing. TASL scripts are used for many types of detection events such as thresholds, successful attack detection, and alerting. By default, all TASL scripts are included on the LCE server; however they can be disabled manually using the /opt/lce/admin/disabled-tasls.txt file described in detail earlier in this document. WORKING WITH THE LCE MANAGER The LCE Manager interface looks similar to the SecurityCenter interface, but does not include vulnerability management. It instead gives the user a means to collect, analyze and report on event activity throughout the enterprise. A single LCE Manager can manage several different LCE servers (daemons), which in turn can manage a large number of LCE clients. INSTALLING THE LCE STANDALONE MANAGER INTERFACE The LCE Manager interface is installed using the same RPM used to install the SecurityCenter. During the installation process, the user is given the option to install either the full SecurityCenter product or the LCE Manager. Copyright Tenable Network Security, Inc. 40

41 LCE MANAGER USE AND OPERATION Please refer to the LCE Manager User Guide for usage and administration guidance. The LCE Manager interface is very similar to that of the SecurityCenter with two distinctions: 1. Options related to vulnerabilities are not available. Event-related options are the same as those provided with the SecurityCenter. 2. The LCE Manager interface provides a single default Organization and Repository. Options for Organization and Repository configuration are not included to LCE Manager users. FOR MORE INFORMATION Tenable has produced a variety of additional documents detailing the LCE s deployment, configuration, user operation and overall testing. These documents are listed here: > Log Correlation Engine Architecture Guide provides a high-level view of LCE architecture and supported platforms/environments > Log Correlation Engine Client Guide how to configure, operate and manage the various Unix, Windows, netflow, OPSEC and other clients > Log Correlation Engine Log Normalization Guide explanation of the LCE s log parsing syntax with extensive examples of log parsing and manipulating the LCE s.prm libraries > TASL Reference Guide explanation of the Tenable Application Scripting Language with extensive examples of a variety of correlation rules > Log Correlation Engine Statistics Daemon Guide configuration, operation and theory of the LCE s statistic daemon used to discover behavioral anomalies > Log Correlation Engine SAN-DAS Guide configuration, operation and theory for using the LCE in large disk array environments Documentation is also available for Nessus, the Passive Vulnerability Scanner and SecurityCenter through the Tenable Support Portal located at There are also some relevant postings at Tenable s blog located at and at the Tenable Discussion Forums located at For further information, please contact Tenable at support@tenable.com, sales@tenable.com or visit our web site at Copyright Tenable Network Security, Inc. 41

42 ABOUT TENABLE NETWORK SECURITY Tenable Network Security, the leader in Unified Security Monitoring, is the source of the Nessus vulnerability scanner and the creator of enterprise-class, agentless solutions for the continuous monitoring of vulnerabilities, configuration weaknesses, data leakage, log management and compromise detection to help ensure network security and FDCC, FISMA, SANS CAG and PCI compliance. Tenable s award-winning products are utilized by many Global 2000 organizations and Government agencies to proactively minimize network risk. For more information, please visit Tenable Network Security, Inc Columbia Gateway Drive Suite 100 Columbia, MD Copyright Tenable Network Security, Inc. 42

43 APPENDIX 1: SAMPLE LCE.CONF FILE lce_options { plugins-directory /opt/lce/daemons/plugins/ correlation-scripts-directory /opt/lce/daemons/plugins/ daemons-directory /opt/lce/daemons/ database-directory /opt/lce/db/ log-directory /opt/lce/admin/log/ key-file /opt/lce/daemons/lce.key pid-file /opt/lce/daemons/lced.pid die-file /opt/lce/daemons/lced.die # The next section determines the interfaces on # which LCE will listen for both syslog messages and # connection requests from clients. For each address # specified, LCE will utilize the corresponding interface. # If this option is used, all clients must be configured # to connect to one of the addresses below. If no addresses # are specified, LCE will listen on all available interfaces. # server-address # The server-port option allows specification of a port # number on which LCE will listen for connection requests # from clients. The clients must be configured to connect # to the port indicated below. server-port # The silo-size variable specifies the maximum amount of # data from matched log events that will be stored in one # indexed file. This may be set to at most 4GB. The option # must be specified in kilobytes, megabytes, or gigabytes # by appending a 'K', 'M', or 'G' to the value. For example, # 512M indicates a 512 megabyte maximum silo size. # The 'number-silos' variable specifies the number of silos. # With a silo-size of 1G and number-silos set to 3, the # system would use a maximum of 3 gigabytes of total storage. silo-size 1024M # The full LCE license supports a maximum of 255 silos, # while the 3-Silo license can utilize only 3 silos. Please # change number-silos below to '3' for the smaller license. number-silos 200 # One IP address may be specified per forward-syslog keyword # which will forward any log event regardless of method # (OPSEC, log file, syslog, windows events, etc.) to a syslog # server. LCE forwards each event to multiple syslog servers # when multiple lines with the forward-syslog keyword are # included. # forward-syslog # forward-syslog # The IP specified below is used to indicate the address of # the Security Center. The setting is used by LCE TASL scripts # that wish to send correlated events. # securitycenter-ip # The lce_queryd process is responsible for high-performance processing of # all query requests. By default, up to about 1 gigabyte of memory is used. # For systems with large amounts of available memory, the following option # can be used to allocate additional memory for the query daemon. This will # increase the number of query results that can be cached, improving response # time during event analysis in SecurityCenter. The option can be specified in Copyright Tenable Network Security, Inc. 43

44 # kilobytes, megabytes, or gigabytes by appending a 'K', 'M', or 'G' to the # value. Note that this value should never be set above 1500M on 32-bit # systems. # additional-query-memory 1G # As events are sent to LCE, if a signature does not exist # for the log event, the data from the log will be stored in # a file named 'notmatched.txt' in the configured database # directory. The maximum number of log entries that will be # stored in the file is specified by the 'save-nonmatched' # variable below. In addition, there is a hard file size # limit of 2GB. If either the specified entry limit or the # 2GB file size limit is exceeded, notmatched.txt will be # deleted and opened again to store subsequent new events. # The save-nonmatched option is useful for seeing what kinds # of events are not being matched by the LCE plugins. save-nonmatched # The 'archive-directory' keyword specifies a directory under which # all log events received by LCE will be saved. The logs are highly # compressed, and are stored according to date using a unique technique # designed to allow historical information to be retained while using # minimal disk space and allowing efficient search. archive-directory /opt/lce/log_archive/ # Log archiving is enabled only when the following line is uncommented. enable-log-archiving # When log archiving is enabled, the following option allows an alert # to be generated when LCE becomes in danger of running out of disk # space to store logs. When disk usage reaches the specified percentage # of capacity, a daily TASL alert will be sent to Security Center. disk-alert-percentage 75 # When the following line is uncommented, client activity will be logged # to the LCE database. This includes connect, disconnect, and failed login # events. log-client-activity # The 'save-all' keyword allows all log events to be stored in a # specific file. When used with the 'logrotate' facility and storing # this file on a different physical file system, or possibly a # network file mount, extremely large amounts of data can be saved # by LCE. The 'save-all' keyword specifies the location of this # file. Any event sent to LCE or collected by an agent will be # stored in the file specified. Please consult the LCE documentation # for advice on configuring the 'logrotate' facility to store data for # a specific length of time, specific size of data, or to simply create # rotating log files. # save-all /opt/lce/db/lce.log # As the lced process collects events, the size of the current silo # grows. When the maximum silo size is reached, a new silo becomes the # active silo and if the maximum number of silos has been reached, an older # silo is overwritten or archived. To assist in disk space management and # performance, the lced process can be configured to execute a script which # invokes some house-keeping, compression and indexing of the collected data. # The provided script can also be invoked manually to force a silo roll. rollover-script /opt/lce/silo-manager.sh # When the maximum number of silos has been reached and an older silo # must be overwritten for the next silo roll, the silo to be overwritten # can first be saved for future use. The following options determine Copyright Tenable Network Security, Inc. 44

45 # whether the database, index and raw files are saved. Valid options for # each setting are "yes" and "no". The location setting allows the silo # to be saved to any mounted volume or other physical location. # # Saving the database alone is sufficient to allow most queries on the # archived data. Retaining the raw files in addition will permit raw # syslog queries, in which the original log message can be viewed. Saving # the index will allow queries on archived data to execute much faster, # but consume additional storage space. # save-database yes # save-index yes # save-raw yes # location /opt/lce/silo_archive/ # Each time the active silo changes, an MD5 checksum is generated for the # previous silo. This allows the intgerity of the.raw and.db data files # to be checked at a later time. The following section allows the MD5 # checksums to be forwarded to one or more off-site locations. The IP # address list may include the Security Center and any syslog servers. # On the Security Center, forwarded checksums will be reported as the # LCE-MD5_Checksums event. # checksum-forward # checksum-forward # The 'auto-update-prms' keyword allows PRM files to be automatically updated. # When enabled, LCE will update the installed PRM files with the latest # versions from Tenable, as well as download any newly published plugins. Both # this and the TASL update are launched whenever a silo rolls. auto-update-prms yes # The 'auto-update-tasls' keyword allows TASL scripts to be automatically # updated. When enabled, any TASL scripts currently installed will be # updated with the latest versions from Tenable. To ensure that only desired # TASL scripts are used, newly available scripts will not be installed. These # can be obtained manually, or with the /opt/lce/daemons/lce_update_plugins.pl # script. auto-update-tasls yes # The following options configure both the automatic and manual update # processes to use a web proxy server when downloading files from Tenable. # When these values are commented out, proxy use is disabled. # proxy-address # proxy-user username # proxy-password password # When a log message is defined in a plugin, LCE provides the option of # specifying a hostname instead of an IP address for the srcip and dstip # fields. In this case, LCE automatically attempts to resolve the provided # hostname to an IP address using DNS. Since the same hostname is typically # encountered multiple times, caching the results of lookups can greatly # increase performance. The following section configures DNS caching in # LCE. The cache-results-days option specifies the number of days for # which to cache a hostname-to-ip mapping before updating the result with # a new lookup. Hosts listed in the pre-cache-file are looked up and stored # when LCE starts. A particular hostname or all domain names with a certain # extension can be excluded using the exclude-domain-name option. In this # case, the matching hosts are looked up at every occurrence. max-memory-usage 100M cache-results-days 10 pre-cache-file /opt/lce/admin/hostlist.txt excluded-domains-file /opt/lce/admin/excluded_domains.txt # excluded-domain-name.edu # excluded-domain-name tenablesecurity.com Copyright Tenable Network Security, Inc. 45

46 # The Log Correlation Engine keeps track of network users on the basis of # their usernames. The following section sets restrictions on which usernames # are considered valid. Any usernames failing to match the below criteria # are disregarded, and "invalid" is reported as the user for their associated # logs. accept-letters yes accept-numbers yes additional-valid-characters -_.@ max-username-characters 20 # The following setting defines a list of usernames whose IP addresses are # not tracked. These usernames will appear with their associated logs, but # LCE will not generate alerts when the users with a matching name switch # to a new IP address. excluded-users-file /opt/lce/admin/untracked_usernames.txt # In order to apply user tracking only to appropriate logs, a list of # trusted plugins is provided in the following file. If a plugin is not # on this list, any specified username will appear with the associated # log if it matches the above criteria, but otherwise will not be tracked. # Each plugin is specified in the file by its plugin ID. trusted-plugins-file /opt/lce/admin/trusted_plugins.txt # Each client that communicates with LCE periodically sends a "heartbeat" # message indicating that it is active. This message contains the hostname # and IP address of the client machine, as well as the revision number of # the application. This information is displayed at the Security Center # console when the following option is enabled. display-heartbeat-messages yes # The Log Correlation Engine TASL processor has the ability to periodically # report the performance of scripts as they are applied to incoming events. # The following option, in minutes, represents how frequently a report will # be generated. The latest report can be found in a file named tasl.log # in the log directory specified above. tasl-report-frequency 60 # In addition to the performance report, the TASL processor logs detailed # technical information related to scripts such as syntax and runtime errors. # This data is written to a file called tasl_scripts.log in the log directory. # Since errors in scripts cause log entries to be generated each time they # are encountered, this log can potentially grow large. When set to yes, the # following option causes the log file to be reset each time the scripts are # automatically updated. As a result, all log messages will be relevant to the # currently installed scripts after an update. clear-tasl-log-on-update yes # In order to maximize performance on multi-processor and multi-core systems, # correlated TASL events are processed in parallel to the receiving of # regular incoming events. Since some tasl scripts can run for an extended # period of time, the primary event processor can potentially receive many # tasl-triggering events while a tasl script is still being executed. In # this case, the tasl job is stored in a queue for later processing. The # following option defines the maximum size of this queue. On systems with # extremely large volumes of data, setting the maximum queue size higher # results in increased performance. If a tasl script designated as being # sampleable is triggered while the queue is full, its callback functions # will not be executed. max-tasl-queue-memory 768M # The following file contains the filenames of tasl scripts that are # designated as being sampleable, which results in the behavior described Copyright Tenable Network Security, Inc. 46

47 # above. sampleable-tasls-file /opt/lce/admin/sampleable_tasls.txt # For logs received via syslog, a sensor name can be assigned to each IP # address sending data to LCE. This sensor name will be associated with # all logs from the designated source, regardless of whether or not another # sensor name is extracted from the log text. syslog-sensors-file /opt/lce/admin/syslog_sensors.txt # The Log Correlation Engine has the ability to receive IDS events from # multiple sources. In addition to being normalized and stored in the # log database, each event will be checked against any Security Center # vulnerability databases. If a host is vulnerable to attack, the event # is marked as such, allowing rules to trigger on this scenario so that # the information can be distributed to the affected administrators. # For each IDS sensor, a sensor name and type must be defined as in the # example below. The supported types are Snort, Bro, RealSecure, Dragon, # IntruVert, IntruShield, NetScreen, NFR, Fortinet, PVS, LCE, Cisco, # TippingPoint-Sensor, and TippingPoint-SMS. IDS { sensor-name Enterprise_Snort sensor-type Snort } IDS { sensor-name Win_Snort sensor-type Snort } # The LCE daemon can accept syslog events from any system. Only clients # that are specified in this section will be allowed to send data to LCE. # This is achieved by associating an IP address and a shared secret key with # every client. Currently, the LCE clients only support authentication known # as the 'auth-secret-key'. Other types of authentication may be available # in future releases. The key specified below for each client must # match the key specified in the LCE client application. The optional # sensor-name field defines a default sensor name to be reported for logs # from a particular client that do not provide a sensor name to their # associated plugin rule. # # Several formats are supported for specifying client information. These # are (1) a single IP address, (2) an IP address with a CIDR range, # (3) optional ranges in the third and fourth octets of the IP address, # and (4) a range specified by start and end addresses. # Examples of each follow. In every case, the authentication and sensor # name defined within the block applies to every client covered by the # chosen notation. # # (1) Example client connecting from localhost with key of 's3kret': client { client-auth auth-secret-key s3kret sensor-name Default_Sensor } # # (2) Example of specifying 64 clients with key 's3kret' using a CIDR range: # client /26 { # client-auth auth-secret-key s3kret # } # # (3) Example of specifying a range of 16 clients, from to : # client { # client-auth auth-secret-key s3kret # } Copyright Tenable Network Security, Inc. 47

48 # # (4) Example of specifying a range of 5 clients, from to # client { # client-auth auth-secret-key s3kret # } } # The following section should be completely uncommented for additional # logging generated by the lced process. WARNING - the debug logic # is extremely verbose and several Tenable customers have created log # files greater than 4 GB. These logs are purely for diagnostic purposes. # debug { # lce-api { # client-auth # } # # matches { # match-success # } # # silo { # silo-switch # } # # match-tree { # tree-matches # } # } stats_options { # Directory containing the LCE events.txt file event-directory /opt/lce/db # Log files are named according to date and stored in # the following directory log-directory /opt/lce/admin/log/stats/ # Syslog alerting # The stats dameon will generate SYSLOG messages to one or more # recipients. By default, a local address of is used to # send messages to the lced services. However, more than one # SYSLOG server can be configured. syslog-alert # The minimum standard deviation that must occur for an event # before we'll alert on it. The higher this number, the more # statistically significant a sequence of events needs to be # before an alert is raised. stddev-threshold 1.0 # If an event occurs more/less than 5.0 standard deviation units # then we'll alert. Setting this value higher, will cut down on # any sequence of events which occur close to the standard deviation. stddev-unit-threshold 5.0 # The number of iterations (days) per-event required before # we'll start alerting for an event. If a large amount of LCE # data is already present, this number should be set low or even # to zero. The stats daemon can be started to read in all or just # part of the existing LCE data. If you have NO LCE data, # you are encouraged to leave this value around 7 which means the # stats daemon won't alert on anything until it has 7 days of Copyright Tenable Network Security, Inc. 48

49 # event data. Tenable recommends not running the stats daemon until # at least 7 days of traffic have been collected. iteration-threshold 7 } # If an event occurs less than 10% of the time then we'll alert. # Even if an event may be statistically significant, that sequence # of events may also occur periodically. For example, 50% of the # time you are within a standard deviation, however, occasionally, # (the other 50%) you have outliers 2 and 3 standard deviations # away. Those outliers may be the cause of 90% of the alerts thrown # in this case. Setting this value to 10, 20 or other values would # only alert for hours which were both out of the allotted standard # deviation, and also have been event counts which have not occurred # before. nhits-frequency-threshold 10.0 # The following sections define your network range. All networks specified # in the first section are included, while the exclude-networks block is # used to make exceptions. include-networks { /24; } exclude-networks { } Copyright Tenable Network Security, Inc. 49

50 APPENDIX 2: TROUBLESHOOTING The following are troubleshooting steps for determining LCE client/server functionality: 1. Install and configure the LCE and clients by following the instructions in the documentation. 2. Verify the clients are connecting by viewing the file /opt/lce/admin/log/client.status. a. If the clients never connect, review configuration. b. If the configuration is correct, then there is a network issue. Check for proxies, firewalls or ACLs that may be blocking traffic. c. If the clients connect but do not stay connected, continue to test. 3. The LCE client will not remain connected with the LCE server unless the client has some data to send. To force a client to forward data to the LCE server, an observed log on the LCE client machine can be appended with entries that are known to cause alerts within SC4. This gives the LCE client some data to send to the server. It is advised to put TEST OF FUNCTIONALITY in the beginning of the log entries to ensure that these tests do not interfere with actual alerts. Check your client logs to ensure communication is taking place. a. Yes? Communication is taking place. Continue to Step 4. b. No? Contact Tenable Support for an LCE Client Issue. 4. Once the logs are appended, check the client.status file. Has it changed? a. Yes? Functionality is working. b. No? Continue with next step. 5. Check SC4 for the IP in question and the time of the test. Were there entries found? a. Yes? Your LCE is functioning properly. However, there may be an issue with the client.status heartbeat. Notify Tenable Support of the issue. b. No? Continue to the next step. 6. Grep the logs in the LCE s notmatched.txt file for the IP in question and the time of test. Were there entries found? a. Yes? Your LCE is functioning and logs are being updated properly. However there may be an issue with the client.status heartbeat. Notify Tenable Support of the issue. b. No? Continue to the next step. 7. Perform a TCPDump on the LCE and capture traffic from the IP of the client in question. Repeat step 3 to force communications. Did you receive traffic? a. Yes? Notify Tenable Support of the issue for further assistance. b. No? You have a network issue. Please work with your network support to troubleshoot the issue. Copyright Tenable Network Security, Inc. 50

51 APPENDIX 3: MANUAL SC4/LCE KEY EXCHANGE A manual key exchange between SecurityCenter and the LCE is normally not required; however, in some cases where remote root login is prohibited or key exchange debugging is required, you will need to manually exchange the keys. For the remote LCE to recognize SecurityCenter, you need to copy the SSH public key of SecurityCenter and append it to the /opt/lce/.ssh/authorized_keys file. The /opt/lce/daemons/lce-install-key.sh script performs this function. The following steps describe how to complete this process: The LCE server must have a valid license key installed and the LCE daemon must be running before performing the steps below. 1. Download the SSH public key for SecurityCenter by logging in as the SecurityCenter administrator user and navigating to the Keys section ( System -> Keys ). 2. Click on Download Key, choose the desired key format (both DSA or RSA work for this process) and then click on submit. 3. Save the key file (SSHKey.pub) to your local workstation. Do not edit the file or save it to any specific file type. 4. From the workstation where you downloaded the key file, use a secure copy program, such as scp or WinSCP to copy the SSHKey.pub file to the LCE system. You will need to have the credentials of an authorized user on the LCE server to perform this step. For example, if you have a user bob configured on the LCE server (hostname lceserver ) whose home directory is /home/bob, the command on a Unix system would be as follows: # scp SSHKey.pub bob@lceserver:/home/bob 5. On the LCE server, as the root user, change the ownership of the ssh key file to lce as follows: # chown lce /home/bob/sshkey.pub Then append the SSH public key to the /opt/lce/.ssh/authorized_keys file with the following steps: # su lce # /opt/lce/daemons/lce-install-key.sh /home/bob/sshkey.pub 6. To test the communication, as the user tns on the SecurityCenter system, attempt to run the id command: # su tns # ssh -C -o PreferredAuthentications=publickey lce@<lce-ip> id If a connection has not been previously established, you will see a warning similar to the following: Copyright Tenable Network Security, Inc. 51

52 The authenticity of host ' ( )' can't be established. RSA key fingerprint is 86:63:b6:c3:b4:3b:ba:96:5c:b6:d4:42:b5:45:37:7f. Are you sure you want to continue connecting (yes/no)? Answer yes to this prompt. If the key exchange worked correctly, a message similar to the following will be displayed: # uid=251(lce) gid=251(lce) groups=251(lce) 7. The IP address of SecurityCenter can be added to the LCE system s /etc/hosts file. This prevents the SSH daemon from performing a DNS lookup that can add seconds to your query times. 8. The LCE can now be added to SecurityCenter via the normal administrator LCE add process documented in the SecurityCenter Administration Guide. Copyright Tenable Network Security, Inc. 52

53 APPENDIX 4: MANUAL SC3/LCE KEY EXCHANGE MANUAL SSH KEY EXCHANGE This process is only required for debugging purposes and custom configurations where the add-security-center.sh script is not able to perform the key exchange. See Appendix 3: Manual SC4/LCE Key Exchange for detailed guidance on performing a manual key exchange between SecurityCenter 4 and LCE. When the Security Center is installed, it runs as user tns. The SSH public keys for user tns are available for download from the Security Center administrator interface. From the Console tab on the Security Center, click on the Add/Remove a Log Correlation Engine selection. At the bottom of this form will be three links for various Secure Shell version 1, version 2 and version 1 and 2 RSA public keys as shown below: Links to download SSH keys The form also details a step-by-step manual process for putting the SSH public keys in place depending on which directory your LCE has been installed. Instructions for an LCE installed in the directory /opt/lce are shown below: Copyright Tenable Network Security, Inc. 53

54 Configuring SSH shared public keys DOWNLOAD KEYS TO THE LOG CORRELATION ENGINE If desired, the SSH public keys can be downloaded via a web interface and then transferred to the system running the LCE. However, it may be easier to use the secure copy (scp) tool to download the files directly to the LCE system. On the LCE, log in as root and reset the password for the lce user. This is required for transferring the SSH key to the LCE. Next, on the Security Center, log in as root and then use the secure copy tool to send the SSH public key to the LCE with the following Copyright Tenable Network Security, Inc. 54