User Guide for Proventia Server IPS for Linux

Transcription

1 IBM Proventia Server Intrusion Prevention System User Guide for Proventia Server IPS for Linux Version 1.0 IBM Internet Security Systems

2 Copyright IBM Corporation 2006, IBM Global Services Route 100 Somers, NY U.S.A. Produced in the United States of America. All Rights Reserved. Protected by U.S. Patent No. 7,093,239. Other patents pending. IBM and the IBM logo are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. ADDME, Ahead of the threat, BlackICE, Internet Scanner, Proventia, RealSecure, SecurePartner, SecurityFusion, SiteProtector, System Scanner, Virtual Patch, X-Force and X-Press Update are trademarks or registered trademarks of Internet Security Systems, Inc. in the United States, other countries, or both. Internet Security Systems, Inc. is a wholly-owned subsidiary of International Business Machines Corporation. Microsoft, Windows, and Windows NT are trademarks of Microsoft Corporation in the United States, other countries, or both. Other company, product and service names may be trademarks or service marks of others. References in this publication to IBM products or services do not imply that IBM intends to make them available in all countries in which IBM operates. Disclaimer: The information contained in this document may change without notice, and may have been altered or changed if you have received it from a source other than IBM Internet Security Systems (IBM ISS). Use of this information constitutes acceptance for use in an AS IS condition, without warranties of any kind, and any use of this information is at the user s own risk. IBM Internet Security Systems disclaims all warranties, either expressed or implied, including the warranties of merchantability and fitness for a particular purpose. In no event shall IBM ISS be liable for any damages whatsoever, including direct, indirect, incidental, consequential or special damages, arising from the use or dissemination hereof, even if IBM Internet Security Systems has been advised of the possibility of such damages. Some states do not allow the exclusion or limitation of liability for consequential or incidental damages, so the foregoing limitation may not apply. Reference herein to any specific commercial products, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by IBM Internet Security Systems. The views and opinions of authors expressed herein do not necessarily state or reflect those of IBM Internet Security Systems, and shall not be used for advertising or product endorsement purposes. Links and addresses to Internet resources are inspected thoroughly prior to release, but the ever-changing nature of the Internet prevents IBM Internet Security Systems, Inc. from guaranteeing the content or existence of the resource. When possible, the reference contains alternate sites or keywords that could be used to acquire the information by other methods. If you find a broken or inappropriate link, please send an with the topic name, link, and its behavior to support@iss.net. December 14, 2007

3 Contents Preface Overview How to Use Proventia Server IPS Documentation Getting Technical Support Chapter 1: to Proventia Server IPS Overview About the Proventia Server IPS Agent Understanding the Intrusion Prevention System Architecture How a Proventia Server IPS Agent Works About Agent Policies Chapter 2: Configuring Firewall Protection Overview About the Firewall Precedence of Firewall Rules Configuring an IP or ICMP Firewall Rule Configuring a TCP or UDP Firewall Rule Deleting a Firewall Rule Chapter 3: Configuring Network Protection Overview Global Action Setting Customizing a Pre-Defined Security Event Signature Chapter 4: Monitoring System Integrity and Policy Compliance Overview Customizing a Pre-Defined Log Monitoring Signature Using Regular Expressions in User-Defined Signatures Monitoring for a Custom WTMP Log Event Monitoring for a Custom Syslog Chapter 5: Configuring Buffer Overflow Exploit Prevention Overview What is Buffer Overflow Exploit Prevention? Configuring Global Buffer Overflow Exploit Prevention Changing the Action for a Monitored Application Excluding an Application from Protection Chapter 6: Maintaining the Proventia Server IPS Agent Overview Section A: Refining Agent Behavior Overview Monitoring Agent Activity Storing Alerts Configuring the Agent Heartbeat Interval Reducing the Number of Alerts Sent to SiteProtector Stop Monitoring a Specific Network Interface Card User Guide for Proventia Server IPS for Linux, Version 1.0 3

4 Contents Changing the IP Address or Host Name of the Agent System Restarting the Proventia Server IPS Agent Section B: Configuring Responses Overview About Responses Configuring Site Group-Level Responses Configuring Agent-Level Responses Section C: Updating Your Agent Overview Updating Your Agent Configuring the List of Update Servers Fine-Tuning Update Settings Section D: Logging Intrusion Attempts Overview Logging Packets from Intrusion Attempts Section E: Fine-Tuning Your Agent Overview Configuring Advanced Tuning Parameters Chapter 7: Troubleshooting Overview Refresh Agent Feature in SiteProtector Not Functioning Seeing Alerts for Traffic From Accepted IP Addresses Agent Providing No Buffer Overflow Exploit Protection Console Reporting a Runlevel_Switched Event Not Seeing Network or Firewall Alerts Agent Appearing as Offline Index IBM Internet Security Systems

5 Preface Overview This guide provides the information you need to configure an IBM Proventia Server Intrusion Prevention System for Linux (Proventia Server IPS) agent. After you read this guide, you should be able to configure a Proventia Server IPS agent to protect your servers from attacks and misuse. Scope This guide contains instructions for configuring, maintaining, and troubleshooting Proventia Server IPS agents. Audience This guide is for security managers who manage Proventia Server IPS agents from the SiteProtector Management System. What s New With the addition of support for 64-bit systems, this guide was updated to reflect that Buffer Overflow Exploit Protection is not currently supported on 64-bit systems. User Guide for Proventia Server IPS for Linux, Version 1.0 5

6 Preface How to Use Proventia Server IPS Documentation Using this guide Refer to this guide as you configure and use Proventia Server IPS agents. Related publications For additional information about agents or about SiteProtector, see the following publications: Installation Guide for Proventia Server IPS for Linux SiteProtector Installation Guide SiteProtector Configuration Guide SiteProtector Policies and Responses Configuration Guide 6 IBM Internet Security Systems

7 Getting Technical Support Getting Technical Support IBM ISS provides technical support through its Web site and by or telephone. The IBM ISS Web site The IBM Internet Security Systems (IBM ISS) Resource Center Web site ( provides direct access to online user documentation, current versions listings, detailed product literature, white papers, and the Technical Support Knowledgebase. Support levels IBM ISS offers three levels of support: Standard Select Premium Each level provides you with 24x7 telephone and electronic support. Select and Premium services provide more features and benefits than the Standard service. Contact Client Services at if you do not know the level of support your organization has selected. Hours of support The following table provides hours for Technical Support at the Americas and other locations: Location Americas All other locations Hours 24 hours a day Monday through Friday, 9:00 A.M. to 6:00 P.M. during their local time, excluding IBM ISS published holidays Note: If your local support office is located outside the Americas, you may call or send an to the Americas office for help during off-hours. Table 1: Hours for technical support Contact information The following table provides electronic support information and telephone numbers for technical support requests: Regional Office North America Electronic Support Connect to the MYISS section of our Web site: Telephone Number Standard: (1) (888) (toll free) (1) (404) Select and Premium: Refer to your Welcome Kit or call your Primary Designated Contact for this information. Latin America support@iss.net (1) (888) (toll free) (1) (404) Table 2: Contact information for technical support User Guide for Proventia Server IPS for Linux, Version 1.0 7

8 Preface Regional Office Electronic Support Telephone Number Europe, Middle East, and Africa (44) (1753) Asia-Pacific, Australia, and the Philippines (1) (888) (toll free) (1) (404) Japan Domestic: (81) (3) Table 2: Contact information for technical support 8 IBM Internet Security Systems

9 Chapter 1 to Proventia Server IPS Overview This chapter provides an overview of the Proventia Server IPS agent and describes how it uses a layered security approach to protect your servers. In this chapter This chapter contains the following topics: Topic Page About the Proventia Server IPS Agent 10 Understanding the Intrusion Prevention System 11 Architecture 12 How a Proventia Server IPS Agent Works 13 About Agent Policies 15 User Guide for Proventia Server IPS for Linux, Version 1.0 9

10 Chapter 1: to Proventia Server IPS About the Proventia Server IPS Agent The Proventia Server IPS agent protects servers from the growing spectrum of threats while enabling the servers to keep data and applications reliable, available, and confidential. The agent combines a proven intrusion prevention system with real-time monitoring and analysis of the server operating system, applications, and network activity to safeguard the server environment from misuse and intrusions. Management Manage Proventia Server IPS agents with SiteProtector Version 2.0, Service Pack 6 or later. Layered protection The Proventia Server IPS agent offers a layered approach to security that provides greater overall protection for your system because of its multiple lines of defense: Component Firewall (FW) Intrusion Prevention System (IPS) Operating System Events (OS Events) Buffer Overflow Exploit Prevention (BOEP) Description The firewall is the first line of defense against a network-based attack. The firewall can block incoming or outgoing packets from particular IP addresses, port numbers, or protocols. It blocks many network attacks before they can affect the system. As IP traffic enters or leaves your system, the IPS analyzes it for malicious content. The IPS drops offending packets, and allows the remaining traffic to continue unhindered. Operating system events detect threats to system integrity and policy compliance through entries in system log files. By monitoring changes to log files, the agent can warn you of system activity and allow you to mitigate damage to your system as a result of malicious activity. The BOEP component is the last line of defense against attacks. It comes into play only after the agent has employed and exhausted all other protection methods. This component blocks worms and other malicious code that attempts to exploit buffer overflow vulnerabilities to propagate or gain access to a system. Depending on the type of exploit, the target, and the selected response, the application or service may need to be restarted. Table 3: Components of the Proventia Server IPS agent 10 IBM Internet Security Systems

11 Understanding the Intrusion Prevention System Understanding the Intrusion Prevention System The Intrusion Prevention System (IPS) analyzes packets as they enter or exit the system. Where most other protection products look at exploits through static signatures, IPS identifies the underlying protocol and determines the associated vulnerabilities. What is the Protocol Analysis Module (PAM)? The Protocol Analysis Module (PAM) combines advanced protocol anomaly detection with proven, signature-based detection technology to interpret network activity and to detect attacks contained within any IP based protocol before they reach an application. Updating PAM IBM Internet Security Systems (IBM ISS) keeps PAM information current through the X- Press Update (XPU) mechanism. The agent periodically checks for updates and downloads any XPUs from an Update Server. For information about updates, see Section C, "Updating Your Agent" starting on page 61. User Guide for Proventia Server IPS for Linux, Version

12 Chapter 1: to Proventia Server IPS Architecture The following figure illustrates the architecture of a Proventia Server IPS agent/ SiteProtector deployment: Figure 1: Proventia Server IPS agent/siteprotector architecture 12 IBM Internet Security Systems

13 How a Proventia Server IPS Agent Works How a Proventia Server IPS Agent Works This topic provides a high-level overview of how a Proventia Server IPS agent protects your system against attacks and misuse. Managing agent behavior To manage the protection the agent provides, and to control how the agent responds to security events, you must configure policies. Configure the following policies to manage agent behavior: Policy Firewall Security Events BOEP OS Events Update Settings Tuning Parameters Agent Properties Agent Responses Description Blocks packets that meet a certain criteria or monitors traffic without blocking packets. Monitors activity and optionally drops packets that might indicate an attack or suspicious activity. Monitors for programmatic behaviors typical of buffer overflow exploits. Monitors system integrity and policy compliance through log monitoring. Defines when product updates should be downloaded and applied. Fine-tunes agent behavior. Defines how the agent manages internal error, warning, and informational alerts and how the agent stores these alerts if communication between the agent and SiteProtector is interrupted. Note: There is one Agent Properties policy for each agent. Defines agent-level response objects, which allow you to centrally configure the agent s user-specified response. Table 4: Proventia Server IPS agent policies Reference: See About Agent Policies on page 15. Heartbeat mechanism Agents use a heartbeat mechanism to connect to SiteProtector. A heartbeat is a periodic communication attempt to check for updated policies. The heartbeat interval indicates the amount of time that elapses between heartbeats. Reference: See Configuring the Agent Heartbeat Interval on page 48. Updating policies When the agent connects to the Console, the agent supplies the Console with information, such as the checksum of the policy the agent is currently using. If you have made configuration changes since the agent last connected to the Console, then the checksum of the policy that resides in the Console differs from checksum of the policy that resides in the agent. SiteProtector detects that the checksum is different, and sends the newer policy to the agent. Note: You can force a policy update between heartbeat intervals by using the SiteProtector Refresh Agent feature. User Guide for Proventia Server IPS for Linux, Version

14 Chapter 1: to Proventia Server IPS Updating agents IBM ISS issues frequent updates for agents. To ensure that agents protect your system effectively, you must stay current with these updates. You configure how frequently the agent checks for updates. When the agent connects to the Update Server, the agent determines whether there are any updates available. If there are updates available, the agent does one of the following, depending on how you have configured the agent: downloads and installs the update updates the SiteProtector Console to indicate that an update is available Reference: For more information about product updates, see Updating Your Agent on page IBM Internet Security Systems

15 About Agent Policies About Agent Policies You should have a carefully planned corporate security policy in place before you attempt to configure policies for a Proventia Server IPS agent. A corporate security policy should allow access to required resources, while also providing protection against the risks that open resources can represent. Policies define the behavior of the agent. You must correctly configure and apply policies to get the protection you desire. Policy inheritance Policy inheritance provides an efficient way to use policies based on parent and child relationships. Child groups or child agents can inherit policies from parent groups, or you can override inheritance from a parent by defining a policy at the child level. An agent or group uses the policy defined at its own level or the policy defined at the nearest parent where a policy of the same type is defined. Important: The effects of policy inheritance can be far reaching. Review the policy inheritance information in the SiteProtector Policies and Responses Configuration Guide before you define policies for a Proventia Server IPS agent. Example You configure the policies for a group of assets called Mail Servers. There are 10 subgroups in the Mail Servers group. If all of the subgroups use the same policy settings, the policy settings can pass from the Mail Servers group to all 10 subgroups. If certain subgroups in the Mail Servers group require customized policy settings, you can define a customized policy at the subgroup level for that policy. This approach is much quicker than configuring the policy settings for all 10 subgroups. Policies that cannot be inherited The Agent Properties policy cannot be inherited as this policy defines the properties for each individual agent installed on a system. Policies that appear in the tree in the left pane If a policy is defined at a particular group or agent, this policy is listed as a member of the group or agent when you expand the group or agent in the navigation pane. When you select a policy in the navigation pane, the contents for that policy appear in the right pane. Policies that appear in a table in the right pane When you select a group or agent in the navigation pane, a list of policies that are in use for the selected object appears in the right pane. Overriding a policy for a group or agent If you need to customize a policy for an agent in a group or for a group in a Site, you can define a different policy (override the current policy) at whichever level you need. When are policy updates applied to the agent? When you make changes to a policy, the changes are saved immediately. It may, however, take some time before you see a change in agent behavior for the following reasons: large groups of agents take longer to update the agent only periodically checks for policy updates Note: You can force an agent to apply an updated policy immediately by using the SiteProtector Refresh Agent feature. While an agent updates a policy (this process only usually takes a few seconds), it allows all traffic to pass through the system. User Guide for Proventia Server IPS for Linux, Version

16 Chapter 1: to Proventia Server IPS 16 IBM Internet Security Systems

17 Chapter 2 Configuring Firewall Protection Overview A firewall can reduce, but not eliminate, threats introduced to your system by networking hosts. Firewall technology can prevent attacks that target network resources by limiting access to your system. The firewall is the computer s first line of defense against a network-based attack. In this chapter This chapter contains the following topics: Topic Page About the Firewall 18 Precedence of Firewall Rules 19 Configuring an IP or ICMP Firewall Rule 20 Configuring a TCP or UDP Firewall Rule 22 Deleting a Firewall Rule 24 User Guide for Proventia Server IPS for Linux, Version

18 Chapter 2: Configuring Firewall Protection About the Firewall Firewall rules work to ensure that only authorized traffic can access the server. Firewall rules can prevent users from accessing certain ports and can control user access to sensitive areas of your network. Firewall rules in Proventia Server IPS agents are based on protocol or port. Types of firewall rules The following table lists the basic types of firewall rules: Type Description IP ICMP TCP UDP An IP rule specifies whether an agent should accept or drop traffic to or from a particular IP address or range of addresses. An Internet Control Message Protocol (ICMP) rule specifies whether an agent should accept or drop IP traffic that contains an ICMP message. You can associate the rule with an IP address or a range of IP addresses. A Transmission Control Protocol (TCP) rule specifies whether an agent should accept communication or drop and reset a connection through a TCP port or range of ports. You can associate the port with an IP address or a range of IP addresses. A User Datagram Protocol (UDP) rule specifies whether an agent should accept or drop traffic through a particular UDP port or range of ports. You can associate the port with an IP address or a range of IP addresses. Table 5: Types of firewall rules Recommended number of rules As a general guideline, you should configure no more than 500 firewall rules. 18 IBM Internet Security Systems

19 Precedence of Firewall Rules Precedence of Firewall Rules The agent applies firewall rules in a specific order when it processes network traffic. Consider how the agent applies firewall rules to design the most effective firewall policy. Firewall rule processing When the agent processes firewall rules, it does the following: applies a specific firewall rule precedence stops processing further rules when a rule matches the packet Firewall rule precedence The agent applies firewall rules in the following order: IP and ICMP rules are always applied before TCP and UDP rules. IP and ICMP rules are applied in the order they are displayed in the IP/ICMP section of the policy. TCP and UDP rules are applied in the order they are displayed in the TCP/UDP section of the policy. Processing until a rule applies The agent processes firewall rules against a packet until one of the following statements is true: A firewall rule matches the packet. There are no more firewall rules to process. Example You define the following firewall rules: Rule 1: An IP rule that accepts packets from a source IP Range from to Rule 2: A TCP rule that drops packets from IP In this example, the TCP packets from will not be dropped because rule 1 accepts them and firewall processing on these packets stops after the agent executes the accept action for the first rule. Built-in priority rules There are certain built-in priority rules that always take precedence over user-defined firewall rules. These rules include rules that allow internal agent communications, agent communication with SiteProtector components, and agent communication with gateways. Handling conflicts between firewall rules Because the agent applies firewall rules in the order they appear in the policy, there is never a conflict in the application of firewall rules; however, this order does determine the effectiveness of your firewall policy. Procedure To change the precedence of a firewall rule: 1. Open the Firewall policy that you want to customize. 2. Select the firewall rule you want to change the precedence for. 3. Click the or the button to move the rule higher or lower in precedence. User Guide for Proventia Server IPS for Linux, Version

20 Chapter 2: Configuring Firewall Protection Configuring an IP or ICMP Firewall Rule The Proventia Server IPS agent can filter both inbound and outbound traffic. To filter packets that use the IP or ICMP protocol, create firewall rules based on source and destination IP addresses. Important: The SiteProtector Refresh Agent feature uses an ICMP message to contact the agent. If you create an ICMP firewall rule that includes your SiteProtector address as a source to block, the Refresh Agent feature will not work. About accept rules When you create an accept firewall rule, note the following: The agent does not generate alerts for accept rules, because this creates too many alerts in SiteProtector. The firewall is only the first line of defense. Traffic may pass through the firewall because of an accept rule, but another component may detect malicious content that triggers an alert. Procedure To configure an IP or ICMP firewall rule: 1. Open the Firewall policy you want to customize. 2. In the IP/ICMP section, do one of the following: click Add to add a new rule click Edit to edit an existing rule 3. Complete the following information, as necessary: Option Enable Name Event Severity Action Protocol Description Enables or disables this rule. Note: You must enable the rule before you can set any of the other options. Descriptive name for this rule. Severity of the event that triggers this rule. Action the agent should take against the packet that triggers this rule. Protocol this rule applies to. 4. In the IP Address section, select the Source tab. 5. In the Address section, specify the IP address(es) this rule applies to, as necessary: To specify... any IP address a single address Then... Select Any. Select Single Address, and then type the IP address. 20 IBM Internet Security Systems

21 Configuring an IP or ICMP Firewall Rule To specify... a range of addresses a Network Address/ #Network Bits (CIDR) Then Select Address Range. 2. Type the IP address that represents the beginning of the range, press TAB, and then type the IP address that represents the end of the range. 1. Select Network Address/#Network Bits (CIDR). 2. In the Address/Mask box, type the IP address, press TAB, and then type the number of network bits. 6. Select the Target tab, and then repeat Step 5 to complete the tab. 7. Select the responses you want the agent to take if it detects this type of event. Reference: See Configuring Agent-Level Responses on page Click OK. User Guide for Proventia Server IPS for Linux, Version

22 Chapter 2: Configuring Firewall Protection Configuring a TCP or UDP Firewall Rule The Proventia Server IPS agent can filter both inbound and outbound traffic. To filter packets that use the TCP or UDP protocol, create firewall rules based on source and destination ports and IP addresses. About accept rules When you create an accept firewall rule, note the following: The agent does not generate alerts for accept rules, because this creates too many alerts in SiteProtector. The firewall is only the first line of defense. Traffic may pass through the firewall because of an accept rule, but another component may detect malicious content that triggers an alert. Procedure To configure a TCP or UDP firewall rule: 1. Open the Firewall policy you want to customize. 2. In the TCP/UDP section, do one of the following: click Add to add a new rule click Edit to edit an existing rule 3. Complete the following information: Option Enable Name Event Severity Action Protocol Description Enables or disables this rule. Note: You must enable the rule before you can set any of the other options. Descriptive name for this rule. Severity of the event that triggers this rule. Action the agent should take against the packet that triggers this rule. Note: For TCP rules, the Drop action includes a TCP reset. Protocol this rule applies to. 4. In the IP Address and Port section, select the Source tab. 5. In the Address section, specify the IP address(es) this rule applies to, as necessary: To specify... any IP address a single address a range of addresses Then... Select Any. Select Single Address, and then type the IP address. 1. Select Address Range. 2. Type the IP address that represents the beginning of the range, press TAB, and then type the IP address that represents the end of the range. 22 IBM Internet Security Systems

23 Configuring a TCP or UDP Firewall Rule To specify... a Network Address/ #Network Bits (CIDR) Then Select Network Address/#Network Bits (CIDR). 2. In the Address/Mask box, type the IP address, press TAB, and then type the number of network bits. 6. In the Port section, specify the port(s) this rule applies to, as necessary: To specify... any port a single port a range of ports Then... Select Any. Select Single Port, and then type the port number. 1. Select Port Range. 2. In the range box, type the port that represents the beginning of the range, press TAB, and then type the port that represents the end of the range. 7. Select the Target tab, and then repeat Step 6 and Step 7 to complete the tab. 8. Select the responses you want the agent to take if it detects this type of event. Reference: See Configuring Agent-Level Responses on page Click OK. User Guide for Proventia Server IPS for Linux, Version

24 Chapter 2: Configuring Firewall Protection Deleting a Firewall Rule If you want the agent to temporarily stop processing a firewall rule, you can disable the rule; however, if you no longer need a firewall rule, you can delete the rule. For example, when a system that you designed specific rules for no longer resides on your network, simply delete the rule to keep your firewall policy manageable. Procedure To delete a firewall rule: 1. Open the Firewall policy you want to delete the firewall rule from. 2. Select the firewall rule you want to delete. Note: To select consecutive rules, click the first rule, press and hold down SHIFT, and then click the last rule. To select nonconsecutive rules, press and hold down CTRL, and then click each rule. 3. Click Remove. 24 IBM Internet Security Systems

25 Chapter 3 Configuring Network Protection Overview Network protection-based signatures monitor network traffic for content that can indicate an attack or other suspicious activity. In this chapter This chapter contains the following topics: Topic Page Global Action Setting 26 Customizing a Pre-Defined Security Event Signature 27 User Guide for Proventia Server IPS for Linux, Version

26 Chapter 3: Configuring Network Protection Global Action Setting If you want all the agents that use the same Security Events policy to take the same action against any malicious traffic they detect, you can set a global action. This option configures all enabled signatures at the same time, so you do not have to configure each enabled signature separately. Procedure To set a global action for all enabled security event signatures: 1. Open the Security Events policy that you want to customize. 2. In the When an intrusion is detected box, select one of the following: Option Block and Alert Alert only Description Blocks any malicious traffic and sends an alert to the Console. Sends an alert to the Console when malicious traffic is detected, but allows the traffic to pass. 26 IBM Internet Security Systems

27 Customizing a Pre-Defined Security Event Signature Customizing a Pre-Defined Security Event Signature The Proventia Server IPS agent comes with pre-defined signatures that analyze network traffic. If the agent detects malicious content, it can block the traffic to protect your system. You can customize certain attributes of these pre-defined signatures to better meet your security needs. Procedure To customize a security event signature: 1. Open the Security Events policy that you want to customize. 2. Click the PLUS SIGN (+) to expand the group that contains the signature you want to customize. Tip: Click the column header to sort the list by that attribute. 3. Select the signature that you want to customize. Note: You can select and edit several signatures at the same time. To select consecutive signatures, click the first signature, press and hold down SHIFT, and then click the last signature. To select nonconsecutive signatures, press and hold down CTRL, and then click each signature. 4. Click Edit. 5. Change the following options as necessary: Option Enable Severity Responses Description Enables or disables this signature. Note: You must enable the signature before you can set any of the other options. Severity level assigned to this signature. Note: The default severity is assigned by IBM ISS; you can change the severity if this type of attack posses a different threat to your system. Indicates the responses the agent should take when this event occurs. Each signature can have any combination of responses or no responses at all. Reference: See Configuring Agent Level Responses on page Click OK. Properties The following table describes the elements of the pre-defined signatures available on the Security Events window: Option Enable Name Attack/Audit Severity Description Enables or disables this signature. Descriptive name for this signature. Type of event detected by this signature. Severity level assigned to this signature. Note: The severity default is assigned by the X-Force. Table 6: Pre-defined security event signature properties User Guide for Proventia Server IPS for Linux, Version

28 Chapter 3: Configuring Network Protection Option Protocol XPU Check Date Block Overridden Severity Overridden Responses Description Indicates the protocol used by this attack. Indicates the XPU release that contained this signature. Indicates the date X-Force created this signature. Indicates whether the default X-Force block response for this signature has been overridden. Indicates whether the default X-Force severity setting for this signature has been overridden. Indicates the responses the agent should take when this event occurs. Table 6: Pre-defined security event signature properties 28 IBM Internet Security Systems

29 Chapter 4 Monitoring System Integrity and Policy Compliance Overview The Proventia Server IPS agent can help you ensure system integrity and security policy compliance by monitoring system log files for suspicious activity. The agent can monitor the following types of log files: text logs WTMP logs Syslog logs In this chapter This chapter contains the following topics: Topic Page Customizing a Pre-Defined Log Monitoring Signature 30 Using Regular Expressions in User-Defined Signatures 31 Monitoring for a Custom WTMP Log Event 32 Monitoring for a Custom Syslog 34 User Guide for Proventia Server IPS for Linux, Version

30 Chapter 4: Monitoring System Integrity and Policy Compliance Customizing a Pre-Defined Log Monitoring Signature The Proventia Server IPS agent comes with several pre-defined log monitoring signatures to monitor system log files for suspicious activity that may indicate a threat to system integrity or a violation of your security policy. You can customize a pre-defined signature to meet your specific needs by editing the severity setting. Procedure To customize a pre-defined log monitoring signature: 1. Open the OS Events policy that contains the signature you want to customize. 2. Click the PLUS SIGN (+) to expand the group that contains the signature you want to customize. Tip: Click the column header to sort the list by that attribute. 3. Select the signature that you want to customize. Note: You can select and edit several signatures at the same time. To select consecutive signatures, click the first signature, press and hold down SHIFT, and then click the last signature. To select nonconsecutive signatures, press and hold down CTRL, and then click each signature. 4. Click Edit. 5. Change the following options as necessary: Option Enable Severity Responses Description Enables or disables this signature. Note: You must enable the signature before you can set any of the other options. Severity level assigned to this signature. Indicates the responses the agent should take when this event occurs. Each signature can have any combination of responses or no responses at all. Reference: See Configuring Agent Level Responses on page Click OK. Properties The following table describes elements of pre-defined log monitoring signatures: Option Enable Name Category Severity Responses Description Enables or disables this signature. Descriptive name for this signature. Common grouping of signature for ease of use. Severity of the event that triggers this signature. Note: The default severity is assigned by IBM ISS. Indicates the response the agent should take when this event occurs. Table 7: Pre-defined operating system event signature properties 30 IBM Internet Security Systems

31 Using Regular Expressions in User-Defined Signatures Using Regular Expressions in User-Defined Signatures When you create a user-defined signature, you frequently use regular expressions to define the signature. You can use regular expressions to do the following: specify the information you want the agent to monitor for configure the information the agent retrieves about the event Because user-defined signatures frequently use regular expressions, you should be familiar with how to use regular expressions. Regular expression libraries used by Proventia Server IPS agents The Proventia Server IPS agent uses the Henry Spencer Regular Expression Library. This is the library on which the Perl scripting language bases its syntax. If you are familiar with regular expressions in Perl, then you can use that knowledge when defining regular expressions for a Proventia Server IPS agent. Where to use regular expressions You can use regular expressions in the following places: wherever you specify the event information the agent should include in responses wherever you define the log content to monitor for User Guide for Proventia Server IPS for Linux, Version

32 Chapter 4: Monitoring System Integrity and Policy Compliance Monitoring for a Custom WTMP Log Event The wtmp file is a log file that keeps track of all logins to and log outs from the system. In Proventia Server IPS you can create your own signatures to monitor the WTMP log for events that the pre-defined WTMP log signatures do not monitor for. About the wtmp log file The wtmp log file resides as a binary file in the file system. Many processes on the operating system update the wtmp log file, which records user login activities and some system process activities. Activities logged in the wtmp log Activities logged in the wtmp log include the following: user/root login using telnet user/root login using rlogin user/root login using an X-window console user/root logout processes spawned by init system boots Prerequisites Before you can monitor wtmp log files, you must know the following information: how to use regular expressions Reference: See Using Regular Expressions in User-Defined Signatures on page 31. how to use exceptions in regular expressions if the information that you want the agent to detect varies the exact information in the logs you want to monitor for the exact information in the logs you want to include in any responses Procedure To monitor for a custom wtmp log event: 1. Open the OS Events policy that you want to add the signature to. 2. Select the WTMP Events tab. 3. Click Add. 4. Select Enable. 5. Provide the following information: Element Severity Name Description Description Severity of the event that triggers this signature. Descriptive name for this signature. Describes the purpose of this signature. 32 IBM Internet Security Systems

33 Monitoring for a Custom WTMP Log Event Element Info User Name Entry Type Init ID Responses Description The name/value pairs that specify the information the agent should include in responses when it detects an event that matches this signature. The supported values are as follows: User name (User) Process ID (ProcID) Time Attack origin (AttackOrg) Destination address (DestAddr) This box configures this signature to do the following: trigger an event when a specific user is involved, if you type the name of the user trigger an event for root logins, if you type root trigger an event for any user, if you leave the box blank The type of system activity that is written to the log file. Reference: See the Help for detailed information about entry types. An optional string of text or regular expression that must be present in the device name field of the wtmp record. For example, pts/2 or tty. Indicates the responses you want the agent to take when it detects this event. Reference: See Configuring Agent Level Responses on page Click OK. User Guide for Proventia Server IPS for Linux, Version

34 Chapter 4: Monitoring System Integrity and Policy Compliance Monitoring for a Custom Syslog You can create a user-defined signature if you need to monitor for an event that the predefined UNIX syslog signatures do not detect. UNIX syslog signatures monitor the local syslog and any syslogs that are forwarded to that system. This topic describes how to create a user-defined syslog signature and how to define the logs the agent should monitor. Prerequisites Before you can monitor the local syslog or any syslogs forwarded to the system, you must know the following: how to use regular expressions Reference: See Using Regular Expressions in User-Defined Signatures on page 31. how to use exceptions in regular expressions if the information that you want the agent to detect varies the exact information in the logs you want to monitor for where the log files that you want to monitor reside Specifying newest or all files When you monitor syslog files, you can specify whether the agent should monitor all the files that match the file name pattern, or if the agent should monitor only the most recently changed file. The agent determines the newest file by comparing the last modification time (not the creation time of the file). File rotation When a log file grows to a certain size, the underlying application may rotate the log to a new log file. If you use a wildcard in the log file name when you specify the log file to monitor and you set the Monitor Newest Only flag, the agent can monitor the log file regardless of the file name. Example: If an application writes to /tmp/mylog1.log, then to /tmp/mylog2.log, and so on, then specifying a log file name such as /tmp/mylog*.log and selecting Monitor Newest Only will monitor the most recent log file. Using wildcards to define syslog file names When you monitor syslog files, you can use wildcards to specify the log file name, as follows: Wildcard Specifies * 0 or more characters? any one character [...] a range of characters Table 8: Wildcards for specifying syslog files File switching Sometimes, to retain log file information, an application renames a log file that becomes large. Subsequent log messages go to a file with the original file name, even though this is really a new file. For this situation, the agent keeps a mark for each log file it monitors. Each time a mark is changed, the agent treats it as a new file, reopens it, and monitors it from the beginning of the file. On a UNIX platform, a file mark is the inode number of the file. 34 IBM Internet Security Systems

35 Monitoring for a Custom Syslog Procedure To monitor for a custom syslog event: 1. Open the OS Events policy that you want to add the signature to. 2. Select the Syslog Events tab. 3. Click Add. 4. Select Enable. 5. Provide the following information: Element Severity Name Description Description Severity of the event that triggers this signature. Descriptive name for this signature. Describes the purpose of this signature. 6. Do you want to generate a response only when certain text appears in a log? Note: The agent will generate a response each time something is written to the log unless you specify that certain text must be present before the agent should generate a response. If yes, go to Step 7. If no, go to Step In the Regular Expression box, type the regular expression the agent should monitor the log files for. 8. Do you want to include specific information in any responses the agent takes when it detects an event that matches this signature? If yes, go to Step 9. If no, go to Step Click Info. 10. Specify a name/value pair for this signature, and then click Add. Reference: See Supported data identifiers on page Repeat Step 10 until you have entered all name/value pairs for this signature, and then click OK. 12. In the Log file information section, click Add, and then provide the following information: Option Log Name Log Paths Monitor Newest Only Description Type the name of the log file this signature applies to. Reference: For information about using wildcards, see Using wildcards to define syslog file names on page 34. Type the full path to the log file this signature applies to. Select to monitor only the newest log file; clear the check box if you want to monitor all the log files that match the Log Name. 13. Click OK. User Guide for Proventia Server IPS for Linux, Version

36 Chapter 4: Monitoring System Integrity and Policy Compliance 14. Select the responses you want the agent to take if it detects this type of event. Reference: See Configuring Agent-Level Responses on page Click OK. Supported data identifiers The following table lists the data identifiers you can use to extract a value from a syslog event: @HostIP Description Used to extract one information field in a log entry where N is the relative number of the field in the entry. Note: When counting the number of the field, start from zero. Used to extract the name of the log file that recorded an event you are monitoring for. Used to extract the IP address of the system that is triggering the event. Example A typical syslog message might have the following format: Mar 15 10:25:30 everest sendmail[28244]: authdes_refresh: keyserv(1m)... To extract the host name from this message, you would use the following values: Name = Host Value To extract the log name from this message, you would use the following values: Name = Host Value To extract the IP address, you would use the following values: Name = Host IP Value {!} Used to extract a substring from a string. {!} is a wildcard that pulls a substring located between two defined text entries in the entry. To capture the user name for a user with a failed login, use the following values: Name = User Value = User {!} failed to Table 9: Data identifiers used in name/value paris 36 IBM Internet Security Systems

37 Chapter 5 Configuring Buffer Overflow Exploit Prevention Overview The Proventia Server IPS agent can prevent intruders from using a buffer overflow to exploit your server. Buffer overflow exploit prevention (BOEP) comes configured to monitor all active processes on your system and to report any buffer overflow exploits it detects. No further configuration is required unless you want to do one of the following: exclude applications from being monitored specify a different action for a certain application specify a different response for certain applications Important: BOEP can only provide protection on supported Red Hat and SUSE platforms; BOEP is not currently supported on 64-bit systems. BOEP and operating system buffer overflow protection When you install BOEP, the agent disables ExecShield, the Red Hat buffer overflow protection feature that comes with the operating system. BOEP and the SecurityFusion Module BOEP events are not included in any analysis performed by the SecurityFusion Module. In this chapter This chapter contains the following topics: Topic Page What is Buffer Overflow Exploit Prevention? 38 Configuring Global Buffer Overflow Exploit Prevention 40 Changing the Action for a Monitored Application 41 Excluding an Application from Protection 42 User Guide for Proventia Server IPS for Linux, Version