Managing Latency in IPS Networks

Transcription

1 Revision C McAfee Network Security Platform (Managing Latency in IPS Networks) Managing Latency in IPS Networks McAfee Network Security Platform provides you with a set of pre-defined recommended settings using which, you can begin monitoring traffic immediately after adding a Sensor to your network. However, to realize the complete effectiveness of the Sensor, optimal policy tuning on the Sensor is necessary. McAfee provides a considerable amount of configuration flexibility on its Sensors to enable you to tune policies - for proactive blocking and reduced false positives. Once the Sensor starts monitoring your network traffic, the next concern would be the Sensor's performance. Continuous stateful inspection of network packets is bound to have an impact on performance. For an inline device, the key performance metrics are attack coverage, throughput, load balancing, latency, and scalability. To balance the trade-off between security effectiveness and performance, McAfee Network Security Platform provides several configurable options to balance the traffic load with minimal latency. Latency through the Sensor can vary significantly based on packet size, complexity of protocol or presence of attack traffic. Average latency through the Sensor is typically observed under 1 milli second in real-world networks. 1

2 This document helps you identify the options that can be considered while deploying the Sensor inline in your network, to maximize attack coverage with optimal performance, and minimal latency. McAfee recommends a 5-step approach to manage latency, as depicted in the figure below. 1 Select the right Sensor model Each Sensor model has a throughput limit. For example, the M-2850 Sensor is rated at 600 Mbps performance. If the Sensor begins to see traffic more than 600 Mbps, the Sensor may not keep up with the traffic and introduce latency. It is important that you stay within the operating parameters of the device you deploy. If you are running traffic at gigabit speeds, then use an M-3050/M-4050/M-6050/M-8000/NS9x00/NS7x00 Sensor, which have a much higher throughput. The following tables provides the throughput, number of connections, average latency for different Sensor models: NS-series: Maximum Type NS9300 NS9200 NS9100 NS7300 NS7200 NS7100 Aggregate Performance Max Throughput with test equipment sending UDP packet size of 1512 Bytes Concurrent Connections up to 70 up to 35 up to 30 up to 15 up to 10 up to 5 32,000,000 16,000,000 13,000,000 10,000,000 5,000,000 3,000,000 2

3 Maximum Type NS9300 NS9200 NS9100 NS7300 NS7200 NS7100 Connections established per second Default number of supported UDP Flows Supported UDP Flows maximum Supported UDP Flows minimum Latency (Average UDP per packet Latency) Virtual IPS: 1,000, , , , , , , , , , , ,000 12,000,000 6,000,000 6,000,000 3,000,000 3,000,000 3,000,000 1,000 1,000 1,000 1,000 1,000 1,000 <100 µs <100 µs <100 µs <100 µs <100 µs <100 µs Maximum Type IPS-VM600 IPS-VM100 Aggregate Performance 600 Mbps 100 Mbps Maximum throughput with test equipment sending UDP packet size of 1518 bytes Up to 1 Concurrent connections 600, ,000 Connections established per second 20,000 6,000 Default number of supported UDP Flows 25,000 10,000 Supported UDP Flows 254,208 39,168 Latency (Average UDP per packet Latency) M-series: Up to 150 Mbps Maximum Type Aggregate Performance Maximum throughput with test equipment sending UDP packet size of 1518 bytes Concurrent connections Connections established per sec. Default number of supported UDP Flows M-8000 M-6050 M-4050 M-3050 M-2950 M-2850 M-1450 M Mbps Up to 20 Up to 10 Up to 4 Up to 2.5 Up to 1.5 Up to Mbps Up to 300 Mbps 100 Mbps Up to 150 Mbps 5,000,000 2,500,000 2,000,000 1,000, , ,000 80,000 40, ,000 60,000 36,000 18,000 15,000 10,000 4,000 2, , , ,000 50,000 50,000 25,000 10,000 5,000 3

4 Maximum Type Supported UDP Flows Latency (Average UDP per packet Latency) M-8000 M-6050 M-4050 M-3050 M-2950 M-2850 M-1450 M ,000,000 1,500, , , , ,500 60,000 30,000 2 Select the right IPS policy When you first add a Sensor inline in your network, start your policy configuration by applying either the Default Inline IPS or Default IDS or Default IPS Attack Settings policy under Policy <Admin Domain Name> Intrusion Prevention IPS Policies. This enables you to begin monitoring your network immediately. Subsequently, pick the policy that best matches your needs, and clone the policy. Then remove any irrelevant attacks, add any additional attacks, and configure appropriate response actions to respond to detected attacks. Some amount of false positives and irrelevant alerts can occur for the first 3 to 4 weeks. Tune your policies to delete attacks that do not apply to your environment to reduce the amount of insignificant alerts generated by your Sensors. For example, if you use only Apache Web servers, you may wish to disable IIS-related attacks. While deploying the Sensors inline on high throughput networks, it is not recommended to use the All-Inclusive without audit and All-Inclusive with audit policies. While these policies enable you to fully analyze your network traffic, these include many audit signatures, which alerts on benign traffic. A very high alert volume causes latency in your network. 3 Tune your policies to suit your network Consider the following options to better tune your IPS policies in McAfee Network Security Platform. 3.1 Custom Signatures McAfee Network Security Platform allows you to create custom attacks in McAfee's proprietary format as well as using Snort rules language. However, the ability to create a custom attack can sometimes turn tricky. For instance, a mistake in implementation of a signature on a high traffic network could cause a large number of alerts to be generated. Create custom signatures under Policy <Admin Domain Name> Intrusion Prevention Advanced Custom Attacks using the following best practices: Verify that the custom attack you intend to create does not duplicate any attack provided by McAfee Network Security Platform. In case of a duplicate, you have the flexibility to use both or just the custom attack instead of the McAfee supplied attack. If you choose to use both, then note that the Sensor raises two alerts for the same attack traffic. If the pattern your signature searches for occurs too often in the network traffic being analyzed, checking each match to determine whether it fulfills a signature condition reduces Sensor performance. Avoid using many L3/L4 fixed-field tests in the custom signature, since it is examined in IP/TCP/UDP header on each packet. Avoid using very common strings that every flow/packet contains such as ".com" in HTTP traffic. 4

5 If you have configured multiple interfaces on the Sensor, wherein one interface is using All Inclusive with Audit, while other interface is using Default Inline IPS, the All Inclusive with Audit gets preference and is applied to the Sensor. This causes performance/latency issues. It is recommended that you apply Default Inline IPS policy on all interfaces to improve performance and avoid latency. Avoid single byte or two-byte string match tokens in the custom signatures. Remove the attacks that have wrong detection logic and may never be triggered. For example, a signature to check dest-ip=" " AND dest-ip=" ". 3.2 HTTP Response Scanning Based on the needs of your organization, you can enable HTTP response inspection for inbound traffic, outbound traffic, or both directions. To enable HTTP response scanning, go to Policy <Admin Domain Name> Intrusion Prevention Policy manager. In the Interfaces tab, double-click the interface for which you want to enable the response scanning. The <Device Name/Interface> panel opens on the right-side. In the Inspection Options section, click the edit icon to edit the already applied policy. The Inspection Options page opens. In the Traffic Inspection tab, under HTTP, select the direction in which you want to enable from the HTTP Response Traffic Scanning drop-down. However, to minimize the potential performance impact on the Sensor: McAfee recommends that you enable HTTP response processing on the outbound traffic. Consider enabling HTTP response on the inbound traffic only if you suspect that your internal Web Server is/ could be compromised. 3.3 Non-standard Ports Network Security Platform detects threats for certain standard protocols irrespective of which ports they run on. However, if a non-standard port is used in your network for a standard protocol, configure the port number under Policy <Admin Domain Name> Intrusion Prevention Advacned Non Standard Ports to reduce latency caused by re-routing of necessary traffic due to non-standard port number. For example, HTTP by default uses port 80 or 8080; therefore, a Sensor reading a packet with port 80 or 8080 attempts to decode that traffic as HTTP traffic. However, if you are running an HTTP server on port 2560, then configure this as the non-standard port for HTTP, on the Manager. Avoid configuring the port number for one standard protocol as a non-standard port for another protocol. 3.4 Latency Monitoring Latency monitor is configured in either of the following modes to take action when high latency is observed: Issue latency-monitor enable action alert-only to generate an alert when a high latency is observed at the Sensor. Issue latency-monitor enable action layer2-forward to generate an alert and also forwards the traffic to layer 2. You can view these alerts in the Real-Time Threat Analyzer of the Manager. You can use the following CLI commands to enable, set sensitivity level, and check the status of latency monitor feature: latency-monitor enable action Enables latency monitoring in the Sensor and also specifies the action to be performed if high latency is observed in the Sensor. 5

6 The following are the actions that can be specified in this command: alert-only (generates an alert when a high latency is observed in the Sensor) put-in-layer2 (generates an alert and also forwards the traffic to layer 2). Alerts that are generated can be seen in the System faults page in the Manager. Syntax: latency-monitor enable action <alert-only put-in-layer2> This command should be executed with a parameter value, else the command is treated as invalid. Example: If layer2-forward is enabled, it is necessary to set the layer 2 mode to be on. Otherwise the layer2-forward action does not get executed. latency-monitor enable action alert-only latency-monitor sensitivity-level Configures the sensitivity level for latency management. Syntax: latency-monitor sensitivity-level high latency-monitor sensitivity-level medium latency-monitor sensitivity-level low latency-monitor restore-inline When a high latency is observed on the Sensor and the latency monitor is configured, the Sensor remains in layer 2 until a layer2 mode deassert is invoked or the Sensor reboots. This command allows the Sensor to come out of layer 2 mode without layer 2 deassert. The Sensor restores to inline from layer 2 if the following conditions are met: The latency monitor has put the Sensor in layer 2 mode. The Sensor is in good health. If the Sensor is in bad health, a deassert cannot be performed and the Sensor reboots. A substantial amount of time has lapsed, as configured using this command, when the Sensor went into layer 2 due to latency. The default time to trigger an automatic layer 2 deassert is 10 minutes. If the latency continues to exist after the Sensor is restored to inline mode, the Sensor behaves according to the current setting of the latency monitor. Syntax: latency-monitor restore-inline enable <10-60> latency-monitor restore-inline disable Parameter Description <10-60> The time in minutes to trigger the restore inline from layer 2. It is counted since the time the Sensor moved into layer 2 state due to high latency. The latency-monitor status command displays the status of the latency monitor feature, and the status of the restore-inline feature of the latency monitor. 6

7 latency-monitor Disables the latency monitoring feature or displays the status of latency monitoring feature. Syntax: latency-monitor <disable status> Default Value: Latency monitoring feature is disabled by default. If disabled, latency monitoring feature does not generate any alert nor forward the traffic to layer 2 when high latency is observed. If latency monitoring is enabled, the following information is displayed. latency monitoring status (enable or disable) configured action (alert-only or layer2-forward) 3.5 Packet Logging Large amount of packet logging will cause adverse impact on Sensor performance. On high throughput network, it is advisable to stick to default or disable packet logs for alerts that are not required. 3.6 Scanning Exceptions There could be certain traffic that you want the Sensor to allow or block without deeper inspection. Configure stateless access rules to bypass inspection for trusted high throughput applications like database backups. The Sensor allows or blocks packets just based on the L4 information in those packets, thereby saving time and resources. Configure scanning exceptions to bypass scanning of traffic from a configured VLAN, TCP, or UDP port. Once set, these rules take precedence over Firewall access rules. [Failover ports and M 8000 interconnect ports cannot be configured for scanning exceptions.] You can configure scanning exceptions under Devices <Admin Domain Name> Devices <Device Name> Setup Advanced Stateless Scanning Exception. 3.7 Access Rules for Fragmented Traffic Configure access rules for fragmented traffic to selectively specify rules for a host (or network) based on which the Sensor skips reassembly handling of the fragmented traffic. This helps in decreasing the latency of the fragmented traffic for the specified network or host. Use this feature only with a trusted host and only if you are receiving extremely high amount of fragmented traffic. For example, use access rules for fragmented traffic if your NFS server is sending huge amount of fragmented traffic through your Sensor. Using this feature, receiving traffic from an unknown host can mean evasion using IP fragments. All fragmented traffic are reassembled prior to processing if the traffic does not match any access rules configured for fragmented traffic. Also note the following: 7

8 You can use access rules for fragmented traffic only with TCP flow violation set to Permit out of-order. Syn cookie should not be used when access rules for fragmented traffic are applied. Firewall logging is not supported when access rules are configured for fragmented traffic. 3.8 Layer 7 Data Collection If you have layer 7 data collection enabled under Devices <Admin Domain Name> Devices <Device Name> Setup Advanced L7 Data Collection, then disable protocols or specific fields within a protocol. This optimizes the Sensor performance. 3.9 Heuristic Web Application Server Protection If you have configured Policy <Admin Domain Name> Intrusion Prevention Inspection Options Policies, then configure only the critical Website paths that you want to protect as it affects Sensor performance Advanced Traffic Inspection Configure inspection for advanced evasions only if necessary under Policy <Admin Domain Name> Intrusion Prevention Inspection Options Policies. On a deployment with a high percentage of good traffic and some traffic that uses evasions, the Sensor throughput could drop. 4 Monitor Sensor Performance Once you have configured/customized your policies, use the Manager to monitor the Sensor performance to detect early signs of performance/latency issues in your network. 4.1 Monitor via CLI Use the following CLI commands to monitor Sensor performance show sensor-load Syntax: show sensor-load Run the show sensor-load command to view the following statistics: the average load of traffic on the Sensor processing elements maximum load of traffic seen on each Sensor processing element show mem-usage Syntax show mem-usage Run the show mem-usage command to review the following counts: Avg. Used TCP and UDP Flows across all PEs Max. Used TCP and UDP Flows on a single PE Avg. Used Fragmented IP Flows across all PEs 8

9 Max. Used Fragmented IP Flows on a single PE Avg. Used ICMP Flows across all PEs Max. Used ICMP Flows on a single PE Avg. Used SSL Flows across all PEs Max. Used SSL Flows on a single PE Avg. Used Fragment Reassembly Buffers across all PEs Max. Used Fragment Reassembly Buffers on a single PE Avg. Used Packet Buffers across all PEs Max. Used Packet Buffers on a single PE Avg. Used Attack Marker Nodes across all PEs Max. Used Attack Marker Nodes on a single PE Avg. Used Shell Marker Nodes across all PEs Max. Used Shell Marker Nodes on a single PE Avg. Used L7 Dcap Alert Buffers across all PEs Max. Used L7 Dcap Alert Buffers on a single PE Max. Used L7 Dcap Alert Buffers on a single PE Avg. Used L7 Dcap flows across all PEs Max. Used L7 Dcap flows on a single PE show inlinepktdropstat <port> Syntax show inlinepktdropstat <port> Run the show inlinepktdropstat command to know how many packets are dropped at the Sensor port. Information displayed includes the count for each of the following categories: IP checksum errors TCP checksum errors UDP checksum errors ICMP checksum errors ACL-related packets dropped Out-Of-Context/Bad packets dropped Sensor cold-start-related packets dropped Off/HdrLen error packets dropped Dropped attack packets (that is, blocked packets) IP reassembly timeout packets dropped TCP Out-Of-Order timeout packets dropped 9

10 Dropped packets containing TCP protocol errors Dropped packets containing UDP protocol errors Dropped packets containing ICMP protocol errors Dropped packets containing IP protocol errors Packets dropped due to the Sensor being out of resources Dropped packets containing CRC errors Dropped IP-spoofed packets ICMPv6 checksum error drop count IPv6 reassembly timeout drop count ICMPv6 Protocol error drop count IPv6 Protocol error drop count Host Quarantine IPv4 packet drop count Host Quarantine IPv6 packet drop count Other Layer-2 error packets dropped IP sanity check packets dropped IPv6 sanity check packets dropped Total IP No Credit Packets dropped Total Rate Limit Packets dropped sensor-datapath-stat-analysis show Syntax sensor-datapath-stat-analysis show Run the sensor-datapath-stat-analysis show command to view a list of Sensor statistics that affects latency: Total packets received Total TCP packets Total UDP packets Total non TCP/UDP packets Total fragments Total duplicate fragments Total attack detected Total alert generated Total alerts dropped without response Total alerts dropped because of filter setting Total logs sent 10

11 Total packets matching L3/L4 UDS Policy Ruleset on Sensor **Analysis of the statistics** Attack dropped without response action Non TCP/UDP Traffic Attack dropped because of filter setting Traffic matching L3/L4 UDS Traffic detected with attack Count of fragments Fragmented traffic Percentage of logs to alerts sent TCP Traffic Snort signature support UDP Traffic 4.2 Monitor via Manager Use the Threat Analyzer and System Faults pages in the Manager to monitor Sensor performance in your network Threat Analyzer Dashboards In the Manager, set the thresholds and enable alarm for Sensor performance under Devices <Admin Domain Name> Devices <Device Name> Troubleshooting Performance Monitoring. Once the thresholds are configured, the core Sensor performance metrics are monitored using the Threat Analyzer. Metrics such as Utilization-Device TCP/UDP Flow, Utilization-Device Throughput, Status of Activities, and Operational Status Summary are displayed in the default NSP Health dashboard. You can also create custom dashboards and monitors to view various other Sensor statistics: Statistics Flows TCP and UDP flow data processed by a device. Checking your flow rates can help you determine if your device is processing traffic normally, while also providing you with statistics such as the maximum number of flows supported and number of active TCP and UDP flows. Statistics IP Spoofing Number of IP spoofing attacks detected by the Sensor. Statistics are displayed per direction. Statistics Malware Malware detected for a given device. Statistics Port Packet Drops Packet drop rate on an interface. Statistics Rate Limiting Estimated number of packets dropped/bytes dropped by the device. You can view rate limiting statistics for each device (per interface), listed under the Devices tab. Statistics Rx/TX Total number of packets received (Rx) and transmitted (Tx) for a given device. Statistics Device Packet Drops Packet drop rate on a device. The statistics is displayed on a per device basis. The statistics includes the count of number of packets dropped by a device due to the configured rate limiting and sanity check failures Operational Status Faults From the Manager Dashboard page, click any fault in the System Health monitor to view the faults in the System Faults page. Watch out for performance and latency related faults: 11

12 Fault Name Device in high latency mode Device latency monitoring configuration is conflicting with layer 2 monitoring configuration Device performance <Utilization - Device CPU, Utilization - Device TCP/UDP Flows, Utilization - Device Throughput, Utilization - Port Throughput> Action The device attempts to automatically recover from the high latency condition. Disable moving Sensor to layer 2 bypass mode on high latency or enable layer 2 pass through monitoring. Check the Sensor, and tune your policies to bring the affected metric below the configured threshold level Number of Alerts If you are receiving the Sensor: Attack Marker Resources Exhausted alerts in high volume, then monitor the percentages reported for attack marker nodes and the load on the Sensor via Sensor CLI interface. If the latency continues, tune your policies to bring down the latency experienced. Copyright 2018 McAfee, LLC McAfee and the McAfee logo are trademarks or registered trademarks of McAfee, LLC or its subsidiaries in the US and other countries. Other marks and brands may be claimed as the property of others. 12 0C00