Keeping your HP ArcSight connectors healthy

Transcription

1 Keeping your HP ArcSight connectors healthy Tracy Barella Chief Services Strategist

2 HP ArcSight Connector Health Agenda What is a Health? Health steps by ArcSight component Connectors Connector Appliances Q & A 2

3 Health overview

4 What is a health check? Purpose The purpose of performing a health check is to identify and remove performance bottlenecks to enable top performance of the HP ArcSight implementation. Minor issues can result in major performance degradations over time impacting system availability and user satisfaction. Performing regular health checks will identify issues allowing them to be remediated quickly and ensure continued top performance of the HP ArcSight implementation. In a nutshell A Health consists of common administrative tasks to verify the ArcSight solution is configured and performing optimally. 4

5 Health steps by ArcSight component Note: It s impossible to cover every scenario in this presentation, so only the common checks will be discussed.

6 Health check steps by ArcSight component Connectors Tip: each ArcSight Component by the order of the Event Flow 1 Connectors Connector appliances ESM Manager ESM Database and storage Loggers Up/Down (Connector or Container) Version Connector Event Rate (by EPS) Cache Logs Configuration 6 Version CPU and Memory Network Settings Configuration Backup It s just simple plumbing!!! Event Throughput Dashboard Current Event Sources Dashboard Hardware and Operating System CPU and Memory Utilization ESM Manager JVM (memory) Utilization Data Monitor Utilization Active List/Session List Utilization Rules Engine Event Persistence (insertion) Performance Error Scheduled Task server.properties Agent and Console Threads DB and Oracle RDA Database Performance Statistics Dashboard Partition (Oracle) Trend Jobs Hardware and Operating System CPU and Memory Utilization Oracle version and patch level check Oracle alert log check Oracle memory parameters check ESM Database Storage CPU, Memory, and EPS In/Out Search Performance Custom Report Performance Receivers and Forwarders Storage Group Index Configuration Configured Alerts Scheduled Task Event Archive and Configuration Backup Logger System Health and Audit Event Forwarding Network Configuration Online Event Storage (Only Software-based or SAN Logger)

7 Connectors Connector (or Container) Up/Down Connector Version Are there any Connectors running a version older than ~1 year? A minimum version of is required to leverage the ESM v5.2 schema. 7

8 Connectors (cont.) Connector Cache All Connectors should have 0 events in the cache If most Connectors are continuously caching = Possible ESM level Event Insertion problem If one or two Connectors are continuously caching = Possible Connector level problem or network issue If a Connector caches for a moment and then clears the cache (batched events) = This is normal Connector Event Rate (by EPS) Are there any Connectors receiving a high event rate? See below for definition of high EPS on common Connector types: Syslog Connector or Point Connector : >= 1,500 EPS Windows Unified Connector: > 500 to 1,000 EPS DB-based Connector or SourceFire estreamer Connector: >= 200 EPS Is the high EPS Connector stable? If not, we should recommend another Connector to spread the load? 8

9 Connectors (cont.) Connector Logs../current/logs/agent.out.wrapper.log Java Heap Memory Utilization Memory utilization Frequency of Full GCs Memory in Red Zone alerts Unexpected Connector restarts Connectivity errors End Devices ArcSight Destinations../current/logs/agent.log Parsing errors DOSProtector Chronic WARN and ERROR messages 9

10 Connectors Connector Logs (cont.) Use Connector LogFu to graph the event flow and memory utilization../current/bin/arcsight agent logfu a 10

11 Connectors (cont.) Connector Configuration Destination Settings Are there more than 2 Destinations on each Connector? Too many Destinations can negatively impact performance of a Connector. Common problems found: Networks and CustomerURI are not applied on every Connector Fields-based Aggregation is not properly applied (by Connector Type) No tuning (Filter Out) applied on high EPS Connectors Settings are not the same on every Destination (ESM, Logger, etc.) 11

12 Connectors (cont.) Connector Configuration (cont.) Only check the following on problematic Connectors discovered in previous checks../current/user/agent/agent.properties Optimal settings are different for each Connector type High EPS Connectors (>1200 EPS) such as Syslog, WUC, Point, and Blue Coat can be tweaked quite a bit here../current/user/agent/agent.wrapper.conf Only increase the Java Heap size if memory issues were found in agent.out.wrapper.log Default Java Heap is 256MB Maximum configurable Java Heap is 1024MB (1 GB) Reminder: If you have 50+ Connectors in your environment, try to stay focused on problematic Connectors! 12

13 Health check steps by ArcSight component Connector Appliances Tip: each ArcSight Component by the order of the Event Flow 1 Connectors Connector appliances ESM Manager ESM Database and storage Loggers Up/Down (Connector or Container) Version Connector Event Rate (by EPS) Cache Logs Configuration 13 Version CPU and Memory Network Settings Configuration Backup Event Throughput Dashboard Current Event Sources Dashboard Hardware and Operating System CPU and Memory Utilization ESM Manager JVM (memory) Utilization Data Monitor Utilization Active List/Session List Utilization Rules Engine Event Persistence (insertion) Performance Error Scheduled Task server.properties Agent and Console Threads DB and Oracle RDA Database Performance Statistics Dashboard Partition (Oracle) Trend Jobs Hardware and Operating System CPU and Memory Utilization Oracle version and patch level check Oracle alert log check Oracle memory parameters check ESM Database Storage CPU, Memory, and EPS In/Out Search Performance Custom Report Performance Receivers and Forwarders Storage Group Index Configuration Configured Alerts Scheduled Task Event Archive and Configuration Backup Logger System Health and Audit Event Forwarding Network Configuration Online Event Storage (Only Software-based or SAN Logger)

14 Connector appliances Connector appliance version check Is the version outdated? Are there any known issues with the current version? Connector appliance CPU and memory check Review the following for excessive utilization: CPU utilization is continuously above 70-80% in Logger Dashboard EPS In is continuously above 5,000 EPS (a single C5400 is designed for 5,000 max EPS) the Connector Appliance s Monitor Dashboards for unusual peaks or drops the System Process Status section of the Connector Appliance If possible, SSH to the Connector Appliance and run commands such as top, df, ifconfig, etc. to perform a deeper dive at the OS level Connector appliance network settings check Common problems to check: Incorrect duplex settings on the network interface DNS or NTP not configured properly Connector appliance configuration backup check The daily Configuration Backup job should be scheduled on all Connector Appliances. 14

15 Additional resources

16 My favorite resources for keeping ArcSight healthy! 1. Any HP Protect presentation on ArcSight best practices or troubleshooting: 2. KB Articles on the HP Support Site 3. Solutions listed in previous Support Tickets 4. HP ArcSight University 5. HP ArcSight product documentation 16

17 Please give me your feedback Session TB3257 Speaker Tracy Barella Please fill out a survey. Hand it to the door monitor on your way out. Thank you for providing your feedback, which helps us enhance content for future events. 17

18 Thank you

19