Improving the Effectiveness of Log Analysis with HP ArcSight Logger 6 A SANS Product Review Written by Dave Shackleford April 2015 Sponsored by HP 2015 SANS Institute
Introduction Most organizations today collect logs and actively use them for monitoring, forensics, troubleshooting, and detecting and tracking suspicious behavior, according to the ninth SANS Log Management Survey, in which 97 percent of organizations reported they are currently collecting and leveraging logs for all of these reasons and more. 1 How well they use logs is another matter entirely. In the same survey, 50 percent of respondents for whom detection and tracking of suspicious behavior was a stated need confirmed that such detection and tracking is moderately difficult to accomplish, with another 30 percent stating that log collection and analysis is difficult for this purpose. Many organizations are also struggling with large amounts of log data from a vast variety of distributed sources and are spending significant amounts of time analyzing logs each week 22 percent of respondents spend more than one full day per week analyzing logs. It s clear that log collection and analysis is a critical aspect for most IT security teams. However, even with the advances in log management techniques seen in recent years, many teams are still struggling to get control of their logs and properly manage them, both effectively and efficiently. We recently reviewed HP ArcSight Logger 6, which includes significant updates over earlier releases. The new Logger s standout features include improved incident analysis and response flexibility, overhauled reporting and monitoring, and general enhancements for ease of use. Our evaluation focused on three areas that HP notably updated and enhanced in Logger 6: We can summarize our review process using this question: How can this tool help Logger 6 performed admirably for all the major use cases, and we found numerous capabilities that would help many organizations improve the effectiveness of their log management. 1 Ninth Log Management Survey Report, October 2014; www.sans.org/reading-room/whitepapers/analyst/ninth-log-management-survey-report-35497 1
Ease of Use We reviewed ArcSight Logger 6 in a test environment that HP installed and configured, simulating many events across 20 logging devices to represent a typical enterprise. The first use case that we explored flexibility, customization and ease of use directly relates to the user friendliness of the dashboards and interfaces available to analysts. Our Figure 1. ArcSight Logger 6 Main Dashboard Although Logger 6 includes a number of stock dashboards (packaged for various roles and job functions), we used a dashboard prebuilt by the ArcSight team to demonstrate what current product users report to be the most popular graphs and charts. The Logger upper left to lower right: Contains the aggregate number of failed login events across all users and platforms. Displays administrative SSH sessions to UNIX platforms; this information can assist in monitoring privileged activities. This panel shows patterns of network traffic throughout the environment, emphasizing services in active use. within the environment. 2
Ease of Use (CONTINUED) Beyond these examples, we noted the flexibility to quickly change between saved dashboards in a variety of different categories. Custom dashboards are usually where security analysts spend their time, looking at aggregate events and trends that allow for available to rapidly switch between saved dashboard views, making it simpler than ever to navigate to the desired dashboards. We quickly switched from this original custom dashboard to another one, labeled Intrusion and Configuration Events, that was configured for us. Much like the main dashboard, the Intrusion and Configuration Events dashboard shows popular and useful Figure 2. Intrusion and Configuration Events Dashboard While reviewing the malicious code activity, we noticed a number of events labeled denial of service (DoS) attack or hostile network discovery activity. 3
Ease of Use (CONTINUED) To get a sense for how simple it is to drill down on events, we simply clicked into the HSBQI BSFB PO UIF FOUSZ GPS *$.1 1BDLFU 'MPPE %PJOH TP QSPWJEFE NPSF HSBOVMBS SFTVMUT BOE BVUPNBUJDBMMZ SFEJSFDUFE VT UP UIF "OBMZ[F DBUFHPSZ BT TIPXO JO 'JHVSF Figure 3. Drilldown Malicious Events for ICMP Packet Flood 5IF TDSFFO JO 'JHVSF QSPWJEFE B XFBMUI of data related to the captured events, including the time of the events, what devices observed the events and which logging engine captured and recorded the events for analysis. We could also easily use this data to build a custom dashboard on the fly, using the top malicious IP addresses or another data type from within the events. To create quick dashboard charts and graphs, all we had to do was click the save button (in the toolbar on the query response page) and choose to save to an existing dashboard or create a new one, as shown Figure 4. Creating a Custom Dashboard on the Fly JO 'JHVSF 4"/4 "/"-:45 130(3". 4
Ease of Use (CONTINUED) The Logger interface also allowed us to easily view the overall status of the monitored systems and events. By selecting the Summary menu item at the top of the dashboard window, we were able to quickly review the number of different event types across devices and endpoint agents that forward events to log collector servers in the test network. Clicking any of the various categories yielded more data, and simple metrics Figure 5. Global Summary of Events Having immediate access to a central view of event count, types, systems and logging platforms (known as receivers in Logger jargon) is invaluable to security operations immediately determine whether a particular system is seeing a higher count of events than normal, which receivers are getting the most logs and events sent to them, and what types and categories of events are being seen most frequently. This visibility allows large, distributed teams to focus on particular types of events or one or more receivers that are seeing higher event counts; teams can then scrutinize those platforms to see the cause of the changes. 5
Ease of Use (CONTINUED) emphasis and details on receivers, events, utilization and processing stats from the ArcSight host and, finally, storage. Figure 6. Logger Monitoring Summary of the Environment This view presents a wide range of data, including CPU usage for the Logger platform over specified time periods, total event flow, receiver status and a list of storage repositories defined for use within the event management infrastructure. This data is valuable for security professionals who need to keep up with changes in performance and events over time, as well as operations teams that need to track how much space is in use for event storage. the direct navigation query field (shown at the top of the screen throughout the UI). This intelligent search query box autopopulates suggestions based on keywords or even just letter combinations and strings that a user types, making it exceedingly simple to locate various dashboard pages, analysis pages, specific data types starts with the term Data and the suggested search options that Logger 6 automatically creates. Figure 7. Dynamic Search Query Field 6
Ease of Use (CONTINUED) The Logger interface was incredibly simple to use. Within seconds, enormous amounts of data were readily visible and available, and finding specific events, dashboards, metrics and other important elements of the monitoring environment was easy. on how Logger works, where to find data of interest, and how to create and monitor custom dashboards. This element is critically important for most enterprises that are struggling with the increasing volume of log data in their environments. The respondents to the latest SANS Log Management Survey were in many cases spending hours or even days each and every week simply analyzing logs and trying to bring log management under control. Security analysts will be as efficient and effective as their log management products are easy to learn and use. Logger 6 should enable any organization to cut the time needed to perform maintenance, keep the systems up and running properly, and track events for security monitoring and response. 7
product its usability and effectiveness for security operations team members who would need to: investigation security issues We began evaluating Logger s capabilities by reviewing some of its monitoring dashboards. The first dashboard we looked at was Login and Connection Activity, shown Figure 8. Login and Connection Activity Dashboard This dashboard displays the total failed logins, both by product (system type) and user name. In our test network, the majority of failed logins occurred within the UNIX environment, which would immediately cause an experienced analyst to wonder:
(CONTINUED) We can also pinpoint the user accounts experiencing the most login failures and determine whether these failures correlate with the failed logins for UNIX servers. Figure 9. Account Login Failure Detail We compared users and the failed logins to their accounts with ease. We then had the option to click on individual users to get more detail on when and where each failed login occurred, as well. Such details are useful for any security analyst who is investigating a potential breach or suspected account compromise, because correlation with specific times and dates of other activities will likely be useful. 9
(CONTINUED) One of the most practical and useful features that can aid in monitoring and investigation activities is the free text search function within the Analyze category. As we found, entering a keyword into the search field triggers Logger to provide options for filter and event selection, as well as a search history, examples and suggestions for additional search operators that fit with the entered keyword. An example of this feature, Figure 10. Free Text Search A more advanced and specific query for netflow and top destination ports was simple to create using Logger s flexible and reasonably intuitive syntax. (An analyst might use such a query when looking for network scanning in the environment or for actively seeking out top data flow destinations.) The syntax for this query was Figure 11. A More Targeted Logger Query 10
(CONTINUED) Although this query is a simple example, Logger has an enormous number of syntax options, so analysts will definitely need to take some time to get comfortable with many of them. destination port, which may indicate (in normal situations) traffic headed to the Network Time Protocol (NTP) service or, alternatively, a new channel for malware distribution or some other attack. (This column appears in light blue.) We easily expanded the query to determine what the top source addresses (senders) are for these data flows, using the syntax Figure 12. Filtering Netflow Source Addresses to Port 123 (Note that Logger retrieved our search operator history, based on the string we entered.) Another example we explored was searching for all information and events related to Logger returned a distillation of all events and IDS platforms producing log and alert Figure 13. Querying All IDS Events 11
(CONTINUED) The results showed us what the IDSes were reporting, which is usually a valuable start to network intrusion analysis. We could query with ease across all such devices by using advanced syntax ( ) to evaluate their responses against a list detected by the test environment s IDS platforms. Figure 14. Searching for Top IDS Attack Categories (This query took just over five seconds to process and report on more than 514,000 aggregate events, doing so in real time.) We kept exploring our use case, entering even more detailed queries and examining known exploits and vulnerabilities in the environment. In particular, we explored a common scenario in enterprise security monitoring environments. The premise in this case was based on a new attack profile identified either by a IDS update. After the IDS sensors were updated with the signatures for this attack, how would an analyst go about seeing whether the signature tripped all the sensors in the environment? 12
(CONTINUED) Enter ArcSight Logger. Because we were concerned with one event type only, we could easily build on the last IDS sensor query we created to find out whether any of our IDS Execute Command, we could add this event name to our existing query, to end up with the following: dynamically updated with the new query. Figure 15. A Targeted Query for a Specific Exploit The results provided us with useful tactical data on which to focus. We could see how many events came in and when those events took place. We could also see which sensors detected the events; such information can help analysts pinpoint what services are targeted and where the attacks are happening. We also noticed that Logger assists analysts in constructing queries by providing operator history). 13
(CONTINUED) We finished this example by finding the top sources of this attack which we could then use for firewall rules, IP blacklists or other monitoring efforts simply by adding the filter Figure 16. Top Malicious Source Addresses With a list in hand of IP addresses that were sending malicious exploits and attacks to systems in our environment, we could add these addresses to firewall filtering and block rules, watch lists for monitoring additional activity, or threat intelligence cases in case they represent part of a larger attack campaign. 14
(CONTINUED) We explored several additional scenarios where information security and IT operations teams may need to monitor user activity in the environment for troubleshooting corporate VPN services to determine who was connecting and how often, using a simple query of Figure 17. Top VPN Access by Source IP Address As before, we could drill down into any areas of the graph, providing further visibility into who was connecting and from where. (Incidentally, this data could also help us in areas such as license or network management.) IP address 10.0.27.221 which took us to a detailed view of exactly when this address connected to the VPN. We also loaded the same query with a saved search that the labeled VPN Connections. 15
(CONTINUED) Figure 18. A Saved Search for VPN Connections collecting and aggregating data, Logger allows the receiver platforms to be peered together, facilitating searches across them all. In addition, analysts can search against the local log repository they re accessing or across them all very simply. features within the Dashboards and Analyze menus that analysts would find tactically useful in their jobs. To summarize our experiences: categories that can quickly get security teams up to speed, whether they are highly intuitive and rapid query creation that returned results in seconds. within multiple areas of the product that users can save for later use. Logger also remembers the most recent history of queries and filters. number of options is available, which may take time for analysts to learn and understand fully. The suggestions provided in the Logger UI go a long way to mitigate this wide span of options. 16
In our final area of review, we looked at the newly enhanced reporting facility in Logger 6. (The previous versions of Logger s reporting engine were highly capable but also complex and potentially challenging to use, by HP s own admission; the new version of graphs used; the reporting dashboard within our test environment included reports for bandwidth usage by source IP address, top IDS alerts and several others the ArcSight product team added to the testbed as examples. Our reporting dashboard appears in Figure 19. Reporting Dashboard 17
(CONTINUED) errors and warnings. Figure 20. Database Reports in Report Explorer
(CONTINUED) Customizing reports in case analysts need to modify parameters such as the period to examine, the device groups from which events should be selected or the storage database report to 30 days worth of events. Figure 21. Report Customization operations teams could easily use such a report to discover database issues. 19
(CONTINUED) canned report that HP includes with the product, based on a SANS reference document. 2 We ran the first log report listed, which showed attempts to gain access to the environment Figure 22. Running the SANS Top 5 Log Reports 2 Top 5 Essential Log Reports, Version 1.0; www.sans.org/security-resources/top5-logreports.pdf 20
(CONTINUED) Reporting is a critical part of security monitoring and event analysis, and the easier it is, the better. Figure 23. The Final SANS Top Failed Logins Report Customizing any report was easy. Selecting the Customize Report link when running a report enables analysts to add new graphs or data, include custom headers and graphics, or add or remove detail to tailor the report for different audiences. The reporting engine was so simple to use that we had a solid grasp on features and navigation within a brief time. Security teams will appreciate how easy it is to create new reports, customize existing reports, and schedule reports to run regularly and deliver security monitoring and event analysis, and the easier it is, the better. 21
Conclusion Security analysts who need to collect and monitor logs look for certain key features in a product: The ability to collect, analyze, and search across logs quickly is paramount. Customization in queries and dashboards will be essential to handle any number of unforeseen cases and scenarios that come up over time. Any log management product should come with a variety of prebuilt reports and offer analysts the ability to create new and customized reports easily. Security teams want the tools they use daily to have features that enable powerful searches across logs and provide the ability to drill down into data for granular viewing. A log management platform should be able to consume many different log data types and formats. HP ArcSight Logger 6 offers analysts all these capabilities and more. We found the product to be intuitive and easy to use, with powerful features that can save analysts time in analyzing and reporting on events within their environments. 22
About the Author is the founder and principal consultant with Voodoo Security, a SANS analyst, of organizations in the areas of security, regulatory compliance, and network architecture and engineering. He is a VMware vexpert and has extensive experience designing and configuring secure virtualized infrastructures. He has previously worked as chief security officer for Configuresoft and CTO for the Center for Internet Security. Dave is the author of the book Virtualization Security (Sybex). Dave currently serves on the board of directors at the SANS Technology Institute and helps lead the Atlanta chapter of the Cloud Security Alliance. Sponsor SANS would like to thank its sponsor: 23