Improving the Effectiveness of Log Analysis with HP ArcSight Logger 6

Transcription

1 Improving the Effectiveness of Log Analysis with HP ArcSight Logger 6 A SANS Product Review Written by Dave Shackleford April 2015 Sponsored by Hewlett Packard Enterprise 2015 SANS Institute

2 Introduction Most organizations today collect logs and actively use them for monitoring, forensics, troubleshooting, and detecting and tracking suspicious behavior, according to the ninth SANS Log Management Survey, in which 97 percent of organizations reported they are currently collecting and leveraging logs for all of these reasons and more. 1 How well they use logs is another matter entirely. In the same survey, 50 percent of respondents for whom detection and tracking of suspicious behavior was a stated need confirmed that such detection and tracking is moderately difficult to accomplish, with another 30 percent stating that log collection and analysis is difficult for this purpose. Many organizations are also struggling with large amounts of log data from a vast variety of distributed sources and are spending significant amounts of time analyzing logs each week 22 percent of respondents spend more than one full day per week analyzing logs. It s clear that log collection and analysis is a critical aspect for most IT security teams. However, even with the advances in log management techniques seen in recent years, many teams are still struggling to get control of their logs and properly manage them, both effectively and efficiently. We recently reviewed HP ArcSight Logger 6, which includes significant updates over earlier releases. The new Logger s standout features include improved incident analysis and response flexibility, overhauled reporting and monitoring, and general enhancements for ease of use. Our evaluation focused on three areas that HP notably updated and enhanced in Logger 6: Flexibility, customization and ease of use Security monitoring, investigation and incident analysis Reporting We can summarize our review process using this question: How can this tool help security analysts and operations teams perform their jobs more effectively? Fortunately, Logger 6 performed admirably for all the major use cases, and we found numerous capabilities that would help many organizations improve the effectiveness of their log management. 1 Ninth Log Management Survey Report, October 2014; 1

3 Ease of Use We reviewed ArcSight Logger 6 in a test environment that HP installed and configured, simulating many events across 20 logging devices to represent a typical enterprise. The first use case that we explored flexibility, customization and ease of use directly relates to the user friendliness of the dashboards and interfaces available to analysts. Our first stop was the main dashboard, shown in Figure 1. Figure 1. ArcSight Logger 6 Main Dashboard Although Logger 6 includes a number of stock dashboards (packaged for various roles and job functions), we used a dashboard prebuilt by the ArcSight team to demonstrate what current product users report to be the most popular graphs and charts. The Logger dashboard shown in Figure 1 provides an at-a-glance view in four categories, listed from upper left to lower right: All Failed Logins by User. Contains the aggregate number of failed login events across all users and platforms. UNIX - All SSH Authentications by User. Displays administrative SSH sessions to UNIX platforms; this information can assist in monitoring privileged activities. NetFlow Top Destination Ports. This panel shows patterns of network traffic throughout the environment, emphasizing services in active use. Intrusion - Malicious Code. Presents all malware-related events and occurrences within the environment. 2

4 Ease of Use (CONTINUED) Beyond these examples, we noted the flexibility to quickly change between saved dashboards in a variety of different categories. Custom dashboards are usually where security analysts spend their time, looking at aggregate events and trends that allow for easy access to more granular datasets. In the Dashboards menu area, a drop-down list is available to rapidly switch between saved dashboard views, making it simpler than ever to navigate to the desired dashboards. We quickly switched from this original custom dashboard to another one, labeled Intrusion and Configuration Events, that was configured for us. Much like the main dashboard, the Intrusion and Configuration Events dashboard shows popular and useful collections of security-related information such as Top Malicious Code Activity (upper left pane), Top Firewall Drops by Source (upper right pane) and others, as shown in Figure 2. Figure 2. Intrusion and Configuration Events Dashboard While reviewing the malicious code activity, we noticed a number of events labeled ICMP Packet Flood (upper left pane of Figure 2); these events can signal a potential denial of service (DoS) attack or hostile network discovery activity. 3

5 Ease of Use (CONTINUED) To get a sense for how simple it is to drill down on events, we simply clicked into the graph area on the entry for ICMP Packet Flood. Doing so provided more granular results and automatically redirected us to the Analyze category, as shown in Figure 3. Figure 3. Drilldown Malicious Events for ICMP Packet Flood The screen in Figure 3 provided a wealth of data related to the captured events, including the time of the events, what devices observed the events and which logging engine captured and recorded the events for analysis. We could also easily use this data to build a custom dashboard on the fly, using the top malicious IP addresses or another data type from within the events. To create quick dashboard charts and graphs, all we had to do was click the save button (in the toolbar on the query response page) and choose to save to an existing dashboard or create a new one, as shown in Figure 4. Figure 4. Creating a Custom Dashboard on the Fly 4

6 Ease of Use (CONTINUED) The Logger interface also allowed us to easily view the overall status of the monitored systems and events. By selecting the Summary menu item at the top of the dashboard window, we were able to quickly review the number of different event types across devices and endpoint agents that forward events to log collector servers in the test network. Clicking any of the various categories yielded more data, and simple metrics such as events per second (EPS) could be viewed in the upper-right corner of the main view, shown in Figure 5. Figure 5. Global Summary of Events Having immediate access to a central view of event count, types, systems and logging platforms (known as receivers in Logger jargon) is invaluable to security operations teams that need to manage a large environment. From this view a security analyst can immediately determine whether a particular system is seeing a higher count of events than normal, which receivers are getting the most logs and events sent to them, and what types and categories of events are being seen most frequently. This visibility allows large, distributed teams to focus on particular types of events or one or more receivers that are seeing higher event counts; teams can then scrutinize those platforms to see the cause of the changes. 5

7 Ease of Use (CONTINUED) Figure 6 shows another view of the entire environment and its performance, with emphasis and details on receivers, events, utilization and processing stats from the ArcSight host and, finally, storage. Figure 6. Logger Monitoring Summary of the Environment This view presents a wide range of data, including CPU usage for the Logger platform over specified time periods, total event flow, receiver status and a list of storage repositories defined for use within the event management infrastructure. This data is valuable for security professionals who need to keep up with changes in performance and events over time, as well as operations teams that need to track how much space is in use for event storage. One of the best ease-of-use features we were able to test was also one of the simplest: the direct navigation query field (shown at the top of the screen throughout the UI). This intelligent search query box autopopulates suggestions based on keywords or even just letter combinations and strings that a user types, making it exceedingly simple to locate various dashboard pages, analysis pages, specific data types and other UI elements. Figure 7 shows a search that starts with the term Data and the suggested search options that Logger 6 automatically creates. Figure 7. Dynamic Search Query Field 6

8 Ease of Use (CONTINUED) The Logger interface was incredibly simple to use. Within seconds, enormous amounts of data were readily visible and available, and finding specific events, dashboards, metrics and other important elements of the monitoring environment was easy. More importantly, any salt-worthy analyst should be able to rapidly get up to speed on how Logger works, where to find data of interest, and how to create and monitor custom dashboards. This element is critically important for most enterprises that are struggling with the increasing volume of log data in their environments. The respondents to the latest SANS Log Management Survey were in many cases spending hours or even days each and every week simply analyzing logs and trying to bring log management under control. Security analysts will be as efficient and effective as their log management products are easy to learn and use. Logger 6 should enable any organization to cut the time needed to perform maintenance, keep the systems up and running properly, and track events for security monitoring and response. 7

9 Security Monitoring and Incident Response The second area we focused on in this review is the real-life applicability of the product its usability and effectiveness for security operations team members who would need to: Monitor event data Quickly identify unusual behavior or events that warrant attention and investigation Use the product as an aid to assessing incidents and identifying the root cause of security issues We began evaluating Logger s capabilities by reviewing some of its monitoring dashboards. The first dashboard we looked at was Login and Connection Activity, shown in Figure 8. Figure 8. Login and Connection Activity Dashboard This dashboard displays the total failed logins, both by product (system type) and user name. In our test network, the majority of failed logins occurred within the UNIX environment, which would immediately cause an experienced analyst to wonder: What is happening in the UNIX environment are we under attack? Are we seeing brute force authentication attacks? Is an application or service rejecting logins for some unknown reason? 8

10 Security Monitoring and Incident Response (CONTINUED) We can also pinpoint the user accounts experiencing the most login failures and determine whether these failures correlate with the failed logins for UNIX servers. Clicking the graph labeled Top Failed Logins by User provided details of specific account activity, as shown in Figure 9. Figure 9. Account Login Failure Detail We compared users and the failed logins to their accounts with ease. We then had the option to click on individual users to get more detail on when and where each failed login occurred, as well. Such details are useful for any security analyst who is investigating a potential breach or suspected account compromise, because correlation with specific times and dates of other activities will likely be useful. 9

11 Security Monitoring and Incident Response (CONTINUED) One of the most practical and useful features that can aid in monitoring and investigation activities is the free text search function within the Analyze category. As we found, entering a keyword into the search field triggers Logger to provide options for filter and event selection, as well as a search history, examples and suggestions for additional search operators that fit with the entered keyword. An example of this feature, with a simple search for the term netflow, appears in Figure 10. Figure 10. Free Text Search A more advanced and specific query for netflow and top destination ports was simple to create using Logger s flexible and reasonably intuitive syntax. (An analyst might use such a query when looking for network scanning in the environment or for actively seeking out top data flow destinations.) The syntax for this query was netflow top dpt, and its search results came back in seconds, as shown in Figure 11. Figure 11. A More Targeted Logger Query 10

12 Security Monitoring and Incident Response (CONTINUED) Although this query is a simple example, Logger has an enormous number of syntax options, so analysts will definitely need to take some time to get comfortable with many of them. The result of this query as Figure 11 shows was that 123 was the top netflow destination port, which may indicate (in normal situations) traffic headed to the Network Time Protocol (NTP) service or, alternatively, a new channel for malware distribution or some other attack. (This column appears in light blue.) We easily expanded the query to determine what the top source addresses (senders) are for these data flows, using the syntax netflow where dpt=123 top sourceaddress, as shown in the query interface in Figure 12. Figure 12. Filtering Netflow Source Addresses to Port 123 (Note that Logger retrieved our search operator history, based on the string we entered.) Another example we explored was searching for all information and events related to our testbed s IDSes. We performed a free-form query for ids, and within a few seconds Logger returned a distillation of all events and IDS platforms producing log and alert events in the environment, as shown in Figure 13. Figure 13. Querying All IDS Events 11

13 Security Monitoring and Incident Response (CONTINUED) The results showed us what the IDSes were reporting, which is usually a valuable start to network intrusion analysis. We could query with ease across all such devices by using advanced syntax (ids AND categorydevicegroup CONTAINS "IDS/ Network" top categorytechnique) to evaluate their responses against a list of most-frequently-used attack types. This query, shown in Figure 14, ranks the attacks detected by the test environment s IDS platforms. Figure 14. Searching for Top IDS Attack Categories (This query took just over five seconds to process and report on more than 514,000 aggregate events, doing so in real time.) We kept exploring our use case, entering even more detailed queries and examining known exploits and vulnerabilities in the environment. In particular, we explored a common scenario in enterprise security monitoring environments. The premise in this case was based on a new attack profile identified either by a member of the IT operations or information security team or through a vendor-supplied IDS update. After the IDS sensors were updated with the signatures for this attack, how would an analyst go about seeing whether the signature tripped all the sensors in the environment? 12

14 Security Monitoring and Incident Response (CONTINUED) Enter ArcSight Logger. Because we were concerned with one event type only, we could easily build on the last IDS sensor query we created to find out whether any of our IDS sensors spotted this attack. Assuming the name of the IDS event was HTTP IIS Root.exe Execute Command, we could add this event name to our existing query, to end up with the following: ids AND categorydevicegroup CONTAINS "IDS/Network" where categorytechnique="/exploit/vulnerability" where name="http IIS Root.exe Execute Command" The query also appears in Figure 15, which shows the view after the dashboard dynamically updated with the new query. Figure 15. A Targeted Query for a Specific Exploit The results provided us with useful tactical data on which to focus. We could see how many events came in and when those events took place. We could also see which sensors detected the events; such information can help analysts pinpoint what services are targeted and where the attacks are happening. We also noticed that Logger assists analysts in constructing queries by providing unobtrusive suggestions, which appear in Figure 15 as Examples (below the search operator history). 13

15 Security Monitoring and Incident Response (CONTINUED) We finished this example by finding the top sources of this attack which we could then use for firewall rules, IP blacklists or other monitoring efforts simply by adding the filter top sourceaddress to the query, as shown in Figure 16. Figure 16. Top Malicious Source Addresses With a list in hand of IP addresses that were sending malicious exploits and attacks to systems in our environment, we could add these addresses to firewall filtering and block rules, watch lists for monitoring additional activity, or threat intelligence cases in case they represent part of a larger attack campaign. 14

16 Security Monitoring and Incident Response (CONTINUED) We explored several additional scenarios where information security and IT operations teams may need to monitor user activity in the environment for troubleshooting or other reasons. First, we looked for the top source IP addresses connecting to the corporate VPN services to determine who was connecting and how often, using a simple query of vpn top sourceaddress, as shown in Figure 17. Figure 17. Top VPN Access by Source IP Address As before, we could drill down into any areas of the graph, providing further visibility into who was connecting and from where. (Incidentally, this data could also help us in areas such as license or network management.) For example, we clicked the first result shown in Figure 17 (indicated by arrow) the IP address which took us to a detailed view of exactly when this address connected to the VPN. We also loaded the same query with a saved search that the ArcSight product team created for this review; this appears in Figure 18 in the row labeled VPN Connections. 15

17 Security Monitoring and Incident Response (CONTINUED) Figure 18. A Saved Search for VPN Connections For larger environments generating many events and numerous Logger receivers collecting and aggregating data, Logger allows the receiver platforms to be peered together, facilitating searches across them all. In addition, analysts can search against the local log repository they re accessing or across them all very simply. We spent by far the most time during our review exploring real-world use cases and features within the Dashboards and Analyze menus that analysts would find tactically useful in their jobs. To summarize our experiences: Logger provides a large number of out-of-the-box dashboards and analysis categories that can quickly get security teams up to speed, whether they are merely setting up monitoring or launching a time-critical investigation. The free-form search capabilities within Logger are spectacular and allowed for highly intuitive and rapid query creation that returned results in seconds. Logger users can easily create dashboards on the fly, as well as filters and queries within multiple areas of the product that users can save for later use. Logger also remembers the most recent history of queries and filters. Although the syntax for creating queries is not overly complex, an enormous number of options is available, which may take time for analysts to learn and understand fully. The suggestions provided in the Logger UI go a long way to mitigate this wide span of options. 16

18 Reporting In our final area of review, we looked at the newly enhanced reporting facility in Logger 6. (The previous versions of Logger s reporting engine were highly capable but also complex and potentially challenging to use, by HP s own admission; the new version of the product offers significantly streamlined reporting.) First, the reporting engine now has a dashboard with the most-frequently-run report types, based on the queries and graphs used; the reporting dashboard within our test environment included reports for bandwidth usage by source IP address, top IDS alerts and several others the ArcSight product team added to the testbed as examples. Our reporting dashboard appears in Figure 19. Figure 19. Reporting Dashboard 17

19 Reporting (CONTINUED) The Report Explorer (in the menu bar at left) is also easy to navigate and offers analysts the ability to run on-demand, prebuilt reports for compliance and monitoring tasks. Figure 20 shows a custom report we ran that searched the collected logs for database errors and warnings. Figure 20. Database Reports in Report Explorer 18

20 Reporting (CONTINUED) Customizing reports in case analysts need to modify parameters such as the period to examine, the device groups from which events should be selected or the storage locations for events is a simple task. In Figure 21, we changed the window for our database report to 30 days worth of events. Figure 21. Report Customization The report ran in seconds, and we could easily export it to a PDF or . Security or operations teams could easily use such a report to discover database issues. 19

21 Reporting (CONTINUED) A more security-centric report that we explored was the SANS Top 5 Log Reports a canned report that HP includes with the product, based on a SANS reference document. 2 We ran the first log report listed, which showed attempts to gain access to the environment through existing accounts, with failed logins as the primary event type. Figure 22 shows us selecting this report, which was easy to find within the Report Explorer menu. Figure 22. Running the SANS Top 5 Log Reports 2 Top 5 Essential Log Reports, Version 1.0; 20

22 Reporting (CONTINUED) The report finished quickly, and the output is shown in Figure 23. Reporting is a critical part of security monitoring and event analysis, and the easier it is, the better. Figure 23. The Final SANS Top Failed Logins Report Customizing any report was easy. Selecting the Customize Report link when running a report enables analysts to add new graphs or data, include custom headers and graphics, or add or remove detail to tailor the report for different audiences. The reporting engine was so simple to use that we had a solid grasp on features and navigation within a brief time. Security teams will appreciate how easy it is to create new reports, customize existing reports, and schedule reports to run regularly and deliver them via to analysts or management for review. Reporting is a critical part of security monitoring and event analysis, and the easier it is, the better. 21

23 Conclusion Security analysts who need to collect and monitor logs look for certain key features in a product: Scalability and performance. The ability to collect, analyze, and search across logs quickly is paramount. Flexibility. Customization in queries and dashboards will be essential to handle any number of unforeseen cases and scenarios that come up over time. Reporting. Any log management product should come with a variety of prebuilt reports and offer analysts the ability to create new and customized reports easily. Powerful analysis tools. Security teams want the tools they use daily to have features that enable powerful searches across logs and provide the ability to drill down into data for granular viewing. Broad support for log and event data. A log management platform should be able to consume many different log data types and formats. HP ArcSight Logger 6 offers analysts all these capabilities and more. We found the product to be intuitive and easy to use, with powerful features that can save analysts time in analyzing and reporting on events within their environments. 22

24 About the Author Dave Shackleford is the founder and principal consultant with Voodoo Security, a SANS analyst, instructor and course author, and a GIAC technical director. He has consulted with hundreds of organizations in the areas of security, regulatory compliance, and network architecture and engineering. He is a VMware vexpert and has extensive experience designing and configuring secure virtualized infrastructures. He has previously worked as chief security officer for Configuresoft and CTO for the Center for Internet Security. Dave is the author of the book Virtualization Security (Sybex). Recently, Dave co-authored the first published course on virtualization security for the SANS Institute. Dave currently serves on the board of directors at the SANS Technology Institute and helps lead the Atlanta chapter of the Cloud Security Alliance. Sponsor SANS would like to thank its sponsor: 23