Log Analysis with. Presenter: Nathan Hunstad May 2015

Transcription

1 Log Analysis with Presenter: Nathan Hunstad May 2015

2 Obligatory Disclaimer This talk represents my own work: I am not representing any employer, organization, or affiliated group, past, present, or future This talk is based on my experiences in my home lab network and not in an enterprise setting This is an overview only and is provided without warranty: do not rely on what you learn here for compliance or legal obligations!

3 What is Log Analysis? Not this:

4 What is Log Analysis? Or this:

5 What is Log Analysis, really? Forensics: Reconstructing events that have already happened Incident Response: Acting on logs in real-time to identify, contain, and remediate security incidents Troubleshooting: Evaluating systems for faults or unintended behavior and fixing as necessary

6 Handling Logs Help!

7 Splunk Splunk captures all kinds of machine data app log files, syslog, text files, configuration files...basically any text data can be ingested Splunk provides a powerful search engine based on MapReduce for fast searching1 Splunk has add-ins that allow for quickly setting up dashboards and reports for common log sources No, I do not work for Splunk 1

8 Splunk Licensing Splunk Enterprise: based on log volume Splunk Free: fewer features, 500MB/day Go over? You will lose search access! But good enough for home use

9 Splunk Licensing Average Logging Volume

10 Adding Data to Splunk

11 Getting Data Into Splunk Splunk Forwarder Install on any system to read log files locally and forward to Splunk Indexer Versions available for Windows, MacOS, Linux, Solaris, HPUX, AIX, and FreeBSD Configure using GUI or edit.conf files

12 Getting Data Into Splunk Listen on port Splunk daemon binds to a port to listen for traffic (TCP or UDP) Typically used with syslog data

13 Getting Data Into Splunk Monitor Files/Directories Splunk daemon monitors individual files or an entire directory for new files/changes to files Computes CRC and bytes read on files to detect changes Can automatically decompress common formats like zip files

14 Getting Data Into Splunk Remote Hosts What if you can't install a forwarder on a remote host (for example, your shared web host?) My Solution: cron job + monitoring files

15 Splunk Basics

16 Indexes Indexes are the logical buckets into which data is stored By default, all data gets stored in the main index, but other indexes can be defined A number of internal indexes exist for tracking Splunk functionality and start with _, such as _internal and _audit Data retention and access control* is done on a perindex basis

17 Buckets Buckets are collections of index data and metadata Buckets age through several stages: Hot, Warm, Cold, and Frozen Not terribly important for home use, but managing retention becomes important for large data sets

18 My Environment Splunk server: located on server running CentOS Feeds from VMs, Windows desktops, EdgeOS router, managed switch, application logs, external website

19 My Environment Data is split up into multiple indexes for logical grouping Indexes for firewall, switch, Linux, Windows, website, and BOINC events, plus a throwaway index for testing

20 Windows Events Events from Security, Application, and System logs

21 Windows Events PerfMon performance monitoring events

22 Linux Events Syslog events

23 Website Logs Multiple access logs

24 Website Logs Apache access_combined

25 Firewall Logs Dropped and specific accepted connections

26 Switch Logs Connected devices

27 Application Logs BOINC (Berkeley Open Infrastructure for Network Computing) events

28 Basic Search Syntax

29 Search Syntax Basic search: just type in what you want to see

30 Search Syntax Limiting by fields

31 Search Syntax Counting events: stats count

32 Search Syntax Top events: top

33 Search Syntax Bucketing events and charting: timechart

34 Security Events

35 Brute Force Windows Using ncrack against RDP

36 Brute Force Windows Success!

37 Brute Force Linux Using Metasploitable ssh_login module and default root_userpass.txt

38 Port Scanning (External) Port Scanning: Same source IP, multiple destination ports

39 Port Scanning (Internal) Port Scanning: Same source IP, multiple destination ports

40 SQL Injection sqlmap against DVWA Apache logs sent to Splunk

41 Blind SQL Injection sqlmap/dvwa

42 XSS Persistent XSS on DVWA

43 Mimikatz Running mimikatz to dump hashes Nothing happens

44 Correlation Transactions: group events together that match a pattern Successful login following failed logins

45 Correlation Show attackers in a table index=linux transaction host,rhost startswith="eventtype=sshd-login-failure" endswith="eventtype=ssh_open" bucket _time span=30m table _time, rhost

46 More Splunking

47 Field Extraction Splunk can handle some log types automatically pretty well, but adding rules for field extraction can help with searching and indexing A number of extractions come with Splunk ready for use, or you can add your own Uses regex for extraction

48 Lookups Uploading CSV files for extracting or expanding on data in logs

49 Lookups Previous timechart, now with names

50 Data Models Data Models are a powerful way of structuring data to generate specialized searches and visualizations Can be used to generate pivot tables and other complex objects

51 Pivot Tables Based on defined data models Display data in tabular format

52 Dashboards Bringing all your data to one spot, with user-selectable attributes

53 Visualizations Looking closer at the Website Attacks dashboard: Logarithmic Y-axis Daily Buckets

54 Visualizations Grouping Events Attacks: index=website joomlafailure sourcetype="php_error" transaction IP maxpause=1h maxevents=5000 where eventcount>1 table _time, IP, eventcount Port Scanning: index=firewall RuleName=WAN*default-D bucket _time span=30 eventstats dc(dpt) AS PortsScanned by SRC, _time where PortsScanned > 5 dedup SRC, PortsScanned table SRC, PortsScanned, _time

55 Visualizations Firewall Drops

56 Geolocation Splunk can Geolocate IP addresses

57 Geolocation Search: index=website joomlafailure sourcetype="php_error" transaction IP maxpause=1h maxevents=5000 where eventcount>1 iplocation IP geostats latfield=lat longfield=lon sum(eventcount)

58 Splunk TAs Splunk comes with TAs (Technology add-on) with predefined field extractions, transformations, and dashboards

59 Happy Logging!