ELK for Enterprise IR Visibility

Transcription

1

2 ELK for Enterprise IR Visibility Mark Goudie, Security Consulting Director Barry Anderson, Security Architect

3 Agenda Introduction Security Challenges IR and Issues ELK Inputs Enrichment Cloud Integration (Office 365) Cisco Offering

4 > whoami Mark Goudie 20+ years security / technology experience 8+ years data breach investigation experience Former author of SANS Top 20 Author SANS Ethics Committee member Inaugural President High Tech Crime Investigation Association (A/NZ) Payment card industry forensic investigator

5 > whoishe Barry Anderson 20+ years at the pointy end of the spear Built one of the first firewalls for a business in Australia Created the Forensicator FATE project for automating digital forensics evidence processing Presented at the 2015 SANS DFIR Summit

6 Security Challenges Changing Business Models Dynamic Threat Landscape Complexity and Fragmentation

7 Security Challenges Changing Business Models Dynamic Threat Landscape Complexity and Fragmentation TALENT CLOUD 60% data in breaches is stolen in hours 12x 45 25% increase in an organization s cybersecurity risk due to IoT 5 10 times more cloud services are being used than known by IT 54% of breaches remain undiscovered for months Demand for security talent Security vendors for some customers

8 Incident Response Worms Viruses Phishing Data loss DDOS Human Mistakes Not-so Human Mistakes

9 Incident Response - Reality No DNS log or we not collecting all DNS logs Cannot see communication in the HTTP proxy logs Not sure if connection was denied We can t see if there is outbound connection We don t have enough log data We don t log on that server/network/segment/system We can t PCAP data We don t have enough process to support It s all in the cloud

10 Visibility is essential Q. What is the common link between these victims? A. Threat persistence caused by a lack of VISIBILITY

11 What do we need visibility of? Have visibility to prevent data theft Focus on detecting active security incidents Must detect before significant data theft Active security incident

12 (A few) Reasons for Failure Being Human Bias We Know it ALL Syndrome Well-intentioned, ill-informed ASSumptions Too slow in response

13 Steps towards a Solution Enable informed decision making Prefetching as much data as you can Understand your network Enrich data sets Let system provide you what you need Increase automation IR Process Prepare before the incident

14 ELK

15 ELK Logstash Collect, Enrich & Transport Data more importantly, Collect, Normalize, Enrich & Transport Data ElasticSearch Search & Analyze Data in Real Time Kibana Explore & Visualize Your Data, or Explore & Visualize; Go Hunting!

16 Logstash What problem does it solve? In our enterprises, logs are stored: In different locations (read: all over the shop) In different formats Dates?! With different permissions Logs may also require domain knowledge to interpret, and so: Are not easily searchable!

17 Logstash: Input, Filter, Output Input Filter Output

18 Logstash - Config: Here is a very simple Logstash configuration. It takes events from standard input, does no filtering and sends them to Elasticsearch and standard output. input { stdin { } } output { elasticsearch { hosts => ["localhost:9200"] } stdout { codec => rubydebug } }

19 Logstash - Input Plugins: Here are some of the available input plugins stdin tcp, udp syslog eventlog (Windows) exec pipe (stream events from a long-running command) rabbitmq, zeromq kafka twitter s3, sqs (AWS) elasticsearch (Inception, anyone?) Beats (files,processes, Windows events

20 Logstash - Filter Plugins: Here are some of the available filter plugins grok grep csv mutate drop geoip dns anonymize

21 Logstash - Output Plugins: Here are some of the available output plugins elasticsearch redis file http jira pager graphite cloudwatch kafka exec rabbitmq, zeromq

22 Logstash Config: Input Capture logs and documents / Events input { # syslog, live via udp and tcp port syslog { port => 5514 type => "syslog" use_labels => false } # archived syslog files, in standard /var/log/* format file { type => "archive-syslog" path => [ "/usr/local/logstash-syslog/**/*" ] sincedb_path => "/var/db/logstash/sincedb" start_position => "beginning" exclude => [ "*.gz", "*.zip" ] } }

23 Elasticsearch Document Based Search Engine JSON based, Apache Lucene Distributed Multi-tenancy API exposure & RESTful Modules for programming and scripting

24 Kibana Visualize your DATA Build and Share queries, Visualization Build and Share Dashboards Export the Views Accessible via Web Easy to use, less learning curve

25 Sample data

26 Kibana

27 Kibana

28 Kibana

29 Kibana Terms Query : Query Elasticsearch Filter : Elasticsearch Filter to get specific results including Time Ranges Panel : View Search results Dashboard : Collection of panels

30 Input

31 How to Send logs to ELK? Syslog /Syslog-NG Logstash-forwarder FileBeat

32 What is FileBeat Replacement for Logstash Forwarder Lightweight Part of collection of shipping tools : Beats

33 ELK: Pipeline Shipper Visualization Indexer Logstash Search and Storage

34 <Your IDS goes here>

35 Suricata/Snort/Bro/SELKS/Security Onion Suricata an IDS built to be compatible with Snort but built for maximum speed Uses all the cores From 2.0 onwards Eve JSON event and alert output Snort Bro an NSM built on a DSL designed for network analysis (also called Bro!) We re using Bro as an example because it s quite interesting, and because it s the one you re least likely to be familiar with.

36 Where Bro Stands Signature IDS HIPS Host BRO Network FLOW Anomaly

37 BRO Logs DNS HTTP Connection SMTP SSH FTP Weird.. etc

38 BRO Log format By default CSV Consist of header describing the columns Can be changed to JSON format (if required)

39 BRO DNS example #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answersttls rejected C1UL0O1FbUwxOR2ol udp in-addr.arpa 1 C_INTERNET 12 PTR 3 NXDOMAIN F F T F F

40 Logstash-forwarder.conf { } "network": { "servers": [ " :5001" ], "timeout": 60 },

41 Secure logs while shipping } { "network": { "servers": [ " :5001" ], "timeout": 60 "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt" },

42 Define logs to ship "files": [ ] { } "paths": [ "/nsm/bro/logs/current/dns.log" ], "fields": { "type": "bro_dns_log" }

43 FileBeat to Send Logs Edit /etc/filebeat/filebeat.yml filebeat: # List of prospectors to fetch data. prospectors: # Each - is a prospector. Below are the prospector specific configurations # Paths that should be crawled and fetched. Glob based paths. - /nsm/bro/logs/current/dns.log

44 FileBeat to Send Logs Contd.. Edit /etc/filebeat/filebeat.yml Contd.. # Possible options are: # * log: Reads every line of the log file (default) # * stdin: Reads the standard in input_type: log # Type to be published in the 'type' field. For Elasticsearch output, document_type: bro_dns_log

45 FileBeat to Send Logs Contd.. Edit /etc/filebeat/filebeat.yml Contd.. # Multiple outputs may be used. output: ### Logstash as output logstash: # The Logstash hosts hosts: [" :5002"]

46 Add Security to Filebeat # Optional TLS. By default is off. tls: # List of root certificates for HTTPS server verifications certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"] Note: Configure local GeoIP database support. geoip: paths: - "/usr/share/geoip/geolitecity.dat"

47 Logstash: Input (FileBeat) input { } beats { port => 5002 type => "bro_dns_log" ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" }

48 Filters

49 Logstash: Filter filter { if [message] =~ /^#/ { drop { } } else {..

50 Logstash: Filter if [type] == "bro_dns_log" { csv { columns => ["ts","uid","id_orig_h","id_orig_p","id_resp_h, separator => " " }

51 Output

52 Logstash: Output output { elasticsearch { hosts => " " } stdout { codec => rubydebug } }

53 Logstash: Multiple Output output { if [type] == bro_dns_log" { elasticsearch { hosts => " index => "bro_conn_log-%{+yyyy.mm.dd}" } } else if [type] == finance_log" { elasticsearch { hosts => "

54 Kibana

55 Kibana : Configure Index

56 Kibana Result

57 Kibana: Discover Events

58 Enrichment

59 Logstash: Add GeoIP geoip { } geoip { } source => "id_orig_h" target => "orig_geoip" source => "id_resp_h" target => "resp_geoip"

60 Logstash: Add ASN geoip { database => "/usr/share/geoip/geoipasnum.dat" source => "id_orig_h" target => "orig_geoip" }

61 Logstash: Add Intel translate { field => id_resp_h" destination => "Intel_hit" directory_path => /etc/logstash/intel_ip.yaml" } Intel_ip.yaml: <field>: <Message> <IP>: Intel received from.on

62 Cloud Integration (Office 365)

63 Office365 Integration Use internal SPAM filtering Use Native Office365 Trace logs Use Native Office365 API recently announced.

64 Office365 Integration step 1 Set-ExecutionPolicy RemoteSigned on the system $credential = Get-Credential

65 Office365 Integration step 2 $credential.password ConvertFrom-SecureString Set-Content c:\scripts\password.txt $proxysettings = New-PSSessionOption -ProxyAccessType IEConfig

66 Office365 Integration step 3 $session = New-PSSession -ConfigurationName Microsoft.Exchange - ConnectionUri -Credential $credential - Authentication Basic -AllowRedirection -SessionOption $proxysettings

67 Office365 Integration step 4 Import-PSSession $session Get-Mailbox Where-Object {_.RecipientTypeDetails -eq UserMailbox } Set- Mailbox Audit enabled $true

68 Office365: Get Traces Get-MessageTrace -StartDate $datestart -EndDate $dateend Export-Csv - NoTypeInformation $OutputCSVFile

69 LogstashForwarder / Filebeat : input { lumberjack { port => 5000 type => "logs" ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" } }

70 Logstash Filter : if [type]=="logs" { csv { separator => "," columns => ["PSComputerName","RunspaceId","Organization","message_id","time","src_user","recipient","subject","recipient_status","dest_ip","src_ip","size","message_tid","sta rt","end","index"] }

71 Logstash Filter: contd.. Drop header if (["PSComputerName"] == "PSComputerName") { } drop { }

72 Kibana: Lets Visualize

73 DashBoard

74 Where to from here?

75 What about once we start to scale? Look at the OpenSOC architecture. This is an architecture designed to scale (1.2Mpps analyzed in realtime?!): Telemetry Capture - Apache Flume Data Bus - Apache Kafka Stream Processor - Apache Storm Long-Term Data Store - Apache Hive 12 Long-Term Packet Store - Apache Hbase Real-Time Index and Search Elasticsearch This has entered the Apache Incubator as Apache Metron.

76 Cisco Managed Security Offerings

77 Active Threat Analytics (ATA) Near real-time analytics Analytics Anomaly detection Zero day threat focus Reduced mean time to respond Operationalization Advanced expertise People Combat security talent shortage Force multiply internal resources Access to actionable sources of intelligence Cisco proprietary telemetry Intelligence Adapted for customer intelligence Industry-specific intelligence Integration of the latest security technology Technology Scalable and modular architecture Extensible platform designed to evolve with market demands

78 ATA Enables: Speed Accuracy Focus Rapid threat detection reduces the mean time to respond High fidelity cuts down on false positives Increased visibility and control illuminates security blind spots Customer Benefits Risk Mitigation Proactive Security Operational Efficiency Strategic Focus Comprehensive Coverage

79 FIREWALL FIREWALL ATA Architecture Overview CUSTOMER PREMISE CISCO DATA CENTER Full Packet Cisco NetFlow Third Party Machine Exhaust Full Packet Capture Sourcefire AMP Sourcefire IDS Netflow and Metadata Extraction Anomaly Detection Collective Security Intelligence Deterministic and Statistical Analytics Big Data Analytics ThreatGrid VPN SOC INTERNET Secure Connection (HTTPS/SSH/IPSec) VPN CUSTOMER 24/7 ACCESS DEDICATED CUSTOMER SEGMENT Investigator Portal Administrative Consoles Authentication Services CMSP PORTAL Dedicated Customer Portal TICKETING Alerting/Ticketing System COMMON SERVICES Threat Intelligence

80 Solution Overview Full packet capture Protocol metadata NetFlow Machine exhaust (logs) Parse + Format Threat Intelligence Feeds Enrich Alert Applications + Analyst Tools Log Mining and Analytics Network Packet Mining and PCAP Reconstruction Big Data Exploration, Predictive Modelling Unstructured telemetry Other streaming telemetry Enrichment Data

81 Q & A

82 Complete Your Online Session Evaluation Give us your feedback and receive a Cisco Live 2016 T-Shirt! Complete your Overall Event Survey and 5 Session Evaluations. Directly from your mobile device on the Cisco Live Mobile App By visiting the Cisco Live Mobile Site Visit any Cisco Live Internet Station located throughout the venue T-Shirts can be collected in the World of Solutions on Friday 11 March 12:00pm - 2:00pm Learn online with Cisco Live! Visit us online after the conference for full access to session videos and presentations.

83 Thank you

84