Dissecting Firepower-FTD & Firepower-Services Design & Troubleshooting

Transcription

1

2 BRKSEC-3455 Dissecting Firepower-FTD & Firepower-Services Design & Troubleshooting Foster Lipkey, Technical Leader Veronika Klauzova, TAC Tech Lead

3 Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space cs.co/ciscolivebot#brksec Cisco and/or its affiliates. All rights reserved. Cisco Public

4 Agenda Introduction Updated FTD Packet Flow Data-Path Improvements Firepower New Features in X Best Practices for Deployments Troubleshooting Tools Exciting Real-World Use-Cases Conclusions

5 Your presenter throughout FTD journey Firepower TAC TL Snort Expert Sourcefire Veteran Automation Enthusiast Foster Lipkey BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 5

6 Your presenter throughout FTD journey Firepower TAC engineer Passionate Linux Admin Love to explore Cisco technologies Veronika Klauzova BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 6

7 Hardware & Software Review

8 NGFW evolution BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 8

9 What platforms can run FTD Software ASA 5500X-Series (5506X-5555X with SSD) BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 9

10 What platforms can run FTD Software Firepower 2100 series BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 10

11 What platforms can run FTD Software Power Console MGMT 8 x optic SFP+ ports Front view 2 x 2.5 SSD Bays Rear view 2x optional NetMods 2 x Power Supply Module Bays 6 x Hot-Swap Fans units Firepower 4100 series BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 11

12 BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 12

13 BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 13

14 BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 14

15 Updated FTD Packet Flow

16 Firepower Threat Defense high level DETECTION ENGINE / Snort Packet Data Transport System (PDTS) DATA-PATH / LINA FXOS BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 16

17 Firepower 2100 architecture overview BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 17

18 Firepower 9300/4100 architecture overview BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 18

19 FTD Packet-Flow Detection Engine / Snort RX YES PDTS DAQ Lina rule-id matched Ingress Interface Existing Conn NO Egress Interface Pre-Filter L3/L4 ACL ALG checks NAT L3, L2 hops VPN Decrypt QoS, VPN Encrypt Data-Path / LINA TX BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 19

20 LINA / Data-Path Detection Engine/ Snort - Architecture BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 20

21 Data-Path Improvements

22 Data-Path improvements / Safe Guards Snort Fail Open When Busy If the buffer going into Snort is 85% full, new flows will be bypassed Snort Fail Open When Down When Snort goes does due to restart for policy deploy, or for any other reason new flows will be bypassed BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 22

23 Data-Path improvements / Safe Guards Device > Device Management [Edit] > Device tab Automation Application Bypass If traffic enters Snort through the buffer and does not provide a verdict back to LINA within configured threshold, Snort is restarted and a core file is generated BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 23

24 Show Time

25 Snort reload instead of restart As of following changes would not cause Snort to be restarted This applies to all FTD devices managed by FMC Policy changes URL Application ID Intrusion Policy NAP policy Policy action Refer to URL categories for the first time in AC rules or remove all existing references to URL categories Turn on/off Application ID Add or Delete Intrusion Polices in AC rules, or Edit Intrusion Policy Attach a NAP policy for the first time to AC Policy Simple SRU update Security Intelligence Typical rule updates without Shared Object (SO) / binary rule updates Changes to Whitelist/Blacklist of URL, DNS entries BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 25

26 Other snort major updates Changes to application detectors display warnings Break HA operation restart snort/s (warning displayed) Memory allocation changed SRU simple rule changes does not cause snort restart, but binary objects do Binary changes are not that frequent Whether snort would affect it depends on system resources BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 26

27 Snort Preserve-Connection When Snort goes down connections with Allow verdict are preserved in LINA Snort does NOT do a mid-session pickup on preserved flows on coming up Does NOT protect against new flows while Snort is down Feature Introduction Can be enabled/disabled from CLISH: configure snort preserve-connection enable/disable BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 27

28 Minimalize network disruption during policy deployment Snort restart behavior depends on Advance settings in Access Control Policy TAC highly recommend to enable: Inspect traffic during policy apply = Yes Without this option Snort always restarts during policy deployment BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 28

29 Snort Restart & Reload Architecture BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 29

30 Show Time

31 Firepower New Features in X

32 New Signed Software Update/Upgrade images Signed images were introduced in Signed images are the.rhel.tar files (caution: DO NOT UNTAR THEM!) Managed FTD device can be upgraded only after FMC is upgraded FTD on platforms 4100 and 9300 series needs to have upgraded FXOS software via Firepower Chassis Manager prior FTD upgrade to version To update an FMC from to release an unsigned upgrade package need to be used (.sh file) Platform Current Version Destination Version Package name to be used FMC Sourcefire_3D_Defense_Center_S3_Upgrade sh FMC Sourcefire_3D_Defense_Center_S3_Upgrade sh.REL.tar BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 32

33 Virtual FDM Allows users to manage virtual platforms using on-box management Only fresh installation to enables FDMv management option Initial setup can be done once, it cannot be relaunched Adding/removing interfaces on already running FTDv requires deregistration of management (all configuration will be lost!) BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 33

34 Threat Intelligence Director Consumes third-party cyber threat intelligence Requirements: FMC and FTD running GB of memory Protect license (IPv4, IPv6, Domain and URL detection) Malware license (SHA-256 detection) Terminology STIX Structured Threat Intelligence expression TAXII transport mechanism for STIX TID is activated under Access Control Policy Advanced tab TID correlation for incident generation is dependent on an exact match! BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 34

35 TID High-Level Architecture Third-Party Cyber Security Intelligence STIX TAXII Flat files Cisco TID on FMC Syncd.pl Sftunnnel (TCP 8305) Observables Can take up to 20 minutes! NGFW / NGIPS (manage device) BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 35

36 TID Troubleshooting Observables type IPv4 and IPv6 addresses Domain names URL s SHA-256 hashes File location /ngfw/var/sf/iprep_download /ngfw/var/sf/sidns_download /ngfw/var/sf/siurl_download /ngfw/var/sf/sifile_download BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 36

37 API bulk rule access insertion, yay! Old behavior: one AC rule can be imported at the time New behavior: we can insert up to 1000 rules within same API request! How cool is that? We can insert rules at specific location (rule number or within specific category/section) After rule insertion, other rules are automatically reordered Rest API can handle if other user is already modifying the same rule set When no position of the rule is defined, it goes to the end of ACP BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 37

38 Best Practices for Deployments (security is our priority)

39 VPN deployment on FTD: things that you might have missed! Cisco Employee working from home attacker Clear-text / un-authenticated session FMC Should been never been allowed The Internet outside inside Cisco network NGFW Anyconnect (encrypted session) FTP Servers BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 39

40 Is your network protected? BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 40

41

42 FTD / IPS is dropping packets HELP! FTD detection engine / IPS bottleneck causing throughput issues Do we have enough processing power / right hardware? What is traffic pattern / volume? (the type, size and protocol of packet) Why not simply enable all of the rules?. Ok, now really, how many Snort signatures are enabled? expensive signatures & local rule IPS alerting load (processing and disk operations) Expensive work on preprocessors BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 42

43 Tuning IPS rules #(TAC tip & trick) Use case: poor performance with default IPS policy baseline for FTP traffic Simplified topology: client (Windows 10) ---1Gbps --- FTD Gbps --- server (Windows 10) Performance measurement results with default policy ~ 380 Mbps Performance measurement after IPS rule tuning ~ 970 Mbps BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 43

44 Full example: performance numbers from field/lab testing Mode Protocol Configuration Throughput Transparent FTP (Filezilla ) Pre-filter policy with Fast-path rule for TCP ports 20 and 21 ~979 Mbps Access Control Policy, Allow rule for TCP ports 20 and 21, IPS connectivity over Security Access Control Policy, Allow rule for TCP ports 20 and 21, IPS Balanced Security and Connectivity Access Control Policy, Allow rule for TCP ports 20 and 21, IPS Security over Connectivity Access Control Policy, Allow rule for TCP ports 20 and 21, IPS Maximum detection Access Control Policy, Allow rule for TCP ports 20 and 21, IPS tuned (base no rules active + 51 active rules) Filter used: ftp metadata:"security-ips drop" Access Control Policy, Allow rule for TCP ports 20 and 21, IPS tuned (base no rules active + 51 active rules) Filter used: ftp metadata:"security-ips drop" ~650 Mbps ~380 Mbps ~340 Mbps ~320 Mbps ~971 Mbps ~800 Mbps + File policy with application protocol FTP (detect all file types and block malware executable s with local malware analysis) BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 44

45 Low IPS performance? rule it out by FTD rule profiling! Edit /ngfw/var/sf/detection_engines/<uuid>/ advanced/perf_monitor.conf config profile_rules: print all, sort avg_ticks, filename /ngfw/var/log/profiling-rules.log config profile_preprocs: print all, sort avg_ticks, filename /ngfw/var/log/profiling-preprocs.log Restart Snort pmtool restartbytype snort Start rule profiling > system support run-rule-profiling BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 45

46 Low IPS performance? rule it out by FTD rule profiling! BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 46

47 Performance graphs from the WebUI Why does Bytes/Packet matter? BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 47

48 Reassembly cost Posted throughput ratings for the Firepower appliances are usually rated at 1518 bytes packets. Smaller packets results in more processing. 1MB of traffic with 1518 bytes/packets = ~ 658 packets 1MB of traffic with 400 bytes/packet = ~ 2500 packets Every packet header must be evaluated and the packet has to be placed into the buffer for re-assembly. The larger number of packets to process requires more CPU time. BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 48

49 Let s talk about the elephant in the room Large flows are generally related backup, database replication, etc. which usually does not require inspection Sort Analysis > Connections for connection size to find top talkers Once we determine the top talkers, and confirm they can be safely ignored, we create trust rule for the IP conversations. Mitigations IAB / Pre-Filter fast-path BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 49

50 Sizing your NGFW / NGIPS throughput considerations Number of Snort instances per FTD platform For Your Reference Platform Snort Instances Platform Snort Instances Platform Snort instances Firepower Firepower Firepower 9300 SM Firepower Firepower Firepower 9300 SM Firepower Firepower Firepower 9300 SM Firepower Firepower Enabling File-Inspection will change these values > pmtool show affinity BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 50

51 Troubleshooting Tools

52 What are main FTD processes and what they do? snort ids_event_processor ids_event_alerter inspects network traffic (pass, block and alert) sends intrusion events to managing device (FMC) sends intrusion events to Syslog or SNMP server wdt-util used for fail-to-wire / hardware bypass sftunnel diskmanager, Pruner Lina Snmpd, ntpd SFDataCorrelator processing events pm (process manager) secure tunnel between managed device and FMC managing disk space and clean up old files Responsible for Firewall functionality like ACL, NAT, Routing etc. SNMP monitoring, responsible for time synchronization responsible for launching and monitoring of all FTD relevant processes and restarting them in case of failure BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 52

53 Process Management - basics FTD Root CLI: ftd-vklauzov:/# pmtool status grep " - " head SFDataCorrelator (normal) - Running mysqld (system,gui,mysql) - Running httpsd (system,gui) - Waiting sftunnel (system) - Running Process name Category Status Process ID BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 53

54 Process Management - basics FMC Root CLI: root@fmc-2:/# pmtool disablebyid sftunnel root@fmc-2:/# pmtool status grep " - " grep sftunnel sftunnel (system) - User Disabled root@fmc-2:/# pmtool enablebyid sftunnel root@fmc-2:/# pmtool status grep " - " grep sftunnel sftunnel (system) - Running 1720 BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 54

55 Data-path and Snort capture points Detection Engine / Snort > capture-traffic 2. snort inbound/outbound firepower# capture out 1. firepower# capture in DATA-PATH 3. data-path outbound data-path inbound BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 55

56 Data-path inbound/outbound - The Wires Never Lie! Data-path/lina (diagnostic cli): firepower# capture in interface INSIDE match icmp any any trace detail Capture name Interface name protocol Source Destination BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 56

57 Data-path stop and delete captures Delete packet captures firepower# no capture in Stop packet captures firepower# no capture in interface inside BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 57

58 Show Time

59 Show Time

60 Show Time

61 Snort Capture - The Wires Never Lie! (1) CLISH: > capture-traffic Options: -s 0 -w capture.pcap icmp and host IP > : ICMP echo request,id 24538,seq 1,length 64 Berkeley Packet Filter syntax same as for tcpdump capturing tool -s 0 means snaplength, in other words no limit for packet size -w filename.pcap indicates to which file you want to write output of data captured by specified filter capture is written to /ngfw/var/common/ folder Copy file out to SCP server: file secure-copy <IP address of server> <username> <location where to copy the file> capture.pcap BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 61

62 Snort Capture - The Wires Never Lie! (2) CLISH: NON-VLAN TAGGED TRAFFIC VLAN TAGGED TRAFFIC > capture-traffic Options: -s 0 -v -n -e (icmp and host ) or (vlan and icmp and host ) 00:50:56:b6:0b:33 > 58:97:bd:b9:73:ee, ethertype 802.1Q (0x8100), length 78: vlan 208, p 0, ethertype IPv4, (tos 0x0, ttl 128, id 5366, offset 0, flags [none], proto ICMP (1), length 60) LINA CLI: IN LINA CLI: OUT firepower# sh cap inside 802.1Q vlan#208 P > : icmp: echo request firepower# sh cap outside > : icmp: echo request BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 62

63 Which ACP rule is being evaluated? Tool that provides the Access Control Rule evaluation status for each flow as we receive packets in real time. NGFW debug needs to have specified at least one filtering condition. >system support firewall-engine-debug Please specify an IP protocol: icmp Please specify a client IP address: Please specify a server IP address: Monitoring firewall engine debug messages > AS 1 I 44 New session > AS 1 I 44 using HW or preset rule order 2, 'allow and inspect', action Allow and prefilter rule > AS 1 I 44 allow action BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 63

64 Show Time

65 Access Control Policy Rule Hit Counters > show access-control-config ===================[ ciscolive ]==================== Description : Default Action : Allow Default Policy : Balanced Security and Connectivity Logging Configuration DC : Disabled Beginning : Disabled End : Disabled Rule Hits : Variable Set : Default-Set... (output omitted)... Policy name # watch /usr/local/sf/bin/sfcli.pl show firewall grep "ciscolive\ Rule\:\ Rule Hits " ===================[ ciscolive ]==================== Rule Hits : [ Rule: allow ] Rule Hits : 14 Rule name BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 65

66 ACP Rule Hit Counters FMC WebUI Analysis -> Custom -> Custom Workflows -> Create Custom Workflow and use Table Connection Events Add page and fill in fields like: Access Control Policy, Access Control Rule, Count, Initiator IP, Responder IP Add Table view BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 66

67 ACP Rule Hit Counters FMC WebUI vs CLISH > show access-control-config [ Rule: DNS and icmp ] Action : Allow Destination Ports : protocol 6, port 53 protocol 17, port 53 protocol 1 protocol 6, port 80 Logging Configuration DC : Enabled Beginning : Enabled End : Enabled Rule Hits : 28 Variable Set : Default-Set (truncated) Why the hit counters do not match? BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 67

68 Capture With Trace GUI Quickly Identify where in the data-path the traffic is impacted BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 68

69 Show Time

70 Show Time

71 CLI Analyzer Contextual help and highlighting Embedded Intelligence File Analysis BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 71

72 Show Time

73 LINA / Data-Path I m a trouble-shooter now System Support Trace Capture w/ trace Capture-traffic Firewall-Engine-Debug BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 73

74 Exciting Real-World Use-Cases

75 Real World Scenario Intermittent network outages following migration to FTD Following a migration from ASAs to FTDs on pair of boarder firewalls intermittent outages occur. BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 75

76 Real World Scenario Using our tools Failed: Working: Sometimes its what the FW didn t do that counts Symptoms: Migration from ASAs to FTDs results in outage under load. When placing ASAs back inline outage does not occur Troubleshooting: Performance review Capture w/ Trace Packet Capture with FTDs inline Packet Capture with ASAs inline Compared the packet captures Root Cause Sysopt connection tcpmss set to 0 Changed to 0 by adding jumbo frames to interface BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 76

77 Real World Scenario HARDWARE ERROR ON LCD HARDWARE ERROR on Firepower sensor LCD panel BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 77

78 Closing

79 Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space cs.co/ciscolivebot#brksec Cisco and/or its affiliates. All rights reserved. Cisco Public

80 Please complete your Online Session Evaluations after each session Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt All surveys can be completed via the Cisco Live Mobile App or the Communication Stations Complete Your Online Session Evaluation Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at Cisco and/or its affiliates. All rights reserved. Cisco Public

81 Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Tech Corner Meet the Engineer 1:1 meetings Related sessions BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 81

82 Veronika Klauzova Foster Lipkey Thank you

83