Dissecting Firepower-FTD & Firepower-Services Design & Troubleshooting

Dissecting Firepower-FTD & Firepower-Services Design & Troubleshooting Veronika Klauzova BRKSEC-3455

Agenda Introduction Updated FTD Packet Flow Data-Path Improvements Best Practices for Deployments Troubleshooting Tools Firepower New Features Exciting Real-World Use-Cases Conclusions BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 3

Cisco Webex Teams Questions? Use Cisco Webex Teams (formerly Cisco Spark) to chat with the speaker after the session How 1 2 3 4 Find this session in the Cisco Events App Click Join the Discussion Install Webex Teams or go directly to the team space Enter messages/questions in the team space Webex Teams will be moderated by the speaker until June 18, 2018. cs.co/ciscolivebot#brksec-3455 BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 4

Your presenter for today Firepower engineer Passionate Linux Admin Love to explore Cisco technologies Veronika Klauzova BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 5

Hardware & Software Review

What platforms can run FTD Software Power Console MGMT 8 x optic SFP+ ports Front view 2 x 2.5 SSD Bays Rear view 2x optional NetMods 2 x Power Supply Module Bays 6 x Hot-Swap Fans units Firepower 4100 series BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 10

Updated FTD Packet Flow

Firepower Threat Defense high level DETECTION ENGINE / Snort Packet Data Transport System (PDTS) DATA-PATH / LINA FXOS BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 15

FTD Packet-Flow Detection Engine / Snort RX YES Lina rule-id matched PDTS DAQ Ingress Interface Existing Conn NO Egress Interface Pre-Filter L3/L4 ACL ALG checks NAT L3, L2 hops VPN Decrypt QoS, VPN Encrypt Data-Path / LINA TX BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 18

Detection Engine/ Snort - Architecture Snort SNORT Decode SI (IPRep) Frag3 (IP Defrag) Stream5 (Reassembly) AppID DAQ LINA / Data-Path Protocol/Application Preprocessors FTP/TELNET HTTP DCE/RPC DNS SIP SSH SSL SunRPC POP IMAP SMTP Others (non-standard) File Policy QoS Classify (FTD only) ACP Eval IPS Policy Before ACP Rules SI (DNS/URL) Specific Threat Detection (Pre-processors) Back Orifice Portscan Rate-Based Attack Sensitive Data IPS Rule Eval BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 19

Data-Path Improvements

Snort reload instead of restart As of 6.2.2 following changes would not cause Snort to be restarted This applies to all FTD devices managed by FMC 6.2.2 Policy changes URL Application ID Intrusion Policy NAP policy Simple SRU update Security Intelligence Policy action Refer to URL categories for the first time in AC rules or remove all existing references Turn on/off Application ID Add or Delete Intrusion Polices in AC rules, or Edit Intrusion Policy Attach a NAP policy for the first time to AC Policy Typical rule updates without Shared Object (SO) / binary rule updates Changes to Whitelist/Blacklist of URL, DNS entries BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 22

Deployment changes causing interruption 6.2.3 SSL VDB version update User Identity Network discovery (http, ftp, msdn) Update of SRU version Max MTU Snort/DAQ version update System Upgrade BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 24

Minimalize network disruption during policy deployment Snort restart behavior depends on Advance settings in Access Control Policy TAC highly recommend to enable: Inspect traffic during policy apply = Yes Without this option Snort always restarts during policy deployment BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 25

Show Time

Other snort major updates Changes to application detectors display warnings Break HA operation restart snort/s (warning displayed) Memory allocation changed SRU simple rule changes does not cause snort restart, but binary objects do Binary changes are not that frequent Whether snort would affect it depends on system resources BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 27

Data-Path improvements / Safe Guards Device > Device Management [Edit] > Device tab Automation Application Bypass If traffic enters Snort through the buffer and does not provide a verdict back to LINA within configured threshold, Snort is restarted and a core file is generated BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 28

Show Time

Let s talk about the elephant in the room Large flows are generally related backup, database replication, etc. which usually does not require inspection Sort Analysis > Connections for connection size to find top talkers Once we determine the top talkers, and confirm they can be safely ignored, we create trust rule for the IP conversations. Mitigations IAB / Pre-Filter fast-path BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 30

Data-Path improvements / Safe Guards Snort Fail Open When Busy If the buffer going into Snort is 85% full, new flows will be bypassed Snort Fail Open When Down When Snort goes does due to restart for policy deploy, or for any other reason new flows will be bypassed BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 31

Snort Preserve-Connection When Snort goes down connections with Allow verdict are preserved in LINA Snort does NOT do a mid-session pickup on preserved flows on coming up Does NOT protect against new flows while Snort is down 6.2.0.2 Feature Introduction Can be enabled/disabled from CLISH: configure snort preserve-connection enable/disable BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 32

Best Practices for Deployments (security is our priority)

VPN deployment on FTD: things that you might have missed! Cisco Employee working from home attacker Clear-text / un-authenticated session Should been never been allowed FMC Cisco network The Internet outside NGFW inside Anyconnect (encrypted session) FTP Servers BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 34

VPN deployment recommendations Use Access Control Policy rules to define what VPN traffic should be allowed and be specific as much as possible Enable Anti Spoofing mechanism on FTD interface terminating VPN do NOT enable command sysopt connection permit-vpn this will remove possibility to use Access Control Policy to inspect traffic from the users Where suitable, create Null route for VPN traffic on FTD as when user connect it overwrite routing table with more specific entry (/32) BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 37

Troubleshooting Tools

Process Management - basics FTD Root CLI: ftd-vklauzov:/# pmtool status grep " - " head SFDataCorrelator (normal) - Running 15278 mysqld (system,gui,mysql) - Running 15109 httpsd (system,gui) - Waiting sftunnel (system) - Running 19857 Process name Category Status Process ID BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 39

Process Management - basics FMC Root CLI: root@fmc-2:/# pmtool disablebyid sftunnel root@fmc-2:/# pmtool status grep " - " grep sftunnel sftunnel (system) - User Disabled root@fmc-2:/# pmtool enablebyid sftunnel root@fmc-2:/# pmtool status grep " - " grep sftunnel sftunnel (system) - Running 1720 BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 40

What are main FTD processes and what they do? snort ids_event_processor ids_event_alerter inspects network traffic (pass, block and alert) sends intrusion events to managing device (FMC) sends intrusion events to Syslog or SNMP server wdt-util used for fail-to-wire / hardware bypass sftunnel diskmanager, Pruner Lina Snmpd, ntpd SFDataCorrelator processing events pm (process manager) secure tunnel between managed device and FMC managing disk space and clean up old files Responsible for Firewall functionality like ACL, NAT, Routing etc. SNMP monitoring, responsible for time synchronization responsible for launching and monitoring of all FTD relevant processes and restarting them in case of failure BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 41

Data-path and Snort capture points Detection Engine / Snort 2. > capture-traffic snort inbound/outbound firepower# capture out firepower# capture in 3. 1. data-path inbound DATA-PATH data-path outbound BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 42

Data-path inbound/outbound The Wires Never Lie! Data-path/lina (diagnostic cli): firepower# capture in interface INSIDE match icmp any any trace detail Capture name Interface name protocol Source Destination BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 43

Data-path stop and delete captures Delete packet captures firepower# no capture in Stop packet captures firepower# no capture in interface inside BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 44

Snort Capture - The Wires Never Lie! (1) CLISH: > capture-traffic Options: -s 0 -w capture.pcap icmp and host 172.16.1.17 IP 172.16.1.17 > 20.20.20.100: ICMP echo request,id 24538,seq 1,length 64 Berkeley Packet Filter syntax same as for tcpdump capturing tool -s 0 means snaplength, in other words no limit for packet size -w filename.pcap indicates to which file you want to write output of data captured by specified filter capture is written to /ngfw/var/common/ folder Copy file out to SCP server: file secure-copy <IP address of server> <username> <location where to copy the file> capture.pcap BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 45

Snort Capture - The Wires Never Lie! (2) CLISH: > capture-traffic NON-VLAN TAGGED TRAFFIC Options: -s 0 -v -n -e (icmp and host 172.16.2.11) or (vlan and icmp and host 172.16.2.11) VLAN TAGGED TRAFFIC 00:50:56:b6:0b:33 > 58:97:bd:b9:73:ee, ethertype 802.1Q (0x8100), length 78: vlan 208, p 0, ethertype IPv4, (tos 0x0, ttl 128, id 5366, offset 0, flags [none], proto ICMP (1), length 60) LINA CLI: IN OUT LINA CLI: firepower# sh cap inside 802.1Q vlan#208 P0 172.16.2.11 > 20.20.20.11: icmp: echo request firepower# sh cap outside 172.16.2.11 > 20.20.20.11: icmp: echo request BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 46

Which ACP rule is being evaluated? Tool that provides the Access Control Rule evaluation status for each flow as we receive packets in real time. NGFW debug needs to have specified at least one filtering condition. >system support firewall-engine-debug Please specify an IP protocol: icmp Please specify a client IP address: 172.16.1.17 Please specify a server IP address: 20.20.20.100 Monitoring firewall engine debug messages172.16.1.17-8 > 20.20.20.100-0 1 AS 1 I 44 New session 172.16.1.17-8 > 20.20.20.100-0 1 AS 1 I 44 using HW or preset rule order 2, 'allow and inspect', action Allow and prefilter rule 0 172.16.1.17-8 > 20.20.20.100-0 1 AS 1 I 44 allow action BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 47

Show Time

Access Control Policy Rule Hit Counters > show access-control-config ===================[ ciscolive ]==================== Description : Default Action : Allow Default Policy : Balanced Security and Connectivity Logging Configuration DC : Disabled Beginning : Disabled End : Disabled Rule Hits : 10 Variable Set : Default-Set... (output omitted)... Policy name # watch /usr/local/sf/bin/sfcli.pl show firewall grep "ciscolive\ Rule\:\ Rule Hits " ===================[ ciscolive ]==================== Rule Hits : 10 ------------------[ Rule: allow ]------------------- Rule Hits : 14 Rule name BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 49

ACP Rule Hit Counters FMC WebUI Analysis -> Custom -> Custom Workflows -> Create Custom Workflow and use Table Connection Events Add page and fill in fields like: Access Control Policy, Access Control Rule, Count, Initiator IP, Responder IP Add Table view BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 50

ACP Rule Hit Counters FMC WebUI vs CLISH > show access-control-config ------[ Rule: DNS and icmp ]------ Action : Allow Destination Ports : protocol 6, port 53 protocol 17, port 53 protocol 1 protocol 6, port 80 Logging Configuration DC : Enabled Beginning : Enabled End : Enabled Rule Hits : 28 Variable Set : Default-Set (truncated) Why the hit counters do not match? BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 51

Show Time

I m a trouble-shooter now LINA / Data-Path System Support Trace Capture w/ trace Capture-traffic Firewall-Engine- Debug BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 56

Firepower New Features in 6.2.2.X / 6.2.3

New Signed Software Update/Upgrade images Signed images were introduced in 6.2.1 Signed images are the.rhel.tar files (caution: DO NOT UNTAR THEM!) FTD on platforms 4100 and 9300 series needs to have upgraded FXOS software via Firepower Chassis Manager prior FTD upgrade to 6.2.2 version Platform Current Version Destination Version Package name to be used FMC 6.2.0 6.2.2 Sourcefire_3D_Defense_Center_S3_Upgrade-6.2.2-81.sh FMC 6.2.1 6.2.2 Sourcefire_3D_Defense_Center_S3_Upgrade -6.2.2-81.sh.REL.tar BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 59

Threat Intelligence Director Consumes third-party cyber threat intelligence Requirements: FMC and FTD running 6.2.2 15 GB of memory Protect license (IPv4, IPv6, Domain and URL detection) Malware license (SHA-256 detection) Terminology STIX Structured Threat Intelligence expression TAXII transport mechanism for STIX TID correlation for incident generation is dependent on an exact match! BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 60

TID High-Level Architecture Third-Party Cyber Security Intelligence STIX TAXII Flat files Cisco TID on FMC Syncd.pl Sftunnnel (TCP 8305) Observables NGFW / NGIPS (manage device) Can take up to 20 minutes! BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 61

TID Troubleshooting Observables type IPv4 and IPv6 addresses Domain names URL s SHA-256 hashes File location /ngfw/var/sf/iprep_download /ngfw/var/sf/sidns_download /ngfw/var/sf/siurl_download /ngfw/var/sf/sifile_download BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 62

API bulk rule access insertion, yay! Old behavior: one AC rule can be imported at the time New behavior: we can insert up to 1000 rules within same API request! We can insert rules at specific location (rule number or within specific category/section) After rule insertion, other rules are automatically reordered Rest API can handle if other user is already modifying the same rule set When no position of the rule is defined, it goes to the end of ACP BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 63

Serviceability requests <6.2.3> CSCvd94909 - Generate backup from FMC CLI Motivation: In case of FMC web interface is down, there was no way to take current snapshot/backup of the system via CLI. BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 64

Serviceability requests <6.2.3> User Identity mappings Display information about user vklauzov: user_map_query.pl -u <username> Display information about user based on IP address: user_map_query.pl -i <IP address> Display manual for the script: user_map_query.pl --help BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 65

Exciting Real-World Use-Cases

Real World Scenario Slow files transfers through FTD using FTP poor performance with default IPS policy baseline for FTP traffic BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 67

Tuning IPS rules #(TAC tip & trick) Use case: poor performance with default IPS policy baseline for FTP traffic Simplified topology: client (Windows 10) ---1Gbps --- FTD 9300 ---1Gbps --- server (Windows 10) Performance measurement results with default policy: ~ 380 Mbps Performance measurement after IPS rule tuning: ~ 970 Mbps BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 68

Full example: performance numbers from field/lab testing Mode Protocol Configuration Throughput Transpar ent FTP (Filezilla 3.29.0 ) Pre-filter policy with Fast-path rule for TCP ports 20 and 21 ~979 Mbps Access Control Policy, Allow rule for TCP ports 20 and 21, IPS connectivity over Security ~650 Mbps Access Control Policy, Allow rule for TCP ports 20 and 21, IPS Balanced Security and Connectivity ~380 Mbps Access Control Policy, Allow rule for TCP ports 20 and 21, IPS Security over Connectivity ~340 Mbps BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 69

Full example: performance numbers from field/lab testing Mode Protocol Configuration Throughput Transpar ent FTP (Filezilla 3.29.0 ) Access Control Policy, Allow rule for TCP ports 20 and 21, IPS Maximum detection ~320 Mbps Access Control Policy, Allow rule for TCP ports 20 and 21, IPS tuned (base no rules active + 51 active rules) Filter used: ftp metadata:"security-ips drop" Access Control Policy, Allow rule for TCP ports 20 and 21, IPS tuned (base no rules active + 51 active rules) Filter used: ftp metadata:"security-ips drop" ~971 Mbps ~800 Mbps + File policy with application protocol FTP (detect all file types and block malware executable s with local malware analysis) BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 70

Low IPS performance? rule it out by FTD rule profiling! Edit /ngfw/var/sf/detection_engines/<uuid>/ advanced/perf_monitor.conf config profile_rules: print all, sort avg_ticks, filename /ngfw/var/log/profiling-rules.log config profile_preprocs: print all, sort avg_ticks, filename /ngfw/var/log/profiling-preprocs.log Restart Snort pmtool restartbytype snort Start rule profiling > system support run-rule-profiling BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 71

Reassembly cost Posted throughput ratings for the Firepower appliances are usually rated at 1518 bytes packets. Smaller packets results in more processing. 1MB of traffic with 1518 bytes/packets = ~ 658 packets 1MB of traffic with 400 bytes/packet = ~ 2500 packets Every packet header must be evaluated and the packet has to be placed into the buffer for re-assembly. The larger number of packets to process requires more CPU time. BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 74

Sizing your NGFW / NGIPS Throughput considerations For Your Reference Number of Snort instances per FTD platform Platform Snort Instances Platform Snort Instances Platform Snort instances Firepower 2110 6 Firepower 4110 11 Firepower 9300 SM-24 24 Firepower 2120 10 Firepower 4120 24 Firepower 9300 SM-36 36 Firepower 2130 14 Firepower 4140 36 Firepower 9300 SM-44 46 Firepower 2140 26 Firepower 4150 48 - - Enabling File-Inspection will change these values > pmtool show affinity BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 75

A little bit of automation to save hours of manual work!!! TAC has Your back! Show Time

Closing

Why Security Beta Programs? Influence Product Roadmap Bugs Fixed for Release Free Product Training Access to Product Teams Enroll today! http://cs.co/security-beta-nomination I feel a personal attachment to your company through the Beta testing we do. you guys are listening to us and you don t realize how rare that is. - Government Insurance Company BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 81

Complete your online session evaluation Give us your feedback to be entered into a Daily Survey Drawing. Complete your session surveys through the Cisco Live mobile app or on www.ciscolive.com/us. Don t forget: Cisco Live sessions will be available for viewing on demand after the event at www.ciscolive.com/online. 1 2 3 4 5 BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 82

Continue your education Demos in the Cisco campus Walk-in self-paced labs Meet the engineer 1:1 meetings Related sessions BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 83

13:30 16:00 13:30 13:30 08:00 13:00 10:30 08:00 08:00 08:00 08:00 Cisco Firepower Sessions: Building Blocks Monday Tuesday Wednesday Thursday BRKSEC-2031 ASA Fleet Management at Scale BRKSEC-2064 NGFWv and ASAv in Public Cloud (AWS and Azure) BRKSEC-3020 Troubleshooting ASA Firewalls BRKSEC-3032 NGFW Clustering Deep Dive BRKSEC-2050 Firepower NGFW Internet Edge Deployment Scenarios BRKSEC-3455 Dissecting Firepower Design & Troubleshooting BRKSEC-3035 Firepower Platform Deep Dive We are here! BRKSEC-2050 BRKSEC-2066 Firepower NGFW Internet Edge Deployment Optimizing Your Firepower/FTD Deployment BRKSEC-2020 Firepower Deployment Data Center & Enterprise Network Edge BRKSEC-2058 Deep Dive into Firepower Manager BRKSEC-3455 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 84

Thank you