Dissecting Firepower-FTD & Firepower-Services Design & Troubleshooting

Transcription

1

2 Dissecting Firepower-FTD & Firepower-Services Design & Troubleshooting Veronika Klauzova BRKSEC-3455

3 Agenda Introduction Updated FTD Packet Flow Data-Path Improvements Best Practices for Deployments Troubleshooting Tools Firepower New Features Exciting Real-World Use-Cases Conclusions BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 3

4 Cisco Webex Teams Questions? Use Cisco Webex Teams (formerly Cisco Spark) to chat with the speaker after the session How Find this session in the Cisco Events App Click Join the Discussion Install Webex Teams or go directly to the team space Enter messages/questions in the team space Webex Teams will be moderated by the speaker until June 18, cs.co/ciscolivebot#brksec-3455 BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 4

5 Your presenter for today Firepower engineer Passionate Linux Admin Love to explore Cisco technologies Veronika Klauzova BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 5

6 Hardware & Software Review

7 NGFW evolution BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 7

8 What platforms can run FTD Software ASA 5500X-Series (5506X-5555X with SSD) BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 8

9 What platforms can run FTD Software Firepower 2100 series BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 9

10 What platforms can run FTD Software Power Console MGMT 8 x optic SFP+ ports Front view 2 x 2.5 SSD Bays Rear view 2x optional NetMods 2 x Power Supply Module Bays 6 x Hot-Swap Fans units Firepower 4100 series BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 10

11 BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 11

12 BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 12

13 BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 13

14 Updated FTD Packet Flow

15 Firepower Threat Defense high level DETECTION ENGINE / Snort Packet Data Transport System (PDTS) DATA-PATH / LINA FXOS BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 15

16 Firepower 2100 architecture overview BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 16

17 Firepower 9300/4100 architecture overview BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 17

18 FTD Packet-Flow Detection Engine / Snort RX YES Lina rule-id matched PDTS DAQ Ingress Interface Existing Conn NO Egress Interface Pre-Filter L3/L4 ACL ALG checks NAT L3, L2 hops VPN Decrypt QoS, VPN Encrypt Data-Path / LINA TX BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 18

19 Detection Engine/ Snort - Architecture Snort SNORT Decode SI (IPRep) Frag3 (IP Defrag) Stream5 (Reassembly) AppID DAQ LINA / Data-Path Protocol/Application Preprocessors FTP/TELNET HTTP DCE/RPC DNS SIP SSH SSL SunRPC POP IMAP SMTP Others (non-standard) File Policy QoS Classify (FTD only) ACP Eval IPS Policy Before ACP Rules SI (DNS/URL) Specific Threat Detection (Pre-processors) Back Orifice Portscan Rate-Based Attack Sensitive Data IPS Rule Eval BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 19

20 Data-Path Improvements

21 Snort Restart & Reload Architecture BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 21

22 Snort reload instead of restart As of following changes would not cause Snort to be restarted This applies to all FTD devices managed by FMC Policy changes URL Application ID Intrusion Policy NAP policy Simple SRU update Security Intelligence Policy action Refer to URL categories for the first time in AC rules or remove all existing references Turn on/off Application ID Add or Delete Intrusion Polices in AC rules, or Edit Intrusion Policy Attach a NAP policy for the first time to AC Policy Typical rule updates without Shared Object (SO) / binary rule updates Changes to Whitelist/Blacklist of URL, DNS entries BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 22

23 Snort reload or restart during policy deployment? BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 23

24 Deployment changes causing interruption SSL VDB version update User Identity Network discovery (http, ftp, msdn) Update of SRU version Max MTU Snort/DAQ version update System Upgrade BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 24

25 Minimalize network disruption during policy deployment Snort restart behavior depends on Advance settings in Access Control Policy TAC highly recommend to enable: Inspect traffic during policy apply = Yes Without this option Snort always restarts during policy deployment BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 25

26 Show Time

27 Other snort major updates Changes to application detectors display warnings Break HA operation restart snort/s (warning displayed) Memory allocation changed SRU simple rule changes does not cause snort restart, but binary objects do Binary changes are not that frequent Whether snort would affect it depends on system resources BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 27

28 Data-Path improvements / Safe Guards Device > Device Management [Edit] > Device tab Automation Application Bypass If traffic enters Snort through the buffer and does not provide a verdict back to LINA within configured threshold, Snort is restarted and a core file is generated BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 28

29 Show Time

30 Let s talk about the elephant in the room Large flows are generally related backup, database replication, etc. which usually does not require inspection Sort Analysis > Connections for connection size to find top talkers Once we determine the top talkers, and confirm they can be safely ignored, we create trust rule for the IP conversations. Mitigations IAB / Pre-Filter fast-path BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 30

31 Data-Path improvements / Safe Guards Snort Fail Open When Busy If the buffer going into Snort is 85% full, new flows will be bypassed Snort Fail Open When Down When Snort goes does due to restart for policy deploy, or for any other reason new flows will be bypassed BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 31

32 Snort Preserve-Connection When Snort goes down connections with Allow verdict are preserved in LINA Snort does NOT do a mid-session pickup on preserved flows on coming up Does NOT protect against new flows while Snort is down Feature Introduction Can be enabled/disabled from CLISH: configure snort preserve-connection enable/disable BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 32

33 Best Practices for Deployments (security is our priority)

34 VPN deployment on FTD: things that you might have missed! Cisco Employee working from home attacker Clear-text / un-authenticated session Should been never been allowed FMC Cisco network The Internet outside NGFW inside Anyconnect (encrypted session) FTP Servers BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 34

35 Is your network protected? BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 35

36 Show Time BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 36

37 VPN deployment recommendations Use Access Control Policy rules to define what VPN traffic should be allowed and be specific as much as possible Enable Anti Spoofing mechanism on FTD interface terminating VPN do NOT enable command sysopt connection permit-vpn this will remove possibility to use Access Control Policy to inspect traffic from the users Where suitable, create Null route for VPN traffic on FTD as when user connect it overwrite routing table with more specific entry (/32) BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 37

38 Troubleshooting Tools

39 Process Management - basics FTD Root CLI: ftd-vklauzov:/# pmtool status grep " - " head SFDataCorrelator (normal) - Running mysqld (system,gui,mysql) - Running httpsd (system,gui) - Waiting sftunnel (system) - Running Process name Category Status Process ID BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 39

40 Process Management - basics FMC Root CLI: root@fmc-2:/# pmtool disablebyid sftunnel root@fmc-2:/# pmtool status grep " - " grep sftunnel sftunnel (system) - User Disabled root@fmc-2:/# pmtool enablebyid sftunnel root@fmc-2:/# pmtool status grep " - " grep sftunnel sftunnel (system) - Running 1720 BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 40

41 What are main FTD processes and what they do? snort ids_event_processor ids_event_alerter inspects network traffic (pass, block and alert) sends intrusion events to managing device (FMC) sends intrusion events to Syslog or SNMP server wdt-util used for fail-to-wire / hardware bypass sftunnel diskmanager, Pruner Lina Snmpd, ntpd SFDataCorrelator processing events pm (process manager) secure tunnel between managed device and FMC managing disk space and clean up old files Responsible for Firewall functionality like ACL, NAT, Routing etc. SNMP monitoring, responsible for time synchronization responsible for launching and monitoring of all FTD relevant processes and restarting them in case of failure BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 41

42 Data-path and Snort capture points Detection Engine / Snort 2. > capture-traffic snort inbound/outbound firepower# capture out firepower# capture in data-path inbound DATA-PATH data-path outbound BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 42

43 Data-path inbound/outbound The Wires Never Lie! Data-path/lina (diagnostic cli): firepower# capture in interface INSIDE match icmp any any trace detail Capture name Interface name protocol Source Destination BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 43

44 Data-path stop and delete captures Delete packet captures firepower# no capture in Stop packet captures firepower# no capture in interface inside BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 44

45 Snort Capture - The Wires Never Lie! (1) CLISH: > capture-traffic Options: -s 0 -w capture.pcap icmp and host IP > : ICMP echo request,id 24538,seq 1,length 64 Berkeley Packet Filter syntax same as for tcpdump capturing tool -s 0 means snaplength, in other words no limit for packet size -w filename.pcap indicates to which file you want to write output of data captured by specified filter capture is written to /ngfw/var/common/ folder Copy file out to SCP server: file secure-copy <IP address of server> <username> <location where to copy the file> capture.pcap BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 45

46 Snort Capture - The Wires Never Lie! (2) CLISH: > capture-traffic NON-VLAN TAGGED TRAFFIC Options: -s 0 -v -n -e (icmp and host ) or (vlan and icmp and host ) VLAN TAGGED TRAFFIC 00:50:56:b6:0b:33 > 58:97:bd:b9:73:ee, ethertype 802.1Q (0x8100), length 78: vlan 208, p 0, ethertype IPv4, (tos 0x0, ttl 128, id 5366, offset 0, flags [none], proto ICMP (1), length 60) LINA CLI: IN OUT LINA CLI: firepower# sh cap inside 802.1Q vlan#208 P > : icmp: echo request firepower# sh cap outside > : icmp: echo request BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 46

47 Which ACP rule is being evaluated? Tool that provides the Access Control Rule evaluation status for each flow as we receive packets in real time. NGFW debug needs to have specified at least one filtering condition. >system support firewall-engine-debug Please specify an IP protocol: icmp Please specify a client IP address: Please specify a server IP address: Monitoring firewall engine debug messages > AS 1 I 44 New session > AS 1 I 44 using HW or preset rule order 2, 'allow and inspect', action Allow and prefilter rule > AS 1 I 44 allow action BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 47

48 Show Time

49 Access Control Policy Rule Hit Counters > show access-control-config ===================[ ciscolive ]==================== Description : Default Action : Allow Default Policy : Balanced Security and Connectivity Logging Configuration DC : Disabled Beginning : Disabled End : Disabled Rule Hits : 10 Variable Set : Default-Set... (output omitted)... Policy name # watch /usr/local/sf/bin/sfcli.pl show firewall grep "ciscolive\ Rule\:\ Rule Hits " ===================[ ciscolive ]==================== Rule Hits : [ Rule: allow ] Rule Hits : 14 Rule name BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 49

50 ACP Rule Hit Counters FMC WebUI Analysis -> Custom -> Custom Workflows -> Create Custom Workflow and use Table Connection Events Add page and fill in fields like: Access Control Policy, Access Control Rule, Count, Initiator IP, Responder IP Add Table view BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 50

51 ACP Rule Hit Counters FMC WebUI vs CLISH > show access-control-config [ Rule: DNS and icmp ] Action : Allow Destination Ports : protocol 6, port 53 protocol 17, port 53 protocol 1 protocol 6, port 80 Logging Configuration DC : Enabled Beginning : Enabled End : Enabled Rule Hits : 28 Variable Set : Default-Set (truncated) Why the hit counters do not match? BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 51

52 Capture With Trace GUI Quickly Identify where in the data-path the traffic is impacted BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 52

53 Show Time

54 CLI Analyzer Contextual help and highlighting Embedded Intelligence File Analysis BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 54

55 Show Time

56 I m a trouble-shooter now LINA / Data-Path System Support Trace Capture w/ trace Capture-traffic Firewall-Engine- Debug BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 56

57 Deep-dive: FTD troubleshooting/debug tools BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 57

58 Firepower New Features in X / 6.2.3

59 New Signed Software Update/Upgrade images Signed images were introduced in Signed images are the.rhel.tar files (caution: DO NOT UNTAR THEM!) FTD on platforms 4100 and 9300 series needs to have upgraded FXOS software via Firepower Chassis Manager prior FTD upgrade to version Platform Current Version Destination Version Package name to be used FMC Sourcefire_3D_Defense_Center_S3_Upgrade sh FMC Sourcefire_3D_Defense_Center_S3_Upgrade sh.REL.tar BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 59

60 Threat Intelligence Director Consumes third-party cyber threat intelligence Requirements: FMC and FTD running GB of memory Protect license (IPv4, IPv6, Domain and URL detection) Malware license (SHA-256 detection) Terminology STIX Structured Threat Intelligence expression TAXII transport mechanism for STIX TID correlation for incident generation is dependent on an exact match! BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 60

61 TID High-Level Architecture Third-Party Cyber Security Intelligence STIX TAXII Flat files Cisco TID on FMC Syncd.pl Sftunnnel (TCP 8305) Observables NGFW / NGIPS (manage device) Can take up to 20 minutes! BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 61

62 TID Troubleshooting Observables type IPv4 and IPv6 addresses Domain names URL s SHA-256 hashes File location /ngfw/var/sf/iprep_download /ngfw/var/sf/sidns_download /ngfw/var/sf/siurl_download /ngfw/var/sf/sifile_download BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 62

63 API bulk rule access insertion, yay! Old behavior: one AC rule can be imported at the time New behavior: we can insert up to 1000 rules within same API request! We can insert rules at specific location (rule number or within specific category/section) After rule insertion, other rules are automatically reordered Rest API can handle if other user is already modifying the same rule set When no position of the rule is defined, it goes to the end of ACP BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 63

64 Serviceability requests <6.2.3> CSCvd Generate backup from FMC CLI Motivation: In case of FMC web interface is down, there was no way to take current snapshot/backup of the system via CLI. BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 64

65 Serviceability requests <6.2.3> User Identity mappings Display information about user vklauzov: user_map_query.pl -u <username> Display information about user based on IP address: user_map_query.pl -i <IP address> Display manual for the script: user_map_query.pl --help BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 65

66 Exciting Real-World Use-Cases

67 Real World Scenario Slow files transfers through FTD using FTP poor performance with default IPS policy baseline for FTP traffic BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 67

68 Tuning IPS rules #(TAC tip & trick) Use case: poor performance with default IPS policy baseline for FTP traffic Simplified topology: client (Windows 10) ---1Gbps --- FTD Gbps --- server (Windows 10) Performance measurement results with default policy: ~ 380 Mbps Performance measurement after IPS rule tuning: ~ 970 Mbps BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 68

69 Full example: performance numbers from field/lab testing Mode Protocol Configuration Throughput Transpar ent FTP (Filezilla ) Pre-filter policy with Fast-path rule for TCP ports 20 and 21 ~979 Mbps Access Control Policy, Allow rule for TCP ports 20 and 21, IPS connectivity over Security ~650 Mbps Access Control Policy, Allow rule for TCP ports 20 and 21, IPS Balanced Security and Connectivity ~380 Mbps Access Control Policy, Allow rule for TCP ports 20 and 21, IPS Security over Connectivity ~340 Mbps BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 69

70 Full example: performance numbers from field/lab testing Mode Protocol Configuration Throughput Transpar ent FTP (Filezilla ) Access Control Policy, Allow rule for TCP ports 20 and 21, IPS Maximum detection ~320 Mbps Access Control Policy, Allow rule for TCP ports 20 and 21, IPS tuned (base no rules active + 51 active rules) Filter used: ftp metadata:"security-ips drop" Access Control Policy, Allow rule for TCP ports 20 and 21, IPS tuned (base no rules active + 51 active rules) Filter used: ftp metadata:"security-ips drop" ~971 Mbps ~800 Mbps + File policy with application protocol FTP (detect all file types and block malware executable s with local malware analysis) BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 70

71 Low IPS performance? rule it out by FTD rule profiling! Edit /ngfw/var/sf/detection_engines/<uuid>/ advanced/perf_monitor.conf config profile_rules: print all, sort avg_ticks, filename /ngfw/var/log/profiling-rules.log config profile_preprocs: print all, sort avg_ticks, filename /ngfw/var/log/profiling-preprocs.log Restart Snort pmtool restartbytype snort Start rule profiling > system support run-rule-profiling BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 71

72 Low IPS performance? rule it out by FTD rule profiling! BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 72

73 Performance graphs from the WebUI BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 73

74 Reassembly cost Posted throughput ratings for the Firepower appliances are usually rated at 1518 bytes packets. Smaller packets results in more processing. 1MB of traffic with 1518 bytes/packets = ~ 658 packets 1MB of traffic with 400 bytes/packet = ~ 2500 packets Every packet header must be evaluated and the packet has to be placed into the buffer for re-assembly. The larger number of packets to process requires more CPU time. BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 74

75 Sizing your NGFW / NGIPS Throughput considerations For Your Reference Number of Snort instances per FTD platform Platform Snort Instances Platform Snort Instances Platform Snort instances Firepower Firepower Firepower 9300 SM Firepower Firepower Firepower 9300 SM Firepower Firepower Firepower 9300 SM Firepower Firepower Enabling File-Inspection will change these values > pmtool show affinity BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 75

76 Real World Scenario Unable to deploy policy Hundreds of sensors affected! BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 76

77 A little bit of automation to save hours of manual work!!! TAC has Your back! Show Time

78 BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 78

79 Real World Scenario HARDWARE ERROR ON LCD BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 79

80 Closing

81 Why Security Beta Programs? Influence Product Roadmap Bugs Fixed for Release Free Product Training Access to Product Teams Enroll today! I feel a personal attachment to your company through the Beta testing we do. you guys are listening to us and you don t realize how rare that is. - Government Insurance Company BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 81

82 Complete your online session evaluation Give us your feedback to be entered into a Daily Survey Drawing. Complete your session surveys through the Cisco Live mobile app or on Don t forget: Cisco Live sessions will be available for viewing on demand after the event at BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 82

83 Continue your education Demos in the Cisco campus Walk-in self-paced labs Meet the engineer 1:1 meetings Related sessions BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 83

84 13:30 16:00 13:30 13:30 08:00 13:00 10:30 08:00 08:00 08:00 08:00 Cisco Firepower Sessions: Building Blocks Monday Tuesday Wednesday Thursday BRKSEC-2031 ASA Fleet Management at Scale BRKSEC-2064 NGFWv and ASAv in Public Cloud (AWS and Azure) BRKSEC-3020 Troubleshooting ASA Firewalls BRKSEC-3032 NGFW Clustering Deep Dive BRKSEC-2050 Firepower NGFW Internet Edge Deployment Scenarios BRKSEC-3455 Dissecting Firepower Design & Troubleshooting BRKSEC-3035 Firepower Platform Deep Dive We are here! BRKSEC-2050 BRKSEC-2066 Firepower NGFW Internet Edge Deployment Optimizing Your Firepower/FTD Deployment BRKSEC-2020 Firepower Deployment Data Center & Enterprise Network Edge BRKSEC-2058 Deep Dive into Firepower Manager BRKSEC Cisco and/or its affiliates. All rights reserved. Cisco Public 84

85 Thank you

86