NfSen and NFDUMP 16th TF-CSIRT Meeting Sept 15th 2005, Lisbon Peter Haag

Transcription

1 NfSen and NFDUMP 16th TF-CSIRT Meeting Sept 15th 2005, Lisbon Peter Haag 2005 SWITCH NfSen/nfdump What I am going to present: Review: What are NfSen and nfdump. The Tools in Action. Plugins. Outlook - what s next SWITCH 2

2 NfSen/nfrdump What is NetFlow? NetFlow is a traffic monitoring technology developed by Cisco Networks. Flowa are unidirectional and contain connection related data such as: Source and destination IP address. Source and destination port. Source and destination AS. Level3 protocol, ToS byte, TCP flags. Logical input and output interfaces. Bytes and packet counters SWITCH 3 NfSen/nfdump With NfSen you can: Display the network traffic situation. Easily navigate through the netflow data. (time based) Drill down from overview to the details down to the specific flows. Profile/monitor specific Networks/Hosts and events. Extensively filtering netflow data. Analyse the netflow data using the web based as well as the command line based interface. Create lots of Top N statistics. Post process the netflow data for reporting and alerting SWITCH 4

3 NfSen 2005 SWITCH 5 NfSen 2005 SWITCH 6

4 NfSen 2005 SWITCH 7 NfSen 2005 SWITCH 8

5 NfSen/nfdump Putting all together: From: To: Summary: monitored:..... Web Front-end Post Processing CLI softflowd pfflowd Input netflow v5, v7 Periodic Update Tasks & Plugins nfdump Backend 2005 SWITCH 9 nfdump nfdump 1.4: Main Netflow Processing Tool: Stores netflow data in time sliced files. CMD line based tool comparable to tcpdump. Written in C! fast. Supports netflow format v5 and v7. Powerful pcap like filter syntax: ( tcp and dst net /16 and src port > 1024 and bytes < 600 ) or ( bps > 1k and Flexible aggregation. Efficient filter engine: > 6 Mio flows/s on 3GHz Intel. Lots of fast statistics Top N Anonymizing of IP addresses. ( Crypto-Pan ) 2005 SWITCH 10

6 nfdump Nfdump: Statistics Processing: Top N statistics about Flow records Any IP addr, src IP addr, dst IP addr Any Port, src Port, dst Port Any AS, src as, dst as Ordered by: Number of Flows Number of Packets Number of Bytes pps ( packets per second ) bps ( bits per second ) bpp ( bytes per packet ) Example: nfdump -r -s record/flows -s ip/packets/bytes/bps -s dstport/flows 2005 SWITCH 11 nfdump nfdump -r /nfcapd n 20 -s ip/bps Top 20 IP Addr ordered by bps: Date first seen Duration IP Addr Flows Packets Bytes pps bps bpp :56: M 2.3 G M :56: M 2.4 G M :56: M 2.0 G M :54: M 2.6 G M :55: M :53: M 3.0 G M :53: M 2.8 G M :58: M :54: M 2.1 G M :56: M :54: M 1.5 G M :52: M 2.1 G M :54: M 1.6 G M :54: M 1.3 G M :54: M 1.4 G M :59: M M :54: M 1.1 G M :54: M 1.0 G M :57: M M :57: M M 979 IP addresses anonymized Time window: Sep :39:52 - Sep :59:56 Flows analysed: matched: , Bytes read: SWITCH 12

7 nfdump nfdump -r /nfcapd o extended -n 20 -s record/pps Top 20 flows ordered by pps: Date flow start Duration Proto Src IP Addr:Port Dst IP Addr:Port Flags Tos Packets Bytes pps bps Bpp Flows :01: TCP :21 -> :53998.AP M :01: TCP :21 -> :1549.AP M :02: TCP :80 -> :51209.A M :04: TCP : > :42965.AP M :00: TCP :80 -> :32887.A...F M :02: TCP :80 -> :1076.AP M :56: TCP :80 -> :49412.AP..F M :04: TCP : > :56273.AP M :02: TCP :80 -> :1909.A M :00: TCP : > :80.A M :03: TCP :80 -> :3927.AP..F M :58: UDP :53 -> : M :02: TCP :80 -> :16066.A M :02: TCP :80 -> :51090.AP M :02: TCP :80 -> :1638.AP M :01: TCP :80 -> :1394.AP M :03: TCP :80 -> :38613.AP M :02: TCP :80 -> :32895.AP M :01: TCP :80 -> :49764.A M :04: TCP :80 -> :3403.A M IP addresses anonymized Time window: Sep :55:02 - Sep :04:59 Flows analysed: matched: 12128, Bytes read: Sys: 0.054s flows/second: Wall: 0.014s flows/second: SWITCH 13 NfSen in Action Overview - Details - Flows 2005 SWITCH 14

8 NfSen in Action Overview - Details - Flows 2005 SWITCH 15 NfSen in Action Overview - Details - Flows 2005 SWITCH 16

9 NfSen in Action Overview - Details - Flows 2005 SWITCH 17 NfSen Profiles: A profile is a specific view on the netflow data with nfdump filters applied. The profile applies to the graphical as well as to the numerical view. Profiles can be created from data in the past. ( static ) Profiles can be created from incoming data ( continuous ) Any views or processing options are available SWITCH 18

10 NfSen/nfdump Example Profiles: Filter: ( udp or tcp ) and port 53 Filter: bytes < 100 Filters may be as complex as the the filter syntax of nfdump allows. Example: ((src net /16 and src port > 1024 ) or dst host and dst port 80) and packets > 1000 and pps > SWITCH 19 NfSen/nfdump Analyse Incident: 2005 SWITCH 20

11 NfSen/nfdump - Plugins Plugins - what for? Monitoring and alerting. Track for known botnet masters and send notifications. Track for possible scanners or DoS attacks, not necessarily visible in the graph. Port Tracking. Backend Plugins are: Simple Perl modules hooked into the NfSen backend. Automatically called at regular 5 Min intervals. Frontend Plugins are: Simple PHP modules hooked into NfSen frontend. Called by selecting the tab SWITCH 21 NfSen/nfdump NfSen Plugins: Web Front-end Post Processing Frontend Plugins Backend Plugins Periodic Update 2005 SWITCH 22

12 NfSen/nfdump Backend Plugins: = ( ['live', 'CatchDos'], ); 1; package CatchDos; use strict; sub Init { Init plugin } End of Init nfsen.conf Report sub run { my $profile = shift; my $timeslot = shift; Notification.pm } End of run Backend Plugin Runs automatically every 5 min output 2005 SWITCH 23 NfSen/nfdump Example Candidates for scanning activities:. Define a nice filter: We like to see flows containing more than xxx packets my $limit = 6000 ; my $nf_filter = duration < 3500 and packets < 3 and bpp < 100 and src as 559 ; Periodic function input: profilename timeslot. Format yyyymmddhhmm e.g sub run { my $profile = shift; my $timeslot = shift; syslog('debug', "CatchScanners run: Profile: $profile, Time: $timeslot"); my %profileinfo = NfSen::ReadProfile($profile); my $netflow_sources = $profileinfo{'sourcelist'}; process all sources of this profile at once = `$nfdump -M $PROFILEDATADIR/$profile/$netflow_sources -r nfcapd.$timeslot -a -A srcip,dstport -l $limit -f $nf_filter`; Process the output and notify the duty team ( IP addresses anonymised ) 2005 SWITCH 24

13 NfSen/nfdump Example Candidates for scanning activities: The plugin processes data with nfdump arguments and filter: From: To: Subject: Scanners Packet limit: > 6000 packets Date flow start Duration Proto Src IP Addr:Port Dst IP Addr:Port Packets Bytes Flows :29: TCP :0 -> : :29: TCP :0 -> : :29: UDP :0 -> : M :29: TCP :0 -> : :32: TCP :0 -> : :32: TCP :0 -> : M :29: TCP :0 -> : M :29: TCP :0 -> : :29: TCP :0 -> : :29: TCP :0 -> : :29: TCP :0 -> : :29: TCP :0 -> : :29: TCP :0 -> : :29: TCP :0 -> : :29: TCP :0 -> : :29: TCP :0 -> : M :29: TCP :0 -> : :29: TCP :0 -> : ( IP addresses anonymised ) 2005 SWITCH 25 NfSen/nfdump Backend / Frontend Plugins: = ( ['live', 'CatchDos'], ); 1; package CatchDos; use strict; sub Init { Init plugin } End of Init sub run { my $profile = shift; my $timeslot = shift; nfsen.conf /* Port Tracker * Plugin */ function PortTracker_Run($plugi) { $portinfo = GetTopN($plugin_id, $_SESSION["${plugin_id} } // End of PortTracker_Run } End of run Backend Plugin Runs automatically every 5 min output Frontend Plugin 2005 SWITCH 26

14 NfSen Plugins: Port Tracker 2005 SWITCH 27 NfSen Planned Plugin: Host behaviour based worm detection: Result of a PhD network security research work in the context of the DDoSVax project at Swiss Federal Institute of Technology Zurich: Idea: Infected hosts show a different behaviour and can be put into different classes: Traffic class: Worm infected hosts tend to send considerably more traffic than they receive. Responder class: If many more hosts start to answer requests, they probably are victims of a worm. Connector class: Worm infected hosts typically open many connections SWITCH 28

15 NfSen DDoS Vax Host behaviour based worm detection: Example: MyDoom.A Traffic (T) T"C T"R"C Connector (C) T"R R"C Responder (R) no class Most interesing for worm detection are cardinalities of class combinations SWITCH 29 NfSen/nfdump SWITCH: Server: 2 x 3GHz 2GB Ram. Debian Linux Kernel TB ( 2TB + 1TB ) AXUS Disk Raid XFS file system. Gigabit Ethernet interfaces. 5min workload avg. ca. 5%. 25GB Netflow data / day. About 41 days of netflow data available SWITCH 30

16 NfSen/nfdump Next Steps - Todo list: NfSen: More Plugins: Alerting Plugin. Anomaly Detection Plugin. ( DDoS VAX Project ETH Zürich ) Portocol, AS Tracker.. nfdump: Netflow v9 IPv6 Related filters: Worm Footprint Tracking first { dst ip <A> dst port 445 bytes > 600 } then { src ip <A> and dst ip and dst port 80 } Post Processing Filters etc SWITCH 31 NfSen/nfdump Summary: Good and flexible tools for all sort of netflow tasks. Network monitoring. Incident Handling. All sort of tracking Open Source Tools under BSD License. Cmd line tool: nfdump Written in C. Runs on most *nix. Tested on Linux Kernel 2.4.* and 2.6.*, FreeBSD, OpenBSD, Solaris. Available at Web based frontend: NfSen Written in PHP and Perl. Extendable using plugins. Available at Possible candidate for the toolset in GN2/JRA SWITCH 32

17 NFSEN 2005 SWITCH 33