Understanding network traffic through Intraflow data

Transcription

1 Understanding network traffic through Intraflow data David McGrew and Blake Anderson FloCon 2016

2 Exploring threat data features at scale pcap pcap2flow json Offline joy pcap2flow json Online exporter collecto r json

3 Flow Monitoring srcip, dstip, srcport, dstport, prot, starttime, stoptime, numbytes, numpackets Observation Export Collection Analysis Storage Observation Observation

4 I need to understand traffic even when it is encrypted I need to understand all the traffic in my network, not just traffic that passes through a security appliance Known threats and malware Evasive applications and tunnels TLS, SSH, and other encrypted traffic (on any port) Monitoring internal traffic Forensics Crypto usage audit

5 Flow Monitoring srcip, dstip, srcport, dstport, prot, starttime, stoptime, numbytes, numpackets Observation Export Collection Analysis Storage Observation Observation New Data Features

6 Intraflow data Information about events or data inside of flows that can be conveniently collected, stored, and analyzed within a flow monitoring framework

7 Intraflow data Economical observation Unidirectional Minimal computation Small snaplen Application/protocol independence Compactness Observation Transmission and storage Composability

8 Architecture Flow Records Classifier

9 Training architecture Malware Detonation Malware Records Training Classifier Benign Records

10 New Telemetry Data Features Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10

11 Enhanced Telemetry Data Types SPLT Sequence of Packet Lengths and Arrival Times src dst Byte Distribution Relative frequency for each byte in a flow Byte Entropy Initial Data Packet

12 Sequence of Packet Lengths and Times Client packets src dst Server packets Time

13 Byte Distribution H T T P / O K f 31 2e f 4b

14 Byte Distribution H T T P / O K f 31 2e f 4b 1

15 Byte Distribution H T T P / O K f 31 2e f 4b 1 1

16 Byte Distribution H T T P / O K f 31 2e f 4b 2 1

17 Byte Distribution H T T P / O K f 31 2e f 4b 1 2 1

18 Byte Distribution for different encodings

19 JSON flow data Conventional flow data Intraflow data Extracted parameters Easy to use with data analytics and machine learning tools

20 Initial Data Packet SYN SYN ACK ACK Data Data Data Data

21 Initial Data Packet SYN SYN ACK IDP ACK Data Data Data Data

22 Experimental results Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22

23 Performance CPU: Intel 2.70GHz 17% utilization at 1.0e7 bytes/sec, 1.2e4 packets/sec Approx 870 Mbits/sec at full utilization RAM: 8Gbyte 2.7% utilization (216 Mbyte) Byte Distribution Everything else Flow lookup, alloc, init

24 Detecting malware with SPLT and Byte Distribution Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26

25 Analytics User Interface

26 Malware Classification

27 Classifying flows as malicious/benign L1-logistic regression SPLT + 5-tuple L1-logistic regression SPLT + 5-tuple + BD

28 Classifying flows as malicious/benign L1-logistic regression SPLT + 5-tuple 160 non-zero parameters 0.01 FDR: 51.11% Total Accuracy: 98.44% L1-logistic regression SPLT + 5-tuple + BD 128 non-zero parameters 0.01 FDR: 98.92% Total Accuracy: 99.81%

29 Intraflow data Economical observation Unidirectional Minimal computation Small snaplen Application/protocol independence Compactness Observation Transmission and storage Composability SPLT 10 packets 10 bytes 10 bytes Byte Distribution No 256 bytes 256 bytes

30 Intraflow data Economical observation Unidirectional Minimal computation Small snaplen Application/protocol independence Compactness Observation Transmission and storage Composability SPLT 10 packets 10 bytes 10 bytes Byte Distribution No 256 bytes 256 bytes 16 bytes

31 Conclusions Intraflow data is feasible to implement, enables useful inferences SPLT is valuable and relatively cheap Byte Distribution is valuable but more costly Training classifiers is key Data fusion

32 Thank You Cisco and/or its affiliates. All rights reserved. Cisco Confidential 34

33 Joy applications pcap pcap2flow json Server Threat Intelligence online pcap2flow json Server Endpoint online pcap2flow json Server SPAN or TAP online pcap2flow json Raspberry Pi Home