HILTI: An Abstract Execution Environment for Deep, Stateful Network Traffic Analysis

Transcription

1 HILTI: An Abstract Execution Environment for Deep, Stateful Network Traffic Analysis Robin Sommer International Computer Science Institute, & Lawrence Berkeley National Laboratory

2 A Tale of Three Open-Source IDS Shared functionality? Essentially none. Suricata Same for packet filters, firewalls, proxies, routers, switches, OS stack 2

3 Deep Packet Inspection Internet Tap Internal Network IDS Example: Finding downloads of known malware. 1. Find and parse all Web traffic. 2. Find and extract binaries. 3. Compute hash and compare with database. 4. Report, and potentially kill, if found. 3

4 DPI Architecture Application User Interface Configuration, logs, alarms Common primitives & idioms but hardly any reuse Analysis Logic Analysis Primitives State Management Protocol Parsing Network Traffic Signature matching, policy enforcement Pattern matching, packet classification, event correlation, multiplexing Flow table, DFAs, request/reply correlation IP, TCP, HTTP, DNS and that even though this stuff is hard. Why? Different low-level structure & data flows. No common language. 4

5 A High-Level Intermediary Language for Traffic Inspection Application User Interface Analysis Logic Analysis Compiler Host Application Firewall rules, IDS signatures, forwarding rules, Analysis Primitives State Management Protocol Parsing Library of Reusable Functionality HILTI Abstract Machine Intermediary language Execution Model LLVM-based compiler Runtime library Reusable components Network Traffic 5

6 Example: BPF Filters host or src net /24 type IP::Header = overlay { hdr_len: int<8> at 0 unpack UInt8InBigEndian (0, 3), version: int<8> at 0 unpack UInt8InBigEndian (4, 7), [...] src: addr at 12 unpack IPv4InNetworkOrder, dst: addr at 16 unpack IPv4InNetworkOrder } bool filter(ref<bytes> packet) { local addr a1, a2 local bool b1, b2, b3 } a1 = overlay.get IP::Header src packet b1 = equal a a1 = overlay.get IP::Header dst packet b2 = equal a b1 = or b1 b2 b2 = equal /24 a1 b3 = or b1 b2 return b3 6

7 Instruction Set Bitsets Booleans CIDR masks Callbacks Closures Channels Debug support Doubles Enumerations Exceptions File i/o Flow control Hashmaps Hashsets IP addresses Integers Lists Packet input Packet classification Packet dissection Ports Profiling Raw data References Regular expressions Strings Structs Unions Time intervals Timer management Timers Times Tuples Vectors/arrays 7

8 HILTI Machine Model Focus Areas Rich Domain-specific Data Types Flexible Control Flow Concurrent Analysis Robust & Secure Execution Comprehensive Host Interface Real-time Performance Debugging & Profiling Support High-level Optimization 8

9 Implementation: The HILTI Toolchain Host Application HILTI Environment LLVM Toolchain App Core C Interface Stubs Analysis Specification Analysis Compiler HILTI Machine Code HILTI Compiler LLVM Bitcode Compiler/ Linker Native Executable Runtime Library Just-in time via C++ API 9

10 Hello, World! module Main import Hilti void run() { call Hilti::print("Hello, World!") } hello.hlt # hilti-build hello.hlt -o a.out &&./a.out Hello, World! # hiltic -j hello.hlt Hello, World! 10

11 Can HILTI support complex applications? 11

12 Application Case Studies BPF Filter Stateful Firewall Protocol Parsing Bro Script Execution 12

13 BinPAC - A Yacc for Network Protocols Grammar example: Parsing SSH banners. SSH-2.0-OpenSSH_3.8.1p1 type SSH::Banner = unit { magic : /SSH-/; version : /[^-]*/; dash : /-/; software: /[^\r\n]*/; } BinPAC compiles grammar into HILTI parser. HILTI compiles parser into executable code just-in-time. Bro plugin integrates parsers at startup. 13

14 Hello, World! type SSH::Banner = unit { magic : /SSH-/; version : /[^-]*/; dash : /-/; software: /[^\r\n]*/; } ssh.pac2 grammar ssh.pac2; protocol analyzer SSH over TCP: parse with SSH::Banner, port 22/tcp; on SSH::Banner -> event ssh_banner(self.version, self.software); ssh.evt event ssh_banner(version: string, software: string) { { print software, version; } ssh.bro # bro -r ssh.trace ssh.evt ssh.bro OpenSSH_3.9p1, 1.99 OpenSSH_3.8.1p1,

15 Application Case Studies BPF Filter Stateful Firewall Protocol Parsing Bro Script Execution 15

16 Bro Scripts Script example: A simple scan detector. global attempts: table[addr] of count &default=0; event connection_rejected(c: connection) { local orig = c$id$orig_h; # Get originator address. local n = ++attempts[orig]; # Increase counter. } if ( n == SOME_THRESHOLD ) # Check for threshold. NOTICE(...); # Alarm. Bro plugin compiles scripts into HILTI code. HILTI compiles that into executable code just-in-time. 16

17 Evaluation Use HILTI plugin for Bro to compare parsing & script execution with a native Bro. Traces: HTTP: 1/25 of Berkeley port 80 traffic. 30GB trace, 52min, 340k messages. DNS: Full Berkeley port 53 traffic. 1GB trace, 10min, 65M messages. Correctness HILTI captures semantics correctly. Performance Let s see. 17

18 Protocol Parsing HTTP 6tanGaUG 683G 643G 241G 1567G HIL7I 852G 450G 21G 258G 1580G 6tanGaUG DNS 1.25x 177G 356G 180G 712G Protocol PDrsing 6cULSt ExHcutLon HIL7I-to-BUo GOuH 2thHU HIL7I 469G 405G 81G 217G 2.65x 1173G 0.0B 0.2B 0.4B 0.6B 0.8B 1.0B 1.2B 1.4B 1.6B 1.8B C38 cycohs 18

19 Bro Scripts HTTP 6tanGaUG 683G 635G 244G 1562G HIL7I 6tanGaUG HIL7I DNS 698G 781G 76G 254G 175G 358G 176G 709G 175G 243G 139G136G 694G 0.68x 1.23x 3UotocoO 3aUsLng 1810G ScriSt ExHcution HIL7I-to-BUo GOuH 2thHU 0.0B 0.5B 1.0B 1.5B 2.0B C38 cycohs 19

20 Summary HILTI: A new platform for network traffic analysis. A compiler-target for host applications to leverage. Provides common data structures and control flow primitives. Case studies demonstrate aptness of design. Packet filter, stateful firewall, protocol parsing, Bro scripts. Initial performance experiments encouraging. Not too different from native applications. It s still a prototype, with lots of potential. Sommer/Vallentin/De Carli/Paxson: HILTI: An Abstract Execution Environment for Deep, Stateful Network Traffic Analysis. ACM IMC

21 The HILTI Vision Performance via Abstraction Transparent improvement under the hood. Integration of non-standard hardware. High-level compiler optimizations. Automatic parallelization. Facilitate Reuse Means and glue to share functionality. HILTI library of common high-level components. HILTI is available under BSD license at De Carli/Sommer/Jha: Beyond Pattern Matching: A Concurrency Model for Stateful Deep Packet Inspection. ACM CCS 2014.