Automated Extraction of Network Protocol Specifications

Transcription

1 Automated Extraction of Network Protocol Specifications ARO MURI on Cyber Situation Awareness Kick Off Meeting Christopher Kruegel Computer Security Group,

2 Motivation Stateful protocol specifications are very useful black-box vulnerability analysis (fuzz testing) deep packet inspection (intrusion detection) network understanding (who is talking to whom, about what, ) application version fingerprinting Not all protocols are open Open specifications can lack semantics Manual reverse engineering is time-consuming and tedious 2

3 Goal Automatic extraction of protocol specifications 1. message format specifications what types of messages are there what fields are they composed of (length, handles, data, ) what is the semantics of data fields (file name, date, content, ) 2. protocol state machine what are acceptable sequences of messages 3

4 Available Data Sources Network traces limited information (no semantics) Server binaries static analysis dynamic analysis 4

5 Approach Mostly dynamic analysis (+ some static analysis) Basic insight Observe how the program processes (parses) input messages Use dynamic taint analysis to observe the data flow (trace) Analyze format of individual messages Cluster set of similar messages (each cluster is one type) Generalize message formats for each message type Use message sequences (sessions) to extract state machine 5

6 System Overview Our system operates in four phases Each phase produces input for the following phase 6

7 Dynamic Taint Analysis Obtain execution trace Run application (server) in a dynamic data tainting environment Send messages to server (using a client), or wait until a client starts a session with the server Assign a label to each input byte Track propagation of labels during the execution Record all executed instruction taint labels of all instruction operands 7

8 Dynamic Taint Analysis Label Input: G E T / H T T P / 1. 0 \r \n \r \n Propagate Labels: EAX c G E BL G 0 push %esi push %ebx mov (%eax),%bl sub $0x1,%ecx 0 1 8

9 System Overview 9

10 Session Analysis Extract message format Structure-forming parts enough information to parse a message length fields delimiter fields hierarchy between delimiters Additional semantics keywords, file names, session ids,.. 10

11 Detecting Length Fields Length fields are used to control a loop over input data Leverage binary static analysis to detect loops Look for loops where an exit condition tests the same taint labels on every iteration The tricky part is detecting the target field Look at labels touched inside length loop remove labels touched in all iterations remaining labels belong to target field May need to merge multiple loops (e.g., memcpy) 11

12 Detecting Delimiters Delimiter is one or more bytes that separate a field or message observation: all bytes in the scope of the delimiter are compared against a part of the delimiter Delimiter field detection create a list of taint labels used for comparisons for each byte value, merge consecutive labels into intervals Intervals indicate delimiter scope nesting gives us a hierarchical structure recursive analysis to break up message 12

13 13

14 Additional Semantics Protocol keywords IP addresses (used in socket calls) File names Echoed fields (session identifier, cookie, ) Pointers (to somewhere else in packet) Unused fields 14

15 Detecting Keywords A keyword is a sequence of (1 or 2 byte) characters that is tested against a constant value adjacent characters being successfully compared to non-tainted values are merged into a string take delimiters into account Ideally, we would want to check if input is being tested against values that are hard-coded in binary Currently, we just check whether the string (of at least 3 bytes) is present in the binary 15

16 System Overview 16

17 Message Clustering After the session analysis phase, we have format specifications for individual messages We want to automatically determine the different message types that appear in the observed application sessions Assume a similar reaction of the server to similar messages (that is, if they have the same type) First step: Find a metric of similarity between messages 17

18 Message Similarity We define several similarity features and distance functions These features can be divided into three groups: input similarity features ( message format ) execution similarity features ( code execution ) impact similarity features ( behavior ) For each group, we compute a similarity score 18

19 Input Similarity Feature Assumption: Messages of the same type have similar fields and structure To compute an input similarity score, we use a modified sequence alignment algorithm (Needleman Wunsch) The sequences of fields for all message formats are compared Similar parts get aligned, exposing differences or missing parts (matches, mismatches, gaps) 19

20 Code Execution Similarity Assumption: Similar messages are handled by similar code For each pair of messages, the sets of system calls process activity (clone, kill, ) invoked functions invoked library functions executed addresses are recorded Then, the Jaccard indices are computed: 20

21 Behavior Similarity Assumption: Similar messages trigger similar behavior in the server application Output and file activity similarity feature where is data written to same socket, network, file, or terminal what data is written static strings, tainted data, configuration file content 21

22 Message Clustering The similarity features are used to compute a distance matrix We apply the partitioning around medoids (PAM) algorithm for clustering PAM needs the desired number of clusters k as a parameter we determine k by employing a generalization of the Dunn index Dunn index is a standard intrinsic measure of clustering quality (cluster separation / cluster compactness) Result: Clusters of messages that are similar (of same type) For each cluster, a generalized message format is generated 17

23 System Overview 23

24 State Machine Inference Goal Use the information on message types and the application sessions that we observed to infer a minimal state machine Technique Find the minimal automaton that is consistent with our training set, without being overly general 24

25 Augmented Prefix Tree Acceptor (APTA) First, we construct an Augmented Prefix Tree Acceptor (APTA) incompletely specified DFA with a state transition graph that is a tree Each branch of the tree represents the sequence of message types within an observed application session Agobot (malware) example with two application sessions: 25

26 Minimization We want to generalize the APTA by merging some states We only want to merge states that correspond to similar states in the server application (otherwise, result is overly permissive) Commonly, in application level protocols, specific messages have to be sent before the server can perform certain actions often, login is necessary before other commands can be executed other commands may lead the server away from these states Identify states where the application can process similar commands based on the sequence of messages that it previously received 26

27 State Labeling To capture this intuition, we use prerequisites A prerequisite is a sequence of messages that the server has received that leads it to a specific state For the server to be in a state where it can meaningfully process a message of type m, it first has to receive a message of type r (always), optionally followed only by messages of certain types Once all prerequisites are computed, each state is labeled with the set of message types that are allowed as input in that state 27

28 State Labeling 28

29 State Machine Minimization Compute minimal consistent DFA from the labeled APTA run Exbar on the labeled state tree Exbar is the state-of-the-art exact algorithm for minimal consistent DFA inference end-state detection simple heuristic: mark end-states by finding messages that only appear last in sessions result is the protocol state machine 29

30 Agobot Example Example state machine (generated from two observed application sessions) Captures the notion that login is necessary for the command, and logout returns to the initial state 30

31 Evaluation State machines for four widely deployed real-world protocols Agobot malware example text-based protocol (modified IRC) we mimicked a bot herder and sent a few commands SMTP applied our system to sendmail daemon used 16 application sessions (sending ) as training set 31

32 Evaluation Server Message Block (SMB) complex, stateful, binary protocol we monitored the smbd daemon used smbclient for creating the training set recorded 31 training sessions, performing file operations Session Initiation Protocol (SIP) text-based protocol often used in VoIP infrastructure Asterisk server connected using 2 client soft-phones, made phone calls, used voice boxes,... training set represents the use cases of calling someone 32

33 SMB State Machine 33

34 Quality of Specifications How good are these results? specifications should parse valid sessions without being too general Parsing success to this end, we parsed real-world network traces with the extracted specifications Protocol #Sessions Parsing success SMTP 31,903 93,5% SIP % SMB 80 90% SMTP: remaining 6.5% used TLS encryption (limitation) SMB: Fails were previously unknown error conditions (files not found, etc.) 34

35 Quality of Message Formats Protocol #Sessions Parsing success SMTP 31,903 93,5% SIP % SMB 80 90% 35

36 Quality of Specifications State machine comparison we manually built reference state machines for the tested protocols performed 50,000 random walks over inferred state machines, and checked if the message sequences are valid in the reference state machines precision: ratio of sequences generated by random walks over the inferred state machine that are accepted by the reference state machine recall: ratio for sequences from random walks over the reference state machine that are accepted by the inferred state machine Protocol Precision Recall Agobot 1 1 SMTP 1 1 SMB 1.58 SIP

37 Comparative Evaluation Compare our state machine inference with known algorithms for inductive inference (based on minimum message length) sk-strings, beams Known algorithms did not provide acceptable performance on our training data Agobot SMTP SMB SIP P R P R P R P R Prospex beams sk-strings(and) sk-strings(or)

38 Conclusion Prospex can automatically infer protocol specifications for stateful protocols Steps of approach extract message specifications automatically identify message types infer the protocol state machine Generate protocol specifications for a stateful fuzzer 38