Efficient Signature Matching with Multiple Alphabet Compression Tables

Transcription

1 Efficient Signature Matching with Multiple Alphabet Compression Tables Shijin Kong Randy Smith Cristian Estan Presented at SecureComm, Istanbul, Turkey

2 Signature Matching Signature Matching a core component of network devices Operation (ideal): For a set of signatures, match all relevant sigs in a single pass over payload Many constraints Evolving, complex signatures Wirespeed operation Limited memory Active adversary

3 Regular Expressions and DFAs Regular expressions standard for writing sigs Buffer overflow: /^RETR\s[^\n]{}/ Format string attack: /^SITE\s+EXEC[^\n]*%[^\n]*%/ DFAs used for matching to input 3

4 DFA Operation State State State State 3 crt_state= 6 next_state= input_byte= if accept(next_state) alert

5 Matching with DFAs Advantages Fast minimal per-byte processing Composable combine many DFAs into one Disadvantages States are heavyweight ( KB each!) State-space explosion occurs when DFAs combined Memory exhausted with only a few DFAs! Workaround: many DFAs matched in parallel

6 Key: Reduce memory usage Strategy: aggressively reduce memory footprint, keep exec time low State State State State 3 Reduce size of transition tables 6 Reduce number of states 6

7 Main Contribution Multiple Alphabet Compression Tables Lightweight, applicable to hardware or software Compatible with other techniques Worst case = average case Results (in software) x to 7x memory reduction 3% - % execution time increase 7

8 Outline Introduction Alphabet Compression Tables Interacting with D FAs Experimental Results

9 Alphabet Compression: core observation State State State State 3 Some input symbols are equivalent; the transitions on those symbols at any state are identical. 6 9

10 Alphabet Compression Tables Alphabet compression table 3 State State State State 3 6 input_byte= index= crt_state= next_state=

11 Even further compression Alphabet compression table 3 State State State State 3 6

12 Even further compression Alphabet compression table 3 State State State State 3 6

13 Even further compression Alphabet compression table 3 State State State State 3 6 3

14 Multiple ACTs ACT ACT 3 3 State State State State 3 6 How do we know which ACT to use with which state?

15 Multiple ACTs ACT input_byte= crt_act= crt_state= ACT 3 3 State State State State 3 6 index= next_act= next_state=

16 Constructing Multiple ACTs Partition states appropriately for example: {S, S, S 3,, S n } { {S, S,}, {S, S 3, S 9,}, } Construct single ACT for each group of states See algorithm in paper 6

17 Partitioning States for ACTs Input: number of ACTs to use m, DFA D Output: a partition of states into m subsets Use greedy, heuristic approach: States = Set of all states in D; while (m>) { Subset = GetEquivClassPartition(States); AddToResult(Subset); States = States Subset; m--; } return Result; 7

18 How many Compression Tables? Eight ACTs is enough Avg trans per state Avg exec time

19 Outline Introduction Alphabet Compression Tables Interacting with D FAs Experimental Results 9

20 ACTs and D FAs Two kinds of redundancy a,b,c S S d e S3 a,b,c S a,b,c Symbols have identical behavior for large subsets of states Compress with (multiple) ACTs

21 ACTs and D FAs Two kinds of redundancy S a,b,c d e S S3 a,b,c e S f c h S e f h a,b,c S c Symbols have identical behavior for large subsets of states Compress with (multiple) ACTs Symbols at many states lead to common next states Compress with D FAs

22 D FAs: core observation For many pairs of states, the transitions for most characters are identical! State State State State 3 6 Idea: store only one copy Kumar et al (Sigcomm 6); Kumar et al (ANCS 6); Becchi et al (ANCS 7)

23 D FAs: core observation For many pairs of states, the transitions for most characters are identical! State State State State 3 6 Idea: store only one copy Kumar et al (Sigcomm 6); Kumar et al (ANCS 6); Becchi et al (ANCS 7) 3

24 D FAs Default transitions State State State State 3 Issue: good compression, potentially heavy runtime cost 6 input_byte= crt_state= next_state=

25 ACTs and D FAs Together Combine ACTs and D FAs to address both kinds of redundancy Procedure:. Apply D FA compression to DFAs. Apply multiple ACT compression to D FA results Only slight modification to ACT construction Add not handled here symbol Deal with default transitions

26 ACTs + D FAs Default transitions State State State State 3 6 6

27 ACTs + D FAs ACT State State State State 3 6 7

28 ACTs + D FAs ACT ACT State State State State 3 6 3

29 ACTs + D FAs index= ACT ACT State State State State index= input= crt_act= crt_state= next_act= next_state= 9

30 Outline Introduction Alphabet Compression Tables Interacting with D FAs Experimental Results 3

31 Experimental Setup HTTP, SMTP, FTP signatures Grouped by protocol and rule set (Snort or Cisco) DFA Set Splitting (Yu, 6) to cluster DFAs Provide memory bound a priori Heuristically combine into as few DFAs as possible Experiment Environment GB traces, run on 3. GHz P Exec time measured with cycle-accurate counters 3

32 Memory vs Time States DFAs States DFAs 6 States DFAs ideal 3

33 Memory vs Time Snort HTTP Cisco IPS HTTP Lowest mem, highest exec! 33

34 Memory vs Time Snort SMTP Cisco IPS SMTP Lowest mem, highest exec! 3

35 Conclusion Multiple alphabet compression tables Lightweight Applicable to hardware or software platforms Compatible with other techniques Provides better time vs. space performance x to 7x memory reduction 3% to % execution time increase Best technique a function of time, memory limits ACTs add superior design points 3

36 Efficient Signature Matching with Multiple Alphabet Compression Tables Thank you 36

37 intentionally blank 37

38 Regular Expressions and DFAs Regular expressions standard for writing sigs Buffer overflow: /^RETR\s[^\n]{}/ Format string attack: /^SITE\s+EXEC[^\n]*%[^\n]*%/ DFAs used for matching to input Header Payload C A 6 site exec % retr % \n \n S r s e t i t e r [^\n] [^\n] % S3 S \n [^\nrs] Match sig! 3

39 Memory Usage DFA ACT D FA ACT + D FA Snort HTTP Snort SMTP Snort FTP DFA ACT D FA ACT + D FA Cisco HTTP Cisco SMTP 9 3. Cisco FTP All results reported in megabytes (MB) 39

40 Memory vs Time Snort FTP Cisco IPS FTP