Intrusion Detection and Malware Analysis

Transcription

1 Intrusion Detection and Malware Analysis Anomaly-based IDS Pavel Laskov Wilhelm Schickard Institute for Computer Science

2 Taxonomy of anomaly-based IDS Features: Packet headers Byte streams Syntactic events Methods Manual modeling: no training, model constructed by hand Learning from clean data: training requires attack-free data Learning from noisy data: training is required but contamination with attacks can be tolerated Online learning: no training, detection on the fly after a short initialization period.

3 Anomaly-based IDS to be discussed PHAD: packet header features, learning from clean data ALAD: TCP connection features and application keywords, learning from clean data PAYL: content byte stream features, learning from clean data (Re)MIND: content byte stream and structured features, learning from noisy data, online learning

4 PHAD features Ethernet IP TCP UDP ICMP size dest hi dest lo src hi src lo proto src port dst port len checksum hlen tos len frag id frag ptr ttl proto checksum src addr dest addr src port dst port seq num ack num hlen flags wsize checksum urg ptr options type code checksum

5 PHAD algorithm Probability of novel feature values Suppose an attribute x has previously assumed r values in n observations (r < n). Then the probability that the next value of x is different from previously seen values is approximately r n. Computation of the anomaly score For each attribute x in the training data compute the ratio r n. For every new packet p, let Q be a set of attributes whose values were not seen in the training data. Let t i be the time since the last anomalous value has been seen in the attribute i. Then the anomaly score of the packet p is: t s(p) = i n i r i Q i

6 PHAD example Training data: {[ A 1 r ], [ ] B, 2 [ ] B, 3 [ ] C, 4 [ ] A, 1 n -values: { } 3/8 1/2 Test data: {[ ] A, 1 Score for the last packet: [ ] D, 2 [ ] B, 3 [ ] A, 2 [ ]} E 5 [ ] C, 3 s(p 4 ) = 2 8/ = [ ]} C 4

7 Feature clustering in PHAD Problem: each PHAD feature is 4 byte long: 2 32 values are possible Storing n r ratios for every possible value is infeasible

8 Feature clustering in PHAD Problem: each PHAD feature is 4 byte long: 2 32 values are possible Storing n r ratios for every possible value is infeasible Feature clustering: Put a cap C on the number of values to be considered for each feature (e.g. C = 32). Every time this cap is exceeded, merge two nearest values (or ranges) into a new range. e.g. (for C = 3): {3-5, 8, 10-15, 20} {3-5, 8-15, 20} A value of an attribute of a packet p is considered unseen if it falls outside of known ranges.

9 PHAD resume Simple algorthm for anomaly scoring (+) High performance (ca. 75,000 packets/s) (+) Easy to implement (ca. 400 lines of C++ code) ( ) Clean training data required ( ) Poor detection rates: 28% on the DARPA 1999 dataset Packet header features: (+) Easy to extract ( ) Are not useful for detection of real exploits

10 ALAD features P(src ip dst ip): a set of client hosts for each host P(src ip dst ip, dst port): a set of client hosts for each service and each host. P(dst ip, dst port): a set of normal servers on a site (unusual values may be indicative of probes) P(tcp flags dst port): usual patterns of TCP connection initiation and closure for each service. P(keywords dst port): typical application-level keywords for each service.

11 ALAD algorithm Detection is carried out at a connection level. For conditional features, the n r models are computed for each pre-conditioner (i.e. dst ip, dst port or a {dst ip, dst port} pair). For the joint feature, the count r is computed for all pairs of values divided by the total number of connection. No feature clustering is performed.

12 ALAD resume Similar algorithm and same problems as PHAD Limited application-level features slightly improve detection of difficult attacks in the DARPA 1999 dataset.

13 PAYL: payload byte sequence analysis Motivation: detect anomalous packet payloads.

14 PAYL: payload byte sequence analysis Motivation: detect anomalous packet payloads. Problem 1: how can one define a normal packet payload?

15 PAYL: payload byte sequence analysis Motivation: detect anomalous packet payloads. Problem 1: how can one define a normal packet payload? Problem 2: how can one measure the degree of anomality of a packet payload?

16 packet, packet, packet, and and there and there is there already is already is already a very a very a clear very clear pattern clear pattern patter with w displays displays displays the the the variability of the of the of the frequency distributions a PAYL features: raw byte histograms The The two The two plots two plots plots characterize two two different two different different distributions same same web same web server, web server, server, port port 80 port for 80 for 80 two for two different two different different lengths, leh other other 1,460 other 1,460 bytes. 1,460 bytes. Clearly, bytes. Clearly, Clearly, a single a single a single monolithic model not not not represent the the the distributions accurately. A byte histogram is an array of size 256 measuring frequency of all possible byte values in a given packet payload. Examples: port 22 (SSH) port 25 (SMTP) port 80 (HTTP) Advantages of byte histograms: Computation of mean and standard deviation Computation of a distance between two histograms

17 PAYL algorithm Training For each observed packet length, compute average byte histograms with standard deviations. Cluster average models for neighboring packet lengths by merging two histograms if their distance is less than some pre-defined threshold t. Anomaly detection For each new packet p compute its distance from the normal profile for the corresponding packet length (cluster): s(p) = 256 f i (p) f i ( p) σ i=1 i ( p)

18 PAYL resume Anomaly detection over packet payload byte sequencies. Reasonable accuracy: over 50% detection rate for remote attacks at 1% FP rate. Simple detection algorithm: computationally efficient. Drawbacks: Packet mode operation: evasion problems! Primitive data structures: only single-byte histograms possible. Clean training data required.

19 MIND: machine learning IDS Motivation: Learning from contaminated data Improvement of accuracy: good detection at extremely small FP rates needed! Incorporation of semantic structure using advanced learning models.

20 MIND: machine learning IDS Motivation: Learning from contaminated data Improvement of accuracy: good detection at extremely small FP rates needed! Incorporation of semantic structure using advanced learning models. Problem 1: How can one compare packets (connections) in a more structural way than simple histograms?

21 MIND: machine learning IDS Motivation: Learning from contaminated data Improvement of accuracy: good detection at extremely small FP rates needed! Incorporation of semantic structure using advanced learning models. Problem 1: How can one compare packets (connections) in a more structural way than simple histograms? Problem 2: How can one learn in the presence of attacks in training data?

22 Learning from clean data Training: find a smallest enclosing sphere min R,c R 2 s.t. x i c 2 R 2, i = 1,..., M. c R

23 Learning from clean data Training: find a smallest enclosing sphere min R,c R 2 s.t. x i c 2 R 2, i = 1,..., M. Detection: compute a distance from the center x c 2 > R alarm x c 2 R normal c R

24 Learning from contaminated data Training: soften constraints using slack variables. min R,c R 2 +η M i=0 ξ i, s.t. x i c 2 R 2 +ξ i, i = 1,..., M, ξ i 0 c R

25 Learning from contaminated data Training: soften constraints using slack variables. min R,c R 2 +η M i=0 ξ i, s.t. x i c 2 R 2 +ξ i, i = 1,..., M, ξ i 0 Constant η controls the acceptable noise rate in the training data. c R

26 Learning from contaminated data Training: soften constraints using slack variables. min R,c R 2 +η M i=0 ξ i, s.t. x i c 2 R 2 +ξ i, i = 1,..., M, ξ i 0 Constant η controls the acceptable noise rate in the training data. Detection: compute a distance from the center. c R

27 Beyond the mathematical abstraction How do we apply this geometric intuition to network security?

28 Beyond the mathematical abstraction How do we apply this geometric intuition to network security? Interesting observation: the only operation on data involved in the learning approach is computation of similarity between two points: x c.

29 Beyond the mathematical abstraction How do we apply this geometric intuition to network security? Interesting observation: the only operation on data involved in the learning approach is computation of similarity between two points: x c. If we can compute similarity between a pair of observed network events, any learning algorithm can be plugged in!

30 Embedding of sequences in metric spaces Sequences 1. blabla blubla blablabu aa 2. bla blablaa bulab bb abla 3. a blabla blabla ablub bla 4. blab blab abba blabla blu Subsequences Histograms of subsequences Geometry 2 3 Features blablabu blablaa blablu blabla bulab ablub blab abla abba blu bla bb aa b a 1 4

31 X = abrakadabra Y = barakobama Embedding example

32 Embedding example X = abrakadabra Y = barakobama X Y X Y a/5 a/4 20 b/2 b/2 4 d/1 k/1 k/1 1 m/1 o/1 r/2 r/ XY = 21.5

33 Embedding example X = abrakadabra Y = barakobama X Y X Y a/5 a/4 20 b/2 b/2 4 d/1 k/1 k/1 1 m/1 o/1 r/2 r/ XY = 21.5 X Y X Y ab/2 ad/1 ak/1 ak/1 1 am/1 ar/1 ba/2 br/2 da/1 ka/1 ko/1 ma/1 ob/1 ra/2 ra/ XY = 77.5

34 Experimental evaluation of MIND Data: 3 weeks of HTTP-Traffic at FIRST (622,734 HTTP requests) 120 attacks generated by Metasploit Quality measure: Receiver Operating Characteristic (ROC) True positive rate ReMIND Snort IDS False positive rate True positive rate ReMIND SSAD Anagram Tokengram False positive rate

35 MIND resume Anomaly detection over connection payloads Excellent accuracy: over 90% detection with no false alarms Learning in the presense of attacks. Sophisticated data structures. Drawbacks: Computationally more involved: efficient data structures and fine-tuning are required. Delay in detection until completion of TCP connections: incremental processing is needed.

36 Lessons learned Modern anomaly detection methods can achieve superior detection accuracy to signature-based IDS with low false alarm rates. Anomaly detection methods must be equipped for learning from contaminated date (or be able to automatically clean data) Efficient data structures for feature extraction are the key to success of anomaly detection.

37 Recommended reading M. Mahoney and P. Chan. Learning nonstationary models of normal network traffic for detecting novel attacks. In Proc. of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pages , M. Mahoney and P. Chan. Learning rules for anomaly detection of hostile network traffic. In Proc. of International Conference on Data Mining (ICDM), K. Rieck and P. Laskov. Language models for detection of unknown attacks in network traffic. Journal in Computer Virology, 2(4): , K. Wang and S. Stolfo. Anomalous payload-based network intrusion detection. In Recent Adances in Intrusion Detection (RAID), pages , 2004.