Casting out Demons: Sanitizing Training Data for Anomaly Sensors Angelos Stavrou,

Transcription

1 Casting out Demons: Sanitizing Training Data for Anomaly Sensors Angelos Stavrou, Department of Computer Science George Mason University Joint work with Gabriela Cretu, Michael E. Locasto, Salvatore J. Stolfo, Angelos D. Keromytis

2 Anomaly Detection (AD) Systems Supervised They are dependent on labeled data, which cannot be prepared for large data sets, eg. network packets Semi-supervised Using a third party sensor for labeling some data as known bad data Dependent on clean data for training Unsupervised Can clean the data by determining the outliers in the training data No good definition for an anomaly other than low probability data

3 Motivation Detection of zero-day attacks (only using AD system) Detection accuracy of all learning-based anomaly detectors depends heavily on the quality of the training data Training data is often poor, severely degrading AD s reliability as detection and forensic analysis tools

4 Rest of the Talk Intuition Local Training Sanitization Distributed Cross-Sanitization Future work Conclusions

5 Intuition Pattern of actions reflected on traces: Regular what we are expecting based on previous observations Abnormal unlikely data requiring further investigation An attack can pass as normal traffic if it is part of the training set Sanitize the training data by using a large set of micro-models where attacks and non-regular data cause a localized or limited pollution of training data

6 Training Dataset Sanitization Attacks and accidental mal-formed requests/data cause a local pollution of training data An attack can pass as normal traffic if it is part of the training set We seek to remove both malicious and abnormal data from the training dataset Related ML algorithms: Ensemble methods [Dietterich00] MetaCost [Domingos99] Meta-learning [Stolfo00]

7 Training Strategies Uniform Time Divide data into multiple blocks micro-datasets with the same time granularity..

8 Training Strategies Multiple Models Divide data into multiple blocks Build micro-models for each block Attacks and non-regular data cause localized pollution.. µmm 1 µm M 22 µmm K

9 Training Strategies Voting Models Divide data into multiple blocks Build micro-models for each block Test all models against a smaller dataset Simple voting: Weighted voting: w i = number of packets.. µmm 1 µm M 22 µmm K used for training µm i Voting algorithm

10 Training Strategies - Sanitization Divide data into multiple blocks Build micro-models for each block Test all models against a smaller dataset Build sanitized and abnormal models sanitized model:.. abnormal model: V = voting threshold µmm 1 µm M 22 µmm K Training phase Abnormal model model Voting algorithm Sanitized model

11 Shadow Sensor Redirection Shadow sensor Heavily instrumented host based anomaly detector akin to an oracle Performs substantially slower than the native application Use the shadow sensor to classify or corroborate the alerts produced by the AD sensors Sanitized model Feasibility and scalability depend on the number of alerts generated by the AD sensor Testing phase Alert? Alert? False false positive False positive Host based IDS Alert Shadow Alert server

12 Overall Architecture µmm 1 µm M 22 µmm K Training phase Testing phase Alert? Voting algorithm Abnormal Malicious model model Sanitized model Alert? False false positive False positive Host based Shadow IDS Alert Alert server For each host, use a large set of training data: Divide data into multiple blocks Build micro-models for each block Test all models against a smaller dataset Sanitize data based on previous step and build the sanitized model Build an abnormal model as well

13 Micro-models Partition a large training dataset into a number of smaller, time delimited training sets => microdatasets where each md i has a time granularity, g AD can be any chosen anomaly detection algorithm T is the training dataset M denotes the normal model produced by AD Attacks and non-regular data cause a localized or limited pollution of training data

14 Voting algorithms Using a second dataset and testing it against M i L j,i = 0 if M i deems the packet P j as normal L j,i = 1 otherwise The generalized label for packet P j where w i is the weight assigned to M i Simple voting: Weighted voting: used for training µm i = proportion of all packets

15 Sanitized and Abnormal Models Sanitized model Abnormal model V = voting threshold

16 Evaluation Proof of concept using two content-based anomaly detectors: Anagram Payl semi-supervised learning (when using Snort) supervised learning (without Snort) analyzing n-gram unsupervised learning analyzing byte(1-gram) frequency distributions

17 Evaluation dataset 300/100/100 hours of real network traffic

18 Voting Techniques Comparison a) Simple voting b) Weighted voting Performance of Anagram sensor after sanitization for www1

19 Datasets Comparison Performance for www and lists for 3-hour granularity when using Anagram

20 AD sensors comparison Sensor FP)(%) FA TP)(%) TA Anagram Anagram)with)Snort Anagram)with sanitization Payl , Payl)with)sanitization ,

21 Signal-to-noise ratio comparison Sensor www1 www lists Anagram Anagram with Snort Anagram with sanitization Payl Payl with sanitization signal-to-noise ratio TP/FP: higher values mean better results

22 Granularity Impact Granularity impact on the performance of the system when using Anagram and Payl

23 Training Dataset Size Impact Impact of the size of the training dataset for www1

24 AD s Internal Threshold Impact Impact of the anomaly detector s internal threshold for www1 when using Anagram

25 Analysis of g and V a) Simple voting b) Weighted voting Performance of Anagram sensor after sanitization

26 Shadow Sensor Performance Evaluation Overall computational requirements of an AD sensor and a host based sensor (e.g. STEM and DYBOC) l is the standard latency of a protected service O s is the shadow server overhead FP is the false positive rate Sensor STEM DYBOC N/A 44*l 1.2*l Anagram 1.031*l *l Anagram=with=Snort *l *l Anagram=with=sanitization *l *l Payl *l *l Payl=with=sanitization *l *l

27 Caveat Emptor & Limitations The presence of a long-lasting attack in the dataset used for computing the micro-models Poisoning all the micro-models

28 AD Distributed Cross-Sanitization Use external knowledge (models) to generate a better local normal model Abnormal models are exchanged across collaborative sites [Stolfo00] re-evaluate the locally computed sanitized models Apply model differencing Remove remote abnormal data from the local normal model

29 Cross-sanitization Direct model differencing Analytic method, difference of the models Indirect model differencing No analytic method, use testing Local sanitized model direct indirect Remote abnormal model

30 Cross-sanitization: Evaluation Model www1 www lists FP (%) TP (%) FP (%) TP (%) FP (%) TP (%) M pois M cross (direct) M cross (indirect) Indirect model differencing is more expensive than the direct model differencing Method www1 www lists direct s s s indirect s s s

31 Future work adversarial scenarios: new techniques to resist training attacks distributed sanitization: a distributed architecture to share models and remove training attacks model updates: updating AD models to accommodate concept drift

32 Conclusions A novel sanitization method that boosts the performance of out-of-the-box anomaly detectors Simple and general method, without significant additional computational cost An efficient and accurate online packet classifier; both in real time and in post-processing forensic analysis

33 Thank you Questions?