UNSUPERVISED LEARNING FOR ANOMALY INTRUSION DETECTION Presented by: Mohamed EL Fadly
|
|
- Cory Montgomery
- 6 years ago
- Views:
Transcription
1 UNSUPERVISED LEARNING FOR ANOMALY INTRUSION DETECTION Presented by: Mohamed EL Fadly
2 Outline Introduction Motivation Problem Definition Objective Challenges Approach Related Work
3 Introduction Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic.
4 What are anomalies Anomaly is a pattern in the data that does not conform to the expected behaviour Also referred to as outliers, exceptions, peculiarities, surprise, etc. Anomalies translate to significant (often critical) real life entities Cyber intrusions Credit card fraud
5 Real World Anomalies Credit Card Fraud An abnormally high purchase made on a credit card Cyber Intrusions A web server involved in ftp traffic
6 Anomaly detection Anomaly detection triggers alarms when the detected object behaves significantly differently from the predefined normal patterns. Hence, anomaly detection techniques are designed to detect patterns that deviate from an expected normal model built for the data. In cybersecurity, anomaly detection includes the detection of malicious activities, e.g., penetrations and denial of service. The approach consists of two steps: training and detection. a) In the training step, machine-learning techniques are applied to generate a profile of normal patterns in the absence of an attack. b) In the detection step, the input events are labeled as attacks if the event records deviate significantly from the normal profile. Subsequently, anomaly detection can detect previously unknown attacks
7 Application of Anomaly Detection Network intrusion detection Insurance / Credit card fraud detection Healthcare Informatics / Medical diagnostics Industrial Damage Detection Image Processing / Video surveillance Novel Topic Detection in Text Mining
8 Intrusion Detection Intrusion Detection: Process of monitoring the events occurring in a computer system or network and analyzing them for intrusions Intrusions are defined as attempts to bypass the security mechanisms of a computer or network Challenges Traditional signature-based intrusion detection systems are based on signatures of known attacks and cannot detect emerging cyber threats Substantial latency in deployment of newly created signatures across the computer system Anomaly detection can alleviate these limitations
9 Problem Definition In anomaly detection, labeled data corresponding to normal behavior are usually available, while labeled data for anomaly behavior are not. Supervised machine-learning methods need attack-free training data. However, this kind of training data is difficult to obtain in real-world network environments. This lack of training data leads to the well-known unbalanced data distribution in machine learning. In the huge volume of network data, the same malicious data repeatedly occur while the number of similar malicious data is much smaller than the number of normal data. The imbalanced data distribution of normal and anomaly data induces a high false-positive rates (FPRs) of supervised intrusion detection systems (IDSs).
10 Problem definition Unsupervised machine learning methods outperform supervised machine-learning methods in updating rules intelligently while the detection rates downgrade. Thus anomaly detection systems can potentially find new attacks, but they generally have a lower accuracy rate for detection and a higher FAR. The problem we must solve is how to minimize the false negative and false positive rates while keeping higher accuracy rates
11 Objective To propose an unsupervised anomaly detection technique that will produce low false positive rates and to overcome challenges in using labeled data sets for supervised learning, such as time consumption, expensiveness, limitation of expertise, and the accuracy of labels in collecting labeled data.
12 Why unsupervised To overcome the problem of using attack-free training data required by supervised learning. Moreover, with the changing network environment or services, patterns of normal traffic will change. The differences between the training and actual data can lead to high FPRs of supervised IDSs. To address these problems, unsupervised anomaly detection emerges to take unlabeled data as input. Unsupervised anomaly detection aims to find malicious information buried in cyberinfrastructure even without prior knowledge about the data labels and new attacks. Subsequently, unsupervised anomaly detection methods rely on the following assumptions: normal data covers majority while anomaly data are minor in network traffic flow or audit logs. That s why most of the solutions to unsupervised anomaly detection are clusteringbased anomaly/outlier detection techniques.
13 Challenges 1. The key challenge is that the huge volume of data with high-dimensional feature space is difficult to manually analyze and monitor. Such analysis and monitoring requires highly efficient computational algorithms in data processing and pattern learning. 2. Much of the data is streaming data, which requires online analysis. 3. It is also difficult to define a representative normal region or the boundary between normal and outlying behavior. As the concept of an anomaly/outlier varies among application domains 4. The labeled anomalies are not available for training/validation. 5. Training and testing data might contain unknown noises 6. Normal and anomaly behaviors constantly evolve.
14 Approach The below aspects should be considered when choosing my approach 1. Nature of input data 2. Availability of supervision 3. Type of anomaly: point, contextual, structural 4. Output of anomaly detection 5. Evaluation of anomaly detection techniques
15 10 Input Data Most common form of data handled by anomaly detection techniques is Record Data Univariate Multivariate Tid SrcIP Start time Dest IP Dest Port Number of bytes Attack :07: No :13: No :14: No :14: No :14: Yes :14: No :14: No :14: Yes :14: No :14: Yes
16 Input Data Nature of data Nature of attributes Binary Categorical Continuous Hybrid Tid SrcIP Duration Dest IP Number of bytes Internal No No Yes No No
17 Supervision Data Label Supervised Anomaly Detection Labels available for both normal data and anomalies Unsupervised Anomaly Detection No labels assumed; based on the assumption that anomalies are very rare compared to normal data Semi-supervised Anomaly Detection Labels available only for normal data Use modified classification model to learn the normal behaviour and then detect any deviations from normal behaviour as anomalous
18 Type of anomaly Point Anomalies: Contextual Anomalies Collective Anomalies
19 Point Anomalies An individual data instance is anomalous w.r.t. the data Y N 1 o 1 O 3 o 2 N 2 X
20 Contextual Anomalies An individual data instance is anomalous within a context Requires a notion of context Also referred to as conditional anomalies Normal Anomaly
21 Collective Anomalies A collection of related data instances is anomalous Requires a relationship among data instances Sequential Data Spatial Data Graph Data The individual instances within a collective anomaly are not anomalous by themselves Anomalous Subsequence
22 Output of anomaly Detection Label Each test instance is given a normal or anomaly label This is especially true of classification-based approaches Score Each test instance is assigned an anomaly score Allows the output to be ranked Requires an additional threshold parameter
23 Evaluation of Anomaly Detection F-value Accuracy is not sufficient metric for evaluation Example: network traffic data set with 99.9% of normal data and 0.1% of intrusions Trivial classifier that labels everything with the normal class can achieve 99.9% accuracy!!!!! Confusion matrix Predicted class NC C Actual NC TN FP class C FN TP Focus on both recall and precision Recall (R) = TP/(TP + FN) Precision (P) = TP/(TP + FP) F measure = 2*R*P/(R+P) anomaly class C normal class NC
24 Evaluation of Outlier Detection ROC Standard measures for evaluating anomaly detection problems: Recall (Detection rate) - ratio between the number of correctly detected anomalies and the total number of anomalies False alarm (false positive) rate ratio between the number of data records from normal class that are misclassified as anomalies and the total number of data records from normal class ROC Curve is a trade-off between detection rate and false alarm rate 0.2 Area under the ROC curve (AUC) is 0.1 computed using a trapezoid rule Detection rate ROC curves for different outlier detection techniques 1 AUC False alarm rate
25 Possible approach Anomaly Detection Point Anomaly Detection Classification Based Nearest Neighbor Based Clustering Based Statistical Others Rule Based Neural Networks Based SVM Based Density Based Distance Based Parametric Non-parametric Information Theory Based Spectral Decomposition Based Visualization Based Contextual Anomaly Detection Collective Anomaly Detection Online Anomaly Detection Distributed Anomaly Detection
26 Clustering Based Techniques Key assumption: normal data records belong to large and dense clusters, while anomalies belong do not belong to any of the clusters or form very small clusters Categorization according to labels Semi-supervised cluster normal data to create modes of normal behavior. If a new instance does not belong to any of the clusters or it is not close to any cluster, is anomaly Unsupervised post-processing is needed after a clustering step to determine the size of the clusters and the distance from the clusters is required fro the point to be anomaly Anomalies detected using clustering based methods can be: Data records that do not fit into any cluster (residuals from clustering) Small clusters Low density clusters or local anomalies (far from other points within the same cluster)
27 Clustering Based Techniques Advantages: No need to be supervised Easily adaptable to on-line / incremental mode suitable for anomaly detection from temporal data Drawbacks Computationally expensive Using indexing structures (k-d tree, R* tree) may alleviate this problem In high dimensional spaces, data is sparse and distances between any two data records may become quite similar. Clustering algorithms may not give any meaningful clusters
28 Related work 1. A Near Real-Time Algorithm for Autonomous Identification and Characterization of Honeypot Attacks ASIA CCS '15 Proceedings of the 10th ACM Symposium on Information, Computer and Communications Security Owezarski presents an unsupervised algorithm - called UNADA for Unsupervised Network Anomaly Detection Algorithm - for identification and characterization of security related anomalies and attacks occurring in honeypots. What is interested that their method does not need any attack signature database, learning phase, or labeled traffic
29 Related Work This algorithm has several advantages 1. It works in a completely unsupervised manner, what makes it able to work on top of any monitoring system, and directly usable, without preliminary configuration or knowledge. 2. It combines robust clustering techniques to avoid classical issues of clustering algorithms, e.g. sensitivity to initial configuration, the required a priori indication of the number of clusters to be identified, or the sensitivity of results when using less pertinent features. 3. It automatically builds simple and small signatures fully characterizing attacks; theses signature can then be used in a filtering security device. 4. It is designed to run in real time by making possible to take advantage of the parallelism of their clustering approach.
30 Evaluation true Positive rates vs False alarms They run their algorithm on the honeypot traffic traces gathered at the University of Maryland They compare the performance of UNADA against three previous approaches for unsupervised anomaly detection: DBSCAN-based, k-means-based, and PCA-based outliers detection.
31 Related Work 2- Enhancing One-class Support Vector Machines for Unsupervised Anomaly Detection Proceeding ODD '13 Proceedings of the ACM SIGKDD Workshop on Outlier Detection and Description Amer et.al have applied apply two modifications in order to make one-class SVMs more suitable for unsupervised anomaly detection: Robust one class SVMs and eta one-class SVMs. The key idea of both modifications is, that outliers should contribute less to the decision boundary as normal instances. Experiments performed on datasets from UCI machine learning repository show that their modifications are very promising: Comparing with other standard unsupervised anomaly detection algorithms, the enhanced one-class SVMs are superior on two out of four datasets. In particular, the proposed eta oneclass SVM has shown the most promising results.
32 Results Dataset used for evaluation ROC Curve Comparing the AUC of SVM based algorithms against other anomaly detection algorithms
33 Related Work An Unsupervised Anomaly Detection Engine With an Efficient Feature set for AODV Published in: Information Security and Cryptology (ISCISC), th International ISC Conference They proposed an anomaly detection engine by collecting decent features and applying robust PCA on the data set. The results showed their features can detect much more attacks either by applying PCA or by applying robust PCA. Their contribution in using the Robust PCA, through having an unsupervised algorithm that detect anomaly more accurate. The robust PCA can form the baseline profile even by existence of malicious nodes in the learning phase Their results show robust PCA cannot be affected by outlier data within the network
34 References 1- Philippe Owezarski A Near Real-Time Algorithm for Autonomous Identification and Characterization of Honeypot Attacks. In Proceedings of the 10th ACM Symposium on Information, Computer and Communications Security (ASIA CCS '15). ACM, New York, NY, USA, Mennatallah Amer, Markus Goldstein, and Slim Abdennadher Enhancing one-class support vector machines for unsupervised anomaly detection. In Proceedings of the ACM SIGKDD Workshop on Outlier Detection and Description (ODD '13). ACM, New York, NY, USA, Houri Zarch, M.K.; Abedini, M.; Berenjkoub, M.; Mirhosseini, A., "An unsupervised anomaly detection engine with an efficient feature set for AODV," Information Security and Cryptology (ISCISC), th International ISC Conference on, vol., no., pp.1,6, Aug Sumeet Dua and Xian Du. Data Mining and Machine Learning in cybersecurity. April 25, 2011 by Auerbach Publications 5- Varun Chandola, Arindam Banerjee, and Vipin Kumar Anomaly detection: A survey. ACM Comput. Surv. 41, 3, Article 15 (July 2009), 58 pages 6- Yingbing Yu A survey of anomaly intrusion detection techniques. J. Comput. Sci. Coll. 28, 1 (October 2012), Phil Simon. Too Big to Ignore: The Business Case for Big Data. Wiley, Taiwo Oladipupo Ayodele. New Advances in Machine Learning. InTech, Harjinder Kaur, Gurpreet Singh, Jaspreet Minhas, A Review of Machine Learning based Anomaly Detection Techniques
Unsupervised Learning for anomaly Intrusion detection
The American University in Cairo Unsupervised Learning for anomaly Intrusion detection Seminar 1 - Report Mohamed EL Fadly 5-17-2015 Table of Contents Introduction... 2 Motivation... 2 Intrusion detection
More informationAnomaly Detection. You Chen
Anomaly Detection You Chen 1 Two questions: (1) What is Anomaly Detection? (2) What are Anomalies? Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behavior
More informationData Mining Classification: Alternative Techniques. Imbalanced Class Problem
Data Mining Classification: Alternative Techniques Imbalanced Class Problem Introduction to Data Mining, 2 nd Edition by Tan, Steinbach, Karpatne, Kumar Class Imbalance Problem Lots of classification problems
More informationAnomaly Detection on Data Streams with High Dimensional Data Environment
Anomaly Detection on Data Streams with High Dimensional Data Environment Mr. D. Gokul Prasath 1, Dr. R. Sivaraj, M.E, Ph.D., 2 Department of CSE, Velalar College of Engineering & Technology, Erode 1 Assistant
More informationInternational Journal of Research in Advent Technology, Vol.7, No.3, March 2019 E-ISSN: Available online at
Performance Evaluation of Ensemble Method Based Outlier Detection Algorithm Priya. M 1, M. Karthikeyan 2 Department of Computer and Information Science, Annamalai University, Annamalai Nagar, Tamil Nadu,
More informationDATA MINING II - 1DL460
DATA MINING II - 1DL460 Spring 2016 A second course in data mining!! http://www.it.uu.se/edu/course/homepage/infoutv2/vt16 Kjell Orsborn! Uppsala Database Laboratory! Department of Information Technology,
More informationWhat are anomalies and why do we care?
Anomaly Detection Based on V. Chandola, A. Banerjee, and V. Kupin, Anomaly detection: A survey, ACM Computing Surveys, 41 (2009), Article 15, 58 pages. Outline What are anomalies and why do we care? Different
More informationDetection of Anomalies using Online Oversampling PCA
Detection of Anomalies using Online Oversampling PCA Miss Supriya A. Bagane, Prof. Sonali Patil Abstract Anomaly detection is the process of identifying unexpected behavior and it is an important research
More informationEvaluation Measures. Sebastian Pölsterl. April 28, Computer Aided Medical Procedures Technische Universität München
Evaluation Measures Sebastian Pölsterl Computer Aided Medical Procedures Technische Universität München April 28, 2015 Outline 1 Classification 1. Confusion Matrix 2. Receiver operating characteristics
More informationOutlier Detection Using Unsupervised and Semi-Supervised Technique on High Dimensional Data
Outlier Detection Using Unsupervised and Semi-Supervised Technique on High Dimensional Data Ms. Gayatri Attarde 1, Prof. Aarti Deshpande 2 M. E Student, Department of Computer Engineering, GHRCCEM, University
More informationA Comparative Study of Locality Preserving Projection and Principle Component Analysis on Classification Performance Using Logistic Regression
Journal of Data Analysis and Information Processing, 2016, 4, 55-63 Published Online May 2016 in SciRes. http://www.scirp.org/journal/jdaip http://dx.doi.org/10.4236/jdaip.2016.42005 A Comparative Study
More informationCS145: INTRODUCTION TO DATA MINING
CS145: INTRODUCTION TO DATA MINING 08: Classification Evaluation and Practical Issues Instructor: Yizhou Sun yzsun@cs.ucla.edu October 24, 2017 Learnt Prediction and Classification Methods Vector Data
More informationNETWORK FAULT DETECTION - A CASE FOR DATA MINING
NETWORK FAULT DETECTION - A CASE FOR DATA MINING Poonam Chaudhary & Vikram Singh Department of Computer Science Ch. Devi Lal University, Sirsa ABSTRACT: Parts of the general network fault management problem,
More informationINFORMATION-THEORETIC OUTLIER DETECTION FOR LARGE-SCALE CATEGORICAL DATA
Available Online at www.ijcsmc.com International Journal of Computer Science and Mobile Computing A Monthly Journal of Computer Science and Information Technology IJCSMC, Vol. 4, Issue. 4, April 2015,
More informationFeature Selection. CE-725: Statistical Pattern Recognition Sharif University of Technology Spring Soleymani
Feature Selection CE-725: Statistical Pattern Recognition Sharif University of Technology Spring 2013 Soleymani Outline Dimensionality reduction Feature selection vs. feature extraction Filter univariate
More informationNDoT: Nearest Neighbor Distance Based Outlier Detection Technique
NDoT: Nearest Neighbor Distance Based Outlier Detection Technique Neminath Hubballi 1, Bidyut Kr. Patra 2, and Sukumar Nandi 1 1 Department of Computer Science & Engineering, Indian Institute of Technology
More informationLarge Scale Data Analysis for Policy
Large Scale Data Analysis for Policy 90-866, Fall 2012 Lecture 9: Anomaly and Outlier Detection Parts of this lecture were adapted from Banerjee et al., Anomaly Detection: A Tutorial, presented at SDM
More informationInternational Journal of Data Mining & Knowledge Management Process (IJDKP) Vol.7, No.3, May Dr.Zakea Il-Agure and Mr.Hicham Noureddine Itani
LINK MINING PROCESS Dr.Zakea Il-Agure and Mr.Hicham Noureddine Itani Higher Colleges of Technology, United Arab Emirates ABSTRACT Many data mining and knowledge discovery methodologies and process models
More informationClassification Part 4
Classification Part 4 Dr. Sanjay Ranka Professor Computer and Information Science and Engineering University of Florida, Gainesville Model Evaluation Metrics for Performance Evaluation How to evaluate
More informationImproved Classification of Known and Unknown Network Traffic Flows using Semi-Supervised Machine Learning
Improved Classification of Known and Unknown Network Traffic Flows using Semi-Supervised Machine Learning Timothy Glennan, Christopher Leckie, Sarah M. Erfani Department of Computing and Information Systems,
More informationCS6375: Machine Learning Gautam Kunapuli. Mid-Term Review
Gautam Kunapuli Machine Learning Data is identically and independently distributed Goal is to learn a function that maps to Data is generated using an unknown function Learn a hypothesis that minimizes
More informationIntrusion Detection System using AI and Machine Learning Algorithm
Intrusion Detection System using AI and Machine Learning Algorithm Syam Akhil Repalle 1, Venkata Ratnam Kolluru 2 1 Student, Department of Electronics and Communication Engineering, Koneru Lakshmaiah Educational
More informationEvaluating Classifiers
Evaluating Classifiers Reading for this topic: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website) Evaluating Classifiers What we want: Classifier that best predicts
More informationAutomatic Detection Of Suspicious Behaviour
University Utrecht Technical Artificial Intelligence Master Thesis Automatic Detection Of Suspicious Behaviour Author: Iris Renckens Supervisors: Dr. Selmar Smit Dr. Ad Feelders Prof. Dr. Arno Siebes September
More informationAn Intelligent Clustering Algorithm for High Dimensional and Highly Overlapped Photo-Thermal Infrared Imaging Data
An Intelligent Clustering Algorithm for High Dimensional and Highly Overlapped Photo-Thermal Infrared Imaging Data Nian Zhang and Lara Thompson Department of Electrical and Computer Engineering, University
More informationAnomaly Detection in Communication Networks
Anomaly Detection in Communication Networks Prof. D. J. Parish High Speed networks Group Department of Electronic and Electrical Engineering D.J.Parish@lboro.ac.uk Loughborough University Overview u u
More informationEVALUATIONS OF THE EFFECTIVENESS OF ANOMALY BASED INTRUSION DETECTION SYSTEMS BASED ON AN ADAPTIVE KNN ALGORITHM
EVALUATIONS OF THE EFFECTIVENESS OF ANOMALY BASED INTRUSION DETECTION SYSTEMS BASED ON AN ADAPTIVE KNN ALGORITHM Assosiate professor, PhD Evgeniya Nikolova, BFU Assosiate professor, PhD Veselina Jecheva,
More informationEvaluating Classifiers
Evaluating Classifiers Reading for this topic: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website) Evaluating Classifiers What we want: Classifier that best predicts
More informationAN IMPROVED DENSITY BASED k-means ALGORITHM
AN IMPROVED DENSITY BASED k-means ALGORITHM Kabiru Dalhatu 1 and Alex Tze Hiang Sim 2 1 Department of Computer Science, Faculty of Computing and Mathematical Science, Kano University of Science and Technology
More informationComparative Analysis of Machine Learning Methods in Anomaly-based Intrusion Detection
Proceedings of the Fourth Engineering Students Conference at Peradeniya (ESCaPe) 2016 Comparative Analysis of Machine Learning Methods in Anomaly-based Intrusion Detection W.D.Y.N. Piyasinghe, K.E.G.A.P.
More informationOUTLIER MINING IN HIGH DIMENSIONAL DATASETS
OUTLIER MINING IN HIGH DIMENSIONAL DATASETS DATA MINING DISCUSSION GROUP OUTLINE MOTIVATION OUTLIERS IN MULTIVARIATE DATA OUTLIERS IN HIGH DIMENSIONAL DATA Distribution-based Distance-based NN-based Density-based
More informationAnomaly Detection in Categorical Datasets with Artificial Contrasts. Seyyedehnasim Mousavi
Anomaly Detection in Categorical Datasets with Artificial Contrasts by Seyyedehnasim Mousavi A Thesis Presented in Partial Fulfillment of the Requirements for the Degree Master of Science Approved October
More informationA Multi-agent Based Cognitive Approach to Unsupervised Feature Extraction and Classification for Network Intrusion Detection
Int'l Conf. on Advances on Applied Cognitive Computing ACC'17 25 A Multi-agent Based Cognitive Approach to Unsupervised Feature Extraction and Classification for Network Intrusion Detection Kaiser Nahiyan,
More informationEvaluation Metrics. (Classifiers) CS229 Section Anand Avati
Evaluation Metrics (Classifiers) CS Section Anand Avati Topics Why? Binary classifiers Metrics Rank view Thresholding Confusion Matrix Point metrics: Accuracy, Precision, Recall / Sensitivity, Specificity,
More informationContents Machine Learning concepts 4 Learning Algorithm 4 Predictive Model (Model) 4 Model, Classification 4 Model, Regression 4 Representation
Contents Machine Learning concepts 4 Learning Algorithm 4 Predictive Model (Model) 4 Model, Classification 4 Model, Regression 4 Representation Learning 4 Supervised Learning 4 Unsupervised Learning 4
More informationDATA MINING AND MACHINE LEARNING. Lecture 6: Data preprocessing and model selection Lecturer: Simone Scardapane
DATA MINING AND MACHINE LEARNING Lecture 6: Data preprocessing and model selection Lecturer: Simone Scardapane Academic Year 2016/2017 Table of contents Data preprocessing Feature normalization Missing
More informationOutlier Detection. Chapter 12
Contents 12 Outlier Detection 3 12.1 Outliers and Outlier Analysis.................... 4 12.1.1 What Are Outliers?..................... 4 12.1.2 Types of Outliers....................... 5 12.1.3 Challenges
More informationRole of big data in classification and novel class detection in data streams
DOI 10.1186/s40537-016-0040-9 METHODOLOGY Open Access Role of big data in classification and novel class detection in data streams M. B. Chandak * *Correspondence: hodcs@rknec.edu; chandakmb@gmail.com
More informationChapter 3: Supervised Learning
Chapter 3: Supervised Learning Road Map Basic concepts Evaluation of classifiers Classification using association rules Naïve Bayesian classification Naïve Bayes for text classification Summary 2 An example
More informationStatistics 202: Data Mining. c Jonathan Taylor. Outliers Based in part on slides from textbook, slides of Susan Holmes.
Outliers Based in part on slides from textbook, slides of Susan Holmes December 2, 2012 1 / 1 Concepts What is an outlier? The set of data points that are considerably different than the remainder of the
More informationCPSC 340: Machine Learning and Data Mining. Outlier Detection Fall 2018
CPSC 340: Machine Learning and Data Mining Outlier Detection Fall 2018 Admin Assignment 2 is due Friday. Assignment 1 grades available? Midterm rooms are now booked. October 18 th at 6:30pm (BUCH A102
More informationCS249: ADVANCED DATA MINING
CS249: ADVANCED DATA MINING Classification Evaluation and Practical Issues Instructor: Yizhou Sun yzsun@cs.ucla.edu April 24, 2017 Homework 2 out Announcements Due May 3 rd (11:59pm) Course project proposal
More informationNetwork Traffic Measurements and Analysis
DEIB - Politecnico di Milano Fall, 2017 Introduction Often, we have only a set of features x = x 1, x 2,, x n, but no associated response y. Therefore we are not interested in prediction nor classification,
More informationComputer Technology Department, Sanjivani K. B. P. Polytechnic, Kopargaon
Outlier Detection Using Oversampling PCA for Credit Card Fraud Detection Amruta D. Pawar 1, Seema A. Dongare 2, Amol L. Deokate 3, Harshal S. Sangle 4, Panchsheela V. Mokal 5 1,2,3,4,5 Computer Technology
More informationADVANCES in NATURAL and APPLIED SCIENCES
ADVANCES in NATURAL and APPLIED SCIENCES ISSN: 1995-0772 Published BYAENSI Publication EISSN: 1998-1090 http://www.aensiweb.com/anas 2017 February 11(2): pages 14-18 Open Access Journal A Novel Framework
More informationIMPROVING THE PERFORMANCE OF OUTLIER DETECTION METHODS FOR CATEGORICAL DATA BY USING WEIGHTING FUNCTION
IMPROVING THE PERFORMANCE OF OUTLIER DETECTION METHODS FOR CATEGORICAL DATA BY USING WEIGHTING FUNCTION 1 NUR ROKHMAN, 2 SUBANAR, 3 EDI WINARKO 1 Gadjah Mada University, Department of Computer Science
More informationCluster Based detection of Attack IDS using Data Mining
Cluster Based detection of Attack IDS using Data Mining 1 Manisha Kansra, 2 Pankaj Dev Chadha 1 Research scholar, 2 Assistant Professor, 1 Department of Computer Science Engineering 1 Geeta Institute of
More informationWeka ( )
Weka ( http://www.cs.waikato.ac.nz/ml/weka/ ) The phases in which classifier s design can be divided are reflected in WEKA s Explorer structure: Data pre-processing (filtering) and representation Supervised
More informationINF4820, Algorithms for AI and NLP: Evaluating Classifiers Clustering
INF4820, Algorithms for AI and NLP: Evaluating Classifiers Clustering Erik Velldal University of Oslo Sept. 18, 2012 Topics for today 2 Classification Recap Evaluating classifiers Accuracy, precision,
More informationKeywords: Clustering, Anomaly Detection, Multivariate Outlier Detection, Mixture Model, EM, Visualization, Explanation, Mineset.
ISSN 2319-8885 Vol.03,Issue.35 November-2014, Pages:7140-7144 www.ijsetr.com Accurate and Efficient Anomaly Detection via Online Oversampling Principal Component Analysis K. RAJESH KUMAR 1, S.S.N ANJANEYULU
More informationFraud Detection using Machine Learning
Fraud Detection using Machine Learning Aditya Oza - aditya19@stanford.edu Abstract Recent research has shown that machine learning techniques have been applied very effectively to the problem of payments
More informationINTRODUCTION TO DATA MINING. Daniel Rodríguez, University of Alcalá
INTRODUCTION TO DATA MINING Daniel Rodríguez, University of Alcalá Outline Knowledge Discovery in Datasets Model Representation Types of models Supervised Unsupervised Evaluation (Acknowledgement: Jesús
More informationCombination of PCA with SMOTE Resampling to Boost the Prediction Rate in Lung Cancer Dataset
International Journal of Computer Applications (0975 8887) Combination of PCA with SMOTE Resampling to Boost the Prediction Rate in Lung Cancer Dataset Mehdi Naseriparsa Islamic Azad University Tehran
More informationCOSC160: Detection and Classification. Jeremy Bolton, PhD Assistant Teaching Professor
COSC160: Detection and Classification Jeremy Bolton, PhD Assistant Teaching Professor Outline I. Problem I. Strategies II. Features for training III. Using spatial information? IV. Reducing dimensionality
More informationInternet Traffic Classification using Machine Learning
Internet Traffic Classification using Machine Learning by Alina Lapina 2018, UiO, INF5050 Alina Lapina, Master student at IFI, Full stack developer at Ciber Experis 2 Based on Thuy T. T. Nguyen, Grenville
More informationContents. Preface to the Second Edition
Preface to the Second Edition v 1 Introduction 1 1.1 What Is Data Mining?....................... 4 1.2 Motivating Challenges....................... 5 1.3 The Origins of Data Mining....................
More informationCS4491/CS 7265 BIG DATA ANALYTICS
CS4491/CS 7265 BIG DATA ANALYTICS EVALUATION * Some contents are adapted from Dr. Hung Huang and Dr. Chengkai Li at UT Arlington Dr. Mingon Kang Computer Science, Kennesaw State University Evaluation for
More informationUsing Real-valued Meta Classifiers to Integrate and Contextualize Binding Site Predictions
Using Real-valued Meta Classifiers to Integrate and Contextualize Binding Site Predictions Offer Sharabi, Yi Sun, Mark Robinson, Rod Adams, Rene te Boekhorst, Alistair G. Rust, Neil Davey University of
More informationCPSC 340: Machine Learning and Data Mining. Outlier Detection Fall 2016
CPSC 340: Machine Learning and Data Mining Outlier Detection Fall 2016 Admin Assignment 1 solutions will be posted after class. Assignment 2 is out: Due next Friday, but start early! Calculus and linear
More informationNMLRG #4 meeting in Berlin. Mobile network state characterization and prediction. P.Demestichas (1), S. Vassaki (2,3), A.Georgakopoulos (2,3)
NMLRG #4 meeting in Berlin Mobile network state characterization and prediction P.Demestichas (1), S. Vassaki (2,3), A.Georgakopoulos (2,3) (1)University of Piraeus (2)WINGS ICT Solutions, www.wings-ict-solutions.eu/
More informationnode2vec: Scalable Feature Learning for Networks
node2vec: Scalable Feature Learning for Networks A paper by Aditya Grover and Jure Leskovec, presented at Knowledge Discovery and Data Mining 16. 11/27/2018 Presented by: Dharvi Verma CS 848: Graph Database
More informationDetection and Deletion of Outliers from Large Datasets
Detection and Deletion of Outliers from Large Datasets Nithya.Jayaprakash 1, Ms. Caroline Mary 2 M. tech Student, Dept of Computer Science, Mohandas College of Engineering and Technology, India 1 Assistant
More informationAn Unsupervised Approach for Combining Scores of Outlier Detection Techniques, Based on Similarity Measures
An Unsupervised Approach for Combining Scores of Outlier Detection Techniques, Based on Similarity Measures José Ramón Pasillas-Díaz, Sylvie Ratté Presenter: Christoforos Leventis 1 Basic concepts Outlier
More informationAnomalydetection in massive datasets
Anomalydetection in massive datasets Ennio Ottaviani OnAIR s.r.l. ennio.ottaviani@onairweb.com University of Genova, Dept. Of Mathematics, SMID ennio.ottaviani@edu.unige.it 1 Outline 1. Introduction 2.
More informationLecture 6 K- Nearest Neighbors(KNN) And Predictive Accuracy
Lecture 6 K- Nearest Neighbors(KNN) And Predictive Accuracy Machine Learning Dr.Ammar Mohammed Nearest Neighbors Set of Stored Cases Atr1... AtrN Class A Store the training samples Use training samples
More informationAdvancements of Data Anomaly Detection Research in Wireless Sensor Networks: A Survey and Open Issues
Sensors 2013, 13, 10087-10122; doi:10.3390/s130810087 Review OPEN ACCESS sensors ISSN 1424-8220 www.mdpi.com/journal/sensors Advancements of Data Anomaly Detection Research in Wireless Sensor Networks:
More informationEvaluating Classifiers
Evaluating Classifiers Charles Elkan elkan@cs.ucsd.edu January 18, 2011 In a real-world application of supervised learning, we have a training set of examples with labels, and a test set of examples with
More informationAn Anomaly-Based Intrusion Detection System for the Smart Grid Based on CART Decision Tree
An Anomaly-Based Intrusion Detection System for the Smart Grid Based on CART Decision Tree P. Radoglou-Grammatikis and P. Sarigiannidis* University of Western Macedonia Department of Informatics & Telecommunications
More informationClassification. Instructor: Wei Ding
Classification Part II Instructor: Wei Ding Tan,Steinbach, Kumar Introduction to Data Mining 4/18/004 1 Practical Issues of Classification Underfitting and Overfitting Missing Values Costs of Classification
More informationChapter 5: Outlier Detection
Ludwig-Maximilians-Universität München Institut für Informatik Lehr- und Forschungseinheit für Datenbanksysteme Knowledge Discovery in Databases SS 2016 Chapter 5: Outlier Detection Lecture: Prof. Dr.
More informationUnsupervised learning in Vision
Chapter 7 Unsupervised learning in Vision The fields of Computer Vision and Machine Learning complement each other in a very natural way: the aim of the former is to extract useful information from visual
More informationEster Bernadó-Mansilla. Research Group in Intelligent Systems Enginyeria i Arquitectura La Salle Universitat Ramon Llull Barcelona, Spain
Learning Classifier Systems for Class Imbalance Problems Research Group in Intelligent Systems Enginyeria i Arquitectura La Salle Universitat Ramon Llull Barcelona, Spain Aim Enhance the applicability
More informationData Clustering With Leaders and Subleaders Algorithm
IOSR Journal of Engineering (IOSRJEN) e-issn: 2250-3021, p-issn: 2278-8719, Volume 2, Issue 11 (November2012), PP 01-07 Data Clustering With Leaders and Subleaders Algorithm Srinivasulu M 1,Kotilingswara
More informationUnsupervised Learning
Unsupervised Learning Unsupervised learning Until now, we have assumed our training samples are labeled by their category membership. Methods that use labeled samples are said to be supervised. However,
More informationClassification Algorithms in Data Mining
August 9th, 2016 Suhas Mallesh Yash Thakkar Ashok Choudhary CIS660 Data Mining and Big Data Processing -Dr. Sunnie S. Chung Classification Algorithms in Data Mining Deciding on the classification algorithms
More informationPCA Based Anomaly Detection
PCA Based Anomaly Detection P. Rameswara Anand 1,, Tulasi Krishna Kumar.K 2 Department of Computer Science and Engineering, Jigjiga University, Jigjiga, Ethiopi 1, Department of Computer Science and Engineering,Yogananda
More informationSELECTION OF OPTIMAL MINING ALGORITHM FOR OUTLIER DETECTION - AN EFFICIENT METHOD TO PREDICT/DETECT MONEY LAUNDERING CRIME IN FINANCE INDUSTRY
P P P-ISSN: 2347-4408 SELECTION OF OPTIMAL MINING ALGORITHM FOR OUTLIER DETECTION - AN EFFICIENT METHOD TO PREDICT/DETECT MONEY LAUNDERING CRIME IN FINANCE INDUSTRY Abstract 1 1 Kannan SP P, Dr. K. SomasundaramP
More informationThanks to the advances of data processing technologies, a lot of data can be collected and stored in databases efficiently New challenges: with a
Data Mining and Information Retrieval Introduction to Data Mining Why Data Mining? Thanks to the advances of data processing technologies, a lot of data can be collected and stored in databases efficiently
More informationFurther Thoughts on Precision
Further Thoughts on Precision David Gray, David Bowes, Neil Davey, Yi Sun and Bruce Christianson Abstract Background: There has been much discussion amongst automated software defect prediction researchers
More informationData Mining. Introduction. Hamid Beigy. Sharif University of Technology. Fall 1395
Data Mining Introduction Hamid Beigy Sharif University of Technology Fall 1395 Hamid Beigy (Sharif University of Technology) Data Mining Fall 1395 1 / 21 Table of contents 1 Introduction 2 Data mining
More informationPattern recognition (4)
Pattern recognition (4) 1 Things we have discussed until now Statistical pattern recognition Building simple classifiers Supervised classification Minimum distance classifier Bayesian classifier (1D and
More informationNetwork Traffic Measurements and Analysis
DEIB - Politecnico di Milano Fall, 2017 Sources Hastie, Tibshirani, Friedman: The Elements of Statistical Learning James, Witten, Hastie, Tibshirani: An Introduction to Statistical Learning Andrew Ng:
More informationK- Nearest Neighbors(KNN) And Predictive Accuracy
Contact: mailto: Ammar@cu.edu.eg Drammarcu@gmail.com K- Nearest Neighbors(KNN) And Predictive Accuracy Dr. Ammar Mohammed Associate Professor of Computer Science ISSR, Cairo University PhD of CS ( Uni.
More informationHKBU Institutional Repository
Hong Kong Baptist University HKBU Institutional Repository HKBU Staff Publication 2016 Traffic outlier detection by density-based bounded local outlier factors Jialing Tang Hong Kong Baptist University,
More informationClustering Algorithms for Data Stream
Clustering Algorithms for Data Stream Karishma Nadhe 1, Prof. P. M. Chawan 2 1Student, Dept of CS & IT, VJTI Mumbai, Maharashtra, India 2Professor, Dept of CS & IT, VJTI Mumbai, Maharashtra, India Abstract:
More informationMining Of Inconsistent Data in Large Dataset In Distributed Environment
Mining Of Inconsistent Data in Large Dataset In Distributed Environment M.Shanthini 1 Department of Computer Science and Engineering, Syed Ammal Engineering College, Ramanathapuram, Tamilnadu, India 1
More informationBasic Concepts in Intrusion Detection
Technology Technical Information Services Security Engineering Roma, L Università Roma Tor Vergata, 23 Aprile 2007 Basic Concepts in Intrusion Detection JOVAN GOLIĆ Outline 2 Introduction Classification
More informationAnomaly Detection Based on Access Behavior and Document Rank Algorithm
Anomaly Detection Based on Access Behavior and Document Rank Algorithm Prajwal R Thakare, M.Tech IT Dept, ASTRA, Bandlaguda, Abstract:-Distributed denial of service (DDoS) attack is ongoing dangerous threat
More informationLecture 25: Review I
Lecture 25: Review I Reading: Up to chapter 5 in ISLR. STATS 202: Data mining and analysis Jonathan Taylor 1 / 18 Unsupervised learning In unsupervised learning, all the variables are on equal standing,
More informationAn Abnormal Data Detection Method Based on the Temporal-spatial Correlation in Wireless Sensor Networks
An Based on the Temporal-spatial Correlation in Wireless Sensor Networks 1 Department of Computer Science & Technology, Harbin Institute of Technology at Weihai,Weihai, 264209, China E-mail: Liuyang322@hit.edu.cn
More informationMIT Samberg Center Cambridge, MA, USA. May 30 th June 2 nd, by C. Rea, R.S. Granetz MIT Plasma Science and Fusion Center, Cambridge, MA, USA
Exploratory Machine Learning studies for disruption prediction on DIII-D by C. Rea, R.S. Granetz MIT Plasma Science and Fusion Center, Cambridge, MA, USA Presented at the 2 nd IAEA Technical Meeting on
More informationSampling Large Graphs for Anticipatory Analysis
Sampling Large Graphs for Anticipatory Analysis Lauren Edwards*, Luke Johnson, Maja Milosavljevic, Vijay Gadepally, Benjamin A. Miller IEEE High Performance Extreme Computing Conference September 16, 2015
More informationData Mining. Introduction. Hamid Beigy. Sharif University of Technology. Fall 1394
Data Mining Introduction Hamid Beigy Sharif University of Technology Fall 1394 Hamid Beigy (Sharif University of Technology) Data Mining Fall 1394 1 / 20 Table of contents 1 Introduction 2 Data mining
More informationPart I. Hierarchical clustering. Hierarchical Clustering. Hierarchical clustering. Produces a set of nested clusters organized as a
Week 9 Based in part on slides from textbook, slides of Susan Holmes Part I December 2, 2012 Hierarchical Clustering 1 / 1 Produces a set of nested clusters organized as a Hierarchical hierarchical clustering
More informationCyber-physical intrusion detection on a robotic vehicle
WIFS 2015 The 7th IEEE International Workshop on Information Forensics and Security Rome, Italy, 16-19 November, 2015 Cyber-physical intrusion detection on a robotic vehicle Tuan Vuong, George Loukas and
More informationDATA MINING FOR NETWORK INTRUSION DETECTION. Vipin Kumar
DATA MINING FOR NETWORK INTRUSION DETECTION Vipin Kumar Army High Performance Computing Research Center Department of Computer Science University of Minnesota http://www.cs.umn.edu/~kumar Collaborators:
More informationScalable Selective Traffic Congestion Notification
Scalable Selective Traffic Congestion Notification Győző Gidófalvi Division of Geoinformatics Deptartment of Urban Planning and Environment KTH Royal Institution of Technology, Sweden gyozo@kth.se Outline
More informationCPSC 340: Machine Learning and Data Mining. Non-Parametric Models Fall 2016
CPSC 340: Machine Learning and Data Mining Non-Parametric Models Fall 2016 Assignment 0: Admin 1 late day to hand it in tonight, 2 late days for Wednesday. Assignment 1 is out: Due Friday of next week.
More informationLecture Notes on Critique of 1998 and 1999 DARPA IDS Evaluations
Lecture Notes on Critique of 1998 and 1999 DARPA IDS Evaluations Prateek Saxena March 3 2008 1 The Problems Today s lecture is on the discussion of the critique on 1998 and 1999 DARPA IDS evaluations conducted
More informationA novel supervised learning algorithm and its use for Spam Detection in Social Bookmarking Systems
A novel supervised learning algorithm and its use for Spam Detection in Social Bookmarking Systems Anestis Gkanogiannis and Theodore Kalamboukis Department of Informatics Athens University of Economics
More information