Machine Learning. No: It Can t Do That! Hadi Nahari hadinahari. Copyright Cognomotiv 2016

Transcription

1 Machine Learning No: It Can t Do That! Hadi Nahari hadi@cognomotiv.com hadinahari

2 Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them Julius Caesar Act 3, Scene II

3 Setup ML + NetSec

4 National Academy of Engineering Grand Challenges for 21 st Century "The best minds of my generation are thinking about how to make people click ads. ---Jeff Hammerbacher

5 Agenda Motivations Machine Learning 101 ML & Network Security What Works, What Doesn t Conclusion 5 / 50

6 Agenda MOTIVATIONS

7 ML Is NOT New This is the 5 th round

8 ML is HOT!! VCs fund ML-companies like crazy Amazing new fields have opened Autonomous driving, behavior analytics, etc. Ton of existing fields have been revived Search, personalization/customization, audio processing, image processing, etc., etc.

9 Mainly because

10 Code Complexity Space Shuttle: ~400K LOC F22 Raptor fighter: ~2M LOC Linux kernel 2.2: ~2.5M LOC Hubble telescope: ~3M LOC Android core: ~12M LOC Future Combat Sys.: ~63M LOC Connected car: ~100M LOC Autonomous vehicle: ~300M LOC 10 / 50

11 Autonomous vehicle: ~300M LOC 50 M LOC Large Hadron Collider: 60 M LOC

12 Usecase Complexity service provider on avg. only five passwords per 40 online accounts per user where to store the tokens???

13 Data Procreation >2 billion GB of new data is created every day B GB to be exact Sparse data: mainly 0s In 93 the information on the internet surpassed all information that humanity had created before it

14 Stack Proliferation Applications HW Architecture(s)

15 Algorithms 15 / 50

16 Algorithms

17 Agenda ML 101

18 Machine Learning (ML) Study of pattern recognition & computational learning theory in Artificial Intelligence (AI) Algorithms to learn from, and make predictions on data As opposed to following strictly static program instructions

19 ML Models Supervised learning Unsupervised learning (Semi-supervised learning) Reinforcement learning

20 Supervised Learning {(labeled) Input} [map] {Expected Output} Find [map] 20 / 50

21 Supervised Learning Model

22 Unsupervised Learning {(unlabled) Input} [map] {Output} Find structure (patterns) in {Input}

23 Unsupervised Learning Model

24 Reinforcement Learning No correct {Input}/{Output} Action, environment, reward

25 Reinforcement Learning Model 25 / 50

26 Main ML Approaches Decision Tree Learning, Association Rule Learning Inductive Logic Programming, Support Vector Machines, Clustering, Bayesian Networks Representation Learning, Genetic Algorithms Similarity and Metric Learning, Sparse Dictionary Learning Artificial Neural Networks (ANN), Deep Learning (DL)

27 Neural Network Interpret an Artificial Intelligence (AI) task as the evaluation of complex functions Facial Recognition: Map a bunch of pixels to a name Handwriting Recognition: Image to a character NN: Network of interconnected simple neurons

28 The Neuron Feed-forward system, made up of two stages: Linear Transformation of data Point-wise application of non-linear function y i =F(ΣW i X i ) i W 1 W 2 W 3 F(x) =max(0,x) (also sigmoid, Rectified Linear Unit (ReLU), etc.) X X X 1 2 3

29 Artificial Neural Network (ANN) Layers and layers of neurons, with many connections Output: Input:

30 Deep Learning (DL) Branch of ML based on a set of algorithms that: Attempt to model high-level data abstractions Are based on learning representations of data Use complex architectures with multiple non-linear transformations Some representations make it easier to learn tasks from examples (e.g. Alpha Go) 30 / 50

31 DNN: Learning Feature Representation Input Result

32 DNN: Feature Engineering Images/video Image Vision features Detection Audio Audio Audio features Speaker ID Text Text Text features Anything humans can do in 0.1 sec, the right, big 10-layer network can do too Text classification, Machine translation, Information retrieval,...

33 ML/DL Improve With Scale Past Present Future Performance ML / DL Many previous methods Data & Compute

34 Agenda ML & NETSEC

35 Intrusion & Intrusion Detection Intrusion is an attempt to compromise CIA (Confidentiality, Integrity, Availability), or to bypass the security mechanisms of a computer or network Intrusion detection is the process of monitoring the events occurring in a computer system or network, and analyzing them for signs of an intrusion 35 / 50

36 3 Main Detection Methodologies Signature-based Detection (SD) Signature: pattern corresponding to known attack or threat SD: process to compare patterns against captured events A.K.A Knowledge-based Detection Anomaly-based Detection (AD) Anomaly is a deviation to normal behavior Profile of normal is derived from monitoring network traffic AD compares normal profile with observed events Stateful Protocol Analysis (SPA) Vendor-developed generic profiles to specific protocols

37 Cybersecurity System Attacks evolve, ergo building defense systems is nontrivial Thus, higher-level & adaptive methodologies are required

38 Adaptive Cybersecurity Data-capturing tools (Libpcap, Winpcap, etc.) capture events from the audit trails of information sources (e.g. network) Data-preprocessing module filters out the attacks from which good signatures have been learned A feature-extractor derives basic features (sequence of syscalls, start time, NetFlow duration, src/dest IP/port, protocol, byte and packet counts Analysis engine implements detection methods for infrastructure anomalies, which may or may not have appeared before

39 Agenda WHAT WORKS WHAT DOESN T

40 Curse of Dimensionality Data volume is massive min. ~100M events per day Much of the data is streaming data Requires inline, real-time analysis Feature space is high dimensional 40 / 50

41 $/Detection Performance Abysmal Looking for every anomaly is cost prohibitive if at all [practically] possible Narrowing down the criteria too much results in false negative Reference data hard to gain due to privacy concerns Simulated data is useless ML was supposed to be better than signature era

42 Husky Recognition

43 Learned Features We built an effective snow recognition model

44 Models: Simple Correlations Simple models are also (usually) wrong

45 Network Anomalies Malicious data packets have a small variety (low type-count), but happen in high frequency Current models are not good at detecting this type of anomaly Anomaly/outlier varies among application domains Labeled anomalies are not available for training/validation 45 / 50

46 Baselining Using ML to detect anomaly is easy when baseline is well-defined and follows simple mathematical model (e.g. Normal Distribution) Most real-world systems don t render a simple baseline (i.e. their behavior is very complex) [!]Sanctity of baseline: nearly 100% of networks are compromised

47 Time Shifting Window problem : algos should be limited to ingest data in chunks that can be processed What if the anomaly is seeded outside that window? Network traffic diversity: usage varies in every session and with new applications window should also be shifted for recurring training Serious impact on performance, real-time, and security

48 There s More How do you trust what the model predicts? i.e. how do we know the model works correctly (husky)? Designing sound evaluation schemes can be more difficult than the detector itself We really don t know how ML works or how to reason about ML models or how to debug them For now it s just magic & voodoo

49 Agenda CONCLUSION

50 Summary ML is a great and necessary technology ML really shines for some classes of problems ML is NOT the best solution for every problem (e.g. NetSec) Obtaining (and training with) useful data remains a challenge ML is just one initial building block of Machine Cognition and Artificial Understanding: there are many more Still a long way before machines can replicate humans! 50 / 50

51 THANK YOU! Hadi Nahari hadinahari

52 Backup

53 References Prof. Karl Friston seminal works ( Why Should I Trust You? Explaining the Predictions of Any Classifier, Carlos Guestrin, et al ( Using Machine Learning in Network Intrusion Detection Systems, Omar Shaya ( Machine Learning Is Not The Answer To Better Network Security, Matt Harrigan ( Machine Learning Algorithm Cheat Sheet, Laura Diane Hamilton, ( Anomaly Detection Approaches for Communicating Networks ( A Survey on Machine Learning Techniques for Intrusion Detection Systems, J. Sing, N.J. Nene, ( Machine Learning Techniques for Anomaly Detection: An Overview, S. Omar, et al, ( Recent Advances in Predictive (Machine) Learning, J.H. Friedman, et al, ( Outside the Closed World: On Using Machine Learning For Network Intrusion Detection, R. Sommer, V. Paxson, (

54 Are Humans Getting Smarter? IQ scores are rising Underlying biological HW declining Intelligence is in decline