Deriving Network Traffic Signatures via Large Graphs

Transcription

1 Deriving Network Traffic Signatures via Large Graphs Ahmed Abdelhadi (PI) Research Assistant Professor

2 Outline Pattern of Life and IoT A Tractable Framework for POL Modeling Dynamic Behavior Anomaly Detection 2

3 Outline Pattern of Life and IoT A Tractable Framework for POL Modeling Dynamic Behavior Anomaly Detection 3

4 Pattern of Life Detection Pattern of Life Calculations Collection and analysis of vast amount of data for repetitive and unique patterns. Changes in data patterns over time points to anomalies. How to leverage POL? Careful Feature Selection Through behavior tracking of targets over time. Importance of POL in Modern Networks Can be leveraged to analyze actual human/m2m interactions. Accurate pointer to global traits and as well as local patterns. Main Goal - Automation of analysis and inference. 4

5 Pattern of Life for Internet of Things Why is IoT good for POL? Presents a rich data environment for pattern modeling Multitude of flows and interactions within IoT networks Enables multi-dimensional analysis of packet based network traffic POL analysis geared to be agnostic to deep packet inspection Modeling node behavior Based on features that can be obtained from NetFlow analysis of IoT networks 1. Traffic rates between nodes 2. Communication Delay and Inter-arrival times 3. Queue Length at each node 4. Avg. no. of connections per node 5. Node Sensitivity based on common graph centrality measures 5

6 Outline Pattern of Life and IoT A Tractable Framework for POL Modeling Dynamic Behavior Anomaly Detection 6

7 Framework for POL in IoT Proposed Approach 1. Big Data collection from IoT Networks (NetFlow data for example) and building large graphs (order of millions of nodes) over time 2. Coarse Parsing: Cluster Large graphs based on standard centrality and sensitivity methods to form smaller sub-graphs (order of thousands of nodes) 3. Fine Parsing: Model Dynamic Behavior within sub-graphs and perform tracking and anomaly detection within each cluster. 4. Automatically monitor node behavior and flag anomalous behavior 7

8 Outline Pattern of Life and IoT A Tractable Framework for POL Modeling Dynamic Behavior Anomaly Detection 8

9 Modeling Dynamic Behavior Pattern of Life from (Big) Sub-Graphs Feedback FEATURE DISCOVERY ROLE DISCOVERY TEMPORAL ROLE TRANSITION TRACKING Big POL Graph Features Traffic rates Delay Queue Length Avg. no. of connections Node Sensitivity Role Models Local Role Discovery Global Role Discovery Transition Tracking Global Transition Tracking Local Transition Tracking Transition based Clustering 9

10 Modeling Dynamic Behavior: Technical Approach Feature Discovery Select large number of features Time dependent feature matrix Collect time stamps Role Discovery for each node Dimensionality Reduction from large to smaller number of roles Non-negative matrix factorization can be used Assume a feature role transformation matrix such that and 10

11 Behavioral Transition Model Use reduced dimensionality of feature space for behavior tracking Transition model: can be learnt over time from role observations and captures the typical behavior of the network. Learning the transition model: 11

12 Behavioral Transition Model Learning Behavior over time: Stacked Transition Model 1 Squared Error minimized to learn transitions 1 R. A. Rossi, J. Neville, B. Gallagher, K. Henderson, Modeling Dynamic Behavior in Large Evolving Graphs, In Proc. WDSM

13 Outline Pattern of Life and IoT A Tractable Framework for POL Modeling Dynamic Behavior Anomaly Detection 13

14 Anomaly Detection and Clustering Stacked Transition model lends itself to anomaly detection Learn typical network behavior transition over time Use typical pattern to forecast future behavior: Detect anomalies by observing High MSE will lead to anomaly detection 14

15 An Example Small Example with 100 nodes Selected 5 roles for each node Each role had values randomly generated from and exponential distribution with parameter A training phase of 1000 time samples was used to train the transition model A testing phase of time steps was used Artificial anomaly was injected at time steps , , in order to test the detection method When only distribution parameter was changed (shuffled per feature from the same set of values), small anomaly was detected. When distribution changed, large anomaly was accurately detected. 15

16