Big Data Challenges in Large IP Networks

Transcription

1 Big Data Challenges in Large IP Networks Feature Extraction & Predictive Alarms for network management Wednesday 28 th Feb 2018 Dave Yearling British Telecommunications plc 2017

2 What we will cover Making data connections for our Customers Each Router 10 line cards Each Line card maps 40 ports Performance monitored (every 15minutes) via 12 metrics 30m Premises 5k Exchanges 1k Edge Routers Sensors on Chassis, CPU.. 5m time series 0.5 billion obs/day The challenge - predicting failures in routers for in time recovery Cannot predict everything (e.g. failure due to power outage, third party damage, ) But perhaps we can Summarise metrics to hourly Extract Features here we focus on one technique Cortical Learning Algorithm (CLA) Predict line card failures & generate informative alarms Solutions need to be deployed on Big Data platform Large numbers of heterogeneous times series Scaling algorithms for real time streaming data, 1 observation at a time More than one mode of failure, failures are rare Predictions need to have low false positive rate

3 Vital for our customers What we do now and our aim IP Traffic and load set to grow massively 4G (and future 5G) relies on 21 CN QoS Customer tolerance of poor service will decrease & 1 Outage can affect many customers Monitor and apply threshold alarms Some straightforward & essential e.g. Temperature Others complex & poorly understood Many false alarms, nothing recorded for near miss Many do not give Next Best Action' Fast Reactive Logic Provides swift reaction but After the event Dynamically monitor customers on demand testing Identifies common network elements Provides a Next Best Action Timely Predictive Logic Currently NO Predictive capability for raw metrics Distil, model and apply On line 1 record at a time Provides a probabilistic view of all network elements augmenting alarms Storing multiple models & data e.g. Card Failure, Degraded Service,

4 The road map for predicting card failures Resource A Metrics For each router, card, port, A B C D E F G H Resource H Metrics but there are Challenges 1. PROCESS For Every Resource/Metric Clean and snap to regular intervals 1. PROCESS Some detectors require windows of time for each point of measurement 2. FEATURE EXTRACTION Run Anomaly Detector Suite Data reduction Produce patterns for Diagnostics 3. PREDICTION Data Volume greatly reduced Only process anomalies Continuous M/Learning Desirable 2. FEATURE EXTRACTION Heterogeneous time series so we are planning an ensemble Running at real time is difficult Business lacks highly specific expertise - PhDs with Lancaster Multivariate structure Missing values 3. PREDICTION Input summary is heuristic but done once for all predictive models Rare events means we have to do sandpit builds Rate of change of probabilities may provide more information Current Ensemble include Cortical Learning Algorithm PELT/Binary Segmentation Hybrid Extreme Stud/Dev HOT SAX 4. APPLY Record Probabilities for each outcome Flag Warnings & visualise probabilities (in space & time) 4. APPLY Need to capture feedback from operations in order to tune Consuming this into the Operations is non trivial

5 Wouldn t it be great if we had an algorithm that behaves just like an operation analyst We need Feature Extraction to flag strangeness from heterogeneous time series METRIC I METRIC II Resource A Resource A Must cope with Periodicity Huge Transients Developing Anomalies Resource B Resource B And detect changes in Baseline Mean Variance Resource C Resource C And have low false alert rate Point of Failure Point of Failure Strangeness here are outliers : an observation (or a subset of observations) which appears to be inconsistent with the remainder of that set of data* * Barnett & Lewis, 1994

6 Cortical Learning Algorithm (CLA) Follows principles of the human neocortex Higher functions accomplished mostly within uniform crinkled surface of neocortex Different functions (vision, speech..) associated with lobes but very importantly Functions operate in the same way & on the same data signals We recognise/solve/produce behaviour by storing & recalling memories as SEQUENCES These SEQUENCES are stored in an INVARIANT and NOT Exhaustive way Continuously compares input with its own predictions on what to expect next In essence CLA is a self learning anomaly detection platform handles streaming data 1 observation at a time requires to store model not time series big advantage with many tuning parameters Please see Numenta Platform for Intelligent Computer (NuPIC) Numenta.org for much more detail on CLA

7 Cortical Learning Algorithm Anomaly Detector in a nutshell Inbound Data Stream ENCODER ENCODE observation at time t into binary vector All Columns relate to some region of input SPARSE DISTRIBUTED REPRESENTATION Cortical Columns representing the Layers Use approach 32 cells per column e.g of 2k bits only 2% (i.e. 40 bits) can be set to 1, and each of these have semantic meaning SPATIAL POOLER TEMPORAL POOLER ANOMALY CALCULATION RECOGNISE input, similar vectors - similar columns are active 1. Calculate Overlap for each column (a degree of how well they map) 2. Activate top 2% of Overlap columns, subject to Inhibition 3. Adjust perm for all synapses on Activated Columns - learning 1. Each synapse which has a degree of permanence perm (0,1) 2. Allows synapse to fire when perm > τ (some threshold) PREDICT next input vector based on the current/previous states For cells in each Active column : 1. Switch all PREDICTIVE cells to ACTIVE, if NONE make ALL cells to ACTIVE 2. Inter column connections are formed/reinforced between all ACTIVE cells in ACTIVE columns to the WINNING CELLS at previous time step 3. Set a cell as most predictive WINNING Cell in each ACTIVE COLUMN (according to a set of rules) ANOMALY based on the correctness of previous prediction(s) (i) % of spatial pooler at t incorrectly predicted via temporal pooler at t-1 (ii) Calculate RECENT Moving Average μ t compare with LONG TERM M/Average μ t Compare tail probability μ t μ t ~N(0,1) σ t 2,048 columns all map back to fixed area on the SDR (all cells in column inherit this) cells have states INACTIVE PREDICTIVE ACTIVE cells connect to cells in other columns These are connected & disconnected by using perm with threshold

8 Cortical Learning Algorithm Experiment & example time series CARD OUTAGE Thought experiment Measure all series at 5 days prior to any Card outage (for Targets - YES) and 15 th Sept (for Controls - NO) Cohorts of circuits YES Any Card Outage in 5 days, 158 observations NO No Outage of any kind, 648 observations Simple Heuristic At each point we sum all weighted anomalies detected in the past 15 days (s.t. anomalies on the t -15 th day are worth 10% compared to t)

9 Cortical Learning Algorithm Experiment & example time series CARD OUTAGE Thought experiment Measure all series at 5 days prior to any Card outage (for Targets - YES) and 15 th Sept (for Controls - NO) Cohorts of circuits YES Any Card Outage in 5 days, 158 observations NO No Outage of any kind, 648 observations Simple Heuristic At each point we sum all weighted anomalies detected in the past 15 days (s.t. anomalies on the t -15 th day are worth 10% compared to t)

10 Cortical Learning Algorithm Experiment Likelihood Threshold 0.95, 15 day exponential weighted window Comparing all circuits within both YES and NO Card Failure Cohorts xperi Day 0 = Day 1 = Day 15 = 0. 1

11 In Summary Recap and Current Focus The Techniques The Business Current Focus CLA..has promise Deals with streaming but Not currently in real time Anomaly flagging can be inconsistent Parameter tuning exploration When to switch off/on learning Other Ensemble methods Typically much faster, but: generates lots of anomalies relies on window processing Require more pre-processing Missing Values - Robust ARIMA & Kalman Filter Multivariate structure not yet dealt with Monitoring these data streams and suggesting intervention remains daunting Outcomes we wish to predict are: not always cleanly defined (e.g. Error occurred somewhere within the router) very rare occurrences Moving the models & detectors to the data difficult multi-tenanted Hadoop environment operational systems security & long release cycles Continued work on Feature Extraction - augmenting with current alarm output Remove redundancy non informative or collinear metrics Collaboration with operational analysts & software engineers Provide Feature Extraction results for root cause analysis