BRKEWN-3033 DNA Assurance: Deep Dive

Transcription

1

2 BRKEWN-3033 DNA Assurance: Deep Dive Jerome Henry, Principal Engineer

3 Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space cs.co/ciscolivebot#brkewn Cisco and/or its affiliates. All rights reserved. Cisco Public

4 Agenda Introduction DNA Architecture Vision and Technology Surfacing Data Data Collection and Processing Trends vs Issues How trending is computed, what are issues Application Experience Anticipate with Sensors Conclusion

5 DNA Architecture

6 The Cost of Doing Business Spent on Network Operations Labor and Tools $60B * Why are companies spending so much? 95% 70% 75% Network Changes Performed Manually Source: McKinsey study conducted for Cisco in 2016 Policy Violations Due to Human Error OpEx Spent on Network Changes and Troubleshooting BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 6

7 Increasing Demands on the Network Digital Disruption 63 million new devices online every second by Complexity 3X spend on network operations vs network 2 Security 6 months to detect breach 3 Lack of Business and IT Insights Slow and Error Prone Operations Unconstrained Attack Surface 1: Gartner Report - Gartner s 2017 Strategic Roadmap for Networking 2. McKinsey Study of Network Operations for Cisco Ponemon Research Institute Study on Malware Detection, Mar 2016 BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 7

8 Why Transform Digitally? According to Harvard Business Review, companies that master digital transformation generate: 9% more revenue than their industry peers, and 26% more profits than their industry peers BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 8

9 Transforming Customer Experiences 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

10 New Business Models 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 10

11 Network Quality is a complex, end-to-end problem Impacts Join/Roam Client Firmware Impacts Both* Impacts Both* Client Density AP Coverage Impacts Both* RF Noise/Interf. Impacts Quality/Throughput WAN Uplink Usage Impacts Both* WLC Capacity... Impacts Quality/Throughput End-User Services Impacts Both* Impacts Quality/Throughput Impacts Join/Roam Configuration WAN QoS, Routing,... Authentication Impacts Join/Roam Addressing CUCM ISE WAN DHCP Mobile Clients APs Office Site Local WLCs Network Services DC Prime * Both = Join/Roam and Quality/Throughput BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 11

12 The Need for a New Network Constantly Learning Support 100X new devices, apps, users Constantly Adapting Respond Instantly to business demands with limited staff and budget Constantly Protecting See and predict issues and threats and respond fast The more you use, the wiser it gets.

13 THE NETWORK. INTUITIVE. Powered by intent, informed by context. I N T E N T L E A R N I N G CONTEXT S E C U R I T Y BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 13

14 Built on Cisco Digital Network Architecture Open Cloud Service Management Automation and Assurance Automation Analytics Principles Programmable Virtualization Security and Compliance API Driven Programmable Physical and Virtual infrastructure Insights and Experiences Security BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 14

15 Introducing DNA Center Realizing vision of the intent-powered intuitive network Policy Automation Assurance and Analytics Translate business intent into network policy Decouple Policy from Network Topology Reduce manual operations and cost associated with human errors Industry Best-Practices Configuration and Policy Compliance Use context to turn data into intelligence Proactive Issue Identification and Resolution 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

16 Assurance: Predict Issues Before They Happen Visibility Learn from the network and clients attached to it Troubleshoot Find root cause faster with granular details Insights See problems before your end users do Automate Recognize changes and inform the self-driving network Predictive Performance Understand how new services will impact service levels Industry s First Self-Predicting Network Analytics Platform 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

17 Where is IT Spending Their Time? Finding the Source of an issue, is a complex, end-to-end problem 43% Client Density Impacts Join/Roam Impacts Both* Client Firmware Impacts Both* RF Noise/Interf. Impacts Both* AP Coverage WLC Capacity Impacts Both* WAN Uplink Usage Impacts Quality/ Throughput Impacts Quality/Throughput WAN QoS, Routing,... Impacts Quality/Throughput End-User Services CUCM Impacts Join/Roam Authentication Configuration Impacts Join/Roam Impacts Both* Addressing Of IT time spent troubleshooting; #1 consumer of time Mobile Clients APs WAN ISE Prime DHCP Local WLCs Source: Cisco DNA Customer Survey, June 2016 Office Site Network Services DC BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 17

18 End-to-end visibility and insights CUCM WAN DHCP Mobile Clients APs Office Site Local WLCs Network Services DC NMS SFCDC SNOW BOX Cloud Apps BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 18

19 The Value of Context By all means go and see this play Theatre Critic (as quoted in an advertisement by the play s performers) BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 19

20 The Value of Context If triviality is what you happen to be wanting, by all means go and see this play Theatre Critic BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 20

21 Contextual Correlation Example ISE Group: Marketing User: George Baker DDI NetFlow AVC Analytics Engine Topology Location MAC: 60:F4:45:78:96:9F Device MAC: B8:8D:12:36:15:22 BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 21

22 Contextual Correlation Example ISE Group: Marketing User: George Baker DDI NetFlow AVC Analytics Engine Topology Location MAC: 60:F4:45:78:96:9F Device MAC: B8:8D:12:36:15:22 Source IP: BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 22

23 Contextual Correlation Example ISE Group: Marketing User: George Baker DDI NetFlow AVC Analytics Engine Dest IP: Dest Port: 80? Topology Location Device MAC: 60:F4:45:78:96:9F MAC: B8:8D:12:36:15:22 Dest Port: 80 Source IP: ? Dest IP: BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 23

24 Contextual Correlation Example ISE Group: Marketing User: George Baker DDI NetFlow Netflow Dest IP: AVC Analytics Engine Dest Port: 80? Topology Location MAC: 60:F4:45:78:96:9F Device MAC: B8:8D:12:36:15:22 Dest Port: 80 Source IP: ? Dest IP: BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 24

25 Contextual Correlation Example ISE Group: Marketing User: George Baker DDI NetFlow Netflow Dest IP: AVC Analytics Engine MAC: B8:8D:12:36:15:22 Source IP: Dest Port: 80 Topology Location MAC: 60:F4:45:78:96:9F Device Dest Port: 80 Dest IP: BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 25

26 Contextual Correlation Example ISE Group: Marketing User: George Baker DDI NetFlow Netflow Dest IP: AVC Analytics Engine MAC: B8:8D:12:36:15:22 Source IP: Dest Port: 80 Topology Location MAC: 60:F4:45:78:96:9F Device Building 24 1 st Floor Dest Port: 80 Dest IP: BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 26

27 Contextual Correlation Example ISE Group: Marketing User: George Baker DDI NetFlow Netflow Dest IP: AVC Analytics Engine MAC: B8:8D:12:36:15:22 Source IP: Dest Port: 80 Topology Location MAC: 60:F4:45:78:96:9F Device Building 24 1 st Floor Dest Port: 80 No Layer 2 QoS marking for Webex Dest IP: BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 27

28 Contextual Correlation and Property Graph Business Applications Finance George Baker App ID: 18 Src IP: Dest Port: 3600? ? Dest IP: RTP DC Forwarding problem here SJC-9 2 nd Floor WAN QoS problem here... Client density problem here... BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 28

29 DNA Assurance From Network Data to Business Insights Network Telemetry Contextual Data Correlation Complex Event Processing Issues Insights Guided Remediation Actions Clients Baseline INSI GHTS Application Network Wireless 66 Over 100+ Actionable Insights Switching 38 Routing 8 BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 29

30 THE NETWORK. INTUITIVE. Powered by intent, informed by context. I N T E N T L E A R N I N G CONTEXT S E C U R I T Y

31 What is Machine Learning? Machine learning is an application of artificial intelligence (AI) that provides systems the ability to automatically learn and improve from experience without being explicitly programmed to do so The process of learning begins with observations of data, and looking for patterns within the data so as to make increasingly better correlations, inferences and predictions The primary aim is to allow these systems to learn automatically without human intervention or assistance and adjust actions accordingly BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 31

32 Machine Learning Project Kairos For Wireless, Wired and IOT For Wireless, Wired Networks and IOT Anomaly detection across hundred of thousands of devices, dozens of thousands of gears and hundreds of heat maps Cognitive Analytics 32 BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public

33 Machine Project Kairos Learning For For Wireless, Wireless, Wired Wired Networks and IOTand IOT End to end visibility Cognitive Analytics Anomaly detection Predictive Analytics The system detects (complex) early signs of an an issue, in the future, with confidence levels BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 33

34 Machine Learning Algorithms build their models using hundreds of inputs RF & EDCA behavioral metrics,.. Device type, OS release, behavioral metrics,... Queuing, Dropping, WRED behavioral metrics Application metrics, user feedback, failure rate,... WAN & core network metrics.. CUCM ISE... and more WAN DHCP Mobile Clients APs Office Site Local WLCs Network Services DC APIC-EM BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 34

35 Surfacing Data

36 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

37 DNA Assurance Landing Page BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 37

38 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

39 39 BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public

40 Self-Healing Network Architectural Requirement #1: Instrumentation ASSURANCE Collect relevant metrics BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 40

41 Self-Healing Network Architectural Requirement #2: On-Device Analytics ASSURANCE Categorize metrics by degrees of relevance BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 41

42 Self-Healing Network Architectural Requirement #3: Telemetry ASSURANCE Collector EM Upload critical metrics off the device to collector(s) (optimally via model-based streaming-telemetry) BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 42

43 Self-Healing Network Architectural Requirement #4: Scalable Storage ASSURANCE Provision long-term storage, retrieval and representation of network metrics and events BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 43

44 Self-Healing Network Architectural Requirement #5: Analytics Engine ASSURANCE Identify anomalies and trends BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 44

45 Self-Healing Network Architectural Requirement #6: Machine Learning ASSURANCE Correlate all data points and permutations for cognitive and predictive analytics BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 45

46 Self-Healing Network Architectural Requirement #7: Guided Troubleshooting ASSURANCE Analytics Engine EM Identify root cause of issues by contextually correlating data BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 46

47 Self-Healing Network Architectural Requirement #8: Self-Remediation ASSURANCE Network Controller EM Analytics Engine EM Do you want to take the recommended action? Yes Always No No Present actionable insights to the operator Solicit input to remediate the root cause Present a self-remediation option BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 47

48 Cisco DNA Architecture Automation and Analytics Cloud Service Management Automation Analytics NCP Network Controller Platform (Network Controller) NCP EM Virtualization Assuring the Intent NDP EM NDP: Network Data Platform (Analytics Engine) Abstraction layer Delivering the Intent Intent Outcome Analyzing the Outcome within the Context of the expressed Intent 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 48

49 Trends vs Issues

50 DNA Assurance From Network Data to Business Insights Network Telemetry Contextual Data Correlation Complex Event Processing Issues Insights Guided Remediation Actions Clients Baseline INSI GHTS Application Network Wireless 66 Over 100+ Actionable Insights Switching 38 Routing 8 BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 50

51 Trend vs Issue We want to help you when issues occur We also want to tell you before issues occur The art of the trend is to see a pattern in the noise BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 51

52 Trending Trend 1 Trend 2 Trend longer scale Issue Use elastic time scale Concatenate client trends Use flexible percentile Lower weight for clients with issues until their percentile becomes significant BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 52

53 SDA and Switching Specific Correlated Insights Border/Edge Reachability Data Plane Policy Plane Client Onboarding Switch Control plane reachability Edge reachability Border reachability Routing protocol MAP server Border and edge connectivity Border node health Access node health Network Services DHCP, DNS, AAA ISE/PxGrid connectivity Border Node policy Edge Node policy Client/Device DHCP Client/Device DNS Client authentication / authorization CPU, Mem, Temp Line-card Modules POE power TCAM Table Total SDA/Switching Insights: 38 issues in DNA-C 1.1 BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 53

54 Router Specific Correlated Insights Router Health Routing Technologies Connectivity High CPU High memory BGP AS mismatch, Flaps OSPF adjacency failure EIGRP adjacency failure Interface High Utilization LAN connectivity down/flap IS SLA to SP gateway connectivity Total Router Insights: 8 issues in DNA-C 1.1 BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 54

55 Proactive Connectivity Assessment for Wired Test your network anywhere at any time IPSLA analyzes IP service levels for services to increase productivity, lower operational costs, and reduce downtime IPSLA tests are run in the fabric network to verify connectivity to control plane, fabric border, fabric edge nodes, and fabric network services such as DHCP, DNS, AAA servers This provides predictive performance capability before issue happens This configuration is done Example by DNA-C ip sla 1 icmp-echo frequency 300! C? Known networks B Unknown networks B E E E E C B Fabric edge Fabric control Fabric border and default border BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 55

56 Wireless Specific Correlated Insights Client Onboarding Client Experience Network Coverage & Capacity Network Device Monitoring Application Performance Sensor Association failures Throughput analysis Coverage hole Availability Web: HTTP & HTTPS Authentication failures IP address failure Client Exclusion Excessive on-boarding time Roaming pattern analysis Sticky client Slow roaming Excessive roaming AP License Utilization Client Capacity Radio Utilization Crash, AP Join Failure High Availability CPU, Memory utilization Flapping AP, Hung Radio POP3, IMAP, Outlook Web Access File Transfer: FTP & TFTP Terminal: Telnet & SSHv2 Excessive authentication time RF, Roaming pattern Power supply failures Excessive IP addressing time Dual band clients prefer 2.4GHz AAA, DHCP reachability Excessive interference Total Insights: 66 issues in DNA-C 1.1 BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 56

57 DNA-C Assurance Apple Insights 1 2 Device Profile Wi-Fi Analytics 3 Assurance Client shares these details 1. iphone 7, ipad Pro 2. ios 11 Client shares these details 1. BSSID 2. RSSI 3. Channel # Client shares these details Error code for why did it previously disconnected Support per device-group Policies and Analytics Insights into the clients view of the network Provide clarity into the reliability of connectivity BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 57

58 Issue Logic

59 Issue Categories Wireless content Onboarding (23) Failed Authentication (9) Failed to receive IP address (2) Failed association (2) Sensor onboarding (4) Excessive onboarding time and client exclusion (2) WLC Tunnel down (4) Connected (26) Sensor reachability (6) Misbehaving clients Ping-pong (6) AP - Interference, flapping (10) WLC AP Roaming (4) Application Performance - Sensor (1) Coverage Poor RSSI (1) (Connected?) Availability (11) AP down, radio down/crash (6) WLC down, HA unreachable (5) Utilization (10) AP CPU, Memory, Ethernet (4) WLC CPU, Memory, Ethernet (6) Environment (1) WLC power supply (1) Compliance/Policy (1) WLC AP license exhausted (1) BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 59

60 DNA Assurance Client Issue Page BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 60

61 What are Issues in Assurance Assurance issues have at least two angles: Population process How we define issues, define their detection, cause and remediation Display process How we display in the UI information about issues, insights, trends BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 61

62 Population Process Building Blocks An issue is potentially composed of up to 4 elements Can t access GW is unreachable Link is flapping because of STP issue Fix STP Convergence Symptom Issue Cause Remediation Advice Sometimes, Symptom and Issue are the same ( can t associate to Wi-Fi ) Sometimes Issue and Cause are the same (e.g. Symptom = AP fails to join WLC; Issue/Cause = WLC reboots) BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 62

63 Population Process Where to Start From? We can work on Issues starting from any of the first 3 blocks Can t access Symptom Why? Association failed No IP address No DNS address No DNS response GW failed The Symptom-based approach is tempting because this is what users report However, its risk is the wild goose chase ( what if it is because ) BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 63

64 Population Process Where to Start From? We can work on Issues starting from any of the first 3 blocks GW is unreachable Issue Why? Router is down IP route issue L2 issue unreachable = ping? Because Issue can be the symptom, this can be a slippery process BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 64

65 Population Process Where to Start From? Third Block is Safest to Work From Link is flapping because of STP issue Cause Causes can be Issues (or not), but they are always traceable events that may have impact on users or network BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 65

66 Population Process Event Listing Working from Causes can be done in many ways, we chose the Follow The Path approach Repeat for each element lifecycle WLC Services Routed Network Client session lifecycle: Join the cell Get an IP Communicate with services Roam etc What may go wrong at this phase? List causes, not symptoms BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 66

67 Population Process Correlation Map Once Causes are defined, we can map them to Issues and Symptoms What Issue can this trigger: Reachability (GW, server) Service failures Performances (slow throughput) Etc. Link is flapping because of STP issue What symptom will be associated: We need to define some of them, but not all of them, as per our Display logic (see later slides) Cause BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 67

68 Population Process Grouping Issues The Correlation Map is a mesh, with many to many relationships A single cause can be source of multiple issues A single issue can result in multiple symptoms A single issue can be cause by multiple different causes Multiple possible symptoms A B C D E GW Reachability issue Multiple possible causes A B C D E BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 68

69 Example Failed to Join, Client Side Timeout Individual client check: auth, assoc, should take 400 ms max* WLC Routed Network Services *Based on baseline BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 69

70 Example Failed to Join, Client Side Timeout Continuous global checks: WLC Routed Network Services Can we reach the gateway? Can we reach the AAA? Is the AAA service running? What is the baseline response time? BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 70

71 Example Failed to Join, Client Side Timeout Individual client check: auth, worked, on time assoc, worked, on time WLC We saw EAP ID request We saw EAP ID response We saw EAP type request Routed Network Services Response not received after timeout and retries BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 71

72 Example Failed to Join, Client Side Timeout Individual client check: All worked well until EAP request Never got the EAP response WLC Routed Network Services We know that the issue is on the client side How far can we dig? AP on WLC okay? (RTT, CAPWAP keepalives, etc.) Other clients on same WLAN okay? Client RSSI/SNR/retries? Is-it an EAP compatibility issue? BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 72

73 Example Failed to Join, Client Side Timeout Surfacing the issue in the UI: We want to be process-efficient, all these checks are built in a Russian doll, and a mesh logic For example: RF problems cause AAA and other issues A broken AP affects many clients A broken AAA affects all new clients Etc. In other words: surface the larger issue first BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 73

74 Client on boarding and connectivity insights Wireless client on-boarding score Wired client on-boarding score DHCP DHCP Wireless client Onboarding Network services AAA DNS Internet service Wired client Onboarding Network services AAA DNS Internet service RF connectivity Association Roaming First hop connectivity BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 74

75 DNA Assurance Client Issue Page BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 75

76 Excessive IP Addressing We know: auth, assoc were okay, on time AAA phases worked, on time Client sent a DHCP request DHCP responded, but late (based on baseline) Why? Network issue Server issue Service issue BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 76

77 Issues - client boarding issue RF connectivity Association RF Link Health On-going Client onboarding DHCP Network services (raise these issues only if no RF connectivit y issues) AAA WebAuth DNS Network connectivity BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 77

78 Issues - client boarding issue RF connectivity Association RF Link Health On-going RF association failed ( specifically, step 2 or 3 ) (2) (Issue) User has poor RF connection for a sustained period of time (1) RSSI > 70 dbm / SNR > 20 db, 2) MCS consistent with signal, 3) Retries below < 15%, 4) Cell channel utilization < 50%) (1) (Issue) WF2 Misbehaving Dual band capable client preferring 2.4 GHz when 5.0 GHz is available Client onboarding Network services (raise these issues only if no RF connectivit y issues) DHCP AAA WebAuth DNS WF1 Failed to obtain IP address pool exhausted (1) (Issue) WF1 Failed to obtain IP address No response from DHCP server (1) (Issue) WF1 Failed Authentication & Key Exchange Invalid username / pwd (1) (Issue) WF1 Failed Authentication & Key Exchange too many failed auth (1) (Issue) WF1 Failed Authentication & Key Exchange Misconfigured PSK (1) (Issue) WF1 Failed 802.1x Authentication & Key Exchange Incorrect EAP method (2) (Issue) WF1 Failed Authentication & Key Exchange Expired RSA certificate (2) (Trend) WF1 Failed Authentication & Key Exchange Active Directory (2) (Issue) WF1 Failed Association Mac Authentication / RADIUS Server (2) (Issue) WF1 Failed Association Mac Authentication / WLC (2) (Issue) WF1 Failed Authentication & Key exchange Various reasons (2) (Issue) (e.g.unexpected response from RADIUS) Identify guest issues: 1) Local WLC, 2) External Web server, 3) ISE/RADIUS Cannot reach DNS server ( reachability cannot ping DNS server ) (1) (Issue) DNS server not responding ( can ping DNA server, but no response from DNS server, config error, or server issue ) (1) (Issue) Network connectivity Cannot reach user specified intranet service (1) (Issue) Cannot reach user specified internet service (1) (Issue) BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 78

79 Life, Once You Have Onboarded Roaming can be an issue Roaming is the same as onboarding, except that we know the client already Client RF may suffer Sticky client Ping-pong client Poor RF client CHDM client Quality of experience can degrade outside of RF issues BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 79

80 Issues - Wireless connected categories Client experience Roaming issues after client has successfully on boarded WF2 Misbehaving Roaming failed (1) (Issue) WF2 Misbehaving Sticky client (1) (FYI) WF2 Misbehaving r Capable client roaming slow (1) (FYI) WF2 Misbehaving i Capable client roaming slow (3) (FYI) WF2 Misbehaving Dual band capable client connects to 2.4 GHz when 5.0 GHz is available during roam (1) (FYI) WF2 Misbehaving Ping pong client (AP/AP) (1) (FYI) WF2 Misbehaving Ping pong client (SSID/SSID) (1) (FYI) WF2 Misbehaving Ping pong client (2.4GHz / 5 GHz) (1) (FYI) Issue - Roaming failed counts by AP (current) (by WLC-future) Issue - Roaming failed counts by location (floor, building, and site) (current) Issue - Roaming failed counts by client type (current) Issue - Roaming failed counts per client (current) Issue - Roaming failed counts over time (current) Issue - Roaming times longer than normal by AP (current+1) Issue - Roaming times longer than normal by location (floor, building, and site) (current+1) Issue - Roaming times longer than normal by client type (current+1) Issue - Roaming times longer than normal per client (current+1) Issue - Roaming times longer than normal over time (current+1) Trends: For each above issue as trends (future) Prediction: Roaming times and roaming failure counts (future) Kairos BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 80

81 Issues - Wireless connected categories Client experience Throughput Issue: Throughput significant drop or spike for overall (all applications) by location (current) Issue: Throughput significant drop or spike for overall (all applications) by AP (current) Issue: Throughput significant drop or spike for overall (all applications) by client type (current) Issue: Throughput significant drop or spike from normal per application by location (floor, building, and site) (current+1) Issue: Throughput significant drop or spike from normal per application by AP (current+1) (by WLC - future) Issue: Throughput significant drop or spike from normal per application by client type (current+1) Trends: For each above issue as trends (future) Prediction: Throughput 1-5 hours in advance for significant drops or spikes for overall (all applications) by AP (current) Prediction: Throughput 1-5 hours in advance for significant drops or spikes for overall (all applications) by application (current+1) Prediction: Throughput 1-5 hours in advance for significant drops or spikes for overall (all applications) by client type (future) Kairos BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 81

82 Wireless Network Control Plane Categories Connectivity AP and WLC Utilization Availability Wireless Network Control plane Device health AP Device health WLC BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 82

83 Wireless Network Control Plane Categories and Issues Connectivity AP and WLC WF5 AP flapping between WLCs (1) (FYI) WF5 AP VLAN mismatch (flexconnect) (2) (FYI) WF4 WLC regulatory domain mismatch between WLC and APs (2) (FYI) WF5 AP WLC join failure AP not joined to its preferred WLC (3) (FYI) Wireless Network Control plane Availability WF4 WLC down (1) (Issue) WF4 WLC power supply failure (1) (FYI) WF4 WLC last reboot crash (1) (FYI) WF4 WLC HA/SSO the primary unit / standby hot (2) (FYI) WF4 WLC HA/SSO HA peer unreachable (2) (FYI) WF4 WLC HA /SSO both WLCs active (2) (FYI) WF4 WLC EoIP mobility tunnel Data channel down / peer WLC up (2) (FYI) WF4 WLC EoIP mobility tunnel control and data channel down / peer WLC up (2) (FYI) WF4 WLC EoIP anchor tunnel data channel down / peer WLC up (2) (FYI) WF4 WLC EoIP anchor tunnel control and data channel down / peer WLC up (2) (FYI) WF5 WLC Excessive inter-wlc roaming (L2) (2) (FYI) WF5 WLC Excessive intra-wlc roaming (L3) (2) (FYI) WF4 WLC Mobility bug (CP) (3) (FYI) Device health WLC BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 83

84 Wireless Network Device Health Issues Wireless Network Control plane Utilization WF4 WLC utilization Radius queue (1) (FYI) WF6 WLC utilization memory (1) (FYI) WF4 WLC AP license utilization (1) (Trend) WF4 WLC AP license exhausted (1) (FYI) WF4 WLC AP Capacity (1) (Trend) WF4 WLC client capacity (1) (Trend) WF6 WLC utilization CPU (1) (FYI) WF6 WLC utilization WLC Ethernet port utilization (Rate/load, Tx/Rx indivual vs bidirectional, duplex speed) (2)(FYI) WF6 WLC low free Mbuf (2) (FYI) WF6 WLC high WQE pool usage (2) (FYI) WF6 WLC high packet pool usage (2) (FYI) WF6 WLC low available timer (2) (FYI) Device health WLC Temperature FAN QoS Table Usage (Queue depth, Buffer) Interface Statistics - Error, Discard, FCS, Runts, resets Queue Utilization wit drop counters BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 84

85 Application Experience

86 Application Health Global BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 86

87 How Should Network App Treatment Quality be Measured? A "service class" represents a set of traffic that requires specific delay, loss, and jitter characteristics from the network. Conceptually, a service class pertains to applications with similar characteristics and performance requirements RFC 4594, Section 1.3 The 3 basic component metrics of Network Application Treatment Quality: Delay / Latency Jitter Loss BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 87

88 Application Experience Client Cisco and/or its affiliates. All rights reserved. Cisco Public 88

89 Sources of Application Source: Health PerfMon KPIs Source: CB-QoS MIB Source: IP SLA Source: ART Source: Netflow DNA Assurance uses multiple sources of data to generate a multi-dimensional Application Health Score, currently based on packet-loss, latency and jitter 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

90 Calculating Jitter and Loss for RTP Apps Jitter is calculated by comparing the timestamps of RTP packets with subsequent sequence numbers Gaps in subsequent RTP packet sequence numbers identifies lost packets RTP Header Format BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 90

91 Calculating Jitter and Loss for TCP Apps Client SYN ACK CND ART SND SYN-ACK Server SND = Server Network Delay CND = Client Network Delay Network Delay (ND) ND = CND + SND Request Request 1 Request 1 (Cont) RT ACK Response Time (RT) t(first response pkt) t(last request pkt) ACK 3 TT X DATA 1 DATA 2 DATA 3 DATA 4 Application Delay (AD) AD = RT SND X ACK 6 DATA 5 DATA 3 DATA 4 Retransmission Response Transaction Time (TT) t(last response pkt) t(first request pkt) DATA 6 Request 2 BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 91

92 How Many Levels of Sensitivity/Tolerance Per Service-Class for Each of the QoS SLAs? RFC 4594 Section 2.3 Figure 2 BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 92

93 How Many Levels of Sensitivity/Tolerance Per Service-Class for Each of the QoS SLAs? Very Low Tolerance Weight = 5 Low Tolerance Weight = 4 Medium Tolerance Weight = 3 High Tolerance Weight = 2 Yes (Tolerant) Weight = RFC 4594 Section 2.3 Figure 2 BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 93 1

94 ITU-T G.114 Guidance for Absolute Latency Mouth-to-Ear Delay includes Packet formation delay ( 20 ms) and de-jitter buffer playout ( 30 ms) BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 94

95 ITU-T G.114 Guidance for Network Latency (Network Latency is Absolute Latency less 50 ms) One-Way Network Latency MOS Score ms ms ms ms ms ms ms ms ms BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 95

96 Standards-Based Packet Loss Targets ITU-T Y.1541 (2011) offers packet-loss recommendations for various traffic classes Y.1541 VTC = Video Teleconference RFC 4594 Multimedia Conferencing BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 96

97 Standards-Based Loss Targets (continued) ITU-T Y.1541 offers loss ( IP Packet Loss Ratio or IPLR) recommendations for various traffic classes A single loss target (1 in 1000) is given for multiple traffic classes NOTE 4 The class 0 and 1 objectives for IPLR are partly based on studies showing that high quality voice applications and voice codecs will be essentially unaffected by a 10-3 IPLR. Comment: The 10-3 IPLR recommendation is associated with codec behavior (i.e. elastic in RFC 4594 terminology BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 97

98 Standards-Based Jitter Targets (continued) ITU-T Y.1541 (2011) includes two new provisional network QoS Classes In this revision, there is agreement that the applications which provided the original rationale for the IPLR in Table 3 have evolved. In at least one case (IPTV) the performance objectives of Table 1 may now be sufficient (see Appendix VIII). Y.1541 IPTV RFC 4594 Broadcast Video Appendix VIII. Effects of IP network performance on digital television transmission QoS Loss Target is 1 in 100,000 This appendix details a part of the analysis behind the specification of provisional network QoS classes 6 and 7 in Table 3. The objective values were selected in order to support digital television transmission. The IP packet loss ratio (IPLR) objective in classes 0 through 4 was insufficient to support this application, as stated in the previous version of this appendix. BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 98

99 How Are Customers Troubleshooting Issues Today? 1. L1-3 Operator receives a report of a poorly-performing app Network App These are 30+ year old tools! 2. Operator asks the client for the IP address of his device (and finds out his/her default gateway) 3. Operator tries to find the IP address of the application-server 4. Operator logs into the default gateway of the client and performs a traceroute to the app-server IP address 5. Operator iteratively pings each device in the path to approximate latency (but not jitter) 6. Operate iterative logs into each device in the path and executes show commands to check for drops on the interface (and possibly the queues) 7. Operator possibly finds drops, but cannot definitively associate them with any particular application (he can only infer) BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 99

100 Troubleshooting Applications The Network vs. App Triage Decision Network problem or App problem? Helpdesk Network is stable. Must be the app. Code is fine. Don t care about anything else Network Ops IT Routing Loop AppDev Dear User: We do not have enough info/context to troubleshoot further and are thus closing your case. BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 100

101 Path Trace Part 1 of 4 BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 101

102 Path Trace Part 2 of 4 (Device Details and Stats) BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 102

103 Path Trace Part 3 of 4 (Interface Details and Stats) BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 103

104 Path Trace Part 4 of 4 (QoS and ACL Stats) BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 104

105 PathTrace How Does it Work? BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 105

106 Anticipate with Sensors

107 Wireless Sensors Proactively Assess Performance Test your network anywhere at any time On-Boarding Tests Association Authentication & Key Exchange IP Addressing DHCP (IPv4) Network tests DNS (IPv4) RADIUS (IPv4) First Hop Router/Default gateway (IPv4) Intranet Host External Host (IPv4) Application tests POP3, IMAP, Outlook Web Access (IPv4) File Transfer: FTP (IPv4) Web: HTTP & HTTPS (IPv4) Sensors act as clients Dedicated Sensor AP1800 R1 Access point Flexible Radio Flexible Radio Assignment Algorithm intelligently identifies excessive radios and seamlessly converts those into Sensor mode without client impact BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 107

108 Wireless Sensor Support Flexible Radio as Sensor (2800/3800) Dedicated AP as Sensor Dual 5 GHz Flexible Radios Software defined radios automatically adjust to dual 5GHz Purpose-built Hardware for Analytics Flexible radios can to provide simultaneous in-line monitoring to DNA for analytics and insights while serving clients (future) 1815/1830/1850 AP 1800s dedicated sensor /1850 2x2 with 2 spatial streams Multiple powering options: PoE Power USB Type C power Direct AC Power Plug Integrated BLE 5GHz. XOR RADIO 2.4GHz. Sensor (Client Testing) BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 108

109 Run Network Tests See What Works and Where BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 109

110 Run Network Tests Hover over a Test to See Results BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 110

111 Filtering the View Select Only the Tests That Matter to You Now BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 111

112 Sensor Test How-To 1a. Create a New Test, Choose Location and Run Interval BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 112

113 Sensor Test How-To 1b. Choose the SSIDs to test BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 113

114 Sensor Test How-To 2. Select the Tests to Run BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 114

115 Sensor Test How-To 3. Select the Sensors to Use BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 115

116 Sensor Test How-To Test Runs at Intervals BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 116

117 Sensor Test How-To As Usual, You can Zoom and Customize BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 117

118 Sensor Test How-To You Will be Able To Define Sensitivity Each sensor needs to test RF, SSID connectivity, then one or multiple services Imagine: 1000 stores per region 10 sensors per store, covering the sensitive areas of each store Each sensor is in range of 7 to 10 APs (we counted 7) Each store has 5 SSIDs (staff, contactor etc) using 802.1X 15 tests (from DHCP, DNS, pings to various servers and application connect test) Math: 1000 x 10 x 7 x 5 x 15 = RADIUS hits per 30 min intervals. Should all sensors test everything? You will be able to define test aggressiveness From only 2 sensors test any given wired service to all test all BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 118

119 Conclusion - What Happens Next

120 AirSense Real-time Client and App Insights to enhance Wireless Assurance Today Low visibility on Client RF health Less granular Client Onboard state No real-time access to Client location Zero visibility on AP anomalies No On-demand insights on App experience Enriched Client RF stats ~6x faster More granular Client Onboard state ~2x faster for 250+ states w/airsense Real-time Client location ~3-5 sec updates AP anomaly detection ~5-10 new issues identified On-demand App experience ~Instant insights (<5 sec) BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 120

121 Top Use Cases that can be solved by AirSense Enhance guided remediation of Enhance guided remediation of Wireless Issues Wireless Issues Real-time and On-Demand Real-time and On-Demand Troubleshooting Troubleshooting Client On-boarding issues due to poor RF condition Automated Anomaly based packet captures DHCP and DNS issues on the client and network side On-Demand RF Scanner (14+ Wireless Actionable insights) VIP Wireless Assurance Real-time SLA Thermometer Active Sensor Testing 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 121

122 Troubleshooting Use Cases using AirSense 1 On-Demand RF Scanner Live RF Network Pause 11:50:32 Record Session Red Spot: Coverage Hole Situation A large logistics company uses Wi-Fi operated Auto-guided vehicles (AGV) in its logistics warehouses. Due to high ceilings and large moving metal structures, these warehouse have dynamic RF scenarios Orange Spot: Browsing-quality Throughput Green Spot: Voice-quality Throughput Red Spot: Coverage Hole KPI List Graph* SNR RSSI MCS Throughput Packet retry Due to little human involvement sometimes these machines don t take optimal path as it failed to download new work order due to temporal coverage hole Solution IT staff uses the On-Demand RF scanner tool to validate AGV s RF coverage during the live operation and detect coverage holes in a highly dynamic RF environment This helps IT staff immensely in RF design, planning and optimization Key Issues addressed Poor RF Zones RF design optimization Sticky client analysis with real-time location VoIP performance and One-way audio issue Key Industries Healthcare, Distribution, Logistics, Retail and Higher Ed. BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 122

123 Troubleshooting Use Cases using AirSense 2 Automated Packet Captures Visual Packet Trace Analyzer (<5 sec) Situation A Financial services giant decided to adopt to BYOD to encourage people to use its Virtual Desktop Infrastructure for a certain critical applications. In the first week, John discovered longer Onboarding time from mobile device compare to laptop and received complaints from several end-users Solution Real-time Anomalies with Auto PCAPs (<5 sec) IT staff turned on the Auto PCAP to capture onboarding and roaming failure anomalies for the BYOD device at a specific site. This allowed IT staff to capture de-authentication packets that are typically observed when the client has driver issues IT staff also had access to detailed PCAP analysis that enabled them to identify the root cause behind onboarding and roaming failures On-boarding and roaming failures Key Issues addressed VOIP Performance issues Sticky client analysis with real-time location Key Industries All segments BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 123

124 Troubleshooting Use Cases using AirSense 3 VIP Service Assurance Visual Packet Trace Analyzer (<5 sec) Situation A large healthcare deployed 5000 vital sign monitor for every in-patient. These vital sign allow doctors to monitor critical parameters on a real-time basis Wi-Fi connection of vital sign monitor starts to get disrupted on an intermitted basis during the middle of the day and if not addressed it can be life threatening Solution IT staff uses Live Troubleshooting tool to perform detailed forensics on both the client state and the location Real-time Client Location Tracking (<5 sec) Using packet trace analyzer, the IT staff is able to visualize frequent roaming and re-authentication failures from the device along with lower RSSI than anticipated in particular location Key Issues addressed On-boarding and roaming failures w/ PCAP Sticky client analysis with real-time location Service Level Assurance for critical apps/users Remote Wireless troubleshooting through Full PCAP Key Industries Healthcare, Distribution, Logistics, Retail, Higher Ed. and MSPs BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 124

125 Forward Looking Ideas What Else Could We Do? Imagine that we linked the DNA Assurance platform to TAC database Issue found: high CPU on platform XYZ TAC knowledge base Issue found with 952 other customers, when command AA is used in combination with interface command BB, changing to command CC achieves the same result but lower CPU condition Suggested fix Would you like me to make the change? Disclaimer: this is not a roadmap, just an exchange of forward looking ideas BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 125

126 Forward Looking Ideas What Else Could We Do? Imagine that we linked the DNA Assurance platform to the CVD Database Click: grade my config Anonymous CVD database call 52 optimization opportunities were found Suggested fixes Would you like me to make the change? Disclaimer: this is not a roadmap, just an exchange of forward looking ideas BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 126

127 Summary The top-of-mind issue for most CIOs is digital transformation Most networking departments today are bogged down by operations, spending the majority of their time manually configuring and troubleshooting their networks Enabling an intent-based closed-loop architecture, including automation and analytics, significantly frees up IT time and resources This techtorial focused specifically on the role of analytics and assurance in the enterprise network architecture Client, Network and Application health monitoring, reporting and troubleshooting were all presented in-depth, as were additional use-cases of DNA Assurance. Additional ongoing-development initiatives were previewed, including Machine Learning, Application Integration (Skype-for-Business example) and Application Dynamics integration BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 127

128 Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space cs.co/ciscolivebot#brkewn Cisco and/or its affiliates. All rights reserved. Cisco Public

129 Please complete your Online Session Evaluations after each session Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt All surveys can be completed via the Cisco Live Mobile App or the Communication Stations Complete Your Online Session Evaluation Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at Cisco and/or its affiliates. All rights reserved. Cisco Public

130 Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Tech Circle Meet the Engineer 1:1 meetings Related sessions BRKEWN Cisco and/or its affiliates. All rights reserved. Cisco Public 130

131 Thank you

132