WiFi Issues In The Wild A view from the cloud

Transcription

1 WiFi Issues In The Wild A view from the cloud #WLPC Phoenix 2018 KN Gopinath (VP of Engineering) Karan Gupta (Senior Researcher)

2 Mojo Cloud Architecture - Inception in 2007, Evolving Continuously 1 Management Plane - Centralized Access Security Engagement SCALE One example: A Cloud instance is managing 100,000 LIVE APs Virtualized Traffic Aggregator Data Tunnel Local Breakout of Data Management traffic to and from cloud Data Integration (Syslog, RESTful APIs) Logging SIEM CRM 3b 3a ROBUST Works even if connectivity to manager is LOST 2 3 Data Plane Flexible 2 Control Plane - Distributed Zero functionality loss if connectivity with cloud breaks

3 Cognitive Wi-Fi Powered Through Cloud 1. Management Plane - Centralized BIG DATA Store key client parameters & run ML algorithms. 4. Cognition Plane - Intelligent 3b. Data tunnel 3a. Local data breakout 3. Data Plane - Flexible 2. Control Plane Distributed SMART EDGE APs Failure Detection Needle in the hay stack 3

4 Wi-Fi User Experience: Connectivity Client Journey Extending the Philosophy of Mojo Packets (launched in 2013) Client Events Sent real-time to the cloud. 4

5 Wi-Fi User Experience: Performance Anomaly Detection & Baselining of various performance parameters computed and stored in the cloud. 5

6 User Application Experience NEW! Launching at WLPC! Deep Packet Inspection, Machine Learning 6

7 Failure Analysis Overview Reference Data Connectivity Failures Performance Failures Conclusion 7

8 Reference Data Anonymized and a subset from our production cloud. Clients 237k+ Applications 400+ Associations 31M Duration 1 week Retail & Hospitality Enterprise Verticals Manufacturing Education 8

9 9 The State of Apps % of Customers Google Google APIs Amazon Web Services Apple Facebook Google Play itunes YouTube Microsoft Akamai icloud gmail Doubleclick Windows Update Twitter MS Online Apple Update Facebook Video MS Office 365 Yahoo Google Ads Amazon Spotify ICMP Adobe Google Drive MS Outlook STUN LinkedIn Snapchat Netflix Video Stream Exchange Online CIFS WhatsApp Media Message Instagram APNS AppNexus Google Analytics Crashlytics Integral Ad Science

10 Wi-Fi Connectivity Failure & Latencies 10. Confidential Information.

11 WiFi almost always gets blamed! 11

12 Client Connectivity Failures Ultimate Truth: Instrumented AP Driver Code to tap into client s state machine. Association Failures Authentication Failures Network Failures AP association limit exceeded Capability mismatch Generic assoc. failure Fast roaming failed RADIUS auth. failure RADIUS Server not reachable Incorrect PSK EAPOL 4-Way handshake failed DHCP failure DNS failure 12

13 The State of WiFi Connectivity Approximately 6% of connections fail. 94% 6% Note that this corresponds to about 5% of the clients. More than 3000 connections happen in our network per minute (on an average). Successful Connections Failed Connections 13

14 Why do we see Connectivity Failures in an otherwise well configured and operational Network? 46% 52% More than 50% of the connection failures are due to wired side issues. 2% Association Authentication Network 14

15 Failure Distribution - We DID notice some connectivity errors that were transient in nature Assoc. Failure 2% DNS Failure 27% Mac Filtering 4% RADIUS Auth Failure 4% RADIUS Server Unresponsive 1% 19% PSK Errors. Possibly due to Guest users and/or BYOD Incorrect PSK 19% 15 DHCP Failure 25% EAPOL Handshake Failure 13% Portal Failures 5% 13% EAPOL Errors. Mostly, transient and selfcorrecting in nature.

16 Connectivity With Respect To 2.4 & 5 GHz Bands 8 7 How many of you think that the connectivity issues are similar in both the bands? % Clients GHz 5GHz Association Authentication Network 16

17 Connectivity Failures With Respect To Verticals % of connections 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Enterprise Manufacturing Education Retail & Hospitality Assocation Authentication Network DHCP/DNS errors predominate in Retail & Hospitality Due to transient nature of guest users. 17

18 State of AAA Latency For Successful Connections % of customers ~ 62% of the customers have their baseline latency below 500ms (0,20] (40,60] (80,100] (120,140] (160,180] (200,220] (240,260] (280,300] (320,340] (360,380] (400,420] (440,460] (480,500] (520,540] Latency (ms) (560,580] (600,620] (640,660] (680,700] (720,740] (760,780] (800,820] (840,860] (880,900] (920,940] (960,980] 18

19 State of DHCP Latency For Successful Connections % of customers ~ 80% of the customers have their baseline latency below 20ms And ~ 90% have their baseline latency below 100ms (0,20] (40,60] (80,100] (120,140] (160,180] (200,220] (240,260] (280,300] (320,340] (360,380] (400,420] (440,460] (480,500] (520,540] (560,580] (600,620] (640,660] (680,700] (720,740] (760,780] (800,820] (840,860] (880,900] (920,940] (960,980] Latency (ms) 19

20 Wi-Fi Performance Failures & Latencies 20. Confidential Information.

21 Client Performance Failures An Overview 300+ counters monitored by an AP, but following four key metrics capture client health. Low Data Rate guideline is 20Mbps. Low RSSI guideline is -70dbm 21

22 The State of WiFi Performance % Clients Low Data Rate (17%) is the dominant factor. 19% of the clients are affected due to performance issues. (vs 5% clients that are affected due to connectivity) 10 0 Low Data Rate Low RSSI Stickiness High Retry Rate Unaffected 22

23 Performance With Respect To 2.4 & 5 GHz Bands 40 % Clients % unique affected 3x of 5GHz 11% unique affected 2.4GHz band has 3 times the performance issue compared to 5GHz. Clients have a tendency to chose 2.4 GHz at lower RSSIs GHz 5GHz Tx power and Cell Size matter 23 Low Data Rate Low RSSI High Retry Rate Stickiness

24 Performance Issues With Respect To Verticals % Clients Low Data Rate Low RSSI High Retry Rate Stickiness 0 Enterprise Retail & Hospitality Manufacturing Education 24

25 State of DNS Latency in % Customers % of customers ~ 70% of the customers have their baseline latency below 100ms (0,20] (40,60] (80,100] (120,140] (160,180] (200,220] (240,260] (280,300] (320,340] (360,380] (400,420] (440,460] (480,500] (520,540] Latency (ms) (560,580] (600,620] (640,660] (680,700] (720,740] (760,780] (800,820] (840,860] (880,900] (920,940] (960,980] 25

26 State of WAN Latency in % Customers % of customers ~ 51 % of the customers have their baseline latency below 100ms 5 0 (0,20] (40,60] (80,100] (120,140] (160,180] (200,220] (240,260] (280,300] (320,340] (360,380] (400,420] (440,460] (480,500] (520,540] Latency (ms) (560,580] (600,620] (640,660] (680,700] (720,740] (760,780] (800,820] (840,860] (880,900] (920,940] (960,980] 26

27 Benchmarking Latencies Across Verticals Latency (ms) - Median Clearly AAA latency needs to be optimized Education Enterprise Manufacturing Retail_Hospitality DHCP DNS WAN AAA 27

28 OCE: Short Connection Times Auth. Frame Higher layer packet containment IE added to assoc. frames Auth. Response FILS AUTH FILS AUTH EAP- RP used to reduce delays; FILS key generated during initial authentication with AAA Assoc. Request Request for IP address can be piggybacked DHCP Messages Assoc. Response AP can send IP address 28 Avoids delays due to security and IP address assignment

29 Key Take-Aways 5% Connectivity Failures observed in production. Can affect user experience Its NOT Wi-Fi Always DHCP & DNS failures dominate Cross customer analysis possible only through cloud. Low link speed and RSSI dominant performance factors. Benchmark organizations against their peers/vertical Need good visualization & automated analysis tools Improve overall user satisfaction of existing deployments 29

30 Abhishek Kunal