JStorm Based Network Analytics Platform. Alibaba Cloud Senior Technical Manager, Biao Lyu

Transcription

1 JStorm Based Network Analytics Platform Alibaba Cloud Senior Technical Manager, Biao Lyu

2 Overview of Alibaba Cloud 18 Regions 150+ Products 1Million+ Customers

3 Comprehensive Networking Product Family 12 products for 5 scenarios Self-Defined Cloud Networking Environment VPC Internet Access Saving BGP Cost Building Hybrid Cloud Global Connection SLB NAT Gateway EIP Shared Flow Package Shared Bandwidth Package Express Connect Cloud Hosting VPN Gateway Smart Access Gateway Cloud Enterprise Network Global Acceleration From Data Center to Global Connection, for every enterprise networking scenarios

4 Networking is getting complicated 25G 10W+ NIC Underlay VxLAN Neutron Hybrid Cloud IoT 5G vcpe 400G DCI Overlay SD-WAN RDMA Vitrual Private Cloud vrouter NFV Virtual switch NSH SDN Segment Routing Large Scale! High Throughput! Low Latency! Rich Features!

5 Challenges from Growing Scale Tenants Servers VMs 1,000,000,000 1,000,000,000 10,000,000,000 Random Failures! Daily Maintenance! Multi-Dimension! Profiling!

6 What we want to achieve Network as a Service Minimum Impact Network Update Precise Diagnose Network Issue Realtime Monitor Network Failure Deep Understanding Network Business

7 Automation to Intelligence Tool Network Update Script Controller Disaster Recovery Network Monitor Network Config Network Diagnose Network Profiling Automation Intelligence

8 QiTian - JStorm Based Network Analytic Platform Flow API WEB Robot Output Analytic Result Basic Monitor Diagnose Scheduling Profiling DB ETL Math Policy Event Notification Packet Route Connectivity LB Recovery User/Device Real-Time Analytic Config JStorm flow, metric, log, topology, device status, etc product, customer, instance, etc SDN Controller Network Devices Business Services Data Source

9 QiTian - Data Sources & Types QiTian SDN Controller operation log, topology Streaming Log, DB, message Log Streaming operation, infrastructure Streaming Log Operation Services Business Services customer, product, instance, income/cost Streaming Log, DB DB Streaming message Streaming access, ddos, waf Streaming Log Security flow, metric, device status, access log, etc Streaming Log Overlay Network Device Underlay Network Device Gateway Router Switch Host VSwitch vborder Device

10 QiTian - High Performance Network Analytics Network Devices Stream Dispatch Streams Local-Shuffling Local Data Merge ETL Telemetry Local-Shuffling Fields-Grouping 100 Million Records/min 100 Million Records/min 1 Million Records/min 1 Million Records/min per Stream

11 QiTian - High Performance Network Analytics Metrics TPS Data 500+ Metrics 2,000,000 Tuples/Sec 100T/Day

12 QiTian - Real-Time Network Monitor Network Devices Basic Steam Processing Policy Steam Processing Event Merge Steam Processing Event Notification Steam Processing Raw Data Rich Meta Data metric vibration Network Event Merged Network Event mail flow, metric, log user, instance, cluster predicted interval issue, failure sms event statistics app msg

13 QiTian - Real-Time Network Monitor metric vibration Major Network Issue Detection abnormal event statistics 90% predicted interval

14 QiTian - Streaming Based Network Diagnose Packet Analytic Config Network Devices Packets Steam Processing Packet Sort flow route Packets Route Calculation Result device latency Latency Calculation Loss Rate Calculation device loss rate Packets Collection Spout Packets Processing Bolt

15 QiTian - Streaming Based Network Diagnose Packet Loss/Latency Diagnose Package Analyze Config Flow Route Packet Analyze

16 QiTian - Intelligent Network Traffic Scheduling Network Devices Reliability Stream Processing Traffic Scheduling Stream Processing Topology Analyze Network Metric Network Event issue, failure Load Calculation Cluster Selection Device Selection Scheduling-Aware Event Spout Traffic Scheduling Processing Bolt SDN Controller Traffic Schedule Config

17 QiTian - Multi-Dimension Network Profiling Business Services Business Data Steam Processing Rich ETL Steam Processing DB Biz Data Biz Data Stream Biz Data Bolt Group 1 In Memory ETL Batch Processing Traffic Pattern msg Network Devices Biz Data Spout Hash Biz Data Bolt Group 2 In Memory ETL Rich Meta Data customer, instance, cluster Business Cost User Profiling Telemetry Biz Data Bolt Group 2 In Memory ETL Telemetry Data Spout Rich ETL Bolt

18 QiTian - Multi-Dimension Network Profiling Income Analyze by Industries Customer Resource Layout BGP Cost

19 What s Next Deployment Elastic Monitor real-time network deployment policy real-time network expansion policy new detection policy based on machinelearning integrated

20 Thanks