Data-Driven Network Opera1ons. France-IX 2016 Avi Freedman

Transcription

1 Data-Driven Network Opera1ons France-IX 2016 Avi Freedman

2 Summary Why Data-Driven Network Opera1ons? The food: data types and sources Requirements and tool types Data Fusion Business-driven use cases: Review Use case: Traffic explora1on and anomaly detec1on Use case: Planning and peering Use case: Network performance analy1cs Use case: Security and breach detec1on Use case: Revenue enhancement Summary 2

3 Why Data Driven Network Opera1ons?

4 What is Data-Driven Network Opera9ons? Using your data to drive your ops + business! Most content companies and enterprises are data and analy1cs driven. Devops is as well (APM, metrics-at-scale). But the network world has some catchup to do. We re now at the point where CPU, systems and methods exist for NetOps. But most of the work remains to be done. 4

5 The Food: Data Types and Sources

6 What kind of data? *flow BGP SNMP and streaming telemetry (i.e. queue depth) Event logs User tags ( Applica1on A, Peer, My Network,Rack#1 ) Server Metrics (RRD, tsdb, Grafana, ) Ac1ve measurement data (probes) LLDP and IGP-based topology data SDN overlay/underlay mappings DNS queries Threat intelligence 6

7 Enhanced NetFlow: Performance and Seman9cs Imagine if we could get performance data from the network: Queue Depth Retransmits per flow TCP latency Applica1on Latency You can (somewhat): Host sogware (Nprobe) Sensors / Taps Webserver logs (Nginx) Cisco AVC supported routers 7

8 Requirements and Tool Types

9 Key Network Operator Requirements Key requirements for modern Data-Driven Network Opera1ons: No data aggrega1on or pre-filtering. Correla1on (fusing) between data types. Full resolu1on searchable and stored for months. FAST. Less than 10s for results. Cannot wait minutes while spelunking. Network-savvy Uis and APIs (understanding rou1ng and prefixes). Detect anomalies. Should not have to watch graphs manually. 9

10 Anomaly Detec9on You shouldn t have to stare at dashboards or watch logs to detect badness. Should be able to look at dimensions such as interface+path+type, or cidr/ prefix, detect unusual pakerns and perform an ac1on. Con1nually. Be able to look at things beyond pps/bps such as retransmits, latency, log.s Be able to detect shigs such as did an asn or IP on a par1cular interface suddenly move from top-x #200 to #2 and that is unusual for this 1me of day. This is doable today, though requires network-savvy data backends and is not yet magic. 10

11 Available Tool Types for Traffic-Centric Analy9cs Current Open Source: pmacct, ntop, SiLK Older Open Source: cflowd, MrTG, AS-PATH, swatch Commercial soaware: Arbor, SevOne, Solarwinds, ManageEngine Do It Yourself Big Data: Kala + {ELK, Hadoop, Druid, Cassandra, Spark} SaaS Big Data: Ken1k, Appneta +(Splunk [for security]) On-Prem Big Data: Cisco Tetra1on, Ken1k, Deepfield We re making progress as most many big data tools support high volumes of data, a large set support high speed queries, and a (not iden1cal) large set support non-aggregated storage and high volumes of data. Most don t yet support data fusion and network savviness about prefixes and paths yet, so DIY efforts must implement. 11

12 Data Fusion

13 Fusing Data Flow or BGP or SNMP or DNS or logs alone are not enough. Star1ng with NetFlow/sFlow/IPFIX alone, which is (IP src/dst, ports, bytes, packets, and some1mes VLAN, MPLS, ). This becomes much richer when combined with: BGP akributes Geography Tags (rack, department, customer ) Config changes and sogware versions Threat intelligence and known-bad IPs Fusing should be near real-1me, performed at ingest and data specific 13

14 Distributed system data ingest 14

15 Pulling it together Legacy non-distributed network tools: Don t scale to handle large amounts of data (space/io/cpu limited). Typically store only aggregates or pre-filtered data. Very limited fusing, only 1 or 2 data types per tool. Limited dimensions and filtering depth, ogen slow. Ogen require large amounts of configura1on + tuning. Big-Data approach: Large scale, billions of records per hour, no aggrega1on, complex fusing. Distributed micro-service architecture. Can scale very wide. ++Hardware. If done right, fast. Real-1me ingest and < 5s queries. Big challenge is adding fusion, network savviness, and of course hardware. 15

16 Business-Driven Use Cases 16

17 Data-Driven Ops + Business Use Cases Network Planning Peering Analy1cs and Abuse Conges1on detec1on Is it the network? Where on the network? Proac1ve aler1ng Distributed DDoS Detec1on What Changed Post Deploy? Security and Breach Detec1on Cost Analy1cs Revenue Iden1fica1on (New + Risk) Enabling Internal Groups 17

18 Use Case: Traffic Exlora1on and Anomaly Detec1on 18

19 Use case: Traffic debugging and inspec9on Why did the interface just double its traffic, now saturated? Customer renewal re-pricing. Where is the traffic leaving my network? Is a peer sending me traffic they shouldn t be? Did a content provider shig their traffic path to me? 19

20 Traffic by geo, filtered by one source interface 20

21 Traffic debugging and inspec9on 21

22 Use case: Anomaly detec9on Traffic from individual top-20 ASN over transit unusually high. Operator no1fied at red line. 22

23 Traffic, anomaly detec9on and annota9on 23

24 Traffic annotated with mul9ple events 24

25 Anomaly detec9on: DDoS detec9on and characteris9cs 25

26 Use Case: Planning and Peering 26

27 Use case: Network Planning Flow-based traffic + BGP can be used to help show: Path, neighbor, transit, origin, and country of traffic. Strategic peering and transit changes that can improve performance and reduce costs. Poten1al new peers and loca1ons to peer. Evaluate the poten1al of new peering exchanges or facili1es. Transit rela1onships that are of high or likle value. Understand ROI before extending backbone links or capacity. 27

28 Network Planning, traffic by BGP HOP 28

29 Network Planning, collapsed path, exclude 1st 29

30 Use Case: Network Performance Analy1cs

31 Use case: Network Performance Analy9cs Flow-based traffic + BGP + network performance data can show: Whether If And And Performance Or Or Or issues are in the applica1on or network layer network, where? in a way exposable to internal development + app opera1ons to pinpoint where issues by peer, or even remote AS hops behind peers prefix DC not (in the network) 31

32 Retransmits enhanced flow example 32

33 Use Case: Security and Breach Detec1on

34 Use case: Network Security Analy9cs Flow-based traffic +/- threat intelligence can show: Poten1ally Anomaly Ideally, And To compromised servers, desktops, or IoT devices. detec1on can ogen uncover these even without threat intel. this can be used to help downstream or internal customers. feed DDoS response if there are local sources/sinks. be a good ci1zen, BCP38 viola1on detec1on can run across the network, even with simple heuris1cs like how many /8s as sources? 34

35 Use Case: Revenue Enhancement

36 Use case: Revenue Enhancement Some1mes not discussed in polite company, but great traffic-based analy1cs can help with the top line as well as margin: Offering Iden1fying high-margin customers lower rates to akract more traffic. large 2 nd and 3 rd hop AS sinks or sources behind peers, to convert to customer rela1onships. 36

37 Summary

38 Summary Networks can produce large amounts of data that will make your life easier. Big Data plalorms are able to consume this data. Specific tools for Network Operators are beginning to appear (free and paid). Paid tools are more specific to network use (UI, easy setup, etc). Free tools have the power but require cobbling together pieces. Much work to be done re fusing data such as logs, changes, alerts, DNS. SaaS providers will provide community views and enable data-sharing. Mul1-source correlated at-scale data is heavily in use for non-network systems. It s 1me vendors and network folks catch up and end the NetOps suffering. 38

39 Ques1ons? Dan Ellis 39

40 Appendix: Architecture

41 Distributed system data ingest 41

42 Distributed system fusing 42