Note. Some History 8/8/2011. TECH 6 Approaches in Network Monitoring ip/f: A Novel Architecture for Programmable Network Visibility

Transcription

1 TECH 6 Approaches in Network Monitoring ip/f: A Novel Architecture for Programmable Network Visibility Steve McCanne - CTO riverbed Note This presentation is for information purposes only and is not a commitment, promise or legal obligation to deliver any new products, features or functionality. The development, release, and timing of any features or functionality described remains at Riverbed's sole discretion. Some History tcpdump wireshark ethereal libpcap CACE/Riverbed BPF winpcap CACE

2 Wireshark When CACE was acquired last fall, Riverbed became the sponsor of the wireshark project Huge following and project momentum 5 million users 500,000 downloads per month Over 2 million lines of code Key component in Riverbed s strategy Continued support of ecosystem and openness But wireshark alone doesn t fully address the broader problem of network visibility... Information Overload A grab bag of point solutions... events snmp mrtg s net logs reports Information Overload Creates a hay stack of information... events snmp mrtg s net logs reports 2

3 The Challenge When lost in the hay stack, simple businesslevel questions cannot be answered... How well is the network delivering our critical applications? How well did we do last week, month, quarter? How can I communicate this to business leaders? Is there a specific problem in delivering these applications? If so, where? What caused or is causing the problem? How do we fix it? Status Quo Monitoring Automatically detect problems Alert administrators Spans whole network Broad audience Troubleshooting Investigate root cause of problem after detection Deep analysis of protocols and applications Isolated point in network Experts only Status Quo Monitoring Typically done with Troubleshooting Typically done with s 3

4 Just What is Flow? WAN specific collection points in the network data store Just What is Flow? traverse the network typically at very high speed records summarizing the s come out at very low speed Flow Info From a set of collection points, gather snapshots every minute or so... each identified by tuple (proto, address, port) network device queried for active s network device queried for active s device returns list of stats to monitoring system Flow 1: #bytes, #pkts, #drops,... Flow 2: #bytes, #pkts, #drops, Flow N: #bytes, #pkts, #drops,... 4

5 Flow Example What can you do? Take a metric like byte count Divide by sampling interval. Plot as time series bandwidth x x x time Flow Example What can you do? Take a metric like byte count Divide by sampling interval. Plot as time series bandwidth x x x time The Status Quo WAN Remote Site Data Center Flow Analyzer 5

6 The Status Quo Packet Analyzer WAN Remote Site Data Center Flow Analyzer The Problem Monitoring and troubleshooting tackled from separate silos Even when portrayed as coupled, existing integration is often loose and hard to leverage Based on decades of development in separate silos It is REALLY HARD to go from knowing a problem exists, to running wireshark at the right place and right time to diagnose the root cause We need to fix this... A Shift A shift needs to happen From point solutions The haystack of point tools To a holistic system A holistic and integrated system-wide view of the network and network services Not claiming to have solved all the problems, but we believe our new architectural approach is a great foundation for moving forward 6

7 Enter Cascade and ip/f FLOW-BASED VISIBILITY PACKET-BASED VISIBILITY Profiler (Centralized Analysis) Pilot (Packet Analysis) Gateway (Flow Collection) Sensor (Packet Inspection) Shark (Packet Capture) REMOTE FLOW/PACKET VISIBILITY Steelhead (Packet Capture & Flow Export) E n t e r C a s c a d e a n d ip/f FLOW-BASED VISIBILITY PACKET-BASED VISIBILITY Profiler (Centralized Analysis) ip/f: integrated s and Gateway (Flow Collection) Sensor (Packet Inspection) REMOTE FLOW/PACKET VISIBILITY Pilot (Packet Analysis) Shark (Packet Capture) Steelhead (Packet Capture & Flow Export) How hard can it be? On the surface, the network visibility problem just doesn t seem that hard Collect router/switch stats and dump to disk Capture s and dump to disk Capture s and dump to disk Analyze the performance database Look through the hay Find the needle 7

8 Flow Overload Some math 500,000 s / min x 200 bytes / stat x 1440 min/ day x 7 days / week = ~1 TB / week Packets even worse... Some math 100 Mb/s x 3600 sec / hr x 24 hours / day x = ~11 TB / day A data navigation problem In TB s of stored data, you need the three magic s that will reveal the problem... 8

9 A data navigation problem And, these three s could be in any of the numerous capture devices... A data navigation problem And, these three s could be in any of the numerous capture devices... ip/f: Flow Plus Packets WAN Remote Site Data Center Single Logical Record 9

10 It s hard to make things easy How Can We Make Network Visibility As Easy As This? A hierarchical storage model WAN Remote Site Data Center Packets (nsec) Micro (1 sec) Flow (1 min) Macro (15+ min) Distributed and centralized WAN DISTRIBUTED CENTRALIZED Packets (nsec) Micro (1 sec) Flow (1 min) Macro (15+ min) 10

11 ... and synchronized WAN DISTRIBUTED CENTRALIZED Packets (nsec) Micro (1 sec) Flow (1 min) Macro (15+ min) Like a brain with many eyes... WAN DISTRIBUTED CENTRALIZED Packets (nsec) Micro (1 sec) Flow (1 min) Macro (15+ min) The ip/f Architecture capture storage layer collector s micro macro distributed centralized 11

12 The ip/f Architecture computation layer capture analytics analytics collector s micro macro distributed centralized The ip/f Architecture protocol layer REST API (XML/HTTP) capture analytics analytics collector s micro macro distributed centralized The ip/f Architecture presentation layer wireshark console console REST API (XML/HTTP) capture analytics analytics collector s micro macro distributed centralized 12

13 The ip/f Architecture Cascade Pilot Cascade Profiler wireshark console console REST API (XML/HTTP) Cascade Gateway capture analytics analytics collector s micro macro Cascade Shark The ip/f Architecture Cascade Pilot Cascade Profiler wireshark console integration console REST API (XML/HTTP) Cascade Gateway capture analytics analytics collector s micro macro Cascade Shark A real example London The network is slow! The problem was somewhere in here Everything is fine here and there... Hong Kong 13

14 ip/f Drill Down Packets Micro Flow Macro ip/f Drill Down Packets Micro Flow Macro ip/f Drill Down Packets Micro Flow Macro 14

15 ip/f Drill Down Packets Micro Flow Macro ip/f Drill Down Packets Micro Flow Macro Find the right Shark box... ip/f Drill Down Packets Micro Flow Macro 15

16 ip/f Drill Down Packets Micro Flow Macro ip/f Drill Down Packets Micro Flow Macro A real example London The network is slow! The problem was somewhere in here Everything is fine here and there... Hong Kong 16

17 Now we can find the needle... London Here it is, took 15 min instead of days to track down xxx xxx Bangalore office Minimize mean-time to innocence Hong Kong So how does it work? What makes the blinky red light come on? Lots of important building blocks but a key foundational technology is behavioral analytics proactive instead of reactive know when something is wrong before getting a call from an angry user alerts without manually setting tens of thresholds adaptive... no manual updates to thresholds when the network changes Behavioral Analytics bandwidth (Mb/s) Order entry system b Week 1 Sun Mon Tue Wed Thu Fri Sat 17

18 Behavioral Analytics bandwidth (Mb/s) Order entry system b Week 1 Week 2 Sun Mon Tue Wed Thu Fri Sat Behavioral Analytics bandwidth (Mb/s) Order entry system b average bandwidth Sun Mon Tue Wed Thu Fri Sat Behavioral Analytics bandwidth (Mb/s) Order entry system b variance Sun Mon Tue Wed Thu Fri Sat 18

19 Behavioral Analytics Order entry system b bandwidth (Mb/s) Sun Mon Tue Wed Thu Fri Sat Behavioral Analytics Order entry system b bandwidth (Mb/s) Sun Mon Tue Wed Thu Fri Sat Behavioral Analytics bandwidth (Mb/s) Order entry system b Sun ANOMALY Mon Tue Wed Thu Fri Sat ALARM 19

20 Behavioral Analytics bandwidth (Mb/s) Order entry system b Problem fixed! Sun Mon Tue Wed Thu Fri Sat Behavioral Analytics Instead of telling the system how to detect a problem, tell it what performance policies you care about The analytics engines learns automatically by observing your network When things change, the engine adapts to changes by learning the new normal Tying it all together Lots of customer discussions over past 9 months A very clear trend emerged... A core set of common needs A long tail of unique needs bread & butter long tail of custom requests 20

21 Tying it all together The solution: infrastructure programmability programmatic access to analytics customizable processing logic ip/f REST layer End users can then integrate across other IT systems customize monitoring for their environment build browser-based mashups that integrate with existing portals and applications The ip/f Architecture wireshark console console REST API (XML/HTTP) capture analytics analytics collector s micro macro distributed centralized ip/f as a Platform CASCADE PILOT CASCADE PROFILER ip/f REST UI gadgets behavioral analytics dissectors capture control reports ip/f Platform info s topology alarms system config. etc... 21

22 ip/f as a Platform CASCADE PILOT CASCADE PROFILER ip/f scripting API ip/f REST UI gadgets behavioral analytics dissectors capture control reports ip/f Platform info s topology alarms system config. etc... ip/f as a Platform CASCADE PILOT... custom apps CASCADE PROFILER ip/f scripting API ip/f REST UI gadgets behavioral analytics dissectors capture control reports ip/f Platform info s topology alarms system config. etc... Some Examples Failover monitor Monitor active connections to a critical server, and make sure clients reconnect to backup server if primary server fails IDS integration When certain alerts are received from snort, reach out to shark appliances and save trace related to suspected threat Google maps mashup Show network health geographically on a google map, and correlate failures and performance degradation with natural disasters 22

23 Summary It s hard to make things easy An emerging industry trend is top-down visibility Monitor, detect problem, troubleshoot There seems to be a gap between vision and reality We think our approach fills the gap ip/f to integrate monitoring and capture Programmable infrastructure to customize the infrastructure for the long tail of requirements Seemless integration with the popular wireshark tool We re working hard to make it all happen 23