UnivMon: Software-defined Monitoring with Universal Sketch

Transcription

1 UnivMon: Software-defined Monitoring with Universal Sketch Zaoxing (Alan) Liu Joint work with Antonis Manousis (CMU), Greg Vorsanger(JHU), Vyas Sekar (CMU), and Vladimir Braverman(JHU)

2 Network Management: Many Monitoring Requirements Flow size distribution Entropy, Traffic Changes Traffic Engineering SuperSpreaders Worm Detection Anomaly Detection Heavy-hitters Accounting. SDN Controller (OpenDayLight etc.)

3 Network Management: Many Monitoring Requirements Flow size distribution Entropy, Traffic Changes Traffic Engineering SuperSpreaders Worm Detection Anomaly Detection Heavy-hitters Accounting. Typical Measurement Questions: Who s sending a lot more traffic than 0min ago? (Change) Who s sending a lot from /6? (Heavy Hitter) Are you being DDoS-ed?

4 Example: A Victim being DDoSed

5 Traditional: Packet Sampling Sample packets at random, aggregate into flows Flow = Packets with same pattern Source and Destination Address and Ports FlowId Counter 2 Flow reports Estimate: FSD, Entropy, Heavyhitters Not good for fine-grained analysis Extensive literature on limitations for many tasks!

6 Application-Specific Sketches Bloom-filter, Count-min Sketch, reversible sketch, etc. Heavy Hitter Entropy Superspreader Application-Level Metric Counter Data Structures Packet Processing Application-Level Metric Counter Data Structures Packet Processing.. Application-Level Metric Counter Data Structures Packet Processing Computation (off router) Monitoring (on router) Traffic Complexity: Need per-metric implementation Recent Example: OpenSketch [NSDI 3] Trend: Many more applications appear!

7 App A Generic Method... App n Control Plane Application-specific Computation Packet Processing Traffic Data Structure Data Plane. Generic 2. Not too expensive 3. Accurate for finegrained analysis

8 Outline Motivation UnivMon System Design UnivMon Algorithm Evaluation

9 Our Design: UnivMon UnivMon Control+ Plane 3.#Metric#Estimation Control plane Metric estimation Monitoring manifest distribution.#distribute# Manifests UnivMon Data Plane Data Plane Traffic data collection Sketch counter updates 2.#Collect#Sketch#counters Late-binding for applications: data plane is general-purpose. One Sketch: no need of memory allocation for multiple tasks

10 Outline Motivation UnivMon Design UnivMon Algorithm Evaluation

11 Theory of Universal Streaming [BO 0, BO 3] Estimated G-sum Universal Sketch As long as g(fi) does not grow monotonically faster than f i 2 G-sum = i= n g(fi) frequency vector is < f,f 2 f n > (A stream of length m with n unique items) When g(x) = x 0, G-sum is # of distinct items. [SuperSpreader] When g(x) = xlog(x), G-sum is the entropy norm. [Entropy]

12 Basics: Streaming Algorithms (A stream S of length m with n unique items) K-th Frequency Moments: F k (S) = i= n k f i F 0 is n: the number of distinct items in S. F is m: the length of S. F 2 is Gini's index of homogeneity. When k>2, the space lower bound is Ω(n 2/k ) [CKS 03] (For current applications, the case of k>2 is not that interesting) Heavy Hitters (Frequent Items): g-heavy item i : g(f i ) > α G-sum for some α Count-min sketch [CM 04] is a popular L-heavy hitter algorithm

13 0 Universal Sketch Data Structure Levels In Parallel H()=, H(5)=, H(2)= H2(5)=, H2(2)= Count Sketch Alg L2 Heavy Hitter Algorithms Heavy Hitter Alg Heavy -2 Hitter + Alg Heavy Hitters (,4), (3,2),(5,2) (,4), (5,2),(2,) log(n) 5 H3(2)= Generate log(n) substreams by zero-one hash funcs H.H log(n) Heavy Hitter Alg + Heavy Hitter Alg Count-Sketch etc. Similar to counting bloom filter (5,2), (2,) (2,)

14 Levels 0 Counters from Universal Sketch (,4), (3,2),(5,2) (,g(4)), (3,g(2)),(5,g(2)) (,4), (5,2), (2,)... Estimating G-sum (,g(4)), (5,g(2)), (2,g()) Estimated G-sum Y0=2g()+2g(2)+g(4) Apply arbitrary g() Y=g()+g(2)+g(4)... log(n) (5,2),(2,) (5,g(2)),(2,g()) (2,) Sum of the g()s (2,g()) Y2=g()+g(2) Y3=g() Recursive Steps: Yi- = 2Yi + new counters repeated counters 3

15 Intuitions of Universal Sketch. Group items into log(n) groups 2. Find (>=) one from each group 3. Recursively sum them all! Idea: Detect items that contribute most to G-sum

16 Putting it together: UnivMon Hash functions and sketch counters Offline Computation

17 Application-specific Packet Our Approach: Sampling UnivMon Solution Approach I OK, want I got to monitor: it. But I Heavy results will I want calculate Hitter, more are not a DDoS, Entropy, OK, I got etc. it large appications! good class enough! of funcs! Controller OK! Universal Flow reports Sketch OK! OK! I need I additional algorithm Count k-ary sketch Min OK! xxx bitmap Sketch for Changes xxx. for DDoS Oh HHno!

18 Outline Motivation UnivMon Design UnivMon Algorithm Evaluation

19 Key Evaluation Questions Feasibility of hardware deployment? Removed expensive data structures Implemented in P4 UnivMon s accuracy Compare to the up-to-date sketch algorithms UnivMon s stability Stabilities over different traces

20 Evaluation Comparison with custom sketches via OpenSketch

21 Evaluation Comparison with custom sketches via OpenSketch

22 Conclusions Traditional packet sampling based approaches have limitations for fine-grained analysis. Custom Sketches (e.g. OpenSketch) Need to know applications beforehand High Implementation cost More expensive on multi-tasks UnivMon: an efficient and general sketching based approach with late-binding on applications.

23 Future Directions Multi-dimensional data Performance optimization for hardware Dynamically change monitoring scope New Theories on Universal Sketch: Sliding Window Model [SODA 6] Functions of One Variable [PODS 6] Symmetric Norms [arxiv 5.0]

24 Thank you!