Power of Slicing in Internet Flow Measurement. Ramana Rao Kompella Cristian Estan

Size: px

Start display at page:

Download "Power of Slicing in Internet Flow Measurement. Ramana Rao Kompella Cristian Estan"

Kerrie Jacobs
5 years ago
Views:

1 Power of Slicing in Internet Flow Measurement Ramana Rao Kompella Cristian Estan 1

2 IP Network Management Network Operator What is happening in my network? How much traffic flows towards a given destination? What are the popular applications in my network? How much should I charge my customer? Flow Measurement helps answer these questions! 2

3 Flow Measurement Flow is defined by the various packet header fields E.g. five-tuple consisting of SIP, DIP, SP, DP and protocol Measuring characteristics of a flow such as number of packets, bytes, TCP flags etc. Aggregates computed from individual flow records collected from various routers Popular applications - aggregation on port number Customer traffic - aggregation on IP address/prefix NetFlow widely used and supported in routers Flow records created on first packet seen Flow records expired on flags, inactive timeout, active timeout 3

4 Contributions New flow measurement algorithms and estimators Flow slicing Multifactor smart sampling A better flow measurement solution Better adaptability Lower resource consumption 4

5 Flow Measurement at Routers Bottleneck 1: CPU Processing Bottleneck 2: Memory Utilization Bottleneck 3: Reporting Bandwidth Packet Arrival Update flow record Create flow if first packet Report expired flows to monitoring station Packet sampling: CPU, Memory, Bandwidth Sample and hold: Memory, Bandwidth Smart sampling (size dependent): Bandwidth 5

6 Effect of sampling algorithms Sampling algorithm Byte counts Packet counts Flow counts (TCP) Flow counts (non-tcp) No sampling exact exact exact exact Packet sampling [cisco] [duffield02] impossible [chaudhuri98] Sample & hold [estan02]??? Flow slicing Smart sampling [duffield01] very in [duffield03] Multifactor smart sampling very very very 6

7 Flow measurement solutions Packet arrival NetFlow Sampled NetFlow Adaptive NetFlow Flow Slices Reduce CPU Reduce Memory N/A Static Sampling Probability Adaptive Sampling Probability Static sampling probability Adaptive flow slicing probability Reduce Bandwidth Smart sampling Smart sampling Smart sampling Multifactor smart sampling Flow record sent to monitoring 7

8 Flow Slices: Control CPU Packet Arrival Packet sampling stage controls CPU usage Choose q as maximum probability required to operate within CPU constraints Packet Sampling with probability q lookup Entry not found Flow memory 8

9 Flow Slices: Control Memory Probability p controls the rate at which records are created Every record expired after a slice duration of t Slice duration controls staleness of data Configurable inactivity timeout Can adapt slicing probability based on traffic mix any time. Packet Arrival Packet Sampling with probability q Flow slicing with probability p lookup Entry not found Create flow Time out after slice duration or after inactivity timeout Flow memory 9

10 Unbiased estimators Produces unbiased estimators for packet, byte and SYN counters Proofs in the paper When a new flow is created after it passes the flow slicing stage: Set the packet counter to 1/p Set the byte counter to b/p (b is size of the first packet) Set the SYN counter to 1/p, if packet has a SYN flag, 0 otherwise 10

11 Solutions Packet arrival NetFlow Sampled NetFlow Adaptive NetFlow Flow Slices Reduce CPU Reduce Memory N/A Static Sampling Probability Adaptive Sampling Probability Static sampling probability Adaptive flow slicing probability Reduce Bandwidth Smart sampling Smart sampling Smart sampling Multifactor smart sampling Flow record sent to monitoring 11

12 Smart sampling [duffield01] Flow records collected need to be transmitted to a centralized location for further analysis Smart sampling reduces bandwidth requirements while preserving accuracy Each record transmitted with a probability min(1,b/z) b is the size of the flow in bytes z is a threshold Large flow records are always transmitted Small flow records are transmitted with probability proportional to their size Can compute byte counts for aggregates from sampled records 12

13 Multi-factor smart sampling A generalization of smart sampling that improves accuracy of byte, packets and SYN count estimates Choose to transmit a record (s, b, a) based on a probability r = min (1, s/z s + b/z b + a/z a ) s size of flow in number of packets b size of flow in number of bytes a SYN count Choosing appropriate z s, z b and z a can balance the requirements of all the estimators Receiver counts the record as (s/r, b/r, a/r), so that estimates are unbiased 13

14 Flow Slices: Putting it all together Packet arrival Packet sampling with probability q Reduces Bottleneck 1: CPU Flow Slicing with probability p Reduces Bottleneck 2: Memory Multifactor Smart Sampling Reduces Bottleneck 3: Bandwidth Flow record sent to monitoring 14

15 Evaluation: Accuracy Two-sided errors indicate unbiasedness Real traces from CAIDA (transit link) Flow slicing probability = 0.8% (1 in 125) Slice duration = 60 seconds Trace length = 1 hour 15

16 Accuracy vs Memory tradeoff Mean Relative Error (Flows > 5000) Decreases rapidly as flows are stored for longer time Mean Relative Error Memory Usage Almost linear Increase in memory Flow Slicing Probability Memory Usage 16

17 Smart vs Multifactor sampling Compare outputs of smart and multifactor smart sampling w.r.t to true value Thresholds used: Z s =1000 packets, Z b =500,000 bytes, Z a =50 flows, for smart sampling Z=50,000 bytes Picked threshold so reduction in records in both cases from 1,700,000 to 190,000 Pkts Bytes SYNs Web 0.3% 0.4% 0.1% 0.7% 1.6% 0.8% Kazaa 0.6% 0.4% 0.1% 0.2% 12.4% 2.4% telnet 0.9% 0.6% 1.0% 0.8% 39.2% 4.9% 17

18 Comparison with ANF ANF sampling probability fixed at 1 in 1024 For Flow Slices, packet sampling q = 1/16 and flow slicing p = 1/64 combined produces 1/1024 Results: Flow Slices perform better than ANF for individual flows Similar errors when aggregated by port (for various applications) 18

19 Comparison with ANF Port Number / range ANF Slices (60s) Slices (180s) Slices (300s) Web (80) 0.5% 0.4% 0.5% 0.3% Kazaa (1214) 1.2% 1.0% 0.8% 1.0% > 50, % 5.4% 3.7% 3.1% % 15.3% 12.2% 10.8% 19

20 Memory Comparison Slice / bin ANF Flow Memory Slices Inactive Volume of Records ANF Slices Inactive 60s 1,195 1, ,764 63,658 68, s 3,141 3, ,229 57,028 61, s 4,158 4, ,730 53,953 57,635 Volume of records roughly similar Flow Slices with inactive timeouts outperforms ANF 20

21 Memory Comparison Slice / bin ANF Flow Memory Slices Inactive Volume of Records ANF Slices Inactive 60s 5,641 5,378 3,065 27,509 25,896 26, s 14,049 14,046 3,944 23,994 22,896 23, s 21,716 21,667 4,218 21,716 21,667 22,841 Volume of records roughly similar Flow Slices with inactive timeouts outperforms ANF 21

22 Conclusions Three critical router constraints CPU, memory and bandwidth Flow Slices provides three different tuning knobs to control all these resources In comparison with current approaches, Consumes less memory for better or comparable accuracy Can preserve accuracy for different measures of traffic with reduced bandwidth usage 22

23 QUESTIONS? 23

The Power of Slicing in Internet Flow Measurement

The Power of Slicing in Internet Flow Measurement Ramana Rao Kompella University of California, San Diego ramana@cs.ucsd.edu Cristian Estan University of Wisconsin-Madison estan@cs.wisc.edu Abstract Network