Data Streaming Algorithms for Efficient and Accurate Estimation of Flow Size Distribution

Transcription

1 Data Streaming Algorithms for Efficient and Accurate Estimation of Flow Size Distribution Jun (Jim) Xu Networking and Telecommunications Group College of Computing Georgia Institute of Technology Joint work with: Abhishek Kumar, Jun Li, Minho Sung, Ellen Zegura, Georgia Tech. Olivier Spatscheck, Jia Wang, AT&T Labs - Research Li Li, Bell Labs Networking area Qualifying Exam: Oral Presentation

2 Outline Motivation and introduction Our data streaming works Space-Code Bloom Filter per flow traffic measurement without per flow state Estimating flow size distribution without per flow state Data streaming techniques for IP traceback Data streaming techniques for unstructured P2P routing Theory and philosophy of network data streaming

3 Motivation for network data streaming Problem: to monitor network links for events such as Elephant flows, Number of distinct flows, average flow size Flow size distribution Per-flow traffic volume Example applications: Traffic engineering Network pricing and billing, Anomaly detection, Network queue management

4 Motivation for network data streaming Monitoring at high speed is challenging packets arrive every 25ns on a 40 Gbps (OC-768) link has to use SRAM for per-packet processing per-flow state too large to fit into SRAM Traditional solution using sampling: Sample a small percentage of packets Process these packets using per-flow state stored in slow memory (DRAM) Using scaling to recover the original statistics, hence high inaccuracy with low sampling rate Fighting a losing cause: higher link speed requires lower sampling rate Conclusion: naive sampling based approach is doomed

5 Network data streaming a smarter solution Computational model: process a long stream of data (packets) in one pass using a small (yet fast) memory Problem to solve: need to answer some queries about the stream at the end or continuously Trick: try to remember the most important information about the stream pertinent to the queries Application: provide a framework for solving many network monitoring problems Difference with data streaming in DB/theory: need to run orders of magnitude faster (i.e., wisdom under fire) Status: extremely young and promising area no book, few papers, and a lot of confusion

6 Data Streaming Algorithm for Estimating Flow Distribution

7 Data Streaming Algorithm for Estimating Flow Distribution Problem: To estimate the probability distribution of flow sizes. In other words, for each positive integer i, estimate n i, the number of flows of size i. Applications: Traffic characterization, network billing/accounting, anomaly detection, etc. Importance: The mother of many other flow statistics such as average flow size (first moment) Definition of a flow: All packets with the same flow-label. The flow-label can be defined as any combination of fields from the IP header, e.g., <Source IP, source Port, Dest. IP, Dest. Port, Protocol>.

8 Related work: Inverting sampled packet traces. Current measurement boxes only collect sampled traces. The approach is to invert the sampled distribution to obtain the actual distribution [Duffield, Lund, & Thorup 2003]. High estimation errors due to low sampling rates. Theoretical limitations to inverting sampled traffic [Hohn & Veitch, IMC 2003].

9 Our approach: network data streaming Main idea: process each and every packet using a fast SRAM to skim most important information Design philosophy: Lossy data structure + Bayesian statistics = Accurate streaming Information loss is unavoidable: (1) memory very small compare to the data stream (2) too little time to put data into the right place Control the loss so that Bayesian statistical techniques such as Maximum Likelihood Estimation can still recover a decent amount of information.

10 Architecture of our Solution System Model Packet stream Header Header Header 1. Update Online Streaming Module 2. Raw streaming result Offline Processing Module 3. Flow distribution

11 Architecture of our Solution Lossy data structure Measurement proceeds in epochs (e.g. 100 seconds). Maintain an array of counters in fast memory (SRAM). For each packet, a counter is chosen via hashing, and incremented. No attempt to detect or resolve collisions. Data collection is lossy (erroneous), but very fast. Each 32-bit counter only uses 9-bit of SRAM (due to [Ramabhadran & Varghese 2003]) At the end of the epoch, the counter array is paged to disk.

12 The shape of the Counter Value Distribution 1e Actual flow distribution m=1024k m=512k m=256k m=128k frequency flow size The distribution of flow sizes and raw counter values (both x and y axes are in log-scale). m = number of counters.

13 Architecture of our solution Bayesian statistics The counter array is processed to obtain the Counter Value Distribution. {m o =# counters with value 0, y 1 =# counters with value 1,, y z =# counters with value z.} Use Bayesian statistics to derive the following quantities from this data: The total no. of flows, n. The total no. of flows with exactly one packet, n 1. The flow distribution φ.

14 Estimating n and n 1 Let total number of counters be m. Let the number of value-0 counters be m 0 Then ˆn = m ln(m/m 0 ) Let the number of value-1 counters be y 1 Then ˆn 1 = y 1 eˆn/m Generalizing this process to estimate n 2, n 3, and the whole flow size distribution will not work Solution: joint estimation using Expectation Maximization

15 Expectation Maximization in a nutshell Let Φ be the parameters we would like to estimate Let Y be the observation Goal: find Φ that maximizes the likelihood function P (Φ Y), i.e., the Maximum Likelihood Estimator. Problem: P (Φ Y), viewed as f(φ, Y), can be hard to evaluate and maximize Solution: P (Φ Y), viewed as g(φ, Y, Γ), is often easy to compute and maximize as a function of Φ. Also E[Γ Φ old, Y] is often easy to evaluate as a function of Γ Obtain Γ = E[Γ Φ old, Y] Maximize g(φ, Y, Γ) to obtain Φ new Repeat the above two steps until the algorithm converges.

16 Estimating the entire distribution, φ, using EM Begin with a guess of the flow distribution, φ ini. Based on this φ ini, compute the various possible ways of splitting a particular counter value and the respective probabilities of such events. This allows us to compute a refined estimate of the flow distribution φ new. Repeating this multiple times allows the estimate to converge to a local maximum. This is an instance of Expectation maximization.

17 Estimating the entire flow distribution an example For example, a counter value of 3 could be caused by three events: 3 = 3 (no hash collision); 3 = (a flow of size 1 colliding with a flow of size 2); 3 = (three flows of size 1 hashed to the same location) Suppose the respective probabilities of these three events are 0.5, 0.3, and 0.2 respectively, and there are 1000 counters with value 3. Then we estimate that 500, 300, and 200 counters split in the three above ways, respectively. So we credit 300 * * 3 = 900 to n 1, the count of size 1 flows, and credit 300 and 500 to n 2 and n 3, respectively.

18 Evaluation Before and after running the Estimation algorithm 1e Actual flow distribution Raw counter values Estimation using our algorithm 1000 frequency flow size

19 Sampling vs. array of counters Web traffic Actual flow distribution Inferred from sampling,n=10 Inferred from sampling,n=100 Estimation using our algorithm frequency flow size

20 Sampling vs. array of counters DNS traffic Actual flow distribution Inferred from sampling,n=10 Inferred from sampling,n=100 Estimation using our algorithm frequency flow size

21 Space Code Bloom Filter for per-flow measurement

22 Space Code Bloom Filter for per-flow measurement Problem: To keep track of the total number of packets belonging to each flow at a high speed link. Network billing/accounting and anomaly detec- Applications: tion Challenges: 1. Traditional techniques such as sampling will not work at high link speed. 2. Existing streaming algorithms only track the identity of the elephants [Estan and Varghese 2002]

23 Architecture and system model of SCBF Measurement proceeds in epochs (e.g. 10 second). Maintain an aggregate synopsis data-structure. Update the data-structure on every packet arrival. Write-only data structure fast updates, low hardware complexity. Copies of the synopsis are paged to disk periodically. 0. New packet arrival Header 1. Process header CPU 2. Write to SCBF SCBF Module SRAM Module 1 SRAM Module 2 3. Paging to disk once "full" Persistent Storage 4. Query 5.Answer

24 Architecture and system model of SCBF Queries provide a flow-label and ask for its size. Obtain raw data from the data-structure and infer the approximate flow size from the raw data (a table lookup process) This provides approximate estimates that have low relative error with high probability.

25 Balls in Bins

26 Balls in Bins Packet Processor

27 Balls in Bins Packet Processor Pick one bin (uniformaly at random)

28 Balls in Bins Packet Processor Mark Bin as occupied

29 Balls in Bins Packet Processor Mark Bin as occupied (by setting the corresponding bit to 1)

30 Balls in Bins - Estimation # Occupied bins =

31 Balls in Bins - Estimation #packets=2 #packets=3 #packets=4 pdf pdf pdf # bins # bins # bins # Occupied bins =

32 Balls in Bins - Estimation #packets=2 #packets=3 #packets=4 pdf pdf pdf # bins # bins Maximum Likelihood estimate # Occupied bins = # bins

33 Using Bloom Filters to Implement Bins Insert (label) h 1, h 2,..., h K h 1, h 2,..., h K Yes Query (label) All 1? No

34 Use of multiple Bloom Filters to implement Multiple Bins (1) (1) (2) (2) h 1, h 2,..., h (2) 1, h2,..., h (1) (l) (l) (l) K h K h 1, h2,..., h K Bloom Filter 1 Bloom Filter 2 Bloom Filter L Insert each ball (packet) in one of l bins (Bloom filters), chosen uniformly at random. While querying, count number of bins (Bloom filters) in which one or more balls (packets) were inserted and estimate the size of the flow from this count.

35 Space Code Bloom Filter - Multiple bins in the same Bloom Filter (1) (1) (2) (2) h 1, h 2,..., h (2) 1, h2,..., h (1) (l) (l) (l) K h K h 1, h2,..., h K Bloom Filter 1 Bloom Filter 2 Bloom Filter L

36 Querying an SCBF Query (label) Yes No Yes Count = (# Yes) (1) (1) (2) (2) h 1, h 2,..., h (2) 1, h2,..., h (1) (l) (l) (l) K h K h 1, h2,..., h K Bloom Filter 1 Bloom Filter 2 Bloom Filter L

37 Extending the range of SCBF An SCBF of size l fills up after about (l ln l) insertions (result from the classic coupon collector s problem). So we can t use it for estimating flow sizes much larger than (l ln l). Objective To provide a constant relative error, irrespective of flow-size. Solution Use multiple SCBFs operating at different resolutions. Small flows will be captured by SCBFs of finer resolution. Large flows will be captured by SCBFs of coarse resolutions.

38 Multi-Resolution SCBF (1) (1) (2) (2) h 1, h 2,..., h (2) 1, h2,..., h (1) (l) (l) (l) K h K h 1, h2,..., h K Bloom Filter 1 Bloom Filter 2 Bloom Filter L SCBF 1 Sampling rate 1 (1) (1) (2) (2) h 1, h 2,..., h (2) 1, h2,..., h (1) (l) (l) (l) K h K h 1, h2,..., h K Bloom Filter 1 Bloom Filter 2 Bloom Filter L SCBF 2 Sampling rate 1/4 (1) (1) (2) (2) h 1, h 2,..., h (2) 1, h2,..., h (1) (l) (l) (l) K h K h 1, h2,..., h K SCBF r Sampling rate 1/4 r 1 Bloom Filter 1 Bloom Filter 2 Bloom Filter L

39 Original vs. estimated flow size. Note that both axes are on logscale. Accuracy of SCBF estimated original Estimated flow length (packets) Original flow length (packets)

40 Accuracy of SCBF using Maximum Likelihood Estimation flows >=10 --> <-- flows >=5 P[relative err < e] (CDF) all flows flows >=2 flows >=3 flows >=5 flows >=10 flows >=100 flows >=3 ---> flows >=100 --> <-- all flows e CDF of relative error for flows of various size

41 Performance of SCBF - complexities and accuracy Computational complexity compute 5 hash-functions and write 5 bits per packet. Space complexity 4 bits of storage required for each packet. Can operate at OC-768 (40 Gbps) with 5 ns SRAM. More than 80% responses are within ±25% of the actual value.

42 Data streaming techniques for IP traceback Efficient and Scalable Query Routing for Unstructured Peer-to-Peer Networks Abhishek Kumar Jun (Jim) Xu Ellen W. Zegura Georgia Institute of Technology

43 Search in Unstructured Peer-to-Peer Networks Flooding is the naive solution. Random walk reduces duplicate queries. Replication of content indices improves both solutions. However, such replication is expensive and not efficient. How about replicating probabilistic information?

44 Search using Probabilistic information Noise Low information Medium information Strong information Noise Content Host (Origin of information) Noise

45 Exponential Decay Bloom Filter EDBF A data structure for succinct representation of probabilistic information. Number if bits used for representing a probability value is proportional to the value. Decay of information with network distance. Easy set union operation.

46 Routing using EDBF Nodes advertise their own content using all bits (strng infromation). Information from neighboring nodes is combined into one EDBF and decayed by a constant factor. The union of this decayed information with the local advertisement is then propagated to neighbors. Efficient propagation of probabilistic information. Delta compression of updates is easy and results in huge savings.

47 Simulation Results % of queries answered SQR OHR Random walk flooding % of queries answered SQR OHR Random walk flooding Hop Limit (a)flat topologies e+06 Hop Limit (b) Flat topologies, log scale on x-axis.

48 Simulation Results % of queries answered SQR+GIA GIA Ultrapeer Hop Limit (c) Hierarchical topologies, log scale on x- axis.

49 Distributed coordinated data streaming a new paradigm A network of streaming nodes Every node is both a producer and a consumer of data streams Every node exchanges data with neighbors, streams the data received, and passes it on further From EE point of view: distributed source coding + distributed channel coding

50 Philosophy of network data streaming View it as special-purpose data compression Parametric or nonparametric statistics? View it as a channel design problem Future work: develop the filter theory for network data streaming

51 Contributions Data streaming algorithms for estimating flow size distribution and per-flow traffic volume, for performing better IP traceback, and for searching more efficiently in P2P networks A new methodology for network data streaming: Lossy data structure + Bayesian statistics = Accurate Streaming A new data streaming paradigm: distributed coordinated streaming Help lay the foundation of this new field.

52 Related publications Kumar, A., Xu, J., Zegura, E. Efficient and Scalable Query Routing for Unstructured Peer-to-Peer Networks, to appear in Proc. of IEEE Infocom Kumar, A., Sung, M., Xu, J., Wang, J. Data Streaming Algorithms for Efficient and Accurate Estimation of Flow Distribution", in Proc. of ACM Sigmetrics 2004/IFIP WG 7.3 Performance 2004, Best Student Paper Award. Li, J., Sung, M., Xu, J., Li, L. Large-Scale IP Traceback in High-Speed Internet: Practical Techniques and Theoretical Foundation", in 2004 IEEE Symp. on Security & Privacy. Kumar, A., Xu, J., Wang, J., Spatschek, O., Li, L. Space-Code Bloom Filter for Efficient Per-Flow Traffic Measurement, in Proc. of IEEE Infocom 2004.