Sampling Challenges. Tanja Zseby Competence Center Network Research Fraunhofer Institute FOKUS Berlin. COST TMA September 22, 2008

Transcription

1 Sampling Challenges Tanja Zseby Competence Center Network Research Fraunhofer Institute FOKUS Berlin

2 Desired Features for Traffic Observation Network-wide: multiple observation points Flexible: change viewpoints Shareable: comparable results But: We can not measure everything everywhere! 2of 14

3 Problem: Resource Limitations Resource Limitations Processing power Transmission Storage Demand depends on Data rates Required granularity Solutions Dedicated Hardware Improved Algorithms Data Selection Additional CPU load for running NetFlow on different routers* 1:100 *source: NetFlow Performance Analysis, Cisco white paper 3of 14

4 Example: Flow Measurements Examples Flow:= packets with common properties all packets from host A to host B all small packets all TCP-SYN packets all HTTP packets with URL Customer Network Customer Network End System Provider Network 4of 14

5 Example: Flow Measurements Packets: <s 1, t 1, c 1 >, <s 2, t 2, c 2 >,... <s N, t N, c N > Packet Attributes at Observation Point s i, sequence number t i arrival time c i content (header, payload) Classification f(c i ) Flows: FlowID 1: <s 1, t 1, c 1 > <s 4, t 4, c 4 > <s 8, t 8, c 8 > FlowID 2: <s 2, t 2, c 2 > <s 3, t 3, c 3 > <s 6, t 6, c 6 > FlowID 3: <s 5, t 5, c 5 > <s 7, t 7, c 7 > <s 9, t 9, c 9 > Aggregation Aggregation Aggregation Flow Records: < N, μ, σ,... > < N, μ, σ,... > < N, μ, σ,... > f f f f f f f f f 5of 14

6 Packet Selection Packets: Selected Packets: <s 1, t 1, c 1 >, <s 2, t 2, c 2 >,... <s N, t N, c N > f(c i ) Filtering Packet Selection f(s i ) or f(t i,) Sampling <s 2, t 2, c 2 >, <s 6, t 6, c 6 >... <s n, t n, c n > Classification Flows: FlowID 1: <s 8, t 8, c 8 > FlowID 2: <s 2, t 2, c 2 > <s 6, t 6, c 6 > FlowID 3: <s 5, t 5, c 5 > <s 9, t 9, c 9 > Aggregation Aggregation Aggregation Flow Records: < Nˆ, μˆ, σˆ,... > f f f < Nˆ, μˆ, σˆ,... > < Nˆ, μˆ, σˆ,... > f f f f f f 6of 14

7 Challenge1: Accuracy Assessment Accuracy Assessment required Achievable accuracy depends on Sampling and estimation method Sampling parameters Population characteristics unknown and highly dynamic Accuracy assessment during measurement For each measurement interval For each flow Based on sampled data Bias Precision 7of 14

8 Estimation Accuracy All Flows random nofn n/n=5% Flows with High Accuracy StdErr rel CIRIL S24D00 ( ) 1 N N σ + μ N μ = N μ n N f f f f f f f VEGAS project, funded by Cisco Systems NetFlow Records CIRIL S24D00, compliant, rel. stderr<0.05 N f, Store sum of squares! x f 2 xf 8of 14

9 Challenge 2: Multipoint Data Selection Goal: Select same packet at different observation points <s A, t A, c 1 > <s B, t B, c 1 > X X? <s A, t A, c 1 > Sampling: f(s i ) or f(t i,) Filtering: f(c i ) Packet Attributes at Observation Point s i, sequence number t i arrival time c i content (header, payload) 9of 14

10 Coordinated Packet Selection Trajectory Sampling [Duffield&Grossglauser, 2001] Packet Content c 1 Hash-function Hash-value [ ] ] Selection Decision f(c 1 )=1 f(c 1 )=0 Goal: Emulate random selection 10 of 14

11 Coordinated Packet Selection Problem1: Hash Input Not all content suitable (e.g. constant among pkts,changeable on path) IPv6 different fields, few data available Middlebox operations (e.g., Network Address Translation) Problem2: Predictability of selection decision [Goldberg&Rexford, 2007]: Crypto-strong PRF with secret key Problem3: Deterministic operation Biased Selection Traffic Dependent Problem4: Variability of traffic Sample size variation 11 of 14

12 FOKUS Contribution Recommended hash input Based on empirical investigations Initial investigations on IPv6 Recommended hash functions Empirical investigations on bias Different traces and functions Standardization Sampling Methods Hash functions Accuracy Assessment Christian Henke, Carsten Schmoll, Tanja Zseby, Empirical Evaluation of Hash Functions for Multipoint Measurements, ACM CCR, Vol. 38, July of 14

13 IETF Standardization IP Flow Information Export (IPFIX) RFC5101 Export of flow information from routers and probes Flexible flow definition Packet Sampling (PSAMP) Export of packet information over IPFIX Specification of packet selection techniques Configuration of packet selection techniques FOKUS contribution IPFIX requirements (RFC3917), implementation guidelines, applicability statement, file format, testing, extensions PSAMP selection techniques FOKUS Open Source IPFIX library 13 of 14

14 Call for Papers: PAM 2009 Passive and Active Measurement Conference PAM 2009 April 1-3, 2009 Seoul, Korea Paper Registration: September 24th, 2008, 23:59 GMT Paper Submission: October 1st, 2008, 23:59 GMT 14 of 14

15 Thank you! FOKUS Open Source IPFIX Library: Measurement data always welcome at: