Origin- des*na*on Flow Measurement in High- Speed Networks

Transcription

1 IEEE INFOCOM, 2012 Origin- des*na*on Flow Measurement in High- Speed Networks Tao Li Shigang Chen Yan Qiao

2 Introduc*on (Defini*ons) Origin- des+na+on flow between two routers is the set of packets that pass both routers. Egress Ingress ISP network Ingress Egress Ø Traverse from one router to the other (direc<onal) Ø Traverse between two routers (undirec<onal) OD flow size: cardinality of the packet set (more details soon ) 2 /20

3 Applica*ons of OD- flow Measurement Network Access PaKern Capacity Planning Intrusion Detec<on System Accoun<ng and Billing Worm Detec<on Scanner Detec<on Discover BoKleneck Anomaly Detec<on Service Provision 3 /20

4 Problem Measure the OD- flow size between any two routers. An Example r1 r2 The OD flow size between r1 and r2 is 1 4 /20

5 Technical Challenges Fast Speed Small Memory Huge Volume 5 /20

6 One Naïve Solu*on Each router: record all packets that pass it Ø Compare two routers set Ø Unrealis<c : many packets (e.g. 100M) Ø Signature, each 160 bits, total: 2GB Ø We only want a few bits for each packet!! p1 p2 p3 6 /20

7 Another Solu*on Each router: maintains a counter for each of other routers Each packet: records all routers it has passed (r1) (r1, r2) r1 r2 r3 id value id Value value id value r2 r3 0 0 r1 r r1 r /20

8 Mo*va*on & Overview Mo<va<on Ø Storing exact info too expensive => sta<s<cal methods. Basic Idea: Each router maintains a bitmap. For each packet p Ø Pseudo- randomly mark one bit in the bitmap Ø Same packet always maps to same bit in different routers r1 r2 p9 p1 p3 p5 p4 p Under- 1 1 es<ma<on Can be stalslcally solved Bitwise when the AND bitmap of two is long bitmaps Over- enough es<ma<on 1 1 p1 p3 p7 p8 p6 p9 8 /20

9 Marking the Packet Informa*on Online: marking packet informa<on Offline: measuring OD flow size Each router maintains a bitmap B with fixed length m, ini<ally all bits are set to zero When it comes a packet p Ø H(p): H(..) is a hash func<on whose output range is [1,..,m] Ø Mark the bit to 1: B[H(p)] := 1 p1 p2 p B m 9 /20

10 Measuring the Size of Each OD Flow All routers report their bitmaps to centralized server Nota<ons: Ø Let S1 and S2 be the set of packets that pass two routers r1 and r2 Ø n1 and n2 be the cardinali<es of S1 and S2, n1 = S1, n2 = S2 Ø nc is the number of common packets that r1 and r2 share What we want to know 10 /20

11 Measuring the Size of Each OD Flow (Cont.) Measuring n1 and n2 Ø [K. Whang, etc. A linear- Lme probabilislc counlng algorithm for database applicalons, ACM Trans. On Database Systems, 1990] p1 p2 p3 p4 p5 p6 p7 p8 p9 B m = 10, V = 4/10 = * ln(0.4) = /20

12 Measuring the Size of Each OD Flow (Cont.) Measuring nc Ø Take a bitwise AND opera<on of B1 and B2, denoted as Bc B p1 p3 p2 Bc 1 1 B p1 p4 p6 p5 Ø A bit in Bc, it is 0 iff the following two condi<ons are BOTH sa<sfied 1. It is not chosen by any packet in 2. It is either not chosen by any packet in or not chosen by any packet in 12 /20

13 Measuring the Size of Each OD Flow (Cont.) The probability for a bit in Bc to remain 0 is Observe in Bc Maximum Likelihood Es<ma<on Ø Select values that produce a distribu<on that gives the observed data the greatest probability 13 /20

14 Measuring the Size of Each OD Flow (Cont.) We can compute nc in the following formula Measurement accuracy 14 /20

15 Simula*ons Simula<on setup Ø n1 and n2 are randomly selected from [100,000, 10,000,000] Ø nc is randomly selected from [100, 50,000] Ø Choose 1,000 different n1, n2, and nc Ø Run 1,000 <mes and show averaged results Most related work: Quasi- likelihood approach (QMLE): Ø Use bucket array for packet storage Ø Derive quasi- probability distribu<on of packet informa<on 15 /20

16 Simula*ons (Cont.) Per- packet Processing Overhead Measurement Accuracy 16 /20

17 Experimental Results Abilene Network Ø 12 routers at different ci<es in US Ø 24 weeks of traffic trace Ø 5 minutes for each dataset (a measurement period) Ø 0.3M to 13M packets for each router in one measurement period 17 /20

18 Experimental Results (Cont.) Per- packet Processing Overhead Measurement Accuracy 18 /20

19 Conclusions Measure OD flow size A bitmap data structure MLE method Experiments 19 /20

20 Thank you

21 Condi*on 1 : It is not chosen by any packet in Each packet: randomly select one bit in B B 1 m A specific bit b in B, each packet has probability Ø 1/m to choose it Ø (1-1/m) not to choose it p The probability for Condi<on 1 is 21 /20

22 Condi*on 2: It is either not chosen by any packet in chosen by any packet in or not A specific bit b in B, each packet has probability Ø 1/m to choose it Ø (1-1/m) not to choose it Ø The probability for b not to be chosen by any packet in Ø is Ø is 22 /20