Scaling Hardware Accelerated Network Monitoring to Concurrent and Dynamic Queries with *Flow

Transcription

1 Scaling Hardware Accelerated Network Monitoring to Concurrent and Dynamic Queries with *Flow John Sonchack, Oliver Michel, Adam J. Aviv, Eric Keller, Jonathan M. Smith

2 Measuring High Speed Networks 00 Gb/s Links > Tb/s Switches

3 Measuring High Speed Networks Debugging Who is causing microbursts? Queue Lengths Drop Counts 00 Gb/s Links > Tb/s Switches 3

4 Measuring High Speed Networks Debugging Who is causing microbursts? Queue Lengths Drop Counts 00 Gb/s Links > Tb/s Switches Traffic Engineering Which flows are colliding? Utilization 4

5 Measuring High Speed Networks Debugging Who is causing microbursts? Queue Lengths Drop Counts 00 Gb/s Links > Tb/s Switches Traffic Engineering Which flows are colliding? Utilization Security Are any hosts compromised? Packet timing statistics 5

6 Measurement Challenges Concurrent Applications High Packet Rates Debugging Who is causing microbursts? Queue Lengths Drop Counts Traffic Engineering Which flows are colliding? Utilization 3.$Tb/s >$00$M$ packets/s >$0$M$ flows/s Security Are any hosts compromised? Packet timing statistics 6

7 Prior Measurement Systems Custom Hardware (e.g., NetFlow) Processing Cluster 7

8 Prior Measurement Systems Custom Hardware (e.g., NetFlow) Processing Cluster Flexibility Applications can t define custom statistics Efficiency Measurement Tput: < M packets / second per core Network Tput: > 00 M packets / second per switch 8

9 Prior Measurement Systems Custom Hardware (e.g., NetFlow) Flexibility Processing Cluster Efficiency Reconfigurable Switch ASICs [Marple SIGCOMM 7, TurboFlow EuroSys 8, Sonata SIGCOMM 8] SELECT packet.length GROUP BY tcp flow AGGREGATE average 9

10 Prior Measurement Systems Custom Hardware (e.g., NetFlow) Flexibility Processing Cluster Efficiency Reconfigurable Switch ASICs [Marple SIGCOMM 7, TurboFlow EuroSys 8, Sonata SIGCOMM 8] Concurrency SELECT packet.length GROUP BY tcp flow AGGREGATE average 0

11 Concurrency Challenges with Reconfigurable ASICs Max Queue Length Total Drop Count Firewall L Forwarding Max Queue Len Drop Ct.

12 Concurrency Challenges with Reconfigurable ASICs Max Queue Length Total Drop Count Std Dev. Packet Size Avg. Flow Duration Avg. Utilization Jitter Avg. Utilization Std Dev Pkt Size Avg. Jitter Firewall L Forwarding Avg. Dur. Max Queue Len Drop Ct.

13 Concurrency Challenges with Reconfigurable ASICs Not Enough room in the datapath for all the queries. Max Queue Length Total Drop Count Std Dev. Packet Size Avg. Flow Duration Avg. Utilization Jitter Avg. Utilization Std Dev Pkt Size Avg. Jitter Firewall L Forwarding Avg. Dur. Max Queue Len Drop Ct. 3

14 Concurrency Challenges with Reconfigurable ASICs Avg. Jitter Firewall L Forwarding Avg. Dur. Max Queue Len Drop Ct. 4

15 Concurrency Challenges with Throughput (Gb/s) Reconfigurable ASICs Recompile P4 ASIC at 9 seconds Recompiling the datapath disrupts measurement and forwarding. Avg. Jitter Timestamp (Seconds) Firewall Max Queue Len L Forwarding Avg. Dur. Drop Ct. 5

16 *Flow: Efficiency and Concurrency? Question: can we leverage reconfigurable ASICs for concurrent monitoring? 6

17 *Flow: Efficiency and Concurrency? Question: can we leverage reconfigurable ASICs for concurrent monitoring? Insight: concurrency challenges are caused by trying to do too much in the ASIC. 7

18 The Main Idea 8

19 The Main Idea Preprocessing Calculation SELECT packet.length GROUP BY tcp flow AGGREGATE max Average Utilization Calculation Preprocessing Jitter Calculation Preprocessing Max Packet Length Calculation Preprocessing 9

20 The Main Idea Average Utilization Calculation Jitter Calculation Max Packet Length Calculation Preprocessing Preprocessing Preprocessing 0

21 The Main Idea Average Utilization Calculation Jitter Calculation Max Packet Length Calculation Common Preprocessing

22 The Main Idea Average Utilization Calculation Max Packet Length Calculation Concurrency Jitter Calculation Common Preprocessing

23 The Main Idea Average Utilization Calculation Max Packet Length Calculation Concurrency Jitter Calculation? Efficiency Common Preprocessing 3

24 The Main Idea Average Utilization Calculation Max Packet Length Calculation Concurrency Jitter Calculation? Efficiency Common Preprocessing 4

25 The Main Idea Concurrency? Average Utilization Calculation Max Packet Length Calculation Jitter Calculation Common Preprocessing Efficiency 5

26 The Main Idea Average Utilization Calculation Max Packet Length Calculation Concurrency Jitter Calculation Measure every packet in > Tb/s traffic with server. Efficiency Common Preprocessing 6

27 The Main Idea Application specific calculation Concurrency *Flow Agent Preprocessing Efficiency *Flow Cache 7

28 Outline Motivation Design Implementation Evaluation 8

29 *Flow Design SELECT ip length GROUP BY tcp flow AGGREGATE sum. Decoupling calculation. Generalize selection & grouping Common Preprocessing App. Specific Calculation 9

30 Decoupling Calculation SELECT ip length GROUP BY tcp flow AGGREGATE sum Key: A -> B Length: 00 Flow Key A -> B C -> D E -> F Length Sum Preprocessing App. Specific Calculation 30

31 Decoupling Calculation SELECT ip length GROUP BY tcp flow AGGREGATE sum Grouped Packet Vector (GPV):!!!!!!!!!!!!!! " { flowkey : packetlength } Flow Key Pkt Len Length Sum Key: A -> B Length: 00 A -> B C -> D E -> F Preprocessing App. Specific Calculation 3

32 Generalizing Selection and Grouping SELECT ip length GROUP BY host AGGREGATE average SELECT timestamp GROUP BY tcp flow AGGREGATE variance Common Preprocessing Grouped Packet Vectors (GPVs) 3

33 Generalizing Selection and Grouping SELECT ip length GROUP BY host AGGREGATE average SELECT timestamp GROUP BY tcp flow AGGREGATE variance!!!!!!!!!!!!!!! "!!!!!!!!!!!!!!!!!!! "!" flowkey :( packetlengths, packettimestamps,... ) > Common Preprocessing Grouped Packet Vectors (GPVs) 33

34 Generalizing Selection and Grouping SELECT ip length GROUP BY host AGGREGATE average SELECT timestamp GROUP BY tcp flow AGGREGATE variance Regroup to any sub-key for KV op. and copy per GPV. export key = <src IP, dst IP, src port, dst port, protocol, link ID> Grouped Packet Vectors (GPVs) Common Preprocessing 34

35 *Flow Architecture Regrouping and statistics calculation. Monitoring Applications Packet feature selection and initial grouping. *Flow Cache Thin interface to grouped packet vectors. Grouped Packet Vectors *Flow Agent 35

36 Outline Introduction Design Implementation Evaluation 36

37 *Flow Implementation Proof-of-concept implementation. Monitoring Applications P4 implementation for 3. Tb/s Barefoot Tofino. *Flow Cache Grouped Packet Vectors *Flow Agent 37

38 Background: Stateful Match Action ASICs Restrictive computational model Match Action Stateful Variables Line rate processing ( packet per cycle) Packet Headers + metadata 38

39 *Flow as a Stateful Match Action Pipeline Slot ID Key Slot ID Packet Count Slot ID Packet Features Buffer Key: d Size: 4 Packet Headers + metadata H A S H 3 a b c

40 *Flow as a Stateful Match Action Pipeline Slot ID Key Slot ID Packet Count Slot ID Packet Features Buffer Key: d Size: 4 Packet Headers + metadata H A S H 3 a b c

41 *Flow as a Stateful Match Action Pipeline Untracked Flow: Replace and copy to software Slot ID Key Slot ID Packet Count Slot ID Packet Features Buffer Key: d Size: 4 Packet Headers + metadata H A S H 3 a b c

42 *Flow as a Stateful Match Action Pipeline Untracked Flow: Replace and copy to software Slot ID Key Slot ID Packet Count Slot ID Packet Features Buffer Key: a Tuples: [0, 400] Key: d Size: 4 Packet Headers + metadata H A S H 3 d b c

43 *Flow as a Stateful Match Action Pipeline Untracked Flow: Replace and copy to software Tracked Flow: Append tuple Slot ID Key Slot ID Packet Count Slot ID Packet Features Buffer Key: b Size: 8 Packet Headers + metadata H A S H 3 d b c

44 *Flow as a Stateful Match Action Pipeline Untracked Flow: Replace and copy to software Tracked Flow: Append tuple Tracked Flow, Buffer Full: rollover buffer Slot ID Key Slot ID Packet Count Slot ID Packet Features Buffer Key: c Size: 0 Packet Headers + metadata H A S H 3 d b c

45 *Flow as a Stateful Match Action Pipeline Untracked Flow: Replace and copy to software Tracked Flow: Append tuple Tracked Flow, Buffer Full: rollover buffer Slot ID Key Slot ID Packet Count Slot ID Packet Features Buffer Key: a Tuples: [9, 500, Key: c Size: 0 Packet Headers + metadata H A S H 3 d b c

46 *Flow as a Stateful Match Action Pipeline Untracked Flow: Replace and copy to software Tracked Flow: Append features Tracked Flow, Buffer Full: rollover buffer Key: a Tuples: [0, 400] Packet Headers + metadata H A S H Slot ID 3 Key a d b c Slot ID 3 Packet Count Slot ID 3 Packet Features Buffer Key: c Tuples: [9, 500, 46

47 Outline Motivation Design Implementation Evaluation 47

48 Evaluation: Concurrency and Efficiency 48

49 Evaluation: Concurrency and Efficiency *Flow Cache Stages ALUs Query Stages ALUs Connection Count Avg. Latency Flowlet Sizes Packet Counts TCP Non-Monotonic TCP Out of Sequence GPV *Flow Cache Vs statistics Compiled Queries [Marple SIGCOMM 7] 49

50 Evaluation: Concurrency and Efficiency The *Flow Cache can service any query but only requires as many hardware resources as ~ compiled query. *Flow Cache Stages ALUs 33 Query Stages ALUs Connection Count 4 0 Avg. Latency 6 Flowlet Sizes 3 Packet Counts 5 7 TCP Non-Monotonic 5 6 TCP Out of Sequence 7 4 GPV *Flow Cache Vs statistics Compiled Queries [Marple SIGCOMM 7] 50

51 Evaluation: Concurrency and Efficiency Throughput (M GPVs/s) Host Profiler Flow Classifier Congestion Debugger Classifies TCP flows based on features. Generates per-host timestamp vectors. Identifies end hosts with packets in queue during microburst. 00 X average GPV rate of 0 GbE Internet link [3] # CPU Cores 5 [3]: CAIDA 05 and 06 trace stats

52 Evaluation: Concurrency and Efficiency Throughput (M GPVs/s) Host Profiler Flow Classifier Congestion Debugger Classifies TCP flows based on features. Generates per-host timing profiles. One GPV processing server can measure One GPV processing terabit rate traffic. server can measure terabit rate traffic. Identifies end hosts with packets in queue during microburst. 00 X average GPV rate of 0 GbE Internet link [3] # CPU Cores 5 [3]: CAIDA 05 and 06 trace stats

53 Outline Motivation Design Implementation Evaluation 53

54 In The Paper On Github Match+Action Layout ASIC Resource Requirements and More Evaluation 3. Tb/s *Flow Cache Prototype for Barefoot Tofino Background 54

55 Conclusion (and Thank You for Listening!) What are the right measurement subtasks to do in hardware? *Flow Applications Statistics Calculation Concurrency Efficiency *Flow Cache Selection and Grouping 55