Low-latency Network Monitoring via Oversubscribed Port Mirroring

Transcription

1 Low-latency Network Monitoring via Oversubscribed Port Mirroring Jeff Rasley, Brent tephens,! Colin Dixon, Eric Rozner,! Wes Felter, Kanak Agarwal,! John Carter, Rodrigo Fonseca

2 elf-tuning Networks Control Loop Examples!! Measurement Traffic Engineering! Failure Detection How fast can we do this? Control Decision

3 elf-tuning Networks 100 ms 1 sec+ Control Loop Examples!! Measurement Traffic Engineering! Failure Detection How fast can we do this? Control Decision

4 elf-tuning Networks 100 ms 1 sec+ Control Loop Examples!! Measurement Traffic Engineering! Failure Detection How fast can we do this? ~100 µs Control Decision

5 elf-tuning Networks 100 ms 1 sec+ Control Loop Examples!! Measurement Traffic Engineering! Failure Detection How fast can we do this? ~10 ms Control ~100 µs Decision

6 elf-tuning Networks 100 ms 1 sec+ Measurement Control Loop Examples!! Traffic Engineering! Failure Detection How fast can we do this? ~10 ms ~100 µs Control Decision

7 Motivation for Faster Control Loops This gets much! worse as you go from! 1 Gbps > 10 Gbps % of flows shorter than x Background TCP flows, Microsoft data center! DCTCP, Alizadeh et al. igcomm Flow Duration at 1Gbps (ms)

8 Motivation for Faster Control Loops This gets much! worse as you go from! 1 Gbps > 10 Gbps % of flows shorter than x Background TCP flows, Microsoft data center! DCTCP, Alizadeh et al. igcomm Flow Duration at 1Gbps (ms)

9 Motivation for Faster Control Loops 100 This gets much! worse as you go from! 1 Gbps > 10 Gbps % of flows shorter than x At 100 ms, 90% are over! before you see them! Flow Duration at 1Gbps (ms) Background TCP flows, Microsoft data center! DCTCP, Alizadeh et al. igcomm 10

10 Why is Measurement low? Traditionally, this didn t need to be fast! Control plane CPUs are typically slow! ampling or port counter polling! Is this likely to get better? Maybe! Faster control plane CPUs could help, still a big gap between CPUs and AICs

11 Our olution: Abuse Port Mirroring Modern switches support port-mirroring! Copies all packets e.g. going out a port to a designated mirror port! Production! Traffic We abuse port mirroring to radically increase the number of samples/sec we get from a switch! We mirror all ports to a single mirror port! Oversubscription approximates sampling (in the data plane) at much higher rates witch

12 Our olution: Abuse Port Mirroring Modern switches support port-mirroring! Copies all packets e.g. going out a port to a designated mirror port! Production! Traffic We abuse port mirroring to radically increase the number of samples/sec we get from a switch! We mirror all ports to a single mirror port! Mirror Port Oversubscription approximates sampling (in the data plane) at much higher rates witch

13 Our olution: Abuse Port Mirroring Modern switches support port-mirroring! Copies all packets e.g. going out a port to a designated mirror port! Production! Traffic We abuse port mirroring to radically increase the number of samples/sec we get from a switch! We mirror all ports to a single mirror port! Mirror Port Oversubscription approximates sampling (in the data plane) at much higher rates witch

14 Architecture A set of collectors receives a stream of samples from mirror ports! Controller Netmap or Intel DPDK for fast processing! ) Reconstruct flow information across all flows in the network! e.g. flow throughput and port congestion! Collector(s) Collectors can interact with an DN controller to implement various applications! H H H H e.g. traffic engineering

15 What Can Go Wrong? Lose input/output port information from packets! Recover meta-data about packets by sharing topology Controller state from the controller! ) When mirror port fills, its drop policy is unknown thus making it hard to calculate throughout! Rate estimation via TCP sequence numbers! Oversubscribed port may occupy switch buffer space, Collector(s) taking away from production traffic! Indeed, buffers were reduced. Latency of production traffic decreased. Negligible increase in packet loss (~0.1%). H H H H

16 What Can Go Wrong? Lose input/output port information from packets! Recover meta-data about packets by sharing topology state from the controller! When mirror port fills, its drop policy is unknown thus making it hard to calculate throughout! Rate estimation via TCP sequence numbers! Oversubscribed port may occupy switch buffer space,! What are you crazy?!! By oversubscribing a mirror port there s a possibility of degrading production traffic.! Cisco Controller Collector(s) ) taking away from production traffic! Indeed, buffers were reduced. Latency of production traffic decreased. Negligible increase in packet loss (~0.1%). H H H H

17 What Can Go Wrong? Lose input/output port information from packets! Recover meta-data about packets by sharing topology Controller state from the controller! ) When mirror port fills, its drop policy is unknown thus making it hard to calculate throughout! Rate estimation via TCP sequence numbers! Oversubscribed port may occupy switch buffer space, Collector(s) taking away from production traffic! Indeed, buffers were reduced. Latency of production traffic decreased. Negligible increase in packet loss (~0.1%). H H H H

18 Results: ample Latency (high congestion) t_2 witch H t_1 Collector H H ender Receiver Latency = t_2 - t_1 Low Congestion! ample Latency:! µs CDF Measurement Latency (ms) IBM G8264 (10Gb) Pronto 3290 (1Gb)

19 What Can You Do With This? TE! tride(8) 100 MiB Workload! CDF of Flow Throughput 16 hosts! etup! k=4 Fat Tree ECMP Optimal tride(8)! Permutation of hosts,! such that each flow! traverses the core IBM G8264 (10 Gb) Hardware

20 What Can You Do With This? TE! tride(8) 100 MiB Workload! CDF of Flow Throughput etup! 16 hosts! k=4 Fat Tree tride(8)! Permutation of hosts,! such that each flow! traverses the core ECMP Optimal Poll 100ms IBM G8264 (10 Gb) Hardware

21 What Can You Do With This? TE! tride(8) 100 MiB Workload! CDF of Flow Throughput etup! 16 hosts! k=4 Fat Tree tride(8)! Permutation of hosts,! such that each flow! traverses the core ECMP Optimal Poll 100ms Our TE IBM G8264 (10 Gb) Hardware

22 Conclusion Using oversubscribed port mirroring we get ~1 million samples / sec.! We get sampling latencies between 100 µs 6ms on real hardware, today.! We improve this by 3 4 orders of magnitude, the state of the art is 100 ms 1 sec+

23 Questions? Thank you!

24 Production Traffic Throughput Ave Per-Flow Tput (Gbps) Mirror No Mirror Congested Output Ports

25 Production Traffic Latency 3 Average Latency (ms) Mirror No Mirror Congested Output Ports

26 Production Traffic Packet Loss 0.16% Percent of Pkts Lost 0.14% 0.12% 0.1% 0.08% 0.06% 0.04% 0.02% 0% Mirror No Mirror Congested Output Ports

27 Flow Rate Estimation Throughput (Gbps) Gbps 6 Gbps Time (s)

28 Rate Estimation Error