Networking (Containers) in Ultra- Low-Latency Environments. Avi Deitcher

Size: px

Start display at page:

Download "Networking (Containers) in Ultra- Low-Latency Environments. Avi Deitcher"

Gregory Flowers
5 years ago
Views:

1 Networking (Containers) in Ultra- Low-Latency Environments Avi Deitcher

2 Who Am I?

3 Who Am I? (not 24601)

training Some startups on the way Avid (if not very good) ice hockey player Long-Kme

4 Who Am I? Life in tech business: 10 yrs large-scale mission-crikcal IT 10+ yrs consulkng & training Some startups on the way Avid (if not very good) ice hockey player Long-Kme lover of great engineering. when used to make a real difference Atomic Inc: Generalist prackkoner Network just one element Product : engineering : operakons (not 24601)

5 A LiVle History

6 A LiVle History Summer 2015 Fintech X: Containerize us! Hint: It is harder than you think and worth it Culture/process > technology QuesKon: Networking? Answer: ScienKfic method

7 A LiVle History Summer 2015 Fall 2016 Fintech X: Containerize us! Hint: It is harder than you think and worth it Culture/process > technology QuesKon: Networking? Answer: ScienKfic method Good prackce demands: 1. Redo tests with new opkons and versions 2. Make tests available 3. Explain it all well

8 What Is Ultra-Low Latency?

9 What Is Ultra-Low Latency? every 100ms of delay costs 1% of sales [1] 1. hvp://home.blarg.net/%7eglinden/stanforddatamining ppt

10 What Is Ultra-Low Latency? every 100ms of delay costs 1% of sales [1] extra 0.5s in search page generakon Kme dropped traffic by 20% [2] 1. hvp://home.blarg.net/%7eglinden/stanforddatamining ppt 2. hvp://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html

11 What Is Ultra-Low Latency? every 100ms of delay costs 1% of sales [1] extra 0.5s in search page generakon Kme dropped traffic by 20% [2] Not. Even. Close. 1. hvp://home.blarg.net/%7eglinden/stanforddatamining ppt 2. hvp://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html

12 Ultra-Low Latency 38 messages in 7 milliseconds 1 message (avg) every 184 u-sec!

13 Two Types of Networking Direct

14 Two Types of Networking Direct Fabric+Overlay

15 maybe four Workload Awareness

16 maybe four Workload Awareness Fabric Awareness

17 Networking OpKons Direct Metal macvlan Bridge/vSwitch (no NAT) net=host SR-IOV Overlay Flannel Weave Docker Overlay Calico (IPIP) Workload Awareness Docker bridge (NAT) Fabric Awareness Calico (NaKve)

Our Tests What We Tested netperf netserver UDP & TCP round-robin Sizes: 300, 500, 1024, 2048 No orchestrakon = complete control 50000 iterakons Law of large numbers Latency (Avg, %iles), CPU

18 Our Tests What We Tested netperf netserver UDP & TCP round-robin Sizes: 300, 500, 1024, 2048 No orchestrakon = complete control iterakons Law of large numbers Latency (Avg, %iles), CPU How We Tested.net Because it had to be metal Wicked smart team Complete test run Network changes Hardware variakons, errors DifferenQals, not absolutes hvps://github.com/deitch/network-tests

19 Local vs. Remote

23 Local Networking Summary SR-IOV horrible latency but great CPU Hold that thought net=host on par with metal macvlan closest virtualized to metal Rest in same range: Latency: 5-10 u-sec overhead CPU: negligible difference Calico (IPIP & nakve) & Docker overlay slightly more performant (margin of error?) Watch out for very large TCP packets

27 Remote Networking Summary Weave (sleeve) adds latency and CPU Reason for fast datapath Again, macvlan best virtualized All the rest: Latency: within 50 u-sec of each other, except SR- IOV with very large TCP packets CPU: similar, but keep an eye on Flannel (UDP)

28 About that SR-IOV Type 1: Intel I350 1Gbps Type 3: Mellanox MT27500 ConnectX-3 10Gbps

29 SR-IOV SR-IOV does not automaqcally mean bewer Switch in network card Trades host CPU for card processor Quality varies drama3cally Even Mellanox far worse locally My 2 : SR-IOV falls further behind due to: Speed of iterakon Open-source Sosware + CPU

30 What else could we do? Ø Other hardware types Ø Other network fabrics Ø Other network overlay versions (we have the data ) Ø Docker macvlan network driver Ø ipvlan Ø IPv6 Ø Kernel and network stack tuning Ø Distant networks Ø Other traffic paverns (mulkcast vs unicast) Ø Other host-to-host encrypkon Ø Other kernel versions Ø Other OSes (Illumos-based?) Ø A whole lot more

31 Headaches (and Thanks) Headaches Weave SYN-(nothing) etcd is touchy Packet L3 network is powerful but unique Macvlan, weave, flannel: all required pings for mac Sexng up bridge w/o NAT, Calico, macvlan was different SR-IOV is complicated and flaky, especially Mellanox netperf with UDP packets can get stuck (Calico-ipip) And a whole lot more (ask me offline) And thanks: Bryan Boreham, Adam Harrison at weave.works Zac Smith, Adam, Aaron, Andy, Lucas, everyone at Packet

32 Conclusions SR-IOV: most of the Kme, just not worth it Performance: Metal (+ net=host): always performs best Direct network++: macvlan is your friend Others: Roughly similar, careful of Weave (sleeve) What s your use case? ULL: Metal/net=host > macvlan > calico > overlay Everything else: Focus on your architecture and skills Pick intelligently: easier, not simple

33 Conclusions SR-IOV: most of the Kme, just not worth it Performance: Metal (+ net=host): always performs best Direct network++: macvlan is your friend Others: Roughly similar, careful of Weave (sleeve) What s your use case? ULL: Metal/net=host > macvlan > calico > overlay Everything else: Focus on your architecture and skills Pick intelligently: easier, not simple

34 @avideitcher QuesKons and help:

An Analysis and Empirical Study of Container Networks

An Analysis and Empirical Study of Container Networks Kun Suo *, Yong Zhao *, Wei Chen, Jia Rao * University of Texas at Arlington *, University of Colorado, Colorado Springs INFOCOM 2018@Hawaii, USA 1