Hashing on broken assumptions

Size: px

Start display at page:

Download "Hashing on broken assumptions"

Lindsay Young
6 years ago
Views:

1 Hashing on broken assumptions Lorenzo Saino Fastly Name of Presentation

2 Problem: Spreading traffic across multiple links, paths, hosts Solutions: Link Aggregation Equal Cost Multipath (ECMP)

3 Link aggregation Combine multiple physical links between network devices into one logical link physical links logical link switch switch

4 Equal Cost Multipath (ECMP) Balance traffic across paths Balance traffic across hosts switch switch switch switch switch host host host

5 Requirements Load balance Traffic must be uniformly spread across next- hops Stateless- but- sticky path pinning All packets of a flow must take the same path

6 Load imbalance Load imbalance reduces system capacity

7 Load imbalance Load imbalance reduces system capacity Perfect load balance

8 Load imbalance Load imbalance reduces system capacity All resources fully utilized

9 Load imbalance Load imbalance reduces system capacity Load imbalance

10 Load imbalance Load imbalance reduces system capacity Unused capacity Cannot take any additional load

11 Quantifying impact of load imbalance L max load of most loaded resource L avg average load U max 2 (0, 1] max attainable utilization Load imbalance: Max attainable utilization: L max L avg =[1, +1) U max = Lmax 1 = L avg L avg L max

12 Quantifying impact of load imbalance Umax L max /L avg

13 Quantifying impact of load imbalance Umax Perfect balance Full utilization L max /L avg

14 Quantifying impact of load imbalance Umax 0.6 X Most loaded resource 1.5x average 33.3% reduction of capacity L max /L avg

15 What happens without path pinning? Same endpoints, different paths: Out- of- order packets Frequent drops of TCP congestion window (CWND) Poor throughput performance Different endpoints: TCP resets

16 TCP resets SYN host host router SYN/ACK ACK RST host

17 Requirements: Load balance Path pinning Solution: Flow- level hashing

18 Flow-level hashing read five tuple hash function packet src IP addr dst IP addr protocol src port dst port next-hop

19 Assumptions Load balance Hashing uniformly spread traffic across next- hops Path pinning Hashing pins packets of a flow to the same path

20 Do these assumptions hold?

21 Assumptions Load balance Hashing uniformly spread traffic across next- hops Path pinning Hashing pins packets of a flow to the same path

22 Hashing quality Two switch models: Switch A Switch B 2^16 five- tuple combinations switch nexthops

23 Switch A Perfect hashing Measured L/Lavg Nexthop rank

24 Switch B 2.0 Measured 1.5 Perfect hashing L/Lavg x 0.5 6x Nexthop index

25 Switch B Vendor claims supporting an arbitrary number of next- hops [1, 256]

26 Switch B Only a subset of next- hops are actually supported

27 Switch B Only a subset of next- hops are actually supported

28 Switch B Only a subset of next- hops are actually supported X X X X X X next-hops don t get any traffic

29 Assumptions Load balance Hashing uniformly spread traffic across next- hops Path pinning Hashing pins packets of a flow to the same path

30 Hashing on IPv4 TOS field Version IHL Type of Service Total Length Identification Flags Fragment Offset Time to Live Protocol Header Checksum Source Address Destination Address Options Padding

31 Hashing on IPv4 TOS field Version IHL Type of Service Total Length Identification Flags Fragment Offset Time to Live Protocol Header Checksum Source Address Destination Address Options Padding

32 Hashing on IPv4 TOS field Version IHL Type of Service Total Length Identification Flags Fragment Offset Time to Live Protocol Header Checksum Source Address Destination Address Options Padding

33 Hashing on IPv4 TOS field RFC Requirements for IP Version 4 Routers explicitly permits to involve the second- to- last bit of the TOS/DS octet in routing decisions RFC Definition of the Differentiated Services Field deprecates the IPv4 Type of Service field redefines it as the Differentiated Services field RFC The Addition of Explicit Congestion Notification (ECN) to IP reserves the last two bits of the DS octet for ECN

34 Hashing on IPv4 TOS field host router host TCP handshake: Hosts negotiate ECN support ECN- capable bits unset host Flow data: ECN- capable bits set Scenario Hosts are ECN capable Router uses IPv4 TOS for hash computation (RFC 1812) TCP handshake flow data

35 IPv6 flow label rewrite host x, x!= 0 y x, x!= 0 middlebox if flow_label!= 0: flow_label = rand() z switch uses IPv6 flow label for hash computation y z host host forbidden by RFC 6437 allowed by RFC 6437

36 SYN proxies SYN proxy switch host host switch host Switches: use ingress interface for hash computation, or use different hash function seeds TCP handshake flow data

37 Conclusions Load balancing There are devices that do not hash traffic uniformly Path pinning Hashing on fields other than five tuples breaks ECMP Ingress port IPv4 TOS IPv6 flow label

38 Recommendations Operators: Ensure that your network devices hash flows uniformly or that could cost you money Disable additional inputs if you do not need extra entropy Vendors: Disable hashing inputs other than five- tuple by default Make hash input fields configurable Make hash seed configurable

39 FIN

Packet Header Formats

A P P E N D I X C Packet Header Formats S nort rules use the protocol type field to distinguish among different protocols. Different header parts in packets are used to determine the type of protocol used