Toward an ebpf-based clone of iptables

Size: px

Start display at page:

Download "Toward an ebpf-based clone of iptables"

Blaise Shields
5 years ago
Views:

1 Toward an ebpf-based clone of iptables Matteo Bertrone, Sebastiano Miano, Jianwen Pi, Fulvio Risso, Massimo Tumolo Netdev 0x12, Montréal (Canada), July 12th, 2018

Objective Started in Nov 2017, with a (applied) research mindset Objective: create a complex network software with ebpf, to assess strengths and weaknesses of the above technology Earlier and, in

2 Objective Started in Nov 2017, with a (applied) research mindset Objective: create a complex network software with ebpf, to assess strengths and weaknesses of the above technology Earlier and, in some sense, orthogonal to bpfilter Create a (partial) clone of iptables, exploiting the ebpf technology Understand the challenges Provide hints about the feasibility of this project Characterize performance of the prototype Possibly use vanilla Linux kernels (no extensions) Identify possible future improvements/directions in ebpf/linux kernel 2

3 TURIN POLYTECHNIC Part I

4 Key objective #1: Preserving iptables semantic Local processes iptables netfilter hooks PREROUTING INPUT FORWARD OUTPUT POSTROUTING TC hooks FILTER Routing Decision XDP hooks NAT ebpf code Routing Decision FILTER FILTER ebpf code NAT netfilter Routing Decision NAT netdev (e.g., eth0) 4 netdev (e.g., eth1)

5 Key objective #1: Preserving iptables semantic We need to support existing services (e.g., security orchestrators) that are based on iptables, which is a different approach compared to bpfilter Two (TC_INGRESS/XDP_INGRESS and TC_EGRESS) hooks, which must emulate three chains No hooks available in ebpf to easily intercept locally terminated/generated traffic The ebpf code has to process only the packets that would hit each specific chain (INPUT, FORWARD, OUTPUT) We have to guess very early which chain will be hit XDP alone is not enough (no support for egress traffic) ebpf code PREROUTING INPUT FORWARD OUTPUT POSTROUTING iptables netfilter hooks TC hooks XDP hooks FILTER Routing Decision Local processes FILTER Routing Decision NAT FILTER ebpf code NAT netfilter Routing Decision NAT netdev (e.g., eth0) 5 netdev (e.g., eth1)

6 bpf-iptables filtering architecture To Linux Stack From Linux Stack TC/XDP Ingress hook TC Egress hook Netfilter Netfilter Ingress Chain Selector [local dst] INGRESS CHAIN Egress Chain Selector [local src] OUTPUT CHAIN FORWARD CHAIN [remote src] From netdev (e.g., eth0) Selects the right chain based on the IP dst/src address in the incoming packet (path prediction). Selection logic is dynamically configured with all IP addresses visible from the root namespace (intercepts NETLINK messages). 6 To netdev (e.g., eth1)

7 Key objective #2: Fast matching algorithm iptables uses a linear search Constraints for the selection of the algorithm: Feasible with available ebpf data structures (maps) Possibility to rearrange the code in multiple ebpf programs to overcome the limitation of the maximum number of instructions per program Chosen Linear Bit Vector Search [1] [1] T. Lakshman and D.Stidialis. High speed policy-based packet forwarding using efficient multi-dimensional range matching. In Proc. ACM Sigcomm 98, Sept

8 Linear Bit Vector Search: Overview Rule #1: iptables A INPUT Rule #2: iptables A INPUT Rule #3: iptables A INPUT Rule #4: iptables A INPUT Rule #5: iptables A INPUT d /8 p tcp --dport 80 j ACCEPT --dport 80 j ACCEPT p udp --dport 53 j ACCEPT p tcp --dport 53 j ACCEPT j ACCEPT Input packet: ip.dst= ip.proto= TCP tcp.dport= Values Dest. IP Protocol Dest. port Matched rules 0/ / Value Matched rules * TCP UDP Value Matched rules * & & = Rule #1 8

FILTERING (INPUT/OUTPUT/FORWARD) CHAIN percpu_array shared across the entire program chain Packet header offsets Each ebpf program can exploit a different matching algorithm (e.g., exact match, longest prefix match, etc).

9 tail call tail call Linear Bit Vector Search: implementation LBVS is implemented with a chain of ebpf programs, connected through tail calls. Each ebpf program is injected only if there are rules operating on that field. FILTERING (INPUT/OUTPUT/FORWARD) CHAIN percpu_array shared across the entire program chain Packet header offsets Each ebpf program can exploit a different matching algorithm (e.g., exact match, longest prefix match, etc). ebpf program #1 ebpf program #2 ebpf program #3 ebpf program #N Map keeping the action for each rule Headers parsing Header parsing is done once and results are kept in a shared map for performance reasons. ip.dst lookup ip1 bitv1 ip2 bitv2 ip3 bitv3 ip.proto lookup Bitwise AND bitvectors * bitv1 udp bitv2 tcp bitv3 Search first matching rule Update counters ACTION (drop / accept) rule1 act1 rule2 act2 rule3 act3 rule1 cnt1 rule2 cnt2 Packet From ebpf hook The loop required to perform the bitwise-and is unrolled; currently supported ~8K rules ( bit words). Bitvector with temporary result percpu_array shared across the entire program chain [Packet] To ebpf hook 9

10 Implementation details Currently supported fields: IP src/dst: Longest Prefix Match (LPM TRIE MAPS) E.g. s /8 d /32 L4 protocol: Exact Match (HASH MAPS) E.g. proto tcp; proto 0x02; proto icmp TCP/UDP ports src/dst: Exact match (HASH MAPS) E.g. sport 80 dport 8080 TCP flags: Set, NotSet and Wildcard (ARRAY MAP, with all combinations) E.g. tcp-flags SYN,ACK SYN Conntrack state (more later) More fields can be added (e.g., netdev where traffic comes from/goes to) Possible future work: dynamically select the best algorithm per each field 10

11 Linear Bit Vector Search: Comments Good Good overall performance, particularly when an high number of rules is involved Feasible with current vanilla ebpf Bad Rule update time could be critical, as it may require second(s) Still linear in the number of rules (timed the number of fields), although executed with x64 parallelism Future steps Among the many options (e.g., HiCuts, Efficuts, etc.) Tuple Merge[1] could be an interesting algorithm to look at Fast online rule update time Should be doable with ebpf In our opinion this is not the most critical point right now. [1] James Daly and Eric Torng. TupleMerge: Building Online Packet Classifiers by Omitting Bits. Proceedings of the 26th International Conference on Computer Communication and Networks (ICCCN). IEEE,

12 Key objective #3: Connection tracking Enable rules operating on connection state E.g., -A INPUT m conntrack ctstate=established j ACCEPT E.g., -A INPUT -s /16 m conntrack ctstate=new j ACCEPT Prototypal ebpf-based implementation, with basic support for TCP/UDP/ICMP 12

13 Filtering architecture with conntrack TC/XDP Ingress hook Netfilter To Linux Stack From Linux Stack Netfilter TC Egress hook Connection tracking Ingress Chain Selector [local dst] Update session Label Packet INGRESS CHAIN Store session Update session Label Packet FORWARD CHAIN Store session Egress Chain Selector [local src] Update session Label Packet OUTPUT CHAIN Store session [remote src] From netdev (e.g., eth0) 13 To netdev (e.g., eth1)

14 Pro and con Limitations No support for the massive validation checks (e.g. on TCP window) performed by Linux conntrack, as well as IP de-fragmentation, which could not be straightforward in ebpf No support for (application-layer) RELATED connections RELATED : the packet belongs to a new connection, but is associated to an existing connection (e.g., FTP data transfer, a VoIP RTP stream) Access to L7 fields and handle upper layer protocols with ebpf could not be doable at the time maing Instead, ICMP errors are recognized and handled Advantages Available also when the ebpf program is attached to XDP hooks 14

$Key objective #4: Preserving iptables syntax iptables A INPUT p tcp \ --dport 53 j ACCEPT$ bpf-iptables (almost vanilla) libiptc (modified) Rule interceptor and translator shell script

bpf-iptables (almost vanilla) libiptc (modified) Rule interceptor and translator shell script

15 Key objective #4: Preserving iptables syntax iptables A INPUT p tcp \ --dport 53 j ACCEPT bpf-iptables A INPUT p tcp \ --dport 53 j ACCEPT iptables (vanilla) libiptc (vanilla) bpf-iptables (almost vanilla) libiptc (modified) Rule interceptor and translator shell script netlink netlink ebpf system calls bpf-iptablesd (REST-based daemon) Kernel (netfilter) Kernel (netfilter) Kernel (ebpf) 15

16 iptables vs. bpf-iptables Allow both executables to coexist on the same machine User can choose the one he likes based on a different executable name, all supporting the same command line bpf-iptables supports a subset of the iptables commands Examples of unsupported cases: Negation: -proto tcp sport!80 Range: -proto tcp dport An unsupported command triggers an error on the command line Probably does not make sense to support all the features of iptables 16

17 PRELIMINARY PERFORMANCE EVALUATION Part II

18 UDP throughput (Mpps) Throughput (packets per second) 1,6 1,4 iptables bpf-iptables 1,2 1 0,8 0,6 0,4 Testing conditions Two Intel i servers, 32GB RAM Two direct 10GbE links (A B A) Server 1: pktgen-dpdk traffic generator (UDP traffic, 64B Ethernet frames) Traffic measured when losses < 1% Server 2: iptables + IP forwarding enabled 0, Number of filtering rules 18

19 # connections per second # connections per second Throughput (TCP connections per second) Page size: 87bytes baseline iptables Testing conditions Number of concurrent connections (apache benchmark) Page size: 11Kbytes bpf-iptables baseline iptables bpf-iptables Two Intel i servers, 32GB RAM One bidirectional direct 10GbE link (A B) 1000 filtering rules active Server 1: traffic generator (ab, different degrees of parallelism) Server 2: iptables + apache web server (iptables forced to run on a single CPU core by configuring interrupt affinity) Number of concurrent connections (apache benchmark) 19

20 Latency (ms) Processing latency (ms) 1,6 1,4 1,2 1,0 0,8 0,6 0,4 0,2 0,0 iptables bpf-iptables Number of filtering rules Testing conditions Two Intel i servers, 32GB RAM Two direct 10GbE links (A B A) Server 1: traffic generator (ping, one ICMP echo req. per second) Server 2: iptables + IP forwarding enabled 20

21 FUTURE WORK Part III

22 #1. Integration with NAT iptables netfilter hooks iptables nat hooks Local processes PREROUTING INPUT FORWARD OUTPUT POSTROUTING TC hooks FILTER Routing Decision XDP hooks NAT ebpf code Routing Decision FILTER FILTER ebpf code NAT netfilter Routing Decision NAT netdev (e.g., eth0) 22 netdev (e.g., eth1)

23 #1. Integration with NAT Netfilter NAT is no longer semantically equivalent when filtering is done with ebpf E.g., in INPUT packets would cross filter and then NAT, while in vanilla Linux they do the opposite We do not have any hook to get packets after the netfilter NAT Hence, we have to implement both NAT and filtering in ebpf Integration for PREROUTING and POSTROUTING is simple, just put a NAT module in front of the ebpf filtering chain OUTPUT NAT is rarely used (as far as we now), so we re not investigating that part Should we really need to add NAT as well? It depends, but even simple apps (e.g., Docker) make use of NAT 23

24 #2. Integration with bpfilter bpfilter has two main objectives Provides a way to intercept iptables command when being injected in the kernel Defines a mechanism that can evaluate some policy rules as early as possible bpfilter does not have a way to preserve the semantic of iptables rules Our work is largely orthogonal to bpfilter We can easily get rid of our code that intercepts policy rules and move to the mechanism provided by bpfilter Possible nice direction to explore: moving the evaluation of some rules as early as possible (while preserving the iptables semantic) 24

25 #3. Improving time for updating rules Rather high compared to other solutions (~1s) Two reasons We should optimize our control plane algorithms ebpf code injection time is not that small, particularly when an high number of programs and maps need to de changed 25

26 LESSONS LEARNED (OR IDEAS FOR DISCUSSION) Part IV

27 #1. ebpf vs. Netfilter: Cooperation or Competition? Local processes iptables netfilter hooks TC hooks XDP hooks Future hooks PREROUTING INPUT FORWARD OUTPUT POSTROUTING FILTER Routing Decision NAT ebpf code Routing Decision FILTER FILTER ebpf code NAT netfilter Routing Decision NAT netdev (e.g., eth0) 27 netdev (e.g., eth1)

28 #1. ebpf vs. Netfilter: Cooperation or Competition? Difficult to enhance netfilter by adding individual components based on ebpf E.g., cannot replace filtering while using the netfilter NAT Adding more hooks around would simplify the integration and coexistence of both technologies From competition Cannot have both technologies at the same time... to cooperation Use the best of both worlds Local processes PREROUTING INPUT FORWARD OUTPUT POSTROUTING E.g., an ebpf hook to intercept local traffic would remove the necessity of the Chain Selector block ebpf code iptables netfilter hooks TC hooks XDP hooks FILTER Routing Decision FILTER Routing Decision NAT FILTER ebpf code NAT netfilter Routing Decision NAT netdev (e.g., eth0) 28 netdev (e.g., eth1)

29 #2. A way for the perfect matching algorithm Hard to find a one size fits all algorithm We chose LBVS because of its run-time performance Other people need an algorithm supporting high frequency rule updates And more ebpf can enable different people to implement the matching algorithm that is the best fits for their objectives Difficult to achieve with other technologies (e.g., current iptables) 29

30 #3. Replacing or offloading iptables? Our work so far has been in the direction of replacing iptables A (almost) fully compatible clone People in our group are even working on NAT Is this the right way to go? What about offloading some iptables rules, e.g., in a dedicated BPF program executed in XDP? E.g., rules that filter traffic coming from a huge set of malicious hosts Preserving the iptables semantic is definitely a must 30

31 #4. Need more work on Connection Tracking We implemented our own minimal conntrack Other people did the same, e.g., Cilium folks But no advanced features, e.g., IP reassembly, related connections, etc, which are currently not feasible in ebpf Is it the right choice to have conntrack as ebpf? Difficult to have all the features of the existing (native) connection tracking We probably need to investigate more 31

32 CONCLUDING REMARKS Part V

33 Conclusions 100% compatible version of iptables does not look a good idea What about ip6tables, arptables and ebtables? A reduced (and faster) version looks more appealing Enables best of both words (command line compatibility, features, performance) Performance characterization Excellent results when an high number of rules are configured Below iptables with a few rules (or none) Rule update not so fast (~ 10 seconds for a database of 1000 rules) Under investigation if new ebpf hooks/helpers could improve the performance For sure, it may enable a better cooperation between the ebpf world and netfilter 33

34 34

A 10 years journey in Linux firewalling Pass the Salt, summer 2018 Lille, France Pablo Neira Ayuso

A 10 years journey in Linux firewalling Pass the Salt, summer 2018 Lille, France Pablo Neira Ayuso What is Netfilter? Not just iptables Image from Wikipedia (J. Engelhardt, 2018)