Comparing Open vswitch (OpenFlow) and P4 Dataplanes for Agilio SmartNICs

Transcription

1 Comparing Open vswitch (OpenFlow) and P4 Dataplanes for Agilio SmartNICs Johann Tönsing May 24, NETRONOME

2 Agenda Contributions of OpenFlow, Open vswitch and P4 OpenFlow features missing in P4, and P4 features missing in OpenFlow Architecture of Open vswitch accelerated by a SmartNIC Architecture of a P4 SmartNIC datapath Performance considerations Next steps: proposals towards convergence and alignment 206 NETRONOME 2

3 Contributions of OpenFlow Architectural Principle: - Logically Centralized, Decoupled Control Model: - Defined by specification: dataplane protocols, pipeline of match-action tables (Soon: dataplane protocols no longer in main specification, dynamic dataplane protocol definition being considered) Interface: - Protocol (handcrafted, binary C struct style) 206 NETRONOME 3

4 Contributions of OpenFlow Architectural Principle: - Logically Centralized, Decoupled Control Well known SDN (and older) principle - not unique to OpenFlow No plans to abandon perhaps add selective delegation, hierarchies Model: - Defined by specification: dataplane protocols, pipeline of match-action tables (Soon: dataplane protocols no longer in main specification, dynamic dataplane protocol definition being considered) Interface: - Protocol (handcrafted, binary C struct style) 206 NETRONOME 3

5 Contributions of OpenFlow Architectural Principle: - Logically Centralized, Decoupled Control Well known SDN (and older) principle - not unique to OpenFlow No plans to abandon perhaps add selective delegation, hierarchies Model: - Defined by specification: dataplane protocols, pipeline of match-action tables (Soon: dataplane protocols no longer in main specification, dynamic dataplane protocol definition being considered) Too general: Table Typing Patterns define what is required by application, supported by implementation Too limited: annoying to have to create specification revisions to add protocols, behavior Interface: - Protocol (handcrafted, binary C struct style) 206 NETRONOME 3

6 Contributions of OpenFlow Architectural Principle: - Logically Centralized, Decoupled Control Well known SDN (and older) principle - not unique to OpenFlow No plans to abandon perhaps add selective delegation, hierarchies Model: - Defined by specification: dataplane protocols, pipeline of match-action tables (Soon: dataplane protocols no longer in main specification, dynamic dataplane protocol definition being considered) Too general: Table Typing Patterns define what is required by application, supported by implementation Too limited: annoying to have to create specification revisions to add protocols, behavior Software usually built using libraries with callable APIs Interface: - Protocol (handcrafted, binary C struct style) Protocol can be a language independent interface (but prefer automated marshalling, e.g. Thrift) Need faster interface mechanism (e.g. callable API) for datapath extension code 206 NETRONOME 3

7 Contributions of P4 Architectural Principle: - Language enabling datapath programming - Deployable with central or distributed control, as switch or part of switch Model: - Datapath protocols and behavior are defined by a program (not enshrined in a standard) Interface: - Evolving - currently callable API is generated 206 NETRONOME 4

8 Contributions of P4 Architectural Principle: - Language enabling datapath programming - Deployable with central or distributed control, as switch or part of switch Can be used with logically centralized decoupled control Interoperability may be limited if embedded in proprietary datapaths Model: - Datapath protocols and behavior are defined by a program (not enshrined in a standard) Interface: - Evolving - currently callable API is generated 206 NETRONOME 4

9 Contributions of P4 Architectural Principle: - Language enabling datapath programming - Deployable with central or distributed control, as switch or part of switch Can be used with logically centralized decoupled control Interoperability may be limited if embedded in proprietary datapaths Model: - Datapath protocols and behavior are defined by a program (not enshrined in a standard) Useful to enable protocols and behavior to rapidly evolve: killer app = flexibility Useful as a unambiguous model of programmable and fixed datapath behavior Interface: - Evolving - currently callable API is generated 206 NETRONOME 4

10 Contributions of P4 Architectural Principle: - Language enabling datapath programming - Deployable with central or distributed control, as switch or part of switch Can be used with logically centralized decoupled control Interoperability may be limited if embedded in proprietary datapaths Model: - Datapath protocols and behavior are defined by a program (not enshrined in a standard) Useful to enable protocols and behavior to rapidly evolve: killer app = flexibility Useful as a unambiguous model of programmable and fixed datapath behavior Convenient for initial experimentation Interface: - Evolving - currently callable API is generated However complicates matters for controllers - need to link with different library for each version of each program Consider generic interface instead 206 NETRONOME 4

11 Contributions of Open vswitch Evaluated Widely deployed virtual switch - Integrated with many controllers, cloud management systems (OpenStack, ODL, ONOS ) Accepted into Linux kernel - Uses Linux kernel facilities where appropriate, e.g. for connection tracking or QoS Developed by community as open source project - Frequently leading OpenFlow standards 206 NETRONOME 5

12 Contributions of Open vswitch Evaluated Widely deployed virtual switch - Integrated with many controllers, cloud management systems (OpenStack, ODL, ONOS ) Great! Accepted into Linux kernel - Uses Linux kernel facilities where appropriate, e.g. for connection tracking or QoS Developed by community as open source project - Frequently leading OpenFlow standards 206 NETRONOME 5

13 Contributions of Open vswitch Evaluated Widely deployed virtual switch - Integrated with many controllers, cloud management systems (OpenStack, ODL, ONOS ) Great! Accepted into Linux kernel - Uses Linux kernel facilities where appropriate, e.g. for connection tracking or QoS Using Linux specific features is problematic for interoperability - e.g. with hardware switches, or DPDK version of OVS Developed by community as open source project - Frequently leading OpenFlow standards 206 NETRONOME 5

14 Contributions of Open vswitch Evaluated Widely deployed virtual switch - Integrated with many controllers, cloud management systems (OpenStack, ODL, ONOS ) Great! Accepted into Linux kernel - Uses Linux kernel facilities where appropriate, e.g. for connection tracking or QoS Using Linux specific features is problematic for interoperability - e.g. with hardware switches, or DPDK version of OVS Developed by community as open source project - Frequently leading OpenFlow standards Great! 206 NETRONOME 5

15 OpenFlow Features not in P4 At any time, the SDN controller can send a flow add or flow modify message: creating a new table or a new control flow path, matching an additional field or invoking an additional action, etc. In P4, actions, tables, and control flow options are pre-declared in the program Solutions to retain dynamic functionality with OpenFlow compatibility: JIT compile program variant with a capability once it is first required [analogous to PyPy] Cache these variants - remember last used (issue: when to garbage collect ) Solutions to avoid need for dynamic functionality Model (e.g. TTP) is used to declare required functionality (e.g. action list patterns) at start time No changes needed to OpenFlow otehrwise (device returns error if functionality unsupported) 206 NETRONOME 6

16 P4 Features not in OpenFlow Too many to list here! Nice! Flexibility can actually be a problem for controllers, cloud management systems: difficult to generically support all datapath programs / program versions Solution: plugin into controller to go along with datapath program Run time interface becomes distributed system infrastructure (RPC) Solution: implement functionality with a well defined (less rapidly evolving) interface Run-time interface generated from program is problematic - too many libraries / modules for controller writers to consume Consider generic interface also: APIs to discover and access objects Generated interface remains useful for fast datapath extension software ( see demo ) 206 NETRONOME 7

17 OVS Acceleration using Agilio SmartNIC (Nova, Neutron) x86 Userspace Open vswitch Subsystem OVS CLI Callable API OVS Agent OpenFlow x86 Kernel Configuration via controller, CLI, or Callable API OVS Kernel DP Match/Act PCIe Agilio Intelligent Server Adapter (NIC) OVS Kernel DP Match/Act 206 NETRONOME 8

18 OVS Acceleration using Agilio SmartNIC (Nova, Neutron) x86 Userspace Open vswitch Subsystem OVS CLI Callable API OVS Agent OpenFlow x86 Kernel Configuration via controller, CLI, or Callable API OVS Kernel DP Match/Act PCIe Agilio Intelligent Server Adapter (NIC) Open vswitch Datapath OVS Kernel DP Match/Act (e.g. Entunnel, Deliver to VM, Send to Port) 206 NETRONOME 8

19 OVS Acceleration using Agilio SmartNIC (Nova, Neutron) x86 Userspace Open vswitch Subsystem OVS CLI Callable API OVS Agent OpenFlow x86 Kernel Configuration via controller, CLI, or Callable API OVS Kernel DP Match/Act PCIe Agilio Intelligent Server Adapter (NIC) Miss Open vswitch Datapath OVS Kernel DP Match/Act (e.g. Entunnel, Deliver to VM, Send to Port) 206 NETRONOME 8

20 OVS Acceleration using Agilio SmartNIC (Nova, Neutron) x86 Userspace Open vswitch Subsystem OVS CLI Callable API OVS Agent OpenFlow Miss x86 Kernel Configuration via controller, CLI, or Callable API OVS Kernel DP Match/Act PCIe Agilio Intelligent Server Adapter (NIC) Miss Open vswitch Datapath OVS Kernel DP Match/Act (e.g. Entunnel, Deliver to VM, Send to Port) 206 NETRONOME 8

21 OVS Acceleration using Agilio SmartNIC (Nova, Neutron) x86 Userspace Open vswitch Subsystem OVS CLI Callable API 2 OVS Agent OpenFlow Miss x86 Kernel Configuration via controller, CLI, or Callable API OVS Kernel DP Match/Act 2 OVS userspace agent populates kernel cache Agilio Intelligent Server Adapter (NIC) Miss Open vswitch Datapath PCIe OVS Kernel DP Match/Act (e.g. Entunnel, Deliver to VM, Send to Port) 206 NETRONOME 8

22 OVS Acceleration using Agilio SmartNIC (Nova, Neutron) x86 Userspace Open vswitch Subsystem OVS CLI Callable API 2 OVS Agent OpenFlow Miss x86 Kernel Configuration via controller, CLI, or Callable API OVS Kernel DP Match/Act 2 OVS userspace agent populates kernel cache 3 Offload datapath: copy match tables, sync stats Agilio Intelligent Server Adapter (NIC) 3 Miss Open vswitch Datapath PCIe OVS Kernel DP Match/Act (e.g. Entunnel, Deliver to VM, Send to Port) 206 NETRONOME 8

23 OVS Acceleration using Agilio SmartNIC (Nova, Neutron) x86 Userspace Open vswitch Subsystem OVS CLI Callable API 2 OVS Agent OpenFlow Miss x86 Kernel DP Ext. Configuration via controller, CLI, or Callable API OVS Kernel DP Match/Act 4 2 OVS userspace agent populates kernel cache 3 Offload datapath: copy match tables, sync stats 4 Datapath extension software Agilio Intelligent Server Adapter (NIC) 3 Miss OVS Kernel DP Match/Act Open vswitch Datapath (e.g. Entunnel, Deliver to VM, Send to Port) PCIe 4 Datapath Extension or Plugin P4 / C in Sandbox 206 NETRONOME 8

24 P4 for SmartNICs with VNF and µvnf chaining Control App Populate tables, display statistics P4 / C Development Environment Edit - Debug Server (x86 - Linux) Run-Time Interface VNF Kernel Mode (C) netdev 2 VNF User Mode (C) DPDK Demonstrated Concepts: P4 and C running on SmartNIC Debug at P4 level and C level VNFs running on x86 server and on SmartNIC P4 traffic steering with metering Integrated in-band telemetry for measurement of VNF performance Agilio SmartNIC SmartNIC Firmware (P4) Match Protocol Other Security µvnf (C) Timestamp µvnf (C) Latency Stats µvnf (C) Timestamp µvnf (C) Latency Stats µvnf (C) PCIe TCP Meter 206 NETRONOME 9

25 OVS on P4 Datapath (Nova, Neutron) x86 Userspace Open vswitch Subsystem OVS CLI Callable API OVS Agent OpenFlow x86 Kernel OVS Kernel DP Match/Act PCIe Agilio Intelligent Server Adapter (NIC) P4 Matching Datapath Extension or Plugin P4 / C in Sandbox 206 NETRONOME 0

26 OVS on P4 Datapath (Nova, Neutron) x86 Userspace Open vswitch Subsystem OVS CLI Callable API OVS Agent OpenFlow x86 Kernel OVS Kernel DP Match/Act PCIe Agilio Intelligent Server Adapter (NIC) P4 Matching P4 Generated Datapath P4 Datapath Extension or Plugin P4 / C in Sandbox 206 NETRONOME 0

27 OVS on P4 Datapath (Nova, Neutron) x86 Userspace Open vswitch Subsystem OVS CLI Callable API OVS Agent OpenFlow x86 Kernel OVS Kernel DP Match/Act PCIe Agilio Intelligent Server Adapter (NIC) Miss P4 Matching P4 Generated Datapath P4 Datapath Extension or Plugin P4 / C in Sandbox 206 NETRONOME 0

28 OVS on P4 Datapath (Nova, Neutron) x86 Userspace Open vswitch Subsystem OVS CLI Callable API OVS Agent OpenFlow Miss x86 Kernel OVS Kernel DP Match/Act PCIe Agilio Intelligent Server Adapter (NIC) Miss P4 Matching P4 Generated Datapath P4 Datapath Extension or Plugin P4 / C in Sandbox 206 NETRONOME 0

29 P4 into OVS Datapath (Nova, Neutron) x86 Userspace Open vswitch Subsystem OVS CLI Callable API OVS Agent OpenFlow x86 Kernel P4 / OVS Matching P4 / OVS PCIe Agilio Intelligent Server Adapter (NIC) P4 Matching 206 NETRONOME

30 P4 into OVS Datapath (Nova, Neutron) x86 Userspace Open vswitch Subsystem OVS CLI Callable API OVS Agent OpenFlow x86 Kernel P4 / OVS Matching P4 / OVS PCIe Agilio Intelligent Server Adapter (NIC) P4 Generated Datapath P4 Matching P4 206 NETRONOME

31 P4 into OVS Datapath (Nova, Neutron) x86 Userspace Open vswitch Subsystem OVS CLI Callable API OVS Agent OpenFlow x86 Kernel P4 / OVS Matching P4 / OVS PCIe Agilio Intelligent Server Adapter (NIC) Fallback P4 Generated Datapath P4 Matching P4 206 NETRONOME

32 P4 into OVS Datapath (Nova, Neutron) x86 Userspace Open vswitch Subsystem OVS CLI Callable API OVS Agent OpenFlow Fallback x86 Kernel P4 / OVS Matching P4 / OVS PCIe Agilio Intelligent Server Adapter (NIC) Fallback P4 Generated Datapath P4 Matching P4 206 NETRONOME

33 P4 instead of OVS (Datapath and Control) x86 Userspace (Nova, Neutron) Open vswitch Subsystem P4 Agent Fallback x86 Kernel P4 or ebpf or XDP Protocol(s) to be discussed Agilio Intelligent Server Adapter (NIC) Fallback P4 Generated Datapath PCIe P4 Matching P4 206 NETRONOME 2

34 Performance Observations (Agilio -CX SmartNIC) Simple scenario SmartNIC accelerated OVS: 35 < core P4 on SmartNIC: 36 < core => Despite high level language overhead, only deploying the packet processing operations required by the application yields high performance More complex scenario Unaccelerated OVS: 22 server cores SmartNIC accelerated OVS: 33 < server core (>60x efficiency) P4 on SmartNIC: 2 < server core (>40x efficiency) => Hand-tuned implementation beats P4 (currently beta compiler: room for improvement) => Both offer significantly better efficiency (throughput per server core) than unaccelerated OVS Also available: Agilio -LX ~80 Mpps 206 NETRONOME 3

35 Next Steps Enable migration from existing OpenFlow network / cloud control interface to a P4 capable control interface (generated / generic) collaborating with e.g. ONF, Open vswitch, OpenStack, ODL, ONOS, NFV bodies Standardize a persistent IR with a simple to parse format (YAML / JSON) and a stable schema collaborating with e.g. OpenSourceSDN.org, ONF Use subset (extract) of IR as model, also as a description of the run-time interface Equivalent:.h for.c In effect a TTP Supports multiple P4 language versions (or, in future, other languages) Behavioral model v2 developers discovered the usefulness of a JSON based IR Enables ecosystem of tools (optimizers, visualizers, front / back end compilers, etc.) Use Agilio SmartNICs to accelerate existing datapaths - OVS (OpenFlow), Contrail vrouter, Linux to implement new datapaths - P4, ebpf/xdp 206 NETRONOME 4

36 Thank you! More information: and NETRONOME