4. Miscellaneous: network virtualization

Transcription

1 4. Miscellaneous: network virtualization Protocols for Data Networks (aka Advanced Computer Networks)

2 Lecture plan 1. B. Pfaff et al., Design and implementation of Open vswitch, NSDI 15 and B. Pfaff et al., Extending Networking into the Virtualization Layer, HotNets T. Koponen et al., Network Virtualization in Multi-tenant Datacenters, NSDI 14 2

3 Lecture plan 1. B. Pfaff et al., Design and implementation of Open vswitch, NSDI 15 and B. Pfaff et al., Extending Networking into the Virtualization Layer, HotNets T. Koponen et al., Network Virtualization in Multi-tenant Datacenters, NSDI 14 3

4 Context Virtualization has changed the way we do computing The goal of most data centers is to have all hosts virtualized The number of virtual machines has already exceeded the number of servers With the proliferation of virtualization, a new network layer is emerging Within the hypervisor 4

5 Motivation Virtualization imposes new requirements Network mobility, as VMs can migrate between hosts Scaling limits with datacenters with hundreds of thousands of VMs Isolation is required for joint-tenant environments But also provides features that make networking easier Virtualization layer can provide information about host arrivals and movements Topology becomes more tractable Networking is composed entirely of leaf nodes However, this placement complicates scaling It is very flexible, as it is in software 5

6 Contribution Typical model of internetworking in virtualized environments is L2 switching Primary concern is providing basic network connectivity Hard to address the challenges that exist in virtualized environments The authors present the design and implementation of Open vswitch (OvS), a capable virtual switch for virtualized environments A software switch that resides within the hypervisor or management domain Exports interface for fine grained control of the forwarding (via OpenFlow) and of configuration (via OVSDB: to configure queues, create/destroy switches, add/remove ports, etc.) Open-source Multi-platform 6

7 Where is Open vswitch Used? Broad support: Linux, FreeBSD, NetBSD, Windows, ESX KVM, Xen, Docker, VirtualBox, Hyper-V, OpenStack, CloudStack, OpenNebula, Widely used: Most popular OpenStack networking backend Default network stack in XenServer 1,440 hits in Google Scholar Thousands of subscribers to OVS mailing lists source: 7

8 Challenge Performance......without the luxury of specialization Design goals flexibility and general-purpose use and high-performance the primary function of a hypervisor: running user workloads The paper basically shows how OvS obtains high performance without sacrificing generality Most of it details their design optimizations through flow caching and other types of caching to reduce CPU usage and increase forwarding rates 8

9 OVS architecture Largest component is ovs-vswitchd, a userspace daemon Essentially the same across operating systems The data path kernel module is usually written specially for the host OS for performance Remains unaware of OpenFlow, simplifying the module This separation is invisible to the controller 9

10 Use case: network virtualization Implication for performance: 100+ hash lookups per packet for tuple space search! (Note: for packet classification tuple space search is used, with one hash table per type of match; the tuple is the key, with the set of fields for the type of match) based on: 10

11 Non-solutions These helped: Multithreading Flow setup distributed to multiple threads/cores Optimistic concurrent techniques Such as userspace RCU (Read-Copy Update) Batching packet processing Increasing performance for flow setup Classifier optimisations Microoptimisations But none really enough Classification is expensive on general-purpose CPUs! based on: 11

12 OVS Cache v1: microflow cache source: 12

13 Speed up with microflow cache From 100+ hash lookups per packet, to just 1! In practice: Tremendous speedup for most workloads Problematic for traffic patterns with short-lived microflows Fundamental caching problem: low hit rate source: 13

14 Solution: more expensive cache If k c << k 0 +k k 24 : benefit! source: 14

15 Naive approach to populating cache Combine all tables into 1! Result: up to n 1 n 2 n 24 flows Crossproduct problem source: 15

16 Lazy approach to populating cache Solution: Build cache of combined megaflows lazily as packets arrive Same (or better!) table lookups as naive approach. Traffic locality yields practical cache size source: 16

17 OVS Cache v2: Megaflow Cache Megaflows are more effective when they match fewer fields Megaflows that match TCP ports are almost like microflows! Contribution: megaflow generation improvements Tuple priority sorting: when a match occurs, no need to search lower priorities Staged lookup: 4 hash tables instead of 1, with matching in multiple stages (metadata only; metadata + L2; metadata + L2 + L3; metadata + L2 + L3 + L4) Prefix tracking: optimization for IP prefixes using a trie data structure that allows matching of only the higher order bits 17 source:

18 Megaflow vs. Microflow Microflow cache: k 0 + k k 24 lookups for first packet in microflow 1 lookup for later packets in microflow Megaflow cache: k c lookups for (almost) every packet k c > 1 is normal, so megaflows perform worse in common case! Best of both worlds would be: k c lookups for first packet in microflow 1 lookup for later packets in microflow source: 18

19 OVS Cache v3: Dual Caches source: 19

20 Evaluation: in production Cache size 99 th percentile = 7k flows OvS limit = 200k entries So it s fine Cache hit rate 97.7% overall Cache is effective CPU usage 80% hypervisors average 5% or less Their traditional target They individually examined the outliers and realised they had a previously unknown bug, which (they believe) was corrected in OvS

21 Evaluation: microbenchmarks For the tests they ran Netperf s TCP_CRR test It repeatedly establishes a TCP connection, sends and receives one byte of traffic, and disconnects Results from a simple flow table design to illustrate benefits of optimizations, in transactions per second (tps) Each optimization reduces the number of kernel flows needed to run the test (representing trips kernel-userspace), so reduces userspace CPU usage While it increases number of masks/tuples (which increase packet classification cost) However, the tradeoff is overall positive 21

22 Lecture plan 1. B. Pfaff et al., Design and implementation of Open vswitch, NSDI 15 and B. Pfaff et al., Extending Networking into the Virtualization Layer, HotNets T. Koponen et al., Network Virtualization in Multi-tenant Datacenters, NSDI 14 22

23 Context and motivation Server virtualization has become the dominant approach for managing computational infrastructures What is lacking to achieve full virtualization? Virtualizing the network What network aspects are important to virtualize? Network topology Different workloads require different topologies How has this problem been solve traditionally? Simple, build multiple physical networks Address space Virtualized workloads operate in the same address space as the physical network Problems? Cannot move VMs to arbitrary locations Cannot change addressing type (if physical is IPv4, VMs are IPv4) 23

24 Alternatives Wait, but we ve had network virtualization for ages! VLANs Virtualize L2 (Ethernet) networks 24

25 Opening parentheses: VLAN 25

26 Motivation Problem 1: what if a CS user moves office to Chemistry, but wants connect to the CS switch? Need to move all cabling Problem 2: one LAN = a single broadcast domain all layer-2 broadcast traffic (ARP, DHCP, unknown location of destination MAC address) must cross entire LAN; no isolation security/privacy issues, efficiency issues (hard to scale) One possibility to solve this problem would be to replace center switch with router Problem 3: inefficient use of switches If you have many groups with a small number of users each, then you will have many ports unused Computer Science Chemistry Physics Fonte: [Kurose2009] 26

27 VLANs port-based VLAN: switch ports grouped (by switch management software) so that single physical switch 1 2 Chemistry (VLAN ports 1-8) CS (VLAN ports 9-15) operates as multiple virtual switches Chemistry (VLAN ports 1-8) CS (VLAN ports 9-16) Fonte: [Kurose2009] 27

28 traffic isolation: frames to/from ports 1-8 can only reach ports 1-8 can also define VLAN based on MAC addresses of endpoints, rather than switch port dynamic membership ports can be dynamically assigned among VLANs How is forwarding done between VLANS? via routing (just as with separate switches) in practice vendors sell combined switches plus routers Port-based VLAN 1 2 Chemistry 7 8 router CS Fonte: [Kurose2009] 28

29 VLANS spanning multiple switches trunk port carries frames between VLANS defined over multiple physical switches Frames forwarded within VLAN between switches can t be vanilla frames (must carry VLAN ID info) 802.1q protocol adds/removed additional header fields for frames forwarded between trunk ports Electrical Engineering (VLAN ports 1-8) Computer Science (VLAN ports 9-15) Ports 2,3,5 belong to EE VLAN Ports 4,6,7,8 belong to CS VLAN Fonte: [Kurose2009] 29

30 802.1Q VLAN frame format Type frame 802.1Q frame 2-byte Tag Protocol Identifier (value: 81-00) Recomputed CRC Tag Control Information (12 bit VLAN ID field, 3 bit priority field like IP TOS) Fonte: [Kurose2009] 30

31 Closing parentheses 31

32 Alternatives Wait, but we ve had network virtualization for ages! VLANs NAT MPLS Virtualize L2 (Ethernet) networks Virtualize IP address space Virtualize physical paths 32

33 Opening parentheses: MPLS 33

34 Multiprotocol label switching (MPLS) Initial goal: high-speed IP forwarding using fixed length label (instead of IP address) fast lookup using fixed length identifier (rather than shortest prefix matching) borrowing ideas from Virtual Circuit (VC) approach but IP datagram still keeps IP address! PPP or Ethernet header MPLS header IP header remainder of link-layer frame label Exp S TTL Fonte: [Kurose2009] 34

35 MPLS capable routers a.k.a. label-switched router forward packets to outgoing interface based only on label value (don t inspect IP address) MPLS forwarding table distinct from IP forwarding tables flexibility: MPLS forwarding decisions can differ from those of IP e.g, use destination and source addresses to route flows to same destination differently (traffic engineering) re-route flows quickly if link fails: pre-computed backup paths Fonte: [Kurose2009] 35

36 MPLS versus IP paths IP routing: path to destination determined by destination address alone R6 R5 R4 R3 D A R2 IP router Fonte: [Kurose2009] 36

37 MPLS versus IP paths MPLS routing: path to destination can be based on source and destination address R6 R5 R4 entry router (R4) can use different MPLS routes to A based, e.g., on source address R2 R3 D A IP-only router MPLS and IP router Fonte: [Kurose2009] 37

38 MPLS signaling Need to modify OSPF, IS-IS link-state flooding protocols to carry info used by MPLS routing e.g., link bandwidth, amount of reserved link bandwidth Entry MPLS router uses RSVP-TE signaling protocol to set up MPLS forwarding at downstream routers R6 R5 R4 modified link state flooding RSVP-TE D A Fonte: [Kurose2009] 38

39 MPLS forwarding tables in out out label label dest interface 10 A 0 12 D 0 8 A 1 in out out label label dest interface 10 6 A D 0 R6 R5 R4 0 R2 in out out label label dest interface A 0 R D 0 A in out R1 out label label dest interface 6 - A 0 Fonte: [Kurose2009] 39

40 Closing parentheses 40

41 Alternatives Wait, but we ve had network virtualization for ages! VLANs NAT MPLS Virtualize L2 (Ethernet) networks Virtualize IP address space Virtualize physical paths What is the problem with these solutions? VLANs don t scale Point solutions, requiring box-by-box configuration No global, unifying abstractions 41

42 Contribution NVP, a Network Virtualization Platform A complete network virtualization solution Allows the creation of virtual networks, each with independent Service models Topologies Addressing architectures over the same physical network 42

43 Network hypervisor abstractions Control abstraction Tenants define logical datapaths that are configured with their control planes Logical datapath = set of logical network elements How are logical datapaths defined? A packet forwarding pipeline (similar to forwarding ASICs) that contains a sequence of lookup tables The pipeline results in a forwarding decision How are logical datapaths implemented? In the software virtual switches Forwarding decisions are done solely on the end hosts! Advantages over ASIC implementations? More flexibility Can match over arbitrary packet header fields 43

44 Network hypervisor abstractions Packet abstraction Packets sent by endpoints are given the same treatment (switching, routing, filtering) as in the tenant s home network 44

45 Network hypervisor architecture What happens when the logical datapaths reaches a forwarding decision? The packet is tunneled over the physical network to the receiving host hypervisor Using several encapsulation mechanisms, such as GRE or STT Allowing the encapsulation of Ethernet frames inside IP packets, for example Host hypervisor decapsulates the packet and sends it to destination VM The physical network sees nothing but ordinary IP traffic 45

46 Opening parentheses: tunneling 46

47 Generic Routing Encapsulation Tunneling Encapsulation with delivery header The addresses in the delivery header are the addresses of the head-end and the tail-end of the tunnel Delivery header / GRE / Private network site / tunnel /16 Public Network /16 Private network site

48 Closing parentheses 48

49 Discussion What network entity configures the software switches? An SDN controller Tunnels work for point-to-point communication. How about multicast and broadcast? A simple multicast overlay is used, adding physical forwarding elements for that purpose (service nodes) Service nodes replicate the packets received How are logical networks interconnected with physical networks? A gateway is used for this purpose 49

50 Design challenges How to accelerate software switching? How to compute all that forwarding state and disseminate it to the switches, avoiding inconsistencies? How to scale the controller cluster? 50

51 Logical datapath implementation NVP uses Open vswitch (OVS) to forward packets The NVP controller cluster configures the OVS remotely using two protocols OpenFlow to inspect and modify the flow tables OVSDB to create and manage overlay tunnels and to discover which VMs are hosted at a hypervisor How is the logical pipeline created? NVP augments the logical flow table in OVS to include a match over the packet s metadata for the logical table identifier NVP modifies each action of a flow entry to write the ID of the next logical flow table and to resubmit the packet back to the OVS flow table This creates the logical pipeline 51

52 Forwarding performance Traditional physical switches classify packets using TCAMs How can we classify packets quickly with software switches, such as OVS? What techniques are explored in NVP? Flow caching Exploits traffic locality All packets belonging to same flow (say, one VM TCP connection) traverse exactly the same set of flow entries The first packet of the flow is sent from the kernel module to userspace But userspace program installs exact-match flows into the flow table in the kernel, so future packets don t leave the kernel Use of hardware offloading techniques TCP segment offloading (TSO) allows the OS to send TCP packets larger than the physical MTU, and then the NIC takes care of the rest Large Received Offload (LRO) does the opposite (again, work offloaded to the NIC) Problem: current Ethernet NICs do not support offloading in the presence of IP encapsulation Solution: use TSS as encapsulation method Add fake TCP header, and then the NIC is capable of performing the standard offloading mechanisms 52

53 Forwarding state computation Forwarding state is computed based on vnics location info and system configuration, and is pushed to transport nodes via OpenFlow Computational model is entirely proactive Is this different from the traditional SDN model? Different, here controllers push all forwarding state down and do not process any packets Is it good or bad? Simplifies scaling of the controller cluster Failure isolation less problems if connectivity to the controller cluster is lost Full computation after every change is computationally inefficient, so incremental computation necessary Problem: very hard to code and to test Solution: they implemented nlog, a domain-specific, declarative language that allows the separation of logic specification from its implementation 53

54 Controller cluster What techniques are used to scale computation? Controllers are arranged in a two-layer hierarchy Separation of concerns eases computation and allows more parallelization What techniques are used to guarantee high availability? There are hot-standbys at both layers 54

55 Evaluation: cold start Simulates bringing the entire system back online after major datacenter disaster Takes around one hour Comments? 55

56 Evaluation: tunnel performance Why is GRE throughput so low? It is incapable of using hardware offloading STT, on the other hand, is capable of having a throughput equivalent to having no encapsulation 56

57 Discussion What were, in your opinion, the seeds of NVP s success? Make logical networks look exactly like current network configurations despite current networks many flaws, they represent a large installed base, and can be used without modification The purpose-built programming language (nlog) easing development while assuring correctness Leveraging the flexibility of software switching Software enabling much faster innovation SDN control centralization Important to have a centralized global view 57

58 Next lecture: fast networking Mandatory L. Rizzo, netmap: A Novel Framework for Fast Packet I/O, USENIX ATC 12 [Optional] S. Garzarella et al., Virtual device passthrough for high speed VM networking, IEEE/ACM ANCS 2015