Cisco Ultra Packet Core High Performance AND Features. Aeneas Dodd-Noble, Principal Engineer Daniel Walton, Director of Engineering October 18, 2018

Size: px

Start display at page:

Download "Cisco Ultra Packet Core High Performance AND Features. Aeneas Dodd-Noble, Principal Engineer Daniel Walton, Director of Engineering October 18, 2018"

Debra Snow
5 years ago
Views:

1 Cisco Ultra Packet Core High Performance AND Features Aeneas Dodd-Noble, Principal Engineer Daniel Walton, Director of Engineering October 18, 2018

2 The World s Top Networks Rely On Cisco Ultra M 600M 200G Deployments Total Subscribers Total Sessions Gbps / System (at par with physical)

Market Evolution Delivering the 1 s 1 Gbps at edge 1 ms latency 1 billion

UPP ASR 5500 Ultra Performance parity Functional parity Scale Distributed

Cloud Native 5G Ultra Ultra Platform with CUPS 5G Network Functions Massive IOT

3 Market Evolution Delivering the 1 s 1 Gbps at edge 1 ms latency 1 billion connections Transition to Virtual Performance 4G Ultra Services Platform UGP USF UPP ASR 5500 Ultra Performance parity Functional parity Scale Distributed Architecture Slicing Low Latency Gig-Speed Automation Containers Micro-Services Cloud Native 5G Ultra Ultra Platform with CUPS 5G Network Functions Massive IOT Multi-Access Automated Lifecycle Management Dynamic Distributed Slices Micro-Services Architecture

4 Capacity Growth - Fact PB 2.5Tbps Peak hour throughput N/A Tier 1 X1.8 Peak hour throughput YoY APJC Tier 2 8PB Total daily volume N/A Tier 1 X2.5 Peak hour throughput YoY APJC Tier 1

5 5G Core is Distributed by Design Centralized Services and Connectivity Highly Distributed and Fragmented Network and Services 5G Logical Layout Actual Layout SMF SMF Red Slice Orange Slice

6 Performance

7 Background Despite being best-known for hardware forwarding, Cisco has always built high performance packet forwarding software Exception path processing CPU-centric products VPP (Vector Packet Processor) began ~2002 VPP has been incorporated into many hardware and software products and more recently has been open-sourced as part of FD.io

8 What is VPP/FD.io?

9 FD.io is Project at Linux Foundation Multi-party Multi-project Software Dataplane High throughput Low Latency Feature Rich Resource Efficient Bare Metal/VM/Container Multiplatform FD.io Scope: Dataplane Management Agents - Control Plane Packet Processing Classify/Transform/Prioritize/Forward/Terminate Network IO - NIC/vNIC cores/threads Bare Metal/VM/Container Dataplane Management Agent Packet Processing Network IO

10 Multiparty: FD.io Members Service Providers Network Vendors Chip Vendors Integrators

11 Multiparty: Contributor/Committer Diversity Yandex Qiniu Universitat Politècnica de Catalunya (UPC)

12 Read more at

13 How does VPP work?

14 Memory is the enemy A single 10GbE port is capable of 14Mpps On a 3.5GHz CPU core, we have 250 cycles/packet Each packet must be processed in 67ns Main (DDR) memory is 70ns away This is >100 clock cycles away On Intel Sandy Bridge CPUs caches are 4/12/30 clock cycles away If we are serious about performance - we must optimize the code for cache and memory operations Programming paradigm shift: Scalar to Vector

15 Primer: Instruction and Data caches Instruction cache (I-Cache) Stores only CPU instructions. Holds branch prediction information Helps pre-fetch the incoming instructions Instruction pipeline Registers Data cache (D-Cache) Fast buffer that contains application data Processor operate on data loaded from memory into the data cache then from cache into the CPU registers Resultant stored into register, then to cache and finally to main memory

In scalar processing the whole code cannot fit into instruction cache Modules processing packet, are loaded

16 Scalar Packet Processing Packet processing Ethernet-Input IPv4 Input IPv4 lookup IPv4 transmit ECMP processing LAG processing Transmit Process only single packet at a time. In scalar processing the whole code cannot fit into instruction cache Modules processing packet, are loaded into instruction cache. E.g.: 7 modules processing a single packet. So 4 packets will cause 7*4=28 cache misses High performance hit, workaround bigger caches.

17 Vector Packet Processing Process more than one packet at a time. Grabs all available packets from Rx device. Form a vector of packets ( frame ) Process frame (vector) using a directed graph of nodes

18 VPP Architecture Packet n Hardware Plugin Vector of n packets hw-accel-input dpdkinput vhost-user-input ethernet-input af-packet-input Input Graph Node Graph Node Packet Processing Graph Skip sftw nodes where work is done by hardware already ip6-rewrite ip6-input ip6- lookup ip6-local ip4-input ip4- lookup ip4-local mpls-input ip4- rewrite arp-input Plugin custom-1 custom-2 custom-3 Plugins are: First class citizens That can: Add graph nodes Add API Rearrange the graph Can be built independently of VPP source tree

19 How Vector packet processing works? Exploits the probability that most packets will follow the same graph Fixes I-cache thrashing I-cache reloaded when all packets are finished a node

20 How Vector packet processing works? For eg, here 4 packets will cause I-cache thrashing only 7 times, compared to 28 in scalar packet processing. Primary problem VPP solving Reducing i-cache misses Reducing d-cache misses

21 What happens when processing diverges? Same process, but for subset of packets. Each node still executes the set of packets that belong to that node. Scheduler takes care of the node execution.

22 Sounds good. How fast?

VPP Performance at Scale Phy-VS-Phy [Gbps]] 600.0 400.0 200.0 0.

0 480Gbps zero frame loss 200Mpps zero frame loss IPv6, 24 of 72 cores IPv4+ 2k Whitelist, 36 of 72 cores

online: IMIX => 342 Gbps,1518B => 462 Gbps https://docs.fd.

M4 Intel C610 series chipset 4 x Intel Xeon Processor E7-8890 v3 (18 cores, 2.

23 VPP Performance at Scale Phy-VS-Phy [Gbps]] [Mpps] Gbps zero frame loss 200Mpps zero frame loss IPv6, 24 of 72 cores IPv4+ 2k Whitelist, 36 of 72 cores Zero-packet-loss Throughput 1518B 64B 1518B 64B [Gbps]] Regular performance characterizations online: IMIX => 342 Gbps,1518B => 462 Gbps [Mpps] B => 238 Mpps 64B 64B for 12 port 40GE Hardware: Cisco UCS C460 M4 Intel C610 series chipset 4 x Intel Xeon Processor E v3 (18 cores, 2.5GHz, 45MB Cache) 2133 MHz, 512 GB Total 9 x 2p40GE Intel XL x 40GE = 720GE!! Latency 18 x 7.7trillion packets soak test Average latency: <23 usec Min Latency: 7 10 usec Max Latency: 3.5 ms Headroom Average vector size ~24-27 Max vector size 255 Headroom for much more throughput/features NIC/PCI bus is the limit not vpp

24 VPP Benefits from Intel Xeon Processor Developments Increased Processor I/O Improves Packet Forwarding Rates FD.io SoftwareFD.io Intel Xeon Hardware Intel Xeon = = Terabit Service SP Platform Ethernet YESTERDAY Socket 0 QPI Socket 1 2 Broadwell Broadwell 2 QPI Server CPU Server CPU 3 Server 2x [2 Sockets] PCle Packet Forwarding Rate [Gbps] % 1,120* Gbps TODAY Socket 0 UPI Socket 1 2 Skylake Skylake 2 UPI Server CPU Server CPU SATA B I O S PCH x8 PCIe 50GE PCIe x16 100GE 1 Network I/O: 160 Gbps Core ALU: 4-wide parallel µops Memory: 4-channels 2400 MHz Max power: 145W (TDP) x8 PCIe 50GE PCIe PCIe x16 100GE x16 100GE 1 Server [2 Sockets] Server [1 Socket] +75% % FD.io Takes Full Advantage of Faster Intel Xeon Scalable Processors No Code Change Required SATA B I O S Lewisburg PCH x8 PCIe 40GE x8 PCIe 50GE PCIe x16 100GE 1 PCIe x8 x8 Network I/O: 280 Gbps Core ALU: 5-wide parallel µops Memory: 6-channels 2666 MHz Max power: 205W (TDP) PCIe 50GE 50GE PCIe PCIe x16 100GE x16 100GE 1 Intel Xeon E5-2699v4 22 Cores, 2.2 GHz, 55MB Cache Intel Xeon v3, v4 Processors Intel Xeon Platinum 8180 Processors * On compute platforms with all PCIe lanes from the Processors routed to PCIe slots. Intel Xeon Platinum Cores, 2.7 GHz, 33MB Cache Breaking the Barrier of Software Defined Network Services 1 Terabit Services on a Single Intel Xeon Server!

25 Features define your customers experience, define how you charge/monetize, identify fraud, protect your RAN assets provide visibility into your mobile network. Do % of features follow from 4G to 5G? How, and when, does Slicing change this?

26 Is hardware still needed?

27 Hardware Cisco has unrivaled expertise in packet forwarding in silicon Cisco ASICs FPGAs GPUs Merchant silicon (NPUs, ASICs) SmartNICs Feature / Performance tradeoffs are limiting Experience with new software architecture is making software stronger than ever before Continue to investigate/prototype

28 Summary

29 Summary Cisco Ultra Packet Core Feature Rich Unmatched for feature/performance Many IP services beyond standards Packet core user planes are changing/adapting Demand for cost / performance New form factor (physical, virtual, containers) 4G data is growing fast ( x per year) CUPS User Plane opportunity for SW deployments 5G is coming Much higher data rates demand multithreaded solutions Needs small, distributed and public cloud UPF

30 Thank You!

A Universal Terabit Network Dataplane

A Universal Terabit Network Dataplane FastData.io Project th of July 07: material developed for Intel Purley Scalable Processor launch. Video clip version: https://www.youtube.com/watch?v=alj0xlevv. Accompanying