Applying Hyper-scale Design Patterns to Routing

Applying Hyper-scale Design Patterns to Routing Hannes Gredler, CTO RtBrick Inc. DEVNET-2064 2

Who am I? CTO at RtBrick, Inc. Past stint: Distinguished Engineer with the other router-vendor 18 Years working experience, developing, deploying and supporting Routing Software Expertise BGP, IS-IS, MPLS 20+ Patents 20+ Proposed Standards http://www.arkko.com/tools/allstats/hannesgredler.html IETF WG co-chair (IS-IS) DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 3

>2013 exposure to Data Center Networks & SR New large-scale data-center network model emerging [draft-ietf-rtgwg-bgp-routing-large-dc] End-to-End Layer-3 routing Fixes issues with L2 switching data plane. Hierarchical Topology CLOS-based Max 5-stages Use of aggregation at TORs DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 4

Got a couple of inconvenient insights Networks have become Anti-Moore direct sourcing from OEM manufacturers in Taiwan Hardware is a Commodity Cost per Bit dropping sharply (USD 400 / 100GBE port) Boutique ASICs viable in 5 years from now? Curated Software Release models approaching EOL Modularization or Custom package selection desired (no-pim, no RSVP, etc.) Pay per-use Different model (node vs. system) for Resiliency Open sourcing of components the new normal Integration of components becomes core competency DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 5

1) The network is the computer John Gage Sun Microsystems DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 6

2) Is it possible to construct a router based on the web 2.0 mindset? Hannes Gredler, 2015 DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 7

Agenda Introduction Multi-Level Architecture Micro-services & APIs Commoditization & Unit Economics Resiliency, system coupling and state recovery Open Source Development & Test Conclusion

Multi-Level Architecture

Hyper-scale Multi-level Architecture DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 10

Forwarding node Translates RIB Objects to local OS representation Tables Routes Nexthops Hardware Prefix Caching Aggregate FIB table (filter specifics) Localize fwd table VPNs DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 11

Protocol I/O node Schema driven protocol serializer /de-serializer Keep alive delegation /absorption Terminal Communication point for Sockets, stdio & file I/O Pre-processing protocol stream (filter BGP PA128) Queuing machinery & Routing Protocol update generation Interface state handling DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 12

Application (route computation) node Data Structure Schema input_proc add: bgp_filter_input chg: bgp_filter_input del: - IB: pre-filter Nbr Neighbor 193.203.0.40 pre-filter Ipv4 RIB Schema driven Data Structure Server Stores Applications Objects Routes, Nexthops, Tables Triggered execution (Add, Chg, Del) of internal/external Application code Python functions C/C++ library calls Executables vfork() APPd DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 13

Putting it all together DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 14

Micro-Services & APIs

Build a system of little components Micro-service architecture is like a UNIX pipeline model Small pieces of software, serving a unique purpose Easy transfer of state from one brick to next Source Filter Sort Filter Sink curl http://192.168.1.1/bds/object grep Received-From: sort grep foo > out DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 16

REST/JSON based APIs DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 17

Database centric / Distributed Data Store bds://local/isis.adj bds://local/bgp.neighbor PUBSUB bds://217.160.181.216/bgp.rib-in bds://local/isis.lsdb.l2 DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 18

Open IPC format = BSON/JSON Binary JSON for memory and I/O efficiency JSON conversion on the fly possible DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 19

Table replication & state flow within a system DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 20

Commoditization & Unit Economics

Compute Strategy: Yahoo vs. Google Few Big vs. Many Small DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 22

Commodity data plane = White-boxes Economy of scale will ultimately render custom-asics obsolete FY2016 systems shipping: 100GB, > 128K FIB entries Disintegration happening soon to enter the Edge Router Business For ease of integration makes no Hardware, no locality, no OS assumptions Unbounded Configuration Possibilities: Single Switch, Cluster of Switches, Co-located x86 Rack Servers. Large FIBs, Small FIBs, SW-based forwarders & Combos thereof DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 23

Commodity control plane = 1RU Rack Servers 16-32 CPU Cores, 64 GB RAM, Solid State Disks app USD 3000 Runs stock Ubuntu / Centos Linux Containers (LXC) dependency management Para-Virtualization DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 24

Resiliency, system coupling and state recovery

Hyper-scale Multi-level Architecture DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 26

Resiliency DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 27

Resiliency snapshot DB to disk DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 28

Resiliency (2) restart based on disk snapshot DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 29

Open Source Development & Test

Open Development Open Source 100 eyes better than 4 eyes, Network effects Long term Maintenance Open Source means sharing of not just Code: Code Test Build Documentation DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 31

Open development (1) Use what is usable No needs to re-invent Linux, event-loops, memory managers Kernel based networking stacks are not usable for a router Debugging hard (GDB live attachment) Experimental forwarding code with no fault-domains in your kernel, really? TCP snapshots / restart. In 2016Q1 we did not have a packet forwarding core Cisco did release fd.io / VPP User space DPDK design aligned with our (religious) believes Most feature complete open-source L3 forwarder Engineered for performance and maintainability DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 32

Open development (2) Kick-ass VPP crew Helped us to implement necessary core-features (indirect next-hop) within two weeks. Good balance between Stability and feature velocity Excellent Continuous Integration & Test Automation (untypical for FLOSS projects) DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 33

Open Development (3) - Ostinato DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 34

Open Development (4) VPP Internet stream generator DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 35

Conclusion

In Conclusion Network Equipment design got to be Distributed, Multi-level Architecture Micro-service based Running on Commodity Hardware System Resilient Open Development / Open Test Cisco Vector Packet Processing (VPP) Best code in the industry (why is this free?) Good Code Governance Establishment of an innovative ecosystem around VPP underway DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 37

rtbrick demo at fd.io booth

Demo hosted at EC2 instance DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 39

Demo dashboard DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 40

Thank you

Complete Your Online Session Evaluation Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card. Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us. Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 42

Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Lunch & Learn Meet the Engineer 1:1 meetings Related sessions DEVNET-2064 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 43