Applying Hyper-scale Design Patterns to Routing

Transcription

1

2 Applying Hyper-scale Design Patterns to Routing Hannes Gredler, CTO RtBrick Inc. DEVNET

3 Who am I? CTO at RtBrick, Inc. Past stint: Distinguished Engineer with the other router-vendor 18 Years working experience, developing, deploying and supporting Routing Software Expertise BGP, IS-IS, MPLS 20+ Patents 20+ Proposed Standards IETF WG co-chair (IS-IS) DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 3

4 >2013 exposure to Data Center Networks & SR New large-scale data-center network model emerging [draft-ietf-rtgwg-bgp-routing-large-dc] End-to-End Layer-3 routing Fixes issues with L2 switching data plane. Hierarchical Topology CLOS-based Max 5-stages Use of aggregation at TORs DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 4

5 Got a couple of inconvenient insights Networks have become Anti-Moore direct sourcing from OEM manufacturers in Taiwan Hardware is a Commodity Cost per Bit dropping sharply (USD 400 / 100GBE port) Boutique ASICs viable in 5 years from now? Curated Software Release models approaching EOL Modularization or Custom package selection desired (no-pim, no RSVP, etc.) Pay per-use Different model (node vs. system) for Resiliency Open sourcing of components the new normal Integration of components becomes core competency DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 5

6 1) The network is the computer John Gage Sun Microsystems DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 6

7 2) Is it possible to construct a router based on the web 2.0 mindset? Hannes Gredler, 2015 DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 7

8 Agenda Introduction Multi-Level Architecture Micro-services & APIs Commoditization & Unit Economics Resiliency, system coupling and state recovery Open Source Development & Test Conclusion

9 Multi-Level Architecture

10 Hyper-scale Multi-level Architecture DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 10

11 Forwarding node Translates RIB Objects to local OS representation Tables Routes Nexthops Hardware Prefix Caching Aggregate FIB table (filter specifics) Localize fwd table VPNs DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 11

12 Protocol I/O node Schema driven protocol serializer /de-serializer Keep alive delegation /absorption Terminal Communication point for Sockets, stdio & file I/O Pre-processing protocol stream (filter BGP PA128) Queuing machinery & Routing Protocol update generation Interface state handling DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 12

13 Application (route computation) node Data Structure Schema input_proc add: bgp_filter_input chg: bgp_filter_input del: - IB: pre-filter Nbr Neighbor pre-filter Ipv4 RIB Schema driven Data Structure Server Stores Applications Objects Routes, Nexthops, Tables Triggered execution (Add, Chg, Del) of internal/external Application code Python functions C/C++ library calls Executables vfork() APPd DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 13

14 Putting it all together DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 14

15 Micro-Services & APIs

16 Build a system of little components Micro-service architecture is like a UNIX pipeline model Small pieces of software, serving a unique purpose Easy transfer of state from one brick to next Source Filter Sort Filter Sink curl grep Received-From: sort grep foo > out DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 16

17 REST/JSON based APIs DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 17

18 Database centric / Distributed Data Store bds://local/isis.adj bds://local/bgp.neighbor PUBSUB bds:// /bgp.rib-in bds://local/isis.lsdb.l2 DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 18

19 Open IPC format = BSON/JSON Binary JSON for memory and I/O efficiency JSON conversion on the fly possible DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 19

20 Table replication & state flow within a system DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 20

21 Commoditization & Unit Economics

22 Compute Strategy: Yahoo vs. Google Few Big vs. Many Small DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 22

23 Commodity data plane = White-boxes Economy of scale will ultimately render custom-asics obsolete FY2016 systems shipping: 100GB, > 128K FIB entries Disintegration happening soon to enter the Edge Router Business For ease of integration makes no Hardware, no locality, no OS assumptions Unbounded Configuration Possibilities: Single Switch, Cluster of Switches, Co-located x86 Rack Servers. Large FIBs, Small FIBs, SW-based forwarders & Combos thereof DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 23

24 Commodity control plane = 1RU Rack Servers CPU Cores, 64 GB RAM, Solid State Disks app USD 3000 Runs stock Ubuntu / Centos Linux Containers (LXC) dependency management Para-Virtualization DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 24

25 Resiliency, system coupling and state recovery

26 Hyper-scale Multi-level Architecture DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 26

27 Resiliency DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 27

28 Resiliency snapshot DB to disk DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 28

29 Resiliency (2) restart based on disk snapshot DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 29

30 Open Source Development & Test

31 Open Development Open Source 100 eyes better than 4 eyes, Network effects Long term Maintenance Open Source means sharing of not just Code: Code Test Build Documentation DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 31

32 Open development (1) Use what is usable No needs to re-invent Linux, event-loops, memory managers Kernel based networking stacks are not usable for a router Debugging hard (GDB live attachment) Experimental forwarding code with no fault-domains in your kernel, really? TCP snapshots / restart. In 2016Q1 we did not have a packet forwarding core Cisco did release fd.io / VPP User space DPDK design aligned with our (religious) believes Most feature complete open-source L3 forwarder Engineered for performance and maintainability DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 32

33 Open development (2) Kick-ass VPP crew Helped us to implement necessary core-features (indirect next-hop) within two weeks. Good balance between Stability and feature velocity Excellent Continuous Integration & Test Automation (untypical for FLOSS projects) DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 33

34 Open Development (3) - Ostinato DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 34

35 Open Development (4) VPP Internet stream generator DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 35

36 Conclusion

37 In Conclusion Network Equipment design got to be Distributed, Multi-level Architecture Micro-service based Running on Commodity Hardware System Resilient Open Development / Open Test Cisco Vector Packet Processing (VPP) Best code in the industry (why is this free?) Good Code Governance Establishment of an innovative ecosystem around VPP underway DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 37

38 rtbrick demo at fd.io booth

39 Demo hosted at EC2 instance DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 39

40 Demo dashboard DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 40

41 Thank you

42 Complete Your Online Session Evaluation Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card. Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us. Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 42

43 Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Lunch & Learn Meet the Engineer 1:1 meetings Related sessions DEVNET Cisco and/or its affiliates. All rights reserved. Cisco Public 43

44