Innovation and Experimentation through SDN and Network Virtualization

JOHN HICKS Network Research Engineer SEPTEMBER, 2015 Innovation and Experimentation through SDN and Network ization

I2 Production User I2 Prototype Internet2 Taxonomy Implemented Using LHCONE NET+ External Provider s Dependencies XSEDE GENI? Layer 3 R&E IP and TR-CPS s I2-Run Specific Hardware Connectors GENI Learning ONOS? General Purpose VLAN - AL2S SDN Controller ESNET NOAA NVS (Network ization ) ized Ethernet ing Hypervisor Ethernet es Circuits and Wavelengths - AL1S Fiber & Optical Transport

Internet2 Goals Support production networking Across Internet2 Integrated with partner networks around the globe Be a leader in advanced networking Show what can be done, don t copy what commercial world is doing Innovation Platform: Abundant Bandwidth (100G +) Deeply Programmable (SDN) Friction Free Science (Science DMZs) Use Cases Domain scientists collaborating in data intensive science Extending local control over far-flung campuses (e.g. US, China, Middle East) Massively Online Courses Etc. [ 4 ]

Internet2 Philosophy Support open, standards-compliant implementation of SDN Strongly resist vendor impulse towards vertical integration Decoupling control plane from data plane enables competition between vendors and competition between controllers Deploy multiple vendors in a common network to force inter-operability Creates a lowest common denominator effect Reflects the reality of the R&E networking community 5-7 networks along the E2E path Support both Production Networking and Innovation Twin goals are definitely at odds but What is the point of R&E networks if we re following? There isn t financial support to build operational-quality R&E networks just for network research Harness the strengths of R&E community to influence the market Open, collaborative, innovative community Collectively we have the power to change the conversation [ 5 ]

Network ization on Internet2

Network ization on Internet2 Control a slice of the national network! Enable: Rapid prototyping of advanced applications Rapid prototyping of new network services Rapid advancement of network research

Network ization on Internet2 TBD ONOS GENI Site MGR TBD TBD TBD [ 10 ]

Notable Milestones to Date April 2012: Internet2 announces intent to build 100G Layer 2 network on an SDN substrate in partnership with Indiana University October 2012: Internet2 AL2S launched on Brocade MLXe-16s in pure OpenFlow mode: First nationwide, open 100G network built on SDN Substrate March 2013: Internet2 AL2S becomes multi-vendor with introduction of Juniper MX-960s in pure OpenFlow mode May 2013: Juniper OpenFlow implementation becomes fully supported December 2013: Mulitpoint VLANs supported June 2014: Network ization implemented through Flowspace Firewall hypervisor August 2014: Partnership with ON.LAB begins October 2014: GENI Sitemon v0.1 becomes first alien controller running on the Internet2 network October 2014: Multi-Domain SDX demonstrated April 2015: ONOS Controller / SDN-IP demonstrated with 3 universities June 2015: Three continent deployment of router-less Layer 3 network using ONOS and SDN-IP [ 11 ]

Internet2 Network Software Stack KEY Vendor Developed Internet2 Developed Internet2 Modified 3rd Party Developed Experimenter Code API SDN-IP API EXP APP API FOAM STITCHING AGGREGATE API UI API EXP APP Oscars API GENI Learning ONOS MD- NOX NOX OSCARS IDCP OpenFlow OpenFlow FIU FSFW Utah FSFW MAX FSFW OpenFlow OpenFlow OpenFlow OpenFlow OpenFlow OpenFlow OpenFlow OpenFlow

Internet2 September, 2015 Current Status OpenFlow 1.0 in production OF 1.3 support in FSFW and in design Experimenting with Brocade 5.8b implementation of OF1.3 Working with Juniper on implementing requirements for OF 1.3 Hypervisor (FlowSpace Firewall1.0.5) in production Supports L2 and L3 matching Vendor Updates (current versions Juniper 13.3, Brocade 5.6dc) Vendor-specific limits do exist. Controller ( 1.1.6) in production Supports Layer 2 Trace Accepting 3 rd party controllers Questionnaire Mininet Emulation of AL2S Test Lab Production GENI Aggregate Manager in production Allow provisioning of a sliver across AL2S as part of a larger GENI slice

Internet2 2015 Plans Continue to support network research on AL2S, and we are in particular interested in understanding and meeting needs of GENI researchers Deploy NSI on AL2S Begin conversations about continued IDCP support ONOS Deployment, with Global Peers as a prototype service Work with vendors to get OF 1.3 Support Brocade -> 5.8x ( now ) in testing Juniper -> In discussion Continue to support and enhance, FSFW Evaluating OF 1.3 support in FSFW Refine Slice Deployment Process Faster? Test for correctness, then safety Testing constraints?

Operating SDN Networks: The Good Possible to build and operate a reliable Layer 2 and Layer 3 network atop an SDN Substrate Possible to support multiple controllers concurrently on an SDN substrate through network virtualization Possible to create a mult-domain SDX using network virtualization Possible to build a global Layer 3 network through software on a routerless network in ~1 month [ 15 ]

Operating SDN Networks: The Bad Vendor implementations of OpenFlow 1.0 have been buggy and incomplete Vendor implementations of OpenFlow 1.3 have been very slow to appear, as well as buggy and incomplete OpenFlow 1.0 and 1.3 standards have too many optional features, making the implementation of new features a painstaking negotiating process with multiple vendors OpenFlow 1.0 and 1.3 specs are sufficiently vague that we had to write supplemental specs to ensure vendor interoperability [ 16 ]

Operating SDN Networks: The Ugly Building a network software stack requires absolutely rigorous testing when any component changes Testing harness becomes the resource bottleneck Testing for safety!= testing for correctness Supporting multiple controllers concurrently on a production network software stack: Requires significant FTE resources Moves slower than researchers are accustomed Requires more productization (logging, release management, documentation) than normally done by researchers [ 17 ]

Takeaways Operating an SDN-based network is doable, today, and has been for a 2+years SDN!= Open SDN SDN = Fully programmable devices Open SDN = Fully programmable, vendor-swappable devices It s too soon to declare winners in the network stack space Controllers: ODL, ONOS, Ryu, etc. Apps: FSFW/, SDN-IP, etc. Declaring a winner raises the narrow waist => Less room for R&E innovation We need crisp, complete, required SDN programming interfaces fully implemented across multiple vendors We need to start tool development to support network operators of SDNbased-networks We need maturation of open source controllers Logging, Documentation, Release Management, Long-Term Support Open Source Testing Harnesses [ 18 ]

JOHN HICKS Network Research Engineer SEPTEMBER, 2015 INNOVATION AND EXPERIMENTATION THROUGH SDN AND NETWORK VIRTUALIZATION jhicks@internet2.edu [ 19 ] September 22, 2015 2013 Internet2

Controlling a Slice on Internet2 Request a slice (email: noc@internet2.edu) Receive a questionnaire from Internet2 NOC Submit questionnaire to Internet2 Download FSFW; try your controller in that environment http://globalnoc.iu.edu/sdn/fsfw.html/ Submit your package Good documentation accelerates process! Good logging accelerates process! Internet2 NOC tests your controller on idream GENI environment Problems -> Go back one step Internet2 deploys your controller on Internet2 Network

What do we want you to do Have well tested, well versioned, and packaged code Provide lots of documentation Provide lots of configurable logging Have a Ticketing/Bug reporting system Provide Installation and Operation instructions Given the FlowSpace be able to generate the proper Configuration for your application Be patient, it s a learning experience for all of us

What do you need to do Provide Enough documentation to setup and configure your application Provide enough logging (to a file) to be able to debug your application If it breaks we will disable your slice, and send you the log, your slice will not be enabled until the problem is fixed Any API (besides OpenFlow) or UI must be secure Provide involved and reactive developers Application should already have been tested with FlowSpace Firewall to verify it will function properly FlowSpace Firewall does not re-write rules, it allows or denies rules. Your app needs to be able to work on a set of VLANs (and they wont be the same VLAN across all devices) Know the FlowSpace you want for your slice es EndPoints Number of flows Interfaces

FAQs I don t have a Brocade or Juniper. Can I develop on the idream GENI platform? No, not really. It s a limited resource with a tight schedule. See if you can find a Juniper or Brocade elsewhere to validate controller functionality. Testing on v is not the same as testing on real world es. Despite the vendoragnostic promise of OpenFlow, be prepared to have vendor-specific details in your controller. Am I controlling production traffic? No, you are controlling your slice. You need to generate traffic into the slice. Am I running my controller? No, the Internet2 NOC is running your controller in a private slice. Make sure the logging is good enough that you can figure out what went wrong. Packaging? Documentation? Logging? This is a research project. I just wrote the controller yesterday, and most of the configuration details in my head. The Internet2 NOC is deploying and running your controller and feeding you the logging results. The better the packaging, documentation, and logging, the more likely your effort will be a success. How good is your OpenFlow 1.3 support? Very good, if you mean OpenFlow 1.0. Otherwise, we re not there yet. My controller requires 10 physical servers on which to run. (As opposed to VMs.) Is that OK? Internet2 has a very limited number of servers on which we can deploy controllers. Please plan to minimize your configuration or supply servers we can deploy.

SDX SDX1

SDX SDX1 SDX2 SDX3

SDX NSI NSI NSI SDX1 SDX2 SDX3

Multi-Domain SDX SDX1 SDX2 SDX3

Multi-Domain SDX SDX2 SDX3

Multi-Domain SDX GENI Controller SDX2 SDX3

Multi-Domain SDX FlowSpace Firewall SDX2 SDX3

Multi-Domain SDX SDX2 SDX3

NORDUnet Multi-Domain SDX Internet2 SDX2 SDX3

MD - Utah MAX Multi-Domain SDX Internet2 FIU

ONOS/SDN-IP Prototype - Goals Working with partners such as ON.LAB to develop and deploy innovative networking technologies Internet2 has deployed ONOS / SDN-IP as a prototype service Goal: Explore new research ideas and create prototype services that might influence the next generation of R&E networks. With this effort, Internet2 is exploring the possibility of using its SDN substrate to eliminate routers entirely from the network (just an idea!!!), Doing so could, eventually, lead to a significant improvements in network operations and significant cost savings in capital and operational expenditures for Internet2. [ 36 ]

Deploying ONOS across Internet2 Indiana Gigapop Clemson University Duke University [ 37 ]

[ 38 ] Extending ONOS to across South America

[ 39 ] Internet2 ONOS Topology

Planned Internet2 / GEANT Topology (Phase 3E) [ 40 ]

Prototype Multi-Domain Layer 2 Backdrop: Internet2 operates a Layer 2 Campuses (e.g. University of Utah) operate a Layer 2 Regional Networks (e.g. MAX) operate a Layer 2 Exchange Points (e.g. AMPATH/FIU) operate a Layer 2 Is there a way to create a Multi-Domain Layer 2? Common capabilities Willingness to collaborate Willingness to contribute to a common project Maintain local control Withdraw at any time Enable (illusion of) global control Control remote administrative domains No change in software, just configuration

ONOS/SDN-IP Prototype - Implementation Initial effort: Clemson, Duke, FIU, Indiana GigaPoP, Internet2, MAX/UMD, and University of Utah have agreed to participate Each campus deployed a router and a perfsonar node. Internet2 deployed ONOS / SDN-IP in a slice over FlowSpace Firewall (FSFW) Each campus peered with Internet2 using BGP. [ 43 ]

ONOS/SDN-IP Prototype Transition to Real Traffic Phase 1: Private AS's and Private IPs with perfsonar boxes attached. (Complete.) Phase 2: Public AS's and Public IPs with perfsonar boxes attached. (Planned.) Phase 3: Real traffic. (Planned) [ 44 ]

ONOS/SDN-IP Prototype Transition to Full Deployment Phase A: Run ONOS on a single Ubuntu server taken from the test lab in the Indianapolis Gigapop without syslog server logging enabled. (Complete.) Phase B: Run 2 instances of ONOS on two Ubuntu servers in Washington DC. (Planned.) Phase C: Test new version of ONOS on two CENTOS servers in the lab. (New version will include ON.LAB commitments to ease operation and manageability concerns expressed by Internet2 NOC.) (Planned) Phase D: Deploy new version of ONOS on two CENTOS servers in the field and log on a remote syslog server. (Planned.) Phase E: Peer Internet2 ONOS instance with GEANT ONOS/ICONA instance (after it's deployed) via BGP. (Planned.) Phase F: Explore multiple ONOS instances, linked using ICONA (supplied by GARR). (Contemplated.) [ 45 ]