Optical considerations for nextgeneration network Inder Monga Executive Director, ESnet Division Director, Scientific Networking Lawrence Berkeley National Lab 9 th CEF Networks Workshop 2017 September 11 th, 2017
Exponential growth is deceptive, and then explosive 2 9/14/2017
Petabytes Jan 2018 Jul 2017 Jan 2017 Jul 2016 Jan 2016 Jul 2015 Jan 2015 Jul 2014 Jan 2014 Jul 2013 Jan 2013 Jul 2012 Jan 2012 Jul 2011 Jan 2011 Jul 2010 Jan 2010 Jul 2009 Jan 2009 Jul 2008 Jan 2008 Jul 2007 Jan 2007 Jul 2006 Jan 2006 Jul 2005 Jan 2005 Jul 2004 Jan 2004 Jul 2003 Jan 2003 Jul 2002 Jan 2002 Jul 2001 Jan 2001 Jul 2000 Jan 2000 Jul 1999 Jan 1999 Jul 1998 Jan 1998 Jul 1997 Jan 1997 Jul 1996 Jan 1996 Jul 1995 Jan 1995 Jul 1994 Jan 1994 Jul 1993 Jan 1993 Jul 1992 Jan 1992 Jul 1991 Jan 1991 Jul 1990 Jan 1990 The traffic growth exponential! ESnet Accepted Traffic: Jan 1990 - Jan 2017 (Log Scale) Actual Exponential regression with 12 month projection 1 EB Jan 2021* 1000.0000 100.0000 Projected volume for Jan 2018: 186.6 PB 56 PB, Jan 2017 Actual volume for Jan 2017: 56.0 PB 10.0000 1.0000 0.1000 0.0100 10x growth every 47 months 0.0010 0.0001 0.0000 Month 3 9/14/2017
Challenge: How do we design an affordable optical substrate that is resilient to optical growth? 4 9/14/2017
Context 5 9/14/2017
ESnet is a dedicated mission network engineered to accelerate a broad range of science outcomes We do this by offering unique capabilities, and optimizing the network for data acquisition, data placement, data sharing, data mobility. 6 9/14/2017 imonga at es dot net
Mission: To Enable and Accelerate Scientific Discovery by Delivering Unparalleled Network Infrastructure, Capabilities, and Tools Potential network service requirements to support tomorrow s scientific collaborations 7 Application-Network Interaction Virtual Private Clouds Distributed Workflow Integration Network Security Services Global Connectivity Superfacility Model Deadline Scheduling Tele-Presence Network Content Caching Remote Control Applications Virtual Private Networks Bulk Data Movement Real Time Data Streamin Named Data Networking
Next-generation network (ESnet6) drivers Exponential growth in network CAPACITY needs 72% year-on-year traffic growth since 1990 Cost effective solution to increase capacity as needed Network Life Cycle: Improve RELIABILITY Replace aging infrastructure Increase the cyber-resiliency of the network Network FLEXIBILITY Support increasingly complex workflow models Flexibility at all layers of the network is needed to support wide spectrum of science requirements 8
Design requirements 1. Capacity Predict usage Determine overheads (e.g. burst multiplier, resiliency requirements, short-term growth trends) 2. Services Document workflows Develop service portfolio* *NB: Service Portfolio in conjunction with architecture design drives the technical requirements 9
Capacity Planning Process 1. Determine predicted baseline usage (for 2020, 2025, and 2030) 1. Perform best-fit growth curve of ingress traffic per router 2. Adjust individual router predictions such that total of all router ingress traffic matches ESnet s 25+ year total traffic growth curve 3. Using historical flow data and predicted ingress traffic data, perform full mesh path computation to determine per link utilization from PEto-PE 2. Strategic capacity planning* (for 2020 and 2025) 1. Add burst overhead bandwidth per link based on historical knowledge 2. Add additional bandwidth to paths based on resiliency strategy 3. Keep in view new projects on the horizon *NB: This is an iterative process as we continue to monitor growth trends as well as field requests for new requirements (e.g. new experiments, etc) 10
Long term modeling and capacity prediction continues to be a challenge ESnet6 Predicted Usage Map in Jan 2030 LHCONE ramps up From 1.7 PB in December 2014 ~10x in 8 months To 18.4 PB in July 2015 100+Tbps speeds at long-haul distances on a single fiber pair is outside the existing optical technology envelope 11
What does capacity really look like for the nextgeneration network? Jan 2021 Bandwidth Capacity Planning Predictions 12 9/14/2017
R&D Phase: Architecture and Technologies Matrix Orchestrators (A) Router and DWDM Ethernet Switch Architecture (B) Packet Transport Router Architecture (C) Router and OTS Architecture (D) Router and PKT/OTN OTS Architecture (E) SDN Router and PKT/OTN OTS Architecture (F) SDN Router and OTS Architecture Layer 3 Routers SDN Routers Layer 2 DWDM Ethernet Switches SDN Switches Layer 1 Transport Router DWDM Alien Wave Optical Transport PKT-OTN Optical Transport Systems P2P Optical Transport Systems 13 Packet Optical Integration Traditional Routed Software Defined Networking
Investigating all possible options Orchestrators (A) Router and DWDM Ethernet Switch Architecture (B) Packet Transport Router Architecture (C) Router and OTS Architecture (D) Router and PKT/OTN OTS Architecture (E) SDN Router and PKT/OTN OTS Architecture (F) SDN Router and OTS Architecture Layer 3 Routers SDN Routers Layer 2 DWDM Ethernet Switches SDN Switches Layer 1 Transport Router DWDM Foreign Alien (alien) Wave Optical transponders Transport Packet switch fabrics PKT-OTN Optical Transport Systems Flexible transponders (or OTN-switch capable transponders) P2P Optical Transport Systems Optical Add/Drop mux (fixed-grid or flexgrid), with or without lambda switching, directionless or not Amplifiers Amplifiers Amplifiers Amplifiers Amplifiers Amplifiers Dark fiber Dark fiber Dark fiber Dark fiber Dark fiber Dark fiber
Experimentation 15 9/14/2017
1. Packet-Optical integration PTX 5000 (2015) http://www.juniper.net/us/en/local/pdf/white papers/2000552-en.pdf 16 9/14/2017
2. 400G testbed (2016) colorless mux/demux (CCMD) Raman amp (SRA) Switchable line amp (XLA) 200G/100G/50G transceiver Ciena equipment included: two colorless mux/demux, two Raman amplifiers, two switchable line amplifiers, two flexible grid wavelength switches Flexible grid wavelength switch 17 9/14/2017 6/27/16
Testbed deployment over loaned fiber 18 6/27/16
Testbed deployment over loaned fiber: Spectrum Analysis Captured prior to super-channel configuration Shows channels spaced 50 GHz apart Both channels running over the full 93.3 km fiber distance, error-free. Total spectrum utilized for the 400G signal: 100 GHz 19 6/27/16
Testbed deployment over loaned fiber: Spectrum Analysis Captured after superchannel configuration Shows channels spaced 37.5 GHz apart (black) Increased spectral efficiency (bit/s/hz) Both channels running over the full 93.3 km fiber distance, error-free. Total spectrum utilized for 400G signal: 75 GHz Example impact on fiber capacity, given 4.4 THz of useable spectrum: 8.8 Tbps (88 100G @ 50 GHz) 11.7 Tbps (117 100G @ 37.5 GHz) 20 6/27/16
3. First Production 400G Service on ESnet5 Goal: Deploy and harden a 400G production service (4x100 GigE), perform applications testing, production run. Two new wavelengths were provisioned, 200G per wave (2x100 GigE payload) Wavelength Selectable Switches (WSSs) are in the path, but are limited to 50 GHz granularity. On BayExpress, the production 400G circuit consumed 100 GHz of spectral bandwidth 2 adjacent 50 GHz channels Comes as close to a super channel as possible in production 21 6/27/16
Direction of file system transfers, 15 PB moved between centers across 400G service NERSC: File system Transfers 400G Oakland Scientific Facility [1] Oakland, CA Berkeley Lab s Shyh Wang Hall [2] Berkeley, CA 22 6/27/16
Thoughts 23 9/14/2017
Packet Optical No standard definition Is it: Integration of physical packet and optical functionality? Same chassis? Limiting and vendor lock-in Logical integration of packet and optical control plane? Same flow paradigm? G-MPLS vision? Transport SDN? Integration of services offered? MEF style Ethernet services Broadened our search from physically integrated products to abstract network architecture (integration of forwarding, control and management plane) 24 9/14/2017
What is Disaggregation? Disaggregation is decoupling of software and hardware, and components inside as well Opposite of Monolithic Usually puts the responsibility of integrated system of disaggregated components on the buyer (in this case the network provider) Whether they do it themselves or pay someone to do it Pros Has the power to simplify buy what you need Cons Specification and responsibility of the working system falls on the integrator/purchaser 25 9/14/2017
Bringing (minimal) disaggregation to packet-optical Vendor A DWDM Router Interface Vendor C Open Optical Line System Vendor A DWDM Router interface Vendor B White Box DCI box Vendor B White Box DCI box 26 9/14/2017
Pulling together a potential optical architecture Segment Routing for TE (Control Plane disaggregation) Dedicated transponder shelf (White Box, DCI, or Vendor integrated) DWDM Optics Packet Optical Integrated NEs Open Line System 27 FlexGrid (to support >200Gbps waves) Colorless support, no fixed filters Directionless for wave provisioning flexibility \
Considerations of future bandwidth needs Scaling the optical layer involves: The cost for additional channels (cost of transceivers pairs) underlying photonic layer allows use of the entire C-band many unused photonic channels or paths still available in the network How far those channels can go (optical signal-to-noise ratio) determines the maximum reach from transmitter to receiver consideration of Shannon limit, channel size, modulation format higher modulations will be required for reaching higher fiber capacities The space, power and cooling required (port density, power efficiency) optimize for lowest Joules per bit minimize optical-electrical conversions besides transceiver cost, our next biggest concern 28
ESnet6 ( Hollow-Core ) Conceptual Architecture Overview Services Edge Programmable, Flexible, Dynamic Hollow Core Programmable, Scalable, Resilient Hollow Core Primary function is to maintain connectivity between Service Edge nodes No per-hop L3 routing within the Core, packets will be (label) switched High capacity bandwidth paths with optical express and line sub-rate support Protection and restoration for (Service) Edge-to-Edge connections Dynamically provisionable bandwidth paths Centralized intelligence for traffic engineering paths 29 Low cost to add capacity as needed Services Edge Primary function is to provide customer service handoff Instantiate services locally at point-of-use (where possible, and using the Core only for connectivity to other service edges) Coordination with Core for edge-to-edge services with TE constraints Reactive functions performed locally with proactive functions orchestrated centrally Highly programmable data and control plane Network Elements (NEs) leveraging SDN concepts to dynamic instantiate (new) services as needed
Disaggregation, Open source, and Optical 30 9/14/2017
Combination of Packet-Optical, White Box and Disaggregation: Telecom Infra Project 31 9/14/2017
Combination of Packet-Optical, White Box and Disaggregation: Telecom Infra Project 32 9/14/2017
We have just started our journey 33 9/14/2017
Thank You and Questions? 34