Data Center Interconnect for the Webscale Era

Independent market research and competitive analysis of next-generation business and technology solutions for service providers and vendors Data Center Interconnect for the Webscale Era A Heavy Reading white paper produced for VIAVI Solutions Inc. AUTHOR: STERLING PERRIN, PRINCIPAL ANALYST, HEAVY READING

INTRODUCTION The cloud computing era is characterized by on-demand workloads, unprecedented computing flexibility, low infrastructure costs and massive scalability (up and down, as required). Controlling the massive data centers, data center infrastructure and applications that make up the cloud are the global webscale Internet companies, including Apple, Amazon, Google, Facebook and Microsoft. As data center infrastructure scales to meet cloud demands, so must the networks required to connect data centers with one another data center interconnect (DCI). This white paper focuses specifically on the DCI requirements and trends for the largest Internet companies, the webscale Internet companies. The first half of the paper defines the unique DCI requirements for webscale providers and the tools they've adopted to succeed. The second half discusses three key technology trends in webscale DCI: disaggregation, open line systems (OLSs) and 100G and beyond. DRIVERS & REQUIREMENTS FOR DCI Heavy Reading estimates that, collectively, the webscale Internet companies spent about $27 billion on network equipment in 2016, or roughly 14 percent of global service provider capex during that year. The bulk of webscale equipment spending goes to servers and switches and other infrastructure that resides within the data center, but a sizeable (and growing) portion is dedicated to optical networking equipment used for connecting data centers together across geographies, including metro, long haul and even submarine routes. This is the data center interconnect (DCI) market. With webscale network equipment capex increasing at 9.6 percent annually, the DCI equipment sub-segment of capex is increasing at an even faster rate. For example, Heavy Reading forecasts that global metro DCI equipment spending will increase at a 22.4 percent CAGR from 2016 to 2021, growing from $726 million in 2016 to $2 billion in 2021. Webscale Requirements Originally, webscale companies bought the available telecom-purpose dense wavelength division multiplexing (DWDM) equipment and forced it to fit their unique application requirements. But the differences between traditional telecom transport and webscale transport requirements are stark. A sampling of these differences is highlighted in Figure 1. Figure 1: Webscale Requirements Mapped Compared to Traditional Telcos Traditional Telco Characteristic OSMINE/TIRKS Integration Proprietary Hardware and Software NMS/Vendor release cycle Scale (10-100 Gbit/s per wavelength) Multiple protocols and services Lowest cost per Gbit/s Webscale Internet Company Requirement Operational Simplicity Open source and open APIs DevOps model Hyperscale (100 Gbit/s+ per wavelength) IP and data center interconnection only Lowest cost, lowest power and smallest footprint per Gbit/s HEAVY READING OCTOBER 2017 DATA CENTER INTERCONNECT FOR THE WEBSCALE ERA 2

Webscale Tools To address their unique requirements as efficiently as possible, webscale providers built and adopted their own set of tools, many of which are spreading beyond webscale to traditional telecom providers. The most significant webscale tools are described in some detail below. Open source code and specifications: Open source code has risen as a means of achieving the standardization goals of global connectivity, openness and efficiency much faster than the historical standards process, and in a way that is ultimately much more open. Whereas traditional consensus-driven standards take years for ratification, open source software projects launch in a matter of months. Open source software successes abound, including OpenStack, Kubernetes, Apache Hadoop, Open vswitch and OpenFlow, among many others. Open source has also spread to hardware initiatives, with the Open Compute Project (OCP) as the classic example. Most recently, the Telecom Infra Project (TIP) seeks to replicate OCP's success in telecom infrastructure. Open APIs: Open application programming interfaces (APIs) decouple software development from hardware development and allow third-party developers (including the service providers themselves) to program and make changes to systems. For webscales, open APIs have given them the ability to rapidly customize systems to their requirements and break away from the rigid hardware vendor development cycles of the past. DevOps: DevOps is the combination of software developers, IT and network operations staff to build, test and release software collaboratively. Benefits of the DevOps methodology include rapid, frequent and more reliable software releases and upgrades, as well placing operations control in the hands of the service providers themselves. The DevOps methodology stands in stark contrast to traditional operator operations support systems (OSSs), which are based on closed software systems that require vendor input for even minor system updates and changes. SDN: Webscale Internet companies pioneered the use of software-defined networking (SDN) in the wide-area network (WAN). Launched in 2011, Google's B4 WAN (originally called G-Scale) used SDN to boost network utilization on its global internal DCI network to more than 95 percent, thus enabling Google to scale its network capacity at lower costs than otherwise possible. In this and other WAN applications, the key benefit SDN is automating tasks that previously required manual/human intervention. Automating tasks delivers opex savings, but the bigger value of automation is in rapid network delivery and network changes, in most cases at speeds and complexity that are not possible manually. SDN is tightly coupled with APIs, and SDN is a key enabler for the disaggregation, open line and bandwidth variable transceiver trends discussed in the next section. Commoditized hardware: In the unending mission to reduce cost per bit for traffic, commodity hardware platforms are essential, wherever possible. While there are industry examples of webscale providers building networking hardware themselves, these providers have made clear that their preference is really to buy network equipment from suppliers as long as equipment meets their cost and performance requirements. Certainly, x86-based servers and switches play a major role today in data center infrastructure, but x86 is not sufficient for much of DCI. Rather, Heavy Reading sees a broader trend toward using pluggable optics and merchant silicon in nearly all applications. Pluggable optics and merchant silicon bring benefits of lower costs, reduced vendor lock-in and the separation of hardware from software that is key to disaggregation. HEAVY READING OCTOBER 2017 DATA CENTER INTERCONNECT FOR THE WEBSCALE ERA 3

KEY WEBSCALE DCI TRENDS Disaggregation of Optical Terminals Disaggregation is one of several trends that started with webscale providers, was pushed to the DWDM equipment suppliers and is now moving to traditional telecom network operators. Heavy Reading defines functional disaggregation in networking as: "The separation of networking equipment into functional components and allowing each component to be individually deployed. Ideally, the function is provided in the smallest form-factor capable of delivering a specific function. Additionally, the equipment should be self-contained, require no additional common equipment to operate, and incorporate open APIs to enable SDN control." One prominent example of disaggregation in action is the TIP's Voyager Open Packet DWDM transponder platform, announced in November 2016. Based on merchant hardware and an open software architecture, Voyager includes a Broadcom Tomahawk packet-switching ASIC, Acacia AC400 DSP ASIC and an open software layer that decouples hardware from software, consistent with the principles of SDN. Significantly, Voyager open transponder trial progress has been rapid and global. As of MWC in February 2017, providers including Equinix (U.S.) Orange (France), MTN (South Africa) and Telia Carrier (pan-europe) had announced various Voyager network trials. ADVA and Coriant are two major equipment suppliers that are supporting the Voyager product. Disaggregation runs counter to the functional converge trend in optical equipment over the past two decades, in which vendors tightly coupled proprietary hardware and software together in advanced systems (such as packet optical transport). The webscale-led disaggregation trend, however, is breaking the convergence hold among traditional suppliers and their traditional telecom operator customers. Participation of traditional hardware suppliers and operators in TIP's Voyager trials illustrates this point. The rapid proliferation of disaggregated commercial DCI gear provides further illustration. Systems vendors that have launched disaggregated DCI systems for the webscale market include Infinera (Cloud Xpress2), ADVA (TeraFlex), Coriant (Groove 30), Ciena (Waveserver and Waveserver Ai), Cisco (NCS 1000), Fujitsu (Infinity T100) and Nokia (1830 PSI 2T), among others. These compact 1RU systems pack 1 Tbit/s capacity or higher and include only the high-speed Ethernet interfaces that DCI applications require. While not full optical "white box" systems to the degree of TIP's Voyager, the new disaggregated DCI products incorporate a far greater degree of openness than in DWDM's past. In particular, open APIs are a key feature of disaggregated DCI that allows webscale providers to access the boxes directly and write their own programs for provisioning, managing and automating functions and tying the systems boxes into their own IT environments completely customized and completely independent of the hardware vendors themselves. Open software programmability is one of SDN's key tenets. Open Line Systems Just like they are doing with the end terminals, webscale providers are changing the way that optical line systems are architected and deployed for DCI. Optical line systems consist of wavelength multiplexers/demultiplexers, optical amplifiers and ROADMs, as well as the control/management of those components. Historically, the line system has been tightly HEAVY READING OCTOBER 2017 DATA CENTER INTERCONNECT FOR THE WEBSCALE ERA 4

coupled with the terminal systems so that both line and terminal are supplied by the same vendor, and other vendors' transponders don't work over someone else's optical line. The end-to-end optical network is a closed and proprietary system. Some webscale providers have concluded that decoupling the line from the terminals can yield major benefits for them moving forward thus extending the disaggregation concept of DCI to include the optical line system, as well the terminals (discussed above). The benefits of an open line system (OLS) include: Rapid technology adoption: Transponders and line systems run on different technology innovation cycles. Specifically, coherent detection and photonic integration (including silicon photonics) are leading to rapid innovation in transponders, while line systems evolve more slowly. Decoupling the line from the terminals allows service providers to advance through several generations of transponder technologies without having to change the line systems. Eliminate vendor lock-in. An OLS naturally eliminates vendor lock-in as service providers can purchase line systems and transponders from different suppliers, share a single line among several transponder suppliers and even source different line components form multiple suppliers (such as ROADMs). This makes suppliers more competitive with each other, speeds industry innovation and lets service providers select best-of-breed solutions. Open SDN control. Open APIs on OLSs allow service providers to break from proprietary control/management systems and provide a path to true SDN control in the optical domain, in which northbound APIs from the elements connect to network orchestrator software that is typically built in-house by the webscale provider. Thus, an OLS with open APIs under a common network orchestrator allows for end-to-end control of multi-vendor networks. Microsoft has been one of the strongest proponents of the OLS, having introduced the concept at OFC 2015. The following year, at OFC 2016, Microsoft presented results of a lab system emulation of an OLS running over 4,000 km of fiber and representing its North American backbone network. Colorless, flexgrid ROADMs were used. In addition to the Microsoft work, TIP is building an OLS within its Open Packet Optical Transport Working Group. This is the same working group responsible for the Voyager disaggregated DWDM system. TIP is building a node-level optical spec that is tailored for a point-to-point OLS configuration with specifications proposed for key optical parameters. Centralized SDN control is also part of TIP's OLS plan. 100G & Beyond With DCI bandwidth continuing its rapid growth, channel capacity is at the heart of transponder/line-side priorities for webscale DCI. Within metro DCI, the weight of service provider spending is rapidly shifting to 100G and higher data rates as those rates become commercially available, all but eliminating the need for 10G in the DCI network. Figure 2 shows Heavy Reading's global metro DCI line-side port forecast by line rate. While 100G transponders will remain the DCI workhorse over the next five years, 200G line rate adoption will be much faster, as service providers seek to squeeze the greatest capacity and highest spectral efficiency out of their fiber. Heavy Reading forecasts that 200G (and higher) HEAVY READING OCTOBER 2017 DATA CENTER INTERCONNECT FOR THE WEBSCALE ERA 5

line shipments will increase at a 53 percent CAGR through 2021 to account for 18 percent of all metro DCI lines shipped in 2021. Single-wavelength coherent 400G transponders are just starting to become commercially available, and metro DCI will be the first application for the new line rate. As costs drop, 400G is expected to become a significant contributor by 2021. Figure 2: Worldwide Metro DCI Line-Side Ports by Rate, 2015-2021 (Units) 160,000 120,000 80,000 40,000 0 2015 2016 2017 2018 2019 2020 2021 Source: Heavy Reading, 2017 10G 40G 100G 200/400G DCI is not just about more bandwidth, and increasing the flexibility of available capacity is a critical requirement for the line. Webscale providers' needs for greater flexibility has sped the availability of multiple modulation formats within transponders. Coherent transponders from many vendors now support multiple modulation formats, including BPSK, QPSK, 8QAM and 16QAM, as well as different options for forward error correction (FEC) coding and software programmability for setting functionality. Using flexible transponders, service providers can software-select the right characteristics to yield the greatest amount of capacity tuned to every reach and set of fiber conditions throughout the network. Thus, some transponders can be configured for 16QAM 200G transmission on a short 50 km DCI link, while others may use 100G QPSK for a 1,000 km link across the country. As discussed above, coherent transmission provides the greatest capacity and modulation flexibility, but some webscale providers are driving 100G direct detect modulation formats for shorter-reach metro DCI applications. Direct detection lacks coherent's tremendous reach, but, on the plus side, direct detect cuts costs, size and power consumption compared to equivalent 100G coherent systems. Given reach limitations, PAM4 is targeted at DCI applications less than 80 km. Of particular interest is PAM4 modulation (though other direct detect variants will likely surface as this application matures). Partnering with Microsoft, module maker Inphi has developed 100G transceivers that use PAM4 silicon, consume 4.5 watts of power per 100G, transmit up to 80 km and plug directly into Layer 2/3 data center switches (eliminating the need for external DWDM boxes for interconnection). At OFC 2017, Microsoft, ADVA and Inphi published a technical paper detailing results of a 4 Tbit/s commercial system and line successfully delivering full capacity over an 80 km link. HEAVY READING OCTOBER 2017 DATA CENTER INTERCONNECT FOR THE WEBSCALE ERA 6

CONCLUSIONS The webscale Internet companies are driving optical networking innovations to meet their unique requirements for interconnecting the data centers that host the cloud's infrastructure and applications. Those requirements include operational simplicity, software and hardware openness, terabit-scale interconnectivity and the lowest cost, lowest power and smallest footprint per Gbit/s. Webscale tools to achieve these requirements include extensive use of open source code and specifications, open APIs, adoption of DevOps processes, centralized and open SDN control, and commoditized hardware wherever possible. Cutting across these webscale tools, Heavy Reading identifies three key DCI technology trends driving innovation over next three years: Disaggregation of terminals: Speeds customized innovation in transponders, drives hardware commoditization, eliminates vendor lock-in and fits end-to-end SDN model for software automation. Open line systems: Separates faster terminal development from slower line-side development cycles, allows mixing and matching of transponder vendors and transponder types, eliminates vendor lock-in and fits end-to-end SDN model. 100G+: Advanced modulation formats (including QPSK, 8QAM, 16QAM, PAM4) coupled with the flexibility of bandwidth variable transponders allows webscale companies to drive the greatest capacity and highest possible spectral efficiency from optical fibers, given variations of reach and fiber types throughout their networks. Finally, we note the criticality of optical testing in DCI, including turn-up and in-service testing. As the cloud becomes the primary model for enterprise applications and as the data center becomes the new central office in telecom, assuring availability and reliability of connections within and between data centers is essential for data center operators and their customers. This includes physical layer testing to ensure properly functioning fibers and connectors and protocol testing up to 400G to cover the proliferating options for modulation and standards. In addition, concepts of disaggregation, open hardware and open software appear first in DCI before spreading to broader applications. Thus, we see white box testing and verification as important for DCI today. HEAVY READING OCTOBER 2017 DATA CENTER INTERCONNECT FOR THE WEBSCALE ERA 7