Ethernet Switching at 10 Gigabit and Above IST

Ethernet Switching at 10 Gigabit and Above IST-2001-33182 Workpackage WP7: Demonstrations Deliverable D7.2: Native Ethernet Transmission Beyond the LAN Version 1.0 Public Issue Date: December 2004 INFORMATION SOCIETY TECHNOLOGIES (IST) PROGRAMME

Contents Contents... 2 Abstract... 3 Introduction... 3 Common foundations... 4 10 GE LAN PHY - Ethernet over dark fibre... 4 10 GE WAN PHY - Ethernet over long-distance networks... 6 Ethernet versus SONET/SDH : OAM over MAN and WAN... 9 Conclusions... 10 Acknowledgements... 11 References... 11 ESTA IST-2001-33182 2 22/12/2004

Abstract Ethernet is the de-facto LAN standard technology. Extending the technology outside the LAN and into the MAN and WAN has lately been the focus of several standardisation bodies and industry forums. We present a practical approach to the problem of using 10 Gigabit Ethernet equipment for building the data transmission layer of long-distance networks. Introduction The purpose of this paper is to explain the potential of the new 10 Gigabit Ethernet local area and wide area interfaces. Using this information we demonstrate how they may be exploited to provide extended LAN segments over long distances. This is of interest to enterprises requiring high capacity off-site links at lowest cost, or to research networks seeking to interconnect widely separated data centres or carriers wishing to offer cost effective solutions to customers. The 10Gbit/s standard offers two PHY interfaces for use at 10Gbit/s, the LAN and the WAN. The LAN PHY is a straight tenfold increase on the more widely known 1Gbit/s LAN interfaces. It is limited to a maximum of 40Kms but the optics defined in the specification are suitable for amplification by any standard amplifier such as those used in wide area network repeaters. We show that by using such amplifiers we have extended a LAN segment to a radius of 550 kilometers and have demonstrated this with real applications running point to point across Denmark. This technique can be employed by any Research Network having access to the POPs in which to install the amplifiers or to a carrier wishing to provide lowest cost high bandwidth services. The WAN PHY is a completely new specification for Ethernet and is intended as an interface between Ethernet LAN segments and the wide area network running SONET/SDH. The specification itself caters to both the datacom and telecom communities but in the authors experience neither community really understands or accepts it. This paper explains the reasons for this and goes on to show how the WAN PHY may be exploited to provide extended segment LAN services point to point across any SONET domain. We have described the extension of a LAN segment to run between Geneva and Tokyo with no loss of functionality using this technique. We note that recent ITU specifications have standardized this approach making it available to commercial exploitation. Finally we explore the management aspects of handling the extension of Ethernet this way to show that the potential exists for correctly managed services using these techniques. ESTA IST-2001-33182 3 22/12/2004

Common foundations The IEEE 802.3 family of standards defines two layers for Ethernet: a physical data transmission (PHY) layer and a medium access control (MAC) layer. The fastest transmission speed to date is defined by the 10 Gigabit Ethernet standards (802.3ae-2002 [1] and 802.3ak-2004 [2]). The IEEE 802.3ae specification defines a transmission method over optical fibre at speeds up to 10 Gbit/s, while the 802.3ak defines a transmission method over Infiniband-type copper cable at similar speeds. The data is transmitted in frames of variable size, with minimum and maximum sizes specified by the standard. The size and content of the frames are defined by the MAC layer. The MAC layer is common to all the Ethernet standards and the IEEE traditionally kept the backward compatibility. The maximum and minimum frame sizes remained virtually unchanged since the adoption of the first 802.3 standard in 1984. The definition of the frame content has been altered slightly along Ethernet s evolution by re-defining existing fields to provide increased functionality. The 10 GE standard defines the first Ethernet that does not support half duplex communications via the Carrier Sense Multiple Access with Collision Detection Algorithm (CSMA/CD). The use of CSMA/CD limited the maximum span of the network to LAN distances in order to detect collisions on the shared transmission medium. By supporting only full-duplex connectivity, the 10 GE frees Ethernet of the CSMA/CD legacy. Henceforth, the distance of a point-to-point connection is only limited by the optical components used for transmitting and propagating the signal. The standard [1] defines a maximum of 40 km reach for a point-to-point connection, using 1550 nm lasers over single mode carrier-grade optical fibre. Two categories of physical layer devices (referred as transceivers later in this paper) are supported by the 10 GE: LAN PHY and WAN PHY. LAN PHY was developed as a linear evolution in the purest Ethernet tradition: faster and further, keeping the Ethernet framing of the outgoing signal. WAN PHY was defined by IEEE with SONET/SDH compatible signalling rate and framing in order to enable the direct attachment of Ethernet to the existing base of long-distance infrastructure. 10 GE LAN PHY - Ethernet over dark fibre Transceivers that allow transmission over distances of 80 km are nowadays available on the market [3], even if the standard officially supports a maximum distance of 40 km. However, due to the chromatic dispersion and attenuation of the light in the optical fibre, it is an open issue whether devices that would allow much longer distances make commercial sense because of the high optical power to be used for compensating the attenuation and also the amount of signal processing to be employed for compensating the dispersion. ESTA IST-2001-33182 4 22/12/2004

A solution available today for extending the reach of a LAN PHY connection is to take advantage of the fact that the 10 GE standard defines the 10GBASE-ER, an optical signal using the 1550nm wavelength. This wavelength is used in traditional telecommunication equipment for carrying the long-distance communications. It is generally accepted that an optical signal centred on the 1550nm wavelength can be transmitted for over 600 km of single mode fibre. Erbium-doped fibre optical amplifiers (EDFA) have to be strategically located on the path in order to compensate for the attenuation. Dispersion-compensating fibre is also required on the path, to compensate for the chromatic dispersion. s BATM T6 Lyngby 1 GE LAN 121 KM 93 KM 38 KM DCF EDFA BATM T6 62 KM 84 KM 75 KM 52 KM LAN 1 GE s Aalborg Figure 1 - Detailed configuration of the 10 GE LAN PHY over dark fibre connection In collaboration with DARENET, the Danish Research Network operator, we demonstrated a 525 km-long LAN PHY point-to-point connection over dark fibre [4]. The configuration is depicted in Figure 1. The XENPAK transceiver employed in the 10 GE ports supports the PMD BIST capabilities as defined by the 802.3ae-2002 standard. This feature allowed for Bit Error Rate (BER) measurements to be performed directly by the switch. We measured a BER rate of 10-14, two orders of magnitude lower than required by the standard. Above a certain number of amplifiers and a certain distance on the optical fibre, the optical signal needs to be fully regenerated before being transmitted towards its destination. The full signal regeneration requires an optical-electro-optical (OEO) transition and consists in three operations: re-amplification, re-shaping and re-timing. The American National Standards Institute (ANSI) and the International Telecommunication Union (ITU) have standardized the SONET/SDH framing and signalling rate for the long-distance communications. Therefore all the regenerators currently deployed in the field can only regenerate SONET/SDH signals. However, in order to follow the traditional 10x increase in data speed, the signalling rate of LAN PHY is 10.31 GBaud due to the 64B/66B coding technique. The data rate and framing make LAN PHY incompatible with the installed wide area network infrastructure, hence the requirement to use a different device for transmitting Ethernet frames at long distances. WAN PHY was introduced by IEEE as a direct gateway from the LAN into the SONET/SDHdominated WAN. ESTA IST-2001-33182 5 22/12/2004

10 GE WAN PHY - Ethernet over long-distance networks WAN PHY has been defined to be compatible with SONET/SDH in terms of signalling rate and encapsulation method. It uses the same transmission rate, 9.95 GBaud, for a payload capacity of 9.58 Gbit/s, using the Synchronous Transport Signal (STS)-192c / Virtual Container (VC)-4-64c frame format. WAN PHY therefore enables the transport of native Ethernet frames over existing long-haul networks. The optical jitter characteristics of WAN PHY lasers were relaxed by comparison to the SONET/SDH standard. The main reason for the relaxed jitter specifications was to provide a less expensive solution by lowering using lower price optoelectronic devices. However, all the WAN PHY ports that we used in our experiments complied with the SONET/SDH optical jitter value. This seems to also be the trend for next-generation WAN PHY based on XFP optics [5]. The XFP multi-source agreement enables the same optical components to be used for SONET/SDH, 10 GE and Fibre Channel. The clock that is used as a reference for WAN PHY transmissions is allowed to be less accurate (20 parts per million ppm instead of 4.6 ppm or a variation of +/- 20 microseconds per second instead of +/- 4.6 microseconds per second). The 20 ppm value is the required timing accuracy of a SONET/SDH connection operating in a special maintenance mode. A SONET/SDH connection operating in production mode is required to have a signal timing accuracy of 4.6 ppm or better. Certain bits of the SONET/SDH management overhead are unused in the WAN PHY frame or have default values. The SONET/SDH ring topology and 50ms restoration time are explicitly excluded from the WAN PHY specification. These are the reasons why the 10 GE standard does not guarantee the strict interoperability of WAN PHY with SONET/SDH equipment. A 10GBASE-W interface is not intended to interoperate directly with interfaces that comply with SONET or SDH standards, or other synchronous networks. Such interoperation would require full conformance to the optical, electrical, and logical requirements specified by SONET or SDH, and is outside the scope and intent of this standard. [1] Instead of a direct interoperability guarantee, IEEE defines a specific piece of equipment (the Ethernet Line Terminating Equipment ELTE) for connecting WAN PHY to SONET/SDH networks. The main tasks of the ELTE are to compensate for differences in clock accuracy and eventually add bits in the frame management overhead. However, no manufacturer has, to date, built an ELTE. The direct attachment of WAN PHY to the existing long-distance infrastructure is the only solution currently available for transmitting native 10 GE frames long haul. Although the WAN PHY standard permits timing and optics that diverge from the SONET/SDH requirements in practice there are not yet any available components that take advantage of the possibilities. Therefore all the existing WAN PHY implementations use SONET compatible laser sources. ESTA IST-2001-33182 6 22/12/2004

The first field deployment experiment of WAN PHY within a trans-european testbed was performed between Geneva and Amsterdam on a wavelength offered by SURFnet [6]. The configuration of the 1700km-long point-to-point connection is presented in Figure 2. Amsterdam DWDM Geneva DWDM 10 GE WAN 10 GE WAN Figure 2 - WAN PHY over between Geneva and Amsterdam The connection was characterized by researchers from CERN using traffic generators and running a mixture of Ethernet frames filled with the Bit Error Rate (BER) test pattern defined in the 10 GE standard [1]. Continuous runs in excess of 98 hours at 100% of the line speed provided results for an equivalent BER of 10-14 [6]. Later, the connection was used together with researchers from the University of Amsterdam for studying long-distance data transfer protocols using servers. These experiments highlighted a failure-free behaviour of the network and limitations in the hardware and software architecture of the computers used as end-nodes. An aggregate rate in excess of 9 Gbit/s was achieved using two servers as traffic sources [6]. A public demonstration of a WAN PHY to connection was organised during the ITU Telecom World 2003 exhibition. Building on our previous experience, the demonstration showcased a transatlantic connection between Geneva and Ottawa via Amsterdam. Two research network operators, CANARIE and SURFnet, provided the long-distance infrastructure by directly interconnecting two of their circuits (figure 3) to form a 10000km point-to-point connection. Ottawa Geneva 10 GE WAN Chicago Amsterdam 10 GE WAN Figure 3 - WAN PHY over between Geneva and Ottawa An equivalent BER of 10-12 was achieved during endurance tests that employed traffic generators for utilising 100% of the available bandwidth. Researchers from University of Carleton and CERN used this connection for studying long-distance data transfer protocols [7]. One single-stream TCP connection between two servers achieved a stable ESTA IST-2001-33182 7 22/12/2004

transmission rate of 5.65 Gbit/s. The rate limitation to an average of 5.65 Gbit/s is still due to the hardware and software of the servers. The ultimate demonstration of WAN PHY over circuits took place in October 2004. A large international collaboration [8] pooled circuits provided by four research network operators (WIDE, Pacific Northwest, CANARIE, SURFnet) to form a testbed for experiments between the University of Tokyo and CERN. Figure 4 shows the network configuration. In addition to the increased number of operator domains involved, different operators used hardware from several manufacturers for building their respective circuits. NI40G Tokyo OME 6500 OME 6500 Geneva NI40G 10 GE WAN Seattle Chicago Amsterdam 10 GE WAN Figure 4 - WAN PHY over between Geneva and Tokyo This connection practically doubled the distance of the Geneva-Ottawa experiment. The new XENPAK modular WAN PHY optical modules were used. As result of server hardware evolution, one single-stream TCP connection reached an average transfer rate of 7.21 Gbit/s. The direct attachment of WAN PHY to existing OC-192/STM-16 SONET/SDH infrastructure is one of the available options for transporting native Ethernet frames over long-distance network infrastructure. The other option currently available is to attach a WAN PHY directly to a SONET/SDH transponder located in Dense Wavelength Division Multiplexing (DWDM) equipment. No OC-192 circuit needs to be defined in this scenario, removing the need for SONET/SDH equipment at the edge of the network and using a DWDM-only meshed network core. On the Geneva-Amsterdam connection provided by SURFnet, we had the opportunity to verify in the field the scenario of connecting WAN PHY directly to transponders in DWDM equipment. The layout of the network is presented in Figure 5. A 91-hours continuous run with traffic generators filling the available bandwidth resulted in the transfer of 365 TB of data without errors [6]. ESTA IST-2001-33182 8 22/12/2004

Amsterdam DWDM Geneva DWDM 10 GE WAN transponder transponder 10 GE WAN Figure 5 - WAN PHY over DWDM between Geneva and Amsterdam We can conclude that WAN PHY provides, for the first time, a native gateway between LANs and existing WANs. The approach we took of using SONET compatible optics to achieve compatibility with the installed infrastructure has since been reinforced with an ITU recommendation, [9] G.8012/Y.1308 which was pre-published in 08/2004. It has adopted SONET clocking (clock accuracy better than 4.6 ppm) both for EoS UNI (Ethernet over SONET User Network Interface) and EoS NNI (Ethernet over SONET Network Node Interface). A WAN PHY conforming to this recommendation will therefore meet the ITU and exceed the IEEE requirements and allow carriers to provide cheaper services for such a connection. It should be reiterated here that this approach is only being recommended for the case of a simple linear extension of an Ethernet segment. It should not be read as necessarily supporting the concept of an Ethernet everywhere solution since it is not scalable enough to match carrier s requirements. Ethernet versus SONET/SDH : OAM over MAN and WAN The operations, administration and maintenance (OAM) features of the today s Ethernet networks are a legacy from the LAN-only times. Ethernets use the same communication channel to transmit both user data and link management information. The management information inherits therefore the asynchronous character of Ethernet and the sending may be delayed in case data frames are in the process of being transmitted. In contrast, SONET/SDH has powerful failure detection, isolation and reporting features via special management channels embedded in every synchronously transmitted frame. It should be noted that the Ethernet OAM functionality applies only to the last point-to-point link of the end-to-end Ethernet path between two devices. The fault detection features of Ethernet are currently restricted to a link down notification, but no indication is provided on whether the failure is due to the transmission medium or to one of the end node interfaces. A Cyclic Redundancy Check (CRC) sequence added to the end of each Ethernet frame allows the detection of bit errors. The quality of the link is monitored in principal by counting the number of frames that are received with CRC errors. The built-in Ethernet link quality monitoring mechanisms hence arguably provide information equivalent to the bit error counters available in the SONET/SDH section management overhead. ESTA IST-2001-33182 9 22/12/2004

Some steps towards introducing operator-oriented features have been made by the introduction of the WAN PHY. Additional features were included in the 802.3ah-2004 standard (Ethernet in the First Mile, [10]) adopted in 2004. The WAN PHY implements a restricted set of the SONET/SDH connection management features. The 802.3ah standard defines critical and non-critical link events that are notified to the peer system at the other end of the link. It also introduces a loopback operating mode for running special diagnostic traffic. In the WAN, it is important that traffic is automatically migrated to an alternate route in case of link failure. The link protection feature of SONET/SDH networks provides fast (50 ms maximum interruption time, enforced by the standard) traffic re-routing in case of a link failure. However, PoS interfaces support only a subset of the SONET/SDH link protection features, the Linear Automatic Protection Switching, which enables rapid switchover to an idle backup link. The Ethernet Link Aggregation standard (802.3ad, [11]) allows for multiple Ethernet point-to-point connections to be bundled together, with the possibility of rapid switchover of traffic in case of physical link failure. However, no switchover time guarantees are enforced by the standard. Being a Layer 2 technology, Link Aggregation allows for using different long-distance transmission technologies for the each of the aggregated links. Also, it enables the simultaneous protection of multiple links using an N + 1 protection scheme. Conclusions The 10 GE standard provides the means for using Ethernet as a link layer technology over MAN and WAN. Today, Ethernet is a full-duplex distance-independent protocol. The span of a point-to-point connection is limited only by the optical components in the case of the LAN PHY. We reached a distance of 525km over dark fibre, using optical amplifiers and dispersion-compensating optical fibre. The WAN PHY can be used for point-to-point connections over the installed long-distance SONET/SDH and DWDM network infrastructure. The longest path we demonstrated reached 18000 km through an circuit provisioned over the backbones of four research network operators. As a point-to-point link layer transmission technology, an Ethernet approach became a viable alternative to SONET/SDH. Significant improvements in terms of scalability, operations, administration, management and bandwidth guarantees are required for Ethernet to be deployed in carrier networks in a topological Ethernet everywhere approach. ESTA IST-2001-33182 10 22/12/2004

Acknowledgements The long-distance connectivity was kindly provided by CANARIE, DARENET, PacificNorthwest, SURFnet, WIDE. We acknowledge the generous contribution in equipment and technical assistance from CERN Openlab, Chelsio Communications, Cisco Systems, Networks, Foundry Networks, Global Crossing, Hewlett-Packard, Intel,, SARA. References [1] IEEE Std. 802.3ae-2002 Amendment: Media Access Control (MAC) Parameters, Physical Layers and Management for 10Gb/s Operation, August 2002. [2] IEEE Std 802.3ak-2004 Carrier Sense Mulitple Access with Collision Detection (CSMA/CD) Access Method and Physical Layer Specifications Amendment: Physical Layer and Management Parameters for 10Gb/s Operation, Type 10GBase-CX4 [3] Optillion TOP-5080 10GBASE-ZR datasheet: http://www.optillion.com/show.php?id=21073 [4] Martin Nordal Petersen, Mikkel Hasse Olesen: 10 Gb/s Non-Regenerated Ethernet Field Trial over 525-km Dark Fiber, proceedings of the OECC-COIN2004 conference. http://www.batm.de/inc/t6/wp/oecc-coin%202004.pdf [5] The XFP Multisource Agreement http://www.xfpmsa.org [6] Catalin Meirosu, Piotr Golonka, Andreas Hirstius, Stefan Stancu, Bob Dobinson, Erik Radius, Antony Antony, Freek Dijkstra, Johan Blom, Cees de Laat: Native 10 Gigabit Ethernet Experiments over Long Distances, accepted for publication in Elsevier s Future Generation Computer Systems Journal [7] Bob Dobinson, René Hatem, Wade Hong, Piotr Golonka, Catalin Meirosu, Erik Radius and Bill St. Arnaud: Transatlantic Native 10 Gigabit Ethernet Experiments: Connecting Geneva to Ottawa, proceedings of the HSMNC 03 conference, LNCS 1960, Springer-Verlag, 2004 [8] http://mail.canarie.ca/mlists/news2004/0102.html [9] http://www.itu.int/rec/recommendation.asp?type=products&lang=e&parent=t-rec-g [10] http://www.ieee802.org/3/efm/ [11] http://www.ieee802.org/3/ad/index.html ESTA IST-2001-33182 11 22/12/2004