Initial Implementations of Point-to-Point Ethernet over SONET/SDH Transport

Transcription

1 ETHERNET TRANSPORT OVER WIDE AREA NETWORKS Initial Implementations of Point-to-Point Ethernet over SONET/SDH Transport Vish Ramamurti, John Siwko, George Young, and Mike Pepe, SBC Laboratories, Inc. ABSTRACT There is considerable interest in using Ethernet over SONET/SDH (EoS) for Ethernet transport in a wide area network. EoS leverages the existing SONET/SDH infrastructure to provide efficient Ethernet transport with excellent OAM capabilities. In our initial evaluations and deployments of EoS, we have encountered some issues, as is common with any new technology. These issues include non-transparent transport of Ethernet information, lack of carrier-class performance monitoring and fault propagation, and non-standardized implementation of basic concepts. Vendors, service providers, and standards bodies should work together to resolve these remaining issues and thus bring forth the full potential of EoS. INTRODUCTION Ethernet technology has evolved to the point where service providers are beginning to deploy this technology in the wide area network. Relying only on native Ethernet for layer 1 transport in the wide area network would mean building an expensive overlay network parallel to the extensive synchronous optical network/synchronous digital hierarchy (SONET/SDH) transport infrastructure carriers have already built. Transporting Ethernet over SONET/SDH (EoS) would make use of existing carrier facilities and operations, administration, and maintenance (OAM) features, and also provide a way to transport Ethernet frames in a bandwidth-efficient manner. SONET/SDH, having been developed as the backbone optical technology for carriage of high-bandwidth voice and data traffic, is characterized by high availability and reliability. Customers naturally expect highly reliable service when Ethernet is transported over SONET/SDH. When mapping Ethernet into SONET/SDH, some implementations have taken the Ethernet switch in a card approach where the transport network element interface card would perform traditional layer 2 functions. This approach presents network management problems since service providers and customers typically use different tools and processes to manage their layer 2 and layer 1 networks. This article focuses on layer 1 transport of Ethernet over SONET (the transport network element tries to be transparent to layer 2) and highlights some of the issues we have found while evaluating early implementations of hardware used to provide the service. This article will concentrate on issues associated with mapping Fast Ethernet (100 Mb/s) and Gigabit Ethernet into SONET. Extensions to other Ethernet rates and mapping into SDH can easily be made. ETHERNET AND SONET SPEEDS When Ethernet port speeds are referred to as 100 Mb/s or 1 Gb/s, these rates refer to the medium access control (MAC) sublayer signaling rates (Fig. 1). Due to physical layer block encoding (e.g., 4B/5B for 100Base-TX or 8B/10B encoding for 1000Base-X), the resulting physical layer speeds are 125 Mb/s and 1.25 Gb/s respectively. The standard time-division multiplexed (TDM) rates SONET systems transport are payload capacities of Mb/s (DS1), Mb/s (VT1.5), Mb/s (DS3), Mb/s (STS- 1), Mb/s (STS-3c), Mb/s (STS- 12c), Mb/s (STS-48c), and Mb/s (STS-192c). As observed, Ethernet physical layer rates are different from the TDM rates. When attempting to carry Ethernet over SONET, one approach could be to map Ethernet into a larger SONET payload capacity. However, such an approach would waste a large portion of the customerpaid SONET transport bandwidth (e.g., nearly 35 percent when 100 Mb/s Ethernet is mapped into an STS-3c and nearly 60 percent when 1 Gb/s Ethernet is mapped into an STS-48c). Virtual concatenation [1] will alleviate this problem to a large extent where individual VT1.5, STS-1, or STS-3c time slots can be logically grouped together to obtain a SONET bandwidth that closely matches the Ethernet rate. For example, seven STS-3cs can be logically combined as an STS-3c-7v virtually concatenated path to carry a Gigabit Ethernet client. In the case of 4B/5B or 8B/10B encoding, the /04/$ IEEE

2 IFS IPG PA Ethernet frame SFD DA SA L/T PYLD FCS IPG PA SFD DA SA IEEE Ethernet MAC frame 1000 Mb/s 8B10B 8B10B 8B10B 8B10B 8B10B 8B10B 8B10B 8B10B 8B10B 8B10B 8B10B 8B10B 8B10B 1250 Mb/s Ethernet frame: Useful portion of the IEEE Ethernet frame in a full-duplex environment DA: Destination address 6 bytes SA: Source address 6 bytes L/T: Length/Type field 2 bytes PYLD: Payload bytes (up to 9000 plus bytes for jumbo frames) FCS: Frame check sequence 4 bytes IPG: Interpacket gap (at least 12 bytes) PA and SFD: Preamble (7 bytes) and start of frame delimiter (1 byte) 8 bytes IFS: Interframe spacing (at least 20 bytes) 8B10B: 10-bit code groups IFS Ethernet frame The physical layer rates of 125 Mb/s in the case of a 100M Ethernet port and 1.25 Gb/s in the case of a Gigabit Ethernet port are the physical layer speeds whether or not Ethernet frames are actually sent at any time. Figure 1. Ethernet layers 2 and 1. SONET overhead Payload capacity GFPH Ethernet frame GFPH Ethernet frame... GFPH SONET frame Ethernet frame GFPH: GFP header 8 bytes Ethernet frame: bytes (9000+ bytes if carrying jumbo frames) SONET: Contiguously and virtually concatenated payloads Figure 2. Ethernet over SONET mapping using GFP-F. Ethernet physical layer adds 25 percent overhead to the layer 2 Ethernet stream. This overhead is in addition to the interframe spacing (IFS), defined as the interpacket gap, preamble, and start of frame delimiter (Fig. 1). The IFS is layer 2 overhead between adjacent Ethernet frames, and carries no useful information in a full duplex link. Therefore, considerable bandwidth savings can be achieved if just the Ethernet frames are mapped into SONET (Fig. 2), and the IFS and physical layer are discarded at the near end and reproduced at the far end. This bandwidth savings enables a customer who does not require line rate Ethernet to pay for only the reduced SONET bandwidth required for the actual transport of Ethernet frames. This is the approach taken with Frame Mapped Generic Framing Procedure (GFP-F) adaptation of Ethernet client signals [2]. A service provider can offer SONET bandwidth options in basic TDM increments (VT1.5, STS-1, or STS-3c), and the customer can buy just the bandwidth that best meets their needs independent of the Ethernet port speed. The impact of a frame-mapped approach, as we will see in detail later, is that certain Ethernet mechanisms cannot be transported transparently over the SONET network. While the GFP standard allows for both frame mapping (GFP-F) and transparent mapping (GFP-T) for EoS, the GFP-T standard is continuing to undergo modifications to efficiently transport line rate and subrate Ethernet [2]. Service standards that preserve all the essential and Ethernet content (Ethernet frames including control frames, Spanning Tree BPDUs, Slow Protocol frames, etc.), while still removing unnecessary overhead such as physical layer encoding and IFS, are desirable. International Telecommunication Union Telecommunication Standardization Sector (ITU-T) Study Group 15 [3] and the Metro Ethernet Forum (MEF) [4] are taking steps toward creating such standards. SUBRATE ETHERNET TRANSPORT DEFINING SUBRATE ETHERNET The physical layer rates of 125 Mb/s in the case of a 100 Mb Ethernet port and 1.25 Gb/s in the case of a Gigabit Ethernet port are the physical layer speeds whether or not Ethernet frames are actually sent at any time. When an Ethernet port is referred to as having a throughput of 1 Gb/s, this corresponds to a stream of Ethernet frames separated by the minimum 20 bytes of IFS. When 1 Gb/s line rate Ethernet is transmitted using only 64-byte frames, the Ethernet data rate is approximately only 760 Mb/s; the remaining 240 Mb/s is made up of the IFS. When the same 1 Gb/s line rate Ethernet is transmitted using only 1518-byte frames, the Ethernet data rate is approximately 980 Mb/s, with the remaining 20 Mb/s made up of IFS. However, we have 65

3 In any definition of data rate, the time period of measurement plays an important role in the measurement of the throughput. In the case of Ethernet, the time period of measurement becomes even more important. 20 bytes 20 bytes Ethernet frame Minimum IFS Ethernet frame Minimum IFS... Figure 3. A 100 Mb/s Ethernet stream and two different 50 Mb/s Ethernet streams. observed that many test equipment and router vendors measure the Ethernet rate as 1 Gb/s in both cases. When an Ethernet port sends traffic at less than the line rate, the IFS must be more than the minimum 20 bytes between at least some of the Ethernet frames. When a router or a test equipment reports an Ethernet rate of 500 Mb/s out of a port, this means that some combination of Ethernet frames and IFS within the traffic stream constitute this rate. However, given that Ethernet frames can range from 64 to 1518 bytes (and even larger with jumbo frames), and IFS can be any value greater than or equal to 20 bytes, it is not exactly clear what the 500 Mb/s rate refers to. The Ethernet rate seems to be the Ethernet port speed times the ratio of the bytes corresponding to Ethernet frame size plus 20 bytes (minimum IFS) for each Ethernet frame within a time period of measurement and all the bytes present within the same time period of measurement. Definitions for Ethernet service traffic rate parameters need to be standardized to avoid conflicts caused by different interpretations of Ethernet rate. MEF drafts [6] are being developed to define bandwidth profiles based on Ethernet data rates (no IFS) and burst sizes. While this is a good approach, the Ethernet data rates (no IFS) used in these profiles are different from what is currently referred to as Ethernet rates (includes IFS). Vendors, service providers, and customers should all come to the same understanding when standards are established. DROPPING OF ETHERNET FRAMES In any definition of data rate, the time period of measurement plays an important role in the measurement of the throughput. In the case of Ethernet, with varying frame lengths and IFS, and the fact that Ethernet frames always leave an Ethernet port at the port speed, the time period of measurement becomes even more important. In Fig. 3, two different 50 Mb/s streams are shown. In both cases, the time period of measurement is 10 ms. However, if the time period of measurement was 5 ms, the first stream would still have a rate of 50 Mb/s for Line rate 100Mb/s Ethernet stream 10 ms Two different 50 Mb/s Ethernet streams Ethernet frame Ethernet frame IFS Ethernet frame IFS... ETH FR 20 bytes Min IFS bytes 104 bytes ETH FR 20 bytes Min IFS ETH FR 10 ms 5 ms 5 ms IFS Ethernet both the 5 ms periods within the 10 ms window. The second stream, however, would have a rate of 100 Mb/s for the first 5 ms and zero Mb/s for the second 5 ms period. When this second 50 Mb/s stream is mapped into a SONET STS-1, if the buffer on the ingress EoS card is not large enough to hold the excess frames during the first 5 ms window, the resulting throughput on the SONET network will be less than 50 Mb/s. So, even though the customer is technically sending data at 50 Mb/s (with a 10 ms time period of measurement), and the SONET payload is expected to handle this speed (after discarding the IFS), the resulting throughput could be as bad as 25 Mb/s (50 Mb/s 5 ms/10 ms) through the SONET network if the ingress buffer cannot hold many frames. No generally accepted time period of measurement or standards for buffer sizes exist today. Once standardized, the policing mechanism being defined by ITU-T SG15 [3] and MEF [4], where bandwidth profile parameters are expressed in terms of committed information rate, excess information rate, committed burst size, and excess burst size, together with a token bucket rate algorithm, could be applied in the design concepts for traffic management. ETHERNET RATE LIMITING One solution to keep the Ethernet data rate lower than the SONET payload rate is for the customer to rate limit traffic at the egress port of their switch or router. However, due to the variability of Ethernet frame sizes and IFS, and due to jitter and latency concerns, many switch/ router implementations have a less than perfect rate limiting scheme. In tests conducted in our laboratory using different switch/router equipment, we have observed that current rate limiting schemes are very frame-size dependent and, for a fixed desired output Ethernet rate, a large variation is observed (sometimes greater than 10 percent) in the actual output rate between the rates corresponding to the smallest (64-byte) and largest (1518-byte) frame sizes. Once standards are available that precisely define Ethernet rates, stricter bounds are needed on Ethernet rate limiting performance. 66

4 FLOW CONTROL The IEEE Ethernet standard [5] defines a flow control mechanism that lets an Ethernet receiver port send PAUSE frames to the transmitter port to throttle traffic. These PAUSE frames contain a time unit parameter that indicates to the transmitting port how long to pause. A PAUSE frame corresponding to zero time units can be sent to request resumption of transmission. From the discussion earlier, it is clear that flow control would be useful for a service provider offering an EoS service. Note that even though frames may not get dropped in the service provider network when flow control is enabled, frames could be dropped somewhere upstream unless either enough buffer space is available along the upstream path or the flow control mechanism is implemented on every link all the way to the traffic source. Ethernet flow control is a MAC layer 2 mechanism that works only between two directly connected layer 2 Ethernet ports. Although point-to-point private line EoS service is a layer 1 service, network equipment (NE) vendor implementations may process MAC layer 2 protocols before mapping frames into SONET. Thus, PAUSE frames sent by the customer s port may be terminated by the MAC layer at the Ethernet port on a service provider s EoS card. However, some customers may want their layer 2 PAUSE frames to be transported all the way to the farend layer 2 customer port. Other customers may prefer a flow control feature being offered by the service provider in order to prevent frames from being dropped in the transport service network. Therefore, it would be useful to have an option to enable flow control on the ingress port of an EoS card, or to disable flow control and permit transparent passage of customer PAUSE frames through the SONET network. A standard to allow for the transparent passage of such layer 2 PAUSE frames through the SONET network is desirable. This standard could even consider options such as permitting multiple sources of PAUSE frames on a transport link. These options for flow control are under consideration for EoS transport services in ITU-T SG15. NONTRANSPARENT TRANSPORT OF ETHERNET AUTO-NEGOTIATION Auto-negotiation (AN) normally works on a link connecting two Ethernet devices to allow them to settle on the speed of connection, type of connection (full or half duplex), ability to send and receive PAUSE frames, and, optionally, to convey remote fault indication information. AN is part of the physical coding sublayer within the IEEE Ethernet protocol [5]. Although an EoS layer 1 service is expected to act like a piece of wire transporting content from a customer port on one side of the network to a customer port on the other side, with respect to AN the service does not always work this way. With frame-based EoS adaptation, vendor NE implementations go up the Ethernet stack to the MAC layer before mapping frames into SONET. Options to transparently transport AN information to the far-end equipment may be desirable. Besides providing more transparent service, this could help the service provider avoid AN-related problems. Today, with AN set to ON on the transport NE, the following have been observed: Interoperability issues occur with certain customer equipment. The Ethernet link may be negotiated to a speed lower than the customer ordered speed. With AN set to OFF on the transport NE, the following have been observed: Ethernet ports on some commercially available switches/routers work only with AN turned ON. If the customer has AN turned ON at his/her end of a Gigabit Ethernet connection and AN is set to OFF at the service provider NE, the Gigabit Ethernet link would be UP on the service provider side. No trouble will be reported anywhere on the service provider network, while in reality the link will be DOWN (inoperable) on the customer side. Certain Ethernet ports on commercially available switches/routers do not operate when a link is disconnected and reconnected on a port with AN OFF, unless the AN option is cycled from OFF to ON and back to OFF. Certain Ethernet switches and routers tie flow control with AN. Flow control will work only if AN is turned ON on these devices. Keeping AN ON helps to create bidirectional link failures even if a fault occurs only in one direction. Layer 2 protocols such as Spanning Tree Protocol (STP) [6] and Link Aggregation Control Protocol (LACP) [5] work efficiently in redirecting traffic only when both directions of a link fail. JUMBO FRAMES AND OTHER PROPRIETARY ETHERNET FRAMES According to IEEE standards [5], Ethernet frames can be between 64 and 1518 bytes long without virtual LAN (VLAN) tagging and up to 1522 bytes long with VLAN tagging. Although these are the standard frame sizes, many commercially available Ethernet switches/routers support frame sizes exceeding 9000 bytes. Frames exceeding the size of standard Ethernet frames are generally called jumbo frames in the industry. Private line EoS service, as a layer 1 transport service, is expected to carry layer 2 Ethernet frames of all sizes. However, many implementations of frame-based EoS adaptation employ off-the-shelf Ethernet components that consider frames larger than 1522 bytes as frames with errors, and drop them. Dropping of other proprietary Ethernet frames has also been observed. SPANNING TREE BPDUS, MAC CONTROL FRAMES, AND SLOW PROTOCOL FRAMES Several layer 2 mechanisms, such as STP and LACP, may transport special Ethernet frames on a link. By current standards, these frames are ter- Auto-negotiation normally works on a link connecting two Ethernet devices to allow them to settle on the speed of connection, type of connection, ability to send and receive pause frames and, optionally, to convey remote fault indication information. 67

5 SONET has its own clocking source, thus the Ethernet source timing is terminated at the Ethernet/SONET interface. A separate Ethernet signal must then be generated at the far end by a second local Ethernet oscillator. Clock 1 Tx 1 Source Figure 4. Ethernet timing in the EoS network. minated by a MAC layer, regardless of whether or not the terminating entity implements the protocol. Frame-based EoS implementations typically terminate the MAC layer at the ingress to the SONET network and thus frequently terminate these frames. An option, as envisioned in [3, 4], to transparently transport these frames while providing a layer 1 EoS service is needed. FRAME-SIZE-DEPENDENT THROUGHPUT If the incoming Ethernet stream has enough frames to fill the SONET payload, the Ethernet data rate combined with the EoS mapping overhead (Fig. 2) should be very close to the SONET payload rate. However, when the frame sizes are small, certain frame-based EoS implementations find it difficult to perform all the processing associated with extracting Ethernet frames from native Ethernet traffic and mapping them into the SONET payload at a speed close to the SONET payload rate. The throughput associated with a layer 1 EoS service should not be layer 2 frame-size-dependent. CLOCKING Clock 2 Ethernet frame loss can occur when the Ethernet source generates traffic at full line rate (100 percent utilization), even when the SONET bandwidth greatly exceeds the required Ethernet bandwidth. Ethernet interfaces have local oscillators on the transmit side. Even if the transmitter clock is sloppy, whatever traffic is sent to the receiver buffer will be removed from the buffer at the same rate it is transmitted, because the receiver derives its timing from the received signal. SONET has its own clocking source; thus, the Ethernet source timing is terminated at the Ethernet/SONET interface. A separate Ethernet signal must then be generated at the far end by a second local Ethernet oscillator (Fig. 4). Therefore, there is no clock transparency from the Ethernet source to the Ethernet destination, and timing between transmitter 3 and the destination is completely independent from clock 1. Assume that the SONET bandwidth is greater than the Ethernet bandwidth, so SONET will not be a bottleneck. In this case, SONET is able to remove frames from buffer 1 as fast as the Ethernet source sends frames to that buffer. SONET stuffing is added as necessary, and then removed when frames are placed at buffer 2. Therefore, the SONET portion of Fig. 4 can be removed, and the figure simplified to Fig. 5. In Fig. 5, the Ethernet source sends frames to SONET Clock 3 Buffer 1 Tx 2 Buffer 2 Tx 3 Buffer 3 Near-end NE Far-end NE Destination buffer 2 at a rate determined by clock 1. Frames are removed from buffer 2 and sent to the destination at a rate determined by clock 3. Assume that frames are being sent at full line rate (100 percent utilization). If clock 3 is faster than clock 1, buffer 2 will be drained by clock 3 s transmitter faster than clock 1 s transmitter can fill it. This situation is not a problem, since the transmitter can insert additional idles whenever the buffer becomes empty. The problem occurs if clock 3 is slower than clock 1. In that case, if the source is sending frames at 100 percent utilization, clock 1 will transmit frames to buffer 2 faster than clock 3 will drain that buffer. Eventually, buffer 2 would fill and frames would be lost. Ethernet standards [5] allow a clock accuracy of ±100 ppm. In the worst case, clock 1 could be +100 ppm and clock 3 at 100 ppm, for a total timing difference of 200 ppm. In a 100Base-T connection, this timing difference would result in transmitter 1 sending 20,000 b/s more than transmitter 3 would send. Once buffer 2 was filled, this excess bit rate would result in the loss of nearly 30 frames/s for 64-byte frames on a 100BASE-T connection. For a GigE connection, the excess bit rate would be 200,000 b/s; for 64- byte frames nearly 300 frames/s would be lost. The same issue can occur with Ethernet switches, since they also block clock transparency between the source and destination. But the problem is less likely to occur in a switch, since their buffer sizes are substantially larger than the buffer sizes in an NE, and thus can withstand a full line rate burst for much longer. One possible solution would be for transmitter 3 to transmit a smaller than standard interpacket gap whenever buffer 2 exceeded a certain threshold. A smaller than standard interpacket gap is permitted by IEEE under certain circumstances in order to deal with clocking issues. However, there is a risk that the destination receiver might not properly handle an interpacket gap smaller than 12 bytes, and treat it as an error. Another way to mitigate the problem would be to substantially increase buffer sizes in NEs. Increasing buffer sizes would allow traffic sources to burst longer at full line rate, but would also increase latency and jitter. A third possible solution would be to use positively biased clocks (with tighter tolerance) in NEs (clock 3). With a positively biased clock, the probability that the source clock (clock 1) is faster than the NE clock (clock 3) is decreased. Finally, if standards are changed to permit multiple PAUSE sources on a link, the far-end NE could use flow control to slow down the source. 68

6 PERFORMANCE MONITORING Performance monitoring (PM) is an important capability to determine the sources of network problems, especially for networks that cover a large geographic area. PM becomes particularly important when a connection passes through multiple distinct network management domains. For example, an EoS connection might begin at a customer device managed by the customer, then pass through carrier equipment managed by the carrier, and finally terminate at a second customer device managed by the customer. Each network management domain will have its own personnel, who typically cannot access equipment outside of their management domain. PM plays a vital role in isolating a problem to a specific network management domain. PM is also important in determining credits for services with service level agreements (SLAs). In a truly transparent service, all valid service frames should be able to pass through the EoS connection. Unfortunately, most frame-based Ethernet mapping NEs do not allow such transparency. Instead, they set themselves up as censors, judging which customer frames are worthy of being passed through and which are to be blocked. An implementation agreement that specifies a consistent treatment of ingress frames in a private line EoS service is desired. The types of frames and frame errors (violations of standards) we have seen blocked by various NEs include the following: control frames Spanning Tree Protocol BPDUs Frames whose destination address is a reserved multicast address control frames LACP frames Any Slow Protocol frame (type field = 8809) PAUSE frames Any MAC control frame (type field = 8808) Frames blocked by current standards Short frames with valid FCS (frame size < ) Long frames with valid FCS (frame size > 1518 bytes) FCS errored frames Frames whose Length field does not match the size of the frame payload (Data and Pad fields) Any frame, once the receiver buffer is full With so many reasons transmitted frames might not reach their destination, it becomes important to determine the underlying cause for each missing frame. The general guiding principle for PM measures is: An NE should have a separate PM measure for every reason a frame is not passed on toward the far-end customer. Note that it is not sufficient to just have one PM measure called Dropped_Frames or, as is found in many NEs, just a couple of global error PM measures. A finer granularity of PM measures is needed to troubleshoot the problem and determine responsibility. Additional PM measures are needed to track the flow of frames over the EoS connection. There should be PM measures that count the number of frames received from the Ethernet/ Clock 1 Tx 1 Source Figure 5. Simplified Ethernet timing. Clock 3 GigE interface, the number of Ethernet frames transmitted across the SONET interface, the number of Ethernet frames received from the SONET interface, and the number of frames transmitted across the Ethernet/GigE interface. These PM measures can help determine the location of missing frames, and help determine whether a customer is receiving his/her contracted throughput. If flow control is enabled on an NE, there should be a PM measure for the number of PAUSE frames generated by the NE that are transmitted across the Ethernet/GigE link. Similarly, if an NE generates any other type of frame (LACP, spanning tree, proprietary slow protocol, MAC control frame, etc.), there should be individual PM measures for each type of generated frame that count the numbers of those frames generated and transmitted. While NEs support binning of SONET PM measures, they generally have not implemented PM binning of their Ethernet PM measures. Ethernet PM measures should be binned for the same reasons SONET PM measures are binned: to track performance over time and isolate a problem to a particular time period. As a minimum, for integration with SONET PM binning there should be a 15-min counter and a 1-day counter for each PM measure. FAULT PROPAGATION When two Ethernet devices are directly connected, the receiver can directly detect any fault that occurs on the link. In an EoS connection, however, a fault might occur at the near-end Ethernet link, or in the SONET transport network (Fig. 6). In such a case, the far-end NE Ethernet transmitter (transmitter 3 in the figure) could simply continue to transmit idles to the destination. The destination would then be unable to distinguish whether a fault has occurred in the connection, or the source simply has nothing to transmit. Eventually upper-layer protocols would time out, but that can take several minutes. It should also be noted that since an EoS connection typically passes through several network management domains, a fault that occurs inside one domain would not be reported to other domains, and personnel in one domain cannot access the status of devices in another domain. For all these reasons, an EoS service needs to have some mechanism to notify the connection destination whenever a fault is detected, regardless of where the fault occurs. Such a fault propagation scheme requires two steps that must be implemented and preferably standardized: Carriage of fault notification across the SONET network Buffer 2 Tx 3 Buffer 3 Destination 69

7 In keeping with the principle of transparent transport, the EoS network should NOT itself cause a fault in the reverse path (i.e., notify the source.) Rather, the EoS network should only propagate the fault to the destination. Tx 1 Source Buffer 1 Tx 2 Near-end NE Figure 6. Fault propagation: Tx 3 transmits idles to the destination when a fault occurs in the near-end link or in SONET. Carriage of fault notification across the Ethernet/GigE link Several alternatives exist for the first step. GFP defines a method of carrying fault information [2]. If GFP is not used, the SONET path indicators PDI-P and AIS-P can be used for notification across the SONET network of faults detected on the Ethernet link and SONET network, respectively [1]. For the second step, the NE must insert a fault condition into the far-end Ethernet/GigE link. The inserted fault condition should be understood by the wide variety of possible Ethernet devices at the destination. One possibility is to cause a loss of signal (LOS) by, for example, turning off the transmitter laser or configuring the electrical signal for high impedance. In keeping with the principle of transparent transport, the EoS network should not itself cause a fault in the reverse path (i.e., notify the source.) Rather, the EoS network should only propagate the fault to the destination. In many cases, a bidirectional link is desired, since several upper layer recovery methods will work or converge faster only if both directions of a link are down. If the destination wishes to shut down the reverse link for this reason, the destination will insert a fault condition into the reverse direction, which can then be propagated to the original source. On the other hand, if unidirectional links are acceptable, the destination will continue to use the reverse link. Care must be taken in a fault propagation scheme to avoid potential lockups. For example, consider a simplistic scheme where detection of a fault at an NE Ethernet receiver always causes insertion of PDI-P, and receipt of PDI-P always causes an NE Ethernet transmitter to turn its signal off. If LOS ever occurs at both sides simultaneously, both NEs will suppress each other s Ethernet transmitters through their mutual insertion of PDI-P. One method to avoid such a lockup in this case would be for an NE to insert PDI-P in response to a detected fault only if it is not currently receiving PDI-P for that path. SONET Buffer 2 Tx 3 Buffer 3 Far-end NE CONCLUSIONS Transporting Ethernet over SONET/SDH leverages the installed base of transport equipment in carrier networks while at the same time taking advantage of OAM features and bandwidth efficiencies associated with carrying Ethernet frames over SONET/SDH. This article highlights some of the issues we have encountered during our initial evaluations and deployments of EoS. These issues include non-transparent transport of Ethernet information, lack of carrier-class performance monitoring and fault propagation, and non-standardized implementation of basic concepts. Vendors, service providers, and standards bodies should work together to resolve these remaining issues and thus bring forth the full potential of EoS transport services on behalf of customers. REFERENCES [1] ANSI T , Synchronous Optical Network (SONET) Basic Description Including Multiplex Structure, Rates and Formats. [2] ITU-T Rec. G.7041, Generic Framing Procedure (GFP), Dec [3] ITU-T Draft Rec. G.8010, Architecture of Ethernet Layer Networks, Oct. 2003; Draft Rec. G.8011, Ethernet over Transport Ethernet Services Framework, proposed Apr. 2004; Draft Rec. G , Ethernet Private Line Service, proposed Apr [4] MEF Tech. Spec., Ethernet Services Model Phase 1, Oct. 2003; MEF Tech. Spec., Ethernet Services Definitions Phase 1, under development; MEF Tech. Spec., Traffic Management Specification Phase 1, under development. [5] IEEE 802.3, IEEE Standard for Information Technology Telecommunications and Information Exchange Between Systems Local and Metropolitan Area Networks Specific Requirements Part 3: Carrier Sense Multiple Access with Collision Detection (CSMA/CD) Access Method and Physical Layer Specifications, [6] IEEE 802.1D, IEEE Standard for Information Technology Telecommunications and Information Exchange Between Systems IEEE Standard for Local and Metropolitan Area Networks Common Specifications Media Access Control (MAC) Bridges, BIOGRAPHIES Destination VISH RAMAMURTI (vramamurti@labs.sbc.com) is a principal member of technical staff at SBC Laboratories, Inc. He is currently responsible for evaluating layer two technologies such as Ethernet, Fibre Channel, and RPR, and their relationship to the transport network. His work involves aiding the transition of the technology from the assessment phase to field deployment. He obtained his Ph.D. in computer engineering from the University of Texas at Austin in JOHN SIWKO has been a senior member of technical staff at SBC Laboratories, Inc. since His primary area of responsibility has been Ethernet as it relates to the transport network. He obtained his Ph.D. in electrical engineering from the University of California at Los Angeles in He has worked for Trilogy Software and Bellcore (now Telcordia). He obtained his Master s and Bachelor s degrees in electrical engineering from Columbia University, New York, and the Massachusetts Institute of Technology, respectively. GEORGE YOUNG is a principal member of technical staff, responsible for broadband transport standards and technology planning, with SBC Laboratories, Inc. He received a B.S.E.E. degree from the Illinois Institute of Technology in He has been planning and developing digital transport facility systems with Ameritech, AT&T, Bellcore (now Telcordia) and SBC since 1972, and currently represents SBC in the ATIS T1X1 Committee, IEEE RPR Working Group, ITU- T Study Group 15, and Optical Internetworking Forum. MIKE PEPE is the executive director of Broadband Transport at SBC Laboratories, Inc. His group is responsible for approval for use testing of transport products including DWDM, next-generation SONET, and digital crossconnect systems for SBC. He has 25 years of telecommunications experience, 15 spent in optical networking. He has significant expertise in digital switching, optical switching, SONET, and DWDM. He received a B.S.E.E. from the University of Texas at Austin. 70