PREPARING FOR FUTURE STRUCTURED CABLING REQUIREMENTS IN THE DATA CENTER: 40/100 GIGABIT ETHERNET AND BEYOND

PREPARING FOR FUTURE STRUCTURED CABLING REQUIREMENTS IN THE DATA CENTER: 40/100 GIGABIT ETHERNET AND BEYOND Cindy Montstream, RCDD/NTS, EE, CPLP Director of elearning and Standards DATA COMMUNICATIONS Legrand, North America

ABSTRACT Articles have been written and presentations have been given about migrating to 40 and 100 Gigabit Ethernet for a few years now. It is often confusing with the many different options when designing the structured cabling system. Many times, manufacturers provide solutions that complement their product offering but don t provide an understanding of what to consider. The best approach is to understand the foundation and key considerations so the correct questions can be asked that will lead to the right solution. This white paper will educate you on the things you need to understand and consider. It will also look at upcoming technology. There are many different cable infrastructure options from manufacturers; some not as good as others. By understanding the key points to consider, you will be better prepared to make the right decision for your data center infrastructure. You will also be better prepared to understand how to support new technology as it evolves. INTRODUCTION The increase in data being used, big data demands, video, social networking, and network changes like virtualization, consolidation and Fiber Channel over Ethernet (FCoE) are all driving the need for 10 gigabits per second (Gbps) data rates. Data centers will be migrating to 40 and 100 Gbps. In fact, data centers already have to support 40 Gbps to now aggregate 10 Gbps links from the edge/access switches to the core routers/servers. The Institute of Electrical and Electronic Engineers 802.3 Working Group (IEEE 802.3) develops standards for Ethernet based Local Area Networks (LANs). Figure 1, taken from the IEEE 802.3 Higher Speed Study Group Tutorial shows that server speeds are doubling every 24 months and core networking equipment is doubling every 18 months. Figure 1: Rate Mb/s 1,000,000 100,000 10,000 Core Networking Doubling 18 mos 1,000 Gigabit Ethernet Server I/O Doubling 24 mos 100 1995 2000 2005 2010 2015 2020 Date Source: Original figure from IEEE802.3 Higher Speed Study Group Tutorial (November 2007); updated and presented for IEEE802.3 Next Generation BASE-T CFI Consensus Building Presentation, July 2012 Plenary. 40 and 100 Gigabit Ethernet (GbE) is here. Many major manufacturers provide equipment that allow a transition to 40/100 GbE or support 40/100 GbE in new installations. Extreme Networks, CISCO and Arista have 40 GbE equipment and CISCO launched modules that support 40 and 100 GbE at CISCO Live in the summer of 2013. Arista has a data center switch that is scalable to 40 and 100 GbE. Brocade, Juniper and other manufacturers have 100 GbE equipment. We will begin by taking a look at the standards that define the minimum requirements for Ethernet application standards and telecommunications structured cabling. The Institute of Electrical and Electronic Engineers (IEEE) organization develops application standards and the Telecommunications Industry Association (TIA) develops telecommunication structured cabling standards. This white paper will discuss the following: IEEE and TIA Standards 40/100 Gbps Transmission 40/100 Gigabit Ethernet Connectivity and Cabling Fiber Considerations When Migrating to 40/100 Gigabit Ethernet What Role Will Copper Play? Other Considerations What s Coming? 100 Gigabit Ethernet 40 Gigabit Ethernet ~3% of server ports in 2015 10 Gigabit Ethernet ~9% of server ports in 2010 10, 40 and 100 Gigabit Ethernet Deployment Examples Conclusions 2

IEEE AND TIA STANDARDS IEEE 802.3 is a working group within the Institute of Electrical and Electronics Engineers (IEEE) professional organization. It is also a collection of IEEE standards produced by the working group defining the physical layer and the media access control layer (MAC) of wired Ethernet (There are other groups responsible for wireless, etc). These standards define technology, generally specific to local area networks, with some wide area network applications. The standards define the physical connections between nodes and/or infrastructure devices like hubs, switches, routers, etc. and various types of copper or fiber cable. The Telecommunications Industry Association (TIA) defines the performance for structured cabling at the component level, link level and channel level to support an application over the distance specified. Sometimes a new category of performance must be defined to support a new application. The purpose of standards is to provide the minimum requirements to guarantee applications will function properly with equipment from any manufacturer. Using TIA structured cabling assures interoperability between components from different manufacturers. 40/100 Gbps TRANSMISSION The IEEE 802.3ba standard, defining 40 Gbps and 100 Gbps Ethernet transmission primarily over optical fiber, was ratified in June 2010. This was based on the IEEE 802.3ae standard defining 10 GbE transmission ratified in 2002, which made development of the standard much easier and faster. IEEE did not develop a completely new transmission definition for 40 Gbps and 100 Gbps transmission over two fibers like 10 GbE. Both 40 GbE and 100 GbE are based on using parallel transmission paths transmitting 10 Gbps; 40 GbE requires four lanes and 100 GbE requires ten lanes for both transmitting and receiving. This was a departure from previous fiber systems. Every application that IEEE802.3 defines has a Physical Medium Dependent (PMD) sublayer as part of the specification. The PMD sublayer defines details of transmission and reception of individual bits on a physical medium. Table 1 lists IEEE s objectives for supporting 40 Gbps Ethernet over specific media. Table 1: IEEE Objectives for 40 Gigabit Ethernet Objective Resulting PMD Description of PMD 100m on OM3 1 MMF 2 (850nm) 150m on OM4 3 MMF 2 (850nm) 10km on SMF 4 (1310nm) 7m over copper 1m over backplane 40GBASE-SR4 40GBASE-LR4 40GBASE-CR4 40GBASE-KR4 1. OM3 is a 50 micron, laser-optimized multimode fiber 2. MMF stands for multimode fiber 3. OM4 is a 50 micron, laser-optimized multimode fiber with higher bandwidth than OM3 4. SMF stands for singlemode fiber 5. Twin-ax cabling is used 40 Gbps PHY using 40GBASE-R encoding over (4) lanes of multimode fiber with a reach up to at least 100m (can support at least 150m over OM4 MMF 2 ) 40 Gbps PHY using 40GBASE-R encoding over (4) wavelength division multiplexing (WDM) lanes of single-mode fiber with a reach up to at least 10km 40 Gbps PHY using 40GBASE-R encoding over (4) lanes of shielded balanced copper cabling 5 with a reach up to at least 7m 40 Gbps PHY using 40GBASE-R encoding over (4) lanes of an electrical backplane with a reach up to at least 1m The objective was to support 40 GbE for at least 100 meters over OM3 50 micron, laser-optimized multimode fiber. With the release of OM4, a 50 micron, laser-optimized multimode fiber (LOMF) with higher bandwidth than OM3, the distance can be extended to 150 meters. There was an objective to support 40 GbE over single-mode for up to 10 kilometers for long reaches. There were also objectives for two copper options; 7 meters over copper and 1 meter over an electrical backplane. Table 2 lists the objectives for supporting 100 GbE over specific media. Table 2: IEEE Objectives for 100 Gigabit Ethernet Objective Resulting PMD Description of PMD 100m on OM3 MMF 1 (850nm) 150m on OM4 MMF 1 (850nm) 10km on SMF 2 (1310nm) 40km on SMF 2 (1310nm) 7m over copper 100GBASE-SR10 100GBASE-LR4 100GBASE-ER4 100GBASE-CR10 100 Gbps PHY using 100GBASE-R encoding over (10) lanes of multimode fiber with a reach up to at least 100m (can support at least 150m over OM4 MMF1) 100 Gbps PHY using 100GBASE-R encoding over (4) wavelength division multiplexing (WDM) lanes on single-mode fiber with a reach up to at least 10km 100 Gbps PHY using 100GBASE-R encoding over (4) wavelength division multiplexing (WDM) lanes on single-mode fiber with a reach up to at least 40km 100 Gbps PHY using 100GBASE-R encoding over (10) lanes of shielded balanced copper cabling 3 with a reach up to at least 7m 1. MMF stands for multimode fiber 2. SMF stands for single-mode fiber 3. Twin-ax cabling is used 3

The objectives for both 40 and 100 GbE are similar. Almost all of the objectives for 40 GbE are also objectives of 100 GbE. 100 GbE includes a longer-reach goal over single-mode for support up to 40 km. Distance will not be an issue when migrating from 40 GbE to 100 GbE since the supported distances are the same (except for the 40km on single-mode fiber). These objectives led to the requirements summarized in table 3 and 4 for 40 GbE (table 3) and 100 GbE (table 4). The tables summarize the signaling, media and distance for both 40 Gigabit Ethernet and 100 Gigabit Ethernet. Table 3: Signaling, Media and Distance for 40 Gigabit Ethernet PMDs 40 Gigabit Ethernet PMD Name 40GBASE-SR4 40GBASE-LR4 40GBASE-CR4 Signaling 4 x 10 Gbps 4 x 10 Gbps 4 x 10 Gbps Media Parallel MMF Duplex SMF Twin-ax Distance 0.5 100m OM3 10km SMF 7m copper cable assembly Table 4: Signaling, Media and Distance for 100 Gigabit Ethernet PMDs 100 Gigabit Ethernet PMD 100GBASE-SR10 100GBASE-LR4 100GBASE-ER4 100GBASE-CR10 Signaling 10 x 10 Gbps 4 x 25 Gbps 4 x 25 Gbps 10 x 10 Gbps Media Parallel MMF Duplex SMF Duplex SMF Twin-ax Distance 100m OM3 10km SMF 40km SMF 7m Copper Cable Some key takeaways are that both 40 GbE and 100 GbE require more than two fibers for transmission over multimode fiber. 40 GbE requires four transmit and four receive multimode fibers, for a total of eight fibers per channel. 100 GbE requires ten transmit and ten receive multimode fibers, for a total of twenty fibers per channel. The single-mode options for 40 GbE and 100 GbE require parallel transmission as well. 40 Gigabit Ethernet over single-mode uses four transmit lanes and four receive lanes, each transmitting at 10 Gbps. 100 Gigabit Ethernet over single-mode uses four transmit and four receive lanes, each transmitting at 25 Gbps. IEEE 802.3ba, the 40 Gbps and 100 Gbps Ethernet transmission standard, specifies signaling over single-mode fiber using wavelength division multiplexing (WDM) transmission. This means that for 40 GbE and 100 GbE over single-mode fiber, each of the four lanes is transmitted at a different wavelength. 40GBASE-LR4 transmission is defined by a center wavelength and wavelength range for each lane. The center wavelengths used for the four lanes are members of the CWDM (Conventional/Course Wavelength Division Multiplexing) grid defined in the ITU-T G.694.2 standard. This standard defines a channel spacing grid using wavelengths from 1271 to 1611 nm, with a channel spacing of 20nm. Table 5 shows the center wavelength and wavelength range for each 40GBASE-LR4 transmission lane. Notes: MTP is a registered trademark of US Conec, Ltd. 4

100GBASE-LR4 and 100GBASE-ER4 define a wavelength range for each lane also. The wavelength range is the same for both 100GBASE PMDs as shown in table 5. These ranges are based on center frequencies that are part of the frequency grid defined in the ITU-T G.694.1 standard. This standard defines a set of frequencies used to designate allowed central frequencies to support dense wavelength division multiplexing (DWDM) applications. This standard supports a variety of channel spacing ranging from 12.5 GHz to 100 GHz and wider, beginning at 193.1 THz. 100GBASE-LR4 and 100GBASE-ER4 lanes use center frequencies from 229 THz to 231.4 THz and are spaced at 800 GHz. Table 5 shows the center frequency, correlating center wavelength and wavelength range for each 100GBASE-LR4 and 100GBASE-ER4 lane. Table 5: Wavelength-Division-Multiplexed Lane Assignments Lane L 0 L 1 L 2 L 3 40GBASE-LR4 Center Wavelength 1271 nm 1291 nm 1311 nm 1331 nm Wavelength Range 1264.5 to 1277.5 nm 1284.5 to 1297.5 nm 1304.5 to 1317.5 nm 1324.5 to 1337.5 nm 100GBASE-LR4 and 100GBASE-ER4 Center Frequency Center Wavelength 231.4 THz 1295.56 nm 230.6 THz 1300.05 nm 229.8 THz 1304.58 nm 229 THz 1309.14 nm Wavelength Range 1294.53 to 1296.59 nm 1299.02 to 1301.09 nm 1303.54 to 1305.63 nm 1308.09 to 1310.19 nm Since the different wavelengths do not interfere with each other when transmitted on a single fiber, all four can be transmitted over one fiber. If the four lanes of the signal were transmitted at the same wavelength then four fibers are needed to separate the lanes as in parallel transmission over multimode. The four receive lanes also use WDM transmission so 40 GbE and 100 GbE channels over single-mode only require a total of two fibers; one transmit fiber and one receive fiber. These cables typically use LC connectors. There is no requirement to associate a particular electrical lane with a particular optical lane since the transceiver is capable of receiving lanes in any order. Both 40 GbE and 100 GbE have a copper option for up to 7 meters using Twinax cable. 802.3ba does not define a twisted-pair option. 40/100 GIGABIT ETHERNET CONNECTIVITY AND CABLE Except for the single-mode long-reach options, all 40/100 Gigabit Ethernet options over fiber use parallel transmission, requiring more than two fibers per channel. Fiber connectivity must be able to terminate more than two fibers. This is a departure from connectivity used in systems supporting up to 10Gigabit Ethernet, which only requires a total of two fibers per channel. The most common connector for transmission over two fibers is the LC. This is the only connector recommended for new installations requiring two fibers for transmission in the TIA data center standard, ANSI/TIA-942-A. This connector is used for 10 GbE and below, as well as the 40/100 GbE single-mode options listed above. With the need to support multiple transmission paths, the Media Dependent Interface (MDI) identified by the IEEE 802.3ba standard for 40 GbE and 100 GbE transmission (when not using WDM) is the MPO style connector. The MPO connector is the connector recommended by the ANSI/TIA-942 data center standard, for applications requiring parallel fiber transmission. The terms MPO and MTP are used interchangeably for this style of connector. MPO is the generic name for this Multi-Fiber Push On connector style. MTP is an MPO style connector and a registered trademark of US Conec, Ltd. 5

Figure 2 below shows an MPO plug and adapter interface. Figure 2: MPO Connector Fiber cable Key Keyway Alignment hole MPO female plug connector flat interface Alignment pin Male MDI as a PMD receptacle for mating with a female MPO plug connector. MPO connectors are typically terminated onto 12 fibers. MPOs may also be terminated onto 24 fibers. There is a keyway for maintaining polarity. (Polarity is covered in more depth later in the paper in the section entitled, Fiber Considerations when Migrating to 40/100 Gigabit Ethernet ). The connector has precision alignment pins or holes to ensure all fibers align properly with the mating connector. The component type (i.e. cassette, adapter panel, trunk cable) usually dictate whether there are pins or holes; pins are usually on the fixed components like cassettes. If not properly cleaned, alignment pins could collect debris around the pins resulting in the two components not mating correctly. IEEE 802.3ba identifies specific positions on an MPO connector to use for transmit and receive. The four transmit and four receive optical lanes of 40GBASE-SR4 (40 GbE over multimode) must occupy the positions shown in figure 3 below. Looking at the end face of the MPO, with the connector key on top, the transmit optical lanes occupy the four leftmost positions and the receive optical lanes occupy the four rightmost positions. There are eight active lanes within twelve total positions, with the four center positions unused. Figure 3: 40G-BASE-SR4 Optical Lane Assignments Tx Tx Tx Tx Rx Rx Rx Rx The 100BASE-SR10 (100 GbE over multimode) requires a total of 20 fibers, 10 transmit and 10 receive. Position assignments are shown below. There are three options, the first being a single-receptacle shown as Option A in figure 4 below. Option A is recommended by IEEE. The two-receptacle options: Option B and Option C are alternatives. Option A uses a 24-position MPO connector with the top center 10 positions allocated for receive and the bottom 10 center positions allocated for transmitting, as shown in figure 4. Figure 4: 100G-BASE-SR10 Optical Lane Assignments Option A: Single connector (recommended) 6

Option B and C use two 12-position MPO connectors. Option B, shown in figure 5, uses side-by-side interfaces. The 10 center positions of the right interface are used for receive and the 10 center positions of the left interface are used for transmit. Figure 5: 100G-BASE-SR10 Optical Lane Assignments Option B: Side by side (alternative) Options C is similar to option B, but uses the stacked layout depicted in figure 6. The ten center positions of the top connector are used for receive and ten center positions of the bottom connector are used for transmit. Figure 6: 100G-BASE-SR10 Optical Lane Assignments Option C: Stacked (alternative) Equipment manufacturers usually play a key role in driving the adoption of a particular MDI (Media Dependent Interface) option. For example, Option A, the single 24-position MPO has more connections in a smaller footprint, making it more complex and therefore more costly to manufacture. Option B, the two 12-fiber side-by-side MPO connectors require twice the width of the other two options. Option C, the two stacked 12-position MPO connectors provide single-width, but takes up more vertical space that could potentially add rack units. IEEE is also standardizing a 100 GbE option that uses four transmit and four receive fibers, for a total of eight fibers. The data rate on each fiber is 25 gigabits per second. Interfaces that active equipment manufacturers adopt will play a significant role in driving a particular MDI option. FIBER CONSIDERATIONS WHEN MIGRATING TO 40/100 GIGABIT ETHERNET A multimode fiber system is the most cost effective fiber solution to use in the data center. Most distances are less than 150 meters; surveys have shown that more than 80% of data centers are equal to or less than 100 meters. Multimode fiber transceivers are much less expensive than single-mode transceivers because they use a vertical cavity surface emitting laser (VCSEL) light source, which is easy to manufacture and package. 7

Although single-mode cable is less expensive, factoring in the total system cost of multimode versus single-mode, multimode becomes significantly less expensive. Some common approaches used in data centers are summarized in table 6 below. Each approach uses short-wavelength (850 nanometer) transmission over multimode fiber. Table 6: Common Data Center Approaches Using Short Wavelength Transmission 10G 40G 100G Signaling 10Gb 10Gb x 4 10Gb x 10 Laser Type VCSEL VCSEL Array VCSEL Array Fiber Type OM3/OM4 OM3/OM4 OM3/OM4 Connector 2 LCs 12-Fiber MPO (2) 12-fiber MPOs or 24-fiber MPO 2 fibers 12 fibers (8 used) 24-fibers (20 used) Number of Fibers Maximum Distance OM3: 300m OM4: 550m OM3: 100+ m OM4: 150+ m 1 1. 150 meters on OM4 requires low loss connectors. This is discussed in the channel insertion section. Note: A 24-fiber MPO can be used instead of the two 12-fiber MPOs OM3: 100+ m OM4: 150+ m 1 The fiber system should be designed around OM3 or OM4 MMF if there are plans to support applications beyond 10 Gbps. OM3 supports 10 GbE up to 300 meters, but only supports 40/100 GbE up to 100m. OM4 supports 10 GbE up to 550 meters, but only supports 40/100 GbE up to 150 meters. If planning to support 40 GbE and/or 100 GbE in the future, the channel cannot be designed for the maximum distances that 10G can support. Both OM3 and OM4 support 10 GbE through 100 GbE for up to 100 meters, but OM4 will support 150 meter lengths if there are distances in the data center that exceed 100 meters. Always design for the application that has the most stringent requirements (usually the fastest data rates) even if the application is a future installation. In addition to selecting the type of fiber, OM3 or OM4, there are several other important considerations when selecting components for a fiber cabling system. These include channel insertion loss, polarity and alignment pins. Channel Insertion Loss/Loss Budget The channel insertion loss is made up of the insertion loss (IL) of the cable, specified as decibels per kilometer (db/km), the insertion loss of all mated connector pairs and the insertion loss of splices in that channel. Referring to table 7, as the data rate increases from 10 Gbps to 40/100 Gbps, the total channel insertion loss or loss budget decreases noticeably. Table 7: Maximum Channel Insertion Loss PMD Name Fiber Type Total Number of Fibers Max Link Length (meters) Max Channel Insertion Loss (dbs) 10 GbE 10GBASE-SR OM3 2 300 2.6 40 GbE 40GBASE-SR4 OM3 8 100 1.9 40 GbE 40GBASE-SR4 OM4 8 150 1.5 100 GbE 100GBASE-SR10 OM3 10 100 1.9 100 GbE 100GBASE-SR10 OM4 10 150 1.5 8

Understanding the impact of each component in the channel loss budget is extremely important when selecting cable and connectors. Often, the cable attenuation performance and bandwidth drive the design of the channel. The impact that a connector can have on the total channel budget can be significant. Figure 7 shows total loss budgets for a 100 meter channel at different data rates common to current Ethernet applications. As data rates progress from 100 Mbps Ethernet-based systems to today s 10 Gbps Ethernet-based systems, the optical loss budgets have shrunk considerably from 11 db to 2.6dB. 40/100 Gbps Ethernet systems have an even smaller budget of 1.9 db when using OM3 or 1.5dB when using OM4. Figure 7: Total Channel Insertion Loss by Application 11.0 db 3.0 db 2.6 db 1.9 db 100 Mbps 1 Gbps 10 Gbps 40/100 Gbps (OM3) If we look at two channel insertion loss budget examples for 2 and 3 mated pairs, including the cable loss for a 100 meter link at 850nm, the importance of connector loss is apparent. Using the standard loss for a multimode fiber cable of 3.5 db/km and an average of 0.50 db loss per mated connector pair (TIA standards allow up to a maximum 0.75 db loss and up to 4 connections), the calculated loss for a 100 meter channel with 2 mated connector pairs is 1.35 db ((3.5db/km * 0.1km) + (0.5 * 2)). Applied to the loss budgets, as shown in figure 8, this is not significant for 100 Mbps systems. However, the insertion loss takes up a little more than half of the 10G budget and almost three-quarters of the 40/100 Gbps budget. Figure 8: Channel Insertion Loss In A 100 Meter Channel with 2-Mated Connector Pairs 11.0 db 1.35 db insertion loss 3.0 db 2.6 db 1.9 db 100 Mbps 1 Gbps 10 Gbps 40/100 Gbps (OM3) If we look at a 3 connector-pair channel, the loss budget goes to 1.85 db ((3.5db/km * 0.1km) + (0.5 * 3)), as shown in figure 9. This is more than 70% of the 10 Gbps budget and almost the entire 40/100 Gbps budget. This would exceed the loss budget using OM4 for 150 meters, which is 1.5 db because of the longer distance, proving insertion loss of a connector is very important. 9

Figure 9: Channel Insertion Loss In A 100 Meter Channel with 3-Mated Connector Pairs 11.0 db 1.85 db insertion loss 3.0 db 2.6 db 1.9 db 100 Mbps 1 Gbps 10 Gbps 40/100 Gbps (OM3) It is important to consider the trade-off. If the IL of one component can be reduced, there will be room for extra loss in another component. For example, if using OM4 at only 100 meters instead of 150 meters, the loss of the cable will be less because IL is directly related to distance (db/km). This can make room for more mated connector pairs. However, all of the IL gain can easily be negated with inferior connector components. Polarity Don t forget to plan for the proper polarity. Maintaining proper polarity guarantees an optical path from the transmit port of one device to the receive port of another device, known as the polarity flip. There are several different methods to maintain polarity, but the different methods may not be interoperable. There are three methods depicted in the TIA standards; methods A, B and C. There are other proprietary methods used by various manufacturers. Each method requires a specific combination of components to maintain polarity. Assuming duplex signaling, using an MPO backbone cable, cassettes and patch cords, the following list shows the component options that are used in specific combinations for each of the polarity methods. The options for components are: MPO-to-MPO backbone cables: Type A, B or C MPO-to-LC cassettes: Method A or Method B Patch cords: Type A-to-A or Type A-to-B Figure 10: A-to-A and A-to-B Patch Cords A B A B A-to-A Patch Cord A-to-B Patch Cord B A B A For example, with duplex signaling, a Method A polarity scheme uses a Method A cassette, Type A trunk cable and a type A-to-B patch cord on one end of the channel and a type A-to-A patch cord on the other end. The transmit to receive flip is done in the patch cord at one end. Method B uses a Method B cassette and trunk cable and an A-to-B patch cord at each end because the flip is done in the cassette and trunk cable. Method C uses a Method A cassette with a Type C trunk cable and A-to-B patch cords at each end. The flip is done in the trunk cable only. Polarity becomes more complicated when migrating to 40/100 GbE because parallel transmission replaces duplex transmission. Parallel optical fiber links integrate multiple transmitters in one transmitter module, multiple fibers in fiber array connectors and multiple receivers in one receiver module. Multiple transmitters and receivers may also be integrated together in a transceiver module. The three methods, A,B and C, are expanded in the ANSI/TIA-568 standard to include links that use parallel signaling in one row (12-fiber MPO) and parallel signals in two rows (24-fiber MPO). Array connectors are keyed to maintain polarity. A keyed MPO connector is shown in figure 11. 10

Figure 11: MPO Plug Fiber Positions Looking at the End of Ferrule with Key Up Alignment Pin Holes Key shown up MPO Ferrule Position 1 Position 12 Alignment Pins When mating connector plugs that use alignment pins, like the MPO connector, it is critical that one plug is pinned and the other plug is unpinned. Because all known transceivers that accept MPO plugs are pinned, they accept only unpinned plugs. Figure 12: MPO Connector With Pins Installed Alignment Pins The pinned connector is typically located inside the panel to help protect the pins from being damaged (i.e. the fixed connector is pinned and the connector that is frequently removed and handled is unpinned). For example, cassettes are typically pinned and trunk cables are typically unpinned. Consult the manufacturer since there may be exceptions required for your design. If not properly cleaned, alignment pins could collect debris around the pins resulting in the two components not mating correctly. WHAT ROLE WILL COPPER PLAY? In addition to defining 40 Gbps and 100 Gbps Ethernet transmission over optical fiber, the IEEE 802.3ba standard defines an option for copper at each data rate: 40GBASE-CR4 and 100GBASE-CR10. Both define transmission over multiple lanes (4 and 10 respectively) of shielded balanced copper cabling (twinax cable) with a reach of up to at least 7 meters. Because of the 7 meter distance, this application is typically used for connection within a rack or in a neighboring rack. There is no specification for 40/100 GbE transmission over twisted pair balanced copper cabling. IEEE initiated work on a standard for 40GBASE T, representing 40 Gbps Ethernet over balanced twisted pair. This technology is what started the work in TIA on the category 8 specification for balanced twisted pair cabling. Category 8 performance will be specified out to 2 GHz and will support a 2-connector, 30 meter channel over balanced twisted pair. The application for this technology is in the Data Center (DC) environment for server-to-switch connections within a row, typically referred to as end-of-row or middle-of-row architectures as shown figure 13. 11

Figure 13: NGBASE-T Application (shown in IEEE Tutorial; July 2012 Plenary) Distance served by NGBASE-T Within the rack Neighboring racks, stranded ports End of row Distance served by CR4 Within the rack Neighboring racks OTHER CONSIDERATIONS Too often when designing data centers, the focus is on selecting cabling with little consideration given to the importance of the other parts of the cabling infrastructure. Earlier we learned how the connector can have a major impact on the loss budget of a channel. The patch cord is a component within the channel that is often also overlooked, but can have a significant impact on the performance of the channel. Low quality patch cords can cause the channel to be below TIA performance requirements, preventing the channel from supporting applications for which it was designed. You may need to consider high-performing, low loss patch cords to meet the budget of the most bandwidth-intensive, highest frequency application for which you are designing. The best performing product can be assembled into a channel, but without the proper infrastructure, the network performance can be compromised. A well-designed physical support infrastructure will make maintaining the network simpler. It is important to consider the density of the cables. Make sure that the physical support infrastructure (e.g. the rack and the vertical cable manager) is able to properly handle the number of cables needed. Keep in mind that there may be a mix of copper and fiber media; copper for connection within the rack and neighboring racks. The rack should be able to properly support multiple media types, each presenting their own challenges. For example, copper twisted pair cable is heavier and has a larger cable OD, requiring a larger bend radius than fiber. Figure 14: Racks and Cabinets Must Provide Proper Cable Management with Easy Accessibility 12

Although fiber cable has a much smaller OD, the greater fiber connection density often results in more fiber cables. Infrastructure must be able to properly support and maintain a proper bend radius and crush resistance. Consider mechanical specifications and bend radius of the optical fiber cable. Proper support and strain relief of the cable at the active equipment ports eliminate the potential of ports being damaged from an unsupported or tightly bent cable. Figure 15: Proper Support of Fiber and Airflow There should be sufficient room to easily access cables and allow proper air flow. Active equipment can breathe side-to-side, front-to-back or top-to-bottom. The rack or cabinet should be able to accommodate each of these configurations. If the active equipment changes, the rack or cabinet should be able to accommodate the new active equipment. Remember that accessories may be required. The vertical managers must be deep enough to allow the management of many cables of different media types, but to also allow for the proper airflow for the active equipment installed. Figure 16: Vertical Managers Must be Sized Appropriately Properly designed infrastructure will help maximize the network performance and simplify maintaining it by making moves, adds and changes more efficient. It also promotes energy efficiency by using passive cooling to manage airflow through network equipment and control heat loads throughout the data center. A properly designed infrastructure will help optimize space constrained network real estate by accommodating higher density equipment and connectivity solutions. The properly designed infrastructure will support current and future needs, facilitating growth in a timely and cost effective way. WHAT S COMING? IEEE has a number of ongoing projects. There is a working group, 802.3bm, which is looking at the next generation of 100 GbE to reduce cost. This standard will reduce the number of transmission lanes from 10 to 4 by transmitting 25 Gbps in each lane instead of 10 Gbps. This will make it very easy to update the infrastructure from 40 GbE to 100 GbE because both use the same number of fibers for transmission. IEEE also has a 400 GbE study group that was formed in March, 2013. The goal is to provide physical layer specifications, supporting the following link distances: At least 100 m over MMF At least 500 m over SMF At least 2 km over SMF At least 10 km over SMF There are also many developments within Fibre Channel, a high-speed network technology primarily used to connect computer data storage. 32G Fibre Channel (GFC) is currently under development. The target link distance is 100 meters over OM4 and 70 meters over OM3. 32 GFC still uses serial transmission with 2 fibers and will use the same external 13

small form factor pluggable (SFP) transceiver modules with LC optical connectors. This will be backward compatible with 8 GFC and 16 GFC. There is a new project looking at 128 GFC. Normally, Fibre Channel doubles in speed, 8GFC, 16 GFC, 32GFC, etc., but 128GFC will be based on 32 GFC. 128 GFC will use 4 x 32 GFC, so it will require the 32GFC standard to be completed first. A port will be able to autonegotiate 128 GFC back to 32 GFC and 16 GFC without user intervention. 64 GFC will be developed once work is complete on 128 GFC. 10, 40, AND 100 GIGABIT ETHERNET DEPLOYMENT EXAMPLES Structured cabling models that follow the guidelines of the standards provide flexibility, facilitate troubleshooting and provide modularity. However, 10 GbE structured cabling systems designed today must be carefully designed to support 40/100 GbE in the future. Figures 17 and 18 show two common scenarios for deploying 10 GbE in data centers. The first example is commonly used to interconnect or cross-connect applications. This scenario uses a 12-fiber MPO-to- MPO fiber trunk with cassettes loaded in fiber enclosures. The cassettes transition from the MPO backbone to simplex LCs (6-pairs) that plug into the equipment. Figure 17: 10 GbE Deployment Example 1 Equipment Example 2 is often used as an interconnect method, using a harness to transition from the MPO interface on the trunk to the discrete LCs, used to connect to the 10 GbE equipment. The 12-fiber MPO-to-MPO trunk plugs into the back side of the MPO adapter plate in the fiber enclosure. The transition to discrete connectors, for the10 GbE equipment, is done with an MPO-to-LC cable harness assembly like the Ortronics HiLOC cable that plugs into the front side of the MPO adapter plate as shown. Figure 18: 10 GbE Deployment Example 2 Equipment Duplex Cords (6x2-fibers) Cassettes in Fiber Enclosures Either scenario can be easily migrated to 40 GbE. Migrating the first example to 40 GbE requires replacing the cassette with an MPO adapter panel in the fiber enclosure; cassettes can be repurposed on the SAN (Storage Area Network) side. The six discrete patch cords are replaced with a 12-fiber MPO array cord. This would provide one 40 GbE channel. Each 40 GbE channel requires one 12-fiber MPO trunk cable and a 12-fiber MPO array cord on each end. Migrating the 10 GbE example in figure 18 to 40 GbE, requires replacing the cable harness assembly with an MPO array cord. As shown in figure 19, each 40 GbE channel requires one 12-fiber MPO trunk cable and a 12-fiber MPO array cord on each end from the adapter panel to the active equipment. Figure 19: 40 GbE Deployment Equipment This cabling solution can be easily migrated to 100 GbE by adding a second 12-fiber MPO trunk cable and replacing the 12-fiber MPO cord with a cord that has a 24-fiber MPO connector at one end and two 12-fiber MPOs on the other, as shown in figure 20. Figure 20: 100 GbE Deployment Equipment 6x2-Fibers Harness 12-Fiber MPO Cord Backbone 12-fibers Cassettes in Fiber Enclosures 12-Fibers 12-Fibers MPO Adapter Panel MPO Adapter Panel Backbone 12-Fibers MPO Adapter Panel MPO Cord (24-Fiber to Two 12-Fiber) MPO Adapter Panel 12-Fiber MPO Backbone Duplex Cords (6x2-fibers) 6x2-Fibers Harness 12-Fiber MPO Cord MPO Adapter Panel MPO Cord (24-Fiber to Two 12-Fiber) Backbone (Two 12-Fiber Cables) MPO Adapter Panel Equipment Equipment Equipment Equipment Other design options include using 8-fiber MPOs (4 positions in the 12-fiber connector are not used) to connect to 40 GbE equipment. The examples shown here were selected because they provide easy upgrades and are common. Using a 12-fiber MPO cord for 40 GbE leaves 4 fibers dark, but the cord can be reused when upgrading to the new cost savings 100 GbE that uses eight total fibers (four transmit fibers and four receive fibers) instead of 20 total fibers. Notes: MTP is a registered trademark of US Conec, Ltd. 14

CONCLUSIONS Before selecting product for your data center design, establish the fastest application your structured cabling will need to support. Multimode fibers systems are more common than single-mode systems for short distances because they are more cost effective. Select at least OM3, however OM4 will provide longer distance support or more connections over shorter distances. The type of connector is determined by the transmission; LC for duplex transmission and MPO/MTP for parallel transmission. Channel insertion loss is the foundation for design, so consider high-performance, low loss components. You will also need to consider the polarity method to be used and then select the correct components to support that method. If using array connectors for parallel transmission, consider which components require pins and which do not. The best option is to work with the manufacturer to make sure the correct components are selected. Don t forget to put as much thought into designing your physical infrastructure as the structured cabling. The connection density in switches, servers and routers is increasing. This means more cable to manage and higher operating temperatures, making properly managed airflow extremely important. The proper design of the infrastructure is critical to support the life of the network and protect the investment. Notes: MTP is a registered trademark of US Conec, Ltd. 15

Europe/Middle East/Africa Great King Street North Birmingham, B19 2LF United Kingdom Phone: +44(0).796.730.9227 Latin America Montes Urales NO. 715 3er Piso Col. Lomas de Chapultepec MEXICO D.F. - C.P. 11000 Phone: +52.55.5201.6450 Asia Pacific 3rd Floor Hang Seng Building, No. 200 Hennessy Road, Wanchai, Hong Kong Phone (HKG): +852.9401.1885 Phone (China): +86.1392.4848.222 Level 2, OVAL Building, Hi-tech City, Madhapur, Hyderabad - 500081, India Phone: +91.40.4433.4234 2014 Legrand All Rights Reserved rev.09 14 INT designed to be better. Global Headquarters 125 Eugene O Neill Drive New London, CT 06320 Phone: +1.800.934.5432 or +1.860.445.3900 Fax: +1.860.405.2992 www.ortronics.com Follow Us