10 Gb Ethernet: Evaluating when and why to upgrade

Size: px

Start display at page:

Download "10 Gb Ethernet: Evaluating when and why to upgrade"

Jemima Sutton
5 years ago
Views:

1 10 Gb Ethernet: Evaluating when and why to upgrade

2 While 10 (GbE) has been broadly available for years, it s still new to most network engineers and can be a challenge to decide when. This expert E-Guide highlights how migrating to 10GbE and debunks some of the most common myths surrounding the technology. Uncover truths about price, requirements, what s in store for the future and more. By: Howard Marks, contributor 10 (GbE) is becoming the new normal, at least in the data center. Adoption has followed the same pattern we saw for the transition from 10 Mbps to 100 Mbps Fast Ethernet and from Fast Ethernet to : Data center and network managers began installing expensive fiber optic ports to ameliorate bottlenecks between switches. As prices for the technology fell, they were able to begin connecting servers as well. Eventually, the high-performance connections spread to broad adoption. However, while prior Ethernet transitions have been primarily about speed, the migration to 10 GbE promises much more. Today, most data centers run multiple networks for different purposes. Servers access storage arrays over dedicated Fibre Channel networks, while users access the same servers over separate, and separately managed, Ethernet networks. The Ethernet network may be further divided -- either physically or through virtual LANs -- into networks for user access, management, security and backup. Most virtual server hosts running VMware, Xen or Hyper-V virtualization software must be connected to several of these networks. That means each server host will have a redundant pair of 1 GbE ports for each network. Surveys of data center Page 2 of 9

3 managers put the average number of Ethernet connections per virtual server host somewhere between six and 10. With greater bandwidth available, a single pair of 10 GbE connections could satisfy all the input and output (I/O) requirements of a typical virtual server host. But the challenge for network and data center managers is not simply to provide bandwidth. They must also ensure that each system, function and application on the network has the amount of bandwidth and network quality of service it needs while attaining interoperability and avoiding bottlenecks. 10 GbE allows data and Much of the buzz accompanying 10 GbE has been about its ability to support the of data and s. During the past 10 years, Fiber Channel storage area networks (SANs) have increased in bandwidth to an average of 4 or 8 Gbps, while iscsi (Internet Small Computer System Interface) SANs typically run over 1 Gbps Ethernet. More significantly, the industry has developed SAN management tools that aren't available for iscsi but give administrators visibility and control of Fibre Channel networks. The Fibre Channel over Ethernet (FCoE) protocol was developed to allow IT organizations to converge SAN and data traffic onto 10 GbE while preserving the security and management models of Fibre Channel. FCoE encapsulates Fibre Channel Protocol (FCP) data packets inside Ethernet frames so that a server can access its Fibre Channel SAN-attached storage through an Ethernet connection. FCoE s designers specified exactly how FCP packets should be encapsulated within Ethernet, but also had a significant problem to solve. Ethernet and Fibre Channel handle network congestion very differently. Fibre Channel devices negotiate the amount of data they intend to send across a link before the data is actually transmitted, using buffer credits. The sender does transmit until the recipient says it has enough memory available to hold the data. Ethernet systems, on the other hand, just send the data. If the line is so busy that the memory in the switch at the other end is too full to hold the new data, the packet is dropped. Page 3 of 9

4 Overcoming packet loss with data center bridging The inherent issue is that, as usually implemented, Fibre Channel lacks a recovery mechanism to retransmit dropped packets. And implementing one would add unacceptable latency to storage traffic. As a result, FCoE has been engineered to take advantage of a series of congestion management enhancements to Ethernet that together are known as data center bridging (DCB). The most significant new feature in DCB is per-priority pause. This capability allows an Ethernet switch to tell a server to stop sending low-priority data until it can flush some data from its memory, rather than waiting until its buffers get full and dropping packets. There are eight possible priority levels, and FCoE uses the highest priority so that storage data can be communicated even during busy network times. The packet loss DCB achieves can boost performance of other protocols, like iscsi, as well. Although FCoE sends storage traffic across ubiquitous Ethernet links, the switches that make up that network must include specific support for FCoE. Some switches, such as the Cisco Nexus 5000 and Brocade 8000, are designed to be placed at the top of each server rack, and include Fibre Channel ports for connections to existing SANs. FCoE-capable switches cost about the same as other 10 GbE switches with DCB, but equipment makers charge extra for the FCoE support features. Enterprises that may not need FCoE capability now can use them as standard top-of-rack switches and add SAN support later. The proponents of FCoE wanted to preserve as much of the Fibre Channel experience as possible, so they built converged network adapters (CNAs) that run with Ethernet drivers for data networking and separate Fibre Channel drivers and management software for FCoE traffic. CNAs allow the host operating system, or hypervisor, to see separate storage and data connections, while the storage administrator sees the interface they are accustomed to, such as QLogic SANsurfer or Emulex ONEconnect. CNAs also offload the protocol processing of FCoE, or in some cases iscsi, from the main processor in the server. This can save between 3% and 5% in a typical eight-core server, which isn't a large percentage but can add up. Page 4 of 9

5 Opening the network to multiple paths Switches with 10 GbE capability also enable Layer 2 multipathing. Traditional Ethernet switches use the spanning tree protocol to prevent network loops that could clog the network with broadcast and other packets. Spanning tree works by enabling one, and only one, path from one switch to another. So, if there are two uplinks from your top-of-rack switch to your core switch, one will be disabled by spanning tree. Layer 2 multipathing uses the TRILL (Transparent Interconnection of Lots of Links) protocol to learn all the paths from one point on the network to another and send packets down the best path on a per-packet basis. TRILL-based networks don t need the hierarchical layering (i.e., edge, aggregation and core) of spanning tree networks and can be connected in any sort of mesh or fabric. They also use all links possible, increasing available bandwidth between any two switches. We are clearly entering the era of 10 data center networking. With higher speeds and new features like DCB and TRILL, we can build bigger, faster and more resilient networks that can carry not just traditional data traffic but also storage via FCoE. Equipment makers are still at the start of introducing new products, and additional capabilities and features will continue to develop. The time spent carefully taking this evolution into account in your data center network will pay off in spades. About the author: Howard Marks is the founder and chief scientist at DeepStorage.net, a networking consultancy based in Hoboken, N.J. In more than 25 years of consulting, he has designed and implemented networks, management systems and Internet strategies at organizations including American Express, JP Morgan, Borden Foods, U.S. Tobacco, BBDO Worldwide and Foxwoods Resort Casino. Mr. Marks is a frequent speaker at industry conferences and the author of more than 100 articles in technical publications. He has also written the book Networking Windows and coauthored Windows NT Unleashed. Page 5 of 9

6 By: Michael Brandenburg, Technical Editor Although 40 and 100 represent the industry's cutting edge, the vast majority of enterprises are still in the midst of a transition from to 10 (GbE) in their networks. While 10 GbE has been broadly available for years, it is still new to most network engineers. With that in mind, we are busting some of the most common myths surrounding the technology in this 10 GbE edition of myth versus reality. Myth: Deploying 10 is too expensive. Reality: While the price per port of 10 GbE is still higher than Gigabit Ethernet, the gap between the two is narrowing. In high-traffic areas of the network, it might actually be more cost effective to use a single 10 GbE port rather than ten Gigabit links to achieve the same level of bandwidth and performance. What may really work: Mike Spanbauer, principal analyst for enterprise networking and data center technology at Current Analysis, suggests that enterprises evaluate the entire cost of an access port, including power consumption and port density. In data centers where power, cooling and physical space are a premium, 10 GbE may actually be the more costeffective option. Myth: 10 only works on expensive optical cables and transceivers. Reality: Engineers have a range of cabling options for 10 GbE, with more on the way. While optical is the primary choice for long-range links between switches in the data center, enterprises can choose lower-cost optical cabling or twinax copper cabling for shorter range or to interconnect within the server rack. Also, many server and switch vendors are preparing to launch products with 10GBase-T, which will offer the convenience of the more common twisted pair cabling with RJ45 connectors for distances up to 100 meters. Page 6 of 9

7 What may really work: Cabling data center networks can be an expensive, time-consuming proposition, so data center managers should look into all of the available media choices to maximize the life span of the physical network. Likewise, many data center managers prefer to standardize on a single, small form-factor pluggable (SPF+) transceiver module and cabling type, according to Spanbauer. So while a mix and match approach to cabling may offer short-term cost savings, installing fiber optic might offer greater investment protection to support 10 GbE and beyond. Myth: Enterprises will have to completely rip and replace switches or do forklift upgrades to support 10. Reality: Many enterprises will discover that the modular switches running in their networks are capable of supporting 10 GbE line cards. Also, many new 10 GbE fixed configuration switches fit into the same footprint as existing, racked switches. What may really work: The need for the additional bandwidth that comes with 10 GbE may quickly spread beyond the data center. A number of n wireless access points could fill a uplink, necessitating 10 GbE links to aggregate the wireless LAN traffic, Spanbauer said. A network manager should consider 10 Gigabit options throughout the entire enterprise network. Myth: 10 is a requirement for building a data center fabric. Reality: Many networking vendors offer data center fabric switches in both Gigabit and 10 GbE port options. These are offered as a low-cost option, and data center managers can take advantage of the any-to-any connectivity and low-latency functionality that a data center offers to existing servers that lack 10 GbE adapters. What may really work: While data center fabrics offer support for Gigabit Ethernet, most network administrators will need 10 GbE to support a virtualized data center, Spanbauer said. The effect of running multiple virtual Page 7 of 9

8 servers on a single physical machine is an increase in network utilization on every network port. Without sufficient bandwidth, the network might become the bottleneck in a highly optimized data center. Myth: The only reason to move to 10 is as part of a network and storage project. Reality: Prior to the rise of server virtualization and data center fabrics, many networking vendors made a significant marketing push for network. Since the physical link is shared by IP and s must have high bandwidth capacity, deploying technologies such as Fibre Channel over Ethernet (FCoE) or iscsi will demand 10. What may really work: Network is one of many drivers for enterprises to make the case for 10, said Spanbauer. Network, virtualization and even high-capacity WAN links are all demanding more bandwidth than can provide. Myth: Isn t 40 Gigabit or 100 Gigabit coming soon? Why would anyone bother with 10? Reality: Switches that support 40 and 100 are just beginning to come to market. However, many of these products are proprietary or based on draft standards. While network switch vendors are pledging to provide an upgrade path from their current offerings and offer future support for the ratified standard, those vendors that do have 40 GbE on the market use it exclusively for interlinking between their own switches. Also, the price per port for 40 and 100 will remain extremely high until the technology is more broadly adopted. What may really work: Network administrators must continually weigh the cost of bleeding-edge technologies against the benefits of the technology. Very high-end data centers may require the bandwidth that 40 or 100 Gigabit Ethernet provides, but like 10 GbE a few years ago, the majority of enterprise networks will wait some time before the price and standards barriers of these technologies are low enough to adopt. Page 8 of 9

Free resources for technology professionals TechTarget publishes targeted technology media that address your need for information and resources for researching products, developing strategy and

9 Free resources for technology professionals TechTarget publishes targeted technology media that address your need for information and resources for researching products, developing strategy and making cost-effective purchase decisions. Our network of technology-specific Web sites gives you access to industry experts, independent content and analysis and the Web s largest library of vendor-provided white papers, webcasts, podcasts, videos, virtual trade shows, research reports and more drawing on the rich R&D resources of technology providers to address market trends, challenges and solutions. Our live events and virtual seminars give you access to vendor neutral, expert commentary and advice on the issues and challenges you face daily. Our social community IT Knowledge Exchange allows you to share real world information in real time with peers and experts. What makes TechTarget unique? TechTarget is squarely focused on the enterprise IT space. Our team of editors and network of industry experts provide the richest, most relevant content to IT professionals and management. We leverage the immediacy of the Web, the networking and face-to-face opportunities of events and virtual events, and the ability to interact with peers all to create compelling and actionable information for enterprise IT professionals across all industries and markets. Related TechTarget Websites Page 9 of 9

WHAT NETWORK VIRTUALIZATION TECHNOLOGY CAN DO FOR YOUR NETWORK TODAY

E-Guide WHAT NETWORK VIRTUALIZATION TECHNOLOGY CAN DO FOR YOUR NETWORK TODAY SearchSDN : S oftware-defined networking (SDN) and network technologies are being combined to fundamentally change our approach