PowerEdge FX2 - Upgrading from 10GbE Pass-through Modules to FN410S I/O Modules

Transcription

1 PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules Dell Networking Solutions Engineering June 06 A Dell EMC Deployment and Configuration Guide

2 Revisions Date Revision Description Authors June 06.0 Initial Release Jim Slaughter, Mark Fisher, Shree Rathinasamy Copyright 06 Dell Inc. or its subsidiaries. All Rights Reserved. Except as stated below, no part of this document may be reproduced, distributed or transmitted in any form or by any means, without express permission of Dell. You may distribute this document within your company or organization only, without alteration of its contents. THIS DOCUMENT IS PROVIDED AS-IS, AND WITHOUT ANY WARRANTY, EXPRESS OR IMPLIED. IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE SPECIFICALLY DISCLAIMED. PRODUCT WARRANTIES APPLICABLE TO THE DELL PRODUCTS DESCRIBED IN THIS DOCUMENT MAY BE FOUND AT: Performance of network reference architectures discussed in this document may vary with differing deployment conditions, network loads, and the like. Third party products may be included in reference architectures for the convenience of the reader. Inclusion of such third party products does not necessarily constitute Dell s recommendation of those products. Please consult your Dell representative for additional information. Trademarks used in this text: Dell, the Dell logo, PowerEdge and Force0 are trademarks of Dell Inc. Other Dell trademarks may be used in this document. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and/or names or their products and are the property of their respective owners. Dell disclaims proprietary interest in the marks and names of others. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and/or names or their products and are the property of their respective owners. Dell disclaims proprietary interest in the marks and names of others. PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

3 Table of contents Revisions... Introduction Typographical conventions Before upgrading... 4 Hardware The Dell PowerEdge FX/FXs and supported modules Dell 0GbE Pass-through Module PowerEdge FN40S I/O Module... 7 Upgrade Overview Step-by-step example Part - Configure the ToR switch..... Part Replace the PTM in Slot A with an FN40S..... Part Replace the PTM in Slot A with an FN40S... 4 A References and additional resources... 6 B Components used in this guide... 7 C Terminology... 8 D FN IOM internal port mapping... 9 D. Quarter-width servers with dual-port CNAs... 9 D. Half-width servers with dual-port CNAs... 0 D. Half-width servers with quad-port CNAs... D.4 Full-width servers with dual-port CNAs... D.5 Full-width Servers with quad-port CNAs... E Support and feedback... 4 PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

4 Introduction This deployment guide covers a step-by-step procedure to minimize network downtime when upgrading from Dell 0 Gigabit Ethernet (0GbE) Pass-through Modules (PTMs) to PowerEdge FN40S Input/Output Modules (IOMs) in a PowerEdge FX chassis. When upgrading, all servers in the chassis may remain up throughout the process. In the scenario covered in this guide, the PTMs are hot-swapped one at a time so that no more than half of the server network ports are down at any point in the upgrade. Upgrading from PTMs to FN40S IOMs provides the following benefits: Adds multilayer switching capability to the chassis Adds redundancy to links going to the top-of-rack (ToR) switch Optimizes east-west traffic by allowing server-to-server communication within the chassis Can significantly reduce cabling at the back of the chassis Additional configuration is not required on the FN40S for many common deployments Note: If FX server network adapters are configured such that a single LACP NIC team spans both PTMs, and therefore will span both FN40S IOMs upon replacement, the FN40S IOMs will need to be configured as Virtual Link Trunking (VLT) peers. This will bring all network ports in the FX chassis down during configuration. Use the PowerEdge FX FN I/O Module VLT Deployment Guide for this scenario.. Typographical conventions This document uses the following typographical conventions: Monospace text Bold monospace text Underlined monospace text Italic monospace text CLI examples Commands entered at the CLI prompt CLI examples that word wrap (text is entered as a single command) Variables in CLI examples. Before upgrading Dell EMC recommends reviewing the Dell PowerEdge FN I/O Module (FN IOM) - Easy Deployment Guide before upgrading to FN40S IOMs. The Easy Deployment Guide covers basic FN IOM tasks such as the following: Resetting to factory defaults Establishing a remote or serial connection to the management interface Disabling and re-enabling Uplink Failure Detection (UFD) 4 PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

5 Hardware This section briefly describes the hardware used in this guide.. The Dell PowerEdge FX/FXs and supported modules The PowerEdge FX and FXs enclosures are U, hybrid, rack-based computing platforms. They combine the density and efficiencies of blades with the simplicity and cost benefits of rack-based systems. Dell PowerEdge FXs (front) with one PowerEdge FC60 server installed Dell PowerEdge FXs (back) with two PowerEdge FN40S IOMs installed Note: This guide applies to either the PowerEdge FX or FXs chassis. The FX does not offer PCIe expansion slots. The FXs includes eight PCIe expansion slots, as Figure shows. 5 PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

6 FX with server and networking modules FX module descriptions 6 PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

7 . Dell 0GbE Pass-through Module The Dell 0GbE PTM provides one-to-one connectivity from blade server network adapter ports to ToR switch ports. It has eight internal ports and eight external SFP+ ports. The pass-through module does not do any switching.. PowerEdge FN40S I/O Module The PowerEdge FN40S IOM is a Layer and switch with eight internal ports and four external SFP+ ports. In addition to full switching and routing features, it supports FCoE, iscsi with data center bridging (DCB), stacking, and VLT. It comes preconfigured for many common deployments. 7 PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

8 Upgrade. Overview Figure 5 shows a fully loaded FX chassis configured with two 0GbE PTMs. Four FC60 servers reside in the front slots, each containing a quad-port converged network adapter (CNA). The PTMs reside in slots A and A in the back of the chassis. This configuration requires eight uplinks from each PTM to the ToR switch, one for each server CNA port. If an uplink fails, the corresponding CNA port goes down until the link is restored. Before Upgrade FX Chassis Eight 0GbE Uplinks from each PTM FC60 Slot PTM-A FC60 Slot FC60 Slot ToR Switch FC60 Slot 4 PTM-A Fully loaded FX chassis before upgrade The same FX chassis with the two PTMs replaced by two FN40S IOMs is shown in Figure 6. The four external ports on each FN40S are configured in a 40GbE LACP port channel connected to the ToR switch. Anywhere from one to four uplinks can be used from each FN40S. The bandwidth of each LACP port channel will equal the sum of the connected links. Dell EMC recommends a minimum of two uplinks from each FN40S for redundancy. If an uplink fails, traffic continues to flow on the remaining uplink(s) in the port channel without affecting connectivity of the server CNAs. 8 PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

9 After Upgrade FX Chassis 40GbE LACP Port Channels FC60 Slot FN40S-A FC60 Slot FC60 Slot ToR Switch FC60 Slot 4 FN40S-A Fully loaded FX chassis after upgrade. Step-by-step example This section provides steps to replace each PTM with an FN40S in a sequence that minimizes network downtime on the servers inside the FX chassis. The chassis and servers remain up and running throughout the upgrade. For clarity in the diagrams, this step-by-step example uses a reduced number of connections as shown in Figure 7 and Figure 8. In this example, there are two FC60 servers in the chassis, and each FC60 server has a dual-port CNA. On each server, one CNA port is untagged in the default vlan and the other port is tagged in VLAN 6. A Dell Networking S4048-ON functions as the ToR switch. Notes: Any Dell or third-party switch that supports 0GbE SFP+ links and LACP may be used as the ToR switch in place of the S4048-ON. See the Dell PowerEdge FN I/O Module (FN IOM) - Easy Deployment Guide for LACP configuration commands on common third-party switches. The FN40S IOMs are at their factory default settings throughout the upgrade. See the Dell PowerEdge FN I/O Module (FN IOM) - Easy Deployment Guide for steps to reset FN IOMs to factory defaults if needed. Replace the VLANs and port numbers in this example with those in your configuration. 9 PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

10 Step-by-Step Example: Before Upgrade FX Chassis FC60 Slot FC60 Slot CNA port CNA port CNA port CNA port port port 5 port port 5 PTM-A PTM-A port port 5 te / te / te / te /4 port port 5 ToR Switch S4048-ON default VLAN VLAN 6 te /9 te /0 Upstream Server port port Step-by-step example - before upgrade with PTMs installed Step-by-Step Example: After Upgrade FX Chassis FC60 Slot CNA port CNA port CNA port te 0/ te 0/5 te 0/ FN40S-A te 0/9 te 0/0 te /5 te 0/9 Po 8 Po 0 te /6 te /7 te /8 ToR Switch S4048-ON te /9 te /0 Upstream Server port port FC60 Slot CNA port te 0/5 FN40S-A te 0/0 Po 8 Po default VLAN VLAN 6 Step-by-step example - after upgrade with FN40S IOMs installed 0 PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

11 .. Part - Configure the ToR switch Notes: By default, all four external (uplink) ports on FN40S IOMs are preconfigured in LACP port channel 8. In this example, two links from each FN40S are used. The corresponding connections on the ToR switch must also be configured as LACP port channels. Port channel numbers 0 and are used on the S4048-ON, but they can be assigned any number supported by the ToR switch. Figure 7 shows that before upgrade, the two PTMs connect to four S4048-ON ports (te /-/4). Figure 8 shows that, upon completion of the upgrade, the FN40S IOMs connect to four different S4048-ON ports (te /5-/8). The connections can reuse the original ports. But, if they are available, using different ports minimizes network downtime during the upgrade.. Configure port channel 0 on the S4048-ON to later connect to FN40S-A port channel 8: S4048-ON(conf)#interface range tengigabitethernet /5-/6 S4048-ON(conf-if-range-te-/5-/6)#no shutdown S4048-ON(conf-if-range-te-/5-/6)#description LINK TO FN40S-A S4048-ON(conf-if-range-te-/5-/6)#port-channel-protocol lacp S4048-ON(conf-if-range-te-/5-/6-lacp)#port-channel 0 mode active S4048-ON(conf-if-range-te-/5-/6-lacp)#exit S4048-ON(conf-if-range-te-/5-/6)#exit S4048-ON(conf)#interface port-channel 0 S4048-ON(conf-if-po-0)#portmode hybrid S4048-ON(conf-if-po-0)#switchport S4048-ON(conf-if-po-0)#description PO TO FN40S-A S4048-ON(conf-if-po-0)#no shutdown Port channel 0 does not require VLAN configuration in this example because it will only carry traffic on the default, untagged VLAN (VLAN ).. Configure port channel on the S4048-ON to later connect to FN40S-A port channel 8: S4048-ON(conf)#interface range tengigabitethernet /7-/8 S4048-ON(conf-if-range-te-/7-/8)#no shutdown S4048-ON(conf-if-range-te-/7-/8)#description LINK TO FN40S-A S4048-ON(conf-if-range-te-/7-/8)#port-channel-protocol lacp S4048-ON(conf-if-range-te-/7-/8-lacp)#port-channel mode active S4048-ON(conf-if-range-te-/7-/8-lacp)#exit S4048-ON(conf-if-range-te-/7-/8)#exit S4048-ON(conf)#interface port-channel S4048-ON(conf-if-po-)#portmode hybrid S4048-ON(conf-if-po-)#switchport S4048-ON(conf-if-po-)#description PO TO FN40S-A S4048-ON(conf-if-po-)#no shutdown S4048-ON(conf-if-po-)#exit PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

12 . Tag port channel in VLAN 6: S4048-ON(conf)#interface vlan 6 S4048-ON(conf-if-vl-6)#tagged port-channel This completes S4048-ON switch configuration. It assigns the correct interfaces to LACP port channels 0 and, and configures the port channels in the appropriate VLANs... Part Replace the PTM in Slot A with an FN40S. Disconnect all cables from the PTM in slot A (PTM-A). A is the top slot.. Remove PTM-A from the chassis. All server CNA ports connected to PTM-A go down. The PTM in slot A (PTM-A) is not affected and continues to function normally.. Insert an FN40S into slot A (FN40S-A). FN40S-A powers up and takes about minutes to boot. After FN40S-A has booted, the server CNA ports will continue to be down until the next step. This is due to a feature called Uplink Failure Detection (UFD) whereby when upstream connectivity is down, the FN IOM disables the downstream, server-facing links. Notes: The FN40S switch console can be monitored using a serial cable, SSH, or telnet session. See the Dell PowerEdge FN I/O Module (FN IOM) - Easy Deployment Guide for console connection instructions and more information on UFD. By default on the FN40S, all internal ports and the uplink port channel (po 8) are untagged in the default VLAN (VLAN ) and tagged in all other VLANS (VLANS -4094). This eliminates the requirement to manually configure VLANs on the FN40S for most common deployments, including this example. 4. Connect cables from S4048-ON ports te /5-/6 to FN40S-A ports te 0/9-0/0. The network now appears as shown in Figure 9. After the connection is made, the LACP port channel between FN40S-A and the S4048-ON comes up. On FN40S-A, UFD detects port channel 8 is up and brings up the internal, server-facing ports (FN40S-A ports te 0/ and te 0/5 in Figure 9). This causes the corresponding server CNA ports to come up. PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

13 Step-by-Step Example: Upgrade Midpoint FX Chassis FC60 Slot FC60 Slot CNA port CNA port CNA port CNA port te 0/ te 0/5 port port 5 FN40S-A PTM-A te 0/9 Po 8 Po 0 te 0/0 te /5 te /6 te / port te /4 port 5 ToR Switch S4048-ON default VLAN VLAN 6 te /9 te /0 Upstream Server port port Upgrade midpoint with one PTM and one FN40S installed 5. After the FC60 server's CNA ports are up, a north-south connectivity check is done by pinging the servers in the FX chassis from an upstream device on all applicable VLANs traversing FN40S-A. The upstream server at successfully pings the FX servers at and through the ToR and FN40S-A on the default VLAN. PTM-A is not affected and functions normally. 6. An east-west connectivity check is done by pinging between the servers in the FX chassis on all applicable VLANs traversing FN40S-A. The FC60 server at successfully pings the FC60 server at through FN40S-A on the default VLAN without going through the ToR switch. Note: If you want to verify east-west traffic bypasses the ToR switch by shutting port channel 8, disable UFD on the FN40S first. Instructions for disabling and re-enabling UFD are in the appendix of the Dell PowerEdge FN IOM Easy Deployment Guide. PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

14 .. Part Replace the PTM in Slot A with an FN40S. Disconnect all cables from the PTM in slot A (PTM-A).. Remove PTM-A from the chassis. All server CNA ports connected to PTM-A go down. FN40S-A is not affected and functions normally.. Insert an FN40S into slot A (FN40S-A). FN40S-A powers up and takes about minutes to boot. After FN40S-A has booted, the server CNA ports will continue to be down until the next step due to the UFD feature. 4. Connect cables from S4048-ON ports te /7-/8 to FN40S-A ports te 0/9-0/0. The network now appears as shown in Figure 0. After the connection is made, the LACP port channel between FN40S-A and the S4048-ON comes up. On FN40S-A, UFD detects port channel 8 is up and brings up the internal, server-facing ports (FN40S-A ports te 0/ and te 0/5 in Figure 0). This causes the corresponding server CNA ports to come up. Step-by-Step Example: After Upgrade FX Chassis FC60 Slot CNA port CNA port CNA port te 0/ te 0/5 te 0/ FN40S-A te 0/9 te 0/0 te /5 te 0/9 Po 8 Po 0 te /6 te /7 te /8 ToR Switch S4048-ON te /9 te /0 Upstream Server port port FC60 Slot CNA port te 0/5 FN40S-A te 0/0 Po 8 Po default VLAN VLAN 6 Both FN40S IOMs installed 5. After the FC60 server's CNA ports are up, a north-south connectivity check is done by pinging the servers in the FX chassis from an upstream device on all applicable VLANs traversing FN40S-A. The upstream server at successfully pings the FX servers at and through the ToR and FN40S-A on VLAN 6. 4 PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

15 6. An east-west connectivity check is done by pinging between the servers in the FX chassis on all applicable VLANs traversing FN40S-A. The FC60 server at successfully pings the FC60 server at via FN40S-A on VLAN 6 without going through the ToR switch. Note: If you want to verify east-west traffic bypasses the ToR switch by shutting port channel 8, disable UFD on the FN40S first. Instructions for disabling and re-enabling UFD are in the appendix of the Dell PowerEdge FN IOM Easy Deployment Guide. All connections have been verified and the upgrade is complete. 5 PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

16 A References and additional resources Dell PowerEdge FN I/O Module (FN IOM) - Easy Deployment Guide Dell TechCenter Networking Home Manuals and Documentation for PowerEdge FX/FXs and Modules PowerEdge FX FN I/O Module VLT Deployment Guide 6 PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

17 B Components used in this guide The components and versions used while developing this guide are shown below. Component Dell 0GbE pass-through module Software Version NA PowerEdge FN40S IOM Dell Networking OS 9.0(0.) Dell Networking S4048-ON ToR switch Dell Networking OS 9.0(0.) PowerEdge FX Chassis Management Controller.0 7 PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

18 C Terminology CNA (Converged Network Adapter): Combines the functionality of a network interface controller and a storage host bus adapter. It connects to storage area networks and local area networks. FN IOA (FN I/O Aggregator): Legacy term for FN IOM. FN IOM (FN I/O Module): FN IOM refers to the switch modules on the rear of the Dell PowerEdge FX/FXs chassis that receive and transmit network IO from the blade servers. This includes the PowerEdge FN40S, FN40T and FN0S. This term does not refer to pass-through modules. LACP (Link Aggregation Control Protocol): LACP is used to control the bundling of several physical ports together to form a single logical channel. LACP allows a network device to negotiate an automatic bundling of links by sending LACP packets to the peer (which is a directly connected device that also implements LACP). NIC Teaming: NIC Teaming allows multiple network interfaces on a computer to be placed into a team to combine bandwidth or provide traffic failover to prevent connectivity loss in the event of a network component failure. Port Channel: Two or more network links bundled together to function as a single link. Port Channels provide increased bandwidth, redundancy and load balancing. ToR (Top of Rack): A switch that is positioned at the top of a server rack in a data center. UFD (Uplink Failure Detection): When upstream connectivity fails, the FN IOM disables the downstream, server-facing links. This feature ensures NIC teaming failovers occur properly if links are lost upstream. The FN IOM uses LACP for the uplink port channel so it is necessary for the corresponding connections at the upstream switch to also be configured as an LACP port channel. 8 PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

19 D FN IOM internal port mapping D. Quarter-width servers with dual-port CNAs The PowerEdge FC40 is an example of a quarter-width server. For quarter-width servers with dual-port CNAs, each CNA port maps to a single port on each of the two FN IOMs. The server slots in the top row are designated a through d, and a through d on the bottom row. See Figure and Table. a b c d a b c d FN IOM A (Top) Internal Ports FN IOM A (Bottom) Internal Ports Server Slot Quarter-width servers with dual-port CNAs Quarter-width servers with dual-port CNAs FN IOM A (Top) Internal Port Numbers FN IOM A (Bottom) Internal Port Numbers a b c d 4 4 a 5 5 b 6 6 c 7 7 d 8 8 Note: Quad-port CNAs are not available for quarter-width servers. 9 PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

20 D. Half-width servers with dual-port CNAs The PowerEdge FC60 is an example of a half-width server. For half-width servers with dual-port CNAs installed, the CNA ports map to a single port on each of the two FN IOMs. Figure and Table present the port mapping for half-width servers with dual-port CNAs. Note: Ports, 4, 6 and 8 are not used when using half-width blades with dual-port adapters. Slot Slot Slot Slot 4 FN IOM A (Top) Internal Ports FN IOM A (Bottom) Internal Ports Half-width servers with dual-port CNAs Server Slot Half-width servers with dual-port CNAs FN IOM A (Top) Internal Port Numbers FN IOM A (Bottom) Internal Port Numbers PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

21 D. Half-width servers with quad-port CNAs For half-width servers with quad-port CNAs installed, the CNA ports map to two ports on each FN IOM. Figure and Table present the port mapping for half-width servers with quad-port CNAs. Slot Slot Slot Slot 4 FN IOM A (Top) Internal Ports FN IOM A (Bottom) Internal Ports Half-width servers with quad-port CNAs Server Slot Half-width servers with quad-port CNAs FN IOM A (Top) Internal Port Numbers FN IOM A (Bottom) Internal Port Numbers,,,4,4 5,6 5,6 4 7,8 7,8 PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

22 D.4 Full-width servers with dual-port CNAs The PowerEdge FC80 is an example of a full-width server. For full-width servers with dual-port CNAs installed, the CNA ports map to two ports on each FN IOM. Figure 4 and Table 4 present the port mapping for full-width servers with dual-port CNAs. SAS/ SATA Slot 0 SAS/ SATA Slot 0 FN IOM A (Top) Internal Ports FN IOM A (Bottom) Internal Ports Full-width servers with dual-port CNAs Server Slot Full-width servers with dual-port CNAs FN IOM A (Top) Internal Port Numbers FN IOM A (Bottom) Internal Port Numbers,, 5, 7 5, 7 PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

23 D.5 Full-width Servers with quad-port CNAs For full-width servers with quad-port CNAs installed, the CNA ports map to four ports on each FN IOM. Figure 5 and Table 5 present the port mapping for full-width servers with quad-port CNAs. SAS/ SATA Slot 0 SAS/ SATA Slot 0 FN IOM A (Top) Internal Ports FN IOM A (Bottom) Internal Ports Full-width servers with quad-port CNAs Server Slot Full-width servers with quad-port CNAs FN IOM A (Top) Internal Port Numbers FN IOM A (Bottom) Internal Port Numbers,,, 4,,, 4 5, 6, 7, 8 5, 6, 7, 8 PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0

24 E Support and feedback Contacting Technical Support Support Contact Information Web: Telephone: USA: Feedback for this document We encourage readers of this publication to provide feedback on the quality and usefulness of this deployment guide by sending an to 4 PowerEdge FX - Upgrading from 0GbE Pass-through Modules to FN40S I/O Modules version.0