> Resilient Data Center Solutions using the VSP 7000 Technical Configuration Guide. Virtual Services Platform Engineering. Avaya Data Solutions

Transcription

1 Virtual Services Platform 7000 Engineering > Resilient Data Center Solutions using the VSP 7000 Technical Configuration Guide Avaya Data Solutions Document Date: Document Number: NN Document Version: 3.0

2 2015 Avaya Inc. All Rights Reserved. Notices While reasonable efforts have been made to ensure that the information in this document is complete and accurate at the time of printing, Avaya assumes no liability for any errors. Avaya reserves the right to make changes and corrections to the information in this document without the obligation to notify any person or organization of such changes. Documentation disclaimer Avaya shall not be responsible for any modifications, additions, or deletions to the original published version of this documentation unless such modifications, additions, or deletions were performed by Avaya. End User agree to indemnify and hold harmless Avaya, Avaya s agents, servants and employees against all claims, lawsuits, demands and judgments arising out of, or in connection with, subsequent modifications, additions or deletions to this documentation, to the extent made by End User. Link disclaimer Avaya is not responsible for the contents or reliability of any linked Web sites referenced within this site or documentation(s) provided by Avaya. Avaya is not responsible for the accuracy of any information, statement or content provided on these sites and does not necessarily endorse the products, services, or information described or offered within them. Avaya does not guarantee that these links will work all the time and has no control over the availability of the linked pages. Warranty Avaya provides a limited warranty on this product. Refer to your sales agreement to establish the terms of the limited warranty. In addition, Avaya s standard warranty language, as well as information regarding support for this product, while under warranty, is available to Avaya customers and other parties through the Avaya Support Web site: Please note that if you acquired the product from an authorized reseller, the warranty is provided to you by said reseller and not by Avaya. Licenses THE SOFTWARE LICENSE TERMS AVAILABLE ON THE AVAYA WEBSITE, ARE APPLICABLE TO ANYONE WHO DOWNLOADS, USES AND/OR INSTALLS AVAYA SOFTWARE, PURCHASED FROM AVAYA INC., ANY AVAYA AFFILIATE, OR AN AUTHORIZED AVAYA RESELLER (AS APPLICABLE) UNDER A COMMERCIAL AGREEMENT WITH AVAYA OR AN AUTHORIZED AVAYA RESELLER. UNLESS OTHERWISE AGREED TO BY AVAYA IN WRITING, AVAYA DOES NOT EXTEND THIS LICENSE IF THE SOFTWARE WAS OBTAINED FROM ANYONE OTHER THAN AVAYA, AN AVAYA AFFILIATE OR AN AVAYA AUTHORIZED RESELLER, AND AVAYA RESERVES THE RIGHT TO TAKE LEGAL ACTION AGAINST YOU AND ANYONE ELSE USING OR SELLING THE SOFTWARE WITHOUT A LICENSE. BY INSTALLING, DOWNLOADING OR USING THE SOFTWARE, OR AUTHORIZING OTHERS TO DO SO, YOU, ON BEHALF OF YOURSELF AND THE ENTITY FOR WHOM YOU ARE INSTALLING, DOWNLOADING OR USING THE SOFTWARE (HEREINAFTER REFERRED TO INTERCHANGEABLY AS "YOU" AND "END USER"), AGREE TO THESE TERMS AND CONDITIONS AND CREATE A BINDING CONTRACT BETWEEN YOU AND AVAYA INC. OR THE APPLICABLE AVAYA AFFILIATE ("AVAYA"). Copyright Except where expressly stated otherwise, no use should be made of the Documentation(s) and Product(s) provided by Avaya. All content in this documentation(s) and the product(s) provided by Avaya including the selection, arrangement and design of the content is owned either by Avaya or its licensors and is protected by copyright and other intellectual property laws including the sui generis rights relating to the protection of databases. You may not modify, copy, reproduce, republish, upload, post, transmit or distribute in any way any content, in whole or in part, including any code and software. Unauthorized reproduction, transmission, dissemination, storage, and or use without the express written consent of Avaya can be a criminal, as well as a civil offense under the applicable law. Third Party Components Certain software programs or portions thereof included in the Product may contain software distributed under third party agreements ("Third Party Components"), which may contain terms that expand or limit rights to use certain portions of the Product ("Third Party Terms"). Information regarding distributed Linux OS source code (for those Products that have distributed the Linux OS source code), and identifying the copyright holders of the Third Party Components and the Third Party Terms that apply to them is available on the Avaya Support Web site: Trademarks The trademarks, logos and service marks ("Marks") displayed in this site, the documentation(s) and product(s) provided by Avaya are the registered or unregistered Marks of Avaya, its affiliates, or other third parties. Users are not permitted to use such Marks without prior written consent from Avaya or such third party which may own the Mark. Nothing contained in this site, the documentation(s) and product(s) should be construed as granting, by implication, estoppel, or otherwise, any license or right in and to the Marks without the express written permission of Avaya or the applicable third party. Avaya is a registered trademark of Avaya Inc. All non-avaya trademarks are the property of their respective owners. Downloading documents For the most current versions of documentation, see the Avaya Support. Web site: Contact Avaya Support Avaya provides a telephone number for you to use to report problems or to ask questions about your product. The support telephone number is in the United States. For additional support telephone numbers, see the Avaya Web site: 2

3 Abstract This document provides configuration procedures for Avaya s Split Multi-Link Trunking and Shortest Path Bridging feature for Virtual Services Platform

4 Table of Contents Figures Avaya Networking Virtual Services Platform 7000 Avaya Fabric Interconnect for the Data Center Fabric Interconnect Fabric Interconnect Stacking Fabric Interconnect Ports Fabric Interconnect Mesh Fabric Interconnect Mesh Optional SPB NNI port Link Metric Fabric Interconnect Cables FI Rear-Port Mode - Configuration Direct Attach Cables PFC-lite SMLT BEB Guidelines Configuration Examples Reference Diagram VSP 7000 FI Mesh VSP 7000 SMLT Cluster VSP 7000 FI Mesh with SMLT Lossless or PFC-lite Configuration VM NIC Teaming Creation of Virtual Switch NIC Teaming Addition of Adaptor Ports to the Virtual Switch VMware NIC Teaming Load Balance Mechanism Reference Documentation

5 Figures Figure 1: Fabric Interconnect Ports... 7 Figure 2 VSP 7000 vtor (Virtual Top-of-Rack) Figure 3: VSP 7000 vtor to SMLT Core Figure 4: SPB Layer 2 Virtualized Service at Edge Figure 5 VSP 7000 dtor (Distributed Top-of-Rack) Figure 6: VSP 7000 dtor to SMLT Core Figure 7: VSP 7000 dtor with SPB on Front Ports Figure 8 FI Stack Rear Ports Figure 9 FI Stack Cabling Example Figure 16 FI Rear Port Details Figure 10: Rear Port set to Raw Mode Figure 11: SPB Deployment Standalone switching Figure 12: SPB Deployment FI Stack Figure 13: FI Mesh Single Switch / Rack Figure 14 FI Mesh Two Switches / Rack Figure 15: FI Mesh Interconnecting Racks Figure 17: Configuration Example Reference Diagram Figure 18: FI Mesh Configuration Example Rear Port Connections Figure 19: FI Mesh Configuration Example Front Port Connections Figure 20: Data Center #2 Configuration Example SMLT Figure 21: VSP 7000 FI Mesh with SMLT

6 Conventions This section describes the text, image, and command conventions used in this document. Symbols Tip Highlights a configuration or technical tip. Note Highlights important information to the reader. Warning Highlights important information about an action that may result in equipment damage, configuration or data loss. Text Bold text indicates emphasis. Italic text in a Courier New font indicates text the user must enter or select in a menu item, button or command: ERS T# show running-config Output examples from Avaya devices are displayed in a Lucida Console font: ERS T# show sys-info Operation Mode: Switch MAC Address: B0-00 PoE Module FW: Reset Count: 83 Last Reset Type: Management Factory Reset Power Status: Primary Power Autotopology: Enabled Pluggable Port 45: None Pluggable Port 46: None Pluggable Port 47: None Pluggable Port 48: None Base Unit Selection: Non-base unit using rear-panel switch sysdescr: Ethernet Routing Switch T-PWR HW:02 FW: SW:v Mfg Date: HW Dev:H/W rev.02 6

7 1. Avaya Networking Virtual Services Platform 7000 Avaya Fabric Interconnect for the Data Center Avaya FI (Fabric Interconnect) provides the new DNA for your Data Centre Network enabling unparalleled scalability and low latency interconnect to address East-West traffic requirements. A significant challenge in many data centers is the growing amount of East-West traffic which requires higher bandwidth between adjacent racks in the data center. Each VSP 7000 features 4 Fabric Interconnect ports as shown in figure 1. Fabric Interconnects Ports Figure 1: Fabric Interconnect Ports The Fabric Interconnect can be used in two mutually exclusive modes: Fabric Interconnect Stacking, in which up to 8 units create a vtor (virtual Top of Rack) or 16 units in a dtor (distributed Top of Rack) delivering up to 10Tbps using two clusters of 8 switches. Fabric Interconnect Mesh, in which the Fabric Interconnect ports operate as multiple highspeed interconnects, allowing the creation of a fully flexible and scalable network mesh. This solution provides high-density 1 or 10 Gigabit Ethernet edge switching for data centers and provides multi-switch resiliency. The highlighted benefits of this solution include: Brings Virtualized Services directly to the Server Featuring Fabric Interconnect which delivers low latency high throughput 5.12Tbps virtual backplane Small form-factor & energy-efficient with front-to-back or back-to-front cooling options Near Zero Down Time Upgrade when deployed as a cluster: All the traffic can be directed through one cluster while the other is being upgraded with minimal interruption of traffic flows. Low Latency Communication: The unique horizontal stacking capability allows very low latency communication amongst devices connected anywhere within the stack. For most communications there is no impact on uplink capacity. This is ideal for rapid communication required for server clustering, as well as for iscsi. High Fault Tolerance: Use of MLT and SMLT redundancy mechanisms allow for negligible latency impact during link failures. Redundancy protocols with significant convergence times such as Spanning Tree and VRRP are no longer required. VENA: Extending the Fabric to the Top of the Rack with Virtual Enterprise Network Architecture Lossless Mode Operation: Advanced high-capacity Switching Fabric which supports flow control and lossless capabilities High Density ToR: Supporting up to Gigabit ports in one virtual chassis. 7

8 Flexible Growth: As the datacenter grows, additional switches, across multiple data cabinets can be added to the stack. New units can be added to the stack without interrupting operations of the current stack. 1.1 Fabric Interconnect The core of the Avaya VSP 7000 is a fifth generation Layer 3 Switching ASIC rated at 1,280Gbps. This provides the Avaya VSP 7000 with incredible capacity to support wire speed I/O and Avaya FI (Fabric Interconnect) Stacking concurrently. The VSP 7000 delivers a new take on the traditional Top-of-Rack Switch requirement. For modest scenarios, switches can be horizontally interconnected, creating a single logical system spanning eight units/racks, or hundreds of VSP 7000s can be flexibly meshed for massive scale-out that uniquely delivers multi-hop and low-latency. Forming a single-tier, Avaya s Distributed ToR is a connectivity solution for the Data Center s primary requirement: high-performance, low- latency, Layer 2 east-west traffic. Utilizing the high-speed virtual backplane capacity, and invoking Ethernet s plug & play advantage, the VSP 7000 empowers simplified, one-touch, edge-only provisioning. Fabric Interconnect can be used in two mutually exclusive modes: Fabric Interconnect Stacking where the rear ports are set as Fabric Interconnect Stack-mode. Up to 8 units create a vtor (virtual Top of Rack) or 16 units in a dtor (distributed Top of Rack) delivering up to 10Tbps using two SMLT clusters of 8 switches. For Fabric Interconnect Stack, the stack operates in the same manner as other Avaya stackable products and features many of the associated benefits of stacking (single IP address for FI stack, hot swap unit replacement, and distributed uplinks with distributed MLT and LAGs). By default, Fabric Interconnect (FI) ports on the rear of the VSP 7000 are configured for Stack-mode. Fabric Interconnect Mesh where the rear ports are configured as rear-port in either Standard (Raw) or SPB modes in which the Fabric Interconnect ports operate as multiple high-speed interconnects, allowing the creation of a fully flexible and scalable network mesh. Depending on the software release, SPB and/or SMLT is supported on the rear-ports as highlighted in the chart below. Standard is a Raw-mode that can support various port configurations and protocols, such as Inter-Switch Trunking (IST) for Switch Clustering and SMLT. Please note that rear-port Standard Mode does not support SPB on the rear ports. 8

9 Desired Deployment Model Needed Rear-port Mode SMLT (IST) Needed SPB enabled Virtual Servers (e.g. ESX) NIC teaming Server NIC teaming (LACP) Minimum Required Software vtor FI Stacking Disabled (= Stacking enabled) No Can be Yes Vport hashing on non- SLT ports Yes On DMLT ports (with or without LACP) ( if need to run SPB on uplinks) dtor FI Stacking with SMLT Disabled (= Stacking enabled) Yes Can be Yes Vport hashing on non- SLT ports - OR - IP hashing on SLT ports Yes On SLT ports (with or without LACP) ( if need to run SPB on uplinks & IST) FI Mesh with SMLT Enabled in raw mode Yes No Yes Vport hashing on non- SLT ports - OR - IP hashing on SLT ports Yes On SLT ports (with or without LACP) SPB Mesh Enabled in SPBM mode No Yes Yes Vport hashing on non- SLT ports No Use Active Standby NICs SPB Mesh with SMLT Enabled in SPBM mode Yes Yes Yes Vport hashing on non- SLT ports - OR - IP hashing on SLT ports Yes On SLT ports (with or without LACP) In Fabric Interconnecting Stacking, with SPB enabled, supports a maximum stack of 2; in the 10.3 release, a stack of 8 is supported. SMLT or IST over rear port Raw-mode is supported staring in release , however, SPB is not supported in rear port Raw-mode. LACP must be disabled on the rear ports prior to enabling an IST on them. In rear port SPB mode, IP routing cannot be enabled and the total number of rear ports is reduced by one - the switch uses port 40 as loopback when rear port SPB is enabled; please see section for rear port numbering. 9

10 1.1.1 Fabric Interconnect Stacking Virtual Top-of-Rack Fabric Interconnect Stacking Virtual Fabric Interconnect stacking (vtor), available starting in software release 10.1, provides stacking for up to 8 switches with support for a total of GE ports. The servers can be attached to two or more different switches in the vtor to provide dual homed active/active operations. Figure 2 VSP 7000 vtor (Virtual Top-of-Rack) Connectivity from each server to the VSP 7000 vtor stack can be made up of 1 to 8 GigE or 10GigE ports with or without VLAN tagging spread across a vtor stack of up to 8 units. The VSP 7000 vtor will typically be provisioned with multiple VLANs to support native VLAN transport between the edge and core as shown in figure 3 below. The core layer can either be a SMLT cluster made up of, for example, two ERS 8000 switches or two VSP 9000 switches. The core SMLT cluster provides true active /active resilience where any type of link or core switch failure will have little to no impact on data traffic. In addition, all the links between the vtor and core are active /active with no requirement to run Spanning Tree. The core / aggregation layer will either transport the L2 VLANs or route the traffic. Typically the SMLT cluster will route the traffic using either Routed Split Multi-Link Trunking Edge (RSMLT Edge) or using VRRP with Backup Master. If there is a requirement to transport a VLAN, for example from one data center to another to allow services like vmotion, the VLAN must be either provisioned on all switches in the path between the two data centers or provisioned as a L2VSN on the interconnecting SPB switches. Either way VRRP with Backup Master will be used at the two data centers with a common VRRP VIP to provide routing to the rest of the network and, the edge to core connections are all active/active routed or switched. 10

11 Figure 3: VSP 7000 vtor to SMLT Core If we have an SPB core and a Layer 2 virtualized service is required between data centers, with SPB provisioned on the edge VSP 7000 as shown in figure 4 below, we can add a L2 virtual service (L2VSN) within minutes by only having to provision the edge VSP 7000 switches. All that is required is simply adding a VLAN with local port members and adding a service identifier (I-SID) on the VSP 7000 switches. As long as the I-SID matches between two or more VSP 7000 switches irregardless of the local VLAN identifier used, we will have L2 connectivity. Further, if L3 routing is required, a VLAN with the same I-SID can be configured on any core or distribution SPB node that supports IP routing allowing for routing to the data center either via the native routing table or a VRF. If redundancy is required, simply configure the same I-SID on another core or distribution node and enable VRRP with backup master. If Avaya s Communication and Orchestration Manager (COM) is used, we can use the VLAN and VSN managers to provision a L2VSN on two or more switches at the same time instead of using the CLI to provision each switch. Figure 4: SPB Layer 2 Virtualized Service at Edge 11

12 Distributed Fabric Interconnect Stacking Starting in the 10.2 software release, dtor is supported as shown in figure 5. Up to 16 switches can be deployed in a dtor supporting a total GE ports. The servers can be attached to two or more different switches in each SMLT cluster providing dual homed active/active operations with the added protection of a stack failure. dtor allows one to upgrade a stack or take a stack out of commission without impacting services. SPB is supported in the dtor starting in the with a stack of two while in the 10.3 release, dtor with a stack of eight switches is supported. Please note that SPB is only supported on the front ports as the rear ports are operating in stack mode. Figure 5 VSP 7000 dtor (Distributed Top-of-Rack) Connectivity from each server to the VSP 7000 dtor stack can be made up of 1 to 8 GigE or 10GigE ports with or without VLAN tagging spread across a dtor stack of up to 16 units. The benefit of using a dtor over a vtor is the server to edge and edge to core connectivity is now spread across two stacks protecting against any type of VSP 7000 stack or link failures. The VSP 7000 dtor will typically be provisioned with multiple VLANs to support native VLAN transport between the edge and core. If the core / aggregation layer is SMLT as shown in figure 6 below, you can either transport the L2 VLANs or route the traffic. Typically the SMLT cluster will route the traffic using either Routed Split Multi-Link Trunking Edge (RSMLT Edge) or using VRRP with Backup Master. If there is a requirement to transport a VLAN, for example from one data center to another to allow services like vmotion, the VLAN must be provisioned on all switches in the path between the two data centers where VRRP with Backup Master will be used at the two data centers with a common VRRP VIP to provide routing to the rest of the network. When IST is enabled, Avaya recommends using SMLT towards the Server for seamless failovers. 12

13 Figure 6: VSP 7000 dtor to SMLT Core In the release (stack of 2) or 10.3 releases (stack up to 8) the dtor can use SPB to connect to the core as shown in figure 7 below. In this case, SPB L2VSNs are supported on the VSP This makes it very easy to extend a VLAN between data centers as the C-VLAN, local port members, and SPB I-SID identifier are only required to be configured on the SPB bridges that requires the service. If there is a need for routing, the same L2VSN I-SID can be provisioned on any SPB node that supports IP Shortcuts or L3VSNs such as the VSP 9000, VSP 4000, or ERS If redundancy is required, simply configure the same I-SID on another VSP 9000, VSP 4000, or ERS 8000 and enable VRRP with backup master. Figure 7: VSP 7000 dtor with SPB on Front Ports 13

14 Fabric Interconnect Ports - Stacking When operating in FI stack mode, consider each pair of FI ports as a single connection. You must use a pair of FI ports (two FI-up or two FI-down) and two FI cables connected to an adjacent unit s FI ports to establish adjacency between switches in a vtor. Figure 8 FI Stack Rear Ports The following diagram illustrates the FI cabling for a 3-unit and a 4-unit stack where every second unit is stacked. Cabling is connected from Fabric Interconnect Up (pair of FI cables) to Fabric Interconnect Down (pair of FI cables). Note that each pair of up/down ports should be thought of as a single connection. Figure 9 FI Stack Cabling Example Fabric Interconnect Stacking Benefits Fabric Interconnect stacking provides the following benefits: Fault-tolerant Load sharing NIC teaming into stack Up to 8 units create a vtor (virtual top-of-rack) delivering up to 5Tbps using two FI cables in parallel between switches for a total of GE ports Up to 16 units in a dtor (distributed top-of-rack) delivering up to 10Tbps using two clusters of 8 switches for up to GE ports High bandwidth and low latency between servers (2.1µs 5.3us) Highly resilient stacking technology with scalable uplinks Flexibility to spread across multiple data cabinets (100s of servers) Ideal for Grid Computing / High-Performance Computing Solutions 14

15 1.1.2 Fabric Interconnect Ports The VSP 7000 by default operates in Fabric Interconnect stacking mode. The VSP 7000 can be provisioned in rear-port mode where the rear Fabric Interconnect ports will be treated as multiple virtual ports over the 4 physical Fabric Interconnect Ports. When in rear-port mode, the VSP 7000 operates in a standalone mode. Two modes of operation are available in rear-port mode, standard or Shortest Path Bridging (SPB). Standard mode allows all the switch standard features minus SPB across the rear ports, i.e. Spanning Tree, OSPF, RIP, etc. In SPB mode, in the 10.2 release Shortest Path Bridging is supported while in the 10.3 both SPB and SMLT will be supported. Hence, when FI Mesh is required, rear-port mode with operational state of SPB needs to be provisioned. The diagram shows the FI port speeds available depending if Standard or SPB operational state is enabled. To provide greater plug n play capability over the virtual ports when rear-port mode is enabled, LACP link aggregation and VLAN tagging are automatically enabled. This ensures that multiple virtual ports which may run within a single cable or if multiple FI cables are run in parallel that all virtual ports are automatically treated as one link. This simplifies any protocol adjacency such as IS-IS or OSPF. When you issue rear-ports mode all virtual ports will have their LACP state set to true, the LACP Admin Key to 4095 and LACP hashing mode be set to advance. Color Physical Fabric Interconnect Port Rear Port Mode Throughput Ports Black FI Up (right) Top Standard 240Gbps (120 FDX) SPB 240Gbps (120 FDX) 34, 35, 36 Red FI Down (left) Top Standard 240Gbps (120 FDX) 38, 39, 40 SPB 160Gbps (80 FDX) 38, 39 Blue FI Up (right) Bottom Standard 80Gbps (40 FDX) SPB 80Gbps (40 FDX) Blue FI Down (left) Bottom Standard 80Gbps (40 FDX) SPB 80Gbps (40 FDX) Figure 16 FI Rear Port Details In FI mesh, it is recommended to connect like color fabrics interconnect ports together, i.e. red port to an adjacent switch red port to get maximum possible throughput. You can connect any color ports together, i.e. a red port to a blue port, however, the port speed will drop to the lower of two ports. In rear-port SPB operational state, virtual port 40 is not available. Hence, the red port is reduced to 160Gbps. In rear port mode, the front panel Up and Down LEDs blink in a quick pattern (125ms) to 15

16 indicate rear-port mode is operational. In the 10.2 release, only SPBM is officially supported in rear port SPB mode. As of SMLT is supported on rear ports. 16

17 1.1.3 Fabric Interconnect Mesh SMLT - Rear Port set to Standard (raw) The rear ports on the VSP 7000 can operate in either stacking mode or rear port mode. Further, in rear port mode, it can operate in one of two modes of operation, standard (raw) or Shortest Path Bridging (SPB). In standard mode of operation, all protocols minus Shortest Path Bridging (SPB) are supported while in SPB mode of operation, SPB is supported. If the rear port is set in standard mode, starting with the release, the rear ports can be used for SMLT. For example, a SMLT square could be provisioned between VSP 7000 cluster switches using the rear ports for the IST on each IST cluster pair and also using the rear ports for the SMLT connection to the other VSP 7000 cluster. This provides a high-speed pipe for both east-west and north-south traffic by simply using stacking cables for both the IST and SMLT ports instead of expensive front panel SFP+ modules and fiber optic cabling. This solution offers very high speed interfaces (80G or 240G) for the IST and SMLT ports while allowing for all up to 32 GigE or 10GigE front panel ports for connectivity to servers or other devices please see section for more details on rear port numbering and port speeds. Please note the following when setting the rear port mode to standard: SPB is not supported when the rear port mode is set to standard (raw) mode The IST must be made up of at minimum two ports, i.e. use both rear port blue ports or you could use either the red or black rear ports as each has 3 ports each The IST must be made up of at minimum two ports, i.e. use both rear port blue ports (ports 33 & 37) or you could use either the red rear ports (ports 38-40) or black rear ports (ports 34-36) o o o If you use the blue rear ports (ports 33 & 37) between two cluster switches, two stacking cables are required If you use the red ports (ports 38-40) between two cluster switches, only one stacking cable is required If you use the black ports (ports 34-26) between two cluster switches, only one stacking cable is required LACP must be disabled prior to enabling IST or SMLT using the rear ports SMLT square and triangle topologies are supported in rear port raw mode; SMTL full mesh is not supported 17

18 Figure 10: Rear Port set to Raw Mode 18

19 Fabric Interconnect (FI) Mesh Rear Port set to SPB Avaya s FI mesh delivers unmatched scalability to address low latency East-West requirements in the Data Centre. Each switch operates as an individual switch connected together via the rear port FI ports with the option of supporting SMLT IST (10.3 release or higher) in rear port SPB mode. The advantage of using FI mesh over FI Stacking is that FI stacking is limited to two SMLT stacks of 8 units each for a total of 16 switches while FI mesh allows one to scale from 4 units up to 480 switches offering up to 15,360 10GE ports with 262.5Tbps Fabric ToR Fabric Interconnect Mesh (Shortest Path Bridging) Fabric Connect (Shortest Path Bridging) was added to the VSP 7000 starting in the 10.2 software release in standalone mode only as illustrated in figure 11. In the 10.2 release, FI Mesh is supported on the rear ports allowing high speed SPB NNI connectivity (80-640Gbps) between VSP 7000s switches by just using stacking cables between VSP 7000 switches instead of expensive front panel SFP+ modules with fiber optic cabling. By using FI Mesh, one can take advantage of the FI Mesh bandwidth for east-west traffic between VSP 7000 switches. Please note that release 10.3 is required to support SPB and SMLT simultaneously. In FI mesh mode without SMLT, the servers connect to the VSP 7000 SPBM UNI interface either via a single port or via NIC teaming using MLT or LACP to a single VSP If the servers are VMware ESX, NIC teaming to different VSP 7000 s can be deployed without requiring either MLT or SMLT providing Port ID or MAC based load balancing is used where each VM is allocated to one of the ESX NICs for traffic in both directions. The SPB NNI from the VSP 7000 connects to the core SPB nodes either using a single port or an aggregated link using MLT or LACP. The VSP 7000 will terminate Layer 2 virtualized service (L2VSN) where the SPB core switches can be provisioned to provide IP connectivity for the Layer 2 virtualized service. For applications such as vmotion, the core switches can be provisioned with VRRP backup master with a common VIP. Figure 11: SPB Deployment Standalone switching 19

20 In the 10.3 release, Shortest Path Bridging (SPB) is supported with SMLT and you can configure both an SMLT IST and SPB on the same ports. This will allow the servers or switches to be dual homed to two VSP 7000 switches providing active/active link aggregation at the SPB UNI interface level. Please note the following when enabling SPB on the VSP 7000: Only SPB L2VSNs are supported o For routing, the L2VSN must be terminated on at least one VSP 9000, VSP 4000, or ERS 8800 that either has IP Shortcuts or L3VSN enabled; it is recommended to terminate the L2VSN on a second VSP 9000, VSP 4000, or ERS 8800 and use VRRP with backup master The IST must be made up of at minimum two ports, i.e. use both rear port blue ports (ports 33 & 37) or you could use either the red rear ports (ports 38 & 39) or black rear ports (ports 34-36) o o o If you use the blue rear ports (ports 33 & 37) between two cluster switches, two stacking cables are required If you use the red ports (ports 33 & 37) between two cluster switches, only one stacking cable is required Please note that one of the red ports (port 40) is used as a SPB loopback port, hence, the number of ports is reduced by one If you use the black ports (ports 34-26) between two cluster switches, only one stacking cable is required LACP must be disabled prior to enabling IST on the rear ports Figure 12: SPB Deployment FI Stack 20

21 Fabric Interconnect Mesh Mesh Architecture Reference The following diagram illustrated the reference in regards to the recommended FI cabling please see section titled Fabric Interconnect Ports Rear Port Mode for more details on port numbering and speeds. Although any FI rear port can be connected to any adjacent FI port, it is recommended to connect likecolored FI ports to get the maximum possible speed. Black Interconnect (240Gbps) Interconnect top switch in rack N to top switch in adjacent (N-1) rack Interconnect bottom switch in rack N to bottom switch in N+1 rack Red Interconnect ( Gbps) Interconnect top switch in rack N to top switch in adjacent (N+1) rack Interconnect bottom switch in rack N to bottom switch in N-1 rack Blue Interconnect (80Gbps) Interconnect 2 switches in same rack (could be leveraged for IST in future release) Interconnect top switch to top switch in adjacent row in DC Interconnect bottom switch to bottom switch is adjacent row in DC (The connection to adjacent rows are shown in parallel in this reference design. These connections could be run from top switch in switch in row N to top switch in row N-1 and then bottom switch in row N to bottom switch in row N+1) 21

22 FI Mesh ToR Single switch / rack Figure 13 as shown below shows a FI Mesh deployed as Top-of-Rack. The servers can be attached to single switch to provide dual homed active/active operations or two switches in the FI mesh in an active/passive mode or active/active mode. Without an IST, for active/active to more than one VSP 7000, if the servers are VMware ESX with NIC teaming, you can connect the servers to different VSP 7000 s without requiring either MLT or SMLT providing Port ID or MAC based load balancing is used where each VM is allocated to one of the ESX NICs for traffic in both directions. If an IST is provisioned between a pair of VSP 7000 switches, if the servers are VMware ESX with NIC teaming, you can connect the servers to each VSP 7000 in the cluster where it is recommended to use IP Hash load balancing. Figure 13: FI Mesh Single Switch / Rack The following diagram illustrates several methods for rear port mode cable connectivity in a 4-unit FI Mesh. The VSP 7000 rear panel red-to-red and black-to-black can be connected in any manner. 22

23 FI Mesh Two Switches / Rack Figure 14 as shown below shows a FI Mesh deployed as a dual Top-of-Rack or Top-of-Rack and Bottomof-Rack. The servers can be attached to single switch to provide dual homed active/active operations to two switches in the FI mesh in an active/passive mode or active/active mode if VMware ESX servers. Figure 14 FI Mesh Two Switches / Rack The following diagram illustrates rear port mode cable connectivity in an 8-unit FI Mesh with one SMLT cluster using both blue ports as the IST via the left two VSP 7000 switches. 23

24 FI Mesh ToR Interconnecting Racks The following diagram illustrates another example of FI mesh interconnecting switches in different racks. By using Fiber Fabric Interconnect Transceiver and Cables, a connection up to 100 meters is possible. Figure 15: FI Mesh Interconnecting Racks 24

25 FI Mesh Benefits FI provides the following benefits: FI mesh SPB scalability from 4 to 480 switches o o o o Providing low latency fabric. Fabric is scalable up to 262.5Tbps. Extensible up to 15,360 10GE ports. Designed to enable either 24/32/48/56/64 10GE ports per rack Fabric Interconnect ports interconnect each VSP7000: o o Rear port o o o Can use mix of different Fabric Interconnect cable lengths. Optimum performance will be achieved when the same color designated ports are connected to one another The rear ports provide up 640Gbps of connectivity LACP and VLAN tagging is automatically enabled for the rear ports The 4 physical Fabric Interconnect ports provide different bandwidth capabilities (see rear-ports feature for more details). o Three different colors are used to represent the different capabilities. Optimum performance will be achieved when the same colored ports are connected to one another. 25

26 1.1.4 Fabric Interconnect Mesh Optional SPB NNI port Link Metric You can configure the link metric to overwrite the default metric value. By configuring the metric, you can specify a preferred path. Low cost reflects high-speed media, and high cost reflects slower media. For the wide metric, the value ranges from 1 to 16,777,215. SPBM uses the L1-SPB metric defined in a new SPB sub-tlv The total cost of a path equals the sum of the cost of each link If a link has different metric values configured at each end of the link, SPBM will use the highest metric value The default value for wide metrics is 10 As an option, you can change the wide metric to the suggested values as shown in the table below to allow the switch to prefer the higher speed NNI links over the lower speed links. Link Speed (Gbps) Interface Type ISIS L1-metric 1 Native Ethernet MLT bundle Native Ethernet MLT bundle Native Ethernet MLT bundle / FI Native Ethernet MLT bundle / FI MLT bundle 13 By default, all Fabric Interconnect ports operating in rear-port mode use the same LACP key (4095). If you modify a rear-port metric, such as the SPBM-L1 Metric, the modification applies to all ports which have the same LACP key. If different metrics are to be used on specific rearports, you will need to set different LACP keys on those ports. 26

27 1.1.5 Fabric Interconnect Cables Fabric Interconnect cables are available in various lengths. The following FI cables are supported: Order Code Description Copper Fabric Interconnect Cables AL E6 AL E6 AL E6 AL E6 VSP 7000 Fabric Interconnect 0.6 m VSP 7000 Fabric Interconnect 1.5 m VSP 7000 Fabric Interconnect 3 m VSP 7000 Fabric Interconnect 5 m Fiber Fabric Interconnect Cables AL E6 VSP 7000 Active Fabric Interconnect 10 m Fiber Fabric Interconnect Transceiver and Cables AL E5 AL E5 VSP 7000 Fabric Interconnect Transceiver (two required for each Fiber Fabric Interconnect cable) VSP 7000 Fiber Fabric Interconnect cable 50 m AL E5 VSP 7000 Fiber Fabric Interconnect cable 100 m The fiber fabric interconnect transceiver requires either the 50 meter or 100 meter cable. Two transceivers are required per cable and must be direct connected, i.e. it is not designed to allow patching of fiber run through a patch panel. 27

28 1.1.6 FI Rear-Port Mode - Configuration Rear-port mode should be provisioned first prior to provisioning the VSP When changing from FI stacking to rear-port mode, the switch configuration will be reset to partial defaults and will require a reboot of the switch. When you enable rear port mode, the switch applies the following default settings to all FI ports on the rear of the chassis: VLAN tagging for rear ports is set to tagall. The LACP administration key is set to CLI Syntax: The LACP operating mode for rear ports is set to active. The LACP rear ports time-out value is set to short. LACP for rear ports is set to enable. rear-port mode [normal spbm] show rear-port mode CLI Example: Operational mode of standard (raw) VSP7000> ena VSP7000# config term Enter configuration commands, one per line. End with CNTL/Z. VSP7000(config)# rear-port mode enable Enabling rear port mode will disable Fabric Interconnect Stack operation. Switch configuration will be reset to partial-defaults. Continue(yes/no)?y Formatting NVRAM: for DOSFS Formatting...OK. VSP7000> enable VSP7000# config term Enter configuration commands, one per line. End with CNTL/Z. VSP7000(config)# show rear-port mode Rear Port Mode: Enabled Normal Rear Port Operational State: Operational Normal VSP7000(config)# no rear-port mode enable Disabling rear port mode will enable Fabric Interconnect Stack operation. Switch configuration will be reset to partial-defaults. Continue(yes/no)?y Formatting NVRAM: for DOSFS Formatting...OK. CLI Example: Operational mode of SPB VSP7000> ena VSP7000# config term Enter configuration commands, one per line. End with CNTL/Z. 28

29 VSP7000(config)# show rear-port mode Rear Port Mode: Disabled Rear Port Operational State: Not Operational VSP7000(config)# rear-port mode enable spbm Enabling rear port mode will disable Fabric Interconnect Stack operation. Switch configuration will be reset to partial-defaults. Continue(yes/no)?y Formatting NVRAM: for DOSFS Formatting...OK. VSP7000> enable VSP7000# config term Enter configuration commands, one per line. End with CNTL/Z. VSP7000(config)# show rear-port mode Rear Port Mode: Enabled SPB Rear Port Operational State: Operational SPB CLI for LACP VSP7000(config)# show lacp agg Aggr ID Trunk Status Type Members Enabled LA Enabled LA VSP7000(config)# show lacp debug member Port AggrId TrunkId Rx State Mux State Partner Port Port AggrId TrunkId Rx State Mux State Partner Port PortDisabled Detached Current Ready Current Ready Current Ready PortDisabled Detached Current Ready Current Ready 39 CLI for Interface and VLAN Tagging VSP7000# show interfaces Status Auto Flow Port Trunk Admin Oper Link LinkTrap Negotiation Speed Duplex Control Enable Down Down Enabled Disabled 40Gbps Full Disable 29

30 34 32 Enable Up Up Enabled Disabled 40Gbps Full Disable Enable Up Up Enabled Disabled 40Gbps Full Disable Enable Up Up Enabled Disabled 40Gbps Full Disable 37 Enable Down Down Enabled Disabled 40Gbps Full Disable Enable Up Up Enabled Disabled 40Gbps Full Disable Enable Up Up Enabled Disabled 40Gbps Full Disable VSP7000# show vlan interface info Filter Filter Untagged Unregistered Port Frames Frames PVID PRI Tagging Name No Yes 1 0 TagAll Port No Yes 1 0 TagAll Port No Yes 1 0 TagAll Port No Yes 1 0 TagAll Port No Yes 1 0 TagAll Port No Yes 1 0 TagAll Port No Yes 1 0 TagAll Port VSP7000#show vlan interface vids Port VLAN VLAN Name VLAN VLAN Name VLAN VLAN Name BVLAN BVLAN BVLAN BVLAN BVLAN BVLAN BVLAN BVLAN BVLAN BVLAN BVLAN BVLAN

31 1.2 Direct Attach Cables The following devices have been validated by Avaya to function and carry traffic when connected to the Virtual Services Platform Using Avaya devices is recommended to provide maximum compatibility and support for the VSP Vendor Model Details Cisco Cisco SFP+ DIRECT ATTACH CABLE 5m Cisco Cisco SFP+ DIRECT ATTACH CABLE 3m Cisco Cisco SFP+ DIRECT ATTACH CABLE 5m HP J9281B HP SFP+ direct attach cable 1 m HP J9283B HP SFP+ direct attach cable 3 m HP J9285B HP SFP+ direct attach cable 7 m HP J9286B HP SFP+ direct attach copper cable 10 m VSP 7000 Series Feature Pack Release adds validation of the following third party SFP + devices: Vendor Model Details HP B21 HP Blade Systems C-Class SFP+ direct attach cable 1 m HP B21 HP Blade Systems C-Class SFP+ direct attach cable 3 m HP B21 HP Blade Systems C-Class SFP+ direct attach cable 5 m 31

32 1.3 PFC-lite In the release, PFC-lite is supported offering a simplified version of Priority Flow Control (PFC). PFC is a lossless QoS mode that provides a mechanism to stop ingress traffic of a given packet priority. Full PFC is described in IEEE 802.1Qbb. In lossy (best-effort) mode: The packet buffers used by the egress queues are over-allocated depending on the - qos agent buffer setting of: regular, large and maximum. When an egress queue buffer fills up, additional packets destined for that egress queue are taildropped until the egress queue becomes only half-full. The egress queue will empty as packets egress the port. This cell over-allocation allows for better packet buffer utilization when the egress queues on some ports temporarily back up due to momentary oversubscription. Lossy mode is the default mode of operation, with buffer allocation typically set to large with between 2 and 4 queues. In Lossless (Pause Frame) Mode: The packet buffer usage is tracked by each packet ingress port. When an ingress threshold is reached, a PAUSE frame is generated on that ingress port (back to the link partner that is generating the packets). Pause frames continue to be generated until the ingress port packet buffer usage drops to half of the Pause threshold The ingress port cell counter will decrement as packets egress the switch. The egress buffer settings are maximized so that no packets are dropped inside the switch due to the egress queues being filled to capacity. When in Lossless (Pause Frame) mode, the VSP 7000 reacts to Pause frames received from a link partner by stopping transmitting of packets on that front-panel port. Please note that Lossless mode is only supported on standalone switches, it is not supported using DMLT/SMLT on a stack configuration. In Lossless-PFC Mode: The packet buffer usage is tracked by packet priority per ingress port, with two priority groups (PG0 & PG1) being tracked per ingress port. Packets with 802.1P value of 3 or corresponding Class of Service mapping are placed in PG1, all other traffic is placed in PG0. Each priority group ingress packets are counted separately. When the PG1 threshold is reached, a PFC frame that stops PG1 packets is generated on that ingress port (back to the link partner that is generating the packets). When the PG0 threshold is reached, a PFC frame that stops all other packets is generated on that ingress port. PFC frames continue to be generated until the PG1 or PG0 ingress packet buffer usage drops to half of the PG1 or PG0 PFC threshold. 32

33 The ingress PG1 and PG0 cell counters will decrement as PG1 and PG0 packets egress the switch. The egress buffer settings are maximized so that no packets are dropped inside the switch due to the egress queues being filled to capacity. Please note that Lossless-PFC mode is only supported on standalone switches, it is not supported using DMLT/SMLT on a stack configuration. Please note, as of release 10.3, when SPBM is enabled, the default buffer mode is spb-scaled. Use this parameter to select an amount of resource sharing specifically designed for large SPBM configurations. When you enable SPBM, this mode is automatically selected; when you disable SPBM, the configuration reverts to the default of large. If you upgrade from Release and SPBM is enabled, you must manually run the qos agent buffer spb-scaled command. 1.4 SMLT BEB Guidelines For the VSP 7000, it is important to not enable the filter-untagged-frame option on the IST port members. The default PVID of all IST ports must be the primary B-VLAN ID. You can check the default PVID by entering the ACLI command show vlan interface info <port list>. Beginning in the 10.4 release, the primary SMLT BEB switch will use the secondary BVLAN as PVID while the secondary BEB will use the primary BVLAN as PVID. The primary SMLT BEB will change the default PVID to the secondary PVID. Prior to the 10.4 release, both BEB s used the primary BVLAN as the PVID. Also, it is recommended to not enable VLACP on the IST. 33

34 1.5 Configuration Examples In reference to figure 17 as shown on the next page, this configuration example will include both SMLT and SPB configuration for the VSP A SPB FI mesh provision on the VSP 7000 bridges in Data Center #1 while SMLT will be provisioned on the VSP 7000 switches in Data Center #2. In summary, this configuration example will be broken down into three sections: 1. VSP 7000 FI Mesh These bridges will be used in Data Center #1 for vmotion and local services Made up of VSP 7000 SPB bridges named 7001, 7002, 7003, and 7004 provisioned as a SPB FI Mesh An L2VSN will be provisioned for vmotion to allow for layer 2 connectivity between Data Center #1 and Data Center #2 Section will cover FI Mesh while section will cover FI Mesh with SMLT 2. VSP 7000 SMLT Cluster These switches will be used in Data Center #2 for vmotion Made up of VSP 7000 switches named 7010 and 7011 o o SMLT will be provisioned for the ESX servers for active / active resilience The vmotion VLAN will be provisioned with the same L2VSN I-SID used by the VSP 7000 FI Mesh layer for connectivity between Data Center #1 and Data Center #2 3. ERS 8800 nodes 8005 and 8006 Made up of ERS 8800 SPB bridges named 8005 and 8006 providing routing to the two data centers for both in-band mgmt to the VSP 7000 s and routing to the IP network for the vmotion VLAN o o o Note any SPB node that support SPB IP routing anywhere in the network could be used where, for this example, we simply selected SPB nodes 8005 and 8006 Terminate the same L2VSN I-SIDs used at Data Center #1 and Data Center #2 with IP Shortcuts enabled allowing for routing to the rest of the network The L2VSN I-SID could just as easily been added to a VRF, but, for this example, we will provide routing to the data centers via the native GRT routing table VRRP backup master resiliency added to both 8005 and 8006 for added redundancy in case of a node failure Also, we will provision the out-of-band management port on the VSP 7000 and show how to provision inband management using an L2VSN. As of release 10.2, both the out-of-band and in-band management interfaces can be used. Please refer to the Switch Clustering using SMLT Technical Configuration Guide, publication number NN , in reference to an configuration example using the VSP 7000 with SMLT clustering using the rear port for both SMLT and IST. 34

35 1.6 Reference Diagram In summary, we will provision the following: Data Center #1 Figure 17: Configuration Example Reference Diagram o Will be provisioned in FI Mesh (rear port set to SPB) using a total of four VSP 7000 switches Section will cover FI mesh with four standalone switches while will cover FI mesh with two SMTL cluster Data Center #2 o Will be provisioned in dtor with SPB although only two switches are used, a total of 16 switches is supported using two stacks of up to 8 switches each 35

36 1.6.1 VSP 7000 FI Mesh For this configuration example, we will configure the FI mesh switches 7001, 7002, 7003, and 7004 as shown in figure 18. Port 3 on switches 7001 and 7003 will be used to connect to the core SPB backbone core bridges (BCB) 8005 and We will also show how to provision the out-of-band management port and using an SPB L2VSN for in-band management. Figure 18 shows the FI rear port cabling used in this configuration example using the FI black ports (ports 34-36) and FI red ports (ports 38 & 39). We will use the default LACP LAG on all switches and change the default LACP key on the FI black port between 7001 and 7004 from 4095 to 4094 to allow for different ISIS L1-metrics to be used. Figure 18: FI Mesh Configuration Example Rear Port Connections Figure 19 shows the front port connections to each server and shared storage assuming each server is a VMware ESXI server with NIC teaming. VLAN 1600 will used for vmotion and shared storage for the ESXi servers while VLANs 500 and 501 will be used for the services VLANs. Figure 19: FI Mesh Configuration Example Front Port Connections 36

37 Set rear port mode to SPB Switch Parameter Value Rear Port 7001, 7002, 7003, 7004 Rear port mode Enabled & SPB Enable rear-port mode on all switches 7024XLS(config)#rear-port mode spbm Enabling rear port mode will disable Fabric Interconnect Stack operation. Switch configuration will be reset to partial-defaults. Continue(yes/no)?y 7024XLS#show rear-port mode Rear Port Mode: Enabled SPB (Loopback Port Reserved) Rear Port Operational State: Operational SPB (Loopback Port Reserved) Option: Change Spanning Tree mode to MSTP If using the VSP 9000 or VOSS switch in the core and using tools such as VLAN Manager in COM, it is recommended to change the Spanning Tree mode to MSTP. Presently VLAN Manager manages VLANs based on spanning tree groups. Option change spanning mode to MSTP on all switches 7024XLS(config)#spanning-tree mode mst New operational mode MSTP will take effect upon reset 7024XLS(config)#boot Reboot the unit(s) (y/n)? yrebooting XLS#show spanning-tree mode Current STP Operation Mode: MSTP Next STP Operation Mode: MSTP System Name Configure system name Switch 7001: 7024XLS(config)#snmp-server name 7001 For switches 7002, 7003, and 7004, use the configuration as shown below 37

38 7024XLS(config)#snmp-server name XLS(config)#snmp-server name XLS(config)#snmp-server name Configure Out-of-Band Management Interface and static route Configure management address and add route 7001(config)#ip mgmt address switch netmask (config)#ip mgmt route (config)#ip mgmt address switch netmask (config)#ip mgmt route (config)#ip mgmt address switch netmask (config)#ip mgmt route (config)#ip mgmt address switch netmask (config)#ip mgmt route (config)#show ip Bootp/DHCP Mode: BootP When Needed Configured In Use Last BootP/DHCP Stack IP Address: Switch IP Address: Switch Subnet Mask: Mgmt Stack IP Address: Mgmt Switch IP Address: Mgmt Subnet Mask: Mgmt Def Gateway: Default Gateway: (config)#show mgmt-port status Port Link

39 1 Up (config)#show ip mgmt route Destination IP Subnet Mask Gateway IP Status ACTIVE SPB Configuration Switch Parameter Value SPB 7001, 7002, 7003, 7004 B-VLANs 4051, 4052 where 4051 is the primary B-VLAN VLAN Names BVLAN-1 and BVLAN-2 IS-IS Area IS-IS Enable SPBM Enable, using instance 1 IS-IS Area SPB Nick Name SPB System-Name SPB Nick Name SPB System-Name SPB Nick Name SPB System-Name SPB Nick Name SPB System-Name 7004 VLAN Change configuration control to automatic 7001(config)#vlan configcontrol automatic 39

40 SPBM Configuration 7001(config)#vlan create 4051 name BVLAN-1 type spbm-bvlan 7001(config)#vlan create 4052 name BVLAN-2 type spbm-bvlan 7001(config)#spbm 7001(config)#router isis 7001(config-isis)#spbm (config-isis)#spbm 1 b-vid primary (config-isis)#spbm 1 nick-name (config-isis)#manual-area (config-isis)#sys-name (config-isis)#exit 7001(config)#router isis enable For switches 7002, 7003, and 7004, use the same configuration as above except for the items shown below 7002(config-isis)#spbm 1 nick-name (config-isis)#sys-name (config-isis)#spbm 1 nick-name (config-isis)#sys-name (config-isis)#spbm 1 nick-name (config-isis)#sys-name

41 LACP Provisioning Please note this section simply shows how to change the LACP key on the SPB rear-ports. This step is not required and is only added for informational purposes to allow different ISIS L1- metric s to be used so you can proceed to section if you wish. If you are interested in changing the default LACP key, please follow the example below. For this example, we will change the LACP key from the default setting of 4095 to a value of 4094 only for the black rear ports (ports 34 to 36) between bridges 7001 & 7004 and 7002 & 7003; this is the return path in the SPB ring we are provisioning. Change LACP key via black rear port (34-36) By default, all rear ports have the same LACP key with a value of On the SPB return path between bridges 7001 & 7004 and 7002 & 7004, we will change the LACP key to Verify default LACP settings. 7004(config)#show lacp aggr Aggr ID Trunk Status Type Members Enabled LA Enabled LA (config)#show lacp port aggr 8223 Admin Oper Trunk Partner Port Priority Lacp A/I Timeout Key Key AggId Id Port Status Active A Short Active Active A Short Active 7004(config)#show lacp port aggr 8224 Admin Oper Trunk Partner Port Priority Lacp A/I Timeout Key Key AggId Id Port Status Active A Short Active Active A Short Active Active A Short Active Change LACP key on FI Mesh Black ports (ports 34-36) to 4094 on bridges 7001 & 7004 and 7002 & (config)#interface ethernet (config-if)#lacp key (config-if)#exit Verify LACP Keys 41

42 7004(config)#show lacp port aggr 8223 Admin Oper Trunk Partner Port Priority Lacp A/I Timeout Key Key AggId Id Port Status Active A Short Active Active A Short Active 7004(config)#show lacp port aggr 8224 Admin Oper Trunk Partner Port Priority Lacp A/I Timeout Key Key AggId Id Port Status Active A Short Active Active A Short Active Active A Short Active 7004(config)#show lacp debug member 34-36,38-39 Port AggrId TrunkId Rx State Mux State Partner Port Current Ready Current Ready Current Ready Current Ready Current Ready 39 Please note that changing the LACP key on the rear-port interfaces which are already in use as SPB NNI connections will cause the IS-IS adjacency to bounce SPB Interface Configuration SPB Interface Configuration Rear ports for this configuration example, we are provisioning only the FI Red (ports 38 & 39) and Black FI (ports 34, 35, & 36) rear ports. As we are using two LACP keys, we need to select one port from each LAG and enable SPB. Note if you select all ports, i.e , and enable SPBM, this will work, but, you will simple get an error message stating ISIS is already enabled on port 39, 35, and 36 simply just ignore this error message. Rear Ports Same configuration on 7001, 7002, 7003, and (config)#interface ethernet 34, (config-if)#isis 7001(config-if)#isis spbm (config-if)#isis enable 7001(config-if)#spanning-tree mstp learning disable 42

43 Front Ports Same configuration on 7001, 7002, 7003, and (config)#interface ethernet (config-if)#isis 7001(config-if)#isis spbm (config-if)#isis enable 7001(config-if)#spanning-tree mstp learning disable 7001(config-if)#exit ISIS L1-metric Optional As an option, we can change the default metric on the FI rear ports and SPB front ports to reflect the actual port speeds. For this example example, port 3 is a 10GigE uplink. Please see section for more details. Switches 7001 and (config)#interface ethernet (config-if)#isis spbm 1 l1-metric (config-if)#exit 7001(config)#interface ethernet (config-if)#isis spbm 1 l1-metric (config-if)#exit 7001(config)#interface ethernet (config-if)#isis spbm 1 l1-metric (config-if)#exit Switches 7002 and (config)#interface ethernet (config-if)#isis spbm 1 l1-metric (config-if)#exit 7002(config)#interface ethernet (config-if)#isis spbm 1 l1-metric (config-if)#exit 43

44 Remove default VLAN from all SPB ports Please note if the default VLAN is removed from the rear port prior to adding a new VLAN, i.e. adding the SPB B-VLANs as in the previous step, LACP will be disabled and will have to be manually re-enabled again. Remove default VLAN (VLAN 1) port membership from SPB enabled ported Same configuration on all switches 7001 to (config)#vlan members remove 1 3,34-39 Verify VLAN membership 7004(config)#show vlan interface vids 3,34-38 Port VLAN VLAN Name VLAN VLAN Name VLAN VLAN Name BVLAN BVLAN BVLAN BVLAN BVLAN BVLAN BVLAN BVLAN BVLAN BVLAN BVLAN BVLAN

45 VLACP SPB front ports only Switch Parameter Value VLACP - Global 7001, 7002, 7003, 7004 VLACP MAC 01:80:c2:00:00:0f State enable VLACP - Interface 7001, 7002, 7003, 7004 Timers Short Time-out-scale 5 State enable Note that by default, LACP is enabled on the FI rear ports where LACP provides the link integrity functionality as well as link aggregation. VLACP - SPB front panel Interface Configuration Front Ports Same configuration on 7001, 7002, 7003, and (config)#vlacp macaddress 01:80:c2:00:00:0f 7001(config)#vlacp enable 7001(config)#interface ethernet (config-if)#lacp timeout short 7001(config-if)#vlacp timeout-scale (config-if)#vlacp enable 7001(config-if)#exit 7001(config)#show vlacp ========================================================================= Vlacp Global Information ========================================================================= Multicast address : 01:80:c2:00:00:0f Vlacp : enabled Vlacp hold time : (config-if)#show vlacp interface 3 =============================================================================== VLACP Information 45

46 =============================================================================== PORT ADMIN OPER HAVE FAST SLOW TIMEOUT TIMEOUT ETH MAC ENABLED ENABLED PARTNER TIME TIME TYPE SCALE TYPE ADDRESS true true yes short :00:00:00:00: Verify ISIS adjacency Verify front panel ISIS adjacency 7001#show isis interface =============================================================================== ISIS Interfaces =============================================================================== IFIDX TYPE LEVEL OP-STATE ADM-STATE ADJ UP-ADJ SPBM-L1-METRIC =============================================================================== Port: 3 pt-pt Level 1 UP UP Trunk: 64 pt-pt Level 1 UP UP Trunk: 63 pt-pt Level 1 UP UP #show isis adjacencies =============================================================================== ISIS Adjacencies =============================================================================== INTERFACE L STATE UPTIME PRI HOLDTIME SYSID HOST-NAME =============================================================================== Port: 3 1 UP 00:15: b4.e0df 8005 Trunk: 64 1 UP 00:09: cb1.5bff.5fdf 7002 Trunk: 63 1 UP 00:09: If the rear port interfaces are not shown as Trunk, verify that only the SPB VLANs 4051 & 4052 are added to rear ports. If the default VLAN (VLAN 1) is a member of the rear ports, the adjacency will display only one port member instead of a trunk for both the FI black and red rear ports. 46

47 CFM By default, a maintenance domain level value of 4 will be used; if you wish, you can change this value to lower or higher level from 0 to 7. For this configuration example, we will provision a unique Maintenance End Point ID for each VSP 7000 switch. Switch Parameter Value CFM 7001, 7002, 7003, 7004 CFM Enabled 7001 Maintenance End Point (MEP) ID Maintenance End Point (MEP) ID Maintenance End Point (MEP) ID Maintenance End Point (MEP) ID 7004 Enable CFM 7001(config)#cfm spbm mepid (config)#cfm spbm enable 7002(config)#cfm spbm mepid (config)#cfm spbm enable 7003(config)#cfm spbm mepid (config)#cfm spbm enable 7004(config)#cfm spbm mepid (config)#cfm spbm enable Verify Operations 7001#l2 ping vlan < > <routernodename mac> <SPB node name> 7001#l2tracerotue < > <mac routernodename> <SPB node name> Example: 7001#l2ping vlan 4051 routernodename 7004 Please wait for l2ping to complete or press CTRL-C to abort :30:18:23:98:65 L2 PING Statistics---- 0(64) bytes of data 1 packets transmitted, 1 packets received, 0.00% packet loss 47

48 round-trip (us) min/max/ave/stdv = 982/982/982.00/ #l2 ping vlan 4052 routernodename 7004 Please wait for l2ping to complete or press CTRL-C to abort :30:18:23:98:65 L2 PING Statistics---- 0(64) bytes of data 1 packets transmitted, 1 packets received, 0.00% packet loss round-trip (us) min/max/ave/stdv = 966/966/966.00/ #l2 traceroute vlan 4051 routernodename 7004 Please wait for l2traceroute to complete or press CTRL-C to abort l2traceroute to 7004 ( ), vlan (fc-a8-41-f6-37-df) ( ) In-band Management via L2VSN For this example, we will create an L2VSN using mgmt VLAN 101 with I-SID that will also be added to nodes 8005 and 8005 that in turn will have IP Shortcuts enabled allowing for in-band IP connectivity. Switch Parameter Value L2VSN for management 7001, 7002, 7003, 7004 Mgmt VLAN 101 I-SID IP address / IP address / IP address / IP address /24 Add in-band L2VSN and IP address 7001(config)#vlan create 101 name mgmt-101 type port 7001(config)#vlan mgmt (config)#ip address netmask default-gateway (config)#i-sid vlan

49 For switches 7002, 7003, and 7004, use the same configuration as above except for the items shown below 7002(config)#ip address netmask default-gateway (config)#ip address netmask default-gateway (config)#ip address netmask default-gateway (config)#show ip Bootp/DHCP Mode: Disabled Configured In Use Last BootP/DHCP Stack IP Address: Switch IP Address: Switch Subnet Mask: Mgmt Stack IP Address: Mgmt Switch IP Address: Mgmt Subnet Mask: Mgmt Def Gateway: Default Gateway: (config)#show vlan Id Name Type Protocol PID Active IVL/SVL Mgmt VLAN #1 Port None 0x0000 Yes IVL No Port Members: ALL 101 mgmt-101 Port None 0x0000 Yes IVL Yes Port Members: NONE 4051 BVLAN-1 B-VLAN None 0x0000 Yes IVL No Port Members: BVLAN-2 B-VLAN None 0x0000 Yes IVL No Port Members: Total VLANs: 4 49

50 7004#show isis spbm i-sid all id =============================================================================== SPBM ISID INFO =============================================================================== ISID SOURCE NAME VLAN SYSID TYPE HOST-NAME =============================================================================== b4.e0df discover e.1f48.f0df discover fca8.41f6.37df discover fca8.41f6.37df discover cb1.5bff.5fdf discover cb1.5bff.5fdf discover c65 discover c65 discover config config 7004 At this point, we should be able to ping the default gateway and the other three VSP 7000 switches. 7002#ping Host is reachable 7002#ping Host is reachable 7002#ping Host is reachable 7002#ping Host is reachable Using the CFM tracetree command, verify the multicast tree via each SPB VLAN for i-sid #l2 tracetree vlan 4051 i-sid Please wait for l2tracetree to complete or press CTRL-C to abort l2tracetree to F-42-A5, vlan 4051 i-sid nickname hops fc-a8-41-f6-37-df -> b4-e0-df fc-a8-41-f6-37-df -> fc-a8-41-f6-37-df -> c-b1-5b-ff-5f-df fc-a8-41-f6-37-df -> c-b1-5b-ff-5f-df b4-e0-df -> e-1f-48-f0-df 50

51 c-b1-5b-ff-5f-df -> f-df c-b1-5b-ff-5f-df -> f-df 7001#l2 tracetree vlan 4052 i-sid Please wait for l2tracetree to complete or press CTRL-C to abort l2tracetree to F-42-A5, vlan 4052 i-sid nickname hops fc-a8-41-f6-37-df -> fc-a8-41-f6-37-df -> c-b1-5b-ff-5f-df fc-a8-41-f6-37-df -> c-b1-5b-ff-5f-df fc-a8-41-f6-37-df -> b4-e0-df > f-df b4-e0-df -> e-1f-48-f0-df & 8006 Configuration IP and L2VSN The following is the minimum condensed configuration for connectivity to SPB nodes 7001 and config terminal # # VLAN CONFIGURATION - PHASE I # vlan create 101 type port-mstprstp 0 vlan i-sid interface Vlan 101 ip address ip vrrp address ip vrrp 11 backup-master enable priority 150 ip vrrp 11 enable exit config terminal # # VLAN CONFIGURATION - PHASE I # vlan create 101 type port-mstprstp 0 vlan i-sid interface Vlan 101 ip address ip vrrp address ip vrrp 11 backup-master enable ip vrrp 11 enable exit 51

52 Adding Service VLANs for data and vmotion For this example, we will create an L2VSN for vmotion using VLAN 1600 with I-SID that will also be added to nodes 8001 and 8002 that in turn will have IP Shortcuts enabled allowing for routing to the data center. Switch Parameter Value L2VSN for Services/vMotion and VMware mgmt 7001, 7002, 7003, 7004 Services/vMotion VLAN 1600 I-SID , 7002, 7003, 7004 VMware mgmt VLAN 61 I-SID , 7002, 7003, 7004 MLT MLT ID = 1 with ports 9 & 10 Create VLANs 7001, 7002, and 7004 Same configuration on all switches 7001(config)#vlan create 61 type port 7001(config)#vlan create 1600 type port 7001(config)#vlan ports 9,10 tagging tagall filter-untagged-frame enable 7001(config)#vlan configcontrol flexible 7001(config)#vlan members add 61,1600 9, (config)#vlan members remove 1 9,10 Verify configuration 7001(config)#show vlan interface info 9,10 Filter Filter Untagged Unregistered Port Frames Frames PVID PRI Tagging Name Yes Yes 1 0 TagAll Port 9 10 Yes Yes 1 0 TagAll Port (config)#show vlan interface vids 9,10 Port VLAN VLAN Name VLAN VLAN Name VLAN VLAN Name VLAN # VLAN #

53 10 61 VLAN # VLAN # Create MLT Same configuration on all switches 7001(config)#mlt 1 enable member 9,10 learning disable Create the L2VSN Same configuration on all switches 7001(config)#i-sid vlan (config)#i-sid vlan & 8006 Configuration IP and L2VSN config terminal # # VLAN CONFIGURATION - PHASE I # vlan create 1600 type port-mstprstp 0 vlan i-sid interface Vlan 1600 ip address ip vrrp address ip vrrp 160 enable exit config terminal # # VLAN CONFIGURATION - PHASE I # vlan create 1600 type port-mstprstp 0 vlan i-sid interface Vlan 1600 ip address ip vrrp address ip vrrp 160 backup-master enable priority 150 ip vrrp 160 enable exit 53

54 and 8006 Core configuration The following is the minimum condensed configuration for connectivity to SPB nodes 7001 and 7002 with IP Shortcuts enabled config terminal spbm spbm ethertype 0x8100 config terminal spbm spbm ethertype 0x8100 # # CLI CONFIGURATION # prompt "8005" # # ISIS SPBM CONFIGURATION # router isis spbm 1 spbm 1 nick-name spbm 1 b-vid primary 4051 spbm 1 ip enable exit # # VLAN CONFIGURATION - PHASE I # vlan create 4051 name "BVLAN-1" type spbmbvlan vlan create 4052 name "BVLAN-2" type spbmbvlan # # PORT CONFIGURATION - PHASE II # interface GigabitEthernet 2/34 name "isis_nni_7001" isis isis spbm 1 isis enable vlacp fast-periodic-time 500 timeout short timeout-scale 5 funcmac-addr 01:80:c2:00:00:0f vlacp enable no spanning-tree mstp msti 62 force-portstate enable exit # # CIRCUITLESS IP INTERFACE CONFIGURATION - GlobalRouter # interface loopback 1 # # CLI CONFIGURATION # prompt "8006" # # ISIS SPBM CONFIGURATION # router isis spbm 1 spbm 1 nick-name spbm 1 b-vid primary 4051 spbm 1 ip enable exit # # VLAN CONFIGURATION - PHASE I # vlan create 4051 name "BVLAN-1" type spbmbvlan vlan create 4052 name "BVLAN-2" type spbmbvlan # # PORT CONFIGURATION - PHASE II # interface GigabitEthernet 2/34 name "isis_nni_7003" isis isis spbm 1 isis enable vlacp fast-periodic-time 500 timeout short timeout-scale 5 funcmac-addr 01:80:c2:00:00:0f vlacp enable no spanning-tree mstp msti 62 force-portstate enable exit # # CIRCUITLESS IP INTERFACE CONFIGURATION - GlobalRouter # interface loopback 1 54

55 ip address / exit # # ISIS CONFIGURATION # router isis ip-source-address is-type l1 manual-area exit router isis enable # # CFM CONFIGURATION - PHASE II # cfm cmac mepid 805 cfm cmac enable cfm spbm mepid 805 cfm spbm enable # # IP REDISTRIBUTION CONFIGURATION - GlobalRouter # router isis redistribute direct redistribute direct enable exit isis apply redistribute direct ip address / exit # # ISIS CONFIGURATION # router isis ip-source-address is-type l1 manual-area exit router isis enable # # CFM CONFIGURATION - PHASE II # cfm cmac mepid 806 cfm cmac enable cfm spbm mepid 806 cfm spbm enable # # IP REDISTRIBUTION CONFIGURATION - GlobalRouter # router isis redistribute direct redistribute direct enable exit isis apply redistribute direct Verify L2VSN Operations Verify L2VSN 7001(config)#show isis spbm i-sid all id =============================================================================== SPBM ISID INFO =============================================================================== ISID SOURCE NAME VLAN SYSID TYPE HOST-NAME =============================================================================== fca8.41f6.37df config cb1.5bff.5fdf discover c65 discover discover (config)#show isis spbm i-sid all id =============================================================================== 55

56 SPBM ISID INFO =============================================================================== ISID SOURCE NAME VLAN SYSID TYPE HOST-NAME =============================================================================== fca8.41f6.37df config cb1.5bff.5fdf discover c65 discover discover b4.e0df discover e.1f48.f0df discover (config)#l2 tracetree vlan 4052 i-sid Please wait for l2tracetree to complete or press CTRL-C to abort l2tracetree to F-48-80, vlan 4052 i-sid nickname hops fc-a8-41-f6-37-df -> fc-a8-41-f6-37-df -> c-b1-5b-ff-5f-df > f-df For I-SID , SPB nodes 7010 and 7011 will be displayed once SPB is enabled of these nodes. Verify MAC Learning Assuming we are sending traffic via the Services VLAN #show mac-address-table spbm i-sid Mac Address Table Aging Time: 300 Learning Enabled Ports ALL Number of addresses: 3 MAC Address I-SID Source Vid BVid Dest-MAC Dest-Sys-Name E A Port 1/ e-1f-48-f0-df C-29-1B Trunk f-e C-C4-1C Port 1/ FF 00-1E-1F-48-F Port 1/ e-1f-48-f0-df B4-E Port 1/ b4-e0-df

57 A Trunk f-e E Port 1/ B Port 1/

58 1.6.2 VSP 7000 SMLT Cluster For this configuration example, we will configure the VSP 7000 SMLT cluster switches 7010 and Figure 20 shows the SMLT IST and SMLT port configuration. Figure 20: Data Center #2 Configuration Example SMLT For the VSP 7000, it is important to not enable the filter-untagged-frame option on the IST port members. Also, the default PVID of all IST ports must be the primary B-VLAN ID; for this example, this will be B-VLAN ID This will happened automatically providing SPB is enable first prior to enabling the IST. 58

59 Option: Change Spanning Tree mode to MSTP If using the VSP 9000 in the core and using tools such as VLAN Manager in COM, it is recommended to change the Spanning Tree mode to MSTP. Option change spanning mode to MSTP on all switches 7024XLS(config)#spanning-tree mode mst New operational mode MSTP will take effect upon reset 7024XLS(config)#boot Reboot the unit(s) (y/n)? yrebooting XLS#show spanning-tree mode Current STP Operation Mode: MSTP Next STP Operation Mode: MSTP System Name Configure system name 7024XLS(config)#snmp-server name XLS(config)#snmp-server name

60 Configure Out-of-Band Management Interface and static route Configure management address and add route 7010(config)#ip mgmt address switch netmask (config)#ip mgmt route (config)#ip mgmt address switch netmask (config)#ip mgmt route (config)#show ip Bootp/DHCP Mode: BootP When Needed Configured In Use Last BootP/DHCP Stack IP Address: Switch IP Address: Switch Subnet Mask: Mgmt Stack IP Address: Mgmt Switch IP Address: Mgmt Subnet Mask: Mgmt Def Gateway: Default Gateway: (config)#show mgmt-port status Port Link Up (config)#show ip mgmt route Destination IP Subnet Mask Gateway IP Status ACTIVE Enable VLACP Globally Enable VLACP Globally 7001(config)#vlacp enable 7001(config)#vlacp macaddress 180.c200.f 60

61 SPB Configuration Switch Parameter Value SPB 7010, 7011 B-VLANs 4051, 4052 where 4051 is the primary B-VLAN VLAN Names BVLAN-1 and BVLAN-2 IS-IS Area IS-IS Enable SPBM Enable, using instance 1 IS-IS Area Default PVID SPB Nick Name SPB System-Name SPB Nick Name SPB System-Name 7001 VLAN Change configuration control to automatic Same configuration on switches 7010 and XLS(config)#vlan configcontrol automatic SPBM Configuration 7010(config)#vlan create 4051 name BVLAN-1 type spbm-bvlan 7010(config)#vlan create 4052 name BVLAN-2 type spbm-bvlan 7010(config)#save config 7010(config)#spbm This operation will require a reboot to configure the loopback port. Are you sure? (y/n)?y 7010(config)#router isis 7010(config-isis)#spbm (config-isis)#spbm 1 b-vid primary

62 7010(config-isis)#spbm 1 nick-name (config-isis)#manual-area (config-isis)#system-id a8df 7010(config-isis)#sys-name (config-isis)#spbm 1 smlt-peer-system-id bc (config-isis)#exit 7010(config)#router isis enable 7010#show isis system-id ========================================== ISIS System-Id ========================================== SYSTEM-ID ========================================== a8df For switch 7011, use the same configuration as above except for the items shown below 7011(config-isis)#spbm 1 nick-name (config-isis)#system-id bc (config-isis)#spbm 1 smlt-peer-system-id a8df 7011(config-isis)#sys-name #show isis system-id ========================================== ISIS System-Id ========================================== SYSTEM-ID ========================================== bc00 Enter the ALCI show isis system-id on each peer switch to get the System ID value for the spbm 1 smlt-peer-system-id xxxx.xxxx.xxxx command. Each switch in the cluster must peer with its peer s System ID value SPB Interface Configuration Since we will be adding an IST interface via the front ports, ports 23 & 24, we will add an MLT with these port members. Create MLT 1 to be used by the IST Same configuration on switches 7010 and (config)#mlt 1 name IST enable member 23,24 learning disable Verify MLT configuration 7010(config)#show mlt 1 Id Name Members Bpdu Mode Status Type 62

63 IST All Basic Enabled Trunk SPB Interface Configuration Switches 7010 and 7011: 7010(config)#interface ethernet 1,23, (config-if)#isis 7010(config-if)#isis spbm (config-if)#isis enable 7010(config-if)#exit 7010(config)#interface ethernet (config-if)#spanning-tree mstp learning disable 7010(config-if)#exit Remove default VLAN from all SPB ports Please note if the default VLAN is removed from the rear port prior to adding a new VLAN, i.e. adding the SPB B-VLANs as in the previous step, LACP will be disabled and will have to be manually re-enabled again. Remove default VLAN (VLAN 1) port membership from SPB enabled ported Same configuration on switches 7010 & (config)#vlan members remove 1 1,23,24 63

64 VLACP - SPB NNI port 1 to core Switch Parameter Value VLACP - Global 7010, 7011 VLACP MAC 01:80:c2:00:00:0f State enable VLACP - Interface 7010, 7011 Timers Short Time-out-scale 5 State enable VLACP Front Ports Same configuration on 7010 and (config)#interface ethernet (config-if)#vlacp timeout short 7010(config-if)#vlacp timeout-scale (config-if)#vlacp enable 7010(config-if)#exit 7010(config)#show vlacp ========================================================================= Vlacp Global Information ========================================================================= Multicast address : 01:80:c2:00:00:0f Vlacp : enabled Vlacp hold time : (config-if)#show vlacp interface 1 =============================================================================== VLACP Information =============================================================================== PORT ADMIN OPER HAVE FAST SLOW TIMEOUT TIMEOUT ETH MAC ENABLED ENABLED PARTNER TIME TIME TYPE SCALE TYPE ADDRESS true true yes short :00:00:00:00:00 64

65 IST Configuration The following port based VLANs will be configured on the SMLT Switch cluster Switch Feature Parameter Value IST MLT ID 1 VLAN IST VLAN IP address /30 Ports 23, IST VLAN IP address /30 Ports 23,24 For the VSP 7000, it is important to not enable the filter-untagged-frame option on the IST port members. Also, the default PVID of all IST ports must be the primary B-VLAN ID; for this example, this will be B-VLAN ID This will happened automatically providing SPB is enable first prior to enabling the IST. Also, it is recommended to not enable VLACP on the IST. Add IST VLAN 2 and add IP address Same configuration on switches 7010 and XLS(config)#vlan create 2 name ist type port 7024XLS(config)#vlan ports tagging tagall 7024XLS(config)#vlan members 2 23, XLS(config)#vlan members remove IP configuration on (config)#interface vlan (config-if)#ip address (config-if)#exit IP configuration on (config)#interface vlan (config-if)#ip address (config-if)#exit Verify VLAN membership 7010(config)#show vlan interface vids

66 Port VLAN VLAN Name VLAN VLAN Name VLAN VLAN Name 23 2 ist 4051 BVLAN BVLAN ist 4051 BVLAN BVLAN Create IST 7010: 7010(config)#interface mlt (config-if)#ist peer-ip vlan (config-if)#ist enable 7010(config-if)#exit 7011: 7011(config)#interface mlt (config-if)#ist peer-ip vlan (config-if)#ist enable 7011(config-if)#exit Verify IST Operation assuming the peer IST has been configured 7010(config)#show ist MLT ID Enabled Running Master Peer IP Address Vlan ID YES YES NO (config)#show smlt =============================================================================== MLT SMLT Info =============================================================================== MLT SMLT ADMIN CURRENT ID ID TYPE TYPE ist ist 66

67 Verify ISIS adjacency Verify front panel ISIS adjacency 7010#show isis interface =============================================================================== ISIS Interfaces =============================================================================== IFIDX TYPE LEVEL OP-STATE ADM-STATE ADJ UP-ADJ SPBM-L1-METRIC =============================================================================== Port: 1 pt-pt Level 1 UP UP Trunk: 1 pt-pt Level 1 UP UP #show isis adjacencies =============================================================================== ISIS Adjacencies =============================================================================== INTERFACE L STATE UPTIME PRI HOLDTIME SYSID HOST-NAME =============================================================================== Port: 1 1 UP 00:24: fdf 8001 Trunk: 1 1 UP 00:03: bc Verify SPB SMLT Virtual B-MAC and SMLT Peer ID 7010(config)#show isis spbm =================================================================== ISIS SPBM Info =================================================================== SPBM B-VID PRIMARY NICK LSDB INSTANCE VLAN NAME TRAP =================================================================== disabled =================================================================== ISIS SPBM SMLT Info =================================================================== SPBM SMLT-SPLIT-BEB SMLT-VIRTUAL-BMAC SMLT-PEER-SYSTEM-ID INSTANCE =================================================================== 1 primary 70:30:18:23:a8:e bc (config)#show isis spbm =================================================================== ISIS SPBM Info =================================================================== SPBM B-VID PRIMARY NICK LSDB INSTANCE VLAN NAME TRAP =================================================================== 67

68 disabled =================================================================== ISIS SPBM SMLT Info =================================================================== SPBM SMLT-SPLIT-BEB SMLT-VIRTUAL-BMAC SMLT-PEER-SYSTEM-ID INSTANCE =================================================================== 1 secondary 70:30:18:23:a8:e a8df CFM By default, a maintenance domain level value of 4 will be used; if you wish, you can change this value to lower or higher level if you like from 0 to 7. For this configuration example, we will provision a unique Maintenance End Point ID for each VSP 7000 switch. Switch Parameter Value CFM 7010, 7011 CFM Enabled 7010 Maintenance End Point (MEP) ID Maintenance End Point (MEP) ID 7011 Enable CFM 7010(config)#cfm spbm mepid (config)#cfm spbm enable 7011(config)#cfm spbm mepid (config)#cfm spbm enable Verify Operations 7010#l2ping vlan < > <routernodename mac> <SPB node name> 7010#l2tracerotue < > <mac routernodename> <SPB node name> Example: 7010#l2 ping vlan 4051 routernodename 7004 Please wait for l2ping to complete or press CTRL-C to abort :30:18:23:98:65 L2 PING Statistics---- 0(64) bytes of data 1 packets transmitted, 1 packets received, 0.00% packet loss round-trip (us) min/max/ave/stdv = 1111/1111/ / #l2 ping vlan 4052 routernodename

69 Please wait for l2ping to complete or press CTRL-C to abort :30:18:23:98:65 L2 PING Statistics---- 0(64) bytes of data 1 packets transmitted, 1 packets received, 0.00% packet loss round-trip (us) min/max/ave/stdv = 1307/1307/ / #l2 traceroute vlan 4051 routernodename 7004 Please wait for l2traceroute to complete or press CTRL-C to abort l2traceroute to 7004 ( ), vlan ( a8-df) ( f-df) ( d-be-23-df) ( b4-e0-df) (3c-b1-5b-ff-5f-df) (fc-a8-41-f6-37-df) ( ) 69

70 In-band Management via L2VSN For this example, we will create an L2VSN using mgmt VLAN 101 with I-SID that will also be added to nodes 8005 and 8005 that in turn will have IP Shortcuts enabled allowing for in-band IP connectivity. Switch Parameter Value L2VSN for management 7010 & 7011 Mgmt VLAN 101 I-SID IP address / IP address /24 Add in-band L2VSN and IP address 7010(config)#vlan create 101 name mgmt-101 type port 7010(config)#vlan mgmt (config)#ip address netmask default-gateway (config)#i-sid vlan 101 For switche 7011, use the same configuration as above except for the items shown below 7011(config)#ip address netmask default-gateway (config)#show ip Bootp/DHCP Mode: Disabled Configured In Use Last BootP/DHCP Stack IP Address: Switch IP Address: Switch Subnet Mask: Mgmt Stack IP Address: Mgmt Switch IP Address: Mgmt Subnet Mask: Mgmt Def Gateway: Default Gateway:

71 7010(config)#show vlan Id Name Type Protocol PID Active IVL/SVL Mgmt VLAN #1 Port None 0x0000 Yes IVL No Port Members: 1-22, ist Port None 0x0000 Yes IVL No Port Members: mgmt-101 Port None 0x0000 Yes IVL Yes Port Members: NONE 1600 vmotion Port None 0x0000 Yes IVL No Port Members: BVLAN-1 B-VLAN None 0x0000 Yes IVL No Port Members: 1, BVLAN-2 B-VLAN None 0x0000 Yes IVL No Total VLANs: 6 Port Members: 1, #show isis spbm i-sid all id =============================================================================== SPBM ISID INFO =============================================================================== ISID SOURCE NAME VLAN SYSID TYPE HOST-NAME =============================================================================== fca8.41f6.37df discover cb1.5bff.5fdf discover c65 discover discover a8df discover bc00 config b4.e0df discover e.1f48.f0df discover 8006 At this point, we should be able to ping the default gateway and all the other VSP 7000 switches. 7010#ping Host is reachable 7010#ping Host is reachable 71

72 7010#ping Host is reachable 7010#ping Host is reachable 7010#ping Host is reachable 7010#ping Host is reachable Using the CFM tracetree command, verify the multicast tree via each SPB VLAN for i-sid #l2 tracetree vlan 4051 i-sid Please wait for l2tracetree to complete or press CTRL-C to abort l2 tracetree to A F-42-A5, vlan 4051 i-sid nickname hops a8-df-> bc a8-df-> f-df bc-00-> df f-df -> d-be-23-df d-be-23-df -> b4-e0-df d-be-23-df -> e0-7b-bc-23-df e0-7b-bc-23-df -> e-1f-48-f0-df b4-e0-df -> 7001 fc-a8-41-f6-37-df fc-a8-41-f6-37-df -> fc-a8-41-f6-37-df -> c-b1-5b-ff-5f-df c-b1-5b-ff-5f-df -> f-df 7011#l2 tracetree vlan 4052 i-sid Please wait for l2tracetree to complete or press CTRL-C to abort l2tracetree to F-42-A5, vlan 4052 i-sid nickname hops bc-00-> a8-df bc-00-> df d-be-23-df -> e0-7b-bc-23-df e0-7b-bc-23-df -> e-1f-48-f0-df e-1f-48-f0-df -> b4-e0-df e-1f-48-f0-df -> f-df f-df -> c-b1-5b-ff-5f-df f-df -> > 7001 fc-a8-41-f6-37-df

73 Add Services/vMotion VLAN Switch Feature Parameter Value VLAN 1600 VLAN 1600 SLT ID to ESX server 65 Port 25 VLAN Ports Members 25 I-SID Add VLAN Same configuration on switches 7010 and (config)#vlan create 1600 name vmotion type port 7010(config)#vlan ports 25 tagging tagall filter-untagged-frame enable 7010(config)#vlan members add (config)#vlan members remove 1 25 Verify Operations 7010(config)#show vlan interface info Filter Filter Untagged Unregistered Port Frames Frames PVID PRI Tagging Name Yes Yes TagAll Port Yes Yes TagAll Port Yes Yes 1 0 TagAll Port (config)#show vlan interface vids Port VLAN VLAN Name VLAN VLAN Name VLAN VLAN Name ist 3051 BVLAN BVLAN ist 3051 BVLAN BVLAN vmotion

74 SMLT Configuration Same configuration on switches 7010 and (config)#interface ethernet (config-if)#smlt (config-if)#exit Verify operations 7010#show smlt ethernet =============================================================================== SLT Info =============================================================================== PORT SMLT ADMIN CURRENT NUM ID TYPE TYPE slt slt Create the L2VSN Same configuration on switches 7010 and (config)#i-sid vlan

75 1.6.3 VSP 7000 FI Mesh with SMLT In the previous example in section 1.5.1, a Fabric Connect Mesh configuration was provisioned using four standalone VSP 7000 switches. In this example, we will use the rear ports 38 and 39 as an IST and create two SMLT clusters consisting of 7001 and 7002 and a second cluster made up of 7003 and Please note that the VSP 7000 requires minimum two ports members for an IST. For this example, we are using the red rear ports (ports 38 and 39) so that only one rear port cable is required. If we used the two bottom blue ports (ports 37 and 38) for the IST, then two rear cables will be required. For the VSP 7000, it is important to not enable the filter-untagged-frame option on the IST port members. Also, the default PVID of all IST ports must be the primary B-VLAN ID; for this example, this will be B-VLAN ID This will happened automatically providing SPB is enable first prior to enabling the IST. 75

76 Figure 21: VSP 7000 FI Mesh with SMLT 76

77 Set rear port mode to SPB Switch Parameter Value Rear Port 7001, 7002, 7003, 7004 Rear port mode Enabled & SPB Enable rear-port mode on all switches 7024XLS(config)#rear-port mode spbm Enabling rear port mode will disable Fabric Interconnect Stack operation. Switch configuration will be reset to partial-defaults. Continue(yes/no)?y 7024XLS#show rear-port mode Rear Port Mode: Enabled SPB (Loopback Port Reserved) Rear Port Operational State: Operational SPB (Loopback Port Reserved) Option: Change Spanning Tree mode to MSTP If using the VSP 9000 in the core and using tools such as VLAN Manager in COM, it is recommended to change the Spanning Tree mode to MSTP. Option change spanning mode to MSTP on all switches 7024XLS(config)#spanning-tree mode mst New operational mode MSTP will take effect upon reset 7024XLS(config)#boot Reboot the unit(s) (y/n)? yrebooting XLS#show spanning-tree mode Current STP Operation Mode: MSTP Next STP Operation Mode: MSTP System Name Configure system name 7024XLS(config)#snmp-server name XLS(config)#snmp-server name XLS(config)#snmp-server name

78 7024XLS(config)#snmp-server name Configure Out-of-Band Management Interface and static route Configure management address and add route 7001(config)#ip mgmt address switch netmask (config)#ip mgmt route (config)#ip mgmt address switch netmask (config)#ip mgmt route (config)#ip mgmt address switch netmask (config)#ip mgmt route (config)#ip mgmt address switch netmask (config)#ip mgmt route (config)#show ip Bootp/DHCP Mode: BootP When Needed Configured In Use Last BootP/DHCP Stack IP Address: Switch IP Address: Switch Subnet Mask: Mgmt Stack IP Address: Mgmt Switch IP Address: Mgmt Subnet Mask: Mgmt Def Gateway: Default Gateway: (config)#show mgmt-port status Unit Link Limit Shutdown Interval Up (config)#show ip mgmt route Destination IP Subnet Mask Gateway IP Status ACTIVE 78

79 Enable VLACP Globally Enable VLACP Globally 7001(config)#vlacp enable 7001(config)#vlacp macaddress 180.c200.f SPB Configuration Switch Parameter Value SPB 7001, 7002, 7003, 7004 B-VLANs 4051, 4052 where 4051 is the primary B-VLAN VLAN Names BVLAN-1 and BVLAN-2 IS-IS Area IS-IS Enable SPBM Enable, using instance 1 IS-IS Area SPB Nick Name SPB System-Name 7001 SMLT Peer System-ID <System-id of IST peer> 7002 SPB Nick Name SPB System-Name 7002 SMLT Peer System-ID <System-id of IST peer> 7003 SPB Nick Name SPB System-Name 7003 SMLT Peer System-ID <System-id of IST peer> 7004 SPB Nick Name SPB System-Name 7004 SMLT Peer System-ID <System-id of IST peer> 79

80 VLAN Change configuration control to automatic 7001(config)#vlan configcontrol automatic SPBM Configuration 7001(config)#vlan create 4051 name BVLAN-1 type spbm-bvlan 7001(config)#vlan create 4052 name BVLAN-2 type spbm-bvlan 7001(config)#spbm 7001(config)#router isis 7001(config-isis)#spbm (config-isis)#spbm 1 b-vid primary (config-isis)#spbm 1 nick-name (config-isis)#manual-area (config-isis)#sys-name (config-isis)#spbm 1 smlt-peer-system-id 3cb1.5bff.5fdf 7001(config-isis)#exit 7001(config)#router isis enable For switches 7002, 7003, and 7004, use the same configuration as above except for the items shown below 7002(config-isis)#spbm 1 nick-name (config-isis)#sys-name (config-isis)#spbm 1 smlt-peer-system-id fca8.41f6.37df 7003(config-isis)#spbm 1 nick-name (config-isis)#sys-name (config-isis)#spbm 1 smlt-peer-system-id (config-isis)#spbm 1 nick-name (config-isis)#sys-name (config-isis)#spbm 1 smlt-peer-system-id c65 Enter the ALCI show isis system-id on each peer switch to get the System ID value for the spbm 1 smlt-peer-system-id xxxx.xxxx.xxxx command. Each switch in the cluster must peer with its peer s System ID value. 80

81 SPB Interface Configuration Since we will be adding an IST interface via the red rear ports, ports 38 & 39, we will have to disable LACP on these ports and add an MLT. Disable LACP on ports 38 and 39 Same configuration on all switches 7024XLS(config)#interface ethernet 38, XLS(config-if)#no lacp aggregation enable 7024XLS(config-if)#lacp mode off 7024XLS(config-if)#exit Prior to enabling the IST, LACP must be disabled on the rear port member that are being used for the IST Create MLT 1 to be used by the IST Same configuration on all switches 7001(config)#mlt 1 name IST enable member 38,39 learning disable Verify MLT configuration 7001(config)#show mlt 1 Id Name Members Bpdu Mode Status Type IST All Basic Enabled Trunk SPB Interface Configuration Rear ports for this configuration example, we are provisioning only the FI Red (ports 38-39) which are also used for the IST and Black FI (ports 34-36) rear ports. For the FI black ports (ports 34 to 36), we simply just need to select one of the rear ports and enable SPB as all ports are using the same LACP LAG. Rear Ports: Same configuration on all switches 7001(config)#interface ethernet 34,38, (config-if)#isis 7001(config-if)#isis spbm (config-if)#isis enable 7001(config-if)#spanning-tree mstp learning disable 7001(config-if)#exit Front Ports: Same configuration on switches 7001 and (config)#interface ethernet (config-if)#isis 81

82 7001(config-if)#isis spbm (config-if)#isis enable 7001(config-if)#spanning-tree mstp learning disable 7001(config-if)#exit Remove default VLAN from all SPB ports Please note if the default VLAN is removed from the rear port prior to adding a new VLAN, i.e. adding the SPB B-VLANs as in the previous step, LACP will be disabled and will have to be manually re-enabled again. Remove default VLAN (VLAN 1) port membership from SPB enabled ported Same configuration on all switches 7001 to (config)#vlan members remove 1 3,

83 VLACP SPB front ports only Switch Parameter Value VLACP - Interface 7001, 7002, 7003, 7004 Timers Short Time-out-scale 5 State enable Note that by default, LACP is enabled on the FI rear ports where LACP provides the link integrity functionality as well as link aggregation. VLACP - SPB front panel Interface Configuration Front Ports Same configuration on 7001, 7002, 7003, and (config)#interface ethernet (config-if)#vlacp timeout short 7001(config-if)#vlacp timeout-scale (config-if)#vlacp enable 7001(config-if)#exit 7001(config)#show vlacp ========================================================================= Vlacp Global Information ========================================================================= Multicast address : 01:80:c2:00:00:0f Vlacp : enabled Vlacp hold time : (config-if)#show vlacp interface 3 =============================================================================== VLACP Information =============================================================================== PORT ADMIN OPER HAVE FAST SLOW TIMEOUT TIMEOUT ETH MAC ENABLED ENABLED PARTNER TIME TIME TYPE SCALE TYPE ADDRESS true true yes short :00:00:00:00:00 83

84 ISIS L1-metric Optional As an option, we can change the default metric on the FI rear ports and SPB front ports to reflect the actual port speeds. Please see section for more details. Switches 7001 and (config)#interface ethernet (config-if)#isis spbm 1 l1-metric (config-if)#exit 7001(config)#interface ethernet (config-if)#isis spbm 1 l1-metric (config-if)#exit 7001(config)#interface ethernet (config-if)#isis spbm 1 l1-metric (config-if)#exit Switches 7002 and (config)#interface ethernet (config-if)#isis spbm 1 l1-metric (config-if)#exit 7002(config)#interface ethernet (config-if)#isis spbm 1 l1-metric (config-if)#exit 84

85 IST Configuration The following port based VLANs will be configured on the SMLT Switch cluster Switch Feature Parameter Value 7001, 7002, 7003, 7004 IST MLT ID 1 VLAN IST VLAN IP address /30 Ports 38, IST VLAN IP address /30 Ports 38, IST VLAN IP address /30 Ports 38, IST VLAN IP address /30 Ports 38,39 For the VSP 7000, it is important to not enable the filter-untagged-frame option on the IST port members. Also, the default PVID of all IST ports must be the primary B-VLAN ID; for this example, this will be B-VLAN ID This will happened automatically providing SPB is enable first prior to enabling the IST. Also, it is recommended to not enable VLACP on the IST. 85

86 Add IST VLAN 2 and add IP address Same configuration on all switches 7024XLS(config)#vlan create 2 name ist type port 7024XLS(config)#vlan members 2 38, XLS(config)#vlan members remove 1 38, IP configuration on (config)#interface vlan (config-if)#ip address (config-if)#exit IP configuration on (config)#interface vlan (config-if)#ip address (config-if)#exit IP configuration on (config)#interface vlan (config-if)#ip address (config-if)#exit IP configuration on (config)#interface vlan (config-if)#ip address (config-if)#exit 86

87 Create IST 7001(config)#interface mlt (config-if)#ist peer-ip vlan (config-if)#ist enable 7002(config-if)#exit (config)#interface mlt (config-if)#ist peer-ip vlan (config-if)#ist enable 7002(config-if)#exit (config)#interface mlt (config-if)#ist peer-ip vlan (config-if)#ist enable 7003(config-if)#exit (config)#interface mlt (config-if)#ist peer-ip vlan (config-if)#ist enable 7004(config-if)#exit Verify IST Operation assuming the SMLT cluster peer is also configured 7001(config)#show ist MLT ID Enabled Running Master Peer IP Address Vlan ID YES YES NO (config)#show smlt =============================================================================== MLT SMLT Info =============================================================================== MLT SMLT ADMIN CURRENT ID ID TYPE TYPE ist ist 87

88 Verify the default VLAN is now the primary B-VLAN ID. Also, make sure the Filter Untagged Frames option is disabled. 7001(config)#show vlan interface info 38,39 Filter Filter Untagged Unregistered Port Frames Frames PVID PRI Tagging Name No Yes TagAll Port No Yes TagAll Port 39 88

89 Verify ISIS adjacency Verify front panel ISIS adjacency 7001(config)#show isis interface =============================================================================== ISIS Interfaces =============================================================================== IFIDX TYPE LEVEL OP-STATE ADM-STATE ADJ UP-ADJ SPBM-L1-METRIC =============================================================================== Port: 33 pt-pt Level 1 DOWN UP Port: 37 pt-pt Level 1 DOWN UP Trunk: 64 pt-pt Level 1 UP UP Trunk: 1 pt-pt Level 1 UP UP Port: 3 pt-pt Level 1 UP UP In the 10.3 release, up to 64 MLT groups are supported. The last MLT group, Trunk 64, is used by default for the LACP LAG for the rear ports. Trunk 1 in this example is used by the IST. 7001#show isis adjacencies =============================================================================== ISIS Adjacencies =============================================================================== INTERFACE L STATE UPTIME PRI HOLDTIME SYSID HOST-NAME =============================================================================== Trunk: 64 1 UP 00:14: cb1.5bff.5fdf 7002 Trunk: 1 1 UP 00:14: Port: 3 1 UP 00:10: b4.e0df 8005 If the rear port interfaces are not shown as Trunk, verify that only the SPB VLANs 4051 & 4052 are added to rear ports and that the default VLAN (VLAN 1) is not a member of the rear ports. The VLAN configuration must be the same on both cluster switches in order for LACP to form a LAG. 89

90 Verify SPB SMLT Virtual B-MAC and SMLT Peer ID 7001 & 7002 SMLT Cluster: 7001(config)#show isis spbm =================================================================== ISIS SPBM Info =================================================================== SPBM B-VID PRIMARY NICK LSDB INSTANCE VLAN NAME TRAP =================================================================== disabled =================================================================== ISIS SPBM SMLT Info =================================================================== SPBM SMLT-SPLIT-BEB SMLT-VIRTUAL-BMAC SMLT-PEER-SYSTEM-ID INSTANCE =================================================================== 1 primary 3c:b1:5b:ff:5f:e0 3cb1.5bff.5fdf 7002(config)#show isis spbm =================================================================== ISIS SPBM Info =================================================================== SPBM B-VID PRIMARY NICK LSDB INSTANCE VLAN NAME TRAP =================================================================== disabled =================================================================== ISIS SPBM SMLT Info =================================================================== SPBM SMLT-SPLIT-BEB SMLT-VIRTUAL-BMAC SMLT-PEER-SYSTEM-ID INSTANCE =================================================================== 1 secondary 3c:b1:5b:ff:5f:e0 fca8.41f6.37df & 7004 SMLT Cluster: 7003(config)#show isis spbm =================================================================== ISIS SPBM Info =================================================================== SPBM B-VID PRIMARY NICK LSDB INSTANCE VLAN NAME TRAP =================================================================== disabled =================================================================== ISIS SPBM SMLT Info =================================================================== SPBM SMLT-SPLIT-BEB SMLT-VIRTUAL-BMAC SMLT-PEER-SYSTEM-ID INSTANCE =================================================================== 1 primary 70:30:18:23:7f:e

91 7004(config)#show isis spbm =================================================================== ISIS SPBM Info =================================================================== SPBM B-VID PRIMARY NICK LSDB INSTANCE VLAN NAME TRAP =================================================================== disabled =================================================================== ISIS SPBM SMLT Info =================================================================== SPBM SMLT-SPLIT-BEB SMLT-VIRTUAL-BMAC SMLT-PEER-SYSTEM-ID INSTANCE =================================================================== 1 secondary 70:30:18:23:7f:e c65 91

92 CFM By default, a maintenance domain level value of 4 will be used; if you wish, you can change this value to lower or higher level if you like from 0 to 7. For this configuration example, we will provision a unique Maintenance End Point ID for each VSP 7000 switch. Switch Parameter Value CFM 7001, 7002, 7003, 7004 CFM Enabled 7001 Maintenance End Point (MEP) ID Maintenance End Point (MEP) ID Maintenance End Point (MEP) ID Maintenance End Point (MEP) ID 7004 Enable CFM 7001(config)#cfm spbm mepid (config)#cfm spbm enable 7002(config)#cfm spbm mepid (config)#cfm spbm enable 7003(config)#cfm spbm mepid (config)#cfm spbm enable 7004(config)#cfm spbm mepid (config)#cfm spbm enable Verify Operations 7001#l2ping vlan < > <routernodename mac> <SPB node name> 7001#l2tracerotue < > <mac routernodename> <SPB node name> 92

93 In-band Management via L2VSN Please see configuration in previous example Adding Services VLANs for data and vmotion Switch Parameter Value L2VSN for vmotion and Management 7001, 7002, 7003, 7004 Services/vMotion VLAN 1600 I-SID SMLT SLT ID 65 Port Members 25 (7001 & 7002) 7001, 7002, 7003, 7004 VMware mgmt VLAN (7003 & 7004) I-SID SMLT SLT ID 65 Port Members 25 (7001 & 7002) 24 (7003 & 7004) Create VLANs Same configuration on all switches. 7001(config)#vlan create 61 type port 7001(config)#vlan create 1600 type port SMLT Create SLT on port 10 using SLT id 65 Same configuration on 7001 & (config)#vlan ports 25 tagging tagall filter-untagged-frame enable 7001(config)#vlan configcontrol flexible 7001(config)#vlan members add 61, (config)#vlan members remove (config)#interface ethernet (config-if)#smlt (config-if)#exit 93

94 Same configuration on 7003 & (config)#vlan ports 24 tagging tagall filter-untagged-frame enable 7001(config)#vlan configcontrol flexible 7001(config)#vlan members add 61, (config)#vlan members remove (config)#interface ethernet (config-if)#smlt (config-if)#exit Verify configuration 7001(config)#show vlan interface info 25 Filter Filter Untagged Unregistered Port Frames Frames PVID PRI Tagging Name Yes Yes 1 0 TagAll Port (config)#show vlan interface vids 25 Port VLAN VLAN Name VLAN VLAN Name VLAN VLAN Name VLAN # VLAN # Assuming the servers, VMware ESX server in our example, have been added on both cluster switches, the SMLT state should display SLT as shown below. 7001#show smlt =============================================================================== MLT SMLT Info =============================================================================== MLT SMLT ADMIN CURRENT ID ID TYPE TYPE ist ist =============================================================================== SLT Info =============================================================================== PORT SMLT ADMIN CURRENT NUM ID TYPE TYPE slt slt 94

95 7003#show smlt =============================================================================== MLT SMLT Info =============================================================================== MLT SMLT ADMIN CURRENT ID ID TYPE TYPE ist ist =============================================================================== SLT Info =============================================================================== PORT SMLT ADMIN CURRENT NUM ID TYPE TYPE slt slt Create the L2VSN Same configuration on switches , and (config)#i-sid vlan (config)#i-sid vlan #show isis spbm i-sid all fca8.41f6.37df config fca8.41f6.37df config fca8.41f6.37df config cb1.5bff.5fdf discover cb1.5bff.5fdf discover cb1.5bff.5fdf discover c65 discover c65 discover c65 discover discover discover discover a8df discover bc00 discover b4.e0df discover

96 b4.e0df discover e.1f48.f0df discover e.1f48.f0df discover

97 1.7 Lossless or PFC-lite Configuration If you wish, you can enable PFC-lite on the VSP 7000 switches providing at least is used. If release 10.2 or less is used, a buffer mode of large, lossless, or maximum can be configured. After a configuration change, the switch must be rebooted for the change to take effect. 7001(config)#qos agent buffer <large lossless lossless-pfc maximum regular spbscaled> 7001(config)#show qos agent Please note, as of release 10.3, when SPBM is enabled, the default buffer mode is spb-scaled. Use this parameter to select an amount of resource sharing specifically designed for large SPBM configurations. When you enable SPBM, this mode is automatically selected; when you disable SPBM, the configuration reverts to the default of large. If you upgrade from Release and SPBM is enabled, you must manually run the qos agent buffer spb-scaled command. 1.8 VM NIC Teaming In this section, we detail the steps required to configure active/active NIC teaming that can be using with either a MLT configuration, SMLT/SLT configuration on a SMLT cluster, or FI mesh. This applies to any ESX server, 4.x or 5.x Creation of Virtual Switch NIC Teaming In the vsphere Client window, select the ESX server where you wish to enable NIC Teaming. Select the Configuration tab and click on the Networking option. It will show the Virtual Switch configuration. 97

98 98

99 For adding a new Virtual Switch, click on Add Networking at the top right corner to open the Add Network Wizard. The Add Network Wizard first displays the type of Connection Type (Virtual Machine/VMkernel/Service Console) to be used. Select Virtual Machine and click on the Next button at the bottom of the wizard. 99

100 Select the Network Access to be used. Choose the Create a virtual switch option to create a new one. Click on the Next button at the bottom of the window to proceed. 100

101 Specify a name for the Virtual Machine Port group and add a VLAN Id if tagging needs is required; i.e. VLAN 58 as shown here. Click on the Next button. 101

102 Click the Finish button to complete this part of the Virtual Switch creation. 102

103 1.8.2 Addition of Adaptor Ports to the Virtual Switch Under vsphere Client, select the ESX server and then select the Networking option under the Configuration tab. Select Properties for the Virtual Switch that was created in the above step. 103

104 This screen will show the Port group that was created and its properties: Number of ports, Security, Traffic Shaping, Failover and Load Balancing configuration. The properties of the Virtual Switch can be changed by clicking the Edit button in the window. Assuming we wish to change the VLAN ID from 58 to 1002 as shown here. If you do not wish to change anything, proceed to the next step. 104

105 Click on the Network Adapters tab, to add/edit network adapter information. 105

106 Click on Add to add a new adapter to the group. Select required ports from the options available. 106

107 Once added, the Edit button can be used to make changes to port configuration. Click Close to complete this step. The Virtual Switch is now configured. 107

108 Go the Summary tab as shown below and click on Edit Settings. 108

109 Go to the Hardware tab as shown below and click on Add. 109

110 Select Ethernet Adapter and Next>. 110

111 Select the NIC team you created above and then click on Next>. Make sure Connect at power on is selected. 111

112 When completed, the NIC teaming network should be displayed in the Summary tab. 112

113 1.8.3 VMware NIC Teaming Load Balance Mechanism VMware load balancing can be provisioned using the following methods when using an Avaya switch: 1. Route based on the originating port virtual port id Default setting Load balance based on vswitch port ID Recommended when no MLT or SMLT is used on an Avaya switch Recommended when using the VSP 7000 in FI mesh (no SMLT) 2. Route based on IP hash Load balance based on src/dst IP Recommended when MLT, DMLT, or SMLT is used on an Avaya switch 3. Route based on source MAC hash Load balance on vnic MAC address Recommended when no MLT or SMLT is used on an Avaya switch Can be used when using the VSP 7000 in FI mesh (no SMLT) 113

114 SMLT Clustering For Switch Clustering applications via either an SMLT or SLT link, select Route based on IP hash via the VMware via the NIC Teaming tab. Select the ESX server, go to the Configuration tab, click on the Networking option and scroll down to the vswitch where you enabled NIC teaming and click on Properties. After the vswitch interface properties window pop s up as shown below, click on Edit. 114

115 Go to the NIC Teaming tab and select Route based on IP hash under Load Balancing and Link status only via Network Failover Detection. The Link status only solely relies on the link status of the network adapters the Beacon Probing option is not support when IP hash is enabled. Also you should see all the NIC s under Active Adapters. 115

116 FI Mesh No SMLT For non-smlt applications such as FI Mesh, go to the NIC Teaming tab and select Route based on the originating virtual port ID under Load Balancing and Link status only via Network Failover Detection. When you select this setting, traffic from a given virtual Ethernet adapter is consistently send to the same physical adapter unless there is a failover to another adapter in the NIC team. If there are multiple VM s, the VMs will be distributed across the multiple NICs, i.e. VM-1 will use vmnic4 while VM-2 will use vmnic5. 116