Unified Compute Platform 4.1.2

Transcription

1 Unified Compute Platform MK-92UCP084-02

2 Hitachi Data Systems. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording, or stored in a database or retrieval system for commercial purposes without the express written permission of Hitachi, Ltd., or Hitachi Data Systems Corporation (collectively, Hitachi ). Licensee may make copies of the Materials provided that any such copy is: (i) created as an essential step in utilization of the Software as licensed and is used in no other manner; or (ii) used for archival purposes. Licensee may not make any other copies of the Materials. "Materials" mean text, data, photographs, graphics, audio, video and documents. Hitachi reserves the right to make changes to this Material at any time without notice and assumes no responsibility for its use. The Materials contain the most current information available at the time of publication. Some of the features described in the Materials might not be currently available. Refer to the most recent product announcement for information about feature and product availability, or contact Hitachi Data Systems Corporation at Notice: Hitachi products and services can be ordered only under the terms and conditions of the applicable Hitachi agreements. The use of Hitachi products is governed by the terms of your agreements with Hitachi Data Systems Corporation. By using this software, you agree that you are responsible for: 1) Acquiring the relevant consents as may be required under local privacy laws or otherwise from authorized employees and other individuals to access relevant data; and 2) Verifying that data continues to be held, retrieved, deleted, or otherwise processed in accordance with relevant laws. Notice on Export Controls. The technical data and technology inherent in this Document may be subject to U.S. export control laws, including the U.S. Export Administration Act and its associated regulations, and may be subject to export or import regulations in other countries. Reader agrees to comply strictly with all such regulations and acknowledges that Reader has the responsibility to obtain licenses to export, re-export, or import the Document and any Compliant Products. Hitachi is a registered trademark of Hitachi, Ltd., in the United States and other countries. AIX, AS/400e, DB2, Domino, DS6000, DS8000, Enterprise Storage Server, eserver, FICON, FlashCopy, IBM, Lotus, MVS, OS/390, PowerPC, RS6000, S/390, System z9, System z10, Tivoli, z/os, z9, z10, z13, z/vm, and z/vse are registered trademarks or trademarks of International Business Machines Corporation. Active Directory, ActiveX, Bing, Excel, Hyper-V, Internet Explorer, the Internet Explorer logo, Microsoft, the Microsoft Corporate Logo, MS-DOS, Outlook, PowerPoint, SharePoint, Silverlight, SmartScreen, SQL Server, Visual Basic, Visual C++, Visual Studio, Windows, the Windows logo, Windows Azure, Windows PowerShell, Windows Server, the Windows start button, and Windows Vista are registered trademarks or trademarks of Microsoft Corporation. Microsoft product screen shots are reprinted with permission from Microsoft Corporation. All other trademarks, service marks, and company names in this document or web site are properties of their respective owners.

3 Contents Preface Intended audience Product version Related document Comments Getting help v v v v vi vi Chapter 1: UCP Systems High Level Design 1 Chapter 2: UCP 4000 Brocade Systems 5 Network Architecture and Configurations Overview 5 Physical Configurations 6 UCP Director Network Management Features 7 UCP Appliance Initial VCS and Spanning Tree Configuration 8 Server Facing Port Configurations 8 Uplink Configurations 9 Brocade VDX Brocade ICX Chapter 3: UCP 4000 Cisco Systems 21 Network Architecture and Configurations Overview 21 Physical Configurations 21 Layer 2 / Layer 3 Mode 23 UCP Director Network Management Features 24 UCP Appliance Initial vpc and Spanning Tree Configuration 25 Server Facing Port Configurations 27 Internal Network Configurations (Layer 3 mode) 27 Network Design Table 29 Contents iii

4 Add Layer 3 License 33 Configure Unicast Routing for VRF ucpmanagement 35 Configure Global Unicast Routing 47 Configure VXLAN 62 Uplink Configurations 82 Layer 2 Uplink Configurations 83 Scenario 1: Connecting to single core network & multi chassis ether channel technology enabled switches 85 Scenario 2: Connecting to single core network & non-multi chassis ether channel technology enabled switches 85 Scenario 3: Connecting to multiple core networks & multi-chassis ether channel technology enabled switches 87 Scenario 4: Connecting to multiple core networks & non-multi chassis ether channel technology enabled switches 88 Layer 3 Uplink Configurations 90 Scenario 1: OSPF Peering to Core Network 90 Scenario 2: BGP Peering to Core Network 91 Scenario 3: Static Routing 92 Chapter 4: UCP 4000E Systems 95 Network Architecture and Configurations Overview 95 Physical Configurations 95 UCP Director Network Management Features 96 UCP Appliance Initial vpc and Spanning Tree Configuration 97 Server Facing Port Configurations 98 Uplink Configurations 99 Scenario 1: Connecting to single core network & multi chassis ether channel technology enabled switches 101 Scenario 2: Connecting to single core network & non-multi chassis ether channel technology enabled switches 101 Scenario 3: Connecting to separate core networks & multi chassis ether channel technology enabled switches 103 Scenario 4: Connecting to separate core networks & non-multi chassis ether channel technology enabled switches 104 iv Contents

5 Preface This guide provides Hitachi Unified Compute Platform (UCP) IP network architecture and configuration examples that are needed to plan and prepare for a UCP installation. Intended audience This book is intended for network administrators and Hitachi Data Systems (HDS) representatives who are involved in installing and configuring UCP. It assumes that you are familiar with the network technologies, network hardware and its command line interfaces. Product version This guide applies to UCP 4000 Revision 3 and UCP 4000E Revision 2 systems. These systems were first introduced with UCP Director 4.1. Related document The following documents contain additional information about productname: UCP Pre-Installation Requirements and Configuration Guide Contains information and procedures you need to be aware of for a successful UCP installation. Contains technical reference information about the networking architecture of UCP systems and provides procedures to help you configure the system to work in your network environment. Preface v

6 Comments UCP Director Administrator's Guide Contains technical and usage information for UCP and UCP Director. Describes how to administer UCP Director through UCP Director web interfaceucp Director web interface with both VMware vcenter and Microsoft SCVMM. UCP Director API Reference Guide Describes how to use the UCP Director API. UCP Director CLI Reference Guide Describes how to use the UCP Director CLI. UCP Director Third-Party Copyrights and Licenses Contains copyright and license information for the third-party software distributed with or embedded in UCP Director. UCP DOC Administrator's Guide Contains technical and usage information for Unified Compute Platform Director Operations Center (UCP DOC). Describes how to administer UCP DOC through UCP DOC Console. UCP DOC API Reference Guide Describes how to use the UCP DOC API. UCP DOC CLI Reference Guide Describes how to use the UCP DOC CLI. Comments Please send us your comments on this document: Include the document title and number, including the revision (for example, -01), and refer to specific sections and paragraphs whenever possible. All comments become the property of Hitachi Data Systems. Thank you! Getting help Hitachi Data Systems Support Portal is the destination for technical support of products and solutions sold by Hitachi Data Systems. To contact technical support, log on to Hitachi Data Systems Support Connect for vi Preface

7 Getting help contact information: Hitachi Data Systems Community is a global online community for HDS customers, partners, independent software vendors, employees, and prospects. It is the destination to get answers, discover insights, and make connections. Join the conversation today! Go to community.hds.com, register, and complete your profile. Preface vii

8 viii Preface

9 1 UCP Systems High Level Design The high level design of the UCP system includes server, network, storage, and software components designed to deliver a complete virtualized data center as an end-to-end solution. The following table lists the components for a UCP 4000 system. Components UCP 4000 Brocade (min-max) UCP 4000 Cisco (min-max) UCP 4000E (min-max) Hitachi CB500 Chassis 2 Port N/A Port N/A Hitachi CB520 Blades HB3-2 port N/A HB4-2 port HB3-4 port N/A HB4-4 port Switches In-chassis (per chassis) 2 Brocade VDX PassThru In-chassis Fiber Channel (per chassis) 2 Fiber Channel 6546 N/A Chapter 1: UCP Systems High Level Design 1

10 Components UCP 4000 Brocade (min-max) UCP 4000 Cisco (min-max) UCP 4000E (min-max) Fiber Channel ToR 2 Brocade or 4 Brocade 6510 N/A ToR 2 Brocade VDX , 4, 6, or 8 Cisco Nexus 9372PX 2 Cisco Nexus 5548UP or 2 Cisco Nexus 9332PQ Management Servers Management Servers Storage 2 Brocade ICX Cisco Nexus 3048 N/A 2-3 T41S-2U nodes Hitachi VSP G200 G400 G600 G800 G1000 F400 F600 F800 G200 G400 G600 G800 G1000 (Bolt-on) F400 F600 F800 UCP 4000 Brocade Appliance Overview 2 Chapter 1: UCP Systems High Level Design

11 UCP 4000 Cisco Appliance Overview (Front View) UCP 4000 Cisco Appliance Overview (Rear View) Chapter 1: UCP Systems High Level Design 3

12 UCP 4000E Appliance Overview 4 Chapter 1: UCP Systems High Level Design

13 2 UCP 4000 Brocade Systems This chapter will cover the network architecture and uplink configuration examples for UCP 4000 Brocade systems. Network Architecture and Configurations Overview In the Brocade Networking model, UCP utilizes Brocade Virtual Cluster Switching (VCS) Technology to eliminate spanning tree, and optimize for east west traffic. All VDX switches in a single UCP instance are configured as VCS Fabric Cluster mode, and form single VCS fabric. UCP also utilizes Virtual LAG (vlag) technology which provides multi-path networking between VCS fabric and your network. The following diagram illustrates the LAN Architecture for the UCP 4000 Brocade model. Chapter 2: UCP 4000 Brocade Systems 5

14 Network Architecture and Configurations Overview UCP 4000 Brocade Network Architecture Physical Configurations The following tables show port utilization of the Brocade VDX 6740 and ICX 7450 switches that can be used on the UCP 4000 Brocade model. Port Utilization of Brocade VDX 6740 Switch Usage Port Counts Speed Uplinks to customer core 8 10Gbps ISL to other Brocade VDX Gbps ISL to ICX 7450 switches 1 10Gbps Management server connectivity 3 10Gbps ISL to CB500 in-chassis switches (Brocade VDX 6746) 4 per CB500 chassis, up to 32 (8 x CB500 chassis) 10Gbps Note: In addition to the above list, 4 x 40G (QSFP) ports are available with additional 40G port license. 6 Chapter 2: UCP 4000 Brocade Systems

15 Network Architecture and Configurations Overview Port Utilization of Brocade ICX 7450 Switch Usage Port Counts Speed Uplinks to customer core 2 1Gbps ISL to other Brocade ICX 7450 (Stacking) 2 10Gbps CB500 SVP connectivity 1 per CB500 chassis, up to 8 (8 x CB500 chassis) 1Gbps Management server BMC connectivity Storage system management connectivity 3 (ICX7450 A side only) 1Gbps 1 1Gbps Switch management connectivity 1 per switch, up to 4 (ICX 7450 A side only) 1Gbps ISL to Brocade VDX 6740 switches 1 10Gbps UCP Director Network Management Features Onboarding Switches The UCP Director supports onboarding of the Brocade VDX 6740 switches and Brocade VDX 6746 In-Chassis switches for management and health monitoring. Switch Health Monitoring The UCP Director monitors the health of the onboard switches and notifies if there are any warnings or errors detected on the switches. Configuration Backup UCP Director will periodically take backup configurations of the switches in its inventory. UCP Director can also manually backup/apply the switch configurations through user request. Configure Host/Cluster Network This allows the user to configure the Brocade VDX 6746 server facing interfaces on a blade by blade basis. The native vlan which is configured and managed by UCP for the management network and additional vlans such as vmotion or Compute vlans can also be managed through this UCP Director feature. UCP utilizes Brocade VCS technology. By adding VLANs to server facing ports on the VDX 6746, blades with the same VLAN can communicate with each other immediately. You don't need to configure the VDX 6740 switch for internal VLAN communication. Chapter 2: UCP 4000 Brocade Systems 7

16 Network Architecture and Configurations Overview Note: For routed traffic or northbound traffic, the uplink ports (portchannel) trunk VLANs need to be configured manually. VLAN DB management - If a new vlan is added through the Configure Host/Cluster Network feature, the vlan is added to the VDX UCP Appliance Initial VCS and Spanning Tree Configuration VCS Mode UCP 4000 supports Fabric Cluster mode only. Important: UCP 4000 Brocade Revision 3 doesn't support VCS Logical Chassis mode and related features. VCS ID UCP Default VCS ID is "10" across entire UCP stack. HDS representatives configure/change VCS ID based on customer engagement information. Important: Forming a single VCS fabric across the customer core network and UCP network is NOT supported. If VCS is enabled on the core network and Brocade VDX switches, please use a different VCS ID. Also, please make sure to use vlag technology to connect them. Rbridge ID HDS representatives assign Rbridge ID with last octet of the switch management IP address. Rapid-PVST Mode Spanning tree rapid-pvst mode is configured with priority setting Sample Spanning tree configuration protocol spanning-tree rpvst bridge-priority 61440! Server Facing Port Configurations The UCP Director manages the server facing ports of Brocade VDX 6746 In- Chassis switch which are connected to the blades. 8 Chapter 2: UCP 4000 Brocade Systems

17 Uplink Configurations When managing the server facing ports, UCP Director sets the port in switchport trunk mode. In addition, it sets the native vlan on the port to the specified management vlan id which is chosen during deployment. It is critical these settings are never modified because the native vlan is used by UCP Director for management functions. Finally, the port is set to allow specific vlans which are configured by UCP Director through the switchport trunk allowed vlan add command. Once user needs to modify the allowed vlans they can perform a Configure host/cluster vlan operation and UCP Director will modify the port configuration accordingly. The "spanning-tree shutdown", "no fabric isl enable", and "no fabric trunk enable" are configured on all of server facing ports during UCP Deployment. UCP Director Software will not touch these configurations. HDS recommends not to change these configurations. Sample Port Configuration interface TenGigabit 214/0/9 description Connecting to blade 0 no fabric isl enable no fabric trunk enable switchport switchport mode trunk switchport trunk allowed vlan add 57,59 switchport trunk tag native-vlan switchport trunk native-vlan 57 spanning-tree shutdown no shutdown! Uplink Configurations Brocade VDX 6740 If the UCP system is based on Brocade network devices, understand which of the following scenarios can be applied, and use the procedure described in that section to connect the UCP system to the datacenter core network. UCP system with Brocade VDX 6740 switch supports Brocade Virtual Link Aggregation (vlag) technology. It enables high-performance / redundant networking across the UCP system and datacenter core networks. Chapter 2: UCP 4000 Brocade Systems 9

18 Uplink Configurations By default, port 1 to 8 on both Brocade VDX 6740 switches are dedicated for connecting core network. And, port channel 10 is configured for these ports as a virtual LAG(vLAG). Note: In addition to the port 1 to 8, 4 x 40G (QSFP) ports are available with additional 40G port license. Typically the following 3 scenarios can be applied. # Scenario Example of Network Topology 1 Connecting to multi chassis ether channel technology enabled switches 2 Connecting to standard switches (non multi chassis channel technology switch) 10 Chapter 2: UCP 4000 Brocade Systems

19 Uplink Configurations # Scenario Example of Network Topology 3 Connecting to Brocade VCS enabled core network using vlag Important: Forming single VCS fabric across datacenter core network and UCP network is NOT supported. If the VCS is enabled on core network Brocade VDX switches, please use vlag technology to connect them. Scenario 1: Connecting to multi chassis ether channel technology enabled switches 1. Login to the Brocade VDX6740 A using SSH. 2. Configure interface port-channel. R1-BR-6740-A-U42# configure terminal Entering configuration mode terminal R1-BR-6740-A-U42(config)# interface Port-channel 10 R1-BR-6740-A-U42(config-Port-channel-10)# vlag ignore-split R1-BR-6740-A-U42(config-Port-channel-10)# switchport R1-BR-6740-A-U42(config-Port-channel-10)# switchport mode trunk R1-BR-6740-A-U42(config-Port-channel-10)# switchport trunk allowed vlan add 103 R1-BR-6740-A-U42(config-Port-channel-10)# spanning-tree bpdu-mac ccc.cccdßThis is required in case of connecting to Cisco switch or Brocade FCX/ICX switch R1-BR-6740-A-U42(config-Port-channel-10)# no spanning-tree shutdown R1-BR-6740-A-U42(config-Port-channel-10)# no shutdown R1-BR-6740-A-U42(config-Port-channel-10)# exit Important: If core network switch is Cisco switch or Brocade ICX/FCX switch, you must configure "spanning-tree bpdu-mac ccc.cccd" on the port-channel for Spanning Tree interoperability. Chapter 2: UCP 4000 Brocade Systems 11

20 Uplink Configurations 3. Add Ten Gigabit interfaces into the port-channel, which connecting to the core switch. R1-BR-6740-A-U42(config)# interface TenGigabit 212/0/1-8 R1-BR-6740-A-U42(conf-if-te-212/0/1-8)# no fabric isl enable R1-BR-6740-A-U42(conf-if-te-212/0/1-8)# no fabric trunk enable R1-BR-6740-A-U42(conf-if-te-212/0/1-8)# channel-group 10 mode active type standard R1-BR-6740-A-U42(conf-if-te-212/0/1-8)# no shutdown 4. Repeat previous steps to the Brocade VDX6740 B Important: Must use same port-channel number on both VDX6740 A & B switches. By default, port-channel 10 is configured on both switches as vlag. 5. Connect cables to the customer switch, and verify the both port channels are up, and all the ports are synchronized status. Important: Please make sure to connect each cable one by one, and each time a connection is made verify the status of the port-channel. R1-BR-6740-A-U42# show interface port-channel 10 Port-channel 10 is up, line protocol is up Hardware is AGGREGATE, address is 0027.f8b Current address is 0027.f8b Description: Connecting to management switches. 6. Verify all the ports are synchronized status. R1-BR-6740-A-U42# show port-channel 10 LACP Aggregator: Po 10 Aggregator type: Standard Ignore-split is enabled Admin Key: Oper Key 0010 Partner System ID - 0x0001,74-8e-f8-80-5c-c0 Partner Oper Key Member ports on rbridge-id 212: Link: Te 212/0/1 (0xD ) sync: 1 * Link: Te 212/0/2 (0xD ) sync: 1 Link: Te 212/0/3 (0xD ) sync: 1 Link: Te 212/0/4 (0xD ) sync: 1 Link: Te 212/0/5 (0xD ) sync: 1 Link: Te 212/0/6 (0xD ) sync: 1 12 Chapter 2: UCP 4000 Brocade Systems

21 Uplink Configurations Link: Te 212/0/7 (0xD ) sync: 1 Link: Te 212/0/8 (0xD ) sync: 1 Scenario 2: Connecting to standard switches (non-multi chassis ether channel technology enabled switches) 7. Login to the Brocade VDX6740 A using SSH. 8. Configure first interface port channel (for example, port-channel 10) R1-BR-6740-A-U42# configure terminal Entering configuration mode terminal R1-BR-6740-A-U42(config)# interface Port-channel 10 R1-BR-6740-A-U42(config-Port-channel-10)# vlag ignore-split R1-BR-6740-A-U42(config-Port-channel-10)# switchport R1-BR-6740-A-U42(config-Port-channel-10)# switchport mode trunk R1-BR-6740-A-U42(config-Port-channel-10)# switchport trunk allowed vlan add 103 R1-BR-6740-A-U42(config-Port-channel-10)# spanning-tree bpdu-mac ccc.cccdßThis is required in case of connecting to Cisco switch or Brocade FCX/ICX switch R1-BR-6740-A-U42(config-Port-channel-10)# no spanning-tree shutdown R1-BR-6740-A-U42(config-Port-channel-10)# no shutdown R1-BR-6740-A-U42(config-Port-channel-10)# exit Important: If core network switch is Cisco switch or Brocade ICX/FCX switch, you must configure "spanning-tree bpdu-mac ccc.cccd" on the port-channel for Spanning Tree interoperability. 9. Add Ten Gigabit interfaces into the port-channel, which will be connecting to core network switch A R1-BR-6740-A-U42(config)# interface TenGigabit 212/0/1-4 R1-BR-6740-A-U42(conf-if-te-212/0/1-4)# no fabric isl enable R1-BR-6740-A-U42(conf-if-te-212/0/1-4)# no fabric trunk enable R1-BR-6740-A-U42(conf-if-te-212/0/1-4)# channel-group 10 mode active type standard R1-BR-6740-A-U42(conf-if-te-212/0/1-4)# no shutdown 10. Configure second interface port-channel (for example, port-channel 11) R1-BR-6740-A-U42(config)# interface Port-channel 11 R1-BR-6740-A-U42(config-Port-channel-11)# vlag ignore-split R1-BR-6740-A-U42(config-Port-channel-11)# switchport R1-BR-6740-A-U42(config-Port-channel-11)# switchport mode trunk R1-BR-6740-A-U42(config-Port-channel-11)# switchport trunk allowed vlan add 103 Chapter 2: UCP 4000 Brocade Systems 13

22 Uplink Configurations R1-BR-6740-A-U42(config-Port-channel-11)# spanning-tree bpdu-mac ccc.cccdßThis is required in case of connecting to Cisco switch or Brocade FCX/ICX switch R1-BR-6740-A-U42(config-Port-channel-11)# no spanning-tree shutdown R1-BR-6740-A-U42(config-Port-channel-11)# no shutdown R1-BR-6740-A-U42(config-Port-channel-11)# exit Add Ten Gigabit interfaces into the port-channel, which connecting to customer s switch B R1-BR-6740-A-U42(config)# interface TenGigabit 212/0/5-8 R1-BR-6740-A-U42(conf-if-te-212/0/5-8)# no fabric isl enable R1-BR-6740-A-U42(conf-if-te-212/0/5-8)# no fabric trunk enable R1-BR-6740-A-U42(conf-if-te-212/0/5-8)# channel-group 11 mode active type standard R1-BR-6740-A-U42(conf-if-te-212/0/5-8)# no shutdown R1-BR-6740-A-U42(conf-if-te-212/0/5-8)# end 11. Repeat previous steps to the Brocade VDX6740 B Important: Must use same port-channel numbers on both VDX6740 A & B switches. 12. Verify the spanning-tree is running R1-BR-6740-A-U42# show spanning-tree brief VLAN 1 Spanning-tree Mode: Rapid Per-VLAN Spanning Tree Protocol Root ID Priority Address 01e a46 Hello Time 2, Max Age 20, Forward Delay 15 Bridge ID Priority Address 01e a46 Hello Time 2, Max Age 20, Forward Delay 15, Tx- HoldCount 6 Migrate Time 3 sec Interface Role Sts Cost Prio Link-type Edge Po 1 DES FWD P2P No Po 2 DES FWD P2P No Po 10 DIS DSC P2P No Po 11 DIS DSC P2P No 13. Connect cables to the customer switch, and verify the both port channels are up, and all the ports are synchronized status. 14 Chapter 2: UCP 4000 Brocade Systems

23 Uplink Configurations Important: Please make sure to connect each cable one by one, and each time a connection is made verify the status of the port-channel. R1-BR-6740-A-U42# show interface port-channel 10 Port-channel 10 is up, line protocol is up Hardware is AGGREGATE, address is 0027.f8b Current address is 0027.f8b Description: Connecting to management switches. R1-BR-6740-A-U42# show interface port-channel 11 Port-channel 11 is up, line protocol is up Hardware is AGGREGATE, address is 0027.f8b Current address is 0027.f8b Description: Connecting to management switches. 14. Verify all the ports are synchronized status. R1-BR-6740-A-U42# show port-channel 10 LACP Aggregator: Po 10 Aggregator type: Standard Ignore-split is enabled Admin Key: Oper Key 0010 Partner System ID - 0x0001,74-8e-f8-80-5c-c0 Partner Oper Key Member ports on rbridge-id 212: Link: Te 212/0/1 (0xD ) sync: 1 * Link: Te 212/0/2 (0xD ) sync: 1 Link: Te 212/0/3 (0xD ) sync: 1 Link: Te 212/0/4 (0xD ) sync: 1 R1-BR-6740-A-U42# show port-channel 11 LACP Aggregator: Po 11 Aggregator type: Standard Ignore-split is enabled Admin Key: Oper Key 0011 Partner System ID - 0x0001,74-8e-f8-80-5c-c1 Partner Oper Key Member ports on rbridge-id 212: Link: Te 212/0/5 (0xD ) sync: 1 * Link: Te 212/0/6 (0xD ) sync: 1 Link: Te 212/0/7 (0xD ) sync: 1 Link: Te 212/0/8 (0xD ) sync: 1 Chapter 2: UCP 4000 Brocade Systems 15

24 Uplink Configurations Scenario 3: Connecting to Brocade VCS enabled core network using vlag Important: Forming a single VCS fabric across the customer core network and UCP network is NOT supported. If VCS is enabled on the core network and Brocade VDX switches, please use a different VCS ID. Also, please make sure to use vlag technology to connect them. 15. Follow the same steps with Scenario 1: Connecting to multi chassis ether channel technology enabled switches. Important: Please make sure no fabric isl enable and no fabric trunk enable are configured on ten gigabit interfaces. These settings will prevent to form single fabric between customer VCS fabric and UCP VCS fabric. R1-BR-6740-A-U42# show running-config interface TenGigabit 212/0/1 interface TenGigabit 212/0/1 description Port-Channel connecting to customer network no fabric isl enable no fabric trunk enable channel-group 10 mode active type standard lacp timeout long shutdown Brocade ICX 7450 UCP system with Brocade networking model utilizes stacking technology of Brocade ICX 7450 switch. It enables highly redundant networking across UCP system and datacenter core networks. Typically the following 2 scenarios can be applied. By default, port 47 to 48 on both Brocade ICX 7450 switches are dedicated for connecting core network. 16 Chapter 2: UCP 4000 Brocade Systems

25 Uplink Configurations # Scenario Example of Network Topology 1 Connecting to multi chassis ether channel technology enabled switches 2 Connecting to standard switches (non multi chassis channel technology switch) Scenario 1: Connecting to multi chassis ether channel technology enabled switches 16. Login to the Brocade ICX 7450 using SSH. 17. Configure port channel (Link Aggregation) SSH@R1-BR-ICX7450-Stacking#configure terminal SSH@R1-BR-ICX7450-Stacking(config)#lag customer_isl dynamic id 10 SSH@R1-BR-ICX7450-Stacking(config-lag-customer_isl)#ports ethernet 1/1/47 to 1/1/48 ethernet 2/1/47 to 2/1/48 SSH@R1-BR-ICX7450-Stacking(config-lag-customer_isl)#primary-port 1/1/47 SSH@R1-BR-ICX7450-Stacking(config-lag-customer_isl)#deploy LAG customer_isl deployed successfully! SSH@R1-BR-ICX7450-Stacking(config-lag-customer_isl)#exit If ports on the customer s switch are configured using trunk VLAN (tagged VLAN), configure tagged VLAN on uplink ports. SSH@R1-BR-ICX7450-Stacking(config)#vlan 100 ß UCP Management VLAN ID SSH@R1-BR-ICX7450-Stacking(config-vlan-100)#no untagged ethernet 1/1/47 SSH@R1-BR-ICX7450-Stacking(config-vlan-100)#tagged ethernet 1/1/47 Added tagged port(s) ethe 1/1/47 to 1/1/48 ethe 2/1/47 to 2/1/48 to port-vlan Verify spanning tree is running Chapter 2: UCP 4000 Brocade Systems 17

26 Uplink Configurations 802-1w --- VLAN 1 [ STP Instance owned by VLAN 1 ] Bridge IEEE 802.1W Parameters: Bridge Bridge Bridge Bridge Force tx Identifier MaxAge Hello FwdDly Version Hold hex sec sec sec cnt ffff748ef8805cc Default Connect cables to the customer switch, and verify all the ports are up. Important: Please make sure to connect each cable one by one, and each time a connection is made verify the status of the port-channel. SSH@R1-BR-ICX7450-Stacking#show lag id 10 Total number of LAGs: 2 Total number of deployed LAGs: 2 Total number of trunks created:2 (121 available) LACP System Priority / ID: 1 / 748e.f880.5cc0 LACP Long timeout: 120, default: 120 LACP Short timeout: 3, default: 3 === LAG "FCX_ISL" ID 10 (dynamic Deployed) === LAG Configuration: Ports: e 1/1/47 to 1/1/48 e 2/1/47 to 2/1/48 Port Count: 4 Primary Port: 1/1/47 Trunk Type: hash-based LACP Key: Deployment: HW Trunk ID 3 Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name 1/1/47 Up Forward Full 1G 1 Yes N/A 0 748e.f880.5cec 1/1/48 Up Forward Full 1G 1 Yes N/A 0 748e.f880.5cec 2/1/47 Up Forward Full 1G 1 Yes N/A 0 748e.f880.5cec 2/1/48 Up Forward Full 1G 1 Yes N/A 0 748e.f880.5cec Scenario 2: Connecting to standard switches (non-multi chassis ether channel technology enabled switches) 20. Login to the Brocade ICX 7450 using SSH. 21. Configure first port channel (Link Aggregation) SSH@R1-BR-ICX7450-Stacking#configure terminal SSH@R1-BR-ICX7450-Stacking(config)#lag customer_isl dynamic id Chapter 2: UCP 4000 Brocade Systems

27 Uplink Configurations ethernet 1/1/47 ethernet 2/1/47 1/1/47 LAG customer_isl deployed successfully! If ports on the customer s switch are configured using trunk VLAN (tagged VLAN), configure tagged VLAN on uplink ports. SSH@R1-BR-ICX7450-Stacking(config)#vlan 100 ß UCP Management VLAN ID SSH@R1-BR-ICX7450-Stacking(config-vlan-100)#no untagged ethernet 1/1/47 SSH@R1-BR-ICX7450-Stacking(config-vlan-100)#tagged ethernet 1/1/47 Added tagged port(s) ethe 1/1/47 ethe 2/1/47 to port-vlan Configure second port channel (Link Aggregation) SSH@R1-BR-ICX7450-Stacking#configure terminal SSH@R1-BR-ICX7450-Stacking(config)#lag customer_isl dynamic id 11 SSH@R1-BR-ICX7450-Stacking(config-lag-customer_isl)#ports ethernet 1/1/48 ethernet 2/1/48 SSH@R1-BR-ICX7450-Stacking(config-lag-customer_isl)#primary-port 1/1/48 SSH@R1-BR-ICX7450-Stacking(config-lag-customer_isl)#deploy LAG customer_isl deployed successfully! SSH@R1-BR-ICX7450-Stacking(config-lag-customer_isl)#exit If ports on the customer s switch are configured using trunk VLAN (tagged VLAN), configure tagged VLAN on uplink ports. SSH@R1-BR-ICX7450-Stacking(config)#vlan 100 ß UCP Management VLAN ID SSH@R1-BR-ICX7450-Stacking(config-vlan-100)#no untagged ethernet 1/1/48 SSH@R1-BR-ICX7450-Stacking(config-vlan-100)#tagged ethernet 1/1/48 Added tagged port(s) ethe 1/1/48 ethe 2/1/48 to port-vlan Verify spanning tree is running SSH@R1-BR-ICX7450-Stacking#show 802-1w --- VLAN 1 [ STP Instance owned by VLAN 1 ] Bridge IEEE 802.1W Parameters: Bridge Bridge Bridge Bridge Force tx Identifier MaxAge Hello FwdDly Version Hold hex sec sec sec cnt ffff748ef8805cc Default Connect cables to the customer switch, and verify all the ports are up. Chapter 2: UCP 4000 Brocade Systems 19

28 Uplink Configurations Important: Please make sure to connect each cable one by one, and each time a connection is made verify the status of the port-channel. SSH@R1-BR-ICX7450-Stacking#show lag id 10 Total number of LAGs: 3 Total number of deployed LAGs: 3 Total number of trunks created:3 (121 available) LACP System Priority / ID: 1 / 748e.f880.5cc0 LACP Long timeout: 120, default: 120 LACP Short timeout: 3, default: 3 === LAG "FCX_ISL" ID 10 (dynamic Deployed) === LAG Configuration: Ports: e 1/1/47 e 2/1/47 Port Count: 2 Primary Port: 1/1/47 Trunk Type: hash-based LACP Key: Deployment: HW Trunk ID 3 Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name 1/1/47 Up Forward Full 1G 1 Yes N/A 0 748e.f880.5cec 2/1/47 Up Forward Full 1G 1 Yes N/A 0 748e.f880.5cec SSH@R1-BR-ICX7450-Stacking#show lag id 11 Total number of LAGs: 3 Total number of deployed LAGs: 3 Total number of trunks created:3 (121 available) LACP System Priority / ID: 1 / 748e.f880.5cc0 LACP Long timeout: 120, default: 120 LACP Short timeout: 3, default: 3 === LAG "FCX_ISL" ID 11 (dynamic Deployed) === LAG Configuration: Ports: e 1/1/48 e 2/1/48 Port Count: 2 Primary Port: 1/1/48 Trunk Type: hash-based LACP Key: Deployment: HW Trunk ID 3 Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name 1/1/48 Up Forward Full 1G 1 Yes N/A 0 748e.f880.5cec 2/1/48 Up Forward Full 1G 1 Yes N/A 0 748e.f880.5cec 20 Chapter 2: UCP 4000 Brocade Systems

29 3 UCP 4000 Cisco Systems This chapter will cover the network architecture, internal network configuration, and uplink configuration examples for UCP 4000 Cisco Systems. Network Architecture and Configurations Overview In the Cisco Networking model, UCP utilizes Cisco Nexus 9300 series, and its advanced network technologies. Physical Configurations The following table shows configurations of the UCP 4000 with Cisco Networking model. Component Spine / Aggregation layer Leaf / Access layer Cisco Nexus 9332PQ Cisco Nexus 9372PX Configuration Detail 1G Management Cisco Nexus 3048 Server Facing Interfaces ISL Interfaces Max CB500 Chassis with 2- Nic server Max CB500 Chassis with 4- Nic server Uplinks per switch 10G Interfaces 40G Interfaces 16 CB500 chassis (4 chassis per 9372PX pairs) 11 CB500 chassis (2 chassis base 9372PX pair, 3 chassis remaining 3 pairs ) 8 x 10G on Cisco Nexus 9372PX or 6 x 40G on Cisco Nexus 9332PQ Chapter 3: UCP 4000 Cisco Systems 21

30 Network Architecture and Configurations Overview The following table shows port usage of the Cisco Nexus 9332PQ, Nexus9372PX and Nexus 3048 switches that can be used on the UCP 4000 with Cisco Networking model. Cisco Nexus 9332PQ Port Usage Usage Port Counts Speed Uplinks to Core Network 6 40Gbps ISL to other Nexus 9332PQ 2 (L2 Mode Only) 40Gbps ISL to Nexus 9372PX 16 40Gbps Cisco Nexus 9372PX Port Usage Usage Port Counts Speed Uplinks to Core Network 8 (Compute Rack #1 Only) 10Gbps ISL to other Nexus 9372PQ 2 40Gbps ISL to Nexus 9332PX 4 40Gbps ISL to Nexus (Compute Rack #1 Only) 10Gbps Management Server Connectivity 3 (Compute Rack #1 Only) 10Gbps CB500 Blade Server Connectivity (Connecting to CB500 In-Chassis Pass-Throu module) Max 32 (Compute Rack #1) Max 48 (Compute Rack #2-4) 10Gbps Cisco Nexus 3048 Port Usage Usage Port Counts Speed Uplinks to core network 2 1Gbps ISL to other Nexus Gbps ISL to Nexus 9372PX 1 10Gbps CB500 SVP connectivity 1 per CB500 chassis, up to 16 (16 x CB500 chassis) 1Gbps Management server BMC connectivity Storage system management connectivity 3 (Nexus 3048 A side only) 1Gbps 1 1Gbps 22 Chapter 3: UCP 4000 Cisco Systems

31 Network Architecture and Configurations Overview Usage Port Counts Speed Switch management connectivity 1 per switch, up to 10 (Nexus 3048 A side only) 1Gbps Layer 2 / Layer 3 Mode UCP 4000 with Cisco offers 2 types of network layer configurations (Layer 2 / Layer 3 mode). Layer 2 Mode This UCP configuration utilizes 9372PX switches in the access layer and 9332PQ switches in the aggregation layer. vpc is configured between 9372PX access switch pairs and the 9332PQ aggregation switch pair. Each 9372PX switch has two 40G interfaces connected to each 9332PQ switch. All interfaces between the access and aggregation are layer 2 trunk-ports configured in back to back vpc port-channels. All vlans are allowed on the vpc port-channels. The spanning tree root is configured on the aggregation layer. During purchase of the UCP system, there is an option to connect the UCP system to the customer network via 8x10G links offered per 9372PX base rack switch or 6x40G links offered per 9332PQ aggregation switch. Those links can be configured as L2 interfaces or L3 interfaces depending on preference. Chapter 3: UCP 4000 Cisco Systems 23

32 Network Architecture and Configurations Overview Layer 3 Mode This configuration utilizes 9372PX switches in the leaf layer and 9332PQ switches in the spine layer. vpc is configured on all the leaf switch peers. The leaf switches are where the layer 2 and layer 3 boundary is formed. Additionally IGP peering is also configured on a vlan interface for redundancy purposes on each leaf switch pair in the event there is an uplink failure. Individual layer 3 links are configured between the spine and leaf switches. During purchase of the UCP system there is an option to connect the UCP system to the customer network via 8x10G links offered per 9372PX base rack switch or 6x40G links offered per 9332PQ spine switch. Those links can be configured in as L2 links or L3 links depending on preference. UCP Director Network Management Features Onboarding Switches (Layer 2 Mode) The UCP Director supports onboarding of the Nexus 9372PX access switches and 9332PQ aggregation switches for management and health monitoring. Onboarding Switches (Layer 3 mode) The UCP Director supports onboarding of the Nexus 9372PX leaf switch only for management and health monitoring. Currently the 9332PQ spine switch onboarding is not supported. 24 Chapter 3: UCP 4000 Cisco Systems

33 Network Architecture and Configurations Overview Switch Health Monitoring The UCP Director monitors the health of the onboard switches and notifies if there are any warnings or errors detected on the switches. Configuration Backup UCP Director will periodically take backup configurations of the switches in its inventory. UCP Director can also manually backup/apply the switch configurations through user request. Configure Host/Cluster Network This allows the user to configure the Nexus 9372PX server facing interface on a blade by blade basis. The native vlan which is configured and managed by UCP for the management network and additional vlans such as vmotion or Compute vlans can also be managed through this UCP Director feature. VLAN DB management (Layer 2 mode) - If a new vlan is added through the Configure Host/Cluster Network feature, the vlan is added to the pair of Nexus 9372PX server facing interfaces as an allowed vlan and is created in the vlan db. In addition, the vlan is added to the Nexus 9332PQ vlan db and any/all additional Nexus 9372PX pairs vlan database in other racks. VLAN DB management (Layer 3 mode) - If a new vlan is added through the Configure Host/Cluster Network feature, the vlan is added to the pair of Nexus 9372PX server facing interfaces as an allowed vlan and is created in the vlan db. UCP Appliance Initial vpc and Spanning Tree Configuration vpc peer-link The vpc peerlink trunk port-channel interface which allows traffic to ingress/egress between vpc peers also allows all vlans by default. vpc keep-alive The vpc keep-alive link is established through the management vrf with the source as MGMT 0 IP and destination as the MGMT 0 IP on the peer. vpc orphan-port suspend In the event that there is a vpc peer-link failure, all server facing vpc orphan ports will be disabled to avoid a vpc dual-active scenario. vpc peer-gateway Peer gateway is configured on vpc peers to act as the gateway even when packets are destined to the vpc peers MAC address. Chapter 3: UCP 4000 Cisco Systems 25

34 Network Architecture and Configurations Overview vpc peer-switch Allows both vpc peers to act as Root Bridge for the vlan. vpc ip arp synchronize Allows both vpc peers to synchronize arp entries for faster ARP learning and convergence. vpc delay restore 240 In the event of switch reboot, the vpc bring-up is delayed by 240 seconds to allow network re-converge before bringing the vpc peer up. Rapid-PVST Mode (Layer 2 mode) Spanning tree rapid-pvst mode is configured with the spanning tree root configured on the 9332PQ aggregation switches with priority The access 9372PX switches have priority setting Rapid-PVST Mode (Layer 3 mode) Spanning tree rapid-pvst mode is configured with the priority setting on the 9372PX leaf switches. vpc port-channels (Layer 2 mode) - Back-to-Back vpc trunk portchannel is configured between aggregation and access switches, enabled all vlans by default. Also, Back-to-Back vpc trunk port-channel is configured between 1G switch (Nexus3048) and access switches on compute rack #1, enabled management vlan by default. vpc port-channels (Layer 3 mode) - Back-to-Back vpc trunk portchannel is configured between 1G switch (Nexus3048) and access switches on compute rack #1, enabled management vlan by default. Sample vpc Configuration vpc domain 901 peer-switch role priority 100 peer-keepalive destination source delay restore 240 peer-gateway auto-recovery ipv6 nd synchronize ip arp synchronize Sample Spanning tree configuration spanning-tree vlan priority Chapter 3: UCP 4000 Cisco Systems

35 Internal Network Configurations (Layer 3 mode) Server Facing Port Configurations The UCP Director manages the server facing ports which are connected to the CB500 chassis via passthru modules that connect to blades. These server facing ports are connected via the 10G ports on the Nexus 9372PX switches. When managing the server facing ports, UCP Director sets the port in switchport trunk mode. In addition, it sets the native vlan on the port to the specified management vlan id which is chosen during deployment. It is critical these settings are never modified because the native vlan is used by UCP Director for management functions. Finally, the port is set to allow specific vlans which are configured by UCP Director through the switchport trunk allowed vlan command. Once user needs to modify the allowed vlans they can perform a Configure host/cluster vlan operation and UCP Director will modify the port configuration accordingly. The "spanning-tree port type edge trunk" and "vpc orphan-port suspend" are configured on all of server facing ports during UCP Deployment. UCP Director Software will not touch these configuration. HDS recommends not to change these configuration. Sample Port Configuration interface 1/1 description to_blade_server switchport mode trunk switchport trunk native vlan 92 switchport trunk allowed vlan spanning-tree port type edge trunk vpc orphan-port suspend Internal Network Configurations (Layer 3 mode) Note: Skip this section if the UCP 4000 System is running in Layer 2 mode. This section describes how to configure the UCP Internal Network after UCP Initial Deployment in Layer 3 mode. Chapter 3: UCP 4000 Cisco Systems 27

36 Internal Network Configurations (Layer 3 mode) In Layer 3 mode, there are a couple of design choices, and configuration options as shown below. These design choices should be determined before starting UCP Internal Network Configuration. Design Options in Layer 3 mode (*This chapter provides sample configurations for bold highlighted options.) Item Routing Protocol First Hop Redundancy Protocol Isolation / Access Control Overlay Network VM Migration Design Options OSPF, ebgp, ibgp, EIGRP, Static Route HSRP, VRRP ACL, VRF for UCP Management Traffic None, VXLAN, VMware NSX Layer 2 VM Migration with VXLAN, Layer 3 VM Migration Important: Layer 3 VM Migration has limited support in UCPv4.1. (Cluster Service Template feature doesn't support ESXi Cluster creation with Layer 3 vmotion vmkernel.) Note: This guide covers following combination of Routing Protocl and VXLAN method listed in below matrix table. Routing Protocol (underlay) VXLAN (Overlay) Covered in this document OSPF VXLAN Flood and Learn Yes ebgp VXLAN Flood and Learn Yes OSFP VXLAN EVPN Yes ebgp VXLAN EVPN Not documented Following table shows overall configuration workflow. # Steps 1 Configure VRF ucpmanagment 2 Configure UCP Management SVI 3 Configure VM Traffic SVI 28 Chapter 3: UCP 4000 Cisco Systems

37 Internal Network Configurations (Layer 3 mode) # Steps 4 Configure Unicast Routing 5 Configure VXLAN (Run this step if Layer 2 VM Migration chosen and/or any overlay network needed for VM traffics.) Network Design Table Before continuing further please print out the following table of network design requirements and host counts to configure the future sections. Two columns are filled with subnet and host default values and used in the configuration examples below. Gather the customer preference of available subnets and hosts in their data-center and substitute accordingly. Also, take note of whether VXLAN will be configured or not. If not, skip past those fields. The notes column lets you know what will be used where. Note: In reality, you may not want to spread out the Spine 9332 to Leaf 9372 subnets across such a large address space to etc. This was used in the example configurations for simplicity. They may provide a /24 address space and ask to break it up into /31 networks for each interface. Network Design Requirements Interface Subnets (Example used in this document) Mas k Hos t Size Hosts Usage (Example used in this documen t) Notes Loopback x/32 /32 10 host s 212, 213, 214, 215, 216, 217, 218, 219, 220, 221 Router-id for global ospf or ebgp Chapter 3: UCP 4000 Cisco Systems 29

38 Internal Network Configurations (Layer 3 mode) Interface Subnets (Example used in this document) Mas k Hos t Size Hosts Usage (Example used in this documen t) Notes Loopback x/32 /32 10 host s 212, 213, 214, 215, 216, 217, 218, 219, 220, 221 Router-id for vrf ucpmanagemen t 9332A to 9372A /31, /31 /31 4 host s 10,11,20,2 1 L3 interfaces 9332A to 9372B /31, /31 /31 4 host s 10,11,20,2 1 L3 interfaces 9332A to 9372C /31, /31 /31 4 host s 10,11,20,2 1 L3 interfaces 9332A to 9372D /31, /31 /31 4 host s 10,11,20,2 1 L3 interfaces 9332A to 9372E /31, /31 /31 4 host s 10,11,20,2 1 L3 interfaces 9332A to 9372F /31, /31 /31 4 host s 10,11,20,2 1 L3 interfaces 9332A to 9372G /31, /31 /31 4 host s 10,11,20,2 1 L3 interfaces 9332A to 9372H /31, /31 /31 4 host s 10,11,20,2 1 L3 interfaces 9332B to 9372A /31, /31 /31 4 host s 10,11,20,2 1 L3 interfaces 30 Chapter 3: UCP 4000 Cisco Systems

39 Internal Network Configurations (Layer 3 mode) Interface Subnets (Example used in this document) Mas k Hos t Size Hosts Usage (Example used in this documen t) Notes 9332B to 9372B /31, /31 /31 4 host s 10,11,20,2 1 L3 interfaces 9332B to 9372C /31, /31 /31 4 host s 10,11,20,2 1 L3 interfaces 9332B to 9372D /31, /31 /31 4 host s 10,11,20,2 1 L3 interfaces 9332B to 9372E /31, /31 /31 4 host s 10,11,20,2 1 L3 interfaces 9332B to 9372F /31, /31 /31 4 host s 10,11,20,2 1 L3 interfaces 9332B to 9372G /31, /31 /31 4 host s 10,11,20,2 1 L3 interfaces 9332B to 9372H /31, /31 /31 4 host s 10,11,20,2 1 L3 interfaces 9372A to 9372B 9372C to 9372D /31 /31 2 host s /31 /31 2 host s 212,213 IGP between Leafs 214,215 IGP between Leafs 9372E to 9372F /31 /31 2 host s 216,217 IGP between Leafs Chapter 3: UCP 4000 Cisco Systems 31

40 Internal Network Configurations (Layer 3 mode) Interface Subnets (Example used in this document) Mas k Hos t Size Hosts Usage (Example used in this documen t) Notes 9372G to 9372H /31 /31 2 host s 218,219 IGP between Leafs Compute Rack 1 SVI ucpmanagemen t Compute Rack 2 SVI ucpmanagemen t Compute Rack 3 SVI ucpmanagemen t Compute Rack 4 SVI ucpmanagemen t x/24 /24 N/A 1,2,3 GW, HSRP x/24 /24 N/A 1,2,3 GW, HSRP x/24 /24 N/A 1,2,3 GW, HSRP x/24 /24 N/A 1,2,3 GW, HSRP Loopback 0 Secondary Loopback 0 Secondary Loopback 0 Secondary Loopback 0 Secondary x/32 /32 1 host x/32 /32 1 host x/32 /32 1 host x/32 /32 1 host 101 VXLAN only 102 VXLAN only 103 VXLAN only 104 VXLAN only Loopback /3 2 /32 1 host 254 VXLAN only Multicast Group x/24 /32 N/A N/A VXLAN only 32 Chapter 3: UCP 4000 Cisco Systems

41 Internal Network Configurations (Layer 3 mode) If the customer is using ebgp as the global routing protocol gather the AS# s as well and fill into the column. Example of ebgp AS Assignment Rack Location Switch Router AS # (Example used in this document) Router-id Compute Rack #1 9332A-U loopback0 Compute Rack #1 9332B loopback0 Compute Rack #1 9372A-U loopback0 Compute Rack #1 9372B-U loopback0 Compute Rack #2 9372C loopback0 Compute Rack #2 9372D-U loopback0 Compute Rack #3 9372E loopback0 Compute Rack #3 9372F-U loopback0 Compute Rack #4 9372G loopback0 Compute Rack #4 9372H-U loopback0 Add Layer 3 License Optional license (N93-LAN1K9 LAN_ENTERPRISE_SERVICES_PKG) is required for enterprise Layer 3 features on Cisco Nexus 9000 switch. The following features are included in this license. Open Shortest Path First (OSPF) Protocol Border Gateway Protocol (BGP) Intermediate System-to-Intermediate System (IS-IS) Protocol (Layer 3 only) Protocol Independent Multicast (PIM), which includes sparse mode, bidirectional mode, and Source-Specific Multicast (SSM) mode Multicast Source Discovery Protocol (MSDP) Policy-Based Routing Generic routing encapsulation (GRE) tunnels Chapter 3: UCP 4000 Cisco Systems 33

42 Internal Network Configurations (Layer 3 mode) Enhanced Interior Gateway Routing Protocol (EIGRP) VXLAN BGP evpn control plane Please install the Layer 3 license on all Nexus 9332PQ and Nexus 9372PX Switches before configuring layer 3 features. It should be included in the order with the UCP purchase. 1. Login to the all Nexus 9372 and 9332 switches using SSH. 2. Verify "LAN_ENTERPRISE_SERVICES_PKG" license is not installed. If it is already installed, skip rest of the steps for this switch, and move to next switch. switch# show license usage Feature Ins Lic Status Expiry Date Comments Count LAN_ENTERPRISE_SERVICES_PKG No - Unused Obtain the serial number of the Nexus 9300 switch through the following command. switch# show license host-id License hostid: VDH=FOX064317SQ 4. Take the serial # that follows the = sign. 5. Obtain the PAK(Product Authorization Key) 6. Locate the Website URL from the PAK key or go to 7. Input the serial # and PAK key and other information the webpage asks. A license file will be generated and ed to you. 34 Chapter 3: UCP 4000 Cisco Systems

43 Internal Network Configurations (Layer 3 mode) 8. Copy the license file to the bootflash of the Nexus 9300 device using supported file transfer command. Below is example using scp. copy scp://username@ /license.lic bootflash://license.lic 9. Login to the switch, and install the license switch# install license bootflash:license.lic Installing license..done 10. Verify license is installed switch# show license usage Feature Ins Lic Status Expiry Date Comments Count LAN_ENTERPRISE_SERVICES_PKG Yes - In use Never Configure Unicast Routing for VRF ucpmanagement In this section, we go over the Nexus 9300 series switches and configurations to bring up OSPF routing protocol for the VRF ucpmanagement. This section explains how to enable and configure OSPF routing between the Nexus 9332PQ spine switch and Nexus 9372PX leaf switches. Important: After the initial deployment by UCP deployment tool, "ucpmanagement" vrf and SVI for ucpmanagement is pre-defined on Cisco Nexus 9372PX and 9332PQ switches. If ACL, Firewall, or other function will be used for network isolation / access control to UCP management network stack, please remove "ucpmanagement" vrf, reconfigure ucpmanagement SVI, and skip this section. Chapter 3: UCP 4000 Cisco Systems 35

44 Internal Network Configurations (Layer 3 mode) Verify Virtual Routing and Forwarding (VRF) VRF s are used to isolate Layer 3 traffic from the default vrf or global routed traffic. This is done so that hypervisor management and UCP management traffic routes are not reachable from the global routing table and makes the management network more secure. By UCP default, "ucpmanagement" vrf is pre-defined on Cisco Nexus 9372PX and 9332PQ switches. 1. Login to the all Nexus 9372 and 9332 switches using SSH. 2. Verify vrf 'ucpmanagement' is configured on all Nexus 9372PX and 9332PQ switches by running following command. R1-CS-9372-A-# show vrf VRF-Name VRF-ID State Reason default 1 Up -- management 2 Up -- ucpmanagement 3 Up -- OSPF for VRF ucpmanagement OSPF is a layer 3 interior gateway protocol (IGP). The feature must be enabled and a process must be created. Finally, OSPF must be enabled on IPv4/v6 individual interfaces which need to be advertised to neighbors. Feature OSPF To enable OSPF on the switch we first need to enable the feature ospf. This needs to be done on all spine and leaf switches. 3. Run following command on all Nexus 9332PQ and Nexus 9372PX switches. R1-CS-9332-A-U42# conf t Enter configuration commands, one per line. End with CNTL/Z. R1-CS-9332-A-U42(config)# feature ospf Router OSPF Process We need to create the router ospf process next. We have named it ucpmanagement and specified the loopback 1 interface ip address as the router-id. The router-id is what neighboring routers will see when uniquely identifying the router. This needs to be done on all spine and leaf switches. Here we create the ospf process, setup the vrf ucpmanagment in the ospf process, assign the router id for peering, and assign the maximum paths. 36 Chapter 3: UCP 4000 Cisco Systems

45 Internal Network Configurations (Layer 3 mode) 4. Run following commands on all Nexus 9332PQ and Nexus 9372PX switches. R1-CS-9332-A-U42# conf t Enter configuration commands, one per line. End with CNTL/Z. R1-CS-9332-A-U42 (config)# router ospf ucpmanagement R1-CS-9332-A-U42 (config-router)# vrf ucpmanagement R1-CS-9332-A-U42 (config-router-vrf)# router-id <- assign via the table below R1-CS-9332-A-U42 (config-router-vrf)# maximum-paths 32 Configure Loopback 1 The loopback 1 interface should be configured on each device, a suggested configuration scheme is listed below. This loopback interface will be used as the router id as well as for other uses. Configuration for Loopback 1 needs to be configured on all spine and leaf switches according to the table below. 5. Run following commands on all Nexus 9332PQ and Nexus 9372PX switches. R1-CS-9332-A-U42(config)# interface loopback1 <- Creates the interface R1-CS-9332-A-U42(config-if)# vrf member ucpmanagement <- assigns to vrf ucpmanagement R1-CS-9332-A-U42(config-if)# ip address /32 <- assigns the ip address R1-CS-9332-A-U42(config-if)# ip router ospf ucpmanagement area <- advertise loopback 1 in the vrf R1-CS-9332-A-U42(config-if)# no shut Rack Location Compute Rack #1 Compute Rack #1 Compute Rack #1 Switch 9332A- U B- 9372A- U39 Loopback 1 IP/ Router-id VRF OSPF Process /32 ucpmanagement ucpmanagement /32 ucpmanagement ucpmanagement /32 ucpmanagement ucpmanagement Chapter 3: UCP 4000 Cisco Systems 37

46 Internal Network Configurations (Layer 3 mode) Rack Location Compute Rack #1 Compute Rack #2 Compute Rack #2 Compute Rack #3 Compute Rack #3 Compute Rack #4 Compute Rack #4 Switch 9372B- U C- 9372D- U E- 9372F- U G- 9372H- U39 Loopback 1 IP/ Router-id VRF OSPF Process /32 ucpmanagement ucpmanagement /32 ucpmanagement ucpmanagement /32 ucpmanagement ucpmanagement /32 ucpmanagement ucpmanagement /32 ucpmanagement ucpmanagement /32 ucpmanagement ucpmanagement /32 ucpmanagement ucpmanagement Subinterfaces Subinterfaces are a division of a physical interface into multiple logical interfaces. We use the subinterfaces for the vrf ucpmanagement so that the entire interface does not need to be allocated for ucpmanagement. The physical interface as well as remaining sub-interfaces can we utilized for other purposes. 6. Configure the subinterfaces on each spine and leaf switch according to the IP Scheme table below. R1-CS-9372-A-U39(config)# interface 1/49.1 <- creates the interface R1-CS-9372-A-U39(config-subif)# mtu 9216 R1-CS-9372-A-U39(config-subif)# encapsulation dot1q 101 <- a unique encapsulation for the subinterface R1-CS-9372-A-U39(config-subif)# vrf member ucpmanagement R1-CS-9372-A-U39(config-subif)# ip address /31 <- set the ip via scheme below R1-CS-9372-A-U39(config-subif)# ip ospf network point-to-point R1-CS-9372-A-U39(config-subif)# ip router ospf ucpmanagement area <- advertise via area 0 R1-CS-9372-A-U39(config-subif)# no shutdown 38 Chapter 3: UCP 4000 Cisco Systems

47 Internal Network Configurations (Layer 3 mode) Note: If configuring the mtu 9216 gives an error configure the mtu 9216 on the parent interface first. Parent interface of 1/49.1 would be 1/49. In the configuration example, we use encapsulation dot1q 101 as the tag for the subinterface. The subinterface is then added to the vrf ucpmanagement and ip address and ospf peering is configured. The interface is now part of the vrf ucpmanagement and isolated from the global routing table. In addition, OSPF peering has been enabled so all neighboring routers will learn the routes. Cisco Nexus9332-A IP Scheme Link # Switch Interface IP Encapsulation Speed Neighbor A- U A- U A- U A- U A- U A- U A- U A- U A- U A- U A- U41 1/1.1 1/2.1 1/3.1 1/4.1 1/5.1 1/6.1 1/7.1 1/8.1 1/9.1 1/10.1 1/ /31 dot1q G 9372A- U /31 dot1q G 9372A- U /31 dot1q G 9372B- U /31 dot1q G 9372B- U /31 dot1q G 9372C /31 dot1q G 9372C /31 dot1q G 9372D- U /31 dot1q G 9372D- U /31 dot1q G 9372E /31 dot1q G 9372E /31 dot1q G 9372F- U39 Chapter 3: UCP 4000 Cisco Systems 39

48 Internal Network Configurations (Layer 3 mode) Link # Switch Interface IP Encapsulation Speed Neighbor A- U A- U A- U A- U A- U41 1/12.1 1/13.1 1/14.1 1/15.1 1/ /31 dot1q G 9372F- U /31 dot1q G 9372G /31 dot1q G 9372G /31 dot1q G 9372H- U /31 dot1q G 9372H- U39 Cisco Nexus9332-B IP Scheme Link # Switch Interface IP Encapsulation Speed Neighbor B B B B B B B B B- 1/1.1 1/2.1 1/3.1 1/4.1 1/5.1 1/6.1 1/7.1 1/8.1 1/ /31 dot1q G 9372A- U /31 dot1q G 9372A- U /31 dot1q G 9372B- U /31 dot1q G 9372B- U /31 dot1q G 9372C /31 dot1q G 9372C /31 dot1q G 9372D- U /31 dot1q G 9372D- U /31 dot1q G 9372E- 40 Chapter 3: UCP 4000 Cisco Systems

49 Internal Network Configurations (Layer 3 mode) Link # Switch Interface IP Encapsulation Speed Neighbor B B B B B B B- 1/10.1 1/11.1 1/12.1 1/13.1 1/14.1 1/15.1 1/ /31 dot1q G 9372E /31 dot1q G 9372F- U /31 dot1q G 9372F- U /31 dot1q G 9372G /31 dot1q G 9372G /31 dot1q G 9372H- U /31 dot1q G 9372H- U39 Cisco Nexus9372-A IP Scheme Link # Switch Interface IP Encapsulation Speed Neighbor A- U A- U A- U A- U39 1/49.1 1/50.1 1/51.1 1/ /31 dot1q G 9332A- U /31 dot1q G 9332A- U /31 dot1q G 9332B /31 dot1q G 9332B- Cisco Nexus9372-B IP Scheme Link # Switch Interface IP Encapsulation Speed Neighbor B- U38 1/ /31 dot1q G 9332A- U41 Chapter 3: UCP 4000 Cisco Systems 41

50 Internal Network Configurations (Layer 3 mode) Link # Switch Interface IP Encapsulation Speed Neighbor B- U B- U B- U38 1/50.1 1/51.1 1/ /31 dot1q G 9332A- U /31 dot1q G 9332B /31 dot1q G 9332B- Cisco Nexus9372-C IP Scheme Link # Switch Interface IP Encapsulation Speed Neighbor C C C C- 1/49.1 1/50.1 1/51.1 1/ /31 dot1q G 9332A- U /31 dot1q G 9332A- U /31 dot1q G 9332B /31 dot1q G 9332B- Cisco Nexus9372-D IP Scheme Link # Switch Interface IP Encapsulation Speed Neighbor D- U D- U D- U D- U39 1/49.1 1/50.1 1/51.1 1/ /31 dot1q G 9332A- U /31 dot1q G 9332A- U /31 dot1q G 9332B /31 dot1q G 9332B- 42 Chapter 3: UCP 4000 Cisco Systems

51 Internal Network Configurations (Layer 3 mode) Cisco Nexus9372-E IP Scheme Link # Switch Interface IP Encapsulation Speed Neighbor E E E E- 1/49.1 1/50.1 1/51.1 1/ /31 dot1q G 9332A- U /31 dot1q G 9332A- U /31 dot1q G 9332B /31 dot1q G 9332B- Cisco Nexus9372-F IP Scheme Link # Switch Interface IP Encapsulation Speed Neighbor F- U F- U F- U F- U39 1/49.1 1/50.1 1/51.1 1/ /31 dot1q G 9332A- U /31 dot1q G 9332A- U /31 dot1q G 9332B /31 dot1q G 9332B- Cisco Nexus9372-G IP Scheme Link # Switch Interface IP Encapsulation Speed Neighbor G G G G- 1/49.1 1/50.1 1/51.1 1/ /31 dot1q G 9332A- U /31 dot1q G 9332A- U /31 dot1q G 9332B /31 dot1q G 9332B- Chapter 3: UCP 4000 Cisco Systems 43

52 Internal Network Configurations (Layer 3 mode) Cisco Nexus9372-H IP Scheme Link # Switch Interface IP Encapsulation Speed Neighbor H- U H- U H- U H- U39 1/49.1 1/50.1 1/51.1 1/ /31 dot1q G 9332A- U /31 dot1q G 9332A- U /31 dot1q G 9332B /31 dot1q G 9332B- Verify OSPF neighborships 7. Verify that the L3 ospf neighborships are in FULL state on all L3 interfaces that are connected to neighboring switches. R1-CS-9332-A-U42# sh ip ospf neighbors vrf ucpmanagement OSPF Process ID UNDERLAY VRF default Total number of neighbors: 8 Neighbor ID Pri State Up Time Address Interface FULL/ - 1d05h Eth1/ FULL/ - 1d05h Eth1/ FULL/ - 2d04h Eth1/ FULL/ - 2d04h Eth1/ FULL/ - 2d04h Eth1/ FULL/ - 2d04h Eth1/ FULL/ - 2d04h Eth1/ FULL/ - 2d04h Eth1/ FULL/ - 1d05h Eth1/ FULL/ - 1d05h Eth1/ FULL/ - 2d04h Eth1/ FULL/ - 2d04h Eth1/ FULL/ - 2d04h Eth1/ FULL/ - 2d04h Eth1/ FULL/ - 2d04h Eth1/ FULL/ - 2d04h Eth1/16.1 Configure OSPF on the UCP Management SVI 8. Configure ospf on ucp management svi of all Nexus9372PX leaf switches (A, B, C, D, E, F, G, H). R1-CS-9372-A-U39(config)# interface vlan 100 <- Use "UCP Management VLAN ID" 44 Chapter 3: UCP 4000 Cisco Systems

53 Internal Network Configurations (Layer 3 mode) R1-CS-9372-A-U39(config-if)# ip router ospf ucpmanagement area <- Advertise the SVI Verify UCP Management SVI Basic UCP Management SVI configurations will be configured after running UCP Deployment Tool (except OSPF). 9. Verify UCP Management SVI by running below commands on all Nexus9372PX switch. R1-CS-9372-A-# show run interface vlan 100 <- Use "UCP Management VLAN ID"!Command: show running-config interface Vlan100!Time: Tue Oct 13 04:34: version 7.0(3)I1(2) interface Vlan100 <- Must be same "UCP Management VLAN ID" across all racks. no shutdown mtu 9216 vrf member ucpmanagement <- vrf "ucpmanagment" by UCP default ip address /24 <- SVI IP Address. See below table. ip router ospf ucpmanagement area hsrp version 2 hsrp 100 preempt ip <- Gateway IP Address. See below table. ip dhcp relay address use-vrf ucpmanagement <- UCP Utility VM (DHCP Server) IP Address Cisco Nexus9372 Management SVI IP Scheme Rack Location Switch SVI IP HSRP Gateway Mgmt Vlan ID Compute Rack #1 Compute Rack #1 Compute Rack #2 Compute Rack #2 9372A-U / B-U / C / D-U / Chapter 3: UCP 4000 Cisco Systems 45

54 Internal Network Configurations (Layer 3 mode) Rack Location Switch SVI IP HSRP Gateway Mgmt Vlan ID Compute Rack #3 Compute Rack #3 Compute Rack #4 Compute Rack #4 9372E / F-U / G / H-U / Configure VM Traffic SVI 10. Configure SVI for VM Traffic as needed, by running below commands on all Nexus9372PX switch. Configure SVI on the 9372 A, C, E, G Leaf switch. R1-CS-9372-C-(config)# int vlan 103 R1-CS-9372-C-(config-if)# description VM Data subnet on rack 2 R1-CS-9372-C-(config-if)# mtu 9216 R1-CS-9372-C-(config-if)# no shut R1-CS-9372-C-(config-if)# ip address /24 <- SVI IP Address R1-CS-9372-C-(config-if)# hsrp version 2 R1-CS-9372-C-(config-if)# hsrp 103 R1-CS-9372-C-(config-if-hsrp)# preempt R1-CS-9372-C-(config-if-hsrp)# priority 100 <- Must be higher priority (smaller value) than B, D, F, H switch to match vpc primary R1-CS-9372-C-(config-if-hsrp)# ip <- Gateway IP address Configure SVI on the 9372 B, D, F, H Leaf switch. R1-CS-9372-D-U39(config)# int vlan 103 <- Must use same UCP Management VLAN ID for all racks. R1-CS-9372-D-U39(config-if)# description VM Data subnet on rack 2 R1-CS-9372-D-U39(config-if)# mtu 9216 R1-CS-9372-D-U39(config-if)# no shut R1-CS-9372-D-U39(config-if)# ip address /24<- SVI IP Address R1-CS-9372-D-U39(config-if)# hsrp version 2 R1-CS-9372-D-U39(config-if)# hsrp 103 R1-CS-9372-D-D39(config-if-hsrp)# preempt 46 Chapter 3: UCP 4000 Cisco Systems

55 Internal Network Configurations (Layer 3 mode) R1-CS-9372-D-U39(config-if-hsrp)# priority 110 <- Must be lower priority (higher value) than A, C, E, G switch to match vpc seconday R1-CS-9372-D-U39(config-if-hsrp)# ip <- Gateway IP address Note: The priority value is different on 9372 A, C, E, G and B, D, F, H. Cisco recommends to match HSRP active/standby and vpc primary/seconday. Configure Global Unicast Routing In this section, we go over the Nexus 9300 series switches and configurations to bring up OSPF or BGP routing protocols. This section explains how to enable and configure OSPF and ebgp routing between the Nexus 9332PQ spine switch and Nexus 9372PX leaf switch. Note: Please decide now whether OSPF or ebgp will be the routing protocol of choice and skip to related section. OSPF OSPF is a layer 3 interior gateway protocol (IGP). The feature must be enabled and a process must be created. Finally, OSPF must be enabled on IPv4/v6 individual interfaces which need to be advertised to neighbors. Feature OSPF 1. To enable OSPF on the switch we first need to enable the feature ospf. This needs to be done on all Nexus 9332 spine and Nexus 9372 leaf switches. R1-CS-9332-A-U42# conf t Enter configuration commands, one per line. End with CNTL/Z. R1-CS-9332-A-U42(config)# feature ospf Router OSPF Process We need to create the router ospf process next. We have named it 100 in this example and specify the loopback 0 primary ip address as the router-id. The router-id identifies the OSPF instance. This needs to be done on all spine and leaf switches. Chapter 3: UCP 4000 Cisco Systems 47

56 Internal Network Configurations (Layer 3 mode) 2. Configure the OSPF Process for all Nexus9332 and Nexus9372 switches, use the table below to complete. R1-CS-9332-A-U42(config)# router ospf 100 R1-CS-9332-A-U42(config-router)# router-id Rack Location Switch OSPF ID Router-id Compute Rack #1 9332A-U Compute Rack #1 9332B Compute Rack #1 9372A-U Compute Rack #1 9372B-U Compute Rack #2 9372C Compute Rack #2 9372D-U Compute Rack #3 9372E Compute Rack #3 9372F-U Compute Rack #4 9372G Compute Rack #4 9372H-U Configure OSPF on VM Traffic SVI 3. If you need VM Traffic SVI to be advertised, please run the following commands on a switch by switch and svi by svi basis. R1-CS-9372-A-U39(config-if)#interface Vlan101 R1-CS-9372-A-U39(config-if)# ip router ospf 100 area ebgp Note: If OSPF was chosen as the routing protocol of choice, please skip this section. Otherwise, please continue. BGP is becoming commonly used in Spine Leaf topologies because of its scalability and flexibility. 48 Chapter 3: UCP 4000 Cisco Systems

57 Internal Network Configurations (Layer 3 mode) Feature BGP 4. To enable BGP on the switch we first need to enable the feature BGP. This needs to be done on all Nexus 9332 and Nexus 9372 switches. R1-CS-9332-A-U42# conf t Enter configuration commands, one per line. End with CNTL/Z. R1-CS-9332-A-U42(config)# feature bgp Router BGP Process We need to create the router bgp process next. The Spine (Nexus 9332) is configured in its own Autonomous System (AS) and each of the Leafs (Nexus 9372) are configured in separate AS s thus external BGP is used. The router-id identifies the BGP instance. 5. Configure the BGP Process for all Nexus9332 and Nexus9372 switches, use the table below to complete. R1-CS-9332-A-U42(config)# router bgp R1-CS-9332-A-U42(config-router)# router-id <- assign the router id Rack Location Switch Router AS # Router-id Compute Rack #1 9332A-U Chapter 3: UCP 4000 Cisco Systems 49

58 Internal Network Configurations (Layer 3 mode) Rack Location Switch Router AS # Router-id Compute Rack #1 9332B Compute Rack #1 9372A-U Compute Rack #1 9372B-U Compute Rack #2 9372C Compute Rack #2 9372D-U Compute Rack #3 9372E Compute Rack #3 9372F-U Compute Rack #4 9372G Compute Rack #4 9372H-U Configure BGP Routing 6. On the Nexus 9332A and Nexus 9332B spine switches, we create the BGP process and configure the advertised networks as well as neighborships. R1-CS-9332-A-U42# conf t Enter configuration commands, one per line. End with CNTL/Z. R1-CS-9332-A-U42(config)#router bgp <- AS number will be same for both Spine switches R1-CS-9332-A-U42(config-router)# router-id <- primary loopback 0 ip address R1-CS-9332-A-U42(config-router)# graceful-restart-helper R1-CS-9332-A-U42(config-router)# log-neighbor-changes R1-CS-9332-A-U42(config-router-af)# address-family ipv4 unicast R1-CS-9332-A-U42(config-router-af)# network /32 <- add primary loopback 0 ip address R1-CS-9332-A-U42(config-router-af)# network /32 <- advertise anycast rp address if VXLAN will be used R1-CS-9332-A-U42(config-router-af)# network /31 <- advertise interface ip as needed... R1-CS-9332-A-U42(config-router-af)# maximum-paths 32 R1-CS-9332-A-U42(config-router-neighbor)# template peer BGPLEAF <- template to save time on neighborships configuration R1-CS-9332-A-U42(config-router-neighbor)# address-family ipv4 unicast R1-CS-9332-A-U42(config-router-neighbor)# default-originate R1-CS-9332-A-U42(config-router-neighbor)# next-hop-self R1-CS-9332-A-U42(config-router-neighbor)# soft-reconfiguration inbound 50 Chapter 3: UCP 4000 Cisco Systems

59 Internal Network Configurations (Layer 3 mode) R1-CS-9332-A-U42(config-router-neighbor)# neighbor remoteas <- create neighborships on all L3 links between spine and leaf layer. Follow the table below. R1-CS-9332-A-U42(config-router-neighbor)# inherit peer BGPLEAF R1-CS-9332-A-U42(config-router-neighbor)# neighbor remoteas R1-CS-9332-A-U42(config-router-neighbor)# inherit peer BGPLEAF... Nexus 9332 A BGP Routing Switch Networks Neighbor Remote-AS Neighbor 9332A-U / A-U A-U / A-U A-U / B-U A-U / B-U A-U / C- 9332A-U / C- 9332A-U / D-U A-U / D-U A-U / E- 9332A-U / E- 9332A-U / F-U A-U / F-U A-U / G- 9332A-U / G- 9332A-U / H-U A-U / H-U A-U / B- 9332A-U /32 N/A N/A N/A 9332A-U /32 N/A N/A N/A Chapter 3: UCP 4000 Cisco Systems 51

60 Internal Network Configurations (Layer 3 mode) Nexus 9332 B BGP Routing Switch Network Statement Neighbor Remote-AS Neighbor 9332B- 9332B- 9332B- 9332B- 9332B- 9332B- 9332B- 9332B- 9332B- 9332B- 9332B- 9332B- 9332B- 9332B- 9332B- 9332B / A-U / A-U / B-U / B-U / C / C / D-U / D-U / E / E / F-U / F-U / G / G / H-U / H-U39 52 Chapter 3: UCP 4000 Cisco Systems

61 Internal Network Configurations (Layer 3 mode) Switch Network Statement Neighbor Remote-AS Neighbor 9332B- 9332B- 9332B / A-U /32 N/A N/A N/A /32 N/A N/A N/A 7. Configure BGP process and configure the advertised networks as well as neighborships on all Nexus 9372 A,B,C,D,E,F,G,H leaf switches. R1-CS-9372-A-U39# conf t Enter configuration commands, one per line. End with CNTL/Z. R1-CS-9372-A-U39(config)# router bgp <- AS number will be different for each rack. R1-CS-9372-A-U39(config-router)# router-id <- primary loopback 0 ip address R1-CS-9372-A-U39(config-router)# address-family ipv4 unicast R1-CS-9372-A-U39(config-router-af)# network /32 <- add primary loopback 0 ip address R1-CS-9372-A-U39(config-router-af)# network /31 <- add network statements for L3 interfaces created beforehand R1-CS-9372-A-U39(config-router-af)# network /31... R1-CS-9372-A-U39(config-router-af)# maximum-paths 32 R1-CS-9372-A-U39(config-router-af)# template peer BGPSPINE <- template to save time on neighborships configuration R1-CS-9372-A-U39(config-router-neighbor)# address-family ipv4 unicast R1-CS-9372-A-U39(config-router-neighbor)# next-hop-self R1-CS-9372-A-U39(config-router-neighbor)# soft-reconfiguration inbound R1-CS-9372-A-U39(config-router-neighbor)# neighbor remote-as <- ibgp neighborship between vpc peers on leaf R1-CS-9372-A-U39(config-router-neighbor)# update-source Vlan10 R1-CS-9372-A-U39(config-router-neighbor)# address-family ipv4 unicast R1-CS-9372-A-U39(config-router-neighbor)# next-hop-self R1-CS-9372-A-U39(config-router-neighbor)# soft-reconfiguration inbound R1-CS-9372-A-U39(config-router-neighbor)# neighbor remoteas <- create neighborships on all L3 links between spine and leaf layer R1-CS-9372-A-U39(config-router-neighbor)# inherit peer BGPSPINE R1-CS-9372-A-U39(config-router-neighbor)# neighbor remoteas R1-CS-9372-A-U39(config-router-neighbor)# inherit peer BGPSPINE Chapter 3: UCP 4000 Cisco Systems 53

62 Internal Network Configurations (Layer 3 mode) R1-CS-9372-A-U39(config-router-neighbor)# neighbor remoteas R1-CS-9372-A-U39(config-router-neighbor)# inherit peer BGPSPINE R1-CS-9372-A-U39(config-router-neighbor)# neighbor remoteas R1-CS-9372-A-U39(config-router-neighbor)# inherit peer BGPSPINE Nexus 9372 A BGP Routing Switch Network Statement Neighbor Remote-AS Neighbor 9372A-U / A-U A-U / A-U A-U / B- 9372A-U / B- 9372A-U / B-U A-U /32 N/A N/A N/A Nexus 9372 B BGP Routing Switch Network Statement Neighbor Remote-AS Neighbor 9372B-U / A-U B-U / A-U B-U / B- 9372B-U / B- 9372B-U / A-U B-U /32 N/A N/A N/A Nexus 9372 C BGP Routing Switch Network Statement Neighbor Remote-AS Neighbor 9372C / A-U C / A-U C / B- 9372C / B- 54 Chapter 3: UCP 4000 Cisco Systems

63 Internal Network Configurations (Layer 3 mode) Switch Network Statement Neighbor Remote-AS Neighbor 9372C / B-U C /32 N/A N/A N/A Nexus 9372 D BGP Routing Switch Network Statement Neighbor Remote-AS Neighbor 9372D-U / A-U D-U / A-U D-U / B- 9372D-U / B- 9372D-U / A-U D-U /32 N/A N/A N/A Nexus 9372 E BGP Routing Switch Network Statement Neighbor Remote-AS Neighbor 9372E / A-U E / A-U E / B- 9372E / B- 9372E / F-U E /32 N/A N/A N/A Nexus 9372 F BGP Routing Switch Network Statement Neighbor Remote-AS Neighbor 9372F-U / A-U F-U / A-U F-U / B- 9372F-U / B- 9372F-U / E- 9372F-U /32 N/A N/A N/A Chapter 3: UCP 4000 Cisco Systems 55

64 Internal Network Configurations (Layer 3 mode) Nexus 9372 G BGP Routing Switch Network Statement Neighbor Remote-AS Neighbor 9372G / A-U G / A-U G / B- 9372G / B- 9372G / H-U G /32 N/A N/A N/A Nexus 9372 H BGP Routing Switch Network Statement Neighbor Remote-AS Neighbor 9372H-U / A-U H-U / A-U H-U / B- 9372H-U / B- 9372H-U / G- 9372H-U /32 N/A N/A N/A Configure the Layer 3 interfaces Note: Please run this section for both OSPF and BGP. 8. Add Layer 3 configurations to the interfaces on all 9332 and 9372 switches according to tables below. R1-CS-9332-A-U42(config)# interface 1/1 R1-CS-9332-A-U42(config-if)# speed R1-CS-9332-A-U42(config-if)# mtu 9216 R1-CS-9332-A-U42(config-if)# ip address /31 <- Refer IP address of each interface in below tables R1-CS-9332-A-U42(config-if)# ip ospf network point-to-point <-- Add this if OSPF is used R1-CS-9332-A-U42(config-if)# ip router ospf 100 area <-- Add this if OSPF is used R1-CS-9332-A-U42(config-if)# no shutdown 56 Chapter 3: UCP 4000 Cisco Systems

65 Internal Network Configurations (Layer 3 mode) Cisco Nexus9332-A IP Scheme Link # Switch Interface IP Neighbor A-U41 1/ / A-U A-U41 1/ / A-U A-U41 1/ / B-U A-U41 1/ / B-U A-U41 1/ / C A-U41 1/ / C A-U41 1/ / D-U A-U41 1/ / D-U A-U41 1/ / E A-U41 1/ / E A-U41 1/ / F-U A-U41 1/ / F-U A-U41 1/ / G A-U41 1/ / G A-U41 1/ / H-U A-U41 1/ / H-U39 Cisco Nexus9332-B IP Scheme Link # Switch Interface IP Neighbor B- 1/ / A-U B- 1/ / A-U B- 1/ / B-U B- 1/ / B-U B- 1/ / C B- 1/ / C B- 1/ / D-U39 Chapter 3: UCP 4000 Cisco Systems 57

66 Internal Network Configurations (Layer 3 mode) Link # Switch Interface IP Neighbor B- 1/ / D-U B- 1/ / E B- 1/ / E B- 1/ / F-U B- 1/ / F-U B- 1/ / G B- 1/ / G B- 1/ / H-U B- 1/ / H-U39 Cisco Nexus9372-A IP Scheme Link # Switch Interface IP Neighbor A-U39 1/ / A-U A-U39 1/ / A-U A-U39 1/ / B A-U39 1/ / B- Cisco Nexus9372-B IP Scheme Link # Switch Interface IP Neighbor B-U38 1/ / A-U B-U38 1/ / A-U B-U38 1/ / B B-U38 1/ / B- Cisco Nexus9372-C IP Scheme Link # Switch Interface IP Neighbor C- 1/ / A-U C- 1/ / A-U41 58 Chapter 3: UCP 4000 Cisco Systems

67 Internal Network Configurations (Layer 3 mode) Link # Switch Interface IP Neighbor C- 1/ / B C- 1/ / B- Cisco Nexus9372-D IP Scheme Link # Switch Interface IP Neighbor D-U39 1/ / A-U D-U39 1/ / A-U D-U39 1/ / B D-U39 1/ / B- Cisco Nexus9372-E IP Scheme Link # Switch Interface IP Neighbor E- 1/ / A-U E- 1/ / A-U E- 1/ / B E- 1/ / B- Cisco Nexus9372-F IP Scheme Link # Switch Interface IP Neighbor F-U39 1/ / A-U F-U39 1/ / A-U F-U39 1/ / B F-U39 1/ / B- Cisco Nexus9372-G IP Scheme Link # Switch Interface IP Neighbor G- 1/ / A-U G- 1/ / A-U G- 1/ / B- Chapter 3: UCP 4000 Cisco Systems 59

68 Internal Network Configurations (Layer 3 mode) Link # Switch Interface IP Neighbor G- 1/ / B- Cisco Nexus9372-H IP Scheme Link # Switch Interface IP Neighbor H-U39 1/ / A-U H-U39 1/ / A-U H-U39 1/ / B H-U39 1/ / B- Configure L3 Peer Adjacency 9. Add vlan, ip and ospf on the interface vlan for the L3 adjacency between vpc peers according to below table on 9372A,B,C,D,E,F,G,H switches. R1-CS-9372-A-U39(config)# Vlan 10 <- VLAN for L3 peer adjacency R1-CS-9372-A-U39(config-vlan)# interface Vlan10 <- ospf/ibgp interface between vpc peers R1-CS-9372-A-U39(config-if)# no shutdown R1-CS-9372-A-U39(config-if)# ip address /31 <- L3 Peer Adjacency SVI IP Address R1-CS-9372-A-U39(config-if)# mtu 9216 R1-CS-9372-A-U39(config-if)# ip router ospf 100 area <-- Add this if OSPF is used L3 adjacency between vpc peers Rack Location Switch Int Vlan 10 Compute Rack #1 9372A-U /31 Compute Rack #1 9372B-U /31 Compute Rack #2 9372C /31 Compute Rack #2 9372D-U /31 Compute Rack #3 9372E /31 Compute Rack #3 9372F-U /31 Compute Rack #4 9372G /31 Compute Rack #4 9372H-U /31 60 Chapter 3: UCP 4000 Cisco Systems

69 Internal Network Configurations (Layer 3 mode) Configure Loopback Configure the loopback 0 interface so the switch loopback interface is reachable. Repeat for 9332A,B,9372A,B,C,D,E,F,G,H. R1-CS-9332-A-U42(config-if)# interface loopback 0 R1-CS-9332-A-U42(config-if)# ip address /32 <- Loopback 0 address below. R1-CS-9332-A-U42(config-if)# ip router ospf 100 area <-- Add this if OSPF is used Loopback 0 Interface IP Address Rack Location Switch Loopback 0 Primary Compute Rack #1 9332A-U /32 Compute Rack #1 9332B /32 Compute Rack #1 9372A-U /32 Compute Rack #1 9372B-U /32 Compute Rack #2 9372C /32 Compute Rack #2 9372D-U /32 Compute Rack #3 9372E /32 Compute Rack #3 9372F-U /32 Compute Rack #4 9372G /32 Compute Rack #4 9372H-U /32 Verify Neighborships Verify OSPF neighborships 11. Verify that the L3 ospf neighborships are in FULL state on all L3 interfaces that are connected to neighboring switches. R1-CS-9332-A-U42# sh ip ospf neighbors vrf ucpmanagement OSPF Process ID UNDERLAY VRF default Total number of neighbors: 8 Neighbor ID Pri State Up Time Address Interface FULL/ - 1d05h Eth1/ FULL/ - 1d05h Eth1/ FULL/ - 2d04h Eth1/ FULL/ - 2d04h Eth1/ FULL/ - 2d04h Eth1/ FULL/ - 2d04h Eth1/6 Chapter 3: UCP 4000 Cisco Systems 61

70 Internal Network Configurations (Layer 3 mode) FULL/ - 2d04h Eth1/ FULL/ - 2d04h Eth1/ FULL/ - 1d05h Eth1/ FULL/ - 1d05h Eth1/ FULL/ - 2d04h Eth1/ FULL/ - 2d04h Eth1/ FULL/ - 2d04h Eth1/ FULL/ - 2d04h Eth1/ FULL/ - 2d04h Eth1/ FULL/ - 2d04h Eth1/16 Verify BGP neighborships 12. Verify that the L3 BGP neighborships are in FULL state on all L3 interfaces that are connected to neighboring switches. R1-CS-9332-A-U42# sh ip bgp summary BGP summary information for VRF default, address family IPv4 Unicast BGP router identifier , local AS number BGP table version is 7, IPv4 Unicast config peers 5, capable peers 4 0 network entries and 0 paths using 0 bytes of memory BGP attribute entries [0/0], BGP AS path entries [0/0] BGP community entries [0/0], BGP clusterlist entries [0/0] Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd d05h d05h d05h d05h d05h d05h d05h d05h d05h d05h d05h d05h d05h d05h d05h d05h 10 Configure VXLAN VXLAN is an overlay technology that encapsulates packets with a VXLAN network identifier (VNI) to connect disjoint layer 2 networks within a layer 3 connected IP network. Traditionally layer 2 networks had to be connected 62 Chapter 3: UCP 4000 Cisco Systems

71 Internal Network Configurations (Layer 3 mode) directly through a series of layer 2 devices. However, the need to connect disjoint layer 2 networks for reasons such as vmotion for high availability or flexibility to add hosts anywhere in the data center; VXLAN allows us the freedom to tunnel traffic over a traditional Layer 3 network and scale with multi-tenancy in mind. Important: Please decide now whether the customer will use VXLAN Flood and Learn or VXLAN/EVPN and skip to section accordingly. Only one VXLAN variant can be configured at a time. VXLAN Flood and Learn Traditional VXLAN uses any layer 3 routing protocol as the underlay and multicast for broadcast, unknown unicast, multicast, and arp flood and learn traffic. VXLAN flood and learn does not support the idea of traffic segregation for mult-tenancy support. ebgp or OSPF underlay is used as the routing protocol of the underlying network infrastructure. This is the IP Network that edge devices will use to communicate. PIM-SM is the multicast routing protocol used for multi-destination traffic such as broadcast, unknown unicast, multicast, and flood and learn for endhost detection. Anycast-RP is used for rendezvous-point redundancy. The Anycast-RP s are configured on the spine switches with the same ip. Important: This VXLAN flood and learn configuration guide assumes the Unicast Routing has been already configured. If not, go back to "Configure Global Unicast Routing" on page 47 section and complete either the OSPF or ebgp configuration. Enable Required Features 1. Enable pim, the multicast routing protocol. This needs to be configured on all Spine switches 9332A and 9332B. R1-CS-9332-A-U42# conf t Enter configuration commands, one per line. End with CNTL/Z. R1-CS-9332-A-U42(config)# feature pim Chapter 3: UCP 4000 Cisco Systems 63

72 Internal Network Configurations (Layer 3 mode) 2. Enable pim, nv overlay, and vn-segment-vlan-based on all Leaf switches 9372A,B,C,D,E,F,G,H. R1-CS-9372-A-# conf t Enter configuration commands, one per line. End with CNTL/Z. R1-CS-9372-A-(config)# feature pim R1-CS-9372-A-(config)# feature nv overlay R1-CS-9372-A-(config)# feature vn-segment-vlan-based Configure tcam-size 3. Change tcam-size for region arp-ether on all Leaf switches 9372A,B,C,D,E,F,G,H. Note: To allocate tcam space for region arp-ether, we need to reallocate tcam space. In this example, we reduce the tcam space from the vacl region, and re-allocate to the arp-ether region. Please select an appropriate region to re-allocate for arp-ether. R1-CS-9372-A-(config)# hardware access-list tcam region vacl 0 Warning: Please save config and reload the system for the configuration to take effect R1-CS-9372-A-(config)# hardware access-list tcam region arp-ether 256 Warning: Please save config and reload the system for the configuration to take effect R1-CS-9372-A-(config)# copy running-config startup-config [########################################] 100% Copy complete. 4. Reload (reboot) all Leaf switches 9372A,B,C,D,E,F,G,H. R1-CS-9332-B-U41# reload!!!warning! there is unsaved configuration!!! This command will reboot the system. (y/n)? [n] y Configure Loopback 0 Secondary Address 5. The secondary loopback 0 address is used by VXLAN for peering between VTEPs when vpc is configured. Configure secondary IP address on loopback 0 of all Nexus 9372 leaf switches. R1-CS-9372-A-# conf t Enter configuration commands, one per line. End with CNTL/Z. 64 Chapter 3: UCP 4000 Cisco Systems

73 Internal Network Configurations (Layer 3 mode) R1-CS-9372-A-(config)# interface loopback0 R1-CS-9372-A-(config-if)# ip address /32 secondary <- this address should be the same on both vpc peers follow ip scheme. Important: If ebgp was configured as the routing protocol make sure to advertise the loopback0 secondary in a network statement. If ospf was configured the ip router ospf command should already be present. Rack Location Switch Loopback 0 Secondary Compute Rack #1 9332A-U41 N/A Compute Rack #1 9332B- N/A Compute Rack #1 9372A-U /32 Compute Rack #1 9372B-U /32 Compute Rack #2 9372C /32 Compute Rack #2 9372D-U /32 Compute Rack #3 9372E /32 Compute Rack #3 9372F-U /32 Compute Rack #4 9372G /32 Compute Rack #4 9372H-U /32 Configure PIM on L3 Interfaces Configure on all layer 3 interfaces on the Nexus 9332 and Nexus 9372 switches. PIM must be enabled on all the Spine or Leaf facing interfaces and loopback Login to the Nexus 9332 A and B switches, and run following commands. R1-CS-9332-A-U42# conf t Enter configuration commands, one per line. End with CNTL/Z. R1-CS-9332-A-U42(config)#interface loopback0 R1-CS-9332-A-U42(config-if)# ip pim sparse-mode <- Enable multicast on the interface R1-CS-9332-A-U42(config-if)#interface 1/1-16 R1-CS-9332-A-U42(config-if-range)# ip pim sparse-mode <- Enable multicast on the interface Chapter 3: UCP 4000 Cisco Systems 65

74 Internal Network Configurations (Layer 3 mode) 7. Login to the Nexus 9372 A, B, C, D, E, F, G, H switches, and run following commands. R1-CS-9372-A-# conf t Enter configuration commands, one per line. End with CNTL/Z. R1-CS-9372-A-(config)#interface loopback0 R1-CS-9372-A-(config-if)# ip pim sparse-mode <- Enable multicast on the interface R1-CS-9372-A-(config-if)#interface 1/49-52 R1-CS-9372-A-(config-if)# ip pim sparse-mode <- Enable multicast on the interface Configure Anycast-RP (Nexus 9332PQ Spine Switch Only) 8. Interface Loopback 254 is configured with the RP Address. PIM must be enabled for multicast to work. Configure exactly the same on 9332A and 9332B. R1-CS-9332-A-U42# conf t Enter configuration commands, one per line. End with CNTL/Z. R1-CS-9332-A-U42(config)#interface loopback254 R1-CS-9332-A-U42(config-if)# ip address /32 R1-CS-9332-A-U42(config-if)# ip pim sparse-mode <- Enable multicast on the interface R1-CS-9332-A-U42(config-if)# ip router ospf 100 area <- Only configure if ospf is the global routing protocol, if ebgp ignore. Here we specify the RP address and multicast group list associated with it. In addition, anycast rp is configured for RP redundancy on both Spines. 9. Configure exactly the same on 9332A and 9332B. R1-CS-9332-A-U42# conf t Enter configuration commands, one per line. End with CNTL/Z. R1-CS-9332-A-U42(config)#ip pim rp-address group-list /24 < is the anycast RP. R1-CS-9332-A-U42(config)#ip pim anycast-rp < is the local RP address. R1-CS-9332-A-U42(config)#ip pim anycast-rp < is the peer RP address. 66 Chapter 3: UCP 4000 Cisco Systems

75 Internal Network Configurations (Layer 3 mode) Rack Location Switch Anycast RP Local RP Peer RP Compute Rack #1 9332A-U Compute Rack #1 9332B Configure RP Address (Nexus 9372PX Leaf Switch Only) 10. Configure the RP address and group-list associated with the RP on Nexus 9372 A,B,C,D,E,F,G,H switches. R1-CS-9372-A-# conf t Enter configuration commands, one per line. End with CNTL/Z. R1-CS-9372-A-(config)# ip pim rp-address group-list /24 Configure VXLAN VTEP (Nexus9372PX Leaf Switch Only) 11. Configure the nve1 interface on all Leafs. R1-CS-9372-A-# conf t Enter configuration commands, one per line. End with CNTL/Z R1-CS-9372-A-(config)# interface nve1 R1-CS-9372-A-(config-if)# source-interface loopback0 R1-CS-9372-A-(config-if)# no shutdown Configure VLAN ID and VXLAN ID mapping 12. To configure an VLAN to be part of VXLAN VTEP, use following commands on all Leaf VXLAN switches that are to be part of VXLAN. Repeat following steps for all necessary VXLAN-VLAN mappings. In this example, we map vlan 50 (L2 Migration VLAN) to vni 10000, and assign vni to multicast group Note: For L2 VM Migration configuration, must use same migration VLAN ID on all Nexus 9372 leaf switches. R1-CS-9372-A-(config)#Vlan 50 R1-CS-9372-A-(config-vlan)# name migration_vlan R1-CS-9372-A-(config-vlan)# vn-segment <- make vlan 50 part of vni R1-CS-9372-A-(config-vlan)# interface nve1 R1-CS-9372-A-(config-if-nve)# member vni R1-CS-9372-A-(config-if-nve-vni)# mcast-group <- assign vni (vlan50) to multicast group Chapter 3: UCP 4000 Cisco Systems 67

76 Internal Network Configurations (Layer 3 mode) Note: Ideally, one VXLAN segment mapping to one IP multicast group is the way to provide the optimal multicast forwarding. However, it is possible to have multiple VXLAN segments share a single IP multicast group to achive the desired VXLAN scalability. Having multiple-tenant VXLAN networks to share a multicast group does not bring any implications to the Layer 2 isolation between the tenant networks. Verify VXLAN Configuration 13. Displays the nve peer status R1-CS-9372-A-# show nve peers Interface Peer-IP State LearnType Uptime Router-Mac nve Up CP 00:01:19 84b8.02ca Displays the vni and relevant info R1-CS-9372-A-# sh nve vni Codes: CP - Control Plane DP - Data Plane UC - Unconfigured SA - Suppress ARP Interface VNI Multicast-group State Mode Type [BD/VRF] Flags nve Up CP L2 [50] SA nve Up CP L2 [60] SA 15. Verify PIM neighbors are formed R1-CS-9332-A-U42# sh ip pim neighbor PIM Neighbor Status for VRF "default" Neighbor Interface Uptime Expires DR Bidir- BFD Priority Capable State /1 00:02:54 00:01:37 1 no n/a /2 00:00:19 00:01:25 1 no n/a /3 00:00:03 00:01:41 1 no n/a /4 00:00:03 00:01:41 1 no n/a /5 00:00:02 00:01:42 1 no n/a /6 00:00:02 00:01:42 1 no n/a /7 00:00:02 00:01:42 1 no n/a /8 00:00:02 00:01:42 1 no n/a R1-CS-9372-A-# sh ip pim neighbor 68 Chapter 3: UCP 4000 Cisco Systems

77 Internal Network Configurations (Layer 3 mode) PIM Neighbor Status for VRF "default" Neighbor Interface Uptime Expires DR Bidir- BFD Priority Capable State /49 00:03:25 00:01:39 1 no n/a /50 00:00:50 00:01:23 1 no n/a /51 00:00:43 00:01:31 1 no n/a /52 00:00:43 00:01:44 1 no n/a Note: If you can ping between vms but cannot ssh or send traffic, make sure that jumbo frames are configured along the entire traffic path. This includes the vswitch or vsphere distributed switch is using mtu In addition, make sure the mtu is set to 9000 on the vm itself such as ifcfgeth0 configuration if the vm is a linux os. Note: In VxLAN flood and learn mode (7.0(3)I1(2) and earlier), the default gateway for VXLAN VLANs should be provisioned on external routing devices. In VXLAN flood and learn mode (7.0(3)I2(1) and later), the default gateway for VXLAN VLAN is recommended to be a centralized gateway on a pair of VPC devices with FHRP (First Hop Redundancy Protocol) running between them. In BGP EVPN, it is recommended to use the anycast gateway feature on all VTEPs. For more information on configuring the default gateway on external routing devices please reference this Cisco whitepaper series-switches/white-paper-c html VXLAN / EVPN VXLAN over EVPN uses any layer 3 routing protocol as the underlay and multicast for Broadcast, unknown unicast, and multicast. Arp flood and learn is avoided and localized with ARP supression. VXLAN EVPN introduces the idea of traffic segregation for mult-tenancy support. Important: This VXLAN over EVPN configuration guide assumes that OSPF has been chosen as the unicast routing protocol and has been configured already. This section does not cover ebgp as the unicast routing protocol. Also, this VXLAN over EVPN configuration requires deep networking knowledge. For more information, please refer to the "Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide". 7-x/vxlan/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_VXLAN_ Configuration_Guide_7x.html Chapter 3: UCP 4000 Cisco Systems 69

78 Internal Network Configurations (Layer 3 mode) OSPF Underlay is used as the routing protocol of the underlying network infrastructure. This is the IP Network that edge devices will use to communicate. Although any unicast routing protocol is supported. MP-iBGP L2VPN EVPN is used as the overlay network or virtual network built over the underlying network infrastructure. This allows traffic segregation for multi-tenancy support. PIM-SM is the multicast routing protocol used for multi-destination traffic such as broadcast, unknown unicast, and multicast. Anycast-RP is used for rendezvous-point redundancy. The Anycast-rp s are configured on the spine switches with the same ip. Anycast gateway is used to have the same gateway and mac address on all leaf layer switches for a locally defined subnet. This is useful when there are multiple VXLAN VTEPs to facilitate a centralized gateway as opposed to having separate gateways. Enable Required Features 16. Run following commands to enable required feature on all Nexus 9332 A and B Spine switches. R1-CS-9332-A-U42(config)# nv overlay evpn R1-CS-9332-A-U42(config)# feature ospf R1-CS-9332-A-U42(config)# feature bgp R1-CS-9332-A-U42(config)# feature pim 17. Run following commands to enable required feature on all Nexus 9372 A,B,C,D,E,F,G,H Leaf switches. R1-CS-9372-A-(config)#nv overlay evpn R1-CS-9372-A-(config)#feature ospf R1-CS-9372-A-(config)#feature bgp R1-CS-9372-A-(config)#feature pim R1-CS-9372-A-(config)#feature vn-segment-vlan-based R1-CS-9372-A-(config)#feature nv overlay Configure Loopback 0 Secondary Address 18. The secondary loopback 0 address is used by VXLAN for peering between VTEPs when vpc is configured. Configure secondary IP address on loopback 0 of all Nexus 9372 leaf switches. 70 Chapter 3: UCP 4000 Cisco Systems

79 Internal Network Configurations (Layer 3 mode) R1-CS-9372-A-# conf t Enter configuration commands, one per line. End with CNTL/Z. R1-CS-9372-A-(config)# interface loopback0 R1-CS-9372-A-(config-if)# ip address /32 secondary <- this address should be the same on both vpc peers follow ip scheme. Important: If ospf was configured the ip router ospf command should already be present. Rack Location Switch Loopback 0 Secondary Compute Rack #1 9332A-U41 N/A Compute Rack #1 9332B- N/A Compute Rack #1 9372A-U /32 Compute Rack #1 9372B-U /32 Compute Rack #2 9372C /32 Compute Rack #2 9372D-U /32 Compute Rack #3 9372E /32 Compute Rack #3 9372F-U /32 Compute Rack #4 9372G /32 Compute Rack #4 9372H-U /32 Configure PIM on L3 Interfaces Configure on all layer 3 interfaces on the Nexus 9332 and Nexus 9372 switches. PIM must be enabled on all the Spine or Leaf facing interfaces and loopback Login to the Nexus 9332 A and B switches, and run following commands. R1-CS-9332-A-U42# conf t Enter configuration commands, one per line. End with CNTL/Z. R1-CS-9332-A-U42(config)#interface loopback0 R1-CS-9332-A-U42(config-if)# ip pim sparse-mode <- Enable multicast on the interface R1-CS-9332-A-U42(config-if)#interface 1/1-16 R1-CS-9332-A-U42(config-if-range)# ip pim sparse-mode <- Enable multicast on the interface Chapter 3: UCP 4000 Cisco Systems 71

80 Internal Network Configurations (Layer 3 mode) 20. Login to the Nexus 9372 A, B, C, D, E, F, G, H switches, and run following commands. R1-CS-9372-A-# conf t Enter configuration commands, one per line. End with CNTL/Z. R1-CS-9372-A-(config)#interface loopback0 R1-CS-9372-A-(config-if)# ip pim sparse-mode <- Enable multicast on the interface R1-CS-9372-A-(config-if)#interface 1/49-52 R1-CS-9372-A-(config-if)# ip pim sparse-mode <- Enable multicast on the interface Configure Anycast-RP (Nexus 9332PQ Spine Switch Only) 21. Interface Loopback 254 is configured with the RP Address. PIM must be enabled for multicast to work. Configure exactly the same on 9332A and 9332B. R1-CS-9332-A-U42# conf t Enter configuration commands, one per line. End with CNTL/Z. R1-CS-9332-A-U42(config)#interface loopback254 R1-CS-9332-A-U42(config-if)# ip address /32 R1-CS-9332-A-U42(config-if)# ip pim sparse-mode <- Enable multicast on the interface R1-CS-9332-A-U42(config-if)# ip router ospf 100 area <- Only configure if ospf is the global routing protocol. 22. Here we specify the RP address and multicast group list associated with it. In addition, anycast rp is configured for RP redundancy on both Spines. Configure exactly the same on 9332A and 9332B. R1-CS-9332-A-U42# conf t Enter configuration commands, one per line. End with CNTL/Z. R1-CS-9332-A-U42(config)#ip pim rp-address group-list /24 < is the anycast RP. R1-CS-9332-A-U42(config)#ip pim anycast-rp < is the local RP address. R1-CS-9332-A-U42(config)#ip pim anycast-rp < is the peer RP address. 72 Chapter 3: UCP 4000 Cisco Systems

81 Internal Network Configurations (Layer 3 mode) Rack Location Switch Anycast RP Local RP Peer RP Compute Rack #1 9332A-U Compute Rack #1 9332B Configure RP Address (Nexus 9372PX Leaf Switch Only) 23. Configure the RP address and group-list associated with the RP on Nexus 9372 A,B,C,D,E,F,G,H switches. R1-CS-9372-A-# conf t Enter configuration commands, one per line. End with CNTL/Z. R1-CS-9372-A-(config)# ip pim rp-address group-list /24 Configure BGP neighborships 24. Configure the following BGP configuration on the spine switches 9332A and 9332B R1-CS-9332-A-U42(config)# router bgp R1-CS-9332-A-U42(config-router)# router-id <- Use appropriate router-id according to table below R1-CS-9332-A-U42(config-router)# address-family ipv4 unicast R1-CS-9332-A-U42(config-router-af)# neighbor remote-as <- repeat from here to "send-community both" for all 9372 leaf switches according to table below R1-CS-9332-A-U42(config-router-neighbor)# update-source loopback 0 R1-CS-9332-A-U42(config-router-neighbor)# address-family ipv4 unicast R1-CS-9332-A-U42(config-router-neighbor-af)# address-family l2vpn evpn R1-CS-9332-A-U42(config-router-neighbor-af)# send-community both R1-CS-9332-A-U42(config-router-neighbor-af)# route-reflector-client Rack Location Switch Router-id Compute Rack #1 9332A-U Compute Rack #1 9332B-U Chapter 3: UCP 4000 Cisco Systems 73

82 Internal Network Configurations (Layer 3 mode) Rack Location Switch Remote-AS # Neighbor Compute Rack #1 9372A Compute Rack #1 9372B-U Compute Rack #2 9372C Compute Rack #2 9372D-U Compute Rack #3 9372E Compute Rack #3 9372F-U Compute Rack #4 9372G Compute Rack #4 9372H-U Configure the following BGP configuration on the Nexus 9372 A,B,C,D,E,F,G,H leaf switches R1-CS-9372-A-(config)#router bgp R1-CS-9372-A-(config-router)# router-id <- modify according to table below. R1-CS-9372-A-(config-router)# address-family ipv4 unicast R1-CS-9372-A-(config-router-af)# neighbor remote-as <- modify according to Router ID of Nexus 9332A Spine switch. R1-CS-9372-A-(config-router-neighbor)# update-source loopback0 R1-CS-9372-A-(config-router-neighbor)# address-family ipv4 unicast R1-CS-9372-A-(config-router-neighbor)# address-family l2vpn evpn R1-CS-9372-A-(config-router-neighbor)# send-community both R1-CS-9372-A-(config-router-neighbor)# neighbor remoteas 65001<- modify according to Router ID of Nexus 9332B Spine switch. R1-CS-9372-A-(config-router-neighbor)# update-source loopback0 R1-CS-9372-A-(config-router-neighbor)# address-family ipv4 unicast R1-CS-9372-A-(config-router-neighbor)# address-family l2vpn evpn R1-CS-9372-A-(config-router-neighbor)# send-community both R1-CS-9372-A-(config-router-neighbor)# vrf TENANT1 R1-CS-9372-A-(config-router-vrf)# address-family ipv4 unicast R1-CS-9372-A-(config-router-vrf)# advertise l2vpn evpn Rack Location Switch Router-id Compute Rack #1 9372A Compute Rack #1 9372B-U Chapter 3: UCP 4000 Cisco Systems

83 Internal Network Configurations (Layer 3 mode) Rack Location Switch Router-id Compute Rack #2 9372C Compute Rack #2 9372D-U Compute Rack #3 9372E Compute Rack #3 9372F-U Compute Rack #4 9372G Compute Rack #4 9372H-U Configure Anycast Gateway MAC (Nexus9372PX Leaf Switch Only) 26. Configure anycast gateway mac address on Nexus 9372 A,B,C,D,E,F,G,H leaf switches. R1-CS-9372-A-(config)# fabric forwarding anycast-gateway-mac 2020.DEAD.BEEF Configure VXLAN VTEP (Nexus9372PX Leaf Switch Only) Configure the nve1 interface on all Nexus 9372 Leaf switches. Source interface loopback 0 will pick up the secondary ip which is the same as the secondary ip on the vpc peer. VNI s are added with arp suppression and associated to a multicast group. 27. Create the nve interface and make vni 30099, 30100, as a member on all Leaf VTEP switches 9372A,B,C,D. R1-CS-9372-A-(config)# interface nve1 R1-CS-9372-A-(config-if-nve)# no shutdown R1-CS-9372-A-(config-if-nve)# source-interface loopback 0 R1-CS-9372-A-(config-if-nve)# host-reachability protocol bgp R1-CS-9372-A-(config-if-nve)# member vni R1-CS-9372-A-(config-if-nve-vni)# mcast-group R1-CS-9372-A-(config-if-nve-vni)# member vni R1-CS-9372-A-(config-if-nve-vni)# suppress-arp R1-CS-9372-A-(config-if-nve-vni)# mcast-group R1-CS-9372-A-(config-if-nve-vni)# member vni associate-vrf 28. Create Vlans and associate to VN-Segment Vlan 2500 is used for assigning to VRF for multi-tenancy purposes, Vlan 99 is a L2 only segment while Vlan 100 is L2/L3 segment. Configure on all switches 9372A,B,C,D. Chapter 3: UCP 4000 Cisco Systems 75

84 Internal Network Configurations (Layer 3 mode) R1-CS-9372-A-(config)# vlan 99 R1-CS-9372-A-(config-vlan)# name L2onlyHostSegment R1-CS-9372-A-(config-vlan)# vn-segment R1-CS-9372-A-(config-vlan)# vlan 100 Warning: Disabling IGMP snooping for VLAN 99. R1-CS-9372-A-(config-vlan)# name L2L3HostSegment R1-CS-9372-A-(config-vlan)# vn-segment R1-CS-9372-A-(config-vlan)# vlan 2500 Warning: Disabling IGMP snooping for VLAN 100. R1-CS-9372-A-(config-vlan)# name FabricBD R1-CS-9372-A-(config-vlan)# vn-segment Configure on all switches 9372A,B,C,D. VRF TENANT1 is assigned vni segment which is same as vlan 2500 s configured above. R1-CS-9372-A-(config)# vrf context TENANT1 R1-CS-9372-A-(config-vrf)# vni R1-CS-9372-A-(config-vrf)# rd auto R1-CS-9372-A-(config-vrf)# address-family ipv4 unicast R1-CS-9372-A-(config-vrf-af-ipv4)# route-target both auto R1-CS-9372-A-(config-vrf-af-ipv4)# route-target both auto evpn R1-CS-9372-A-(config-vrf-af-ipv4)# address-family ipv6 unicast R1-CS-9372-A-(config-vrf-af-ipv6)# route-target both auto R1-CS-9372-A-(config-vrf-af-ipv6)# route-target both auto evpn 30. Configure on all switches 9372A,B,C,D. Vlan 100 has svi created in anycast gateway mode. Vlan 99 does not have one because it is L2 mode only. R1-CS-9372-A-(config)# interface vlan 100 R1-CS-9372-A-(config-if)# no shutdown R1-CS-9372-A-(config-if)# mtu 9216 R1-CS-9372-A-(config-if)# vrf member TENANT1 Warning: Deleted all L3 config on interface Vlan100 R1-CS-9372-A-(config-if)# ip address /24 R1-CS-9372-A-(config-if)# fabric forwarding mode anycast-gateway 31. Configure on all switches 9372A,B,C,D. Vlan 2500 is created as well. R1-CS-9372-A-(config)# interface vlan 2500 R1-CS-9372-A-(config-if)# description FabricBD R1-CS-9372-A-(config-if)# no shutdown R1-CS-9372-A-(config-if)# mtu Chapter 3: UCP 4000 Cisco Systems

85 Internal Network Configurations (Layer 3 mode) R1-CS-9372-A-(config-if)# vrf member TENANT1 Warning: Deleted all L3 config on interface Vlan2500 R1-CS-9372-A-(config-if)# ip forward 32. Configure on all switches 9372A,B,C,D. EVPN configurations. R1-CS-9372-A-(config)# evpn R1-CS-9372-A-(config-evpn)# vni l2 R1-CS-9372-A-(config-evpn-evi)# rd auto R1-CS-9372-A-(config-evpn-evi)# route-target import auto R1-CS-9372-A-(config-evpn-evi)# route-target export auto R1-CS-9372-A-(config-evpn-evi)# vni l2 R1-CS-9372-A-(config-evpn-evi)# rd auto R1-CS-9372-A-(config-evpn-evi)# route-target import auto R1-CS-9372-A-(config-evpn-evi)# route-target export auto nve # VNI Mcast Group Vlan N/A 2500 Configure Additional VLAN Configure following configurations on all switches 9372A,B,C,D exactly the same. 33. Add the vni to the nve interface. R1-CS-9372-A-(config)# interface nve 1 R1-CS-9372-A-(config-if-nve)# member vni R1-CS-9372-A-(config-if-nve-vni)# suppress-arp R1-CS-9372-A-(config-if-nve-vni)# mcast-group EVPN configurations. R1-CS-9372-A-(config-if-nve-vni)# evpn R1-CS-9372-A-(config-evpn)# vni l2 R1-CS-9372-A-(config-evpn-evi)# rd auto No VLAN id configured, unable to generate auto RD R1-CS-9372-A-(config-evpn-evi)# route-target import auto R1-CS-9372-A-(config-evpn-evi)# route-target export auto Chapter 3: UCP 4000 Cisco Systems 77

86 Internal Network Configurations (Layer 3 mode) 35. Create the vlan. R1-CS-9372-A-(config-evpn-evi)# vlan 101 R1-CS-9372-A-(config-vlan)# name L2L3HostSegment2 R1-CS-9372-A-(config-vlan)# vn-segment Create the anycast gateway. R1-CS-9372-A-(config-vlan)# interface vlan 101 Warning: Disabling IGMP snooping for VLAN 101. R1-CS-9372-A-(config-if)# no shutdown R1-CS-9372-A-(config-if)# mtu 9216 R1-CS-9372-A-(config-if)# vrf member TENANT1 Warning: Deleted all L3 config on interface Vlan101 R1-CS-9372-A-(config-if)# ip address /24 R1-CS-9372-A-(config-if)# fabric forwarding mode anycast-gateway nve # VNI Mcast Group Vlan Verify VXLAN Configuration 36. Displays the vrf and corresponding vni R1-CS-9372-A-# show nve vrf VRF-Name VNI Interface Gateway-MAC TENANT nve1 78ba.f9ad.87f3 37. Displays the nve peer status R1-CS-9372-A-# show nve peers Interface Peer-IP State LearnType Uptime Router-Mac nve Up CP 00:01:19 84b8.02ca Displays the vni and relevant info 78 Chapter 3: UCP 4000 Cisco Systems

87 Internal Network Configurations (Layer 3 mode) R1-CS-9372-A-# sh nve vni Codes: CP - Control Plane DP - Data Plane UC - Unconfigured SA - Suppress ARP Interface VNI Multicast-group State Mode Type [BD/VRF] Flags nve Up CP L2 [99] nve Up CP L2 [100] SA nve Up CP L2 [101] SA nve n/a Up CP L3 [TENANT1] 39. Displays the vxlan interfaces R1-CS-9372-A-# show vxlan interface Interface Vlan VPL Ifindex LTL HW VP ========= ==== =========== === ===== Po1 99 0x x Po x x Po x x Po x509c3000 0x show bgp l2vpn evpn summary R1-CS-9372-A-# show bgp l2vpn evpn summary BGP summary information for VRF default, address family L2VPN EVPN BGP router identifier , local AS number BGP table version is 139, L2VPN EVPN config peers 2, capable peers 2 19 network entries and 37 paths using 2824 bytes of memory BGP attribute entries [22/3168], BGP AS path entries [0/0] BGP community entries [0/0], BGP clusterlist entries [6/24] Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd w1d w1d show bgp l2vpn evpn R1-CS-9372-A-# show bgp l2vpn evpn BGP routing table information for VRF default, address family L2VPN EVPN BGP table version is 139, local router ID is Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *- valid, >-best Chapter 3: UCP 4000 Cisco Systems 79

88 Internal Network Configurations (Layer 3 mode) Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r- redist, I-injected Origin codes: i - IGP, e - EGP,? - incomplete, - multipath, & - backup Network Next Hop Metric LocPrf Weight Path Route Distinguisher: :32868 (L2VNI 30001) * i[2]:[0]:[0]:[48]:[ ]:[0]:[ ]/ i *>i i *>i[2]:[0]:[0]:[48]:[ ]:[32]:[ ]/ i * i i Route Distinguisher: :32817 * i[2]:[0]:[0]:[48]:[ a0]:[0]:[ ]/ i *>i i * i[2]:[0]:[0]:[48]:[ a0]:[32]:[ ]/ i *>i i Route Distinguisher: :32817 * i[2]:[0]:[0]:[48]:[ d60]:[0]:[ ]/ i *>i i * i[2]:[0]:[0]:[48]:[ d60]:[32]:[ ]/ i *>i i Route Distinguisher: :32868 *>i[2]:[0]:[0]:[48]:[ ]:[0]:[ ]/ i * i i *>i[2]:[0]:[0]:[48]:[ ]:[32]:[ ]/ i * i i Route Distinguisher: :32817 *>i[2]:[0]:[0]:[48]:[ d60]:[0]:[ ]/ i * i i *>i[2]:[0]:[0]:[48]:[ d60]:[32]:[ ]/ i * i i Route Distinguisher: :32868 *>i[2]:[0]:[0]:[48]:[ ]:[0]:[ ]/ i * i i * i[2]:[0]:[0]:[48]:[ ]:[32]:[ ]/ i *>i i Route Distinguisher: :3 (L3VNI 50000) *>i[2]:[0]:[0]:[48]:[ d60]:[32]:[ ]/ i * i i 80 Chapter 3: UCP 4000 Cisco Systems

89 Internal Network Configurations (Layer 3 mode) *>i[2]:[0]:[0]:[48]:[ ]:[32]:[ ]/ i * i i *>i[2]:[0]:[0]:[48]:[ a0]:[32]:[ ]/ i 42. show l2route evpn mac all R1-CS-9372-A-# show l2route evpn mac all Topology Mac Address Prod Next Hop (s) d60 BGP a0 Local Eth1/ BGP b8.02ca.9625 VXLAN show l2route evpn mac-ip all R1-CS-9372-A-# show l2route evpn mac-ip all Topology ID Mac Address Prod Host IP Next Hop (s) BGP Verify PIM neighbors are formed 9332A(config-if-range)# sh ip pim neighbor PIM Neighbor Status for VRF "default" Neighbor Interface Uptime Expires DR Bidir- BFD Priority Capable State /1 00:02:54 00:01:37 1 no n/a /2 00:00:19 00:01:25 1 no n/a /3 00:00:03 00:01:41 1 no n/a /4 00:00:03 00:01:41 1 no n/a /5 00:00:02 00:01:42 1 no n/a /6 00:00:02 00:01:42 1 no n/a /7 00:00:02 00:01:42 1 no n/a /8 00:00:02 00:01:42 1 no n/a R1-CS-9372-A-# sh ip pim neighbor PIM Neighbor Status for VRF "default" Neighbor Interface Uptime Expires DR Bidir- BFD Priority Capable State Chapter 3: UCP 4000 Cisco Systems 81

90 Uplink Configurations /49 00:03:25 00:01:39 1 no n/a /50 00:00:50 00:01:23 1 no n/a /51 00:00:43 00:01:31 1 no n/a /52 00:00:43 00:01:44 1 no n/a Note: If you can ping between vms but cannot ssh or send traffic, make sure that jumbo frames are configured along the entire traffic path. This includes the vswitch or vsphere distributed switch is using mtu In addition, make sure the mtu is set to 9000 on the vm itself such as ifcfgeth0 configuration if the vm is a linux os. Save All Switch Configurations 45. Once all configurations are complete, please save all switch configurations on Nexus 9332 A,B, Nexus 9372 A,B,C,D,E,F,G,H switches to prevent configuration loss in the future. R1-CS-9332-A-U42# copy running-config startup-config Uplink Configurations With UCP 4000 with Cisco Networking model, 8 x 10G ports on Nexus 9372PX or 6 x 40G ports on Nexus 9332PQ can be utilized for uplink connectivity. Also, 2 x 1G ports on Nexus 3048 are allocated for uplink connectivity. The following tables and diagrams show default port allocations for uplink connectivity. Switch Uplink Ports Speed Total Uplink Bandwidth Note Nexus 9372PX Port #33-40 (8 ports per switch) 10Gbps 160Gbps Compute Rack #1 only Nexus 9332PQ Port #26-32 (6 ports per switch) 40Gbps 480Gbps Nexus 3048 Port #47-48 (2 ports per switch) 1Gbps 4Gbps Layer 2 mode only Nexus 9372PX Uplink Ports 82 Chapter 3: UCP 4000 Cisco Systems

91 Uplink Configurations Nexus 9332PQ Uplink Ports Nexus 3048 Uplink Ports Also, UCP 4000 with Cisco Networking model supports Layer 2 connectivity and Layer 3 connectivity to the core network. Layer 2 Uplink Configurations Typically the following 4 scenarios can be applied. # Scenario Example of Network Topology 1 Connecting to multi chassis ether channel technology enabled switches. Chapter 3: UCP 4000 Cisco Systems 83

92 Uplink Configurations # Scenario Example of Network Topology 2 Connecting to standard switches (non multi chassis channel technology switch) 3 Connecting to multi chassis ether channel technology enabled switches & Multiple Core Networks 4 Connecting to standard switches (non multi chassis channel technology switch) & Multiple Core Networks The following scenarios show the example of detail configuration. Important: In the Double-sided vpc configuration, the vpc domain identifiers must be different between the upstream switches vpc domain and the UCP vpc domain. During UCP Deployment, HDS support will configure the vpc domain ID using the ID which HDS collects during preengagement. If the domain ID needs to be changed later, please contact HDS support. Note: Uplink configurations are identical between Nexus 9372PX (10G port), Nexus 9332PQ (40G port), and Nexus 3048 (1G port) except port speed setting and port number. 84 Chapter 3: UCP 4000 Cisco Systems

93 Uplink Configurations Scenario 1: Connecting to single core network & multi chassis ether channel technology enabled switches Configure the uplink ports with one vpcs on the Nexus switches. Ensure cross-connectivity of the physical connections to provide redundancy. 1. Login to the Nexus 9372 A and B, or 9332 A and B which be used for uplink connection using SSH. 2. Configure the interface port-channel on both switches interface Port-channel 10 description port-channel to core switch switchport mode trunk switchport trunk allowed vlan <VLANs> spanning-tree port type normal <-- Cisco Recommendation : Do not enable Bridge Assurance (BA) on vpc. speed <-- Configure this if ports are 40G ports vpc Configure the upstream ports for 1 st vpc on both switches interface 1/26 switchport mode trunk switchport trunk allowed vlan <VLANs> speed <-- Configure this if ports are 40G ports channel-group 10 mode active Ensure that the vpc is correctly configured by checking vpc status #show vpc vpc status id Port Status Consistency Reason Active vlans Po10 up success success <Data & Management VLANs> Scenario 2: Connecting to single core network & non-multi chassis ether channel technology enabled switches Configure the uplink ports with two vpcs on the Nexus switches. Ensure cross-connectivity of the physical connections to provide redundancy. Chapter 3: UCP 4000 Cisco Systems 85

94 Uplink Configurations 5. Login to the Nexus 9372 A and B, or 9332 A and B which be used for uplink connection using SSH. 6. Configure the two interface port-channels on both switches interface Port-channel 10 description port-channel to core switch A switchport mode trunk switchport trunk allowed vlan <VLANs> spanning-tree port type normal <-- Cisco Recommendation : Do not enable Bridge Assurance (BA) on vpc. speed <-- Configure this if ports are 40G ports vpc 10 interface Port-channel 11 description port-channel to core switch B switchport mode trunk switchport trunk allowed vlan <VLANs> spanning-tree port type normal speed <-- Configure this if ports are 40G ports vpc Configure the upstream ports for 1 st vpc on both switches interface 1/26 switchport mode trunk switchport trunk allowed vlan <VLANs> speed <-- Configure this if ports are 40G ports channel-group 10 mode active Configure the upstream ports for 2 nd vpc on both switches interface 1/28 switchport mode trunk switchport trunk allowed vlan <Data & Management VLANs> speed <-- Configure this if ports are 40G ports channel-group 11 mode active Ensure that the vpc is correctly configured by checking vpc status #show vpc vpc status id Port Status Consistency Reason Active vlans Chapter 3: UCP 4000 Cisco Systems

95 Uplink Configurations 10 Po10 up success success <Data & Management VLANs> 11 Po11 up success success <Data & Management VLANs> Scenario 3: Connecting to multiple core networks & multi-chassis ether channel technology enabled switches Configure the uplink ports with two vpcs on the Nexus switches. Ensure cross-connectivity of the physical connections to provide redundancy. 10. Login to the Nexus 9372 A and B, or 9332 A and B which be used for uplink connection using SSH. 11. Configure the two interface port-channels on both switches interface Port-channel 10 description port-channel to management core network switchport mode trunk switchport trunk allowed vlan <VLANs for management> spanning-tree port type normal <-- Cisco Recommendation : Do not enable Bridge Assurance (BA) on vpc. speed <-- Configure this if ports are 40G ports vpc 10 interface Port-channel 11 description port-channel to data core network switchport mode trunk switchport trunk allowed vlan <VLANs for data> spanning-tree port type normal speed <-- Configure this if ports are 40G ports vpc Configure the upstream ports for 1 st vpc on both switches interface 1/26 switchport mode trunk switchport trunk allowed vlan <VLANs for management> speed <-- Configure this if ports are 40G ports channel-group 10 mode active Configure the upstream ports for 2 nd vpc on both switches interface 1/28 switchport mode trunk switchport trunk allowed vlan <VLANs for data> speed <-- Configure this if ports are 40G ports Chapter 3: UCP 4000 Cisco Systems 87

96 Uplink Configurations channel-group 11 mode active Ensure that the vpc is correctly configured by checking vpc status #show vpc vpc status id Port Status Consistency Reason Active vlans Po10 up success success <VLANs for Management> 11 Po11 up success success <VLANs for Data> Scenario 4: Connecting to multiple core networks & non-multi chassis ether channel technology enabled switches Configure the uplink ports with four vpcs on the Nexus switches. Ensure cross-connectivity of the physical connections to provide redundancy. 15. Login to the Nexus 9372 A and B, or 9332 A and B which be used for uplink connection using SSH. 16. Configure the four interface port-channels on both switches interface Port-channel 10 description port-channel to management core switch A switchport mode trunk switchport trunk allowed vlan <VLANs for Management> spanning-tree port type normal <-- Cisco Recommendation : Do not enable Bridge Assurance (BA) on vpc. speed <-- Configure this if ports are 40G ports vpc 10 interface Port-channel 11 description port-channel to management core switch B switchport mode trunk switchport trunk allowed vlan <VLANs for Management> spanning-tree port type normal speed <-- Configure this if ports are 40G ports vpc 11 interface Port-channel 12 description port-channel to data core switch A switchport mode trunk switchport trunk allowed vlan <VLANs for Data> spanning-tree port type normal speed <-- Configure this if ports are 40G ports 88 Chapter 3: UCP 4000 Cisco Systems

97 Uplink Configurations vpc 12 interface Port-channel 13 description port-channel to data core switch B switchport mode trunk switchport trunk allowed vlan <VLANs for Data> spanning-tree port type normal speed <-- Configure this if ports are 40G ports vpc Configure the upstream ports for 1 st vpc on both switches interface 1/1 switchport mode trunk switchport trunk allowed vlan <VLANs for Management> channel-group 10 mode active 18. Configure the upstream ports for 2 nd vpc on both switches interface 1/2 switchport mode trunk switchport trunk allowed vlan <VLANs for Management> channel-group 11 mode active 19. Configure the upstream ports for 3 rd vpc on both switches interface 1/3 switchport mode trunk switchport trunk allowed vlan <VLANs for Data> channel-group 12 mode active 20. Configure the upstream ports for 4 th vpc on both switches interface 1/4 switchport mode trunk switchport trunk allowed vlan <VLANs for Data> channel-group 13 mode active 21. Ensure that the vpc is correctly configured by checking vpc status #show vpc vpc status id Port Status Consistency Reason Active vlans Chapter 3: UCP 4000 Cisco Systems 89

98 Uplink Configurations Po10 up success success <Management VLANs> 11 Po11 up success success <Management VLANs> 12 Po12 up success success <Data VLANs> 13 Po13 up success success <Data VLANs> Layer 3 Uplink Configurations Typically the following 3 scenarios can be applied. # Scenario 1 OSPF Peering to Core Network 2 BGP Peering to Core Network 3 Static Routing Important: Uplink connection on Nexus 3048 switch is not covered in Layer 3 mode. If uplink on Nexus 3048 configuration in layer 3 mode is required, please contact HDS support. Note: Uplink configurations are identical between Nexus 9372PX (10G port) and Nexus 9332PQ (40G port) except port speed setting and port number. Scenario 1: OSPF Peering to Core Network Configure the uplink ports with OSPF Peering. 22. Login to the Nexus 9372 A and B, or 9332 A and B which be used for uplink connection using SSH. 23. Configure layer 3 interfaces on Nexus 9372 A and B (or Nexus 9332 A and B) R1-CS-9372-A-(config)# int e 1/33 R1-CS-9372-A-(config-if)# description L3 link to customer network R1-CS-9372-A-(config-if)# ip address /24 <- enter ip and netmask provided by customer R1-CS-9372-A-(config-if)# mtu Chapter 3: UCP 4000 Cisco Systems

99 Uplink Configurations R1-CS-9372-A-(config-if)# ip router ospf 100 area R1-CS-9372-A-(config-if)# no shut 24. Verify that the ospf neighborships are in FULL state on all L3 interfaces that are connected to neighboring switches. R1-CS-9372-A-# sh ip ospf neighbors vrf ucpmanagement OSPF Process ID UNDERLAY VRF default Total number of neighbors: 8 Neighbor ID Pri State Up Time Address Interface FULL/ - 1d05h Eth1/ FULL/ - 1d05h Eth1/2... Scenario 2: BGP Peering to Core Network 25. Login to the Nexus 9372 A and B, or 9332 A and B which be used for uplink connection using SSH. 26. Configure layer 3 interfaces on Nexus 9372 A and B (or Nexus 9332 A and B) R1-CS-9372-A-(config)# int e 1/33 R1-CS-9372-A-(config-if)# description L3 link to customer network R1-CS-9372-A-(config-if)# ip address /31 <- enter ip and netmask provided by customer R1-CS-9372-A-(config-if)# mtu 9216 R1-CS-9372-A-(config-if)# no shut 27. Add the network statement and neighbor statement to bgp configuration. R1-CS-9372-A-# conf t Enter configuration commands, one per line. End with CNTL/Z. R1-CS-9372-A-(config)#router bgp <- AS number R1-CS-9372-A-(config-router-af)# network /31 <- add network statement for L3 interfaces that connects to customer core network. R1-CS-9372-A-(config-router-neighbor)# template peer BGPCORE <- template to save time on neighborships configuration R1-CS-9372-A-(config-router-neighbor)# address-family ipv4 unicast R1-CS-9372-A-(config-router-neighbor)# next-hop-self R1-CS-9372-A-(config-router-neighbor)# soft-reconfiguration inbound Chapter 3: UCP 4000 Cisco Systems 91

100 Uplink Configurations R1-CS-9372-A-(config-router-neighbor)# neighbor /31 remote-as <- establish the neighborship with the ip address on directly connected customers core link. R1-CS-9372-A-(config-router-neighbor)# inherit peer BGPCORE 28. Verify that the BGP neighborships are in FULL state on all L3 interfaces that are connected to neighboring switches. R1-CS-9372-A-# sh ip bgp summary BGP summary information for VRF default, address family IPv4 Unicast BGP router identifier , local AS number BGP table version is 7, IPv4 Unicast config peers 5, capable peers 4 0 network entries and 0 paths using 0 bytes of memory BGP attribute entries [0/0], BGP AS path entries [0/0] BGP community entries [0/0], BGP clusterlist entries [0/0] Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd d05h d05h Scenario 3: Static Routing If customer has decided not to use L3 peering through a routing protocol with the 9332 or 9372, a default route can be added on the switches that are connected to the customer network via a L3 interface. This way the customer network can be reached. 29. Login to the Nexus 9372 A and B, or 9332 A and B which be used for uplink connection using SSH. 30. Configure layer 3 interfaces on Nexus 9372 A and B (or Nexus 9332 A and B) R1-CS-9372-A-(config)# int e 1/33 R1-CS-9372-A-(config-if)# description L3 link to customer network R1-CS-9372-A-(config-if)# ip address /31 <- enter ip and netmask provided by customer R1-CS-9372-A-(config-if)# mtu 9216 R1-CS-9372-A-(config-if)# no shut 31. [BGP case]add the default route into routing bgp configuration. R1-CS-9372-A-(config)#router bgp <- AS number R1-CS-9372-A-(config-router-af)# network /0 92 Chapter 3: UCP 4000 Cisco Systems

101 Uplink Configurations 32. [OSPF case]add the default route into routing ospfconfiguration. R1-CS-9372-A-(config)#router ospf 100 R1-CS-9372-A-(config-router)# default-information originate 33. Verify that the BGP neighborships are in FULL state on all L3 interfaces that are connected to neighboring switches. R1-CS-9372-A-# ip route / <- fill in the ip address s with the customer's ip address of the L3 interface. Chapter 3: UCP 4000 Cisco Systems 93

102 94 Chapter 3: UCP 4000 Cisco Systems

103 4 UCP 4000E Systems This chapter will cover the network architecture and uplink configuration examples for UCP 4000E Systems. Network Architecture and Configurations Overview In the UCP 4000E model, UCP utilizes Cisco Nexus 5548UP switch, and leverages its Storage Network technology (FCoE). UCP 4000E Network Architecture Physical Configurations The following table shows port usage of the Cisco Nexus 5548UP switches. Chapter 4: UCP 4000E Systems 95

104 Network Architecture and Configurations Overview Cisco Nexus 5548UP Port Usage Usage Port Counts Speed Uplinks to Core Network 4 10Gbps ISL to other Nexus 5548UP 4 10Gbps CB500 Blade Server Connectivity (FCoE) (Connecting to CB500 In-Chassis Pass-Throu module) Max 24 10Gbps CB500 SVP Connectivity 3 1Gbps Management Server Connectivity 3 10Gbps Storage System Management Connectivity 1 1Gbps Storage System Connectivity (Native FC) 8 8 or 16Gbps UCP Director Network Management Features Onboarding Switches The UCP Director supports onboarding of the Nexus 5548UP for management and health monitoring. Switch Health Monitoring The UCP Director monitors the health of the onboard switches and notifies if there are any warnings or errors detected on the switches. Configuration Backup UCP Director will periodically take backup configurations of the switches in its inventory. UCP Director can also manually backup/apply the switch configurations through user request. Configure Host/Cluster Network This allows the user to configure the Nexus 5548UP server facing interface on a blade by blade basis. The native vlan which is configured and managed by UCP for the management network and additional vlans such as vmotion or Compute vlans can also be managed through this UCP Director feature. VLAN DB management - If a new vlan is added through the Configure Host/Cluster Network feature, the vlan is added to the pair of Nexus 5548UP server facing interfaces as an allowed vlan and is created in the vlan db. 96 Chapter 4: UCP 4000E Systems

105 Network Architecture and Configurations Overview FC Zone management - UCP Director will configure FC Zone for storage facing FC ports and server facing vfc ports. UCP Appliance Initial vpc and Spanning Tree Configuration vpc peer-link The vpc peerlink trunk port-channel interface which allows traffic to ingress/egress between vpc peers also allows all vlans by default. vpc keep-alive The vpc keep-alive link is established through the management vrf with the source as MGMT 0 IP and destination as the MGMT 0 IP on the peer. vpc orphan-port suspend In the event that there is a vpc peer-link failure, all server facing vpc orphan ports will be disabled to avoid a vpc dual-active scenario. vpc peer-gateway Peer gateway is configured on vpc peers to act as the gateway even when packets are destined to the vpc peers MAC address. vpc peer-switch Allows both vpc peers to act as Root Bridge for the vlan. vpc ip arp synchronize Allows both vpc peers to synchronize arp entries for faster ARP learning and convergence. vpc delay restore 240 In the event of switch reboot, the vpc bring-up is delayed by 240 seconds to allow network re-converge before bringing the vpc peer up. Rapid-PVST Mode Layer 2 mode) Spanning tree rapid-pvst mode is configured with the priority vpc port-channels - vpc trunk port-channel is configured on uplink ports during initial deployment. Sample vpc Configuration vpc domain 901 peer-switch role priority 100 peer-keepalive destination source delay restore 240 peer-gateway auto-recovery ipv6 nd synchronize Chapter 4: UCP 4000E Systems 97

106 Network Architecture and Configurations Overview ip arp synchronize Sample Spanning tree configuration spanning-tree vlan priority Server Facing Port Configurations The UCP Director manages the server facing ports which are connected to the CB500 chassis via passthru modules that connect to blades. When managing the server facing ports, UCP Director sets the port in switchport trunk mode. In addition, it sets the native vlan on the port to the specified management vlan id which is chosen during deployment. It is critical these settings are never modified because the native vlan is used by UCP Director for management functions. Finally, the port is set to spanning-tree port type edge trunk and whichever vlans are allowed are configured by UCP Director through the switchport trunk allowed vlan command. Once user needs to modify the allowed vlans they can perform a Configure host/cluster vlan operation and UCP Director will modify the port configuration accordingly. The "spanning-tree port type edge trunk" and "vpc orphan-port suspend" are configured on all of server facing ports during UCP Deployment. UCP Director Software will not touch these configuration. HDS recommends not to change these configurations. Also, Virtual FC interfaces are created and configured during UCP Deployment. This vfc port is bound to server facing ports, and used for FCoE traffic. Sample Port Configuration interface 1/17 description to_blade_server switchport mode trunk switchport trunk native vlan 92 switchport trunk allowed vlan spanning-tree port type edge trunk vpc orphan-port suspend interface vfc17 bind interface 1/17 no shutdown 98 Chapter 4: UCP 4000E Systems

107 Uplink Configurations Uplink Configurations The UCP 4000E system shares a pair of Cisco Nexus 5548UP switches for data traffic and management traffic. The Cisco 5548UP switches act as a single switch in Virtual Port Channel (vpc) configuration for spanning tree topology determination. The different options for upstream connectivity to the production network infrastructure are dependent on whether the immediate upstream pair of switches is in vpc or spanning-tree configuration. By default, port 1 to 4 on both Cisco Nexus 5548UP switches are dedicated for connecting core network. And, port channel 10 is configured for these ports as a vpc. Typically the following 4 scenarios can be applied. # Scenario Example of Network Topology 1 Connecting to multi chassis ether channel technology enabled switches. & Single Core Network Chapter 4: UCP 4000E Systems 99

108 Uplink Configurations # Scenario Example of Network Topology 2 Connecting to standard switches (non multi chassis channel technology switch) & Single Core Network 3 Connecting to multi chassis ether channel technology enabled switches. & Separated Core Network 4 Connecting to standard switches & Separated Core Network The following scenarios show the example of detail configuration. Important: In the Double-sided vpc configuration, vpc domain identifiers must be different between upstream switches vpc domain and UCP vpc domain. During UCP Deployment, HDS support will configure vpc domain ID using the ID which HDS corrects during pre-engagement. If the domain ID need to be changed later, please contact HDS support. 100 Chapter 4: UCP 4000E Systems