Cisco Metacloud Controller Bundle Design and Implementation Guide v2.3

Transcription

1 Cisco Metacloud Controller Bundle Design and Implementation Guide v Cisco and/or its affiliates. All rights reserved. This document is Cisco Public.

2 Table of Contents Preface... 4 Overview... 5 Components... 5 Architecture Overview... 6 Starter Configuration... 6 General Purpose Configuration... 6 High Performance Configuration... 7 Installing the Metacloud Controller... 7 Installation Checklist... 7 Data Center Requirements... 8 Deployments Beyond 40 Servers... 8 Step 1 Starter Bundle Installation... 9 Required Cables... 9 Cisco 1001-X Series Aggregation Services Routers 2.5G Throughput... 9 Installing the Cisco Switches with M12PQ Modules Cabling for Starter Configuration Port Map for Physical Connections Step 1 General Purpose Installation without 9332 Switches Required Cables Cisco 1001-X Series Aggregation Services Router 10G Throughput Installing the Cisco Switches with M12PQ Modules Cabling for General Purpose Configuration without 9332 Switches Port Map for Physical Connections Step 1 General Purpose Installation with 9332 Switches Required Cables Cisco 1001-X Series Aggregation Services Router 10G Throughput Installing the Cisco Switches with M12PQ Modules Installing the 9332PQ Switches for Expansion Cabling for General Purpose Configuration with 9332 Switches Port Map for Physical Connections Step 1 High Performance Installation without 9332 Switches

3 Required Cables Installing Cisco 1002-X Series Aggregation Services Routers 20G Throughput Cisco Switches with M12PQ Modules Cabling for High Performance Configuration Port Map for Physical Connections Step 1 High Performance Installation with 9332 Switches Required Cables Cisco 1002-X Series Aggregation Services Routers 20G Throughput Cisco Switches with M12PQ Modules Installing the 9332PQ Switches for Expansion Cabling for High Performance Configuration with 9332PQ Switches Port Map for Physical Connections Step 2 Cabling Compute Servers to the Switches Step 3 Cabling for Remote Console (Out-of-Band) Connectivity Cisco 2901 w/ HWIC-16A and 2 CAB-HD8ASYNC Terminal Server Cabling for Remote Console (Out-of-Band) Connectivity (without 9332 Switches) Cabling for Remote Console (Out-of-Band) Connectivity (with 9332PQ Switches) Configuring and Establishing Remote Console Connectivity Providing the Required Network Information Running the Initial Configuration Template Applying the Configuration Step 4 Connecting the UCS C220 Control Plane Servers Step 5 Configure the BIOS on UCS Servers and Passwords on Switches Configure the BIOS on UCS Servers Configure Passwords on Switches Step 6 Open Network Ports Administrative Outbound Connections OOB Outbound Connections OOB Inbound Connections Default VLAN IDs Enable Traffic from Metacloud to Floating IP addresses Next Steps

4 Preface OpenStack is a free and open source Infrastructure-as-a-Service (IaaS) cloud computing project released under the Apache License. It enables enterprises and service providers to offer on-demand computing resources by provisioning and managing large networks of virtual machines. Cisco Metacloud, formerly known as Cisco OpenStack Private Cloud, is a software as a service offering from Cisco that allows organizations to take advantage of the speed and agility that come with OpenStack without having to manage or operate the underlying infrastructure components, including OpenStack services. Organizations that adopt Cisco Metacloud experience many of the same benefits they would in a public cloud but enjoy the added flexibility, reliability, and security of their own data centers. Using Cisco Unified Computing System (UCS), Cisco 9000 Series switches, and Cisco Aggregation Services Routers, known as the Controller Bundle, Cisco OpenStack Private Cloud provides a next-generation data center platform that unites computing, network, storage access, and virtualization into a single cohesive system. The Cisco Metacloud Controller Bundle provides a validated architecture for deploying the Cisco Metacloud service, and the ability to grow incrementally as capacity needs change. This combination of the Cisco UCS platform and Cisco OpenStack Private Cloud accelerates your IT by enabling faster deployments, greater flexibility of choice, efficiency, and lower risk. This Cisco Validated Design document provides design and implementation guidance for deploying the Cisco Metacloud Controller Bundle to enable a customer s multi-tenant cloud services. The Cisco Metacloud Controller Bundle for Cisco Metacloud Design and Implementation Guide is the first edition reference architecture with accompanying Bill of Materials (BOM). As the Cisco Metacloud solution is updated to support changes to OpenStack or advanced features that Cisco may offer there will be subsequent editions of this design guide. 4

5 Overview The Cisco Metacloud Controller Bundle is a converged infrastructure solution based on Cisco UCS,, and Aggregation Services Routers that provide a convenient, consistent, pre-configured hardware and network architecture that has been validated for Cisco Metacloud usage. The extensive validation and documentation provided in this Cisco Validated Design clearly explains the steps involved in the installation of the Controller Bundle solution. The Cisco Metacloud solution contains the following key components: Cisco Metacloud service subscription Cisco Metacloud Controller Bundle o o o o Cisco UCS C-Series servers for cloud orchestration Cisco 9300 Series Switches Cisco Aggregate Services Router (ASR) Cisco 2901 Integrated Services Router (ISR) Cisco UCS or other servers, that meets the minimum requirements, for Cisco Metacloud. Components The Cisco Metacloud Controller Bundle for Cisco Metacloud offers 3 different configurations options and includes the following components: Three UCS C220 M4 Servers Provide a 3-node control plane cluster to ensure high availability of key private cloud services. Two ASR1000 Series Routers Provide up to 20Gb/s network throughput from the cloud s network to the customer s data center and represents the north/south demarcation of the cloud. South of these devices are managed by Cisco, north is the customer s network. Two Switches Provide top of rack (ToR) and aggregation switching, including providing a path for future compatibility with Cisco ACI. Additional pairs of 9000 Series switches can be added to these to expand server capacity. One 2901 Integrated Services Router Provides out of band (OOB) access to all the servers and networking components of the Metacloud Controller Bundle. 5

6 Architecture Overview The Cisco Metacloud Controller Bundle for Cisco Metacloud offers 3 configuration options. The following diagram details the physical architecture of the Controller Bundle. Starter Configuration The Starter configuration supports up to 40 compute servers and offers up to 2.5Gbps of bandwidth throughput. The throughput of a configuration includes the total amount of network traffic coming into and out of the ASR. This includes traffic to the cloud from an external customer network or from the cloud to an upstream gateway (north/south traffic), and traffic between subnets within the Availability Zone (east/west traffic). This configuration uses the Cisco ASR1001-X Services router and uses 2 of the 6 onboard 1G ports. Note The Cisco ASR1001-X Services router comes with two 10G ports, the 2.5G license does not enable these ports and they will be inactive until an appropriate license is purchased. General Purpose Configuration The General Purpose configuration supports up to 100 compute servers and up to 10Gbps of bandwidth throughput. The throughput of a configuration includes the total amount of network traffic coming into and out of the ASR. This includes traffic to the cloud from an external customer network or from the cloud to an upstream gateway (north/south traffic), and traffic between subnets within the Availability Zone (east/west traffic). This configuration uses the Cisco ASR1001-X Services router and uses one of the two onboard 10G ports. Note The Cisco ASR1001-X Services router comes with two 10G ports, the 10G license only enables one of these ports with the other being inactive until an appropriate license is purchased. 6

7 High Performance Configuration The High Performance configuration supports up to 400 servers and up to 20Gbps of bandwidth throughput. The throughput of a configuration includes the total amount of network traffic coming into and out of the ASR. This includes traffic to the cloud from an external customer network or from the cloud to an upstream gateway (north/south traffic), and traffic between subnets within the Availability Zone (east/west traffic. This configuration uses the Cisco ASR1002-X Services router and uses two 10G SPA ports as well as two 10G XFP transceivers. Installing the Metacloud Controller One of the main advantages and key differentiators of the Cisco Metacloud is the ease with which it is operationalized. The high-level steps in deploying and configuring the Controller Bundle are listed below. More detailed instructions are provided later in this document. Installation Checklist Use the following checklist to ensure all steps are completed: Step 1. Rack and cable the controller bundle hardware (routers and switches, controller servers) Step 1 depends on your configuration, choose from the following options before continuing on to Step 2. Starter Bundle Installation (Step 1) General Purpose Installation without 9332 Switches (Step 1) General Purpose Installation with 9332 Switches (Step 1) High Performance Installation without 9332 Switches (Step 1) High Performance Installation with 9332 Switches (Step 1) Step 2. Rack and cable the Capacity Servers. Step 3. Cable and Configure the Cisco 2901 Integrated Services Router. Step 4. Cable the Control Plane Servers. Step 5. Configure the BIOS on UCS Servers and Password on 9000 Switches. Step 6. Open network ports (Firewall ACLs, etc) to enable remote operation. After these steps are completed, the Metacloud Support will perform the following: Verify network connectivity. Validate equipment is wired as required. Remote installation, monitoring, and operation of the Cisco Metacloud service. 7

8 You must follow the instructions in this guide to configure the hardware as specified. Configure the BIOS/CIMC setting on the UCS servers (example: enable DHCP, enable & set VLAN ID to 1000). Ensure the BIOS setting is correct for all servers, factory-shipped configurations may not be correct. ISR 2901 Configuration script and specifications for the slot number for placement of modules must be exact. The script will fail if the specifications are not met. For example, Async module must be in slot 0. Follow the wiring/connecting instructions exactly. Connect devices to specified ports as indicated, the installation scripts are written according to what is connected to specific ports ON ISR, ASR, ). The script will fail if the specifications are not met. Contact Metacloud Support for any questions during your installation. Data Center Requirements The following are requirements and recommendations for the environment the Controller Bundle configuration is deployed in: The ASR1000 Series routers require network access from all sources that need to access instances in the cloud and should be redundantly connected to two independent network paths. The 2901 ISR requires either a public or private IP address and the ability to initiate a VPN connection to the Metacloud Operations Center. Direct access through SSH or console to the 2901 router is also required. This may require changes to the outbound firewall rules and routes. All servers and networking equipment in both the Controller Bundle and the cloud compute capacity should be connected to redundant power sources within the data center rack. A local computer with a console cable appropriate for configuring Cisco devices is required for the initial software configuration of the 2901 ISR. A monitor and keyboard are required to update BIOS settings. You must also complete the cabling and configuration steps Configuring Your Metacloud Installation to complete your installation. Deployments Beyond 40 Servers Cisco Metacloud can support up to 400 physical servers in one pod using switches and a pair of 9332PQ switches. By introducing the 9332PQ switches, Metacloud can aggregate up to 10 pairs of switches and connect up to 400 individual servers. If you plan on expanding past 40 servers at any time, it is highly recommended that you install the 9332PQ switches during the initial installation of Metacloud to mitigate downtime during future expansions. 8

9 Step 1 Starter Bundle Installation When you have completed these steps, go to Cabling Compute Servers to the Switches (Step 2) to continue your installation. Required Cables Cables are NOT provided with your Cisco Metacloud Controller hardware. The table below provides guidance on the required cables. From To Cable Type Quantity ISR Both 9K & Both ASRs High Density 8-port EIA-232 Async Cable Included with ISR ISR Customer s OOB Internet access RJ45/ Cat6 Ethernet 1 ISR Both N9K Mgmt RJ45/ Cat6 Ethernet (1G) 2 ISR Both N9K Data RJ45/ Cat6 Ethernet (1G) 2 ISR Both ASRs (mgmt.) RJ45/ Cat6 Ethernet (1G) 2 Each of the two 93180YC- Each of the two 93180YC- Customer Routers Cat6 Ethernet (1G) 4 Both ASRs Cat6 Ethernet (1G) 4 1st 2nd Multi-mode Fiber with LC connector Three UCS Control servers BOTH 9K switches Twinax cable eg: SFP- H10GB-CU2M= +Each Compute Server BOTH 9K switches ++Twinax cable eg: SFP- H10GB-CU3M= 4 6 Compute Servers x2 +If non-cisco UCS servers are used as compute servers that connect to the ISR, then each of their console ports must connect to a Catalyst This cable could vary for non-ucs compute servers and may require RJ45 SFP cable in 9K switches. Cisco 1001-X Series Aggregation Services Routers 2.5G Throughput Accessories for each (1) ASR1001-X Series router (2x10GB, 6x1GE ports and 2.5G IPBase License): One Cisco ASR1000 Series 8GB DRAM One Cisco ASR1000 Series IOS XE UNIVERSAL Two Cisco 1000BASE-T SFP transceivers 9

10 To install the Cisco 1001-x Series Aggregation Services Routers: 1. Insert two 1000BASE-T SFP transceivers into ports GE0 and GE1 of the ASR1001-X. 2. Repeat for the second ASR1001-X router. 3. Mount the two ASR1001-X routers into the rack. Installing the Cisco Switches with M12PQ Modules Accessories for each (1) switch: One 9300 Accessory Kit Five 1000BASE-T SFP Four QSFP40G BiDi Short-reach Transceiver One 12-port 40G Uplink Module To install the switches: 1. Insert four 40G QSFP+ BiDi transceivers into 40G ports 51 to 54 of the uplink module. 2. Insert five 1000BASE-T SFP transceivers into ports 44 to 48 of the base module. 3. Repeat for the second switch. 4. Mount the two switches in the rack. 10

11 Cabling for Starter Configuration To connect routers and switches: 1. Connect the two 1G interfaces from each ASR1001-X Series router slot GE0 and GE1 to ports 47 and 48 of each corresponding switch. 2. Connect the router s 1G connections to port 45 and 46 on each corresponding switch. 3. Cross-connect the four 40G connections to ports 51 to 54 between the two switches. 11

12 Port Map for Physical Connections This table lists the required physical cabling for the above-mentioned connections: Source Hostname Source Model Source Slot- Port Destination Hostname Destination Model Destination Slot-Port Customer Device YC- 45 Customer Device YC- 46 Customer Device YC- 45 Customer Device YC- 46 ASR1001X-1 ASR1000 Series GE0-1 ASR1001X-1 ASR1000 Series GE1-1 ASR1001X-2 ASR1000 Series GE0-2 ASR1001X-2 ASR1000 Series GE YC YC YC YC YC YC YC YC YC YC YC YC- 54 When you have completed these steps, see Cabling Compute Servers to the Switches (Step 2) to continue your installation. 12

13 Step 1 General Purpose Installation without 9332 Switches When you have completed these steps, go to Cabling Compute Servers to the Switches (Step 2) to continue your installation. Required Cables Cables are NOT provided with your Cisco Metacloud Controller hardware. The table below provides guidance on the required cables. From To Cable Type Quantity ISR Both 9K & Both ASRs High Density 8-port EIA-232 Async Cable Included with ISR BOM ISR Customer s OOB Internet access RJ45/ Cat6 Ethernet 1 ISR Both N9K Mgmt RJ45/ Cat6 Ethernet (1G) 2 ISR Both N9K Data RJ45/ Cat6 Ethernet (1G) 2 ISR Both ASRs (mgmt.) RJ45/ Cat6 Ethernet (1G) 2 1st 1st Customer Router ^Multi-mode Fiber with LC connector 1 2nd 2nd Customer Router ^Multi-mode Fiber with LC connector 1 1st 1st ASR Multi-mode Fiber with LC connector 1 2nd 2nd ASR Multi-mode Fiber with LC connector 1 1st 2nd 93180YC- Multi-mode Fiber with LC connector 4 Three UCS Control servers BOTH 9K switches Twinax cable eg: SFP-H10GB- CU2M= 6 +Each Compute Server BOTH 9K switches ++Twinax cable eg: SFP-H10GB- CU3M= Compute Servers x2 +If non-cisco UCS servers are used as compute servers that connect to the ISR, then each of their console ports must connect to a Catalyst This cable could vary for non-ucs compute servers and may require RJ45 SFP cable in 9K switches. ^Fiber Optic connection recommended to customer routers. Copper possible (IF using SMF or Copper connections THEN replace SFP-10G-SR in N9K BOM). Cisco 1001-X Series Aggregation Services Router 10G Throughput Accessories for each (1) ASR1001-X Series router (2x10GB, 6x1GE ports and 10G IPBase License): One Cisco ASR1000 Series 8GB DRAM One Cisco ASR1000 Series IOS XE UNIVERSAL One Cisco 10GBASE-SR SFP Module 13

14 To install the Cisco 1001-X Series Aggregation Services Routers: 1. Insert a 10GBASE-SR SFP module into port TE0 on each of the ASR1001-X routers. 2. Repeat for the second ASR1001-X router. 3. Mount the two ASR1001-X routers into the rack. Installing the Cisco Switches with M12PQ Modules Accessories for each (1) switch: One 9300 Accessory Kit Two 10GBASE-SR SFP Module One 1000BASE-T SFP Four QSFP40G BiDi Short-reach Transceiver One 12-port 40G Uplink Module To install the Cisco Switches with M12PQ Modules: 1. Insert four 40G QSFP+ BiDi transceivers into 40G ports 51 to 54 of the uplink module. 2. Insert two 10GBASE-SR SFPs into ports 45 and 47 of the base module. 3. Insert one 1000BASE-T SFP into port 44 of the base module. 4. Repeat for the second switch. 5. Mount the two switches in the rack. 14

15 Cabling for General Purpose Configuration without 9332 Switches To connect the ASR1001-X routers and Switches: 1. Connect one 10G interfaces from each ASR1001-X Series router slot TE0 to port 47 on each corresponding switch. 2. Connect the router s 10G connection to port 45 on each corresponding switch. 3. Cross-connect the four 40G connections to ports 51 to 54 between the two switches. 15

16 Port Map for Physical Connections The physical cabling for above-mentioned connections are listed in this table: Source Hostname Source Model Source Slot- Port Destination Hostname Destination Model Destination Slot-Port Customer Device-1 Customer Device-2 10G G ASR1001X-1 ASR1000 Series TE YC- ASR1001X-2 ASR1000 Series TE YC YC YC YC YC YC YC YC YC- 54 When you have completed these steps, see Cabling Compute Servers to the Switches (Step 2) to continue your installation. 16

17 Step 1 General Purpose Installation with 9332 Switches When you have completed these steps, go to Cabling Compute Servers to the Switches (Step 2) to continue your installation. Required Cables Cables are NOT provided with your Cisco Metacloud Controller hardware. The table below provides guidance on the required cables. From To Cable Type Quantity ISR Both 9332 High Density 8-port EIA-232 Async Cable Included with ISR BOM ISR Customer s OOB Internet access RJ45/ Cat6 Ethernet 1 ISR Both N9332 Mgmt RJ45/ Cat6 Ethernet (1G) 2 ISR Both N9332 Data RJ45/ Cat6 Ethernet (1G) 2 ISR Both ASRs (mgmt.) RJ45/ Cat6 Ethernet (1G) 2 1 st st Customer Router ^Multi-mode Fiber with LC connector 1 2nd nd Customer Router ^Multi-mode Fiber with LC connector 1 1 st pair of Both 9332 switches ^Multi-mode Fiber with LC connector 4 Additional pairs of 93180s (if present) Both 9332 switches ^Multi-mode Fiber with LC connector 4 for each additional pair of 93180s 1 st st ASR 2 nd nd ASR MPO-8 LC Duplex QSFP SR4 Optical Fanout Cable MPO-8 LC Duplex QSFP SR4 Optical Fanout Cable st 2 nd 93180YC- Multi-mode Fiber with LC connector 4 3 rd 4 th Multi-mode Fiber with LC connector 4 Three UCS Control servers First YC- switches Twinax cable eg: SFP-H10GB- CU2M= 6 +Each Compute Server BOTH 9K switches ++Twinax cable eg: SFP-H10GB- CU3M= Compute Servers x2 +If non-cisco UCS servers are used as compute servers that connect to the ISR, then each of their console ports must connect to a Catalyst This cable could vary for non-ucs compute servers and may require RJ45 SFP cable in 9K switches. ^Fiber Optic connection recommended to customer routers. Copper possible (IF using SMF or Copper connections THEN replace SFP-10G-SR in N9K BOM). 17

18 Cisco 1001-X Series Aggregation Services Router 10G Throughput Accessories for each (1) ASR1001-X Series router (2x10GB, 6x1GE ports and 10G IPBase License): One Cisco ASR1000 Series 8GB DRAM One Cisco ASR1000 Series IOS XE UNIVERSAL One Cisco 10GBASE-SR SFP Module To install the Cisco 1001-X Series Aggregation Services Routers: 1. Insert a 10GBASE-SR SFP module into port TE0 on each of the ASR1001-X routers. 2. Repeat for the second ASR1001-X router. 3. Mount the two ASR1001-X routers into the rack. Installing the Cisco Switches with M12PQ Modules Accessories for each (1) switch: One 9300 Accessory Kit Two 10GBASE-SR SFP Module One 1000BASE-T SFP Four QSFP40G BiDi Short-reach Transceiver One 12-port 40G Uplink Module To install the Cisco Switches with M12PQ Modules: 1. Insert four 40G QSFP+ BiDi transceivers into 40G ports 51 to 54 of the uplink module. 2. Insert two 10GBASE-SR SFPs into ports 45 and 47 of the base module. 3. Insert one 1000BASE-T SFP into port 44 of the base module. 4. Repeat for each switch. 5. Mount the four switches in the rack. 18

19 Installing the 9332PQ Switches for Expansion The wiring for the 9332PQ switches for expansion is slightly different. The 9332PQ switches will be connected to every switch and both ASRs in the controller. When deploying 9332PQ switches, the Cisco ASR 1000 Series routers will be connected to the 9332PQ switches instead of the switches. Also, because the 9332PQ switches only have 40G ports the ASR 2.5G license will not be supported with expansion and the 9332PQ. To connect the routers and switches: 1. Connect one 10G interface from each ASR1001-X Series router slot T0 to the 9332PQ. The 9332PQ switch only supports 40G ports, you must connect the Cisco 40-Gigabit-to- 4x10-Gigabit breakout cables optical fanout cables to ports 26 and 27. These optical fanout cables enable the four 10G ports to connect the ASR 1000 Series routers. Do not connect both ASR 1000 Series 10G ports to a single optical fanout cable. 2. Cross-connect the four 40G connections on ports 29 to 32 of each 9332PQ switches. 3. Cross-connect a 40G connection from each switch to the same port of each 9332PQ switch (port 1 on each 9332PQ for the first switch, port 2 for the second switch, port 3 for the third switch, etc). 19

20 Cabling for General Purpose Configuration with 9332 Switches TO-DO Heather please change *26 and 27 use breakout cable label to *26 & 27 use optical fanout cable ) 20

21 Port Map for Physical Connections The physical cabling for above-mentioned connections are listed in this table: Source Hostname Source Model Source Slot-Port Destination Hostname Destination Model Destination Slot-Port Customer Router PQ PQ 27(use optical fanout cable for 10G) Customer Router PQ PQ 27(use optical fanout cable for 10G) ASR1001X-1 ASR1001X-2 ASR1000 Series ASR1000 Series TE0 9332PQ PQ 26 (use optical fanout cable for 10G) TE1 9332PQ PQ 26 (use optical fanout cable for 10G) 9332PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ 1 21

22 PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ 4 When you have completed these steps, see Cabling Compute Servers to the Switches (Step 2) to continue your installation. Step 1 High Performance Installation without 9332 Switches When you have completed these steps, go to Cabling Compute Servers to the Switches (Step 2) to continue your installation. Required Cables Cables are NOT provided with your Cisco Metacloud Controller hardware. The table below provides guidance on the required cables. From To Cable Type Quantity ISR Both 9K & Both ASRs High Density 8-port EIA-232 Async Cable Included with ISR ISR Customer s OOB Internet access RJ45/ Cat6 Ethernet 1 ISR Both N9K Mgmt RJ45/ Cat6 Ethernet (1G) 2 ISR Both N9K Data RJ45/ Cat6 Ethernet (1G) 2 ISR Both ASRs (mgmt.) RJ45/ Cat6 Ethernet (1G) 2 1st 1st Customer Router ^Multi-mode Fiber with LC connector 2 2nd 2nd Customer Router ^Multi-mode Fiber with LC 2 22

23 connector 1st 1st ASR Multi-mode Fiber with LC connector 2nd 2nd ASR Multi-mode Fiber with LC connector 1st 2nd Multi-mode Fiber with LC connector Three UCS Control servers BOTH 9K switches Twinax cable eg: SFP-H10GB- CU2M= +Each Compute Server BOTH 9K switches ++Twinax cable eg: SFP- H10GB-CU3M= Compute Servers x2 +If non-cisco UCS servers are used as compute servers that connect to the ISR, then each of their console ports must connect to a Catalyst This cable could vary for non-ucs compute servers and may require RJ45 SFP cable in 9K switches. ^Fiber Optic connection is recommended to customer routers. If you are using SMF or Copper connections replace SFP- 10G-SR in N9K BOM. Installing Cisco 1002-X Series Aggregation Services Routers 20G Throughput Accessories for each (1) ASR1002-X router (2x10GB, 6x1GE ports and 36G IPBase License): One Cisco ASR1000 Series 4GB DRAM One Cisco ASR1000 Series IOS XE UNIVERSAL Two 10GBASE-SR Module and XFP Transceivers To install the Cisco 1002-X Series Aggregation Services Routers: 1. Insert two 10G Shared Port Adapters (SPA) into slots 1 and 2 of each ASR1002-X. 2. Insert a single XFP 10G transceiver into each SPA. 3. Repeat for the second ASR1002-X router. 4. Mount the two ASR1002-X routers into the rack. Cisco Switches with M12PQ Modules Accessories for each (1) switch 23

24 One 9300 Accessory Kit Four 10GBASE-SR SFP Module One 1000BASE-T SFP Four QSFP40G BiDi Short-reach Transceiver One 12-port 40G Uplink Module To install the Cisco switches: 1. Insert four 40G QSFP+ BiDi transceivers into 40G ports 51 to 54 of the uplink module. 2. Insert four 10GBASE-SR SFPs into ports 45 through 48 of the base module. 3. Insert one 1000BASE-T SFP into port 44 of the base module. 4. Repeat for the 2 nd switch. 5. Mount the two switches in the rack. 24

25 Cabling for High Performance Configuration To connect the ASR1002-X Routers and Switches: 1. Connect the two 10G interfaces on slots 1 and 2 of each ASR1002-X Series router to the 10G ports 47 and 48 of each corresponding switch. 2. Connect the router s 10G connections to port 45 and 46 of each corresponding switch. 3. Cross-connect the four 40G connections on ports between the two switches. 25

26 Port Map for Physical Connections The physical cabling for above-mentioned connections are listed in this table: Source Hostname Source Model Source Slot- Port Destination Hostname Destination Model Destination Slot-Port Customer Router YC- 45 Customer Router YC- 46 Customer Router YC- 45 Customer Router YC- 46 ASR1002X-1 ASR1000 Series 0/1/ YC- ASR1002X-1 ASR1000 Series 0/2/ YC- ASR1002X-2 ASR1000 Series 0/1/ YC- ASR1002X-2 ASR1000 Series 0/2/ YC YC YC YC YC YC YC YC YC YC YC YC YC

27 Step 1 High Performance Installation with 9332 Switches When you have completed these steps, go to Cabling Compute Servers to the Switches (Step 2) to continue your installation. Required Cables Cables are NOT provided with your Cisco Metacloud Controller hardware. The table below provides guidance on the required cables. From To Cable Type Quantity ISR Both 9332 HD 8-port EIA-232 Async Cable Included with ISR BOM ISR Customer s OOB Internet access RJ45/ Cat6 Ethernet 1 ISR Both N9332 Mgmt RJ45/ Cat6 Ethernet (1G) 2 ISR Both N9332 Data RJ45/ Cat6 Ethernet (1G) 2 ISR Both ASRs (mgmt.) RJ45/ Cat6 Ethernet (1G) 2 1 st st Customer Router ^Multi-mode Fiber with LC connector 1 2nd nd Customer Router ^Multi-mode Fiber with LC connector 1 1 st pair of Additional pairs of 93180s (if present) Both 9332 switches ^Multi-mode Fiber with LC connector 4 Both 9332 switches ^Multi-mode Fiber with LC connector 4 for each additional pair of 93180s 1 st st ASR MPO-8 LC Duplex QSFP SR4 Optical Fanout Cable 2 nd nd ASR MPO-8 LC Duplex QSFP SR4 Optical Fanout Cable st 2 nd Multi-mode Fiber with LC connector 4 3 rd 4 th Multi-mode Fiber with LC connector 4 3 UCS Control servers First pair 93180YC- switches Twinax cable: SFP-H10GB-CU2M= 6 +Each Compute Server BOTH 9K switches ++Twinax cable: SFP-H10GB-CU3M= Compute Servers x2 +If non-cisco UCS servers are used as compute servers that connect to the ISR, then each of their console ports must connect to a Catalyst This cable could vary for non-ucs compute servers and may require RJ45 SFP cable in 9K switches. ^Fiber Optic connection is recommended to customer routers. If you are using SMF or Copper connections replace SFP- 10G-SR in N9K BOM. 27

28 Cisco 1002-X Series Aggregation Services Routers 20G Throughput Accessories for each (1) ASR1002-X router (2x10GB, 6x1GE ports and 36G IPBase License): One Cisco ASR1000 Series 4GB DRAM One Cisco ASR1000 Series IOS XE UNIVERSAL Two 10GBASE-SR Module and XFP Transceivers To install the Cisco 1002-X Series Aggregation Services Routers: 1. Insert two 10G Shared Port Adapters (SPA) into slots 1 and 2 of each ASR1002-X. 2. Insert a single XFP 10G transceiver into each SPA. 3. Repeat for the second ASR1002-X router. 4. Mount the two ASR1002-X routers into the rack. Cisco Switches with M12PQ Modules Accessories for each (1) switch One 9300 Accessory Kit Four 10GBASE-SR SFP Module One 1000BASE-T SFP Four QSFP40G BiDi Short-reach Transceiver One 12-port 40G Uplink Module To install the Cisco switches: 1. Insert four 40G QSFP+ BiDi transceivers into 40G ports 51 to 54 of the uplink module. 28

29 2. Insert four 10GBASE-SR SFPs into ports 45 through 48 of the base module. 3. Insert one 1000BASE-T SFP into port 44 of the base module. 4. Repeat for each switch. 5. Mount the four switches in the rack. Installing the 9332PQ Switches for Expansion The wiring for the 9332PQ switches for expansion is slightly different. The 9332PQ switches will be connected to every switch and both ASRs in the controller. When deploying 9332PQ switches, the Cisco ASR 1000 Series routers will be connected to the 9332PQ switches instead of the switches. Also, because the 9332PQ switches only have 40G ports the ASR 2.5G license will not be supported with expansion and the 9332PQ. To connect the routers and switches: 1. Connect one 10G interface from each ASR1002-X Series router slot T0 to the 9332PQ. The 9332PQ switch only supports 40G ports, you must connect the Cisco 40-Gigabit-to- 4x10-Gigabit breakout cables optical fanout cables to ports 26 and 27. These optical fanout cables enable the four 10G ports to connect the ASR 1000 Series routers. Do not connect both ASR 1000 Series 10G ports to a single optical fanout cable. 2. Cross-connect the four 40G connections to ports 29 to 32 of each 9332PQ switch. 29

30 3. Cross-connect a 40G connection from each switch to the same port of each 9332PQ switch (port 1 on each 9332PQ for the first switch, port 2 for the second switch, port 3 for the third switch, etc). Cabling for High Performance Configuration with 9332PQ Switches 30

31 Port Map for Physical Connections The physical cabling for above-mentioned connections are listed in this table: Source Hostname Source Model Source Slot-Port Destination Hostname Destination Model Destination Slot-Port Customer Router-1 If 40G 9332PQ PQ 27 Customer Router-2 If 40G 9332PQ PQ 27 Customer Router-1 Customer Router-2 ASR1002X-1 ASR1002X-1 ASR1002X-2 ASR1002X-2 ONLY if 10G connection ONLY if 10G connection ASR1000 Series ASR1000 Series ASR1000 Series ASR1000 Series 9332PQ PQ 27 (use optical fanout cable E1/27/1 and E1/27/2) 9332PQ PQ 27 (use optical fanout cable E1/27/1 and E1/27/2) 10G 0/1/0 9332PQ PQ 26 (use optical fanout cable E1/26/1 and E1/26/2) 10G 0/2/0 9332PQ PQ 26 (use optical fanout cable E1/26/1 and E1/26/2) 10G 0/1/0 9332PQ PQ 26 (use optical fanout cable E1/26/1 and E1/26/2) 10G 0/2/0 9332PQ PQ 26 (use optical fanout cable E1/26/1 and E1/26/2) 9332PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ

32 PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ PQ 4 32

33 Step 2 Cabling Compute Servers to the Switches Note This step can be completed after the Controller Bundle hardware has been set up. All compute servers should be dual connected to the switches. Each connection should use the compute server s 10Gb network interface to connect to the switches. If you are using Cisco C-Series servers for compute, the 10Gb network ports on the MLOM module should be directly connected to the two switches. Information on how to install Cisco C-Series servers can be found on the Cisco website ( If you are using Cisco servers, a Cisco Fabric Interconnect should only be used if using Cisco B-Series servers. All other supported Cisco compute servers should connect directly to the Cisco switches. If you are using Cisco B-Series servers for compute, they must be connected to a pair of Cisco Fabric Interconnects. The Cisco Fabric Interconnects should be dual connected to the switches. Please refer to the Cisco Fabric Interconnect documentation on the Cisco.com website for how the servers should be connected to the Cisco Fabric Interconnects ( Use the following Port Map to ensure the correct wiring is implemented. Source Hostname Source Model Source Slot- Port Destination Hostname -1 10G 1/4 Compute Server G 1/5 Compute Server G 1/6 Compute Server G 1/7 Compute Server G 1/8 Compute Server G 1/9 Compute Server G 1/10 Compute Server G 1/11 Compute Server G 1/12 Compute Server G 1/13 Compute Server G 1/14 Compute Server G 1/15 Compute Server G 1/16 Compute Server G 1/17 Compute Server G 1/18 Compute Server 15 33

34 -1 10G 1/19 Compute Server G 1/20 Compute Server G 1/21 Compute Server G 1/22 Compute Server G 1/23 Compute Server G 1/24 Compute Server G 1/25 Compute Server G 1/26 Compute Server G 1/27 Compute Server G 1/28 Compute Server G 1/29 Compute Server G 1/30 Compute Server G 1/31 Compute Server G 1/32 Compute Server G 1/33 Compute Server G 1/34 Compute Server G 1/35 Compute Server G 1/36 Compute Server G 1/37 Compute Server G 1/38 Compute Server G 1/39 Compute Server G 1/40 Compute Server G 1/41 Compute Server G 1/42 Compute Server G 1/43 Compute Server G 1/4 Compute Server G 1/5 Compute Server G 1/6 Compute Server 3 34

35 -2 10G 1/7 Compute Server G 1/8 Compute Server G 1/9 Compute Server G 1/10 Compute Server G 1/11 Compute Server G 1/12 Compute Server G 1/13 Compute Server G 1/14 Compute Server G 1/15 Compute Server G 1/16 Compute Server G 1/17 Compute Server G 1/18 Compute Server G 1/19 Compute Server G 1/20 Compute Server G 1/21 Compute Server G 1/22 Compute Server G 1/23 Compute Server G 1/24 Compute Server G 1/25 Compute Server G 1/26 Compute Server G 1/27 Compute Server G 1/28 Compute Server G 1/29 Compute Server G 1/30 Compute Server G 1/31 Compute Server G 1/32 Compute Server G 1/33 Compute Server G 1/34 Compute Server 31 35

36 -2 10G 1/35 Compute Server G 1/36 Compute Server G 1/37 Compute Server G 1/38 Compute Server G 1/39 Compute Server G 1/40 Compute Server G 1/41 Compute Server G 1/42 Compute Server G 1/43 Compute Server 40 36

37 Step 3 Cabling for Remote Console (Out-of-Band) Connectivity The ISR 2901 Configuration script and specifications for the slot number for placement of modules must be exact. The script will fail if the specifications are not met. Cisco 2901 w/ HWIC-16A and 2 CAB-HD8ASYNC Terminal Server 1 16-Port Async HWIC 1 512MB DRAM 1 256MB Compact Flash 2 High Density 8-port EIA-232 Async Cable ( Octal Cables ) 1 Eight port 10/100/1000 Ethernet switch interface card To install the Cisco 2901 Integrated Services Router (Terminal Server): 1. Install the 16-port Async HWIC module on EHWIC slot 0 (the rightmost slot) of the 2901 router. 2. Install the 8-port Ethernet switch module on EHWIC slot 2 and 3 of router. This module occupies two slots. 3. Mount the 2901 Integrated Services Router with the modules in the rack and connect the two octal cables to the Async HWIC module. Cabling for Remote Console (Out-of-Band) Connectivity (without 9332 Switches) Connecting the 2901 ISR to the Cisco ASR1000 Series, Cisco switch console ports and management ports provides remote console connectivity for Cisco Metacloud Operations. The required physical connections are illustrated below: 37

38 Connecting the C2901 Async Module Two sets of octal cables connect the console ports of the Cisco ASR1000 Series routers and Cisco switches to the Cisco ISR There are 4 connections required for this setup; P1, P2, P3, and P4. These connections are at the bottom octal cable. Follow the instructions below to connect the consoles: 1. Connect the octal cable to the Async module. 2. Connect the cables to the following console ports: P1 connect to the first ASR1000 Series router console port. P2 connect to the second ASR1000 Series router console port. P3 connect to the first switch console port. P4 connect to the second switch console port. Connecting the 2901 Router To connect the 2901 Router: 38

39 1. Connect Gigabit Ethernet G0/0 (RJ45) to an uplink to the Internet for out-of-band (OOB) access. 2. Connect Gigabit Ethernet G0/1 (RJ45) to port 44 of the first switch. Connecting the 2901 Ethernet Switch Module The 8-port switch module is used to connect the management ports of the ASR1000 Series routers and 93180YC- switches. P1 connect to the first ASR1000 Series router s management port (G0). P2 connect to the second ASR1000 Series router s management port (G0). P3 connect to the first switch s management port (mgmt0). P4 connect to the second switch s management port (mgmt0). P5 connect to port 44 of the second switch. Cabling for Remote Console (Out-of-Band) Connectivity (with 9332PQ Switches) Connecting the 2901 ISR to the Cisco 2960, Cisco ASR1000 Series, Cisco 9332PQ, and Cisco 93180YC- switch console ports and management ports provides remote console connectivity for Cisco Metacloud Operations. The required physical connections are illustrated below: 39

40 Connecting the C2901 Async Module Two sets of octal cables connect the console ports of the Cisco ASR1000 Series routers and Cisco switches to the Cisco ISR There are 9 connections required for this setup; P1 to P8 and P1 on the second octal cable. These connections are at the bottom octal cable. Follow the instructions below to connect the consoles: 1. Connect the octal cable to the Async module. 2. Connect the cables to the following console ports: P1 connect to the first ASR1000 Series router console port. P2 connect to the second ASR1000 Series router console port. P3 connect to the first 9332PQ console port. P4 connect to the second 9332PQ console port. P5 connect to the 2960 console port. P6 connect to the first switch console port. P7 connect to the second switch console port. P8 connect to the third switch console port. P1 (second Async module) connect to the fourth switch console port. Connecting the 2901 Router To connect the 2901 Router: 1. Connect Gigabit Ethernet G0/0 (RJ45) to an uplink to the Internet for out-of-band (OOB) access. 2. Connect Gigabit Ethernet G0/1 (RJ45) to port 48 of the Connecting the 2960 Ethernet Switch Module Connect the management ports of the ASR1000 Series routers, 9332PQ switches, and switches to the connect to the first ASR1000 Series router s management port (G0). 2 connect to the second ASR1000 Series router s management port (G0). 3 connect to the first 9332PQ router s management port (G0). 4 connect to the second 9332PQ router s management port (G0). 5 connect to the first switch s management port (mgmt0). 6 connect to second switch s management port (mgmt0). 7 connect to third switch s management port (mgmt0). 8 connect to the fourth switch s management port (mgmt0) to 9332 Data Connections To connect the 2960 to the 9332PQ switches: 1. Connect port TE1/0/1 on the 2960 to port 25 on the first 9332PQ. 40

41 2. Connect port TE1/0/2 on the 2960 to port 25 on the second 9332PQ. 41

42 Configuring and Establishing Remote Console Connectivity The first-time configuration process involves gathering a few configuration details of the local network, inserting them into the configuration template provided, and using a computer connected to the console port of the Cisco ISR 2901 to apply the completed configuration. The initial bootstrap process is a one-time manual process. All future maintenances and upgrades will be done remotely by Metacloud Support as part of operating the service. Providing the Required Network Information You must provide network information to Metacloud Support to complete your installation. To provide the required network information: A publicly-routable IP Address for the ISR2901 or a private address that you can translate to a public IP for the purposes for establishing a VPN connection (OOB_WAN_IP) The gateway IP address (OOB_WAN_GW) on your device serves as a gateway for the above IP Running the Initial Configuration Template Complete the configuration template during the initial turn-up on the ISR2901 router. The initial password is changed by the Cisco Metacloud Operations Center once this setup is completed. The ISR2901 router comes with a default username and password as cisco/cisco. To run the configuration template: 1. Log into the ISR 2901 router. 2. Perform write erase and reload to boot up with a clean configuration. 3. Select a temporary password and replace the TEMP_PASSWORD in the configuration with the new password. You will then provide this password to the Metacloud Support team so that they can continue configuring the ISR After the ISR 2901 configuration has been completed, the Metacloud Support team will update the password to match current operating standards. 4. Paste the following configuration with the actual IP info for OOB_WAN_IP and OOB_WAN_GW. 42

43 ! hostname c2901! enable secret <TEMP_PASSWORD>! ip cef! license boot module c2900 technology-package securityk9!! username temp_admin privilege 15 secret <TEMP_PASSWORD>! no ip domain lookup ip domain name metacloud.in! crypto key generate rsa general-keys modulus 1024! ip ssh version 2! vrf definition wan! address-family ipv4 exit-address-family! access-list 101 permit any access-list 101 deny ip any any! interface GigabitEthernet0/0 vrf forwarding wan ip address <OOB_WAN_IP> ip access-group 101 in no shut! ip route vrf wan <OOB_WAN_GW>! line con 0 exec-timeout 15 0 logging synchronous 43

44 login local line 0/0/0 0/0/15 session-timeout 30 exec-timeout 30 0 disconnect-character 126 logging synchronous no exec transport input ssh transport output none line vty 0 15 exec-timeout 15 0 login local transport input ssh End Applying the Configuration Connect a computer s USB port to the Cisco 2901 console port via a Cisco console cable and a USB port adapter, and use a terminal application to access the console interface of the 2901 router. Configure the serial port settings as follows: Bits per second 9600 Data bits 8 Parity None Stop bits Flow control None Verify successful configuration by pinging an IP address elsewhere on the Internet, such as A successful attempt returns the following: #ping Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds:!!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms If the configuration is unsuccessful, verify the information is obtained correctly or contact Metacloud Support for assistance. 44

45 Step 4 Connecting the UCS C220 Control Plane Servers Carefully identify and isolate the three control plane servers that were ordered as part of the controller bundle and then connect them as specified below. Compute and control servers are not interchangeable. It is easy to confuse compute and control servers since they may be in the same shipment and appear very similar. It is necessary to ensure you correctly identify and connect the control plane servers. To connect the control plane servers: 1. For the first controller, locate the onboard MLOM module with two 10G ports. 2. Connect the first port to the 10G port 1 of first switch. 3. Connect the second port to the 10G port 1 of second switch. 4. Repeat this for the second and third controllers and connect to port 2 and 3 on each switch. 45

46 Step 5 Configure the BIOS on UCS Servers and Passwords on Switches Ensure the BIOS setting is correct for all servers, factory-shipped configurations are not always the correct configuration. You must configure the BIOS/CIMC setting on the UCS servers for example, enable DHCP, enable and set the VLAN ID to Configure the BIOS on UCS Servers For each UCS node (controllers and compute servers), the USB keyboard, video, and USB mouse can be connected through the front panel of the server using a Cisco-provided dongle. To configure the BIOS for EVERY UCS server (controllers and compute nodes): 1. Access the configuration screen by pressing <F8> on the following screen. 2. Enter the default username and password admin / password. Change the password to Cisco123. Metacloud Support will change this password once remote console connectivity is established. 3. Confirm the BIOS/CIMC settings are configured as follows: Address set to DHCP Active-active mode VLAN ID enabled and set to VLAN 1000 Factory-shipped BIOS configurations may not be the correct configuration for your installation. 46

47 4. Press <F10> to save. 5. Press <ESC> to exit the BIOS configuration. Configure Passwords on Switches To enable access your switches you must provide the Metacloud Support team with the initial admin password you created. To configure the password: 1. Connect the switch console port to a console terminal. You must disconnect the console port to the Octal cable (2901) to make this connection. 2. Power on the switch. The boot process steps display on the console terminal. 3. On the Abort Auto Provisioning and continue with normal setup?(yes/no)[n]: prompt, enter yes. 4. During the System Admin Account Setup step respond to the following prompts: Enter yes for the Do you want to enforce secure password standard (yes/no) [y]: prompt. Create an admin password at the Enter the password for "admin": prompt. Confirm the newly created admin password on the next prompt. 5. During the Basic System Configuration Dialog VDC: 1 process, enter no for the Would you like to enter the basic configuration dialog (yes/no): prompt. 6. Exit the console. The boot process and admin password set up are now complete. 7. Send the admin password you created during this process to the Metacloud Support Team. 8. Disconnect the switch from the terminal console and reconnect the console port to the Octal cable (2901). 47

48 Note Ensure that you send the admin password you created in the procedure above to the Metacloud Support team so that your installation can be completed. The Metacloud Support Team will change the password during the installation process. 48

49 Step 6 Open Network Ports For the Cisco Metacloud service to communicate with the Cisco Metacloud Operations Center, you must allow connectivity from your network to Cisco Metacloud. Have your network team allow communication on the following ports in your network. Administrative Outbound Connections Port Number Protocol Source IP Destination IP Function NA ICMP ASR External Network Any of the below Connectivity Testing 443 TCP ASR External Network Image Store (VM Images, Firmware) 443 TCP ASR External Network Configuration Management System 443 TCP ASR External Network PKI services 443, 8443 TCP ASR External Network Portal (expected 2017) 80, 443 TCP ASR External Network , , Package Repositories 8443 TCP ASR External Network , Monitoring TCP ASR External Network , Remote Admin Channel TCP ASR External Network , Log Monitoring TCP ASR External Network DB Backups 53 TCP/UDP ASR External Network , , , Zone Transfer Servers 53 TCP/UDP ASR External Network Any IP Name Resolution (if not provided by customer) 123 TCP/UDP ASR External Network Any IP Time Synchronization (if not provided by customer) 1194 TCP ASR External Network Management VPN Tunnel OOB Outbound Connections Port Number Protocol Source IP Destination IP Function NA ICMP ISR2901 WAN IP* , Connectivity Testing 500 UDP ISR2901 WAN IP* , OOB VPN Tunnel 4500 UDP ISR2901 WAN IP* , OOB VPN Tunnel NA ESP ISR2901 WAN IP* , OOB VPN Tunnel 49