HPE Performance Optimized Datacenter (POD) Networking Guide Abstract This document provides networking guidance for the various HPE PODs. Part Number: 663145-003R November 2015 Edition: 4
Copyright 2011, 2015 Hewlett Packard Enterprise Development LP The information contained herein is subject to change without notice. The only warranties for Hewlett Packard Enterprise products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. Hewlett Packard Enterprise shall not be liable for technical or editorial errors or omissions contained herein.
Contents Overview... 4 Networking introduction... 4 Network best practices... 4 Planning considerations... 4 Network hardware not requiring customization in the HPE POD rack... 5 Network hardware requiring customization in HPE POD infrastructure... 6 Addressing airflow with HPE Network Switchbox Enclosure Kit (SKU AT053A)... 8 HPE network switchbox enclosure... 8 Cisco MDS 9500 multilayer director... 10 Cisco Catalyst 6500-E... 11 Other hardware considerations... 11 Cabling... 12 POD cable management... 12 POD 240a cable management... 12 POD 40c and POD 20c cable management... 13 Integrating additional IT at the customer site... 14 Network access... 17 Network entry... 17 Network demarcation... 19 Frequently asked questions... 21 Documentation feedback... 22 Contents 3
Overview Networking introduction This guide is a primer and does not cover all networking best practices for the PODs. The customer has responsibility for providing the airflow direction of the network devices that are intended to be installed, especially if the devices are customer-furnished equipment. This must be managed by the account team and the network solution architect. Devices that require hot aisle servicing or maintenance must be placed in racks that allow full rear access. This document includes hardware that is optimized for the HPE POD 240a, HPE POD 40c, and HPE POD 20c airflow cooling and hardware that will need special consideration. Network best practices The following best practices are established for all POD network configurations: Ensure that customer intent is finalized and documented. Assign a network specialist as early as possible (from the customer side) to create and finalize the POD network design and configuration to include point-to-point specification. Refer to this guide to understand which network devices require customization and, if possible, install network devices recommended by Hewlett Packard Enterprise. Use the network switch enclosure option, SKU AT053A, for those devices that utilize side-to-side air-flow cooling, and use side panels, SKU AT960A, as required. Understand the hardware size specifications of the equipment being placed inside the racks of the POD to ensure interference limitations are taken into consideration. Arrange to have the external telecom box sized according to the customer's requirement. Hewlett Packard Enterprise does not provide this box. Plan accordingly for high cable counts aggregating to core network racks. Planning considerations The POD can support various Tier x interconnects via an External Telecom Panel or direct termination at the rack level via patch panels. An External Telecom Panel enables each customer to adjust to fit their specific site requirements. The customer should plan to either install an External Telecom Panel as part of their site design, or plan another solution for demarcation in to the POD. In planning the correct solution, the customer must consider: What Telecom panel rating is required? What is the size of the Telecom panel (H x W x D)? Where is the entry point for the Telecom panel? Which hardware components are required for termination? What is the hardware placement within the Telecom panel? Overview 4
How many fiber/copper devices are being terminated? What are the length of service loops within the Telecom panel? Is hardware designed to be used only in racks? Is this single or multi-homed? Are the network specifications available? Are site and network drawings available? If you require Hewelett Packard Enterprise to provide an External Telecom Panel with termination hardware, specifications must be clearly identified by the customer to Hewelett Packard Enterprise, and will involve lead time and cost incremental to the basic solution. Network hardware not requiring customization in the HPE POD rack NOTE: For current information, see the appropriate product website. For more information, see the following: ProCurve 6600 Series HPE 5800AF Series HPE 12500 Series HPE S9508E-V Router HPE A6616 Router HPE 8808-V Router Arista 7048T, 7050 switches Cisco Nexus 2000 Series Fabric Extender o If the Cisco Nexus 2000 must be positioned with ports flush to the rear of the POD rack, custom mounting brackets will be required. Cisco Nexus 5000 Series Cisco Nexus 7010 Cisco 4948 E(-F) Cisco 6509-V-E Quanta LB4M QLogic 12800-360 Additional information: Several of these previously listed devices have the potential of supporting up to 864 ports, which can result in a large volume of cables, and must be placed in a rack that has full rear-door access to facilitate the maintenance or the replacement of hardware. The recessing of switches to be flush with the back of the rack will require the prevention of air flow recirculation to eliminate the mixing of hot aisle and IT section air. Overview 5
Network hardware requiring customization in HPE POD infrastructure CAUTION: The IT section and hot aisle air separation begins on the front side of the rack at the RETMA rails. Any hardware with air inlets on the side will intake hot aisle air and have the potential to overheat. Devices that work best inside the POD utilize a front-to-back airflow model. Any devices designed with side-to-side airflow need special consideration. The network solution architect of the account team should understand hardware air-flow requirements prior to requesting a formal legal quote from the respective order management team. The following network hardware does require a customized air-flow solution: HPE MSR30-xx series router Cisco 3700 series Cisco 3750-E The following figure shows the airflow for the 3750-E. Cisco 3750-X The following figure shows the airflow for the 3750-X, which includes air inlets on the port side and are not found in the 3750-E model. Cisco 4948 Cisco Catalyst 6500-E Overview 6
Cisco 3800 series router Cisco 2350 series switch Cisco MDS 9124 multilayer fabric switch Cisco 9500 series multilayer director F5 BIG IP 6800 series Digi CM Procurve 2600, 2900, 5400 Avocent Cyclades ACS48 Switches that are not itemized above must be evaluated by the POD network SA to determine if a customized solution is required. All applicable hardware and technical specifications for the devices must be provided to the POD network specialist in the initial engagement stage in order to mitigate last-minute customizations. All customizations impact price, schedule, and resources. NOTE: Some hardware components are labeled as having front-to-back airflow. However, the front of the component, as identified by the vendor, might need to be reoriented in order to prevent overheating and enable proper cooling, as the following image illustrates. Overview 7
Addressing airflow with HPE Network Switchbox Enclosure Kit (SKU AT053A) HPE network switchbox enclosure Devices that are not optimized for front-to-back airflow require an Network Switchbox Enclosure Kit (SKU AT053A) which has been designed to alleviate overheating. The following figure is a diagram of the AT053A kit. During the design phase, the additional RU space required for the top and bottom of the enclosure must be considered. Both the base and the top of the enclosure use 1RU, which leaves 3RU open within the enclosure. Even though there are 4RU open spaces, an empty RU is required between devices for air flow and cable management. The following images show the side air flow. Addressing airflow with HPE Network Switchbox Enclosure Kit (SKU AT053A) 8
AT053A Rack with two side to side air flow switches Item Component Description 1 Air baffle top The top of the air baffle that allows side air flow. 2 Air baffle bottom The bottom of the air baffle that allows side air flow. AT053A Rack with three side to side air flow switches Item Component Description 1 Air baffle top The top of the air baffle that allows side air flow. 2 Air baffle bottom The bottom of the air baffle that allows side air flow. For efficient airflow, maintain a minimum space of 1RU between devices when stacking more than two devices together. IMPORTANT: If installing a single 1RU switch in a Network Switch Enclosure, you must leave one empty RU either above or below the switch to allow for air circulation. Addressing airflow with HPE Network Switchbox Enclosure Kit (SKU AT053A) 9
Devices that are installed in the Network Switch Enclosure and require more than 3RU of space will need additional side panels from the (Network Switch Enclosure Sidepanel Kit (SKU AT960A). The rack openings on the back side of the AT053A Network Switch Enclosure must be filled with blanking panels. Cisco MDS 9500 multilayer director This device requires 3RU minimum space below and above the chassis to provide air flow. The device also requires a Network Switchbox Enclosure Kit (SKU AT053A) as depicted in the previous illustration, as well as the Network Switch Enclosure Sidepanel Kit side panels (SKU AT960A) on the inlet side only. Do not install side panels on the exhaust side of the device. To enable adequate airflow along the top, bottom, and side of the air-inlet side of the chassis, the MDS 9500 must be placed on a support rail and secured to the rack with custom mounting brackets, allowing the chassis to be recessed in the rack. Care must be taken to also maintain hot and cold air-flow separation. The 9500 series switches have complex air flow and will require additional parts and customization. The account SA is responsible for engaging directly with the POD network SA for additional guidance. Rack-mount support brackets from Cisco are included with this device and are required for installation. These brackets are not removable. Power cables for this device must be secured at a level above the chassis to enable fan tray removal. All network cables must be secured to the side of the rack. Addressing airflow with HPE Network Switchbox Enclosure Kit (SKU AT053A) 10
Cisco Catalyst 6500-E This device requires 2RU minimum space below and 3RU above the chassis. The device also requires a Network Switchbox Enclosure Kit (SKU AT053A) and an Network Switch Enclosure Side panel Kit (SKU AT960A). The Cisco Catalyst 6500-E does not require a side panel kit on the exhaust side of the chassis. To enable adequate airflow along the top, bottom, and side of the air-inlet side of the chassis, the 6500-E must be placed on a support rail and secured to the rack with custom mounting brackets, allowing the chassis to be recessed in the rack. Care must be taken to also maintain hot and cold air-flow separation. The 6500 series switches have complex air flow and will require additional parts and customization for successful operation. The account SA is responsible for engaging directly with the POD network SA for additional guidance. Other hardware considerations To maintain hot and cold air seperation while the TFT is in the open position, you must place TFT and KVM devices inside an Network Switchbox Enclosure Kit (SKU AT053A). NOTE: Each customer has individual site requirements, and requires a customized Statement of Work from Hewlett Packard Enterprise Technical Services. Addressing airflow with HPE Network Switchbox Enclosure Kit (SKU AT053A) 11
Cabling POD cable management The customer must create and finalize the POD network design and configuration to include point-to-point cabling specification prior to the receipt of the purchase order. The customer network solution architect will work with the Hewlett Packard Enterprise account team network solution architect to finalize the POD network strategy. The following figure illustrates the rack dimensions that must be considered. Item Component Description 1 RETMA rail 29-inches from the front to the back 2 Rack 35-inches from the front to the back POD 240a cable management The POD 240a utilizes a wire-tray basket system suspended above the racks and positioned towards the rear of the rack for cable management. Feature Specification Location Cable tray over rack aisle Cable tray connecting IT sections across the hot aisle Cable tray across the hot aisle connecting the IT sections Clearances Must maintain a minimum of 2 in between the top of the racks and the bottom of the cable tray Must maintain a minimum of 91.9 in between base of cable tray and the floor Dimensions of cable 16 in wide x 2 in deep x 10 ft long tray sections over racks Cabling 12
Feature Specification Quantity of cable tray sections over racks Dimensions of cable tray sections across the hot aisle Quantity of cable tray sections across the hot aisle connecting the IT sections Four per rack aisle Eight total per POD 240a 8 in wide x 2 in deep x 98 in long Three per POD 240a The following image illustrates the location of the cable tray in the hot aisle. The inset image illustrates how the cable tray hangs from the ceiling of the hot aisle. Top view shown POD 40c and POD 20c cable management For large quantities of cables in the hot aisle, order one of the following: POD 40c; SKU AT965A, POD 40ft Cable Cover Kit POD 20c; SKU AT966A, POD 20ft Cable Cover Kit Feature Specification Location Quantity of floor cable tray sections in hot aisle Dimensions of floor cable tray sections across the hot aisle Floor tray in hot aisle Six per POD 40c: 4 long, 2 short Three per POD 20c: 1 long, 2 short 9 in wide x 3.75 in deep Cabling 13
The following images illustrate the cable tray. Integrating additional IT at the customer site When installing additional hardware in the POD on location, follow these guidelines: Maintain hot and cold air separation within the racks. PDUs in the POD are not equipped with L6-20 connectors. POD 240a In the POD 240a, avoid rack placement for network switches in racks 1 and 22. Cabling 14
Top view shown POD 40c Item Component Description 1 Dock side The side of the POD 240a that provides direct access to IT section A, IT section B, and the hot aisle. 2 Truck/power side The side of the POD 240a that provides access to all main power components. In the POD 40c, avoid rack placement for network switches in racks 1, 2, 5, 6, 7, 11, 12, 16, 17, 18, 21 and 22. POD 40c G2 In the POD 40c G2, avoid rack placement for network switches in racks 4, 8, 12, 16, and 20. NOTE: In the following illustration, the racks that are shaded provide clear access to the back of the rack via the rear doors. Cabling 15
POD 20c In the POD 20c, avoid rack placement for network switches in racks 1, 5, 6, and 10. Cabling 16
Network access Network entry POD 240a The POD 240a has six network access points at the rear of the POD 240a above the cargo doors. NOTE: The connection portal location and configuration might look different, depending on the POD design. Rear cargo end view shown POD 40c and POD 20c The POD 40c and POD 20c have six network access penetrations in vertical alignment on the rear cargo door. Network access 17
Rear cargo end view shown NOTE: The cargo doors might look different, depending on the POD design. The POD 40c G2 also has six network access penetrations in vertical alignment on the water connect side of the POD. Water connection end view shown The POD 40c G2 also has a single network access penetration on the water connect side of the POD. Network access 18
Water connection end view shown Network demarcation Network configurations vary; therefore, the customer is responsible for installing external telecom panel access. The POD does not ship with an external telecom panel to serve as a demarcation point of entry into the POD. Therefore, customers must specify the appropriate external telecom panel. An external telecom panel is not required to extend network cables from outside the POD. A patch panel installed in a rack to distribute connectivity can serve as a demarcation point. However, if the customer decides to install an external telecom panel, the following figure shows the location of a panel. Network access 19
POD external telecom panel example The external telecom panel model can be applied to any POD version as long as the size, rating, cable count capacity, and mounting location requirements are met. Network access 20
Frequently asked questions 1. Can a customer use mounting ear brackets on the equipment in the racks in the POD? When installing through the front of the rack (IT section), yes. If equipment must be installed from the back of the rack (hot aisle), Hewlett Packard Enterprise recommends rail kits instead of mounting ear brackets for devices to be used in the racks. The customer must ensure that rail kits are available for the third-party equipment that is ordered from their respective vendor. Hewlett Packard Enterprise does not recommend using mounting ear brackets in the racks on devices installed from the back of the rack. 2. Is plenum-rated cable needed in the POD? Hewlett Packard Enterprise recommends that you confirm with local agencies to understand permit requirements. 3. How do I plan for network connectivity from the outside world in to the POD and what should I take into consideration? The POD can support various Tier X interconnects via an external telecom panel or direct termination at the rack level to patch panels. Since each customer requires a customized solution, the customer can normally procure and install an external telecom panel or patch panels for direct termination to fit the specific site requirements. The customer should plan to install an external telecom panel or direct termination as part of their site design for demarcation to the POD. In planning the correct solution, the customer must consider: o What is the telecom panel rating requirement? o What is the size of telecom panel (H x W x D)? o What is the entry point for the telecom panel? o What hardware components are required for termination? o What is the placement of hardware within telecom panel? o What are the fiber/copper quantities being terminated? o What are local codes for network cabling requirements (plenum, non-plenum, or LSZH rated)? o What are the lengths of the service loops required within the telecom panel? o Is the hardware being specified in the telecom panel designed to be used only in racks? o Is connectivity single or multi-homed? o Are network specifications available for the required hardware? o Are drawings available of the design? Frequently asked questions 21
Documentation feedback Hewlett Packard Enterprise is committed to providing documentation that meets your needs. To help us improve the documentation, send any errors, suggestions, or comments to Documentation Feedback (mailto:docsfeedback@hpe.com). When submitting your feedback, include the document title, part number, edition, and publication date located on the front cover of the document. For online help content, include the product name, product version, help edition, and publication date located on the legal notices page. Documentation feedback 22