One Dell Way Round Rock, Texas 78682 www.dell.com DELL ENTERPRISE WHITEPAPER MECHANICAL AND THERMAL CONSIDERATIONS FOR RACK DEPLOYMENT OF THE DELL POWEREDGE M-SERIES Dominick Lovicott Enterprise Thermal Engineering
THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. For more information, contact Dell. Information in this document is subject to change without notice. 2 P age
System Fan Exhaust System Fan Exhaust System Fan Exhaust 1 INTRODUCTION Cooling efficiency of data center deployed IT hardware can be a key contributor to data center operating expenditures. The hardware and systems management integrated into the PowerEdge M1000e Modular Blade Enclosure and associated M-Series blades have been designed to achieve high cooling efficiency. There are still opportunities for customers, however, to implement best practices in racking their servers and arranging cable management in their data center to help ensure that the M1000e is operating at its highest efficiency. This paper will discuss cable management, rack airflow management, and label placement best practices for installation of the PowerEdge M-Series. For more information regarding the thermal design of the PowerEdge M-Series and associated M-Series blades please refer to the Thermal Design of the Dell PowerEdge M-Series White paper. <http://www.dell.com/downloads/global/products/pedge/en/pedge_mseries_thermal_design.pdf> 2 CABLE MANAGEMENT The PowerEdge M1000e has a wide variety of connector and cabling options for the various modular components. The component layout for the Dell PowerEdge M1000e Modular Enclosure is shown in Figure 1. Installation of an M1000e chassis includes the connection and management of the following cables: Power Distribution Units (PDU) Inlet Power Cables: connect the power source to the PDUs PDU Jumper Power Cables: connect the PDUs to the Power Supply Units (PSUs) I/O Cables: connect the I/O modules to the customer network Chassis Management Controller (CMC) and Keyboard Video Mouse (KVM) Cables Addressing the multitude of cabling scenarios available from the wide array of PDU, PSU and I/O options in the M1000e is beyond the scope of this discussion. However, the following sections will define best practices that can be applied to a majority of cabling configurations. The discussion will include cable routing zones, strain relief, providing access to modules, trimming, and rack related issues. For a comprehensive guide to cable configurations please see the interactive M1000e Rack and Cable Advisor. <http://support.dell.com/support/topics/global.aspx/support/enterprise_support/en/cabling?c=us&l=en&s=gen> CMC KVM CMC IO Modules IO Modules Power Supply Units Figure 1: Location of modular components on the back of the M1000e Modular Enclosure 3 P age
2.1 PDU Mounting and Power Cables PDU inlet power cables connect PDUs to the main power source. Jumper power cables connect the PDUs to the PSUs. Dell recommends that the PDU installation and power cable routing be the first step when configuring a rack for the M1000e. Please refer to the PDU installation guide and M1000e PDU Selection and Installation White paper for information on PDU options, selection, and installation. <http://www.dell.com/downloads/global/products/pedge/en/pe_m1000e_selection_whitepaper.pdf> 2.2 I/O Cables Dell recommends routing I/O data cables vertically from the M1000e I/O modules for most applications. Routing cables vertically from the I/O modules prevents cables from obstructing the fan exhaust and thereby minimizes impact on airflow and cooling. Figure 2, shows the routing of data cables from two M1000e chassis downward and exiting the bottom of the rack. In cases where an M1000e has a small number of data cables, it may be possible to attach the cables to the sides of the rack and route them vertically in the 0-U space. Additionally the 0-U space may be used for the routing of the inlet power cables. Figure 2: Routing of data cables vertically downward from I/O bays 2.3 Cable Enumerator The M1000e Rail Kit is provided with a set of cable enumerators that can be used to organize and label I/O cable bundles. Each cable enumerator has 8 slots into which a cable can be secured. Each slot has three different cable securing shapes for holding cables of different shapes and diameters. One side of the enumerator is marked with the numbers 1 and 8, and the other side is marked with numbers 9 and 16. Figure 3 shows the two sides of the cable enumerator. 4 P age
Figure 3: Cable enumerator showing shape of cable slots The cable enumerators are designed to improve serviceability by preserving the relative position of the cables in a cable bundle when the cables are disconnected. Cables from the first eight I/O ports of an I/O module would be secured into the slots on the side of an enumerator labeled 1 and 8. The remaining cables from the I/O module would be secured into the slots on the side of an enumerator labeled 9 and 16. Figure 4, shows a set of enumerators secured in place on cable bundles. 2.4 Access to Components Figure 4: Cable enumerators secured to cable bundles The rear modules of the M1000e (power supplies, I/O modules, fan modules, etc) are hot swappable. By routing cables vertically from the I/O bays, access is provided to the fan modules and most of the power supplies. However, cabling in this zone can interfere with access to some of the power supplies and I/O modules (if multiple M1000e chassis are deployed in the rack). By securing the I/O cables to the Strain Relief Bar with the hook and loop fastener straps, the cable bundles can be slid sideways along 5 P age
the Strain Relief Bar. In this manner, access can be gained to the components that are behind the cable bundles, as shown in figure 5. Figure 5: Power supply covered by I/O cables (left), power supply exposed by sliding cables sideways on strain relief bar (center), access granted to power supply (right) 2.5 Rear Door Exit Area and Alternate Door Kit I/O cables can exit the rack at either the top or the bottom. The standard Dell PowerEdge 4210 (42-U) and PowerEdge 2410 (24-U) rack enclosures have an exit area at the bottom of the right rear door for downward directed cables, and two circular cutouts in the roof of the rack for upward directed cables. If the openings in the Dell 42-U rack are insufficient for routing the cables in an M1000e deployment, an alternative rear door kit may be recommended. 2.5.1 Standard Door Kit The standard rear door assemblies of the Dell PowerEdge 4210 and 2410 racks have two cable exit location options: One 60 mm x 250 mm cable exit located at the bottom of the right rear door for cables that exit downward Two 110 mm diameter cable exit holes in the roof of the racks for cables that exit upward Exit areas are shown in Figure 6. If the quantity of exiting I/O cable is light to moderate, then these openings may be sufficient. Figure 6: Exit areas in bottom right door (left) and rack roof (right) for standard door kits of Dell PowerEdge 4210 and 2410 racks 6 P age
2.5.2 Alternate Door Kit Dell offers an alternate rear door kit for the 4210 rack that has a 60 mm x 500 mm cable exit across the bottom of both rear doors, and a removable panel at the top of each door to provide an exit area of 150mm x 500 mm. Exit areas of the alternate door kit are shown in figures 7 and 8. Figure 7: Exit areas at bottom of alternate door kit for PowerEdge 4210 rack enclosure Figure 8: Top of alternate door kit for PowerEdge 4210 rack enclosure showing removable panels in place (left) and removed (right) In general, the alternate door kit is recommended for customers using the Dell PowerEdge 4210 rack who are: Installing three or more M1000e chassis with high number of Ethernet or Infiniband I/O cables that must be routed out of the rack either upward or downward. Using 60 Amp single phase power and are routing the power inlet cables upward; the circular cutouts in the roof of the rack are not large enough to allow the 60 Amp plug to pass through. An alternate door kit is not required or offered for the PowerEdge 2410 rack. The cable routing requirements for the 2410 rack are not as demanding because this rack is only 24-U tall and can hold a maximum of two M1000e chassis. 2.6 Strain Relief Bars The Strain Relief Bar provided in the M1000e rail kit can be used to both hold the weight of the I/O cables and assist in the routing of I/O and jumper power cables. The bar has two mounting methods. In the default position, the Strain Relief Bar is attached by screws to the threaded nut in the rear of the rails. This positions the Stain Relief Bar in the vertical space underneath the I/O module bay and above the power supplies. This is the best position for providing strain relief to data cables that are being routed downward towards the bottom of the rack, as shown in figure 9. Alternatively, the Strain Relief 7 P age
Bar can be mounted in any open hole in the rack by using the cage nuts that are provided with the M1000e Rail Kits. When cables are routed upward, the strain relief should be utilized above the I/O bay. Dell has two recommended placement options for the Strain Relief Bar to accomplish this. First, the Strain Relief Bar can be placed in an open hole above the chassis, as shown in figure 10. If multiple M1000e are deployed in the rack, then the Strain Relief Bar in its default position between the PSU and I/O Bay can provide the support for the cables originating from the lower chassis. Strain relief may be needed above the top chassis. For this, the strain relief bar from the bottom chassis can be mounted in an open space above the top chassis with the provided cage nuts. IO Cables secured to Strain Relief Bar (highlighted in box) Figure 9: I/O cables routed downward from chassis and attached to the Strain Relief Bar IO Cables secured to Strain Relief Bar (highlighted in box) Figure 10: I/O cables routed upward from chassis and attached to the Strain Relief Bar Both data cables and power jumper cables can be secured to the Strain Relief Bar by use of the hook and loop fastener strips that are provided in the M1000e Rail Kit. See the Rack Installation Guide for details on how to use these straps. 2.7 Airflow Exit It is important to consider the cabling s effect on airflow and cooling of a system. Cables can obstruct the airflow path and therefore reduce the cooling efficiency. Figure 11, illustrates the extreme case of a full rack of blade chassis configured with redundant Ethernet I/O modules. The cable bundles in this case are routing in-line with the I/O modules to reduce obstruction to the system fans. Dell s vertical fan 8 P age
layout was designed to minimize cabling impact on airflow. In contrast, horizontally distributed fans can compromise airflow due to cable routing. As shown in Figure 12, areas behind fan exhausts (in red) should not have cables or cable clusters obstructing the airflow. Areas (in green) behind the I/O modules, CMC and KVM are recommended for cable clusters and should not obstruct the airflow path. Cables running over the power supply exhausts may be unavoidable. In this case it is important to utilize the Strain Relief Bars to maintain a minimum distance from the exhaust to allow airflow to exit the PSU. Figure 11: Dell recommended cable management for a full rack of M1000e blade enclosures Figure 12: Areas to avoid placement of cables (red) and suggested areas for cable clusters (green) that will help minimize airflow blockage on the back of a Dell PowerEdge M1000e Modular Blade chassis 3 HOT AIR RECIRCULATION Racked servers in a datacenter environment can be affected by hot air recirculation within the rack. Hot air recirculation occurs when exhaust air is pulled from the hot aisle through recirculation paths in the rack into the inlet of the server. This increase in inlet temperature can induce higher fan speeds and fan power consumption. The overall ability to cool the system can also be reduced due to the higher inlet temperature. 9 P age
Hot air recirculation can have a unique effect on modular blade servers. The hot recirculated air can be drawn into a server module at the outer edge of the enclosure. Due to the isolated airflow from one blade to the next, these outer blades may only draw in this recirculated hot air, while the blades closer to the center of the enclosure will draw fresh cold-aisle air. The outer blades with the higher ambient temperature may then drive higher fan speeds, reducing the cooling efficiency of the deployment. Implementing rack recirculation closeouts can help eliminate this recirculation and increase the cooling efficiency of the modular blade server. 3.1 0-U Blanking Panels For currently shipping PowerEdge 4210 and 2410 rack enclosures, hot air recirculation is addressed with standard 0-U blanking panels. Dell offers a 0-U blanking panel as a retrofit for legacy racks that were not equipped with this feature. This rack feature is specifically designed to eliminate recirculation of hot air through the 0-U space. Figure 13, shows a Dell M1000e blade chassis deployed with and without 0-U blanking panels. Figure 14, shows a picture of the 0-U blanking panel. The Dell 0-U blanking panel part numbers are as follows: 42U Dell rack GN414 24U Dell rack - MT813 0-U Hot Air Recirculation Path 0-U Blanking Panel Figure 13: Dell M1000e rack deployed with (left), and without (right) 0-U recirculation paths Figure 14: Dell 0-U Blanking Panel 10 P age
3.2 U Blanking panels Servers can be deployed in racks that are not fully populated. As with the optional 0-U recirculation paths, the depopulated U slots are a clear path for hot air recirculation. Dell offers U blanking panels to help reduce the amount of hot air recirculation (see figure 14 below). Blade chassis are not immune to this type of recirculation. To help achieve the highest amount of efficiency of the Dell M1000e Blade servers it is recommended that all non-populated U slots have blanking panels installed. The Dell U blanking panel part numbers are as follows: 1U - JJ599 2U - MJ347 3U - KJ451 6U - HJ569 Depopulated U Space Recirculation Path Populated U Blanking Panel Figure 15: Empty U space (above) and blanking panel (below) above a rack deployed Dell blade chassis 4 LABEL/TAG PLACEMENT Labels and tags are commonly used to track or identify hardware in a rack. The labels should be placed on areas of the chassis that do not contain perforation or air inlets. Labels that cover air inlets can dramatically reduce the amount of airflow the fans can pull through the system. Reducing the amount of airflow reduces the cooling capability of the system and can cause the fans to draw more power to compensate for airflow loss. Label placement is particularly important for the Dell PowerEdge M-series blades servers. The chassis was specifically designed to help minimize airflow resistance of the server. Placement of a label on any of the perforation will limit the ability to cool as well as lead to an increase in fan power. Locations of the various inlet perforations for the Dell M1000e enclosure and M-series blade servers are shown in 11 P age
Figure 16. Labels should not be placed in these areas. Labels can be placed on the blade handle and chassis bezel without impacting airflow, as shown in Figure 17. IO Module Inlet Ventilation Blade Inlet Ventilation Power Supply Inlet Ventilation Figure 16: Dell PowerEdge M1000e modular blade chassis illustrating the inlet ventilation for the blade, power supply and I/O module subsystems Blade Handle Chassis Bezel Figure 17: Dell PowerEdge M1000e modular blade chassis illustrating the recommended labeling areas. 12 P age
5 CONCLUSION The Dell PowerEdge M1000e modular blade system was designed to operate over a wide range of operating conditions. While the hardware and systems management were designed for energy-efficient operation, the deployment of the modular M1000e in a rack/datacenter environment requires some considerations to take full advantage of the hardware s energy efficient design characteristics. Implementation of the suggested best practices cable management will provide the least airflow restriction for the fans and provide for simplified cable management. The rack 0-U closeout feature and empty U blanking panels of the Dell rack addresses hot air recirculation effects experienced by racked servers. These combined effects will help maximize the return on investment of deploying the Dell PowerEdge M-Series in the datacenters of today and tomorrow. 13 P age