DESCRIPTION The Auburn University College of Engineering Computational Fluid Dynamics Cluster is built using Dell M1000E Blade Chassis Server Platform. The Cluster will consist of (4) M1000E Blade Chassis Fat Nodes, utilizing (16) M610 half-height Intel dual socket quad-core Nehalem 2.80 GHz processors, 24GB RAM, (2) 160GB SATA drives (RAID0/1) and single Operating System image (CentOS). Each M610 blade server will be connected internally to the chassis via a Mellanox Quad Data Rate (QDR) InfiniBand switch for creation of the ScaleMP vsmp Foundation solution stack. Each M1000E Fat Node will be interconnected via 10 GbE Ethernet using M6220 blade switch stacking modules for parallel clustering using OpenMPI. Each M1000E Fat Node also has independent 10GbE Ethernet connectivity to the Brocade Turboiron 24X Core LAN Switch for login access to each Fat Node, if desired, and consistent NFS mounting of the external persistent storage. This Solution Stack will provide each node with 128 cores @ 2.80 GHz Nehalem, 384GB RAM and 5.1TB RAW internal storage with a total of 512 cores @ 2.80 GHz, 1.536TB shared memory RAM, and 20.48TB RAW internal storage. HPCC Theoretical Performance calculated @ 5.735 teraflops with memory bandwidth @ 35 GB/s. Persistent external storage will be provided using (1) Dell PowerVault MD3000 SAS connected 15TB SATA RAW capacity via 10 GbE NFS connectivity to each fat node via R610 NFS Server. Battery backup for the cluster will be provided by (4) APC Smart-UPS SURT8000RMXLT6U UPS (6400 Watts / 8000 VA, Input 208V / Output 208V, Interface Port DB-9 RS-232, RJ-45 10/100 Base-T, Smart-Slot, Extended runtime model, Rack Height 6 U). This UPS will provide single phase 208V battery-backed power to each component in the cluster architecture. Installation Services will consist of Dell Americas Merge Center Rack and Stack Service for complete installation and cabling of all hardware components into (2) Dell 4210 Racks. The Racks will be delivered fully populated and ready to power up. Further Onsite Installation will be provided by Dell Professional HPC Services Group to install and configure the ScaleMP vsmp Foundation software, Operating System, Ethernet switches, NFS mounting to persistent storage and InfiniBand switch setup. Additionally, (5) days of ScaleMP Enhanced Performance Engineering Phone Support Services are included for Application Optimization and Performance Tuning for initial Installation (application tuning, library selection, etc). About ScaleMP ScaleMP is the leader in virtualization for high-end computing, providing higher performance and lower Total Cost of Ownership (TCO). The innovative Versatile SMP (vsmp) architecture aggregates multiple x86 systems into a single virtual x86 system, delivering an industry-standard, high-end symmetric multiprocessor (SMP) computer. Using software to replace custom hardware and components, ScaleMP offers a new, revolutionary computing paradigm. vsmp Foundation aggregates up to 16 x86 systems to create a single system with 4 to 32 processors (128 cores) and up to 4 TB of shared memory. Solutions based on vsmp Foundation provide superior performance at prices significantly lower than traditional SMPs and comparable to managed x86 clusters. These solutions conserve the advantages of the traditional SMP operating model (ease of installation and management as well as large memory) while leveraging the broad x86-based applications portfolio. For cluster customers, they can continue to run their parallel applications at the same levels of performance, without the complexities of managing multiple OS s, application provisioning, cluster file system, cluster interconnect management. In addition, the I/O architecture is significantly simplified and less expensive (particularly for fast scratch storage). Cluster customers also get a very versatile computing platform, being able to run large memory and high core count jobs in addition to their parallel workloads and the programming model is also more flexible, being able to run MPI, OpenMP, multithreaded and legacy code at optimum performance for High Performance (HPC) environments.
PowerEdge M1000e Blade chassis Overview POWEREDGE M1000E The Dell PowerEdge M1000e Modular Blade Enclosure is a breakthrough in enterprise server architecture. Built from the ground up to combat datacenter sprawl and IT complexity, the M1000e delivers one of the most energy efficient, flexible, and manageable blade server product on the market. Flexible and scalable, the M1000e is designed to support future generations of blade technologies regardless of processor/chipset architecture. The M1000e is optimized for use with Dell s M600 and M605 blade servers. capacity Holds up to sixteen server blades power 3 (non-redundant) or 6 (redundant) 2360 watt hot plug power supplies Based on Dell s Energy Smart Technologies, the M1000e Power Supplies deliver greater levels of efficiency, even at very low levels of utilization Redundant Power Supplies support 3+3 (AC redundancy), 3+1 (Power Supply Redundancy), or 3 + 0 (non-redundant) modes System supports new Dynamic Power Supply Engagement functionality, which (if enabled) puts lightly loaded power supplies into standby mode, driving up the utilization and thus the efficiency on the active supplies Power Supplies Require 200+ volt AC input Dell offers a wide range of Power Distribution options for the M1000e with 20A, 30A, 60A single phase or 30A three phase options cooling M1000e Chassis comes standard with 9 hot pluggable, redundant fan modules Based on Energy Smart Technologies, M1000e fans are a breakthrough in power and cooling efficiency. The fans deliver low power consumption, but also use next generation low flow technology to ensure the lowest possible amount of fresh air is consumed to cool the enclosure ENCLOSURE i/o modules Up to six total I/O modules for three redundant fabrics, featuring Ethernet FlexIO Switches providing on demand stacking and uplink scalability. PowerConnect M6220 Ethernet Switch Cisco Catalyst Blade Switch M 3032 Cisco Catalyst Blade Switch M 3130G Cisco Catalyst Blade Switch M 3130X Dell Ethernet Pass-Through Module - 16 x 10/100/1000Mb copper RJ45 connections Brocade M4424 SAN I/O Module 4Gb Fibre Channel Pass-Through Module 16 x FC 1/2/4Gb SFP ports management Dell OpenManage System Management Integration into 3rd party management solutions via Dell s Preferred Partner Program 1 (standard) or optional 2nd (redundant) Chassis Management Controller(s) (CMC) for remote management Single point of connection from management network to idrac on each of the blades and the management interfaces on the integrated I/O Modules Optional Integrated Avocent keyboard, video and mouse (ikvm) switch ports Integrated KVM switch ports (both analog and digital options) 9-pin serial and RJ-45 for systems management chassis Form Factor: 10U Height: 44cm (17.3in.) Width: 44.7cm (17.6in.) Depth: 75.4cm (29.7in.)
DELL SINGLE VIRTUAL SYSTEM FOR HPC The Dell PowerEdge M1000e, coupled with the vsmp Foundation aggregation technology, provides a supercomputing system with a single Operating System and large shared memory running on a high core count of Intel Xeon processors. This solution provides over 1.5 TFLOP, 1TB shared memory and over 90 GB/s of memory bandwidth *. The Dell Single Virtual System for HPC provides an alternative to traditional SMP systems as well as simplification for cluster deployments by offering a fat-node cluster infrastructure: Unlike traditional SMPs that are costly and expose end users to vendor lock in, Dell s Single Virtual System allows customers to run OpenMP, MPI and large memory jobs with up to 128 cores and 1TB RAM on standard x86 hardware. This solution significantly simplifies the operational tasks associated with clusters, such as multiple Operating Systems, application installation per node, and storage management. The solution consolidates a typical 256 node cluster into a 16 node fat node cluster, greatly reducing the number of managed entities. In addition, it allows greater flexibility for users and programmers, as MPI, OpenMP, and large memory applications can share the same infrastructure. In addition, Storage infrastructure is simplified as well, with better I/O performance by aggregating the local drives for scratch storage. The innovative ScaleMP Versatile SMP (vsmp) architecture aggregates multiple x86 systems into a single virtual x86 system, delivering an industry-standard, high-end symmetric multiprocessor (SMP) computer. ScaleMP is using software to replace custom hardware and components, to offer a new, revolutionary computing paradigm. The vsmp architecture utilizes the Dell PowerEdge M1000e and does not require any custom parts. Its key value is the utilization of software to provide the chipset services that are otherwise required in creating traditional multi-processor systems. vsmp Foundation provides cache coherency, shared I/O and the system interfaces (BIOS, ACPI), which are required by the OS. The vsmp architecture is implemented in a completely transparent manner; no additional device drivers are required and no modifications to the OS or the applications are necessary. Requirements vsmp Foundation uses the following components: Multiple industry standard x86 systems, such as the Dell PowerEdge M600 (note that processor speeds and amount of memory across systems do not have to be the same), InfiniBand infrastructure in the form of HCA's and switch, vsmp Foundation Devices (USB) are used to load the vsmp Foundation software to the blade. These USB flash devices are plugged into each blade system board and are pre loaded with vsmp Foundation. One System Once loaded into the memory of each of the systems, vsmp Foundation aggregates the compute, memory and I/O capabilities of each system and presents a unified virtual system to both the Operating System and the applications running above the OS. vsmp Foundation uses a software-interception engine in the form of a Virtual Machine Monitor (VMM) to provide a uniform execution environment. vsmp Foundation also creates the required BIOS and ACPI environment to provide the OS (and the software stack above the OS) a coherent image of a single system. The Versatile SMP (vsmp) Architecture Coherent Memory vsmp Foundation maintains cache coherency between the individual boards using multiple advanced coherency algorithms. These complex algorithms operate concurrently on a per-block basis, based on realtime memory activity access patterns. vsmp Foundation leverages board local-memory together with best-of-breed caching algorithms to minimize the effect of interconnect latencies. Shared I/O vsmp Foundation aggregates I/O resources across all boards into a unified PCI hierarchy and presents them as a common pool of I/O resources to the OS and the application. The OS is able to utilize all the system storage and networking controllers to provide high-i/o system capabilities. Versatile System vsmp Foundation aggregates system boards with different processor speeds, varied memory amounts or dissimilar I/O devices. This is a unique capability among x86 shared memory systems: A homogenous system with up to 32 sockets (128 cores) and 1 TB RAM, delivering more than 1.5 TFLOPS should be used for compute intensive applications. For applications that are memory intensive and not computeintensive, an imbalanced configuration using both high speed and low speed processors can be architected. With such an imbalanced configuration, vsmp Foundation will aggregate only the high speed processors, while not exposing the low speed processors to the Operating System. Such a configuration allows reduced costs and power consumption, providing large memory and top system performance. Similarly, the customer can mix and match I/O expansion options to fit application needs, making it possible to deliver the industry's most versatile and flexible high-end x86 systems. Coupled with the price/performance attributes, solutions based on vsmp Foundation provide customers the best value for their money. ScaleMP vsmp Foundation has been deployed at over 100 sites around the world, including global 2000 companies.
Simplified Cluster Using the Dell Single Virtual System simplifies cluster infrastructure. The traditional cluster deployment will have four components: Compute nodes. Head node. Fast clustered storage for scratch (low density, high bandwidth). Main storage for home directories, applications, and data (high density, low bandwidth). Assuming such a cluster with 288 nodes, the Dell Single Virtual System, will aggregate it to 24 fan-node cluster, where each node will have 128 cores, 512 GB RAM, and will remove the need for the clustered storage. In essence from management standpoint the solution will have: 24 node cluster, where each node will have 128 cores, 512 GB RAM, and over 9TB internal storage. Each node will include ScaleMP s vsmp Foundation, RHEL Linux installed on Dell PowerEdge M1000e with 16 M600 nodes in Single Virtual System configuration. Each of the 24 nodes will be connected to two GigE switches. The GigE network will also include 6 login nodes and the Research Storage (connected with 10 GigE). Performance The system s capability as measured with SPECcpu_base_rate2000, shows 70%-80% speedup going to 128 cores (results measured on a 16 board configuration with 32 quad-core 2.33 GHz Xeon processors (E5345).
Furthermore, the solution shows linear memory bandwidth scalability. The results were measured on a system with 16 boards and 32 quad-core 2.33 GHz Xeon processors (E5345). The system has a total of 1024 GB shared memory RAM. Please note that this system has slow memory on the last two system boards (cores 112-128), due to a lack of availability of faster memory modules at the time of the benchmark.
Dell OpenManage for HPCC Dell has a systems management strategy that focuses on industry standard technologies by developing alliances with leading systems management vendors. This strategy allows Dell to provide a synergistic approach to the development of tools that integrate into the Enterprise frameworks provided by these vendors. This protects current infrastructure investment while providing a complimentary set of management tools. OpenManage provides the necessary tools for comprehensive hardware monitoring of your HPC Compute Cluster. OpenManage Assistants Enterprise hardware management is a key component for deploying systems to the data center. Dell OpenManage recognizes this need by offering standards based tools that address the areas of configuration, event, asset and problem management. Dell Server Assistant: Provides streamlined out of box setup of your Dell Server. By utilizing patented factory installation technology, Microsoft, LINUX or Novell operating systems can be installed in minutes minimizing resource deployment cycle times. IT Assistant: Is a web based management console that includes features such as system discovery, event management, status polling, asset reporting, remote system configuration, paging, and e-mail event notification for instrumented Dell servers. IT Assistant can be used as a standalone console or can be easily integrated into your Enterprise management framework. Powerful capabilities include launch of RAID management and remote server management utilities. Dell OpenManage Server Administrator: A highly-secure web tool for managing individual servers providing server status, asset and inventory information, audit trails and the ability to perform diagnostics and updates from virtually anywhere at anytime. Scripting capabilities utilizing the full command line interface allows for changes to multiple nodes in the cluster. Dell Out of Band Management: Hardware based operating system independent solutions for delivering out-of-band and in-band server management. This is particularly important for remote sites without local IT staff to monitor the health of servers. An administrator can take control of a remote server from a central location, diagnose the problem, and in many cases fix the problem thereby avoiding the time and cost of dispatching an administrator. OpenManage Connections Dell OpenManage Connections is software that snaps into the major enterprise frameworks, Computer Associates Unicenter TNG, Tivoli, and Hewlett-Packard OpenView, and allows customers to effectively manage Dell systems. From these enterprise frameworks, customers can monitor critical software applications such as those that enable e-commerce. This provides Auburn University College of Engineering with a complete solution for integration of Dell systems into the existing management framework. Dell OpenManage addresses current and future needs for systems management. Dell works closely with our customers to continuously improve and develop tools for managing systems. This relationship enables Auburn University College of Engineering to provide this direction and feedback for future OpenManage products and tools.