QLogic TrueScale InfiniBand and Teraflop Simulations

Transcription

1 WHITE Paper QLogic TrueScale InfiniBand and Teraflop Simulations For ANSYS Mechanical v12 High Performance Interconnect for ANSYS Computer Aided Engineering Solutions Executive Summary Today s challenging economic climate and globally competitive markets make lowering manufacturing costs more critical than ever. Whether you re an aerospace engineer designing a new wing for the latest commercial jet or an automotive engineer simulating multi-car crashes for the next generation of hybrids, the ability to run simulations faster will increase productivity, speed time-to-market, and improve your company s bottom line. High Performance Computing (HPC) is a valuable tool in assisting manufacturers with improving product design and product quality, while reducing cost. QLogic TrueScale InfiniBand along with the latest in multi-core processor technology makes teraflop computing available to organizations at affordable prices. HPC clusters that utilize the QLogic InfiniBand TrueScale interconnect can exploit ANSYS Mechanical v12 improvements specifically developed for HPC environments. ANSYS Mechanical v12 has improved the Dsparse solver, memory handling, load balancing and I/O processing giving it the ability to scale across an increased number of nodes and processors delivering performance in excess of a teraflop. ANSYS v12 improved features, along with affordable HPC clusters, enable organizations to make use of more complex models characterized by millions of degrees-of-freedom (MDoF). QLogic, working in concert with ANSYS, tested ANSYS v12 on range of configurations and interconnects. The results of this testing demonstrated that ANSYS v12 can run effectively on a high core count cluster in a multi-user/multi-job environment without impacting individual job performance. This testing demonstrated the ability to expand the scope and scale of ANSYS Mechanical on HPC cluster implementations.

2 Business Value of HPC Cluster Improve Time-to-Solutions by Completing Simulations Faster When running optimization programs that refine a design, reducing the time-to-solution is one of the keys to success. Interconnecting your cluster computing network using QLogic TrueScale solutions can reduce this critical metric the end result is faster time-tomarket and higher product quality levels. Cluster configurations are 90 times faster than a single CPU workstation/server Up to 300% faster when using TrueScale InfiniBand versus Gigabit Ethernet Maximize Designer/Engineering Productivity through Simultaneous Use of the Cluster Optimizing the productivity of an engineering team is a key to remaining competitive in these challenging economic times. HPC clusters making use of TrueScale InfiniBand improves productivity by delivering faster results for each engineer through support of concurrent simulations. Concurrent simulations execute within 2 to 5% of their single run times. Shared resources deliver performance that makes the entire engineering team more effective and productive. Better Designs by Running Larger Data Sets without Dramatic Increases in Run Time QLogic TrueScale InfiniBand solutions enable companies to effectively scale cluster configurations, run larger data sets and expand simulation complexities in computer-aided design. Results for large models are delivered faster. A 3 MDoF model completes in less than eleven minutes of wall clock time. Reduced Product Development Costs A TrueScale-based HPC cluster enables organizations to increase design complexity and perform more detailed simulations to verify and validate designs. All of this can drive cost and time out of the entire development process. Reduce the need for physical prototyping by leveraging more complex models in a compressed timeframe. This approach can reduce or eliminate much of the need to do physical prototyping. Reduced Costs of Computing Infrastructure By leveraging HPC cluster compute and storage resources, industrial engineering customers utilizing QLogic s high performance TrueScale InfiniBand solutions can reduce the costs and complexities of deploying and operating HPC clusters. TrueScale-based clusters reduce the number of servers needed to deliver the equivalent performance level of an Ethernet cluster. In addition to the server savings there are incremental ongoing savings in power and cooling as a result of using fewer servers to achieve the same system throughput. TrueScale InfiniBand protects your cluster investment by delivering real performance when incremental computation resources are added. ANSYS v12 benchmark tests show that gigabit Ethernet only provides marginal performance gains as a cluster is scaled. Gigabit Ethernet passes a point-of-diminishing returns for clusters larger than 16-nodes/128-cores. QLogic High Performance Cluster Technology QLogic offers a comprehensive end-to-end InfiniBand product portfolio that includes Multi-Protocol Fabric Directors, Edge Fabric Switches, Host Channel Adapters (HCAs) and a complete software suite to install, operate, and maintain your high performance interconnect fabric. QLogic offers the most comprehensive and flexible interconnect fabric solutions on the market. Applications needing 12-to-864 InfiniBand ports can be supported via a single chassis. Multi-chassis fabrics supporting thousands of host nodes can be constructed to meet the most demanding compute cluster requirements. This offering, combined with the industry s only Fabric Management tools enabling an administrator to install and boot a fabric in minutes, helps to satisfy the growing demand for highperformance computational clusters and grids. Benchmark Test The configuration tested consisted of 32 compute nodes and one NFS server node. Each compute node was comprised of an AMD server with dual 2.5 Ghz Quad core Barcelona CPUs and 16 GB memory. The HPC cluster had a total of 256-cores and 512GB of memory. TrueScale InfiniBand provided a 20Gbps interconnect between each node for MPI and file system I/O traffic. A TCP Gigabit Ethernet interconnect was tested for comparison purposes. ANSYS v12.0 using HPMPI was used to execute the BIGBM-7 benchmark test. This test is a 3 MDoF model using the direct sparse solver of a solid 95 elements problem. This presents a worst case scenario for a direct solver problem. The following characterizes the details of the BIGBM-7 simulation test: NSG-WP09001 D A 2

3 2 Gbytes size of assembled stiffness matrix 13 Gbytes memory required memory to to run out-of-core on a single core 93 Gbytes size of matrix factor. Despite being a large memory requirement for one node, it represents only 20% of available memory on the test cluster. 500 Gbytes I/O required to enable application to factor once and solve one load. The off-server I/O was eliminated by using cluster s total available memory Value of QLogic TrueScale InfiniBand over TCP/IP - Gigabit Ethernet The improvements in ANSYS v12 enhance its ability to leverage the latest in today s HPC computing technologies. The choice of interconnect for the HPC cluster becomes an important consideration for maximizing the new features and benefits of ANSYS v12. InfiniBand provides the following advantages over Ethernet: Performance (low latency/high throughput) Price/Performance Scalability Concurrent Simulations Stability Scalable Bandwidth (enables network consolidation) Price/Performance: A TrueScale InfiniBand cluster requires significantly less server resources to complete an equivalent simulation when a comparison is made to a gigabit Ethernet based cluster. It takes 4-servers/32- cores utilizing TrueScale to produce equivalent results of an 8-server/64-core cluster inter-connected with gigabit Ethernet. Achieving real saving around computational resources and ongoing operational expenses, such as power and cooling, makes a notable improvement in the bottom line. Scalability: The value of TrueScale InfiniBand is evident with larger simulations. The test (BIGBM-7) is a simulation that involves 3 MDoF. This scaled linearly across the TrueScale InfiniBand Interconnect, cutting the time-to-solution nearly in half sequentially from 16 to 256 cores. The same tests executed via gigabit Ethernet reached its point-ofdiminishing return with 16-nodes/128-cores. Concurrent Jobs: TrueScale-based clusters provides the flexibility and performance needed to support a dynamic set of concurrent simulations requirements. Virtually partitioning the cluster into smaller virtual configurations is one way to effectively create multiple, low-node-count clusters, each capable of supporting concurrent simulations. This is a great way for multiple engineers to simultaneously share the cluster resources, thereby making a whole engineering team more productive. Alternately, a single engineer has the option to run multiple variations on a design giving, providing the option to quickly identify the best design alternative. Performance: When time-to-solution (TTS) is critical, then the value of InfiniBand based cluster becomes evident. QLogic TrueScale InfiniBand delivers time-to-solution improvements from 35 to 300% depending upon the size and scale of the cluster. TrueScale InfiniBand begins to show its performance advantage with as few as 16-cores and continues to increase as the cluster size is expanded. For example, TrueScale delivers 300% better time-to-solution over Ethernet for a 32-node/256-core cluster. NSG-WP09001 D A 3

4 Another approach is to submit multiple larger jobs across a high node count cluster. For example, a 32-node/256-core cluster equipped with a total of 512GB of available RAM can be used by two engineers to effectively and efficiently support two similar jobs requiring 160GB of memory space simultaneously with negligible job interaction. This scenario was tested and proven to work efficiently. A hybrid concurrent simulations option is a combination of the first and second methods. Multiple engineers have the ability to run their small jobs across smaller segmented cluster nodes as described in the first method, while another engineer was running a larger job(s) across the entire cluster. This combination method produces excellent results as long as the simulations do not exceed the maximum memory and/or the maximum cores available by the cluster. Stability: The TrueScale InfiniBand was very stable without any tuning across all runs of the BIGBM-7 benchmark including the concurrent simulations tests with no sever or network issues. The concurrent simulation runs on the Ethernet network did not complete even after repeated attempts due to unresolved issues. Bandwidth: The bandwidth capabilities of InfiniBand up to 40 gigabits/ second can be used to architect a solution that consolidates data networking, clustering communications and storage access over single fabric. This has the benefit of significantly lowering overall power, real estate and management overhead in data centers. InfiniBand s enhanced Quality of Service (QoS) capabilities support running and managing multiple applications, workloads and traffic classes. As an example, during the BIGBM-7 runs the InfiniBand 20 Gbps fabric was used to concurrently run the MPI inter-processor communication and NFS traffic. Ansys Corporation ANSYS, Inc. develops and globally markets engineering simulation software and technologies widely used by engineers and designers across a broad spectrum of industries. The company focuses on the development of open and flexible solutions that enable users to analyze designs directly on the desktop, providing a common platform for fast, efficient, and cost-conscious product development, from design concept to final-stage testing and validation. The company and its global network of channel partners provide sales, support, and training for customers. QLogic Corporation QLogic is a leading supplier of high performance clustering and storage networking solutions, which include host adapters and InfiniBand switches that are used by major corporations and government facilities. The company delivers a broad and diverse portfolio of products that includes InfiniBand switches and InfiniBand host channel adapters for the fast growing, high performance computing market. The company is also a leading supplier of Fibre Channel HBAs, blade server embedded Fibre Channel switches, Fibre Channel stackable switches, iscsi HBAs, and iscsi routers. QLogic products are delivered to small-to-medium businesses and large enterprises around the world via its channel partner community. QLogic products are also powering solutions from leading companies like Cisco, Dell, EMC, Hitachi Data Systems, HP, IBM, Network Appliance and Sun Microsystems. QLogic is a member of the S&P 500 Index. NSG-WP09001 D A 4

5 Disclaimer Reasonable efforts have been made to ensure the validity and accuracy of these performance tests. QLogic Corporation is not liable for any error in this published or the results thereof. Variation in results may be a result of change in configuration or in the environment. QLogic specifically disclaims any warranty, expressed or implied, relating to the test results and their accuracy, analysis, completeness or quality. Corporate Headquarters QLogic Corporation Aliso Viejo Parkway Aliso Viejo, CA Europe Headquarters QLogic (UK) LTD. Surrey Technology Centre 40 Occam Road Guildford Surrey GU2 7YG UK +44 (0) QLogic Corporation. Specifications are subject to change without notice. All rights reserved worldwide. QLogic, the QLogic logo and TrueScale are trademarks or registered trademarks of QLogic Corporation. ANSYS and ANSYS Mechanical are trademarks or registered trademarks of ANSYS, Inc. AMD and Opteron are trademarks or registered trademarks of Advanced Micro Devices, Inc. HP is a trademark of the Hewlett-Packard Company. InfiniBand is a trademark and service mark of the InfiniBand Trade Association. All other brand and product names are trademarks or registered trademarks of their respective owners. Information supplied by QLogic Corporation is believed to be accurate and reliable. QLogic Corporation assumes no responsibility for any errors in this brochure. QLogic Corporation reserves the right, without notice, to make changes in product design or specifications. NSG-WP09001 D A 5