Technical guide. Windows HPC server 2016 for LS-DYNA How to setup. Reference system setup - v1.0

Transcription

1 Technical guide Windows HPC server 2016 for LS-DYNA How to setup Reference system setup - v DYNAmore Nordic AB LS-DYNA / LS-PrePost

2 1 Introduction - Running LS-DYNA on Windows HPC cluster Contents 1 Introduction - Running LS-DYNA on Windows HPC cluster Assumptions Nomenclature Net resources Reference Windows HPC cluster HPC Cluster software components Hardware File server and File Share setup mpp/ls-dyna and MS MPI Message Passing Interface Using the HPC-Cluster from a CAE-user perspective License server for LS-DYNA Verify that the system works Alternative hardware and MPI software Copyright DYNAmore Nordic AB 1 LS-DYNA and Windows HPC

3 1 Introduction - Running LS-DYNA on Windows HPC cluster 1 Introduction - Running LS-DYNA on Windows HPC cluster The purpose of this guide is to be an aid when choosing hardware, setting up, and using a Windows HPC cluster for LS-DYNA for explicit and implicit analysis for small to medium sized workgroups of CAE-users, i.e users, and small to medium sized cluster, i.e. 40 to about 500 cores. To this end a reference system is described that is within the above scope. It is assumed in this guide that LS-DYNA is used in a pure Microsoft Windows environment. It is assumed that the reader has good knowledge about Windows, Windows Server 2016, networking, and user administration using Active Directory. The described Windows HPC reference system is of course useful also for other CAE-software, but is not covered here. Please note that there are alternative setups using Windows HPC that may be more suitable depending on the situation. 2 Assumptions An all Windows environment is assumed: Workstations for the CAE-users running Windows Professional or Enterprise (versions 7, 8, or 10). The reference system assumes Windows 10 Professional. Windows user management using Active Directory. HPC Server for LS-DYNA running Microsoft Server 2016 with Microsoft HPC Pack Analysis types Implicit or explicit analysis with smp and mpp/ls-dyna. No analyses with extreme IO-requirements, such as out of core implicit analysis or large metal forming analysis with adaptive mesh generation. If this is the case, then some modification of the software configuration may be needed to reach optimal performance. 3 Nomenclature As far as possible the nomenclature from the documentation of Microsoft Windows HPC-Pack 2016 is used. HPC High Performance Computing MPI Message Passing Interface standardized message passing for parallel computing designed to function on a wide variety of networking hardware, software, and operating systems. mpp (as in mpp/ls-dyna) message passing parallel, an mpp software uses message passing to be able to solve a problem in parallel spanning multiple cores, CPUs and/or Compute nodes DYNAmore Nordic AB 2 LS-DYNA and Windows HPC

4 4 Net resources 4 Net resources Microsoft HPC Pack 2016 documentation at technet.microsoft.com Microsoft HPC Pack 2016 Update 1, available at technet.microsoft.com 5 Reference Windows HPC cluster In Figure 1 below the reference system is shown. Head node with Fileserver Windows HPC Cluster InfiniBand Switch Compute node(s) CAE Workstations Company Active Directory server Gigabit Ethernet Figure 1: Reference system 5.1 HPC Cluster software components Components, software and function: Clients (from which jobs are submitted to the HPC Cluster): Workstations with Windows 10 Pro, LSTC WinSuite, Microsoft HPC-Pack 2016 Update 1 (Client utilities installation, which installs the Job Manager and tools needed by LSTC WinSuite). o Function: On the Client/Workstation the simulation model ( input file ) is created, stored on the File server in a suitable folder and submitted as a simulation job to the Head node. Results are viewed on the Client/Workstation. HPC Head node with Fileserver Microsoft Server 2016 Standard, LS-DYNA License Server, Microsoft HPC-Pack 2016 Update 1 (Head node Installation). o Function: The HPC Head node receives the simulation jobs from the Clients, puts them in a queue and then starts them as soon as sufficient resources are available on the Compute nodes. The results from the simulation jobs are store on the File server DYNAmore Nordic AB 3 LS-DYNA and Windows HPC

5 5 Reference Windows HPC cluster Compute node(s) Microsoft Server 2016 Standard, Microsoft HPC-Pack 2016 Update 1 (Compute node installation). o Function: The Compute nodes read the simulation job data from the File server, run the simulation jobs, and store the result files on the File server. All above servers/workstations and their users are registered in the company Active Directory. The users that can access and start jobs on the HPC Cluster are referred to as the CAE-users. Usually a Group in the Active Directory is created HPCgroup so that all users belonging to this group have appropriate access to the HPC Head node, File server Shares, and Compute node(s). When submitting jobs to the HPC Cluster, they are submitted to the HPC Head node and thus this is the only network server name the CAE-users need to know, e.g. HPCSrv. Notes: Microsoft 2016 HPC pack Update 1 is a monolithic installation file that contains options to install both Clients utilities, Head node, Compute node(s) et c. The reference HPC Cluster system has a file server dedicated to the HPC-Cluster, this is often a good choice as the HPC Cluster can generate a lot of IO and data. It is assumed that all servers and Compute node(s) in the HPC Cluster are attached to the company network are reachable from the Workstations. Other options are possible, e.g. with a private network for the HPC Cluster, but these are not explored here. LSTC WinSuite is a complete installation of LSTC products for Windows 7, 8, 10 computers. It contains LS-DYNA, LS-PrePost, LS-TaSC, LS-OPT, LS-Run, Manuals, Training material etc. LS-Run is a command center and is used e.g. to start and queue LS-DYNA simulations on the local workstation or remote Windows and Linux HPC Clusters. LSTC WinSuite can be remote installed on Workstations. 5.2 Hardware The hardware selection was made in Q Workstations Windows 10, 32 GB RAM, Professional level graphics card for CAD (OpenGL), 1 TB Hard drive, single 4-core Xeon CPU, Gigabit Ethernet card, Full HD display HPC Head node with File server 32 GB RAM, single 8 core Xeon CPU, 6x4 TB SAS Hard drives, RAID 10 controller card for SAS, Gigabit Ethernet card, Hard drives use RAID 10 for performance. Compute node(s) 2018 DYNAmore Nordic AB 4 LS-DYNA and Windows HPC

6 6 mpp/ls-dyna and MS MPI Message Passing Interface 192 GB RAM, 2x1 TB Hard drives (RAID 1), dual Xeon SP 6148 CPUs (20 cores/cpu), Mellanox ConnectX-3 InfiniBand cards, Gigabit Ethernet card InfiniBand Switch used for MPI Notes Mellanox SX port infiniband switch, QSFP+ connector cables for connection of Switch and Compute nodes For implicit analysis, more memory per node may be needed on the Compute node(s). Instead of Hard drives on the Compute Nodes, using SSDs may provide significantly better performance for certain types of analyses, but not for the types assumed here, see Section 2. In the system, InfiniBand is only used for MPI-communication between the LS-DYNA-processes. All other network traffic (SMB, TCP, UDP et c) is carried by the Gigabit Ethernet network. 5.3 File server and File Share setup The following file shares are available on the Head node/file server \\HPCSrv : \\HPCSrv\projects with subfolders for each CAE-user or project to store input files and simulation results. This share is also used by the Compute nodes during the simulations. This share should be accessible by all CAE-Users on their Workstations as well as on the compute nodes, else they will not be able to use the HPC Cluster \\HPCSrv\software contains subfolders o lsdyna: the LS-DYNA executables: both smp and mpp-versions in double and single precision. Microsoft HPC pack 2016 uses MS-MPI, thus mpp/ls-dyna binaries should be installed whose label include the phrase msmpi, e.g. lsdyna_mpp_s_r920_winx64_ifort131_msmpi.exe. o installation: Windows HPC Pack 2016 Update 1 installation file and LSTC WinSuite installation file. This is for convenience when adding new Clients (Workstations). 6 mpp/ls-dyna and MS MPI Message Passing Interface The parallel software mpp/ls-dyna works by splitting up the simulation model in N equally sized pieces and each piece is then handled by a separate mpp/ls-dyna process. For efficiency only one mpp/ls-dyna process should by run on each physical core (thus it is generally recommended to turn of hyperthreading or alternatively use methods such as pinning described in the MPI documentation). The different mpp/ls-dyna processes need to communicate to solver the total prob DYNAmore Nordic AB 5 LS-DYNA and Windows HPC

7 7 Using the HPC-Cluster from a CAE-user perspective lem and to that end MPI is used. As quick communication is crucial for performance, the MPI communication is made using a fast network such as InfiniBand between Compute nodes and methods such as user space memory copy between processes on the same Compute node. In the reference HPC Cluster, the MPI implementation is used that is included with Microsoft HPC Pack 2016: MS-MPI (Microsoft MPI). The experience is that it is safe to upgrade the MS MPI version on the HPC Cluster as newer versions, with e.g. bug fixes, are released by Microsoft. For more information on MPI and MS MPI, see technet.microsoft.com (search for MS MPI). 7 Using the HPC-Cluster from a CAE-user perspective The CAE-users uses the system in the following manner: Notes: 1. Create the LS-DYNA input file, e.g. main.k, using a preprocessor, e.g. using LS-PrePost or other pre-processor. 2. Save the input file in the users project folder on the Head Node/File Server \\HPCSrv\project\John\Project34\Crashsim12\main.k 3. Open LS-Run (part of LSTC WinSuite) select the appropriate LS-DYNA binary, solution options, and the input file \\HPCSrv\project\John\Project34\Crashsim12\main.k. Submit the simulation to the HPCSrv HPC Cluster queue. See also Figure 2 for an illustration. As soon as resources, i.e. Compute nodes, are available the simulation will be started. 4. The progress of the job can be monitored using LS-Run, when finished the results are available for postprocessing in \\HPCSrv\project\John\Project34\Crashsim12 (default location). Multiple jobs can be started, they are then run as soon as Compute node resources are available. More information on how to start an LS-DYNA analysis using LS-Run is available under the Help-menu in LS-Run. Jobs submitted by LS-Run to a Windows HPC Cluster are submitted to the standard Windows HPC server queue and thus cooperate/co-exist with other jobs submitted to the HPC Cluster, e.g. from other CAE-software DYNAmore Nordic AB 6 LS-DYNA and Windows HPC

8 8 License server for LS-DYNA Figure 2: LS-Run 8 License server for LS-DYNA The network license is installed by running LS-Run on the license server, e.g. the Head node. All Compute node must be able to access the license server and vice versa. The license is installed from the License option in LS-Run by importing the license file. To be able to use LS-DYNA on the cluster it is necessary to specify License type = network and Server hostname = LicenseServerHostname in the License menu in LS-Run on the computer submitting the job. 9 Verify that the system works To verify that the system works run a benchmark model on 1 or more cores. Suitable benchmarks for e.g. explicit analysis are included with LSTC WinSuite or can be found at Explicit simulation models are recommended as first benchmark. Check that the runtimes are reduced as the number of cores are increased, including over multiple cores. For large explicit analysis (more than about elements per used core, near linear reduction of runtimes are expected when scaling from 1 to more than 100 cores). For larger models LS-DYNA can effectively use up to more than cores in a single analysis simulation. To verify node connectivity and the performance of the MPI/InfiniBand-interconnect one can also use the mpipingpong test included in HPC Pack 2016 (see HPC Pack documentation at technet.microsoft.com). 10 Alternative hardware and MPI software There are alternatives to the hardware and software components selected for the HPC Cluster described here. A noncomplete list is: MPI: Intel MPI, IBM Platform MPI CPU: AMD EPYC Network for MPI: Intel Omni-Path 2018 DYNAmore Nordic AB 7 LS-DYNA and Windows HPC

9 11 Copyright 11 Copyright All trademarks, service marks, trade names, product names and logos appearing in his document are the property of their respective owners, including in some instances Livermore Software Technology Corporation (LSTC) DYNAmore Nordic AB 8 LS-DYNA and Windows HPC