MSC Nastran Explicit Nonlinear (SOL 700) on Advanced SGI Architectures

Transcription

1 MSC Nastran Explicit Nonlinear (SOL 700) on Advanced SGI Architectures Presented By: Dr. Olivier Schreiber, Application Engineering, SGI Walter Schrauwen, Senior Engineer, Finite Element Development, MSC Software April 10, 2013

2 Agenda SGI, Introduction SGI and MSC Software SGI Solutions for CAE SGI architectures MSC Nastran SMP, DMP technologies Benchmarks Effect of processor frequency, Turbo, Hyperthreading, Memory, Interconnect, Filesystem, running on subset of cores, scaling Summary 2

3 SGI's focus SGI: The Trusted Leader in Technical Computing Business Applications Business Computing Big Data Technical Applications Technical Computing Redundancy Workload Optimized Scale & Speed 3

4 SGI CAE Customers 4

5 SGI Partnership with MSC Software MSC Software builds on all SGI supported platforms, software stacks: x86_64, LP64, ILP64 NUMA, multi-core, DMP, SMP Linux Distribution Popular Choices: RHEL, SLES IntelMPI, PlatformMPI, OpenMPI md release started to include SGI Math kernels licensed by MSC Software. 15.2mil DOF SOL101 automotive industry engine model shows performance boost on x86_64 (results, courtesy MSC Software) 5

6 SGI Solution - Mixed Workflow Environment SGI UV Minimizes time to solution. Ultimate flexibility running shared and distributed memory applications in one system using a single OS instance Ideal for Scale-up and Scale-Out SGI UV Mixed Workflow SGI Rackable SGI Rackable or ICE X cluster Cost-effective Solution & performance leader for solvers or InfiniBand or GigE Fabric SGI CXFS, Lustre, Gluster, Panasas SGI ICE X 6

7 SGI Rackable: Highly configurable cluster Suggested: Intel Xeon 8-core 2.6GHz E5-2670, 6-core 2.9GHz E SGI C2108 RP2 Summit (head node) SGI C2112 4RP4 Steelhead (compute nodes) 500 GB SATA mirror Raid 1 on head node 2 striped 500 GB SATA local scratch (compute nodes) 4/8 GB Memory/core (2GB per million elements) IB QDR or FDR interconnect SGI InfiniteStorage 5000 (IS5000) for scratch/storage Altair PBSPro Batch Scheduler v11 SLES or RHEL, SGI Performance Suite, Accelerate 7 20u, 42u racks available

8 SGI ICE X: Highly scalable, integrated, cable-free cluster Suggested: Intel Xeon 8-core 2.6GHz E5-2670, 6-core 2.9GHz E Up to 72 nodes (144 sockets, 1152 cores per rack) Up to two 2.5 SATA drives for local swap/scratch 4/8 GB Memory/core (2GB per million elements) Integrated IB FDR interconnect Hypercube/Fat Tree Single or Dual-plane network topology (Multi-rail for separation MPI & I/O traffic, or message splitting) SGI InfiniteStorage 5000 (IS5000) for scratch/storage SLES or RHEL, SGI Performance Suite, Accelerate Altair PBSPro Batch Scheduler v11 SGI ICE X 8

9 SGI UV2: Highly scalable, Shared Memory, latest generation x86 system Suggestions: Air/water cooled 19 42U rack H:79.5 xw:31.3 xd: sockets (512 cores), 16 TB shared memory/rack Xeon 8 core 2.7GHz E5-4650, 6 core 2.9GHz E Graphics Card, GP-GPU and Intel Phi support SGI NUMALink 6 Interconnect SGI InfiniteStorage 5000 (IS5000) for scratch/storage SLES or RHEL SGI Performance Suite, Accelerate Altair PBSPro Batch Scheduler v11 with CPUSET MOM 9 SGI UV 2000

10 Customer Example: Challenge: Hamilton Sundstrand wants low administration/ maintenance system for fast turn-around times in commercial, regional, corporate and military aircraft integrated component systems development. Solution: UV cores Intel Xeon GHz, 2TB RAM, IS4600 Storage/fast scratch, Altair PBSPro batch scheduling Applications: MSC Nastran, LS-DYNA, ANSYS, STAR-CCM+ Impact: Significant less system administration, more ease of use, IO scalability improvement, leverage of RAM across system to run large MSC Nastran jobs. Frees aircraft engineers to focus on core competencies. No additional IT staff investment. 10

11 SGI Cyclone Completing The Need For Customers Traditional Data Center Modular Data Center Cloud Scale & Control Modular & Mobile On Demand SGI Customers can move freely between environments 11 11

12 SGI Cyclone Solution Customer A Customer B Customer C Customer D Customer E SGI Firewall Customer Login SGI UV: Minimize time to solution. Ultimate flexibility running shared and distributed memory applications in one system using single OS instance. Customer Login SGI UV Customer Login SGI ICE Customer Login SGI Rackable Customer Login SGI ICE X or Rackable cluster: Cost-effective solution & performance leader for all solver runs Analysis types: Bandwidth performance, large statics, dynamics, NVH, Multi-physics. Crash/impact, CFD 12

13 Industry Context: New hardware technologies that extend computational performance: MSC Nastran's computational technologies to exploit them: Shared Memory Parallel (SMP) Distributed Memory Parallel (DMP) Hybrid mode (combination of above two paradigms) 13

14 MSC NASTRAN version: Shared Memory Parallelism (SMP)(80's)(v68): OpenMP Application Programming Interface around computational loops. Distributed Memory Parallelism (DMP)('90s)(v2001): MPI API around physical domain decomposition (more granularity). Paradigms map on two different system hardware levels: inter-node or cluster parallelism (memory local to each node) DMP only. intra-node or multi-core parallelism (memory shared by all cores of each node)--both SMP and DMP Submittal procedures should ensure: Placement of processes and threads across cores and nodes Control of process memory allocation to stay within node capacity Use of adequate scratch files across nodes or network 14

15 MSC NASTRAN execution control Submittal line control options: sdirectory=pathname for scratch files directories mem=explicit memory allocation for in-core processing smp=number of OpenMP threads dmp=number of distributed processes hosts={host1:...:hostn} 15

16 Scratch space filesystem topology NAS (Network Attached Storage) DAS (Direct Attached Storage) Root drive (/tmp) Memory filesystem (/dev/shm) Linux filesystem buffering (DAS) 16

17 Hardware Nomenclature node=host=blade(=chassis) Identified by MAC address/ip address. Comprises 2 sockets (or more) for: Processors with four (quad-core), six (hexa-core), eight or twelve cores each CPU=core LS-DYNA 17 UC 2012

18 Benchmarks Models have been selected for increasing size and changes in solver, material, site and phenomenon modeled-- indicated by their names. 18

19 Small model EulerSnowBoxCompression' Soil compressed by structure to test snow yield model. 240K el. 19

20 Medium model RoeAirBunkerBlast, 880k el. Explosive is ignited outside bunker, blast wave hits bunker, bunker fails, explosive gas enters bunker. Flow between Euler domains is modeled. Uses second order Roe solver 20

21 Large model RoeAirMineBlast 2 x 800k el. Blast wave expands below car model with opening, gas flows inside car. Flow between Euler domains is simulated. Uses the second order Roe solver 21

22 Largest model EulerSoilVehicleBlast 2.2m el. Simulates soil covered JWL explosive below a vehicle. Uses biased Euler mesh. Blast location volume has especially fine Euler mesh to capture blast in soil 22

23 CPU frequency effect on performance Nominal ratio: 1.11, effect only

24 Turbo Boost effect on performance Noticeable gains 24

25 Running with more nodes Dataset dependent, number of processes is relevant parameter Scaling up to 256 processes Job rating falls above 128 processes 25

26 Interconnect effect on performance Dataset dependent 26

27 Giga bit Ethernet interconnect No scalability beyond 64 processes or 4 nodes Job rating levels off at 64 processes Job rating continues to increase up to 128 processes 27

28 Leaving cores unused across nodes Dataset dependent Job rating increases at constant number of processes when increasing number of nodes. Job rating decreases when spreading fixed number of processes across more nodes. 28

29 Leaving cores unused across nodes, benefit 64 processes run across 4 nodes, small FDR 64 processes run across 16 nodes, small FDR 29

30 Leaving cores unused across nodes, no benefit 64 processes run across 4 nodes, medium FDR 64 processes run across 16 nodes, medium FDR 30

31 Anomaly 2 processes on 2 nodes take 60% more time than on 1 node 31

32 Anomaly 2 processes run on 1 node, medium FDR 2 processes run on 2 nodes, medium FDR 32

33 Results observations recapitulation: Good scaling for DMP beyond one node/16 cores. HyperThreading not beneficial due to communication costs Infiniband can be twice as fast as GigE interconnect Frequency and Turbo matter with no I/O bottleneck if DAS or RAM filesystem used (not NAS) More RAM accommodates Linux buffer cache All effects depend on datasets Upgrading a single system attribute like CPU frequency, interconnect, number of cores per node, RAM speed brings diminishing returns if others unchanged. Trades can be made based on metrics such as dataset turnaround times or throughput, acquisition, licensing, energy, facilities, maintenance costs to minimize. 33

34 MSC NASTRAN offers integrated shared and distributed parallel computational features applicable to SGI advanced computer architectures, such as Rackable, ICE X and UV with Intel Xeon Processors, IB QDR, FDR, NUMAlink 6, Interconnects hardware, Linux, SGI Performance Suite, Accelerate software to allow greatly enhanced workflow optimization possibilities. Thank you Olivier Schreiber oliviers@sgi.com Tony DeVarco tdevarco@sgi.com Walter Schrauwen walter.schrauwen@mscsoftware.com 34