From Sand Dynamics to Tank Dynamics Using Advanced Computing to Accelerate Innovation and Improve Designs in Mechanical Engineering

Transcription

1 From Sand Dynamics to Tank Dynamics Using Advanced Computing to Accelerate Innovation and Improve Designs in Mechanical Engineering Associate Professor Dan Negrut NVIDIA CUDA Fellow Simulation-Based Engineering Lab Wisconsin Applied Computing Center Department of Mechanical Engineering University of Wisconsin Madison NVIDIA GTC May 16,

2 Acknowledgements Collaborators: Alessandro Tasora University of Parma, Italy Mihai Anitescu Argonne National Lab, USA Lab Students: Aaron Bartholomew Makarand Datar Toby Heyn Naresh Khude Justin Madsen Hammad Mazhar Dan Melanz Spencer O'Rourke Arman Pazouki Andrew Seidl Rebecca Shotwell Financial support National Science Foundation, Career Award Army Research Office (ARO) US Army TARDEC FunctionBay, S. Korea NVIDIA Caterpillar MSC.Software Advanced Micro Devices (AMD) 2

3 Multibody Dynamics: What is it? [simulated in commercial package] Purpose: understand/optimize performance before building prototype 3

4 Multibody Dynamics: What is it? [simulated in commercial package] 4

5 Classical Computational Dynamics, Constrained Equations of Motion 5

6 Tomorrow s Multibody Dynamics Problem Applications transitioning from multi-body to many-body dynamics Bodies interacting through friction/contact/impact Large translations/large deformations (Everything is nonlinear) Bodies are compliant, might undergo large deformations Bodies might interact with fluid (FSI) Simulate multi-phsyics using at least one GPU. Preferably, several GPUs. Better yet, combine also with multiple cores 6

7 CAE: Looking Ahead How is the Rover moving along on a slope with granular material? What wheel geometry is more effective? 7

8 Frictional Contact Simulation [Commercial Solution] Model Parameters: Spheres: 60 mm diameter and mass kg Forces: smoothing with stiffness of 1E5, force exponent of 2.2, damping coefficient of 10.0, and a penetration depth of 0.1 Simulation length: 3 seconds 8

9 Simulating large engineering problems remains a challenge 9

10 Lab s Research Heterogeneous Cluster 10

11 Lab s Research Heterogeneous Computing Cluster More than 25,000 GPU scalar processors Can manage about 75,000 GPU parallel threads at full capacity More than 1000 CPU cores Mellanox Infiniband Interconnect, 40Gb/sec About 0.7 TB of RAM More than 20 Tflops DP The issues is not hardware availability. Rather, it is producing modeling and solution techniques that can leverage this hardware 11

12 Heterogeneous Computing Template (HCT): A Research-Grade Software Infrastructure for Large Scale Computational Dynamics Simulation Goal, lab s research effort: shape up the future of physics-based simulation Develop a Heterogeneous Computing Template (HCT) that leverages emerging hardware architectures and suitable algorithms to solve open engineering problems Targeted emerging hardware architectures : Clusters of CPUs and GPUs (accelerators) More than 100 CPU cores, tens of GPU cards, tens of thousands of GPU cores Focus on open engineering problems Vehicle mobility, granular dynamics, soil modeling, tire/terrain modeling, FSI, etc. 12

13 HCT: Five Major Components Computational Dynamics requires Advanced modeling techniques Strong algorithmic (applied math) support Proximity computation Domain decomposition & Inter-domain data exchange Post-processing (visualization) HCT represents the library support, the associated API, and the embedded tools that support this five component abstraction 13

14 Multi-Physics targeted Computational Dynamics requires Advanced modeling techniques Strong algorithmic (applied math) support Proximity computation Domain decomposition & Inter-domain data exchange Post-processing (visualization) 14

15 HCT: Support for Advanced Modeling Techniques Modeling: what does it mean? The process of formulating a set of governing differential equations that captures the multi-physics associated with the engineering problem of interest Modeling decisions are consequential Good modeling places you at an advantage when it comes to simulating hard problems 15

16 Multi-Body Dynamics w/ DVI 16

17 Mixing 50,000 M&Ms on the GPU 17

18 Dealing with Compliant Bodies 18

19 Compliant Body Dynamics [w/ friction and contact] 19

20 Modeling, Dynamics of Systems with Compliant Elements Finite Element node coordinates e k é T T T k k k æ ö æ ö æ ö k T ( ), r, r, r = r ç x ç y è ø è ø èç z ø êë T ù úû The global position vector of an arbitrary point on the beam centerline is r(x; e) = S(x)e The shape function matrix for this element is defined as S = [ S I S I S I S I ] Î = - x + x 2 = x - x + x = 3x - 2 x, 4 = (- x + x ) s 1 3 2, s l( 2 ) s s l x = x / l 20

21 Element Mass Matrix and Element Elastic Force M is the symmetric consistent mass matrix of element defined as M A - c/s area, - density, l - element length l 0 T = Aò r S Sdx The vector of the element elastic forces is determined using the strain energy as e11 Q s T l T l T æ e ö 11 k ( e11) + ( k) ò ç ò ç è ø 0 0 æ U ö æ ö = = EA dx EI dx çè e ø e è e ø k - axial strain - magnitude of curvature vector 21

22 Compliant Body Dynamics [w/ friction and contact] 22

23 Deformable Bodies with Constraints 23

24 Fluid-Solid Interaction Example Separating living/dead cells 24

25 Multi-Physics: Multi-Body Dynamics & Fluid Dynamics v e e d v g g v d f v f c a vc b v a v b CG.. 25

26 SPH Simulation of Flow Over Granular Bed DEM-based frictional contact Ellipsoidal particles 1e6 total SPH particles, 1e3 ellipsoids 5/24/2012 University of Wisconsin - Simulation Based Engineering Lab 26

27 SPH Simulation of Dense Particulate Flow DEM-based frictional contact 1e6 total SPH particles, 1e3 ellipsoids 5/24/2012 University of Wisconsin - Simulation Based Engineering Lab 27

28 Multi-Physics targeted Computational Dynamics requires Advanced modeling techniques Strong algorithmic (applied math) support Proximity computation Domain decomposition & Inter-domain data exchange Post-processing (visualization) 28

29 HCT: Novel modeling techniques Seeking to solve numerically the equations of motion robustly & effectively in a heterogeneous hardware environment 29

30 Traditional Discretization Scheme positions time step index Mass Mat. speeds Applied Forces Reaction impulses Complementarity Condition Stabilization term Coulomb 3D fricion model 30 (Stewart, 1998)

31 The Cone Complementarity Problem (CCP) First order optimality conditions lead to Cone Complementarity Problem Introduce the convex hypercone and its polar hypercone: CCP assumes following form: Find such that 31

32 The Quadratic Programming Angle The relaxed EOM represent a cone-complementarity problem (CCP) The CCP captures the first-order optimality condition for a quadratic optimization problem with conic constraints: Notation used: 32

33 CCP Solution Algorithm [mapped on the GPU] 33

34 h =.0001 [s] m g = k spheres r = 3.5 mm μ =.46 s 2 ω = π rad sec Anchor width = 5 [cm] 34

35 200,000 Bodies & 10 kg Anchor 35

36 1 Million Rigid Spheres [parallel on the GPU] 36

37 Objective Function Value [1K bodies, 3525 contacts] The red line has 1000 dots on it; i.e.,1000 Jacobi sweeps The green & blue lines have 100 dots on them; i.e.,100 changes of active set Method GPMINRES-no p GPMINRES-no p (not plotted above) GPMINRES-p Iterations 1000 MinRes Its. [within 100 changes of active set] MinRes Its. [within 1000 changes of active set] 100 MinRes Its. [within 100 changes of active set] Final Objective Function Value γ min γ max Computation Time [sec] Jacobi

38 Multi-Physics targeted Computational Dynamics requires Advanced modeling techniques Strong algorithmic (applied math) support Proximity computation Domain decomposition & Inter-domain data exchange Post-processing (visualization) 38

39 600,000 Bodies Moving & Colliding [on the GPU] 39

40 Example: Ellipsoid-Ellipsoid CD 1 1 d P - P ( M M ) c ( b - b ) n d P1 P2 d P1 P2, i i i i j i j i j A : Rotation Matrix P 1 2 P 1 1 ( M Mcc M) c T 3 i 2 8 i 2 P 1 3 T T c c ( M Mcc M) c M 3 5 i j 8 32 j i 1 3 [ T c c ) ) T c (c M M + Mc( M ] ( ) 2 8 i i j 2 T M Mcc M 3 i j c 2 T M = AR A R diag( r1, r2, r3 ) b : Translation of ellipsoids center T n Mn x z c 2 1 n 2 y P 2 40

41 Collision Detection Broad phase Draws on an Axis Aligned Bounding Box (AABB) approach Narrow phase Draws on Minkowski Portal Refinement 41

42 Multiple-GPU Collision Detection Assembled Quad GPU Machine Processor: AMD Phenom II X4 940 Black Memory: 16GB DDR2 Graphics: 4x NVIDIA Tesla C1060 Power supply 1: 1000W Power supply 2: 750W 42

43 Software/Hardware Setup Main Data Set Results 16 GB RAM Open MP Thread 0 Thread 1 Thread 2 Thread 3 Quad Core AMD Microprocessor CUDA GPU 0 GPU 1 GPU 2 GPU 3 Tesla C1060 4x4 GB Memory 4x30720 threads 43

44 Time (Sec) Spheres Contacts vs. Time Quad Tesla C1060 Configuration Contacts (Billions) 44

45 X Speedup Speedup - GPU vs. CPU (Bullet library) [results reported are for spheres] GPU: NVIDIA Tesla C1060 CPU: AMD Phenom II Black X4 940 (3.0 GHz) Contacts (Millions) 45

46 Multi-Physics targeted Computational Dynamics requires Advanced modeling techniques Strong algorithmic (applied math) support Proximity computation Domain decomposition & Inter-domain data exchange Post-processing (visualization) 46

47 Juggling World Record: 64 People Juggling (of all places) in Madison, Wisconsin 47

48 Computation Using Multiple CPUs 48

49 HCT: Domain decomposition & Inter-domain data exchange Relates to the ability to divide the simulation into chunks and have multiple CPUs/GPUs exchange data during simulation as needed Elements leave one subdomain to move to a different one Key issues: Dynamic load balancing Establish a dynamic data exchange protocol (DDEP) between sub-domains X 3 v 3 Y v v 1 v 4 5 v

50 0.5 Million Bodies on 64 Cores [Penalty Approach, MPI-based] 50

51 Computation Using Multiple CPUs 51

52 Multi-Physics targeted Computational Dynamics requires Advanced modeling techniques Strong algorithmic (applied math) support Proximity computation Domain decomposition & Inter-domain data exchange Post-processing (visualization) 52

53 HCT: Visualization and Post-Processing Rendering very complex scenes with more than one million components Rendering takes longer than simulating Pursuing a rendering pipeline that draws on multiple CPUs and GPUs 53

54 Track Simulation Parameters: Driving speed: 1.0 rad/sec Length: 12 seconds Time step: sec Computation time: 18.5 hours Particle radius: m Terrain: 284,715 particles Inertia parameters of track are fake 54

55 Dual Track Footprint 55

56 Simulation of MRAP Impacted by Debris [work in progress] 56

57 M113 Tank Simulation 57

58 Validation. 58

59 Model Simulate Validate Validation at microscale Validation at macroscale University of Parma, Italy Whenever possible, validate against established codes or analytical results Experimental testing is expensive, time consuming, and requires a different set of skills 59

60 Flat Hopper Tests Video recording from a test 60

61 Flat Hopper Tests 3D rendering from a simulation (4x slower than real-time) 61

62 Flat Hopper Tests Comparison experimental - simulated Experimental Simulated 62

63 Granular Flow: Experimental Validation CPU connection Beads Nanopositioner controller Load cell Translational stage Nanopositioner 63

64 Flow Measurement, 500 micron Spheres 64

65 Flow Simulation, 500 micron Spheres 65

66 Weight [N] Flow Measurement Results, 3mm Gap Size Time [sec] 66

67 Weight [N] Flow Measurement Results, 2.5mm Gap Size Time [sec] 67

68 Weight [N] Flow Measurement Results, 2mm Gap Size Time [sec] 68

69 Weight [N] Flow Measurement Results, 1.5mm Gap Size Time [sec] 69

70 SPH Validation Transient Poiseuille flow Series solution fx vx ( y, t) y( y L) 2 n fl x (2n 1) y sin( ) 3 3 (2n 1) L 2 2 (2n 1) exp( t) 2 L Simulation vs. Analytical results 5/24/2012 University of Wisconsin - Simulation Based Engineering Lab 70

71 SPH Validation Lateral migration of circular cylinder in plane Poiseuille flow Lateral migration of spherical particle in pipe flow The results are in agreement with experiments and other numerical methods: Segre-Silberberg (Nature, no. 189, pp. 2, 1961), Inamuro et al. (International Journal of Multiphase Flow, vol. 26, no. 12, pp , 2000), Pan and Glowinski (Journal of Computational Physics, vol. 181, no. 1, pp , 2002). 5/24/2012 University of Wisconsin - Simulation Based Engineering Lab 71

72 Deformable Body Contact Validation x-position of beam tip as a function of time 72

73 Deformable Body Contact Validation y-position of beam tip as a function of time 73

74 ANCF Energy Conservation 74

75 Conclusions/Putting Things in Perspective Goal: investigate how computing can catalyze over the next 10 years advances in Science and innovation in Engineering Reaching the goal Develop an experimentally validated Heterogeneous Computing Template (HCT) Use HCT to advance state of the art in physics-based simulation 75

76 HCT: Looking Ahead Release 0.9 scheduled for August 2012 C++ libraries, Python wrapper Free download, 76

77 Details reported in [1] A. Pazouki, H. Mazhar, and D. Negrut, "Parallel Contact Detection between Ellipsoids with Applications in Granular Dynamics," Mathematics and Computers in Simulation, p. DOI: /j.matcom , [2] D. Negrut, A. Tasora, H. Mazhar, T. Heyn, and P. Hahn, "Leveraging parallel computing in multibody dynamics," Multibody System Dynamics, pp. 1-23, DOI /s y, [3] D. Negrut, A. Tasora, M. Anitescu, H. Mazhar, T. Heyn, and A. Pazouki, "Solving Large Multi-Body Dynamics Problems on the GPU " in GPU Gems 4, The Jade Edition: Morgan Kaufmann Publishers. vol. 2, W. Hwu, Ed., ed, [4] A. Tasora, D. Negrut, and M. Anitescu, "GPU-Based Parallel Computing for the Simulation of Complex Multibody Systems with Unilateral and Bilateral Constraints: An Overview," in Computational Methods in Applied Sciences: Multibody Dynamics. vol. 23, K. Arczewski, et al., Eds., ed: Springer Netherlands, 2011, pp [5] H. Mazhar, T. Heyn, and D. Negrut, "A scalable parallel method for large collision detection problems," Multibody System Dynamics, vol. 26, pp ,

78 Thank You. Simulation-Based Engineering Lab Wisconsin Applied Computing Center 78