Early Experiences Porting the NAMD and VMD Molecular Simulation and Analysis Software to GPU-Accelerated OpenPOWER Platforms

Transcription

1 Early Experiences Porting the NAMD and VMD Molecular Simulation and Analysis Software to GPU-Accelerated OpenPOWER Platforms John E. Stone, Antti-Pekka Hynninen, James C. Phillips, Klaus Schulten Theoretical and Computational Biophysics Group Beckman Institute for Advanced Science and Technology University of Illinois at Urbana-Champaign Center for Accelerated Application Readiness, Oak Ridge National Laboratory International Workshop on OpenPOWER for HPC Frankfurt, Germany, June 23, 2016

2 Goal: A Computational Microscope Study the molecular machines in living cells Ribosome: target for antibiotics Poliovirus

3 Number of atoms Computational Biology s Insatiable Demand for Processing Power HIV capsid Ribosome 10 6 ATP Synthase STMV 10 5 Lysozyme ApoA

4 Our Interest in GPU-Accelerated OpenPOWER Platforms OpenPOWER roadmap includes: NVLink interconnect to GPUs, 5x-12x PCIe 3.0 perf Support for a variety of advanced DRAM and persistent memory systems Performance of NAMD and VMD on current petascale systems such as Blue Waters and Titan benefits greatly from GPU accelerators DOE Summit and Sierra systems based on Power9 + Volta in late 2017

5 Porting and Testing NAMD and VMD on Crest Power8 systems at ORNL CAAR Crest test systems at the ORNL Center for Accelerated Application Readiness IBM S822L nodes: Dual 10-core IBM POWER8 CPUs, support up to 8-way SMT per physical CPU core 256GB RAM NUMA arch. w/ 4 NUMA-nodes, 384GB/sec peak mem. bandwidth 4x NVIDIA Tesla K40m GPUs, w/ PCIe 3.0 interconnect Testbed to prepare for Summit ORNL Summit

6 CDIMMs P8 CPU IBM S822L CDIMMs SMP A Bus SMP X Bus SMP A Bus P8 CPU PHB PHB PHB PCIe 3.0 x16 PCIe 3.0 x16 PCIe 3.0 x8 GPU GPU IB NIC CDIMMs P8 CPU SMP A Bus SMP X Bus SMP A Bus PHB PHB PHB PCIe 3.0 x8 PCIe 3.0 x16 PCIe 3.0 x16 IB NIC GPU GPU CDIMMs P8 CPU

7

8 NAMD Performance Improvements and OpenPOWER Evaluation For Blue Waters and Titan, CUDA kernels for nonbonded force calculation provide the greatest fraction of GPU speedup New kernels developed for explicit solvent and implicit solvent non-bonded force calculation Performance of new kernels evaluated on POWER8+K40m (Crest) vs. Cray XK7 (BW, Titan) NAMD performance is contingent on many factors, so much work still remains to be done

9 NAMD Non-bonded Force Optimizations for the GPU Use warp-synchronous programming to exchange data between GPU threads using registers rather than shared memory Improve average work efficiency with improved cutoff test tile organization and by exploiting Newton s third law

10 Explicit Solvent Speedup

11 Implicit Solvent (GBIS) Speedup

12 Comparison of Crest vs. roughly comparable Titan nodes Evaluated performance of new NAMD kernels and overall simulation runtime Single node of Crest vs. four nodes of Titan: Crest node: dual POWER8 DCMs and four Tesla K40m GPUs Titan node: single AMD Opteron 6276 and one Tesla K20X Relative performance shown for Crest normalized by Titan Tesla K40m should run ~7% faster than Tesla K20X

13 POWER8+K40m vs. Cray XK7 (1-node Crest vs. 4-nodes Titan)

14 VMD OpenPOWER Performance Evaluation VMD performance characteristics are not dominated by any one kernel or algorithm VMD makes extensive use of GPUs, but there are still many routines that remain CPU-only code paths at present We evaluated performance of several visualization and analysis algorithms with differing characteristics to obtain a broad view of OpenPOWER s performance traits

15 Compiler Evaluation Many compiler choices will be available for OpenPOWER platforms We did some VMD microbenchmark comparisons between GCC and XLC to evaluate their ability to vectorize simple loops etc. IBM XLC was generally more performant, ranging from 7% to 50% faster than GCC One spectacularly bad case for GCC generated VSX vector code that ran 12x slower than XLC, and 11x slower than GCC generating scalar code All subsequent tests were performed with XLC

16 Tachyon Ray Tracing Performance VMD supports high performance ray tracing with built-in RT engines Tachyon: multi-platform hardware-neutral OptiX: NVIDIA GPUs, not available on POWER yet Tachyon characteristics: No hand-coded or other explicit vectorization Construction and traversal of linked lists, spatial acceleration data structures, lots of scalar floating point arithmetic Potential memory latency hiding benefit from SMT

17

18

19 Detailed Tachyon Double-Precision Performance w/ Varying SMT

20 GPU-Accelerated MDFF Cross Correlation GPUs enable laptops and desktop workstations to handle tasks that would have previously required a cluster, or a very long wait GPU-accelerated petascale supercomputers enable analyses were previously impractical, allowing detailed study of very large structures such as viruses GPU-accelerated MDFF Cross Correlation Timeline Regions with poor fit Regions with good fit

21

22

23 MDFF Cross Correlation Hand-coded POWER8 VSX yielded 20% performance gain vs. XLC vectorization Hand-coded x86 SSE yielded only 12% performance gain vs. ICC vectorization POWER8 gained nothing from SMT w/ hand-coded VSX

24 Hand-Coded VSX Instructions. vector float xdelta = *(( vector float *) &sxdelta[0]); // aligned load vector float xdist = vec_splats(sxdist); xdist = vec_add(xdist, xdelta); vector float ydist = vec_splats(sydist); vector float zdist = vec_splats(szdist); vector float xdist2 = vec_mul(xdist, xdist); vector float ydist2 = vec_mul(ydist, ydist); vector float zdist2 = vec_mul(zdist, zdist); vector float dist2 = vec_splats(yzdist2); dist2 = vec_add(dist2, xdist2);.

25 Molecular Orbital Performance Hand-coded POWER8 VSX yielded a speedup of ~2x for MO kernel We saw that SMT benefits scalar C++ by up to 1.85x, but not the hand-coded VSX versions of the same loops

26 Limitations/Issues Encountered No support for GPU peer-to-peer DMAs over PCIe (yet?) POWER CPU core/thread indexing schemes are the transpose of what is done on x86 platforms NUMA thread mapping on POWER8 is complex due to variable SMT modes, dynamic CPU states, etc. No API for querying SMT modes! Code to query SMT state is a few hundred lines, reading /proc /sys etc. as implemented in IBM processor management utilities HPC users need clean APIs that make NUMA pinning and thread indexing much simpler

27 Future Work Continued tuning of NAMD kernels, identification of OpenPOWER-specific overheads for GPU-accelerated kernels VMD molecular dynamics trajectory clustering analysis Preliminary result: POWER8 is ~2.5x faster than dual Xeon 2687W-v3 due to much greater peak memory bandwidth Evaluate OpenACC on POWER Support for GPU hardware OpenGL acceleration, EGL, OptiX GPU ray tracing, and NVENC GPU hardware video encoding

28 Acknowledgements Theoretical and Computational Biophysics Group, University of Illinois at Urbana-Champaign Center for Accelerated Application Readiness, Oak Ridge National Laboratory Funding: DOE INCITE, ORNL Titan: DE-AC05-00OR22725 NSF Blue Waters: NSF OCI , PRAC The Computational Microscope, ACI , ACI NIH support: 9P41GM104601, 5R01GM

29

30 Related Publications Research/gpu/ Immersive Molecular Visualization with Omnidirectional Stereoscopic Ray Tracing and Remote Rendering. John E. Stone, William R. Sherman, and Klaus Schulten.High Performance Data Analysis and Visualization Workshop, IEEE International Parallel and Distributed Processing Symposium Workshop (IPDPSW), (In-press) High Performance Molecular Visualization: In-Situ and Parallel Rendering with EGL. John E. Stone, Peter Messmer, Robert Sisneros, and Klaus Schulten.High Performance Data Analysis and Visualization Workshop, IEEE International Parallel and Distributed Processing Symposium Workshop (IPDPSW), (In-press) Evaluation of Emerging Energy-Efficient Heterogeneous Computing Platforms for Biomolecular and Cellular Simulation Workloads. John E. Stone, Michael J. Hallock, James C. Phillips, Joseph R. Peterson, Zaida Luthey-Schulten, and Klaus Schulten.25th International Heterogeneity in Computing Workshop, IEEE International Parallel and Distributed Processing Symposium Workshop (IPDPSW), (In-press) Atomic Detail Visualization of Photosynthetic Membranes with GPU-Accelerated Ray Tracing. J. E. Stone, M. Sener, K. L. Vandivort, A. Barragan, A. Singharoy, I. Teo, J. V. Ribeiro, B. Isralewitz, B. Liu, B.-C. Goh, J. C. Phillips, C. MacGregor-Chatwin, M. P. Johnson, L. F. Kourkoutis, C. Neil Hunter, and K. Schulten. J. Parallel Computing, (In-press) Chemical Visualization of Human Pathogens: the Retroviral Capsids. Juan R. Perilla, Boon Chong Goh, John E. Stone, and Klaus SchultenSC'15 Visualization and Data Analytics Showcase, 2015.

31 Related Publications Research/gpu/ Visualization of Energy Conversion Processes in a Light Harvesting Organelle at Atomic Detail. M. Sener, J. E. Stone, A. Barragan, A. Singharoy, I. Teo, K. L. Vandivort, B. Isralewitz, B. Liu, B. Goh, J. C. Phillips, L. F. Kourkoutis, C. N. Hunter, and K. Schulten. SC'14 Visualization and Data Analytics Showcase, ***Winner of the SC'14 Visualization and Data Analytics Showcase Runtime and Architecture Support for Efficient Data Exchange in Multi-Accelerator Applications. J. Cabezas, I. Gelado, J. E. Stone, N. Navarro, D. B. Kirk, and W. Hwu. IEEE Transactions on Parallel and Distributed Systems, (In press) Unlocking the Full Potential of the Cray XK7 Accelerator. M. D. Klein and J. E. Stone. Cray Users Group, Lugano Switzerland, May GPU-Accelerated Analysis and Visualization of Large Structures Solved by Molecular Dynamics Flexible Fitting. J. E. Stone, R. McGreevy, B. Isralewitz, and K. Schulten. Faraday Discussions, 169: , Simulation of reaction diffusion processes over biologically relevant size and time scales using multi-gpu workstations. M. J. Hallock, J. E. Stone, E. Roberts, C. Fry, and Z. Luthey-Schulten. Journal of Parallel Computing, 40:86-99, 2014.

32 Related Publications Research/gpu/ GPU-Accelerated Molecular Visualization on Petascale Supercomputing Platforms. J. Stone, K. L. Vandivort, and K. Schulten. UltraVis'13: Proceedings of the 8th International Workshop on Ultrascale Visualization, pp. 6:1-6:8, Early Experiences Scaling VMD Molecular Visualization and Analysis Jobs on Blue Waters. J. Stone, B. Isralewitz, and K. Schulten. In proceedings, Extreme Scaling Workshop, Lattice Microbes: High performance stochastic simulation method for the reaction diffusion master equation. E. Roberts, J. Stone, and Z. Luthey Schulten. J. Computational Chemistry 34 (3), , Fast Visualization of Gaussian Density Surfaces for Molecular Dynamics and Particle System Trajectories. M. Krone, J. Stone, T. Ertl, and K. Schulten. EuroVis Short Papers, pp , Immersive Out-of-Core Visualization of Large-Size and Long-Timescale Molecular Dynamics Trajectories. J. Stone, K. L. Vandivort, and K. Schulten. G. Bebis et al. (Eds.): 7th International Symposium on Visual Computing (ISVC 2011), LNCS 6939, pp. 1-12, Fast Analysis of Molecular Dynamics Trajectories with Graphics Processing Units Radial Distribution Functions. B. Levine, J. Stone, and A. Kohlmeyer. J. Comp. Physics, 230(9): , 2011.

33 Related Publications Research/gpu/ Quantifying the Impact of GPUs on Performance and Energy Efficiency in HPC Clusters. J. Enos, C. Steffen, J. Fullop, M. Showerman, G. Shi, K. Esler, V. Kindratenko, J. Stone, J Phillips. International Conference on Green Computing, pp , GPU-accelerated molecular modeling coming of age. J. Stone, D. Hardy, I. Ufimtsev, K. Schulten. J. Molecular Graphics and Modeling, 29: , OpenCL: A Parallel Programming Standard for Heterogeneous Computing. J. Stone, D. Gohara, G. Shi. Computing in Science and Engineering, 12(3):66-73, An Asymmetric Distributed Shared Memory Model for Heterogeneous Computing Systems. I. Gelado, J. Stone, J. Cabezas, S. Patel, N. Navarro, W. Hwu. ASPLOS 10: Proceedings of the 15 th International Conference on Architectural Support for Programming Languages and Operating Systems, pp , 2010.

34 Related Publications Research/gpu/ GPU Clusters for High Performance Computing. V. Kindratenko, J. Enos, G. Shi, M. Showerman, G. Arnold, J. Stone, J. Phillips, W. Hwu. Workshop on Parallel Programming on Accelerator Clusters (PPAC), In Proceedings IEEE Cluster 2009, pp. 1-8, Aug Long time-scale simulations of in vivo diffusion using GPU hardware. E. Roberts, J. Stone, L. Sepulveda, W. Hwu, Z. Luthey-Schulten. In IPDPS 09: Proceedings of the 2009 IEEE International Symposium on Parallel & Distributed Computing, pp. 1-8, High Performance Computation and Interactive Display of Molecular Orbitals on GPUs and Multi-core CPUs. J. Stone, J. Saam, D. Hardy, K. Vandivort, W. Hwu, K. Schulten, 2nd Workshop on General-Purpose Computation on Graphics Pricessing Units (GPGPU-2), ACM International Conference Proceeding Series, volume 383, pp. 9-18, Probing Biomolecular Machines with Graphics Processors. J. Phillips, J. Stone. Communications of the ACM, 52(10):34-41, Multilevel summation of electrostatic potentials using graphics processing units. D. Hardy, J. Stone, K. Schulten. J. Parallel Computing, 35: , 2009.

35 Related Publications Research/gpu/ Adapting a message-driven parallel application to GPU-accelerated clusters. J. Phillips, J. Stone, K. Schulten. Proceedings of the 2008 ACM/IEEE Conference on Supercomputing, IEEE Press, GPU acceleration of cutoff pair potentials for molecular modeling applications. C. Rodrigues, D. Hardy, J. Stone, K. Schulten, and W. Hwu. Proceedings of the 2008 Conference On Computing Frontiers, pp , GPU computing. J. Owens, M. Houston, D. Luebke, S. Green, J. Stone, J. Phillips. Proceedings of the IEEE, 96: , Accelerating molecular modeling applications with graphics processors. J. Stone, J. Phillips, P. Freddolino, D. Hardy, L. Trabuco, K. Schulten. J. Comp. Chem., 28: , Continuous fluorescence microphotolysis and correlation spectroscopy. A. Arkhipov, J. Hüve, M. Kahms, R. Peters, K. Schulten. Biophysical Journal, 93: , 2007.