Amrita Mathuriya HPC Application Engineer DCG/Intel Corporation November 2016

Transcription

1

2 Optimizations of B-spline based SPO evaluations in QMC for Multi/many-core Shared Memory Processors Amrita Mathuriya HPC Application Engineer DCG/Intel Corporation November 2016 In collaboration with Jeongnim Kim (Intel), Victor Lee(Intel) Ye Luo, Anouar Benali (Argonne National Laboratory) Luke Shulenburger (Sandia National Laboratories )

3 Presenter: Amrita Mathuriya HPC application Engineer at HPC Ecosystem Application Engineering Team; working for code modernization and optimization on Xeon and Xeon Phi Working at Intel for past 8 Years. Expert at algorithms and optimizations for IA architectures. Worked on HPC applications in areas of Computational Geometry, Optical Proximity Correction (OPC), Electromagnetics, Computational Biology, Quantum Monte Carlo. Working on code modernization for Intel Xeon and Xeon Phi architectures. MS in Computer Science with the specialization in Computational Science and Engineering from Georgia Tech, USA under the guidance of Professor David Bader. Obtained B. Tech degree in Computer Science from Indian Institute of Technology (IIT) Roorkee, India. 3

4 Systems KNC: Intel Xeon Phi coprocessor 7120P GHz, 4-way Intel Hyper-Threading Technology, Memory: MB Intel Many-core Platform Software Stack Version OS Version : el7.x86_64 Intel Xeon Phi 7250P (code-named Knights Landing, KNL), 68 cores, 1.4GHz with 16GB MCDRAM (used in flat mode), cluster boot mode=quad, Turbo=enable. KNL used in Quad/Flat mode. Intel Xeon E5-2697v4(BDW) node single socket, 18 cores HT 145W (E5-2697v4 w/128gb RAM DDR *16GB DIMMS. Bluegene/Q (BG/Q) processor from Mira Supercomputer, at Argonne National lab facility. Compilers and MPI and math library. icc version (gcc version compatibility) Intel(R) MPI Library for Linux* OS, Version Build (build id: 14053) 4

5 Agenda KNL overview and motivation Intro to quantum Monte Carlo and QMCPACK Current status of QMCPACK Analysis of CORAL graphite benchmark Optimizations to B-spline based SPO evaluations for QMC Summary 5

6 6

7 Important Characteristics of KNL Increasing core count per node on both Intel Xeon and Xeon Phi processors. Large SIMD units AVX512 supporting 16 single precision floating point simultaneously. Two level Cache system L1/L2 and high memory bandwidth. 7

8 How to gain performance? Scalability Enable data sharing with hybrid parallelism using MPI + threading. Design and implement scalable algorithms SIMD Parallelism adapt Data layouts to enable efficient vectorization. Efficiently utilize caches and memory bandwidth with Tiling (cache-blocking). 8

9 Roofline Performance Analysis on KNL Peak GFLOPS at (0.22 Flops/Byte GFLOPS NOW, 7% Scalar add peak 0.22 VGH roofline performance model for N=2048. Circles denote GFLOPS at the cache-aware AI. 7% of peak GFLOPS achieved with the current AoS version Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, visit Intel Performance Benchmark Limitations. 9

10 Performance Portable on Intel Xeon, Intel Xeon Phi and BG/Q Processors Optimizations for efficiently utilizing SIMD units and caches. SoA data layout transformation. Tiling or AoSoA data layout transformation. Nested thread parallelization to reduce time-to-solution and memory usage. Optimization work done on KNC. Later ported on KNL works out of the box. Optimizations result in significant performance improvement on BG/Q. 10

11 QMCPACK An open-source US-DOE flagship many-body ab initio quantum Monte Carlo (QMC) code for computing the electronic structure of atoms, molecules, and solids. (a) DMC charge-density of AB stacked graphite and (b) the ball-andstick rendering and the 4-Carbon unit cell in blue. Parallel efficiency of QMCPACK on US-DOE facilities. The legend shows the MPI tasks and OpenMP threads of the reference computing unit (CU) and the maximum number of nodes on each platform. J. Kim, K. P. Esler, J. McMinis, M. A. Morales, B. K. Clark, L. Shulenburger, and D. M. Ceperley, Hybrid algorithms in quantum monte carlo, Journal of Physics: Conference Series, vol. 402, no. 1, p , [Online]. Available: 11

12 Diffusion Monte Carlo Schematics Old configurations Random Walking Possible new configurations New configurations w=0.8 ensemble w=1.6 Ensemble evolves according to w=2.4 Diffusion Drift Branching w=0.3

13 How is QMCPACK parallelized QMCPACK utilizes OpenMP to optimize memory usage and to take advantage of the growing number of cores per SMP node. Walkers within a MPI task is distributed among the cores in CPU. Big common data is shared by all the walkers like wave function coefficients. Frequency stops increasing. Node count stops growing. Nodes are getting more powerful but require applications to expose more concurrency. Free lunch is over. On node performance is challenging. 13

14 QMCPACK status Pretty good and we can even do better! Excellent MPI & OpenMP parallel efficiency at the walker level All in double precision except 3D cubic B-Spline. Work done recently to implement mixed precision. Speeds up by x. SIMD efficiency low Basically scalar performance with few exceptions B-Spline SSE/SSE2/QPX Distance tables with QPX Array of Structure (AoS) for D-dim N-particle attributes, e.g., R (N,3), Gradients (N,3), Hessian matrices (N,9) 14

15 CORAL Benchmark KNL Profiling 4x4x1 AB-stacked graphite 64 carbon 256 electrons Coral Benchmark Profile on KNL 18% 29% 18% 34% Einspline Distance Table Jastraw Others The three compute kernels account for 80% run time in QMCPACK on KNL. 15

16 QMC: Single particle orbital (SPO) representation with B- spline basis set One Dimensional cubic B-spline function Tensor product in each Cartesian direction, Representation for 3D orbital, Precomputed coefficients 4D Read only array. Stored in SOA format, P[nx][ny][nz][N] Provided by DFT or HF computations using Quantum Espresso 16

17 Simplified miniqmc Only contains B-spline evaluation routines. Mimics the computational and data access patterns of B- spline SPO evaluations in QMC. Random position generation B-spline SPO evaluation kernels 17

18 Data Layout Performance Considerations Array-of-Structs (AoS) Struct-of-Arrays (SoA) Hybrid (AoSoA) x y z x y z x x x x x x x x y y z z x y z x y z y y y y y y x x y y z z x y z x y z z z z z z z x x y y z z Pros: Logical for expression of physical abstractions in 3D or higher dimensions. Pros: Contiguous loads/stores for efficient vectorization. Pros: Potentially useful for increasing cache locality. Also supports efficient vectorization. 18

19 Pseudocode - VGH Computes value, gradient, Hessian at random (x,y,z) Random Strided access for output arrays. Data access pattern of read-only B-spline coefficients P at a random position (x; y; z) and j0=floor(y/dy) etc. The outermost x dimension is not shown. 19

20 SoA transformation for output arrays x x x x x x y y y y y y z z z z z z Output arrays in SoA (Structure of arrays) format 20

21 How to evaluate performance of QMC Rate of Monte Carlo sample generations (throughputs) per resource For the miniapp, Throughput = (number of evaluations)/(t) Evaluations = (Number of walkers) X (Number of iterations) X (Number of splines) T = Time per call of a function ( such as VGH ) Throughput represents work done on a node. Ideally, it should stay constant across problem sizes. 21

22 VGH throughput by AoS-to-SoA transformation Higher the better 2x-4x Performance improvement for small to medium problem sizes. Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, visit Intel Performance Benchmark Limitations. 22

23 Pseudocode - VGH Computes value, gradient, Hessian at random (x,y,z) Random Strided access for output arrays. Data access pattern of read-only B-spline coefficients P at a random position (x; y; z) and j0=floor(y/dy) etc. The outermost x dimension is not shown. 23

24 Why low performance for large N? AoS-to-SoA improves SIMD efficiency But, caches can be utilized better Reduction on the arrays G& H of size N Streaming access at 4x4x4 block Pressure on resources with large N, e.g., TLB Reduction of output arrays over 64N values How to keep the write data in L1/L2 HUB How to maximize LLC sharing Core Core 24

25 AoSoA Data Layout Transformation x x x x x x y y y y y y z z z z z z Tiled Input array Tiled output arrays Data access pattern of read-only B-spline table a) Current b) Tiled Efficient cache utilization, by tiling both input and output arrays along the innermost dimension. 25

26 Performance gain with tiling/aosoa - Higher the better VGH Performance with SoA to AoSoA transformation (tiling) AoSoA helps achieve sustained throughput across problem sizes for all architectures. Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, visit Intel Performance Benchmark Limitations. 26

27 VGH throughput with tiling, higher the better BDW peak at 64 The tiled input array fits in L3 cache. KNC, KNL peak at 512 For tile size > 512, output arrays fall out of caches. Performance of VGH at N = 2048 with respect to tile size. Tiling improves performance for all three processors. Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, visit Intel Performance Benchmark Limitations. 27

28 Hybrid OpenMP/MPI Parallelism in QMCPACK Current parallelism over walkers (Nw). Working set size in QMCPACK grows with number of walkers. Parallelizing each walker update Specifically, for Intel Xeon Phi, with large number of cores/threads, next level of parallelism becomes essential for strong scaling. Parallel efficiency of QMCPACK on US-DOE facilities. The legend shows the MPI tasks and OpenMP threads of the reference computing unit (CU) and the maximum number of nodes on each platform. J. Kim, K. P. Esler, J. McMinis, M. A. Morales, B. K. Clark, L. Shulenburger, and D. M. Ceperley, Hybrid algorithms in quantum monte carlo, Journal of Physics: Conference Series, vol. 402, no. 1, p , [Online]. Available: 28

29 Parallelism within a walker nested threading Reduces memory requirement and time to solution on a node, by reducing the number of walkers on a node. miniqmc replaces OpenMP nested threading with manual assignment of work. #pragma omp parallel Strong Scaling:- Independent execution of tiles in different threads. 29

30 Strong Scaling Results on KNL Speedup on KNL w.r.t. number of walkers per thread. Reduces time to solution by ~14x with 16 threads per walker Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, visit Intel Performance Benchmark Limitations. 30

31 Strong Scaling Results on KNL Performance of VGH at N = 2048 with respect to tile size. Speedup on KNL w.r.t. number of walkers per thread. Reduces time to solution by ~14x with 16 threads per walker Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, visit Intel Performance Benchmark Limitations. 31

32 Roofline Performance Analysis on KNL SoA, ~7% of peak GFlops SoA data layout conversion Increases cache aware AI from 0.22 to 0.32 ~7% of the achievable peak. 1.5x speedup wrt. AoS version à 0.32 VGH roofline performance model for N=2048. Circles denote GFLOPS at the cache-aware AI and X (b) the best performance (AoSoA) on DDR. Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, visit Intel Performance Benchmark Limitations. 32

33 Roofline Performance Analysis on KNL AoSoA, 11% of peak GFlops AoSoA version increases cache reuse with the same AI. Better cache utilization. ~2.25x gain in performance à 0.32 VGH roofline performance model for N=2048. Circles denote GFLOPS at the cache-aware AI and X (b) the best performance (AoSoA) on DDR. Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, visit Intel Performance Benchmark Limitations. 33

34 Roofline Performance Analysis on KNL AoSoA, 3.3x speedup With MCDRAM AoSoA version with MCDRAM ~3.3x faster than DDR. VGH roofline performance model for N=2048. Circles denote GFLOPS at the cache-aware AI and X (b) the best performance (AoSoA) on DDR. Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, visit Intel Performance Benchmark Limitations. 34

35 Roofline performance analysis on BDW AoSoA, ~50% of achievable GFlops 660 GFLOPS SP Vector FMA Peak VGH roofline performance model for N=2048. Circles denote GFLOPS at the cache-aware AI Performance improved to ~50% of peak GFLOPS with the AoSoA version. Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, visit Intel Performance Benchmark Limitations. 35

36 Performance Summary The improvements are portable to 4 types of CPUs, even from different vendors. Significant speedups even on BG/Q. On VGH routine BGQ BDW KNC KNL SOA and basic 1.9x 1.7x 2.6x 1.7x AoSoA/Tiling 2.7x 3.7x 5.2x 2.3x Strong scaling 5.2x 6.4x 35.2x 33.1x Number of threads per walker (The optimal tile size) 2(32) 2(32) 8(256) 16(128) 36

37 Symmetric Distance table computation AoS to SoA transformation of particle positions. Speedup vs. Problem Size Higher the better Speedup wrt. BDW Baseline KNL 50x Faster with SoA data layout Number of electrons KNL Baseline(256 TH) BDW Opt(2MPI/36TH) KNL Opt(256 TH) KNL used in Quad/Cache mode for these experiments Here, TH = threads. BDW has 2 sockets for these experiments. Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, visit Intel Performance Benchmark Limitations. 37

38 Results Array of structures (AOS) to structure of arrays (SOA) transform helps achieve efficient vectorization. Tiling for better memory access helps achieve approximately constant throughput across problem sizes. Nested parallelism over the AoSoA objects on KNL helps reduce the time-tosolution by ~14x speedup with 16 threads. Optimizations result in significant performance gain on all three distinct cache-coherent architectures. 38

39 Ways we increased the performance! SIMD Parallelism SoA data layout adaption. Efficient cache utilization Tiling/Cache-Blocking. Scalability Next level of threading to reduce time to solution. Takes advantage of reduced working set size. 39

40 Reference Amrita Mathuriya, Ye Luo, Anouar Benali, Luke Shulenburger, Jeongnim Kim Optimization and parallelization of B-spline based orbital evaluations in QMC on multi/ many-core shared memory processors arxiv:

41 Legal Disclaimers INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS. Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling , or go to: Knights Landing and other code names featured are used internally within Intel to identify products that are in development and not yet publicly announced for release. Customers, licensees and other third parties are not authorized by Intel to use code names in advertising, promotion or marketing of any product or services and any such use of Intel's internal code names is at the sole risk of the user Intel, Look Inside, Xeon, Intel Xeon Phi, Pentium, Cilk, VTune and the Intel logo are trademarks of Intel Corporation in the U.S. and other countries. *Other names and brands may be claimed as the property of others. Copyright 2016 Intel Corporation 41

42 Legal Disclaimers Optimization Notice Intel's compilers may or may not optimize to the same degree for non-intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice revision #

43 Legal Disclaimers Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark* and MobileMark*, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more information go to Estimated Results Benchmark Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Software Source Code Disclaimer: Any software source code reprinted in this document is furnished under a software license and may only be used or copied in accordance with the terms of that license. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 43

44 Thank you for your time Amrita Mathuriya

45 Backup

46 SymmetricDTD::moveonsphere Code Sinippet SoA For efficient auto-vectorization with the compiler Three separate arrays for X, Y and Z instead of a single array with (x, y, z) as a data member. Similar SOA (structure of arrays) data layout for the output array. AoS 46