ZKI-Tagung Supercomputing, Okt Parallel Patterns for Composing and Nesting Parallelism in HPC Applications

Transcription

1 ZKI-Tagung Supercomputing, Okt Parallel Patterns for Composing and Nesting Parallelism in HPC Applications Hans Pabst Software and Services Group Intel Corporation

2 Agenda Introduction Parallel Patterns Today What about HPC? Summary 3

3 What is a Parallel Pattern? Design patterns An encoded expertise to capture that quality without a name that distinguishes truly excellent designs A small number of patterns can support a wide range of applications Parallel pattern A common occurring combination of task distribution and data sharing (data access).

4 Parallel Patterns Stencil Superscalar sequence Reduce Partition Gather/scatter Speculative selection Map Scan and Recurrence Pack and Expand Nest Pipeline Search and Match *

5 Why Parallel Patterns? Pattern: DSL for parallel programming Ready to use pieces efficiently mapping concurrency to hardware parallelism Less error-prone (data races, etc.) More productive and what is the reality? Parallel programming is still difficult. Standard languages progress slowly. 6

6 References Details a pattern language for parallel algorithm design Represents the author's hypothesis of how to think about parallel programming Source code samples in MPI, OpenMP and Java Patterns for Parallel Programming, Timothy G. Mattson, Beverly A. Sanders, Berna L. Massingill, Addison-Wesley, 2005, ISBN

7 References (cont.) Teaches parallel programming in a new more effective manner. It s about effective parallel programming. Not about any specific hardware. (c) 2012, publisher: Morgan Kaufmann

8 References (cont.) e/guyb/papers/ble90.pdf ation/hpc/composerxe/enus/2011update/tbbxe/design_patterns.pdf Arch D. Robinson: Composable Parallel Patterns with Intel Cilk Plus, IEEE,

9 Finding Concurrency Practical approach: optimizing sequential code* Use profiler, such as OProfile or Intel VTune Amplifier XE Identify the hotspots of an application; then: Check if the hotspots consist of independent tasks Check if the hotspots can execute independently Theoretical approach: design document Examine the design components, services, etc. Find components that contain independent operations Other approach: parallel patterns (Practical?) Find parallel pattern. Make the pattern explicit in the code. Use library based model e.g., C++11, Intel TBB Use language primitive e.g., C++11, Intel Cilk Plus * Parallelizing a sequential algorithm may never lead to the best known parallel algorithm.

10 Intel Advisor Tool for scalability and what-if analysis Modeling: use code annotations to introduce parallelism Evaluation: estimate speedup and check correctness GUI-driven assistant (5 steps) Productivity and safety Parallel correctness is checked based on a correct program Non-intrusive API It s not auto-parallelization It s not modifying the code

11 Intel Advisor: Five Steps 1. Survey the application (profiler) 2. Annotate the application (API) 3. Check suitability (estimation) 4. Check correctness (analysis) 5. Apply parallel model (user) Idea: Correctly parallelized application

12 Intel Advisor: Quicksort Example template<typename I> void serial_qsort(i begin, I end) { typedef typename std::iterator_traits<i>::value_type T; if (begin!= end) { const I pivot = end - 1; const I middle = std::partition(begin, pivot, std::bind2nd(std::less<t>(), *pivot)); std::swap(*pivot, *middle); ANNOTATE_SITE_BEGIN(Parallel Region); ANNOTATE_TASK_BEGIN(Left Partition); serial_qsort(begin, middle), ANNOTATE_TASK_END(Left Partition); } } ANNOTATE_TASK_BEGIN(Right Partition); serial_qsort(middle + 1, end)); ANNOTATE_TASK_END(Right Partition); ANNOTATE_SITE_END(Parallel Region); * Pure Quicksort (i.e., not attempt to avoid the tail recursion).

13 Intel Advisor: Fortran w/ Annotations subroutine sgemv(res, mat, vec, nrows, ncols) implicit none integer :: nrows, ncols real(kind=4), intent(out), dimension(0:nrows-1) :: res real(kind=4), intent(in), dimension(0:ncols-1) :: vec real(kind=4), intent(in), dimension(0:nrows*ncols-1) :: mat integer :: i, u, v ANNOTATE_SITE_BEGIN("parallel region") do i = 0, nrows 1 end do ANNOTATE_ITERATION_TASK("I") u = i * ncols v = u + ncols res(i) = DOT_PRODUCT(mat(u:v), vec) ANNOTATE_SITE_END("parallel region") end subroutine 14

14 Intel Advisor: Fortran (cont.) Subset of Intel Advisor annotations #define ANNOTATE_SITE_BEGIN(NAME) #define ANNOTATE_SITE_END(NAME) call annotate_site_begin(name) call annotate_site_end() #define ANNOTATE_ITERATION_TASK(NAME) call annotate_iteration_task(name) #define ANNOTATE_TASK_BEGIN(NAME) call annotate_task_begin(name) #define ANNOTATE_TASK_END(NAME) call annotate_task_end() #define ANNOTATE_LOCK_ACQUIRE(ADDRESS) call annotate_lock_acquire(address) #define ANNOTATE_LOCK_RELEASE(ADDRESS) call annotate_lock_release(address) Enable source code preprocessing $ ifort -fpp source_code_with_macros.f90 Advisor module (advisor_annotate) and library -I$ADVISOR_XE_2013_DIR/include/intel64 -L$ADVISOR_XE_2013_DIR/lib64 -ladvisor 15

15 Intel Advisor: Finding Concurrency Introduce as-if parallelism and evaluate it safely Works for C, C++, and Fortran Threading model agnostic 16

16 Agenda Introduction Parallel Patterns Today What about HPC? Summary 17

17 OpenCL* Elemental functions in particular, or kernel functions in general Similar to vectorizable loop body e e e e e e tiling Tiling serves multiple things Maps to memory hierarchy ND range loop blocking; N>1: nested loops Synchronization

18 Fortran Standardization advances quickly ( 03, 08, 13) Contrary to a dying language Cornerstones Array notation ( slices ) Elemental functions Fortran Co-array 19

19 C++: Intel Threading Building Blocks (Library Based Approach) Generic Parallel Algorithms Efficient scalable way to exploit the power of multi-core without having to start from scratch Miscellaneous Thread-safe timers Task scheduler The engine that empowers parallel algorithms that employs taskstealing to maximize concurrency Threads OS API wrappers Concurrent Containers Concurrent access, and a scalable alternative to containers that are externally locked for thread-safety TBB Flow Graph Thread Local Storage Scalable implementation of thread-local data that supports infinite number of TLS Synchronization Primitives User-level and OS wrappers for mutual exclusion, ranging from atomic operations to several flavors of mutexes and condition variables Memory Allocation Per-thread scalable memory manager and false-sharing free allocators 20

20 C++: Intel Threading Building Blocks (Library Based Approach) Generic Parallel Algorithms Efficient scalable way to exploit the power of multi-core without having to start from scratch Miscellaneous Thread-safe timers Task scheduler The engine that empowers parallel algorithms that employs taskstealing to maximize concurrency Threads OS API wrappers Concurrent Containers Concurrent access, and a scalable alternative to containers that are externally locked for thread-safety TBB Flow Graph Thread Local Storage Scalable implementation of thread-local data that supports infinite number of TLS Synchronization Primitives User-level and OS wrappers for mutual exclusion, ranging from atomic operations to several flavors of mutexes and condition variables Memory Allocation Per-thread scalable memory manager and false-sharing free allocators 21

21 C++: Intel Threading Building Blocks (Generic Algorithms) Loop parallelization parallel_for parallel_reduce - load balanced parallel execution - fixed number of independent iterations parallel_deterministic_reduce - run-to-run reproducible results parallel_scan - computes parallel prefix y[i] = y[i-1] op x[i] parallel_do Parallel Algorithms for Streams - Use for unstructured stream or pile of work - Can add additional work to pile while running parallel_for_each - parallel_do without an additional work feeder pipeline / parallel_pipeline - Linear pipeline of stages - Each stage can be parallel or serial in-order or serial out-of-order. - Uses cache efficiently Parallel sorting parallel_sort Parallel function invocation parallel_invoke - Parallel execution of a number of userspecified functions Computational graph flow::graph - Implements dependencies between tasks - Pass messages between tasks 22

22 C++11 a.k.a. C++0B it s late but not too late. Core language ext. memory consistency model Standard library parallelism can now rely on mem. consistency model What about Pthreads? Not portable*. Lack of memory consistency model 23 * Remember, Intel Architecture supports sequentially consistent atomics efficiently ( strong memory consistency model ).

23 C++11 Core language Lambda expressions, type inference, etc. Range-based for loops TLS, atomics Standard Template Lib. (STL) std::thread, std::async Synchronization (atomics, locks, and conditions) No tasks! struct background { background() : i(0), thread(work, this) {} ~background() { thread.join(); } static void work(background* self) { ++self->i; // do something } int i; std::thread thread; }; // work in the background background task;

24 N Threads M Tasks Thread vs. Task HW view thread Stream of instructions (along with a given state) Hardware resource e.g., core0 User view task It s not so interesting who will be the actual worker. OS view Stores the machine state (registers, etc.) At least one thread per process User code Code that runs in an own process space (separate from the OS kernel) Scheduler

25 Intel Cilk Plus Tasking model with only three keywords Library based as far as Hyperobjects are concerned Really, the Plus does not mean C++... It is for C and C++ programmer Vectorization model Array notation and SIMD functions ( elemental ) Pragmas: almost all are promoted to OpenMP 4.0 Array Notation is accepted in GNU* GCC mainline! 26

26 Intel Cilk Plus: Writing Vector Code Array Notation A[:] = B[:] + C[:]; Elemental Function declspec(vector) float ef(float a, float b) { return a + b; } A[:] = ef(b[:], C[:]); SIMD Directive #pragma simd for (int i = 0; i < N; ++i) { A[i] = B[i] + C[i]; } Auto-Vectorization for (int i = 0; i < N; ++i) { A[i] = B[i] + C[i]; } 27

27 Intel Cilk Plus and OpenMP 4.0 The programmer (i.e. you!) is responsible for correctness Available clauses (both OpenMP and Intel versions) PRIVATE FIRSTPRIVATE LASTPRIVATE --- like OpenMP REDUCTION COLLAPSE (OpenMP 4.0; for nested loops) LINEAR (additional induction variables) SAFELEN (OpenMP 4.0) VECTORLENGTH (Intel only) ALIGNED (OpenMP 4.0) ASSERT (Intel only; vectorize or die! ) 28

28 Intel Cilk Plus: Targeting Multicore void qsort(int* begin, int* end) { if (begin!= end) { int* pivot = end 1; int* middle = std::partition(begin, pivot, std::bind2nd(std::less<int>(), *pivot)); std::swap(*pivot, *middle); cilk_spawn qsort(begin, middle); qsort(middle + 1, end); cilk_sync; } } (std::bind2nd is deprecated in C++11) 29

29 Intel Cilk Plus: Lock Free (Reducers or Hyperobjects) int accumulate(const int* array, std::size_t size) { } reducer_opadd<int> result(0); cilk_for (std::size_t i = 0; i < size; ++i) { result += array[i]; } return result.get_value(); 30

30 Intel Cilk Plus: Array Notation Correspond to vector processing (SIMD) Explicit construct to express vectorization Compiler assumes no aliasing of pointers Synonyms array notation, array section, array slice, vector Syntax [start:size], or [start:size:stride] [:] all elements* * only works for array shapes known at compile-time 31 Intel Confidential

31 Intel Cilk Plus: Notation Example Array notation: y[0:10:10] = sin(x[20:10:2]); Corresponding loop: for (int i = 0, j = 0, k = 20; i < 10; ++i, j += 10, k += 2) { y[j] = sin(x[k]); } 32 Intel Confidential

32 Intel Cilk Plus: Array Operators Most C/C++ operators work with array sections Element-wise operators a[0:10] * b[4:10] (rank and size must match) Scalar expansion a[10:10] * c Assignment and evaluation Evaluation of RHS before assignment a[1:8] = a[0:8] + 1 Parallel assignment to LHS ^ temp! Gather and scatter a[idx[0:1024]] = 0 b[idx[0:1024]] = a[0:1024] c[0:512] = a[idx[0:512:2]] 33

33 Intel Cilk Plus: Array Op. (cont.) Index generation a[:] = sec_implicit_index(rank) Shift operators b[:] = sec_shift (a[:], signed shift_val, fill_val) b[:] = sec_rotate(a[:], signed shift_val) Cast-operation (array dimensionality) e.g., float[100] float[10][10] 34

34 Intel Cilk Plus: Array Reductions Reductions Built-in sec_reduce_add(a[:]), sec_reduce_mul(a[:]) sec_reduce_min(a[:]), sec_reduce_max(a[:]) sec_reduce_min_ind(a[:]) sec_reduce_max_ind(a[:]) sec_reduce_all_zero(a[:]) sec_reduce_all_nonzero(a[:]) sec_reduce_any_nonzero(a[:]) User-defined result sec_reduce (initial, a[:], fn-id) void sec_reduce_mutating(reduction, a[:], fn-id) 35

35 Intel Cilk Plus: SIMD Functions declspec(vector) void kernel(int& result, int a, int b) { result = a + b; } void sum(int* result, const int* a, const int* b, std::size_t size) { cilk_for (std::size_t i = 0; i < size; i += 8) { const std::size_t n = std::min(size - i, 8); kernel(result[i:n], a[i:n], b[i:n]); } } * For example, the remainder could be also handled separately (outside of the loop). 36

36 Intel Threading Runtimes: Summary Open source d Intel Threading Building Blocks Intel Cilk Plus Intel OpenMP* 37

37 Agenda Introduction Parallel Patterns Today What about HPC? Summary 38

38 Intel TBB*: Affinity Partitioner Applicable to parallel_for, parallel_reduce, and parallel_scan void sum(int* result, const int* a, const int* b, std::size_t size) { static affinity_partitioner partitioner; parallel_for( blocked_range<std::size_t>(0, size), [=](const blocked_range<std::size_t>& r) { for (std::size_t i = r.begin(); i!= r.end(); ++i) { result[i] = a[i] + b[i]; } }, partitioner ); } * That s not HPC!!! Well, that s no religion and yes it s C++ and people use it for HPC 39

39 OpenMP 4.0: proc_bind clause Give the system more information about the mapping needed at each parallel region void fft_func(...) { nthr = mkl_domain_get_max_threads(mkl_fft); # pragma omp parallel num_threads(nthr), proc_bind(spread) { // do an FFT } } void blas_func(...) { nthr = mkl_domain_get_max_threads(mkl_blas); # pragma omp parallel num_threads(nthr), proc_bind(close) { // execute a BLAS routine } } Summary Gives you the affinity you re looking for Actual implementations may currently be expensive It is already supported in the Intel Compiler 14.0 You can play with it right away! 40

40 Hybrid Parallelism: MPI + Threading Pinning and Affinity I_MPI_PIN_DOMAIN=socket KMP_AFFINITY=compact,1 Intel Xeon Phi MPI + Offload MPI symmetric model Host/coprocessor only mpiexec.hydra $* \ -host $HOST -n 1 \ -env I_MPI_PIN_DOMAIN=socket \ -env OMP_NUM_THREADS=2 \ -env KMP_AFFINITY=compact,1 \ $ROOT/compile/linux-isc-xeon-mpi${OGL}/tachyon $ROOT/scenes/teapot.dat \ -camfile $ROOT/scenes/teapot.cam \ -nosave \ : \ -host $HOST -n $RNKS \ -env I_MPI_PIN_DOMAIN=socket \ -env OMP_NUM_THREADS=30 \ -env OMP_SCHEDULE=dynamic \ -env KMP_AFFINITY=compact,1 \ $ROOT/compile/linux-isc-xeon-mpi/tachyon $ROOT/scenes/teapot.dat \ -camfile $ROOT/scenes/teapot.cam \ -nosave \ : \ -host mic0 -n $MICR \ -env LD_LIBRARY_PATH=$MIC_LD_LIBRARY_PATH \ -env I_MPI_PIN_DOMAIN=node \ -env OMP_NUM_THREADS=224 \ -env OMP_SCHEDULE=dynamic \ -env KMP_AFFINITY=balanced \ -env KMP_PLACE_THREADS=${MICT}T \ $ROOT/compile/linux-isc-mic-mpi/tachyon $ROOT/scenes/teapot.dat \ -camfile $ROOT/scenes/teapot.cam \ -nosave \ : \ -host mic1 -n $MICR \ -env LD_LIBRARY_PATH=$MIC_LD_LIBRARY_PATH \ -env I_MPI_PIN_DOMAIN=node \ -env OMP_NUM_THREADS=224 \ -env OMP_SCHEDULE=dynamic \ -env KMP_AFFINITY=balanced \ -env KMP_PLACE_THREADS=${MICT}T \ $ROOT/compile/linux-isc-mic-mpi/tachyon $ROOT/scenes/teapot.dat \ -camfile $ROOT/scenes/teapot.cam \ -nosave \ $NULL 41

41 Agenda Introduction Parallel Patterns Today What about HPC? Summary 42

42 Summary Prerequisites for composing and nesting parallelism Tasks in addition to (or instead of) plain threads Runtime dynamic scheduler (nesting!) Challenges Scheduling and load balancing Determinism and reproducibility 43

43

44 Legal Disclaimer & Optimization Notice INFORMATION IN THIS DOCUMENT IS PROVIDED AS IS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO THIS INFORMATION INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. Copyright 2013, Intel Corporation. All rights reserved. Intel, the Intel logo, Xeon, Xeon Phi, Core, VTune, and Cilk are trademarks of Intel Corporation in the U.S. and other countries. Optimization Notice Intel s compilers may or may not optimize to the same degree for non-intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice revision # Copyright 2013, 2012, Intel Corporation. All rights reserved.