Shared-Memory Programming Models

Transcription

1 Shared-Memory Programming Models Parallel Programming Concepts Winter Term 2013 / 2014 Dr. Peter Tröger, M.Sc. Frank Feinbube

2 Cilk C language combined with several new keywords Different approach to OpenMP pragmas Developed at MIT since 1994 (!) Initial commercial version Cilk++ with C / C++ support Since 2010, offered by Intel as Cilk Plus Official language specification to foster other implementations Meanwhile maintained as GCC branch (similar to OpenMP) Support for Windows, Linux, and MacOS X Basic concept of serialization Any Cilk program compiled as concurrent code has the same execution semantics as the serial version

3 Intel Cilk Plus 3 Three keywords to express potential parallelism cilk_spawn: Asynchronous function call Runtime decides, spawning is not mandated cilk_for: Allows loop iterations to be performed in parallel Runtime decides, parallelization is not mandated cilk_sync: Wait until all spawned calls are completed Barrier for cilk_spawn activity Runtime decided the level of parallelism, performs work stealing Strand: Instruction sequence in-between a change of parallelism Reducers: Lock-free private views on variables Notation for SIMD array operations and SIMD functions Serialization: Cilk keyword become ordinary statements, code semantics are expected to remain the same

4 Intel Cilk Plus 4 Strand concept makes it possible to express every program as directed acyclic graph (DAG) cilk_spawn fib(n-2) fib(n-1) Strand Strand [cilkplus.org] cilk_sync Strand return x+y Implicit cilk_sync Continuation / Strand

5 Intel Cilk Plus 5 [cilkplus.org]

6 Intel Cilk Plus 6 Accumulator / reduction algorithm Compute one result value by updating it with every computational step (that may be parallelized) Same reduction concept as with OpenMP and others Problem of avoiding data races [software.intel.com]

7 Intel Cilk Plus 7 Express accumulated result as reducer pointer variable to get automated locking Parallel reducer operations are promised to be in serial ordering [software.intel.com]

8 Intel Cilk Plus 8 Express accumulated result as reducer pointer variable to get automated locking Parallel reducer operations are promised to be in serial ordering [software.intel.com]

9 Intel Cilk Plus 9 Parallel tree search Resulting list is always in-order Left subtree Root Right subtree Stable semantics regardless of parallelization [software.intel.com]

10 Intel Cilk Plus 10 Predefined reducers for C and C++, custom reducers supported Optimized internal operation based on strands concept Each strand gets a private view on the reducer variable No locking during update When strands join again, the reducer merges the operations

11 Intel Cilk Plus 11 Cilk support the high-level expression of array operations Gives the runtime a chance to parallelize work Intended for data parallel element operations without any ordering constraints New operator [:] Specify data parallelism on an array array-expression[lowerbound : length : stride] Multi-dimensional sections are supported: a[:][:] Short-hand description for complex loops A[:]=5 for (i = 0; i < 10; i++) A[i] = 5; A[0:n] = 5; A[0:5:2] = 5; for (i = 0; i < 10; i += 2) A[i] = 5; A[:] = B[:]; A[:] = B[:] + 5; D[:] = A[:] + B[:]; func (A[:]);

12 Intel Cilk Plus 12 Array notation can be used inside conditions if (5 == a[:]) results[:] = "Matched ; else results[:] = "Not Matched"; Function mapping is executed in parallel with no specific order A[:] = pow(b[:], c); In C++, this works with any defined operator A[:] = B[:] + C[:]; // A[:] = operator+(b[:], C[:]); Several predefined reduction macros applicable to array sections sec_reduce_add, sec_reduce_mul, sec_reduce_max, sec_reduce_min, sec_reduce_all_zero, sec_reduce_any_zero Array sections can be used as array indices for gather / scatter C[:] = A[B[:]] (gather), A[B[:]] = C[:] (scatter)

13 Intel Threading Building Blocks (TBB) 13 Portable C++ library, toolkits for different operating systems Also available as open source version Complements basic OpenMP / Cilk features Loop parallelization, parallel reduction, synchronization, explicit tasks High-level concurrent containers hash map, queue, vector, set High-level parallel operations prefix scan, sorting, data-flow pipelining, deterministic reduce Unfair scheduling approach, to favor threads having data in cache Supported for cache-aware memory allocation Comparable: Microsoft C++ Concurrency Runtime

14 Intel Math Kernel Library (MKL) 14 Intel library with hand-optimized functions for... Highly vectorized and threaded linear algebra Basic Linear Algebra Subprograms (BLAS) API, confirms to de-facto standards in high-performance computing Vector-vector, matrix-vector, matrix-matrix operations Fast fourier transforms (FFT) Single precision, double precision, complex, real,... Vector math and statistics functions Random number generators and probability distributions Spline-based data fitting C or Fortran API calls Beats any automated compiler optimization

15 Easy Mappings [Dig] 15 Oracle Java Intel TBB MS.Net TPL Parallel For ParallelArray parallel_for Parallel.For Concurrent Collections ConcurrentHashMap,... concurrent_hash_map,... Atomic Classes AtomicInteger,... atomic<t> Interlocked ForkJoin Task Parallelism ForkJoinTask framework task Task, ReplicableTask

16 Lock-Free Programming 16 Lock-free programming as a way of sharing data without maintaining locks Prevents deadlock and live-lock conditions Goal: Suspension of one thread never prevents another thread from making progress (e.g. synchronized shared queue) Blocking by design does not disqualify the lock-free realization Algorithms rely on hardware support for atomic operations Read-Modify-Write (RMW) operations Compare-And-Swap (CAS) operations These operations are typically mapped in operating system API

17 Lock-Free Programming 17 void LockFreeQueue::push(Node* newhead) { for (;;) { // Copy a shared variable (m_head) to a local. Node* oldhead = m_head; // Do some speculative work, not yet visible to other threads. newhead->next = oldhead; // Next, attempt to publish our changes to the shared variable. // If the shared variable hasn't changed, the CAS succeeds and we return. // Otherwise, repeat. if (_InterlockedCompareExchange(&m_Head, newhead, oldhead) == oldhead) return; } }#

18 Sequential Consistency 18 Consistency model where the order of memory operations is consistent with the source code Important for lock-free algorithm semantic Not guaranteed by some processor architectures (e.g. PowerPC) Java and C++ support the enforcement of sequential consistency std::atomic<int> X(0), Y(0); int r1, r2; void thread1() { X.store(1); r1 = Y.load(); } void thread2() { Y.store(1); r2 = X.load(); }# r1 and r2 never become zero at the same time Compiler generates additional memory fences and RMW operations Still does not prevent from memory re-ordering due to instruction re-ordering by the compiler itself

19 Functional Programming 19 Programming paradigm that treats execution as function evaluation -> map some input to some output Contrary to imperative programming No longer focus on statement execution for state modification Programmer no longer specifies control flow explicitly Side-effect free computation through avoidance of local state -> referential transparency (no demand for some control flow) Typically strong focus on immutable data as language default -> instead of altering values, return altered copy One foundation: Alonzo Church s lambda calculus from the 1930 s First functional language was Lisp (late 50s) Trend to add functional programming features into imperative languages

20 Imperative to Functional 20 alert("i'd like some Spaghetti!");# alert("i'd like some Chocolate Moose!");# Optimize function SwedishChef( food ) # {# alert("i'd like some " + food + "!");# }# SwedishChef("Spaghetti");# SwedishChef("Chocolate Moose");# # alert("get the lobster");# PutInPot("lobster");# PutInPot("water");# alert("get the chicken");# BoomBoom("chicken");# BoomBoom("coconut");# Optimize function Cook( i1, i2, f ) {# alert("get the " + i1);# f(i1); f(i2); } # # Cook( "lobster", "water", PutInPot);# Cook( "chicken", "coconut", BoomBoom): # Anonymous Function function Cook( i1, i2, f ) {# alert("get the " + i1);# f(i1); f(i2); } # # Cook( "lobster", "water", # function(x) { alert("pot " + x); } );# Cook( "chicken", "coconut", # function(x) { alert("boom " + x); } );#

21 Imperative to Functional 21 map() does not demand particular operation ordering var a = [1,2,3];# for (i=0; i<a.length; i++) {# a[i] = a[i] * 2; # } # # for (i=0; i<a.length; i++) {# alert(a[i]); # }# function map(fn, a)# {# for (i = 0; i < a.length; i++)# {# a[i] = fn(a[i]);# }# }# map( function(x){return x*2;}, a );# map( alert, a );#

22 Imperative to Functional 22 function sum(a) {# var s = 0;# for (i = 0; i < a.length; i++)# s += a[i];# return s;# }# function join(a) {# var s = "";# for (i = 0; i < a.length; i++)# s += a[i];# return s; }# alert(sum([1,2,3])); # alert(join(["a","b","c"]));# map() and reduce() functions do not demand particular operation ordering function reduce(fn, a, init){# var s = init;# for (i = 0; i < a.length; i++)# s = fn( s, a[i] );# return s;# }# # function sum(a){# return reduce( function(a, b){ return a + b; }, a, 0 );# }# # function join(a){# return reduce( function(a, b){ return a + b; }, a, "" );# }#

23 Imperative to Functional - Python 23 # Nested loop procedural style for finding big products xs = (1,2,3,4) ys = (10,15,3,22) bigmuls = [] for x in xs: for y in ys: if x*y > 25: bigmuls.append((x,y)) print bigmuls [David Merz] print [(x,y) for x in (1,2,3,4) for y in (10,15,3,22) if x*y > 25]

24 Functional Programming 24 Higher order functions: Functions as argument or return value Pure functions: No memory or I/O side effects If the result of a pure expression is not used, it can be removed A pure function called with side-effect free parameters has a constant result Without data dependencies, pure functions can run in parallel A language with only pure function semantic can change evaluation order Functions with side effects (e.g. printing) typically do not return results Recursion as replacement for looping (e.g. factorial) Lazy evaluation possible, e.g. to support infinite data structures Perfect foundation for implicit parallelism...