Advanced Shared-Memory Programming

Transcription

1 Advanced Shared-Memory Programming Parallel Programming Concepts Winter Term 2013 / 2014 Dr. Peter Tröger, M.Sc. Frank Feinbube

2 Shared-Memory Parallelism 2 Libraries and language extensions for standard languages OpenMP for C / C++ Java /.NET concurrency functionality Cilk derivate of C++ Different levels of abstractions Process model, thread model, task model Specialized programming languages for NUMA systems: Partitioned Global Address Space (PGAS)... for computational applications: Fortran / HPF... for implicit parallelism: Functional languages Profiling

3 PGAS Languages 3 Typically, the developer is shielded from memory hierarchy aspects through the operating system (PRAM thinking) Partitioned global address space (PGAS) approach Driven by high-performance computing community Modern approach for large-scale NUMA Precondition for Exascale computing Explicit notion of memory partition per processor Data is designated as local (near) or global (possibly far) Programmer is aware of NUMA nodes and can handle data and task placement explicitly Only way to allow performance optimization in systems with deep memory hierarchies

4 PGAS Libraries 4 PGAS languages Unified Parallel C (Ansi C) Co-Array Fortran / Fortress (F90) Titanium (Java), Chapel (Cray), X10 (IBM), All research, no wide-spread solution on industry level Core data management functionality can be re-used as library Global-Address Space Networking (GASNet) Used by many PGAS languages - UPC, Co-Array Fortran, Titanium, Chapel Aggregate Remote Memory Copy Interface (ARMCI) Blocking / non-blocking API, MPI compatibility Kernel Lattice Parallelism (KeLP) C++ class framework based on MPI

5 Unified Parallel C (UPC) 5 Extension of C for HPC on large-scale supercomputers Initiative started 1996 at the University of Berkeley Meanwhile consortium for language specification Latest language specification v1.3 from 2013 Extension of ISO C 99 with An explicitly parallel execution model An explicit shared memory consistency model Synchronization primitives Memory management primitives Support for major HPC platforms (Tru64, HP-UX, Cray, Altix, ) Support from commercial debuggers

6 Unified Parallel C (UPC) 6 Execution environment Each UPC program consists of a set of threads SPMD execution of UPC threads with flexible placement Threads may allocate shared and private data Access to this data can be local or shared Shared access can be strict or relaxed with respect to memory consistency Strict-only access to variables leads to sequential consistency Relaxed operations are finished before a strict operation

7 Unified Parallel C 7 Data is by default private, exists as copy per thread New data type qualifier shared for shared thread data Shared data has affinity for a particular UPC thread Primitive / pointer / aggregates: Affinity with UPC thread 0 Array type: cyclic affinity per element, block-cyclic affinity, partitioning Pointers to shared data consist of thread ID, local address, and position #include <upc_relaxed.h> #define N 100*THREADS shared int v1[n], v2[n], v1plusv2[n]; void main() { int i; for(i=0; i<n; i++) if (MYTHREAD==i%THREADS) v1plusv2[i]=v1[i]+v2[i]; }

8 Unified Parallel C 8 shared int A[100] Distributes array cyclically across all thread memories shared [B] int A[100] Distributes chunks of size B across all threads shared [*] int A[100] Position all array elements in thread 0 Explicit synchronization primitives upc_lock, upc_unlock, upc_lock_attempt, upc_lock_t upc_barrier, upc_notify, upc_wait Collective operations (upc_all_broadcast)

9 Unified Parallel C 9 Each memory reference / statement can be annotated Strict modifier: Sequential consistency (references from the same thread are in order) Relaxed modifier: Issuing thread sees sequential consistency, no other guarantees Manual optimization still needed, but data management is encapsulated in abstract keywords

10 Unified Parallel C 10 Loop parallelization with upc_forall Assignment of field elements to threads must be done explicitly with fourth parameter Identify thread by shared pointer Distribute in round-robin fashion according to fixed number Block-wise assignment shared int a[100], b[100], c[100]; int i; upc_forall(i=0; i<100; i++; &a[i]) a[i]=b[i]*c[i]; upc_forall(i=0; i<100; i++; i) a[i]=b[i]*c[i]; upc_forall(i=0; i<100; i++; (i*threads)/100) a[i]=b[i]*c[i];

11 X10 11 Parallel object-oriented PGAS language by IBM Java derivate, compiles to C++ or pure Java code Different binaries can interact through common runtime X86 and Power support Transport: Shared memory, TCP/IP, MPI, CUDA, Linux, MacOS X, Windows, AIX Full developer support with Eclipse environment Fork-join execution model, instead of SPMD as in MPI One application instance runs at a fixed number of places Each place has a private copy of static variables main() method runs automatically at place 0 Each place typically represents a NUMA node

12 X10 12 Local Heap Global Reference Local Heap Activities Activities Place 0 Place N Parallel tasks, each operating in one place of the PGAS Direct variable access only in local place of the global space Tasks mapped to places, potentially on different machines Implementation One operating system process per place, manages thread pool Work-stealing scheduler, queue with pending async s

13 X10 13 Local Heap Global Reference Local Heap Activities Activities Place 0 Place N async S Creates a new task that executes S, returns immediately S may reference all variables in the enclosing block finish S Execute S and wait for all transitively spawned tasks (barrier) If one task throws an exception, all others are finished first

14 Example 14 public class Fib {! public static def fib(n:int) {! if (n<=2) return 1;! val f1:int;! val f2:int;! finish {! async { f1 = fib(n-1); }! f2 = fib(n-2);! }! return f1 + f2;! }!! public static def main(args:array[string](1)) {! val n = (args.size > 0)? int.parse(args(0)) : 10;! Console.OUT.println("Computing Fib("+n+")");! val f = fib(n);! Console.OUT.println("Fib("+n+") = "+f);! }! }!

15 X10 15 Local Heap Global Reference Local Heap Activities Activities Place 0 Place N atomic S Execute S atomically, with respect to all other atomic blocks S must not create concurrency, access only local data when(c) S Suspend current task until c, then execute S atomically

16 Example 16 class Buffer[T]{T isref,t haszero} { protected var datum:t = null; class Account { public var value:int; def transfer(src:account, v:int) { atomic { src.value -= v; this.value += v; } } } public def send(v:t){v!=null} { when(datum == null) { datum = v; } } public def receive() { when(datum!= null) { val v = datum; datum = null; return v; } } }

17 Tasks Can Move 17 Local Heap Global Reference Local Heap Activities Activities Place 0 Place N at(p) S Execute statement S at place p, block current task at(p) e Evaluate expression e at place p and return the result at(p) async S Create new task at p to run S, return immediately

18 Example 18 class HelloWholeWorld { public static def main(args:rail[string]) { finish for(p in Place.places()) at(p) async Console.OUT.println(p + says + args(0)); Console.OUT.println( Bye ); } } $ x10c++ HelloWholeWorld.x10 $ X10_NPLACES=4./a.out hello Place(0) says hello Place(2) says hello Place(3) says hello Place(1) says hello Bye

19 X10 Object Model 19 Object live in a single place Tasks shift their place, not objects Bring the work to the data, not the other way around Global references can be created and used explicitely val ref:globalref[rail[int]] = GlobalRef(rail); at transparently copies the reachable object graph Compiler identifies reachable parts of the object graph Runtime copies the neccessary data Global references are serialized, not the content Special support for arrays as non-reference type

20 X10 Example: Parallel Sum 20 public class ParaSum {! public static def main(argv:rail[string]!) {! val id = (i:int) => i; // integer identity function! x10.io.console.out.println("sum(i=1..10)i = " + sum(id, 1, 10));! val sq = (i:int) => i*i; // integer square function, inline def. used instead! x10.io.console.out.println("sum(i=1..10)i*i = " + sum((i:int)=>i*i, 1, 10)); }!! public static def sum(f: (Int)=>Int, a:int, b:int):int {! val s = Rail.make[Int](1);! s(0) = 0;! finish {! for(p in Place.places) {! async{ // Spawn async at each place to compute its local range!! val ppartialsum = at(p) sumforplace(f, a, b);! atomic { s(0) += ppartialsum; } // add partial sums! }}}! return s(0) } // return total sum! private static def sumforplace(f: (Int)=>Int, a:int, b:int) {! var accum : Int = 0;! // each processor p of K computes f(a+p.id), f(a+p.id+k), f(a+p.id+2k), etc.! for(var i : Int = here.id + a; i <= b; i += Place.places.length {! accum += f(i); }! return accum;! }}!

21 Fortran 21 Programming language for scientific computing Developed in the 1950 s by IBM FORTRAN 66: First ANSI-standardized version FORTRAN 77: Structured Programming FORTRAN 90: Modular Programming, Array Operations FORTRAN 03: Object-Oriented Programming FORTRAN 08: Concurrent Programming Primary language in high-performance computing [Wikipedia]

22 Fortran 22 [Wikipedia] Array operation

23 High-Performance Fortran (HPF) 23 High-Performance Fortran as Fortran 95 extension Minimal set of extensions for classical Fortran language Data-parallel programming model Expression of parallelism, data distribution and alignment!hpf$ TEMPLATE!HPF$ PROCESSORS data objects!hpf$ ALIGN template!hpf$ DISTRIBUTE abstract processors with grid topology implementation dependent grid mapping physical processors with arbitrary topology Two-level mapping of data objects to abstract processors Array elements are aligned with template structure Template elements are distributed to abstract processors

24 High-Performance Fortran (HPF) 24 PROCESSOR directive Declare rectangular processor arrangement Defined by number of dimensions and extend per direction Final mapping (abstract -> physical) is not part of HPF TEMPLATE directive Template defined by number of dimensions and extends Abstract space of indexed positions Does not occupy memory at run-time Each data structure has a natural template Example: For array, index space that is identical

25 High-Performance Fortran (HPF) 25 HPF compiler directives as structured comments Hints for the compiler No change to the program semantics DISTRIBUTE directive Allows template to be distributed (BLOCK / CYCLIC) Any dimension of the template can be collapsed or replicated on a processor grid!hpf$ PROCESSORS P(4)!HPF$ TEMPLATE T(16)!HPF$ DISTRIBUTE T(BLOCK) ONTO P !HPF$ DISTRIBUTE T(BLOCK(5)) ONTO P !HPF$ DISTRIBUTE T(CYCLIC) ONTO P !HPF$ DISTRIBUTE T(CYCLIC(2)) ONTO P

26 High-Performance Fortran (HPF) 26 ALIGN directive TRANSPOSITION AND STRIDE SHIFT AND STRIDES Specify that some data objects should be mapped as others Intention to optimize operations that act on both Many options: shifts, strides, transposition of indices,... ALIGN and DISTRIBUTE are static data mappings REAL, DIMENSION(12,6) :: A!HPF$ TEMPLATE T(12,12)!HPF$ ALIGN A(I,J) WITH T(2*J-1,I) RELATIVE ALIGNMENT INTEGER, DIMENSION(4,4) :: B!HPF$ TEMPLATE T(12,12)!HPF$ ALIGN B(I,J) WITH T(2:12:3,1:12:3) COLLAPSE Part of subroutine declaration Dynamic versions at runtime REALIGN REDISTRIBUTE REAL, DIMENSION(8,8) :: C,D!HPF$ TEMPLATE T(12,12)!HPF$ ALIGN C(:,:) WITH T(:,:)!HPF$ ALIGN D(I,J) WITH T(I+5,J+5) REAL, DIMENSION(8,12) :: E!HPF$ TEMPLATE T(12)!HPF$ ALIGN(*.:) WITH T(:)

27 27 High-Performance Fortran (HPF)

28 HPF Data Parallelism 28 With parallel array assignments With FORALL statements FORALL(I=1:100,J=1:100) B(I,J)=1.0 FORALL(I=1:100,J=1:100) A(I,J)=B(I,J) FORALL(I=1: 98,J=3:100) A(I,J)=B(I+2,J-2) FORALL(I=1:100,J=1:100,B(I,J).GT.0) A(I,J)=2.*B(I,J) With INDEPENDENT directive Instruct compiler that instructions in the following FORALL statement have no data dependency With PURE functions Functions with syntactical restrictions to produce no sideeffects, mandatory in FORALL body With some of the intrinsic data functions

29 Declarative Programming 29.NET Language Integrated Query (LINQ) General purpose query facility, e.g. for databases or XML Declarative standard query operators var query = from p in products! where p.name.startswith("a")! orderby p.id! select p;!! foreach ( var p in query ) {! Console.WriteLine ( p.name );! }! PLINQ is parallelizing the execution of queries on objects and XML Declarative style of LINQ allows seamless transition to parallel version of the code IEnumerable<T> data =...; var q = data.where(x => p(x)).orderby(x => k(x)).select(x => f(x)); foreach (var e in q) a(e); IEnumerable<T> data =...; var q = data.asparallel().where(x => p(x)).orderby(x => k(x)).select(x => f(x)); foreach (var e in q) a(e);

30 Functional Programming 30 Programming paradigm that treats execution as function evaluation -> map some input to some output Contrary to imperative programming No longer focus on statement execution for state modification Programmer no longer specifies control flow explicitly High-level solution Side-effect free computation through avoidance of local state -> referential transparency (no demand for some control flow) Typically strong focus on immutable data as language default -> instead of altering values, return altered copy One foundation: Alonzo Church s lambda calculus from the 1930 s First functional language was Lisp (late 50s) Trend to add functional programming features into imperative languages (anonymous functions, fiter, map, )

31 Imperative to Functional 31 alert("i'd like some Spaghetti!");! alert("i'd like some Chocolate Moose!");! Optimize function SwedishChef( food )! {! alert("i'd like some " + food + "!");! }! SwedishChef("Spaghetti");! SwedishChef("Chocolate Moose");!! alert("get the lobster");! PutInPot("lobster");! PutInPot("water");! alert("get the chicken");! BoomBoom("chicken");! BoomBoom("coconut");! Optimize function Cook( i1, i2, f ) {! alert("get the " + i1);! f(i1); f(i2); }!! Cook( "lobster", "water", PutInPot);! Cook( "chicken", "coconut", BoomBoom):! Anonymous Function function Cook( i1, i2, f ) {! alert("get the " + i1);! f(i1); f(i2); }!! Cook( "lobster", "water",! function(x) { alert("pot " + x); } );! Cook( "chicken", "coconut",! function(x) { alert("boom " + x); } );!

32 Imperative to Functional 32 map() does not demand particular operation ordering var a = [1,2,3];! for (i=0; i<a.length; i++) {! a[i] = a[i] * 2;! }!! for (i=0; i<a.length; i++) {! alert(a[i]);! }! function map(fn, a)! {! for (i = 0; i < a.length; i++)! {! a[i] = fn(a[i]);! }! }! map( function(x){return x*2;}, a );! map( alert, a );!

33 Imperative to Functional 33 function sum(a) {! var s = 0;! for (i = 0; i < a.length; i++)! s += a[i];! return s;! }! function join(a) {! var s = "";! for (i = 0; i < a.length; i++)! s += a[i];! return s; }! alert(sum([1,2,3]));! alert(join(["a","b","c"]));! map() and reduce() functions do not demand particular operation ordering function reduce(fn, a, init){! var s = init;! for (i = 0; i < a.length; i++)! s = fn( s, a[i] );! return s;! }!! function sum(a){! return reduce( function(a, b){ return a + b; }, a, 0 );! }!! function join(a){! return reduce( function(a, b){ return a + b; }, a, "" );! }!

34 Imperative to Functional - Python 34 # Nested loop procedural style for finding big products xs = (1,2,3,4) ys = (10,15,3,22) bigmuls = [] for x in xs: for y in ys: if x*y > 25: bigmuls.append((x,y)) print bigmuls [David Merz] print [(x,y) for x in (1,2,3,4) for y in (10,15,3,22) if x*y > 25]

35 Functional Programming 35 Higher order functions: Functions as argument or return value Pure functions: No memory or I/O side effects If the result of a pure expression is not used, it can be removed A pure function called with side-effect free parameters has a constant result Without data dependencies, pure functions can run in parallel A language with only pure function semantic can change evaluation order Functions with side effects (e.g. printing) typically do not return results Recursion as replacement for looping (e.g. factorial) Lazy evaluation possible, e.g. to support infinite data structures Perfect foundation for implicit parallelism...

36 Functional Programming 36 Pure functions save state on the stack as function parameters But: Applications must have side effects The application task is to modify state Goal is to limit side-effects and to concentrate them Many popular new functional languages JVM-based: Clojure, Scala (parts of it) Common Lisp, Erlang, F#, Haskell, ML, Ocaml, Scheme

37 Clojure 37 Dynamically typed, functional programming language Derivation from Lisp Runs on JVM >= v5, allows Java interoperability Variations with backend for.net and JavaScript Major goal: Easier development of data-parallel applications Clojure operation: Function, macro, or special form Special forms are implemented by the compiler Typical keywords (new, throw, monitor-enter, if, let,...) Three ways of sharing mutable data in a safe way Refs: Coordinated access by software-transactional memory Atoms: Synchronized access to one data item Agents: Asychronous access to one data item

38 Clojure 38 (Function) Data

39 39 Clojure

40 Fortress (== Secure Fortran ) 40 Oracle / Sun Programming Language Research Group, Guy L. Steele (Scheme, Common Lisp, Java) Language designed for (mathematical) high-performance computing Dynamic compilation, type inference Growable language: Prefer library over compiler Mathematical notation Everything is an expression, some having void value (e.g. while, for, assignment) Source code can be rendered in ASCII, Unicode, or as image Functional programming concepts, but also Scala / Haskell derivations PT 2013

41 Fortress - Comparison to C 41 No memory management, all handled by runtime system Implicit instead of explicit threading Set of types similar to C library Fortress program state: Number of threads + memory Fortress program execution: Evaluation of expressions in all threads Component model supported, interfaces can be imported and exported Components live in the,fortress database, interaction through shell

42 Fortress Syntax 42 Adopt math whenever possible Integer, naturals, rationals, complex number, floating point Support for units and dimensions Everything is an expression, () is the void value Statements are void-type expressions (while, for, assignment, binding) Some statements have non-() values (if, do, try, case, spawn,...) if x 0 then x else -x end atomic x := max(x, y[k]) Generators: j:k - range, j#n - n consecutive integers from j,...

43 Fortress Basics 43 Object: Fields and methods Traits: Set of abstract / concrete methods (extended interface concept) Every object extends a set of traits trait Boolean extends BooleanAlgebra Boolean,,,,,false,true comprises { true, false } opr (self, other: Boolean): Boolean opr (self, other: Boolean): Boolean opr (self): Boolean end object true extends Boolean opr (self, other: Boolean) = other opr (self, other: Boolean) = self opr (self) = false end...

44 Fortress - Functions 44 Functions Static (nat or int) parameters One variable parameter Optional return value Optional body expression Result comes from evaluation of the body do-end expression: Sequence of expressions with implicit parallel execution, last defining the blocks result Supports also do syntax for explicit parallelism do factorial (10) also do factorial (5) also do factorial (2) end

45 Fortress - Parallelism 45 Parallel programming as necessary compromise, not as primary goal Implicit parallelism wherever possible, supported by functional approach Evaluated in parallel: function / method arguments, operator operands, tuple expressions (each element evaluated separately), loop iterations, sums Loop iterations are parallelized Generators generate values in parallel, called functions run in parallel Race condition handling through atomic keyword, explicit spawn keyword for i <- 1:5 do print(i ) print(i ) end for i <- sequential(1:5) do print(i ) print(i ) end

46 Asynchronous Programming? 46 Huge hype around asynchronous programming A model to implement I/O concurrency Explicit avoidance of any threading overhead Focus on fast context switch between many activities Often implemented by event-driven programming style Activities perform callback when they are done Whole API modeled around this idea JavaScript, Node.JS, Python Twisted, Mostly based on libevent wrapper library for /dev/poll, kqueue, epoll, Mainly targets the problem of blocking code on I/O activity No standalone solution for speedup, but helps with scaling

47 Profiling 47 Many available profiling tools for shared-memory parallelism Sampling profiler Sporadic recording of application state Time-driven: Uniform time period between samples Event-driven: Uniform event number between samples Original code remains unchanged Small impact on execution behavior, can find race conditions Low overhead (hardware-based ~ 2%, software-based ~ 5%) Instrumenting profiler Modification of original application with measurement code Allows gathering of all possible events Higher accuracy than sampling, but also higher overhead Data gathering is one part, vizualisation a different one

48 Intel VTune 48 Commercial profiling tool, for command-line or GUI Plugins for Eclipse and Visual Studio Support for Fortran, C, C++, Java,.NET, Assembly Linux and Windows Standard features Hotspot analysis where is the most time spent? Concurrency analysis are all cores well utilized? Lock analysis which locks are hot? User-mode sampling Sampling library is dynamically attached via LD_PRELOAD Set up timer per thread, then issue signal and take sample Hardware-event based sampling

49 Performance Monitoring Unit 49 one PMU per core one PMU in uncore region elapsed cycles L1, L2 cache events processed instructions uncore bound QPI events L3 cache events memory controller events

50 Example 50 Hotspot Analysis (Whetstone) Top-Down View (Call Tree) Bottom-Up View

51 51 Example

52 52 Example

53 53 Example