Parallel Programming Models & Their Corresponding HW/SW Implementations

Transcription

1 Lecture 3: Parallel Programming Models & Their Corresponding HW/SW Implementations Parallel Computer Architecture and Programming

2 Today s theme is a critical idea in this course. And today s theme is: Abstraction vs. implementation Conflating abstraction with implementation is a common cause for confusion in this course.

3 An example: Programming with ISPC

4 ISPC Intel SPMD Program Compiler (ISPC) SPMD: single *program* multiple data

5 Recall: example program from last class Compute sin(x) using Taylor expansion: sin(x) = x - x 3 /3! + x 5 /5! - x 7 /7! +... for each element of an array of N floating-point numbers void sinx(int N, int terms, float* x, float* result) for (int i=0; i<n; i++) float value = x[i]; float numer = x[i] * x[i] * x[i]; int denom = 6; // 3! int sign = - 1; for (int j=1; j<=terms; j++) value += sign * numer / denom numer *= x[i] * x[i]; denom *= (2*j+2) * (2*j+3); sign *= - 1; result[i] = value;

6 sin(x) in ISPC Compute sin(x) using Taylor expansion: sin(x) = x - x 3 /3! + x 5 /5! - x 7 /7! +... #include sinx_ispc.h int N = 1024; int terms = 5; float* x = new float[n]; float* result = new float[n]; // initialize x here C++ code: main.cpp // execute ISPC code sinx(n, terms, x, result); SPMD programming abstraction: ISPC code: sinx.ispc export void sinx( uniform int N, uniform int terms, uniform float* x, uniform float* result) // assume N % programcount = 0 for (uniform int i=0; i<n; i+=programcount) int idx = i + programindex; float value = x[idx]; float numer = x[idx] * x[idx] * x[idx]; uniform int denom = 6; // 3! uniform int sign = - 1; Call to ISPC function spawns gang of ISPC program instances All instances run ISPC code in parallel Upon return, all instances have completed for (uniform int j=1; j<=terms; j++) value += sign * numer / denom numer *= x[idx] * x[idx]; denom *= (2*j+2) * (2*j+3); sign *= - 1; result[idx] = value;

7 sin(x) in ISPC Compute sin(x) using Taylor expansion: sin(x) = x - x 3 /3! + x 5 /5! - x 7 /7! +... C++ code: main.cpp #include sinx_ispc.h int N = 1024; int terms = 5; float* x = new float[n]; float* result = new float[n]; // initialize x here Sequential execution (C code) Call to sinx() Begin executing programcount instances of sinx() (ISPC code) // execute ISPC code sinx(n, terms, x, result); SPMD programming abstraction: Call to ISPC function spawns gang of ISPC program instances All instances run ISPC code in parallel Upon return, all instances have completed ISPC compiler generates SIMD implementation: Number of instances in a gang is the SIMD width of the hardware (or a small multiple of) ISPC compiler generates binary (.o) with SIMD instructions C++ code links against object file as usual sinx() returns. Completion of ISPC program instances. Resume sequential execution Sequential execution (C code)

8 sin(x) in ISPC Interleaved assignment of elements to instances #include sinx_ispc.h int N = 1024; int terms = 5; float* x = new float[n]; float* result = new float[n]; // initialize x here // execute ISPC code sinx(n, terms, x, result); ISPC Keywords: C++ code: main.cpp programcount: number of simultaneously executing instances in the gang (uniform value) programindex: id of the current instance in the gang. (a non-uniform value: varying ) uniform: A type modifier. All instances have the same value for this variable. Its use is purely an optimization. Not needed for correctness. export void sinx( uniform int N, uniform int terms, uniform float* x, uniform float* result) // assumes N % programcount = 0 for (uniform int i=0; i<n; i+=programcount) int idx = i + programindex; float value = x[idx]; float numer = x[idx] * x[idx] * x[idx]; uniform int denom = 6; // 3! uniform int sign = - 1; ISPC code: sinx.ispc for (uniform int j=1; j<=terms; j++) value += sign * numer / denom numer *= x[idx] * x[idx]; denom *= (2*j+2) * (2*j+3); sign *= - 1; result[idx] = value;

9 sin(x) in ISPC Blocked assignment of elements to instances #include sinx_ispc.h int N = 1024; int terms = 5; float* x = new float[n]; float* result = new float[n]; // initialize x here C++ code: main.cpp // execute ISPC code sinx(n, terms, x, result); ISPC code: sinx.ispc export void sinx( uniform int N, uniform int terms, uniform float* x, uniform float* result) // assume N % programcount = 0 uniform int count = N / programcount; int start = programindex * count; for (uniform int i=0; i<count; i++) int idx = start + i; float value = x[idx]; float numer = x[idx] * x[idx] * x[idx]; uniform int denom = 6; // 3! uniform int sign = - 1; for (uniform int j=1; j<=terms; j++) value += sign * numer / denom numer *= x[idx] * x[idx]; denom *= (j+3) * (j+4); sign *= - 1; result[idx] = value;

10 sin(x) in ISPC Compute sin(x) using Taylor expansion: sin(x) = x - x 3 /3! + x 5 /5! - x 7 /7! +... #include sinx_ispc.h int N = 1024; int terms = 5; float* x = new float[n]; float* result = new float[n]; // initialize x here C++ code: main.cpp // execute ISPC code sinx(n, terms, x, result); ISPC code: sinx.ispc export void sinx( uniform int N, uniform int terms, uniform float* x, uniform float* result) foreach (i = 0... N) float value = x[i]; float numer = x[i] * x[i] * x[i]; uniform int denom = 6; // 3! uniform int sign = - 1; foreach: key language construct Declares parallel loop iterations ISPC assigns iterations to program instances in gang (ISPC implementation will perform static assignment, but nothing in the abstraction prevents a dynamic assignment) for (uniform int j=1; j<=terms; j++) value += sign * numer / denom numer *= x[i] * x[i]; denom *= (2*j+2) * (2*j+3); sign *= - 1; result[idx] = value;

11 ISPC: abstraction vs. implementation Single program, multiple data (SPMD) programming model - This is the programming abstraction - Program is written in terms of this abstraction Single instruction, multiple data (SIMD) implementation - ISPC compiler emits vector instructions (SSE4 or AVX) - Handles mapping of conditional control flow to vector instructions Semantics of ISPC can be tricky - SPMD abstraction + uniform values (allows implementation details to peak through abstraction a bit)

12 ISPC discussion: sum reduction Compute the sum of all array elements in parallel export uniform float sumall1( uniform int N, uniform float* x) uniform float sum = 0.0f; foreach (i = 0... N) sum += x[i]; return sum; export uniform float sumall2( uniform int N, uniform float* x) uniform float sum; float partial = 0.0f; foreach (i = 0... N) partial += x[i]; // from ISPC math library sum = reduceadd(partial); return sum; sum is of type uniform float (one copy of variable for all program instances) x[i] is uniform expression (different value for each program instance) Result: compile-time type error

13 ISPC discussion: sum reduction Compute the sum of all array elements in parallel Each instance accumulates a private partial sum (no communication) Partial sums are added together using the reduceadd() cross-instance communication primitive. The result is the same for all instances (uniform) export uniform float sumall2( uniform int N, uniform float* x) uniform float sum; float partial = 0.0f; foreach (i = 0... N) partial += x[i]; ISPC code at right will execute in a manner similar to handwritten C + AVX intrinsics implementation below. * const int N = 1024; float* x = new float[n]; mm256 partial = _mm256_broadcast_ss(0.0f); // from ISPC math library sum = reduceadd(partial); return sum; // populate x for (int i=0; i<n; i+=8) partial = _mm256_add_ps(partial, _mm256_load_ps(&x[i])); float sum = 0.f; for (int i=0; i<8; i++) sum += partial[i]; * If you understand why this implementation complies with the semantics of the ISPC gang abstraction, then you ve got good command of ISPC.

14 ISPC tasks The ISPC gang abstraction is implemented by SIMD instructions on one core. So... all the code I ve shown you in the previous slides would have executed on only one of the four cores of the GHC 5205 machines. ISPC contains another abstraction: a task that is used to achieve multi-core execution. I ll let you read up about that.

15 Today Three parallel programming models - Abstractions presented to the programmer - Influence how programmers think when writing programs Three machine architectures - Abstraction presented by the hardware to low-level software - Typically reflect implementation Focus on differences in communication and cooperation

16 System layers: interface, implementation, interface,... Parallel Applications Abstractions for describing concurrent, parallel, or independent computation Abstractions for describing communication Programming model (way of thinking about things) Compiler and/or parallel runtime Language or library primitives/mechanisms OS system call API Operating system support Micro-architecture (hardware implementation) Hardware Architecture (HW/SW boundary) Blue italic text: abstraction/concept Red italic text: system interface Black text: system implementation

17 Example: expressing parallelism with pthreads) Parallel Application Abstraction for describing parallel computation: thread Programming model pthread_create() pthread library implementation System call API OS support: kernel thread management x86-64 modern multi-core CPU Blue italic text: abstraction/concept Red italic text: system interface Black text: system implementation

18 Example: expressing parallelism (ISPC) Parallel Applications Abstractions for describing parallel computation: 1. For specifying simultaneous execution (true parallelism) 2. For specifying independent work (potentially parallel) Programming model ISPC language (call ISPC function, foreach construct) ISPC compiler System call API OS support x86-64 (including AVX vector instructions) single-core of CPU Note: ISPC has the task language primitive for multi-core execution (not discussed here)

19 Three models of communication (abstractions) 1. Shared address space 2. Message passing 3. Data parallel

20 Shared address space model (abstraction) Threads communicate by reading/writing to shared variables Shared variables are like a big bulletin board - Any thread can read or write Thread 1 x Memory shared Thread 2 between threads Thread 1: Thread 2: int x = 0; x = 1; int x; while (x == 0) print x;

21 Shared address space model (abstraction) Threads communicate by: - Reading/writing to shared variables - Interprocessor communication is implicit in memory operations - Thread 1 stores to X. Later, thread 2 reads X (observes update) - Manipulating synchronization primitives - e.g., mutual exclusion using locks Natural extension of sequential programming model - In fact, all our discussions have assumed a shared address space so far Think: shared variables are like a big bulletin board - Any thread can read or write

22 Shared address space (implementation) Option 1: threads share an address space (all data is sharable) Option 2: each thread has its own virtual address space, shared portion of address spaces maps to same physical location Virtual address spaces Physical mapping Image credit: Culler, Singh, and Gupta

23 Shared address space HW implementation Any processor can directly reference any memory location Dance-hall organization Interconnect examples Processor Processor Processor Processor Processor Processor Processor Processor Bus Local Cache Local Cache Local Cache Local Cache Memory Memory Interconnect Processor Crossbar Processor Processor Processor Processor Memory I/O Processor Processor Processor Memory Memory Memory Memory Memory Memory Multi-stage network Symmetric (shared-memory) multi-processor (SMP): - Uniform memory access time: cost of accessing an uncached* memory address is the same for all processors (* caching introduces non-uniform access times, but we ll talk about that later)

24 Shared address space architectures Commodity x86 examples Memory Intel Core i7 (quad core) (network is a ring) Memory Controller Core 1 Core 2 On chip network Core 3 Core 4 AMD Phenom II (six core)

25 SUN Niagara 2 Note size of crossbar: about die area of one core Processor Processor L2 cache Memory Processor Processor Processor Crossbar Switch L2 cache L2 cache Memory Memory Processor Processor L2 cache Memory Eight cores Processor

26 Non-uniform memory access (NUMA) All processors can access any memory location, but... cost of memory access (latency or bandwidth) is different for different processors Processor Processor Processor Processor Local Cache Local Cache Local Cache Local Cache Memory Memory Memory Memory Interconnect Problem with preserving uniform access time: scalability - GOOD: costs are uniform, BAD: but memory is uniformly far away NUMA designs are more scalable - High bandwidth to local memory; BW scales with number of nodes if most accesses local - Low latency access to local memory Increased programmer effort: performance tuning - Finding, exploiting locality

27 Non-uniform memory access (NUMA) Example: latency to access location x is higher from cores 5-8 than cores 1-4 Example: modern dual-socket configuration X Memory Memory Memory Controller Core 1 Core 2 Memory Controller Core 5 Core 6 On chip network Core 3 Core 4 Core 7 Core 8 AMD Hyper-transport / Intel QuickPath

28 SGI Altix UV 1000 (PSC s Blacklight) 256 blades, 2 CPUs per blade, 8 cores per CPU = 4096 cores Single shared address space Interconnect: fat tree Fat tree Image credit: Pittsburgh Supercomputing Center

29 Shared address space summary Communication abstraction - Threads read/write shared variables - Manipulate synchronization primitives: locks, semaphors, etc. - Logical extension of uniprocessor programming - But NUMA implementation requires reasoning about locality for perf Hardware support to make implementations efficient - Any processor can load and store from any address - NUMA designs more scalable than uniform memory access - Even so, costly to scale (see cost of Blacklight)

30 Message passing model (abstraction) Threads operate within independent address spaces Threads communicate by sending/receiving messages - Explicit, point-to-point - send: specifies buffer to be transmitted, recipient, optional message tag - receive: specifies buffer to store data, sender, and (optional) message tag - Messages may be synchronous or asynchronous send(x, 2, tag) match! recv(y, 1, tag) Address Y Address X Thread 1 address space Thread 2 address space Image credit: Culler, Singh, and Gupta

31 Message passing (implementation) Popular library: MPI (message passing interface) Challenges: buffering messages (until application initiates receive), minimizing cost of memory copies Hardware need not implement system-wide loads and stores - Connect complete (often commodity) systems together - Parallel programs for clusters! Cluster of workstations (Infiniband network) IBM Blue Gene/P Supercomputer Image credit: IBM

32 Correspondence between programming models and machine types is fuzzy Common to implement message passing abstractions on machines that support a shared address space in hardware Can implement shared address space abstraction on machines that do not support it in HW (via less efficient SW solution) - Mark all pages with shared variables as invalid - Page-fault handler issues appropriate network requests Keep in mind what is the programming model (abstractions used to specific program) and what is the HW implementation

33 The data-parallel model

34 Data-parallel model Rigid computation structure ff Historically: same operation on each element of an array - Matched capabilities of 80 s SIMD supercomputers - Connection Machine (CM-1, CM-2): thousands of processors, one instruction - And also Cray supercomputer vector processors - Add(A, B, n) this was one instruction on vectors A, B of length n Matlab is another good example: A + B (A, B are vectors of same length) Today: often takes form of SPMD programming - map(function, collection) - Where function may be a complicated sequence of logic (e.g., a loop body) - Application of function to each element of collection is independent - In pure form: no communication between iterations of map - Synchronization is implicit at the end of the map

35 Data parallelism in ISPC // main C++ code: const int N = 1024; float* x = new float[n]; float* y = new float[n]; // initialize N elements of x here absolute_value(n, x, y); Think of loop body as function (from previous slide) foreach construct is a map Collection is implicitly defined by array indexing logic // ISPC code: export void absolute_value( uniform int N, uniform float* x, uniform float* y) foreach (i = 0... N) if (x[i] < 0) y[i] = - x[i]; else y[i] = x[i];

36 Data parallelism in ISPC // main C++ code: const int N = 1024; float* x = new float[n/2]; float* y = new float[n]; // initialize N/2 elements of x here absolute_repeat(n/2, x, y); Think of loop body as function foreach construct is a map Collection is implicitly defined by array indexing logic // ISPC code: export void absolute_repeat( uniform int N, uniform float* x, uniform float* y) foreach (i = 0... N) if (x[i] < 0) y[2*i] = - x[i]; else y[2*i] = x[i]; y[2*i+1] = y[2*i]; Also a valid program! Takes absolute value of elements of x, repeats them twice in output vector y

37 Data parallelism in ISPC // main C++ code: const int N = 1024; float* x = new float[n]; float* y = new float[n]; // initialize N elements of x Think of loop body as function foreach construct is a map Collection is implicitly defined by array indexing logic shift_negative(n, x, y); // ISPC code: export void shift_negative( uniform int N, uniform float* x, uniform float* y) foreach (i = 0... N) if (i > 1 && x[i] < 0) y[i- 1] = - x[i]; else y[i] = x[i]; This program is non-deterministic! Possibility for multiple iterations of the loop body to write to same memory location Data-parallel model provides no specification of order in which iterations occur primitives for fine-grained mutual exclusion/ synchronization)

38 Data parallelism the more formal way Note: this is not ISPC syntax // main program: const int N = 1024; stream<float> x(n); // define collection stream<float> y(n); // define collection // initialize N elements of x here // map absolute_value onto x, y absolute_value(x, y); Data-parallelism expressed in this functional form is sometimes referred to as the stream programing model Streams: collections of elements. Elements can be processed independently // kernel definition void absolute_value( float x, float y) if (x < 0) y = - x; else y = x; Kernels: side-effect-free functions. Operate element-wise on collections Think of kernel inputs, outputs, temporaries for each invocation as a private address space

39 Stream programming benefits // main program: const int N = 1024; stream<float> input(n); stream<float> output(n); stream<float> tmp(n); foo(input,tmp); bar(tmp, output); Functions really are side-effect free! (cannot write a non-deterministic program) Program data flow is known: Predictable data access facilitates prefetching. Inputs and outputs of each invocation are known in advance: prefetching can be input tmp output employed to hide latency. foo bar Producer-consumer locality. Can structure code so that outputs of first kernel feed immediately into second kernel. Values are never written to memory! Save bandwidth! These optimizations are responsibility of stream program compiler. Requires sophisticated compiler analysis.

40 Stream programming drawbacks // main program: const int N = 1024; stream<float> input(n/2); stream<float> tmp(n); stream<float> output(n); stream_repeat(2, input, tmp); absolute_value(tmp, output); Need library of ad-hoc operators to describe complex data flows. (see use of repeat operator at left to obtain same behavior as indexing code below) Cross fingers and hope compiler generates code intelligently Kayvon s experience: This is the achilles heel of all proper dataparallel/stream programming systems. If I just had one more operator... // ISPC code: export void absolute_value( uniform int N, uniform float* x, uniform float* y) foreach (i = 0... N) if (x[i] < 0) y[2*i] = - x[i]; else y[2*i] = x[i]; y[2*i+1] = y[2*i];

41 Gather/scatter: Two key data-parallel communication primitives Map absolute_value onto stream produced by gather: // main program: const int N = 1024; stream<float> input(n); stream<int> indices; stream<float> tmp_input(n); stream<float> output(n); Map absolute_value onto stream, scatter results: // main program: const int N = 1024; stream<float> input(n); stream<int> indices; stream<float> tmp_output(n); stream<float> output(n); stream_gather(input, indices, tmp_input); absolute_value(tmp_input, output); absolute_value(input, tmp_output); stream_scatter(tmp_output, indices, output); (ISPC equivalent) export void absolute_value( uniform float N, uniform float* input, uniform float* output, uniform int* indices) foreach (i = 0... n) float tmp = input[indices[i]]; if (tmp < 0) output[i] = - tmp; else output[i] = tmp; (ISPC equivalent) export void absolute_value( uniform float N, uniform float* input, uniform float* output, uniform int* indices) foreach (i = 0... n) if (input[i] < 0) output[indices[i]] = - input[i]; else output[indices[i]] = input[i];

42 Gather instruction: gather(r1, R0, mem_base); Gather from buffer mem_base into R1 according to indices specified by R0. Array in memory: base address = mem_base Index vector: R0 Result vector: R1 SSE, AVX instruction sets do not directly support SIMD gather/scatter (must implement as scalar loop) Gather coming with AVX 2 in 2013 Hardware supported gather/scatter does exist on GPUs. (still an expensive operation)

43 Data-parallel model summary Data-parallelism is about program structure In spirit, map a single program onto a large collection of data - Functional: side-effect free execution - No communication among invocations In practice that s how most programs work But... most popular languages do not enforce this - OpenCL, CUDA, ISPC, etc. - Choose flexibility/familiarity of imperative syntax over safety and complex compiler optimizations required for functional syntax - It s been their key to success (and the recent adoption of parallel programming) - Hear that PL folks! (functional thinking is great, but lighten up!)

44 Three parallel programming models Shared address space - Communication is unstructured, implicit in loads and stores - Natural way of programming, but can shoot yourself in the foot easily - Program might be correct, but not scale Message passing - Structured communication as messages - Often harder to get first correct program than shared address space - Structure often helpful in getting to first correct, scalable program Data parallel - Structure computation as a big map - Assumes a shared address space from which to load inputs/store results, but severely limits communication between iterations of the map (goal: preserve independent processing of iterations) - Modern embodiments encourage, but don t enforce, this structure

45 Modern trend: hybrid programming models Shared address space within a multi-core node of a cluster, message passing between nodes - Very, very common - Use convenience of shared address space where it can be implemented efficiently (within a node) Data-parallel programming models support synchronization primitives in kernels (CUDA, OpenCL) - Permits limited forms of communication CUDA/OpenCL use data-parallel model to scale to many cores, but adopt shared-address space model allowing threads running on the same core to communicate.

46 Los Alamos National Laboratory: Roadrunner Fastest computer in the world in 2008 (no longer true) 3,240 node cluster. Heterogeneous nodes. Cluster Node IBM Cell CPU IBM Cell CPU IBM Cell CPU IBM Cell CPU AMD CPU Core 1 Core 2 Core 3 Core 4 Core 1 Core 2 Core 3 Core 4 Core 1 Core 2 Core 3 Core 4 Core 1 Core 2 Core 3 Core 4 Core 1 Core 2 16 GB Memory Core 5 Core 6 Core 5 Core 6 Core 5 Core 6 Core 5 Core 6 AMD CPU (Address Space) Core 7 Core 8 Core 7 Core 8 Core 7 Core 8 Core 7 Core 8 Core 1 Core 2 4 GB Memory 4 GB Memory 4 GB Memory 4 GB Memory (Address space) (Address space) (Address space) (Address space) Network

47 Summary Programming models provide a way to think about parallel programs. They provide abstractions that admit many possible implementations. But restrictions imposed by abstractions are designed to reflect realities of hardware communication costs - Shared address space machines - Messaging passing machines - It is desirable to keep abstraction distance low so programs have predictable performance, but want it high enough for code flexibility/portability In practice, you ll need to be able to think in a variety of ways - Modern machines provide different types of communication at different scales - Different models fit the machine best at the various scales - Optimization may require you to think about implementations, not just abstractions