A taxonomy of hardware parallelism

Transcription

1 GPU Programming A taxonomy of hardware parallelism Christian Lessig 1

2 Parallel programming [Serial] algorithms have improved faster than clock over the last 15 years. [Parallel] computers are unlikely to be able to take advantage of these advances because they require new programs and new algorithms. Gordon Bell (1992) G. Bell, Massively parallel computers: why not parallel computers for the masses?, in The Fourth Symposium on the Frontiers of Massively Parallel Computation, 1992, pp

3 Parallel programming [Serial] algorithms have improved faster than clock over the last 15 years. [Parallel] computers are unlikely to be able to take advantage of Even worse: You have to know your architecture! these advances because they require new programs and new algorithms. Gordon Bell (1992) G. Bell, Massively parallel computers: why not parallel computers for the masses?, in The Fourth Symposium on the Frontiers of Massively Parallel Computation, 1992, pp

4 Instruction level parallelism Out-of-order execution: processor extracts parallelim from instruction stream D. R. Kaeli, P. Mistry, D. Schaa, and D. P. Zhang, Heterogeneous computing with OpenCL 2.0, Ch. 2. 4

5 Instruction level parallelism Out-of-order execution: processor extracts parallelim from instruction stream D. R. Kaeli, P. Mistry, D. Schaa, and D. P. Zhang, Heterogeneous computing with OpenCL 2.0, Ch. 2. 5

6 Instruction level parallelism Very long instruction word processors: compiler extracts parallelim from program code D. R. Kaeli, P. Mistry, D. Schaa, and D. P. Zhang, Heterogeneous computing with OpenCL 2.0, Ch. 2. 6

7 Flynn s classification data instruction stream 7

8 Flynn s classification 8

9 Flynn s classification 9

10 Flynn s classification 10

11 Flynn s classification 11

12 Flynn s classification existing parallel architectures 12

13 Distributed memory Scales to arbitrary number of processors Communication via message passing e.g. MPI commons/4/40/beowulf.png High latency and limited bandwidth Used in super-computers 13

14 Shared memory Limited to less than 100 processors Communication via shared memory Low latency and high bandwidth Access to shared resources needs to be synchronized 14

15 Parallel Architectures Distributed memory Shared memory

16 Parallel Architectures Distributed memory Shared memory high latency, low bandwidth + low latency, high bandwidth 16

17 Parallel Architectures Distributed memory Shared memory high latency, low bandwidth + large number of processors + low latency, high bandwidth limited number of processors 17

18 Taxonomy instruction stream single cluster multiple explicit multi-core implicit single-core gpu shared distributed memory 18

19 Taxonomy instruction stream single single-core cluster multiple explicit multi-core implicit gpu shared distributed memory 19

20 Taxonomy instruction stream single single-core cluster multi-core multiple explicit implicit gpu shared distributed memory 20

21 Taxonomy instruction stream single single-core multi-core multiple implicit explicit cluster gpu shared distributed memory 21

22 Taxonomy instruction stream single single-core gpu multi-core multiple implicit explicit cluster shared distributed memory 22

23 Further reading J. L. Hennessy and D. A. Patterson, Computer architecture: a quantitative approach, fourth edition. Morgan Kaufmann,