Parallel Programming. Michael Gerndt Technische Universität München

Transcription

1 Parallel Programming Michael Gerndt Technische Universität München

2 Contents 1. Introduction 2. Parallel architectures 3. Parallel applications 4. Parallelization approach 5. OpenMP 6. Data dependences 7. Program transformations 8. MPI 9. Other programming models 10.Programming tools

3 Organization Slides and narration will be available in the web. Please, be on time for the lecture! Please, ask questions! Please, contribute to the lecture! Exercises are very important!

4 Books for lecture Hennessy, Patterson: Computer Architecture - A quantitative Approach. Morgan Kaufmann, 2011.Standardwerk Patterson, Hennessy: Rechnerorganisation und entwurf, 2008, Standardwerk Tanenbaum: Structured Computer Organization. Pearson Studium, 2013, 6. Auflage, Standardwerk David E. Culler, Jasweinder Pal Singh, Anoop Gupta: Parallel Computer Architecture: A Hardware/Software Approach, Morgan Kaufmann, 1999, ISBN Ian Foster Designing and Building Parallel Programs. MPI and OpenMP Standards Randy Allen, Ken Kennedy, Optimizing Compilers for Modern Architectures: A Dependence-based Approach

5 Lecture Part of CSE and Master Informatics Dates Lecture Wednesday, 8:15-9:45, Interimshörsaal 2 Exercises Central tutorial meeting by Andreas Wilhelm Monday, 16:15-17:45, Interimshörsaal 2 Personal advisory session to be announced Contents Pthreads, OpenMP and MPI Individual programs to be parallelized The exercises are important. Bonus of 0.3, for submission of OWN correct solutions

6 Student presentations Short presentations (10 minutes) TOP 500 systems GPU and Xeon Phi accelerators You can propose a presentation One presentation per lecture Good presentation helps to improve your grade

7 Exam and Grading Exam Final exam will cover theory and exercises Exams are available on the web site. Programming will be more this time. Bonus of +0.3 for a good student presentation Do not forget to register for the exam in TUM-Online

8 HPC Applications

9 Goals of Parallel Computing Reduction of execution time Increased extensibility and configurability Possibly better fault tolerance

10 Performance Goal Speedup speedup( p processors) performance( p processors) performance(1 processor) Scientific computing: performance=work/time Efficiency time(1 processor) speedup( p processors) time( p processors) speedup( p processors ) efficiency( p processors ) p

11 Speedup based on Throughput Performance = throughput = transactions / minute speedup( p tpm( p processor) processors) tpm(1 processors)

12 Parallelism for Performance Processor Bit-level up to 128 Bit Instruction-level: pipelining, functional units,vectorization Latency gets very important, branch-prediction Toleration of latency Memory: multiple memory banks IO: hardware DMA, Raid arrays Multiple processors

13 Introduction to Parallel Architectures

14 Classification Parallel Systems SIMD MIMD Distributed Memory Shared Memory MPP NOW Cluster UMA NUMA ccnuma nccnuma COMA

15 Classification Parallel systems Parallel computers SIMD (Single Instruction Multiple Data): Synchronized execution of the same instruction on a set of data MIMD (Multiple Instruction Multiple Data): Asynchronous execution of different instructions. M. Flynn, Very High-Speed Computing Systems, Proceedings of the IEEE, 54, 1966

16 MIMD computers Distributed Memory - DM (multicomputer) Building blocks are nodes with private physical address space. Communication is based on messages. Shared Memory - SM (multiprocessor) System provides a shared address space. Communication is based on read/write operation to global addresses.

17 Shared Memory Uniform Memory Access UMA : (symmetric multiprocessors - SMP): Centralized shared memory, accesses to global memory from all processors have same latency. Non-uniform Memory Access Systems - NUMA (Distributed Shared Memory Systems - DSM): memory is distributed among the nodes, local accesses much faster than remote accesses.