An Introduction to Parallel Programming

Transcription

1 F 'C 3 R'"'C,_,. HO!.-IJJ () An Introduction to Parallel Programming Peter S. Pacheco University of San Francisco ELSEVIER AMSTERDAM BOSTON HEIDELBERG LONDON NEW YORK OXFORD PARIS SAN DIEGO SAN FRANCISCO SINGAPORE SYDNEY TOKYO Morgan Kaufmann Publishers is an imprint of Elsevier M< ~ AVUCAt."'

2 Preface Acknowledgments About the Author CHAPTER CHAPTER Why Parallel Computing?.... Why We Need Ever-Increasing Performance.... Why We're Building Parallel Systems Why We Need to Write Parallel Programs How Do We Write Parallel Programs?..... What We'll Be Doing Concurrent, Parallel, Distributed The Rest of the Book.... A Word of Warning.... Typographical Conventions Summary Exercises Parallel Hardware and Parallel Software..... Some Background The von Neumann architecture..., Processes, multitasking, and threads Modifications to the von Neumann Model The basics of caching Cache mappings Caches and programs: an example , Virtual memory Instruction-level parallelism Hardware multithreading Parallel Hardware SIMD systems MIMD systems Interconnection networks Cache coherence Shared-memory versus distributed-memory Parallel Software Caveats Coordinating the processes/threads Shared-memory XV XVlll xix ix

3 x Distributed-memory Programming hybrid systems Input and Output Performance Speedup and efficiency Amdahl's law Scalability Taking timings Parallel Program Design An example Writing and Running Parallel Programs Assumptions Summary Serial systems Parallel hardware Parallel software Input and output Performance Parallel program design Assumptions Exercises CHAPTER 3 Distributed-Memory Programming with MPI Getting Started Compilation and execution MPiprograms MPLinitandMPLFinalize Communicators, MP LComm_s i ze and MPLComm_rank SPMD programs Communication MPLSend MPLRecv Message matching The status_p argument Semantics of MPLSend and MPLRecv Some potential pitfalls The Trapezoidal Rule in MPI The trapezoidal rule Parallelizing the trapezoidal rule

4 p xi Dealing with 1/ Output Input Collective Communication Tree-structured communication MPLRedu ce Collective vs. point-to-point communications... los MP LA ll reduce Broadcast Data distributions Scatter Gather Allgather MPI Derived Datatypes Performance Evaluation of MPI Programs Taking timings Results Speedup and efficiency Scalability A Parallel Sorting Algorithm Some simple serial sorting algorithms Parallel odd-even transposition sort Safety in MPI programs Final details of parallel odd-even sort Summary Exercises Programming Assignments CHAPTER 4 Shared-Memory Programming with Pthreads Processes, Threads, and Pthreads Hello, World Execution Preliminaries , Starting the threads Running the threads Stopping the threads Error checking Other approaches to thread startup Matrix-Vector Multiplication Critical Sections

5 xii Busy-Waiting Mutexes Producer-Consumer Synchronization and Semaphores Barriers and Condition Variables Busy-waiting and a mutex Semaphores Condition variables Pthreads barriers Read-Write Locks Linked list functions A multi-threaded linked list Pthreads read-write locks Performance of the valious implementations Implementing read-wlite locks Caches, Cache Coherence, and False Shating Thread-Safety Incorrect programs can produce correct output Summary Exercises Programming Assignments CHAPTER 5 Shared-Memory Programming with OpenMP Getting Started Compiling and running OpenMP programs The program Errorchecking The Trapezoidal Rule A first OpenMP version Scope of Variables The Reduction Clause The parallel for Directive Caveats Data dependences Finding loop-carried dependences Estimating rr More on scope More About Loops in OpenMP: Sorting Bubble sort Odd-even transposition sort Scheduling Loops The schedule clause The static schedule type

6 p xiii The dynamic and guided schedule types The runtimescheduletype Which schedule? Producers and Consumers Queues Message-passing Sending messages Receiving messages Termination detection Startup The atomic directive Critical sections and locks Using locks in the message-passing program c ri tical directives, atomic directives, or locks? Some caveats Caches, Cache Coherence, and False Sharing Thread-Safety Incorrect programs can produce correct output Summary Exercises Programming Assignments CHAPTER 6 Parallel Program Development Two n-body Solvers The problem Two selial programs Parallelizing the n-body solvers A word about Parallelizing the basic solver using OpenMP Pm allelizing the reduced solver using OpenMP Evaluating the OpenMP codes Parallelizing the solvers using pthreads Parallelizing the basic solver using MPI Parallelizing the reduced solver using MPI Performance of the MPI solvers Tree Search Recursive depth-first search Nonrecursive depth-first search Data structures for the serial implementations Performance of the serial implementations Parallelizing tree search

7 xiv A static parallelization of tree search using pthreads A dynamic parallelization of tree search using pthreads Evaluating the pthreads tree-search programs Parallelizing the tree-search programs using OpenMP Performance of the OpenMP implementations Implementation of tree search using MPI and static partitioning Implementation of tree search using MPI and dynamic partitioning A Word of Caution Which API? Summary Pthreads and OpenMP MPI Exercises Programming Assignments CHAPTER 7 Where to Go from Here References Index