Structured Parallel Programming

Transcription

1 Structured Parallel Programming Patterns for Efficient Computation Michael McCool Arch D. Robison James Reinders ELSEVIER AMSTERDAM BOSTON HEIDELBERG LONDON NEW YORK OXFORD PARIS SAN DIEGO SAN FRANCISCO SINGAPORE SYDNEY TOKYO Morgan Kaufmann Publishers is an imprint of Elsevier M< rtorc.lui "AU I MANN

2 Contents \ " J\..,.:...,, ~! Listings ':.,.' xv Preface... ~.". : xx Preliminaries xxiii CHAPTER Introduction Think Parallel Performance Motivation: Pervasive Parallelism Hardware Trends Encouraging Parallelism Observed Historical Trends in Parallelism Need for Explicit Parallel Programming Structured Pattern-Based Programming Parallel Programming Models Desired Properties..... Abstractions Instead of Mechanisms Expression of Regular Data Parallelism Composability Portability of Functionality Performance Portability Safety, Determinism, and Maintainability Overview of Programming Models Used When to Use Which Model? Organization of this Book Summary CHAPTER Background Vocabulary and Notation Strategies Mechanisms Machine Models Machine Model Key Features for Performance Flynn's Characterization Evolution Performance Theory Latency and Throughput Speedup, Efficiency, and Scalability v

3 vi Contents..3 Power Amdahl's Law Gustafson-Barsis' Law Work-Span Model Asymptotic Complexity Asymptotic Speedup and Efficiency Little's Formula Pitfalls Race Conditions Mutual Exclusion and Locks Deadlock Strangled Scaling Lack of Locality Load Imbalance Overhead Summary PART I PATTERNS CHAPTER 3 Patterns Nesting Pattern Structured Serial Control Flow Patterns Sequence Selection Iteration Recursion Parallel Control Patterns Fork-Join Map Stencil Reduction Scan Recurrence Serial Data Management Patterns Random Read and Write Stack Allocation Heap Allocation Closures Objects

4 Contents vii Parallel Data Management Patterns Pack Pipeline Geometric Decomposition Gather Scatter Other Parallel Patterns Superscalar Sequences Futures Speculative Selection Workpile Search Segmentation Expand Category Reduction Term Graph Rewriting Non-Deterministic Patterns Branch and Bound Transactions Programming Model Support for Patterns CilkPlus Threading Building Blocks OpenMP Array Building Blocks OpenCL Summary CHAPTER 4 Map Map Scaled Vector Addition (SAXPY) Description of the Problem Serial Implementation TBB Cilk Plus Cilk Plus with Array Notation OpenMP ArBB Using Vector Operations ArBB Using Elemental Functions OpenCL

5 ~----- viii Contents 4.3 Mandelbrot Description of the Problem Serial Implementation TBB Cilk Plus Cilk Plus with Array Notations OpenMP ArBB OpenCL Sequence of Maps versus Map of Sequence Comparison of Parallel Models Related Patterns Stencil Workpile Divide-and-conquer Summary CHAPTER Collectives Reduce Reordering Computations Vectorization Tiling Precision Implementation Fusing Map and Reduce Explicit Fusion in TBB Explicit Fusion in Cilk Plus Automatic Fusion in ArBB Dot Product Description of the Problem Serial Implementation SSE Intrinsics TBB Cilk Plus OpenMP c.3. ArBB Scan Cilk Plus TBB ArBB OpenMP Fusing Map and Scan

6 Contents ix Integration Description of the Problem Serial Implementation Cilk Plus OpenMP TBB ArBB Summary CHAPTER Data Reorganization......, Gather General Gather Shift Zip Scatter Atomic Scatter Permutation Scatter Merge Scatter Priority Scatter Converting Scatter to Gather Pack Fusing Map and Pack Geometric Decomposition and Partition Array of Structures vs. Structures of Arrays Summary CHAPTER Stencil and Recurrence Stencil Implementing Stencil with Shift Tiling Stencils for Cache : Optimizing Stencils for Communication Recurrence Summary CHAPTER Fork-Join Definition Programming Model Support for Fork-Join Cilk Plus Support for Fork-Join TBB Support for Fork-Join OpenMP Support for Fork-Join Recursive Implementation of Map Choosing Base Cases

7 p x Contents. Load Balancing Complexity of Parallel Divide-and-Conquer Karatsuba Multiplication of Polynomials Note on Allocating Scratch Space Cache Locality and Cache-Oblivious Algorithms Quicksort Cilk Quicksort TBB Quicksort Work and Span for Quicksort Reductions and Hyperobjects Implementing Scan with Fork-Join Applying Fork-Join to Recurrences Analysis Flat Fork-Join Summary CHAPTER 9 Pipeline Basic Pipeline Pipeline with Parallel Stages Implementation of a Pipeline Programming Model Support for Pipelines Pipeline in TBB Pipeline in Cilk Plus More General Topologies Mandatory versus Optional Parallelism Summary PART II EXAMPLES CHAPTER 0 Forward Seismic Simulation Background Stencil Computation Impact of Caches on Arithmetic Intensity Raising Arithmetic Intensity with Space-Time Tiling Cilk Plus Code ArBB Implementation Summary

8 Contents xi 0 3 o CHAPTER K-Means Clustering Algorithm K-Means with Cilk Plus Hyperobjects K-Means with TBB Summary CHAPTER Bzip Data Compression The Bzip Algorithm : Three-Stage Pipeline Using TBB Four-Stage Pipeline Using TBB Three-Stage Pipeline Using Cilk Plus Summary CHAPTER 3 Merge Sort Parallel Merge TBB Parallel Merge Work and Span of Parallel Merge Parallel Merge Sort Work and Span of Merge Sort Summary CHAPTER 4 Sample Sort Overall Structure Choosing the Number of Bins Binning TBB Implementation Repacking and Subsorting Performance Analysis of Sample Sort For C++ Experts Summary CHAPTER Cholesky Factorization Fortran Rules! Recursive Cholesky Decomposition Triangular Solve Symmetric Rank Update Where Is the Time Spent? Summary

9 xii Contents APPENDICES APPENDIX A Further Reading A. Parallel Algorithms and Patterns A. Computer Architecture Including Parallel Systems A.3 Parallel Programming Models APPENDIX B Cilk Plus Shared Design Principles with TBB Unique Characteristics Borrowing Components from TBB Keyword Spelling ci l k_for ci l k_s pa wn and ci l Lsync Reducers (Hyperobjects) B.. C++ Syntax B.. CSyntax ' Array Notation B.. Specifying Array Sections B.. Operations on Array Sections B..3 Reductions on Array Sections B..4 Implicit Index B.. Avoid Partial Overlap of Array Sections #p r a gma s i md Elemental Functions B.0. Attribute Syntax Note on C Notes on Setup History Summary APPENDIX c T C. Unique Characteristics C. Using TBB C.3 parall el_for C.3. bl oc ked_r ange C.3. Partitioners C.4 pa rall el_redu ce C. parall eldeterministi c_r educe C.G parall el pi pel i ne C. parall el_i nvok e

10 p Contents xiii C. task_group C.9 task C.9. empty_task C.0 atomic C. enumerabl e_thread_speci fi c C. NotesonC++ll C.3 History C.4 Summary APPENDIX D C++ll Declaring with auto : Lambda Expressions std:: move APPENDIX E Glossary Bibliography Index