Structured Parallel Programming Patterns for Efficient Computation

Transcription

1 Structured Parallel Programming Patterns for Efficient Computation Michael McCool Arch D. Robison James Reinders ELSEVIER AMSTERDAM BOSTON HEIDELBERG LONDON NEW YORK OXFORD PARIS SAN DIEGO SAN FRANCISCO SINGAPORE SYDNEY TOKYO Morgan Kaufmann Publishers is an imprint of Elsevier M<

2 Contents Listings Preface Preliminaries xv xix xxiii CHAPTER 1 Introduction Think Parallel Performance Motivation: Pervasive Parallelism Hardware Trends Encouraging Parallelism Observed Historical Trends in Parallelism Need for Explicit Parallel Programming Structured Pattern-Based Programming Parallel Programming Models Desired Properties Abstractions Instead of Mechanisms Expression of Regular Data Parallelism Composability Portability of Functionality Performance Portability Safety, Determinism, and Maintainability Overview of Programming Models Used When to Use Which Model? Organization of this Book Summary 38 CHAPTER 2 Background Vocabulary and Notation Strategies Mechanisms Machine Models Machine Model Key Features for Performance Flynn's Characterization Evolution Performance Theory Latency and Throughput Speedup, Efficiency, and Scalability 56 V

3 vi Contents Power Amdahl's Law Gustafson-Barsis' Law Work-Span Model Asymptotic Complexity Asymptotic Speedup and Efficiency Little's Formula Pitfalls Race Conditions Mutual Exclusion and Locks Deadlock Strangled Scaling Lack of Locality Load Imbalance Overhead Summary 75 PART I PATTERNS CHAPTER 3 Patterns Nesting Pattern Structured Serial Control Flow Patterns Sequence Selection Iteration Recursion Parallel Control Patterns Fork-Join Map Stencil Reduction Scan Recurrence Serial Data Management Patterns Random Read and Write Stack Allocation Heap Allocation Closures Objects 97

4 Contents vii 3.5 Parallel Data Management Patterns Pack Pipeline Geometrie Decomposition Gather Scatter Other Parallel Patterns Superscalar Sequences Futures Speculative Selection Workpile Search Segmentation Expand Category Reduction Term Graph Rewriting Non-Deterministic Patterns Branch and Bound Transactions Programming Model Support for Patterns CilkPlus Threading Building Blocks OpenMP Array Building Blocks OpenCL Summary 118 CHAPTER 4 Map Map Scaled Vector Addition (SAXPY) Description of the Problem Serial Implementation TBB CilkPlus Cilk Plus with Array Notation OpenMP ArBB Using Vector Operations ArBB Using Elemental Functions OpenCL 130

5 viii Contents 4.3 Mandelbrot Description of the Problem Serial Implementation TBB Cilk Plus Cilk Plus with Array Notations OpenMP ArBB OpenCL Sequence of Maps versus Map of Sequence Comparison of Parallel Models Related Patterns Stencil Workpile Divide-and-conquer Summary 143 CHAPTERS Collectives Reduce Reordering Computations Vectorization Tiling Precision Implementation Fusing Map and Reduce Explicit Fusion in TBB Explicit Fusion in Cilk Plus Automatic Fusion in ArBB DotProduct Description of the Problem Serial Implementation SSE Intrinsics TBB Cilk Plus OpenMP ArBB Scan Cilk Plus TBB ArBB OpenMP Fusing Map and Scan 166

6 Contents ix 5.6 Integration Description of the Problem Serial Implementation CilkPlus OpenMP TBB ArBB Summary 177 CHAPTER 6 Data Reorganization Gather General Gather Shift Zip Scatter Atomic Scatter Permutation Scatter Merge Scatter Priority Scatter Converting Scatter to Gather Pack Fusing Map and Pack Geometric Decomposition and Partition Array of Structures vs. Structures of Arrays Summary 197 CHAPTER 7 Stencil and Recurrence Stencil Implementing Stencil with Shift Tiling Stencils for Cache Optimizing Stencils for Communication Recurrence Summary 207 CHAPTER 8 Fork-Join Definition Programming Model Support for Fork-Join Cilk Plus Support for Fork-Join TBB Support for Fork-Join OpenMP Support for Fork-Join Recursive Implementation of Map Choosing Base Cases 217

7 x Contents 8.5 Load Balancing Complexity of Parallel Divide-and-Conquer Karatsuba Multiplication of Polynomials Note on Allocating Scratch Space Cache Locality and Cache-Oblivious Algorithms Quicksort Cilk Quicksort TBB Quicksort Work and Span for Quicksort Reductions and Hyperobjects Implementing Scan with Fork-Join Applying Fork-Join to Recurrences Analysis Flat Fork-Join Summary 251 CHAPTER 9 Pipeline Basic Pipeline Pipeline with Parallel Stages Implementation of a Pipeline Programming Model Support for Pipelines Pipeline in TBB Pipeline in Cilk Plus More General Topologies Mandatory versus Optional Parallelism Summary 262 PART II EXAMPLES CHAPTER 10 Forward Seismic Simulation Background Stencil Computation Impact of Caches on Arithmetic Intensity Raising Arithmetic Intensity with Space-Time Tiling Cilk Plus Code ArBB Implementation Summary 277

8 Contents CHAPTER 11 K-Means Clustering Algorithm K-Means with Cilk Plus Hyperobjects K-Means with TBB Summary 289 CHAPTER 12 Bzip2 Data Compression The Bzip2 Algorithm Three-Stage Pipeline Using TBB Four-Stage Pipeline Using TBB Three-Stage Pipeline Using Cilk Plus Summary 297 CHAPTER 13 Merge Sort Parallel Merge TBB Parallel Merge Work and Span of Parallel Merge Parallel Merge Sort Work and Span of Merge Sort Summary 305 CHAPTER 14 Sample Sort Overall Structure Choosing the Number of Bins Binning TBB Implementation Repacking and Subsorting Performance Analysis of Sample Sort For C++ Experts Summary 313 CHAPTER 15 Cholesky Factorization Fortran Rules! Recursive Cholesky Decomposition Triangular Solve Symmetric Rank Update Where Is the Time Spent? Summary 322

9 xii Contents APPENDICES APPENDIX A Further Reading 325 A.1 Parallel Algorithms and Patterns 325 A.2 Computer Architecture Including Parallel Systems 325 A.3 Parallel Programming Models 326 APPENDIX В Cilk Plus 329 B.1 Shared Design Principles with TBB 329 B.2 Unique Characteristics 329 B.3 Borrowing Components from TBB 331 B.4 Keyword Spelling 332 B.5 cilk_for 332 B.6 ci 1 k.spawn and ci 1 k_sync 333 B.7 Reducers (Hyperobjects) 334 B.7.1 C++ Syntax 335 B.7.2 С Syntax 337 B.8 Array Notation 338 B.8.1 Specifying Array Sections 339 B.8.2 Operations on Array Sections 340 B.8.3 Reductions on Array Sections 341 B.8.4 Implicit Index 342 B.8.5 Avoid Partial Overlap of Array Sections 342 B.9 //pragma simd 343 B. 10 Elemental Functions 344 B.10.1 Attribute Syntax 345 B.11 NoteonC++ll 345 B. 12 Notes on Setup 346 B.13 History 346 B. 14 Summary 347 APPENDIX С TBB 349 С1 Unique Characteristics 349 C.2 Using TBB 350 C.3 parallel-for 351 C.3.1 Ы ocked_range 351 C.3.2 Partitioners 352 C.4 paral 1 el.reduce 353 C.5 paral1el.deterministic.reduce 354 C.6 paral 1 el_pi pel i ne 354 C.7 paral 1 el_i nvoke 354

10 Contents xiii C.8 task_group 355 C.9 task 355 C.9.1 empty_ta s к 356 C.10 atomic 356 C.11 enumerabl e.thread.speci fic 358 C.12 Notes onc++ll 358 C.13 History 359 C.14 Summary 360 APPENDIX D C D.1 Declaring with auto 361 D.2 Lambda Expressions 361 D.3 std : -.move 365 APPENDIXE Glossary 367 Bibliography 391 Index 397