A Pattern Language for Parallel Programming

Transcription

1 A Pattern Language for Parallel Programming Tim Mattson Beverly Sanders Berna Massingill

2 Motivation Hardware for parallel computing is everywhere : clusters, SMP workstations, NUMA Big Iron. Software to run on these systems is not. Knowledge needed to make effective use of hardware is mostly limited to high-end HPC community. How to disseminate this knowledge (to programmers, domain experts) so that sequential software is rare?

3 A Possible Long-Term Solution A layered solution stack focused on the algorithm designer, not the hardware: A pattern language for parallel programming. A component-based framework. Low-level portable APIs for parallel computing. Supporting middleware. We believe you must start at the top: Get the pattern language right first and you stand a better change of doing other layers right.

4 A Shameless Plug A pattern language for parallel algorithm design with examples in MPI, OpenMP, and Java. This is our hypothesis about how expert parallel programmers think about parallel programming. Now available at a bookstore near you!

5 What s a Design Pattern? High-quality solution to frequently occurring problem in some domain. Written in consistent format to allow readers to quickly understand context and solution. Named so that pattern names provide a vocabulary for discussing solutions (as has happened in object-oriented oriented programming).

6 What s a Pattern Language? Carefully structured collection of patterns. Not a programming language. Can embody a design methodology, so user works through patterns to develop complex design using language s patterns.

7 The Pattern Language s Structure A software design can be viewed as a series of refinements. We consider the process in terms of 4 design spaces which add progressively lower-level elements to the design. Design Space The Evolving Design Finding Concurrency Algorithm Structure Supporting Structures Tasks, shared data, partial orders Thread/process structures, schedules Source code organization, shared data Implementation Mechanisms Thread/process mgmt, interaction

8 Parallel Software: Where the design spaces fit in during software development Finding Concurrency Original Problem Algorithm Structure Units of execution + new shared data for extracted dependencies Supporting struct. & impl. mech. Tasks, shared and local data Program SPMD_Emb_Par () { Program SPMD_Emb_Par () TYPE { Program *tmp, *func(); SPMD_Emb_Par () global_array TYPE { Program *tmp, Data(TYPE); *func(); SPMD_Emb_Par () global_array TYPE { Res(TYPE); *tmp, Data(TYPE); *func(); int N = global_array get_num_procs(); TYPE Res(TYPE); *tmp, Data(TYPE); *func(); int id int= N get_proc_id(); = global_array get_num_procs(); Res(TYPE); Data(TYPE); if (id==0) int int= setup_problem(n,data); N global_array get_proc_id(); = get_num_procs(); Res(TYPE); for (int if (id==0) int I= 0; int I<N;I=I+Num){ = setup_problem(n,data); Num get_proc_id(); = get_num_procs(); tmp for (int if = (id==0) func(i); int I= 0; I<N;I=I+Num){ = setup_problem(n,data); get_proc_id(); Res.accumulate( tmp for (int if = (id==0) func(i); I= 0; tmp); I<N;I=I+Num){ setup_problem(n, Data); } Res.accumulate( tmp for (int = func(i); I= ID; tmp); I<N;I=I+Num){ } } Res.accumulate( tmp = func(i, tmp); Data); } Res.accumulate( tmp); } } } } Corresponding source code

9 The Finding Concurrency Design Space Start with a specification that solves the original problem, finish with a decomposition of the problem into tasks, plus an analysis of shared data and task dependencies (partial ordering). Start Dependency Analysis Group Group Tasks Tasks Order Order Tasks Tasks Data Data Sharing Sharing Decomposition Data Data Decomposition Decomposition Task Task Decomposition Decomposition Design Design Evaluation

10 The Algorithm Structure Design Space Select overall program organization to exploit concurrency identified in previous step. Start Organize By Flow of Data Organize By Tasks Organize By Data Regular? Irregular? Linear? Recursive? Linear? Recursive? Pipeline Pipeline Event-Based Event-Based Coordination Coordination Task Task Parallelism Parallelism Divide Divide and and Conquer Conquer Geometric Geometric Decomposition Decomposition Recursive Recursive Data Data

11 The Supporting Structures Design Space High-level constructs affecting large-scale organization of the source code. Program Structure SPMD SPMD Master/Worker Master/Worker Loop Loop Parallelism Parallelism Data Structures Shared Shared Data Data Shared Shared Queue Queue Distributed Distributed Array Array Fork/Join Fork/Join

12 The Implementation Mechanisms Design Space The primitives of parallel computing. Book s examples are in Java, OpenMP, and MPI. This design space discusses key issues more generically. Not properly design patterns, included to make the pattern language self-contained. UE* Management Thread Thread control control Process Process control control Synchronization Memory Memory sync/fences sync/fences Barriers Barriers Mutual Mutual Exclusion Exclusion Communication Message Message passing passing Collective Collective comm comm Other Other comm comm * UE = Unit of execution

13 Open questions How close did we come to getting it right (identifying right/useful patterns)? We ve heard from patterns people; now we need to hear from domain experts. Our patterns are modeled on 20 years of experience with HPC. Are they too narrow in scope? Our patterns come from an old-fashioned, procedural mindset and are not tied to modern object-oriented oriented software design concepts. Should we be more GoF-like? Lowest-level design space was not represented as patterns. Does this suggest a need to be more abstract and a redesign of that space?