Structured approaches for multi/many core targeting
|
|
- Darleen Newton
- 6 years ago
- Views:
Transcription
1 Structured approaches for multi/many core targeting Marco Danelutto M. Torquati, M. Aldinucci, M. Meneghin, P. Kilpatrick, D. Buono, S. Lametti Dept. Computer Science, Univ. of Pisa CoreGRID Programming model Institute Sept. 18, 2012, IRILL Paris
2 Contents Introduction Hardware Programming models FastFlow FastFlow intro Core FastFlow details FastFlow improvements Distributed FastFlow New skeletons Co-processor targeting Perspectives ParaPhrase Ocaml perspectives
3 Hardware Hardware scenario General purpose computing devices multi-core 4 8/16 per socket decreased core complexity multiple thread contexts per core NUMA
4 Hardware Hardware scenario c Co-processors GPUs more cores & SMPs direct access to memory hierarchy RDMA & cooperation within clusters Many core 50 to 100 cores simpler interconnection networks much simpler cores
5 Programming models Programming model shift Good sequential code parallelized to tune performance
6 Programming models Programming model shift Good sequential code parallelized to tune performance Good parallel code whit seq portions optimized to tune performance subject to parallelisation constrain
7 Programming models Programming model scenario Distributed memory MPI Shared memory OpenMP data parallel task parallel data flow (DAG) Co-processors CUDA (OpenMP)
8 Programming models Programming model scenario Distributed memory MPI Shared memory OpenMP data parallel task parallel data flow (DAG) Co-processors CUDA (OpenMP) relatively low level sync&comm entirely in charge of the application programmer require quite deep target hw knowledge to tune performance efficient co-processor exploitation nightmare
9 Programming models Skeletons and design patterns Algorithmic skeletons since 90s, from HPC community languages (P3L), libraries (eskel, Skipper, Calcium/Skandium, SkeTo, Muskel,...) key concept efficient, parametric, language constructs/library entries modelling common parallel patterns available to the application programmers
10 Programming models Skeletons and design patterns Algorithmic skeletons since 90s, from HPC community languages (P3L), libraries (eskel, Skipper, Calcium/Skandium, SkeTo, Muskel,...) key concept efficient, parametric, language constructs/library entries modelling common parallel patterns available to the application programmers Parallel design patterns since 00s, from SW engineering community key concept recipes to program common parallel patterns structured (layered): concurrency finding, algorithm structure, support structure, mechanism design spaces
11 Programming models Patterns (the skeleton way...) Berkeley report feasible way to attack parallel programming and to form new generations of programmers
12 Programming models Patterns (the skeleton way...) Berkeley report feasible way to attack parallel programming and to form new generations of programmers Algorithmic skeleton community promoting since decades the concept with notable frameworks (SkeTo, OcamlP3L)
13 Programming models Patterns (the skeleton way...) Berkeley report feasible way to attack parallel programming and to form new generations of programmers Algorithmic skeleton community promoting since decades the concept with notable frameworks (SkeTo, OcamlP3L) User community (application programmers) asking more and more abstraction to hide (encapsulate) parallelism exploitaion
14 Programming models Patterns (major commercial actors) Intel TBB include simple patterns (e.g. pipeline) elaborate on the design pattern concept
15 Programming models Patterns (major commercial actors) Intel TBB include simple patterns (e.g. pipeline) elaborate on the design pattern concept Microsoft TPL include simple patterns (data and stream parallel) evolving to macro data flow
16 Programming models Patterns (major commercial actors) Intel TBB include simple patterns (e.g. pipeline) elaborate on the design pattern concept Microsoft TPL include simple patterns (data and stream parallel) evolving to macro data flow Google MapReduce further elaborating on pattern composition (Flume)
17 FastFlow intro FastFlow (What/Why) Parallel programming framework algorithmic skeleton approach C++ based streaming process networks targeting shared memory architectures
18 FastFlow intro FastFlow (What/Why) Parallel programming framework algorithmic skeleton approach C++ based streaming process networks targeting shared memory architectures extremely efficient in inter-concurrent activity communication (in the range of 10nsecs) better or comparable efficiency/performance w.r.t. standard programming environments (OMP, Cilk, TBB)
19 FastFlow intro FastFlow (What/Why) Parallel programming framework algorithmic skeleton approach C++ based streaming process networks targeting shared memory architectures extremely efficient in inter-concurrent activity communication (in the range of 10nsecs) better or comparable efficiency/performance w.r.t. standard programming environments (OMP, Cilk, TBB) raising the level of abstraction exposed to the user impacting programmer efficiency and time-to-deploy
20 FastFlow intro FastFlow kernel layered design incremental implementation of efficient inter-thread communication mechanisms (efficient, lock-free, wait-free bounded and un-bounded SPSC queue SPMC and MPSC queues) simple streaming patterns provided as customizable skeletons (pipeline and task farm) software accelerator abstraction
21 Core FastFlow details FastFlow queue(s) Built on previous work: only hosts pointers nanoseconds mapping 1 mapping 2 mapping 3 push and pop operations NULL is a special (non queue-able) value mapping 1 mapping 2 mapping 3 buffer size active wait on pop from empty and push on full (bounded) queues unbounded version (supporting buffer re-use) nanoseconds buffer size
22 Core FastFlow details FastFlow ff node Main concept: concurrent activity with associated input and output queue wrapper of seq code or skeleton arbitrary compositions: sequential ff node farm or pipeline three methods: 1. void * svc(void *) computes a result out of a task 2. int svc init(void) called once before starting the application code 3. void svc end(void) called before application shutdown
23 Core FastFlow details Stream parallel skeletons pipeline arbitrary number of stages farm Emitter-StringOfWorkers-Collector or Master/Worker feedback channel only in outer skeleton, feeds back (partial)results from last stage output to first stage input free compostion quite complex process graphs
24 Core FastFlow details Stream parallel skeletons pipeline arbitrary number of stages farm Emitter-StringOfWorkers-Collector or Master/Worker feedback channel only in outer skeleton, feeds back (partial)results from last stage output to first stage input free compostion quite complex process graphs
25 Core FastFlow details Accelerator interface Standard streaming programs: first stage generates the input stream middle stage computes results last stage outputs results Accelerator mode: create skeleton program offload tasks to be computed when needed get results asynchronously uses spare cores
26 Core FastFlow details Overall (FastFlow) Skeletons: template process networks (skeletons?) customizable semantics Shared memory: capabilities (pointers) passed between threads consistent usage of the capabilities in charge to the programmer extremely efficient comms (fwd: see Tilera implementation) C++: suitable abstraction mechanisms to provide skeletons/templates
27 Distributed FastFlow Distributed FastFlow TCP/IP FastFlow queues: on top of ZMQ, same signature as internal queues unicast, broadcast, scatter, on-demand, fromany, gatherall protocols Distributed skeletons: Either input or output channel distributed queue SPMD-like model to launch sender and receiver nodes supports coordinated execution of FastFlow programs on different NOW PEs
28 Distributed FastFlow Distributed FastFlow Completion tims (secs) x512 Eth 512x512 Inf 256x256 Eth 256x256 Inf Parallelism degree Impact of network bandwidth (MM benchmark)
29 Distributed FastFlow Distributed FastFlow Host A Host B 6000 N. of tasks Eth(256) Inf(256) Eth(512) Inf(512) Network (matrix dimensions) Impact of network bandwidth (MM benchmark)
30 Distributed FastFlow Distributed FastFlow Serialization/deserialization Homogenous machine assumption in charge to the application programmer (!) prepare method when sending iovec of address, lenght pairs prepare and unmarshalling when receiving prepare buffers rebuild structure from memory segments extremely versatile but also low level...
31 New skeletons Data parallel skeletons Original FastFlow: data parallel implemented with farm emitter decompose data to tasks collector rebuild result from partial results ff map skeleton: leaf node (two tier model) splitter policy in charge to the programmer efficient implementation using farm like template supports map, stencil and reduce like data parallel computations
32 Co-processor targeting FastFlow and co-processors Two different kind of co-processors considered: 1. GP-GPUs suitable for data parallel computations only available almost everywhere, possibily in multiple instances extremely cheap and power efficient 2. many core, general purpose co-processors POSIX thread compliant ranging in the 10 to 100 of (simple) cores not so cheap nor largerly available (e.g. Tilera, Intel MIC)
33 Co-processor targeting FastFlow for Tilera Pro64 Different memory subsystem re-engineering the SPSC queue barrier needed (weak store consistency) #ifdef x86_64 // x86 32/64-bit: no memory fence is needed. #define WMB() asm volatile ("": : :"memory") #endif #ifdef tile // Tilera: using a compiler intrinsic for memory fence. #define WMB() insn_mf(); #endif Clock Cycles SPSC uspsc k 8k k 8k TilePro64@866MHz Nehalem E7@2.0GHz
34 Co-processor targeting FastFlow for Tilera Pro64 Cache coherence protocol may be disabled may be customized Experiments: Hash home node (HHN) No coherency (NHN) Fixed home node (FHN) (Protocol directed allocation) Speedup ideal HHN NHN FHN FastFlow worker threads Matrix multiplication (stream of 64x64 matrixes)
35 Co-processor targeting FastFlow for Tilera Pro64 Speedup/Scalability ideal Speedup Scalability FastFlow worker threads
36 Co-processor targeting FastFlow for Tilera Pro64
37 Co-processor targeting FastFlow & GPUs Two-tier model: Stream parallel skeletons upper part of the skeleton tree Data parallel skeletons lower part of the skeleton tree Sequential wrappers skeleton tree leaves
38 Co-processor targeting FastFlow & GPUs Two-tier model: Stream parallel skeletons upper part of the skeleton tree Data parallel skeletons lower part of the skeleton tree Sequential wrappers skeleton tree leaves GPU code embedded into ff nodes
39 Co-processor targeting FastFlow & GPUs Two-tier model: Stream parallel skeletons upper part of the skeleton tree Data parallel skeletons lower part of the skeleton tree Sequential wrappers skeleton tree leaves GPU code embedded into ff nodes any kind of code to manage GPUs: Cuda, OpenCL but also SkePu,... single/multiple GPU management + kernel scheduling under user responsibility
40 Co-processor targeting FastFlow & GPUs #define SKEPU_CUDA UNARY_FUNC(iters, float, a, {a++; return(a);} ) class MapStage:public ff_node { void *svc(void * task) { skepu::vector<float> * v = (skepu::vector<float> *) task; skepu::vector<float> * r = new skepu::vector<float>(nn); skepu::map<iters> skepumap(new iters); skepumap(*v, *r); free(task); return((void *) r); } };
41 Co-processor targeting FastFlow & GPUs Alternatively: user worker function provided as fixed signature C++ function wrappers for GPUs and CPUs e.g. OpenMP for CPUs CUDA/OpenCL for GPUs
42 Co-processor targeting Sample CPU wrapping template<typename IN, typename OUT> class ff_map_1d_cpu : public ff_node { private: OUT (*f)(in input); public: ff_map_1d_cpu(out (*func)(in)) { f = func; } void * svc(void * task) { task_1d<in>* task1 = (task_1d<in>*) task; IN* input_array = task1->get_array(); size_t size = task1->get_size(); OUT* result = new OUT[size]; #pragma omp parallel for schedule(dynamic) for(size_t i = 0; i < size; i++) { result[i] = f(input_array[i]); } return (void*) new task_1d<out>(result, size); } }
43 Co-processor targeting Sample GPU wrapping Classic C function (with known types): out_type f(in_type arg) {... } Wrapped through macro: #define MapKernel(func_name, out_type, in_type, arg_name) \ device out_type func_name(in_type arg_name); \ global void MapKernel(in_type *input, out_type *outpu { \... data -> GPU... kernel computation... results -> CPU }
44 Co-processor targeting Problems Reliable and efficient GPU targeting within the ff node: CUDA? OpenCL? OpenAAC? kernel parameters? GPU access: One ff node controller per GPU? (CUDA 5.0 up to 32 concurrent controllers per (Kepler 110) GPU) Multi GPU management: special ff node controller Host/GPU memory traffic: CUDA 4.0: direct access to Host (pinned) memory CUDA 5.0: RDMA through PCIe (?)
45 ParaPhrase ParaPhrase perspective EU FP7 STREP project StAndrews, RGU, QUB, Erlang Solutions (UK), UNIPI, UNITO (IT), SCCH (A), Stuttgard (D), Mellanox (Israel) parallelism managed through patterns, refactored and optimized, eventually compiled to skeleton code indipendent but similar toolchains for Erlang and C++/FastFlow
46 ParaPhrase ParaPhrase perspective FastFlow candidate framework to validate project methodology FastFlow skeletons introduce through user assisted refactoring of seq code FastFlow skeleton optimization through refactoring CPU/GPU targeting managed automatically (refactoring + mapping) run time adaptation of refactoring/mapping choices
47 ParaPhrase ParaPhrase perspective FastFlow candidate framework to validate project methodology FastFlow skeletons introduce through user assisted refactoring of seq code FastFlow skeleton optimization through refactoring CPU/GPU targeting managed automatically (refactoring + mapping) run time adaptation of refactoring/mapping choices Erlang pure functional + CSP style processes with send/receive FastFlow-like skeleton set experimented on real applications
48 Ocaml perspectives Ocaml for multicores Experience with parmap possibility to exploit pretty efficient medium grain parallelism relying on processes rather than threads (by necessity) quite an amount of different parallel libraries for Ocaml proof-of-concept need for some language parallelism from different applicative domains common features
49 Ocaml perspectives Ocaml for multicores Experience with parmap possibility to exploit pretty efficient medium grain parallelism relying on processes rather than threads (by necessity) quite an amount of different parallel libraries for Ocaml proof-of-concept need for some language parallelism from different applicative domains common features Wish list (high level, tentative): efficient native mechanisms supporting customizable parallel patterns minimal disruption w.r.t. current Ocaml program practices
50 Ocaml perspectives Ocaml for multicores Efficient native mechanisms efficient threading mechanisms (PThreads?) efficient comm/synch mechanisms (FastFlow?) Customizable patterns: customizable patterns à la FastFlow + abstraction à la Ocaml Minimal disruption w.r.t. current Ocaml program practices: application programmer exposed with high level, domain specific patterns completely embedding parallelism management composable + sequential embedding
51 Ocaml perspectives Ocaml: inheriting from FastFlow Accelerator: FastFlow accelerator (farm) each worker gets a (shared memory) closure and computes it needs access to Ocaml (full) interpreter within an indipendent (pinned) thread Communication mechanisms: completely re-usable in threaded Ocaml environment grants fine grain comms and sycnhs Efficient allocator: built on top of base FastFlow layers greatly improves performance of programs with large number of small mallocs
52 Ocaml perspectives Conclusions FastFlow mature technology targeting different state-of-the-art architectures including Intel (Nehalem, Sandy Bridge), AMD (Magny Cours), Tilera (Tile64pro), Nvidia (GeForce, Tesla) (through SkePu) comparable or better performance w.r.t. standard environments completely open source (sourceforge/mc-fastflow) suitable to support different experiments with other (sequential) programming environments
53 Ocaml perspectives Any questions? (remember: this is team work with Torquati, Aldinucci, Meneghin, Kilpatrick, Buono, Lametti,...)
FastFlow: targeting distributed systems
FastFlow: targeting distributed systems Massimo Torquati ParaPhrase project meeting, Pisa Italy 11 th July, 2012 torquati@di.unipi.it Talk outline FastFlow basic concepts two-tier parallel model From single
More informationEfficient streaming applications on multi-core with FastFlow: the biosequence alignment test-bed
Efficient streaming applications on multi-core with FastFlow: the biosequence alignment test-bed Marco Aldinucci Computer Science Dept. - University of Torino - Italy Marco Danelutto, Massimiliano Meneghin,
More informationMarco Danelutto. May 2011, Pisa
Marco Danelutto Dept. of Computer Science, University of Pisa, Italy May 2011, Pisa Contents 1 2 3 4 5 6 7 Parallel computing The problem Solve a problem using n w processing resources Obtaining a (close
More informationFastFlow: targeting distributed systems Massimo Torquati
FastFlow: targeting distributed systems Massimo Torquati May 17 th, 2012 torquati@di.unipi.it http://www.di.unipi.it/~torquati FastFlow node FastFlow's implementation is based on the concept of node (ff_node
More informationEfficient Smith-Waterman on multi-core with FastFlow
BioBITs Euromicro PDP 2010 - Pisa Italy - 17th Feb 2010 Efficient Smith-Waterman on multi-core with FastFlow Marco Aldinucci Computer Science Dept. - University of Torino - Italy Massimo Torquati Computer
More informationAn efficient Unbounded Lock-Free Queue for Multi-Core Systems
An efficient Unbounded Lock-Free Queue for Multi-Core Systems Authors: Marco Aldinucci 1, Marco Danelutto 2, Peter Kilpatrick 3, Massimiliano Meneghin 4 and Massimo Torquati 2 1 Computer Science Dept.
More informationParallel Programming using FastFlow
Parallel Programming using FastFlow Massimo Torquati Computer Science Department, University of Pisa - Italy Karlsruhe, September 2nd, 2014 Outline Structured Parallel Programming
More informationMarco Danelutto. October 2010, Amsterdam. Dept. of Computer Science, University of Pisa, Italy. Skeletons from grids to multicores. M.
Marco Danelutto Dept. of Computer Science, University of Pisa, Italy October 2010, Amsterdam Contents 1 2 3 4 Structured parallel programming Structured parallel programming Algorithmic Cole 1988 common,
More informationNVIDIA Think about Computing as Heterogeneous One Leo Liao, 1/29/2106, NTU
NVIDIA Think about Computing as Heterogeneous One Leo Liao, 1/29/2106, NTU GPGPU opens the door for co-design HPC, moreover middleware-support embedded system designs to harness the power of GPUaccelerated
More informationAn innovative compilation tool-chain for embedded multi-core architectures M. Torquati, Computer Science Departmente, Univ.
An innovative compilation tool-chain for embedded multi-core architectures M. Torquati, Computer Science Departmente, Univ. Of Pisa Italy 29/02/2012, Nuremberg, Germany ARTEMIS ARTEMIS Joint Joint Undertaking
More informationStructured parallel programming
Marco Dept. Computer Science, Univ. of Pisa DSL 2013 July 2013, Cluj, Romania Contents Introduction Structured programming Targeting HM2C Managing vs. computing Progress... Introduction Structured programming
More informationIntroduction to FastFlow programming
Introduction to FastFlow programming SPM lecture, November 2016 Massimo Torquati Computer Science Department, University of Pisa - Italy Objectives Have a good idea of the FastFlow
More informationOverview of research activities Toward portability of performance
Overview of research activities Toward portability of performance Do dynamically what can t be done statically Understand evolution of architectures Enable new programming models Put intelligence into
More informationAccelerating code on multi-cores with FastFlow
EuroPar 2011 Bordeaux - France 1st Sept 2001 Accelerating code on multi-cores with FastFlow Marco Aldinucci Computer Science Dept. - University of Torino (Turin) - Italy Massimo Torquati and Marco Danelutto
More informationPredictable Timing Analysis of x86 Multicores using High-Level Parallel Patterns
Predictable Timing Analysis of x86 Multicores using High-Level Parallel Patterns Kevin Hammond, Susmit Sarkar and Chris Brown University of St Andrews, UK T: @paraphrase_fp7 E: kh@cs.st-andrews.ac.uk W:
More informationThe multi/many core challenge: a pattern based programming perspective
The multi/many core challenge: a pattern based programming perspective Marco Dept. Computer Science, Univ. of Pisa CoreGRID Programming model Institute XXII Jornadas de Paralelismo Sept. 7-9, 2011, La
More informationIntroduction to FastFlow programming
Introduction to FastFlow programming Hands-on session Massimo Torquati Computer Science Department, University of Pisa - Italy Outline The FastFlow tutorial FastFlow basic concepts
More informationMarco Aldinucci Salvatore Ruggieri, Massimo Torquati
Marco Aldinucci aldinuc@di.unito.it Computer Science Department University of Torino Italy Salvatore Ruggieri, Massimo Torquati ruggieri@di.unipi.it torquati@di.unipi.it Computer Science Department University
More informationModern Processor Architectures (A compiler writer s perspective) L25: Modern Compiler Design
Modern Processor Architectures (A compiler writer s perspective) L25: Modern Compiler Design The 1960s - 1970s Instructions took multiple cycles Only one instruction in flight at once Optimisation meant
More informationEvolution of FastFlow: C++11 extensions, distributed systems
Evolution of FastFlow: C++11 extensions, distributed systems Massimo Torquati Computer Science Department, University of Pisa Italy torquati@di.unipi.it PARAPHRASE Summer School 2014 Dublin, Ireland, June
More informationAddressing Heterogeneity in Manycore Applications
Addressing Heterogeneity in Manycore Applications RTM Simulation Use Case stephane.bihan@caps-entreprise.com Oil&Gas HPC Workshop Rice University, Houston, March 2008 www.caps-entreprise.com Introduction
More informationModern Processor Architectures. L25: Modern Compiler Design
Modern Processor Architectures L25: Modern Compiler Design The 1960s - 1970s Instructions took multiple cycles Only one instruction in flight at once Optimisation meant minimising the number of instructions
More informationLecture 13: Memory Consistency. + a Course-So-Far Review. Parallel Computer Architecture and Programming CMU , Spring 2013
Lecture 13: Memory Consistency + a Course-So-Far Review Parallel Computer Architecture and Programming Today: what you should know Understand the motivation for relaxed consistency models Understand the
More informationTechnology for a better society. hetcomp.com
Technology for a better society hetcomp.com 1 J. Seland, C. Dyken, T. R. Hagen, A. R. Brodtkorb, J. Hjelmervik,E Bjønnes GPU Computing USIT Course Week 16th November 2011 hetcomp.com 2 9:30 10:15 Introduction
More informationParallel Computing. Hwansoo Han (SKKU)
Parallel Computing Hwansoo Han (SKKU) Unicore Limitations Performance scaling stopped due to Power consumption Wire delay DRAM latency Limitation in ILP 10000 SPEC CINT2000 2 cores/chip Xeon 3.0GHz Core2duo
More informationTrends and Challenges in Multicore Programming
Trends and Challenges in Multicore Programming Eva Burrows Bergen Language Design Laboratory (BLDL) Department of Informatics, University of Bergen Bergen, March 17, 2010 Outline The Roadmap of Multicores
More informationUniversità di Pisa. Dipartimento di Informatica. Technical Report: TR FastFlow tutorial
Università di Pisa Dipartimento di Informatica arxiv:1204.5402v1 [cs.dc] 24 Apr 2012 Technical Report: TR-12-04 FastFlow tutorial M. Aldinucci M. Danelutto M. Torquati Dip. Informatica, Univ. Torino Dip.
More informationAccelerating sequential programs using FastFlow and self-offloading
Università di Pisa Dipartimento di Informatica Technical Report: TR-10-03 Accelerating sequential programs using FastFlow and self-offloading Marco Aldinucci Marco Danelutto Peter Kilpatrick Massimiliano
More informationThe Stampede is Coming: A New Petascale Resource for the Open Science Community
The Stampede is Coming: A New Petascale Resource for the Open Science Community Jay Boisseau Texas Advanced Computing Center boisseau@tacc.utexas.edu Stampede: Solicitation US National Science Foundation
More informationCSCI 402: Computer Architectures. Parallel Processors (2) Fengguang Song Department of Computer & Information Science IUPUI.
CSCI 402: Computer Architectures Parallel Processors (2) Fengguang Song Department of Computer & Information Science IUPUI 6.6 - End Today s Contents GPU Cluster and its network topology The Roofline performance
More informationParallel Programming on Larrabee. Tim Foley Intel Corp
Parallel Programming on Larrabee Tim Foley Intel Corp Motivation This morning we talked about abstractions A mental model for GPU architectures Parallel programming models Particular tools and APIs This
More informationThe Era of Heterogeneous Computing
The Era of Heterogeneous Computing EU-US Summer School on High Performance Computing New York, NY, USA June 28, 2013 Lars Koesterke: Research Staff @ TACC Nomenclature Architecture Model -------------------------------------------------------
More informationIntroduction to parallel computers and parallel programming. Introduction to parallel computersand parallel programming p. 1
Introduction to parallel computers and parallel programming Introduction to parallel computersand parallel programming p. 1 Content A quick overview of morden parallel hardware Parallelism within a chip
More informationAdaStreams : A Type-based Programming Extension for Stream-Parallelism with Ada 2005
AdaStreams : A Type-based Programming Extension for Stream-Parallelism with Ada 2005 Gingun Hong*, Kirak Hong*, Bernd Burgstaller* and Johan Blieberger *Yonsei University, Korea Vienna University of Technology,
More informationParallelism. Parallel Hardware. Introduction to Computer Systems
Parallelism We have been discussing the abstractions and implementations that make up an individual computer system in considerable detail up to this point. Our model has been a largely sequential one,
More informationHow to write code that will survive the many-core revolution Write once, deploy many(-cores) F. Bodin, CTO
How to write code that will survive the many-core revolution Write once, deploy many(-cores) F. Bodin, CTO Foreword How to write code that will survive the many-core revolution? is being setup as a collective
More informationParallel Programming Libraries and implementations
Parallel Programming Libraries and implementations Partners Funding Reusing this material This work is licensed under a Creative Commons Attribution- NonCommercial-ShareAlike 4.0 International License.
More informationData Parallel Algorithmic Skeletons with Accelerator Support
MÜNSTER Data Parallel Algorithmic Skeletons with Accelerator Support Steffen Ernsting and Herbert Kuchen July 2, 2015 Agenda WESTFÄLISCHE MÜNSTER Data Parallel Algorithmic Skeletons with Accelerator Support
More informationPROGRAMOVÁNÍ V C++ CVIČENÍ. Michal Brabec
PROGRAMOVÁNÍ V C++ CVIČENÍ Michal Brabec PARALLELISM CATEGORIES CPU? SSE Multiprocessor SIMT - GPU 2 / 17 PARALLELISM V C++ Weak support in the language itself, powerful libraries Many different parallelization
More informationResources Current and Future Systems. Timothy H. Kaiser, Ph.D.
Resources Current and Future Systems Timothy H. Kaiser, Ph.D. tkaiser@mines.edu 1 Most likely talk to be out of date History of Top 500 Issues with building bigger machines Current and near future academic
More informationPARALLEL DESIGN PATTERNS
CHAPTER 6 PARALLEL DESIGN PATTERNS Design patterns have been introduced in the 90 to describe simple and elegant solutions to specific problems in object-oriented software design. Design patterns capture
More informationIntel Thread Building Blocks
Intel Thread Building Blocks SPD course 2015-16 Massimo Coppola 08/04/2015 1 Thread Building Blocks : History A library to simplify writing thread-parallel programs and debugging them Originated circa
More informationMaster Program (Laurea Magistrale) in Computer Science and Networking. High Performance Computing Systems and Enabling Platforms.
Master Program (Laurea Magistrale) in Computer Science and Networking High Performance Computing Systems and Enabling Platforms Marco Vanneschi Multithreading Contents Main features of explicit multithreading
More informationDesign Principles for End-to-End Multicore Schedulers
c Systems Group Department of Computer Science ETH Zürich HotPar 10 Design Principles for End-to-End Multicore Schedulers Simon Peter Adrian Schüpbach Paul Barham Andrew Baumann Rebecca Isaacs Tim Harris
More informationStarPU: a runtime system for multigpu multicore machines
StarPU: a runtime system for multigpu multicore machines Raymond Namyst RUNTIME group, INRIA Bordeaux Journées du Groupe Calcul Lyon, November 2010 The RUNTIME Team High Performance Runtime Systems for
More informationCMSC 714 Lecture 6 MPI vs. OpenMP and OpenACC. Guest Lecturer: Sukhyun Song (original slides by Alan Sussman)
CMSC 714 Lecture 6 MPI vs. OpenMP and OpenACC Guest Lecturer: Sukhyun Song (original slides by Alan Sussman) Parallel Programming with Message Passing and Directives 2 MPI + OpenMP Some applications can
More informationComputing architectures Part 2 TMA4280 Introduction to Supercomputing
Computing architectures Part 2 TMA4280 Introduction to Supercomputing NTNU, IMF January 16. 2017 1 Supercomputing What is the motivation for Supercomputing? Solve complex problems fast and accurately:
More informationTowards a codelet-based runtime for exascale computing. Chris Lauderdale ET International, Inc.
Towards a codelet-based runtime for exascale computing Chris Lauderdale ET International, Inc. What will be covered Slide 2 of 24 Problems & motivation Codelet runtime overview Codelets & complexes Dealing
More informationParallel Patterns for Window-based Stateful Operators on Data Streams: an Algorithmic Skeleton Approach
Parallel Patterns for Window-based Stateful Operators on Data Streams: an Algorithmic Skeleton Approach Tiziano De Matteis, Gabriele Mencagli University of Pisa Italy INTRODUCTION The recent years have
More informationIntroduction: Modern computer architecture. The stored program computer and its inherent bottlenecks Multi- and manycore chips and nodes
Introduction: Modern computer architecture The stored program computer and its inherent bottlenecks Multi- and manycore chips and nodes Motivation: Multi-Cores where and why Introduction: Moore s law Intel
More informationIntel Thread Building Blocks
Intel Thread Building Blocks SPD course 2017-18 Massimo Coppola 23/03/2018 1 Thread Building Blocks : History A library to simplify writing thread-parallel programs and debugging them Originated circa
More informationIntroduction to Parallel Programming For Real-Time Graphics (CPU + GPU)
Introduction to Parallel Programming For Real-Time Graphics (CPU + GPU) Aaron Lefohn, Intel / University of Washington Mike Houston, AMD / Stanford 1 What s In This Talk? Overview of parallel programming
More informationHigh Performance Computing on GPUs using NVIDIA CUDA
High Performance Computing on GPUs using NVIDIA CUDA Slides include some material from GPGPU tutorial at SIGGRAPH2007: http://www.gpgpu.org/s2007 1 Outline Motivation Stream programming Simplified HW and
More informationSupporting Advanced Patterns in GrPPI, a Generic Parallel Pattern Interface
Supporting Advanced Patterns in GrPPI, a Generic Parallel Pattern Interface David del Rio, Manuel F. Dolz, Javier Fernández. Daniel Garcia University Carlos III of Madrid Autonomic Solutions for Parallel
More informationCOMPUTER ORGANIZATION AND DESIGN The Hardware/Software Interface. 5 th. Edition. Chapter 6. Parallel Processors from Client to Cloud
COMPUTER ORGANIZATION AND DESIGN The Hardware/Software Interface 5 th Edition Chapter 6 Parallel Processors from Client to Cloud Introduction Goal: connecting multiple computers to get higher performance
More informationCUDA GPGPU Workshop 2012
CUDA GPGPU Workshop 2012 Parallel Programming: C thread, Open MP, and Open MPI Presenter: Nasrin Sultana Wichita State University 07/10/2012 Parallel Programming: Open MP, MPI, Open MPI & CUDA Outline
More informationParallel Programming Principle and Practice. Lecture 9 Introduction to GPGPUs and CUDA Programming Model
Parallel Programming Principle and Practice Lecture 9 Introduction to GPGPUs and CUDA Programming Model Outline Introduction to GPGPUs and Cuda Programming Model The Cuda Thread Hierarchy / Memory Hierarchy
More informationOpenACC/CUDA/OpenMP... 1 Languages and Libraries... 3 Multi-GPU support... 4 How OpenACC Works... 4
OpenACC Course Class #1 Q&A Contents OpenACC/CUDA/OpenMP... 1 Languages and Libraries... 3 Multi-GPU support... 4 How OpenACC Works... 4 OpenACC/CUDA/OpenMP Q: Is OpenACC an NVIDIA standard or is it accepted
More informationFinite Element Integration and Assembly on Modern Multi and Many-core Processors
Finite Element Integration and Assembly on Modern Multi and Many-core Processors Krzysztof Banaś, Jan Bielański, Kazimierz Chłoń AGH University of Science and Technology, Mickiewicza 30, 30-059 Kraków,
More informationCommunicating Process Architectures in Light of Parallel Design Patterns and Skeletons
Communicating Process Architectures in Light of Parallel Design Patterns and Skeletons Dr Kevin Chalmers School of Computing Edinburgh Napier University Edinburgh k.chalmers@napier.ac.uk Overview ˆ I started
More informationParticle-in-Cell Simulations on Modern Computing Platforms. Viktor K. Decyk and Tajendra V. Singh UCLA
Particle-in-Cell Simulations on Modern Computing Platforms Viktor K. Decyk and Tajendra V. Singh UCLA Outline of Presentation Abstraction of future computer hardware PIC on GPUs OpenCL and Cuda Fortran
More informationProgramming Models for Multi- Threading. Brian Marshall, Advanced Research Computing
Programming Models for Multi- Threading Brian Marshall, Advanced Research Computing Why Do Parallel Computing? Limits of single CPU computing performance available memory I/O rates Parallel computing allows
More informationParallel stochastic simulators in systems biology: the evolution of the species
FACOLTÀ Università degli Studi di Torino DI SCIENZE MATEMATICHE, FISICHE E NATURALI Corso di Laurea in Informatica Tesi di Laurea Magistrale Parallel stochastic simulators in systems biology: the evolution
More informationPool evolution: a parallel pattern for evolutionary and symbolic computing
This is an author version of the contribution published by Springer on INTERNATIONAL JOURNAL OF PARALLEL PROGRAMMING DOI: 10.1007/s10766-013-0273-6 Pool evolution: a parallel pattern for evolutionary and
More informationGenerating Efficient Data Movement Code for Heterogeneous Architectures with Distributed-Memory
Generating Efficient Data Movement Code for Heterogeneous Architectures with Distributed-Memory Roshan Dathathri Thejas Ramashekar Chandan Reddy Uday Bondhugula Department of Computer Science and Automation
More informationFPGA-based Supercomputing: New Opportunities and Challenges
FPGA-based Supercomputing: New Opportunities and Challenges Naoya Maruyama (RIKEN AICS)* 5 th ADAC Workshop Feb 15, 2018 * Current Main affiliation is Lawrence Livermore National Laboratory SIAM PP18:
More informationProductive Performance on the Cray XK System Using OpenACC Compilers and Tools
Productive Performance on the Cray XK System Using OpenACC Compilers and Tools Luiz DeRose Sr. Principal Engineer Programming Environments Director Cray Inc. 1 The New Generation of Supercomputers Hybrid
More information2 Introduction to parallel computing. Chip Multiprocessors (ACS MPhil) Robert Mullins
2 Introduction to parallel computing Robert Mullins Overview Parallel computing platforms Approaches to building parallel computers Today's chip-multiprocessor architectures Approaches to parallel programming
More informationParallel Programming on Ranger and Stampede
Parallel Programming on Ranger and Stampede Steve Lantz Senior Research Associate Cornell CAC Parallel Computing at TACC: Ranger to Stampede Transition December 11, 2012 What is Stampede? NSF-funded XSEDE
More informationCUDA PROGRAMMING MODEL Chaithanya Gadiyam Swapnil S Jadhav
CUDA PROGRAMMING MODEL Chaithanya Gadiyam Swapnil S Jadhav CMPE655 - Multiple Processor Systems Fall 2015 Rochester Institute of Technology Contents What is GPGPU? What s the need? CUDA-Capable GPU Architecture
More informationHigh Performance Computing (HPC) Introduction
High Performance Computing (HPC) Introduction Ontario Summer School on High Performance Computing Scott Northrup SciNet HPC Consortium Compute Canada June 25th, 2012 Outline 1 HPC Overview 2 Parallel Computing
More informationPortability and Scalability of Sparse Tensor Decompositions on CPU/MIC/GPU Architectures
Photos placed in horizontal position with even amount of white space between photos and header Portability and Scalability of Sparse Tensor Decompositions on CPU/MIC/GPU Architectures Christopher Forster,
More informationAMath 483/583, Lecture 24, May 20, Notes: Notes: What s a GPU? Notes: Some GPU application areas
AMath 483/583 Lecture 24 May 20, 2011 Today: The Graphical Processing Unit (GPU) GPU Programming Today s lecture developed and presented by Grady Lemoine References: Andreas Kloeckner s High Performance
More informationIntroduction to Parallel Computing. CPS 5401 Fall 2014 Shirley Moore, Instructor October 13, 2014
Introduction to Parallel Computing CPS 5401 Fall 2014 Shirley Moore, Instructor October 13, 2014 1 Definition of Parallel Computing Simultaneous use of multiple compute resources to solve a computational
More informationCompiling for GPUs. Adarsh Yoga Madhav Ramesh
Compiling for GPUs Adarsh Yoga Madhav Ramesh Agenda Introduction to GPUs Compute Unified Device Architecture (CUDA) Control Structure Optimization Technique for GPGPU Compiler Framework for Automatic Translation
More informationET International HPC Runtime Software. ET International Rishi Khan SC 11. Copyright 2011 ET International, Inc.
HPC Runtime Software Rishi Khan SC 11 Current Programming Models Shared Memory Multiprocessing OpenMP fork/join model Pthreads Arbitrary SMP parallelism (but hard to program/ debug) Cilk Work Stealing
More informationOpenMP for next generation heterogeneous clusters
OpenMP for next generation heterogeneous clusters Jens Breitbart Research Group Programming Languages / Methodologies, Universität Kassel, jbreitbart@uni-kassel.de Abstract The last years have seen great
More informationOpenACC 2.6 Proposed Features
OpenACC 2.6 Proposed Features OpenACC.org June, 2017 1 Introduction This document summarizes features and changes being proposed for the next version of the OpenACC Application Programming Interface, tentatively
More informationHigh performance Computing and O&G Challenges
High performance Computing and O&G Challenges 2 Seismic exploration challenges High Performance Computing and O&G challenges Worldwide Context Seismic,sub-surface imaging Computing Power needs Accelerating
More informationCME 213 S PRING Eric Darve
CME 213 S PRING 2017 Eric Darve Summary of previous lectures Pthreads: low-level multi-threaded programming OpenMP: simplified interface based on #pragma, adapted to scientific computing OpenMP for and
More informationCopyright Khronos Group, Page 1 SYCL. SG14, February 2016
Copyright Khronos Group, 2014 - Page 1 SYCL SG14, February 2016 BOARD OF PROMOTERS Over 100 members worldwide any company is welcome to join Copyright Khronos Group 2014 SYCL 1. What is SYCL for and what
More informationChapter 3 Parallel Software
Chapter 3 Parallel Software Part I. Preliminaries Chapter 1. What Is Parallel Computing? Chapter 2. Parallel Hardware Chapter 3. Parallel Software Chapter 4. Parallel Applications Chapter 5. Supercomputers
More informationGPU Fundamentals Jeff Larkin November 14, 2016
GPU Fundamentals Jeff Larkin , November 4, 206 Who Am I? 2002 B.S. Computer Science Furman University 2005 M.S. Computer Science UT Knoxville 2002 Graduate Teaching Assistant 2005 Graduate
More informationOpenCL TM & OpenMP Offload on Sitara TM AM57x Processors
OpenCL TM & OpenMP Offload on Sitara TM AM57x Processors 1 Agenda OpenCL Overview of Platform, Execution and Memory models Mapping these models to AM57x Overview of OpenMP Offload Model Compare and contrast
More informationComputer Architecture and Structured Parallel Programming James Reinders, Intel
Computer Architecture and Structured Parallel Programming James Reinders, Intel Parallel Computing CIS 410/510 Department of Computer and Information Science Lecture 17 Manycore Computing and GPUs Computer
More informationIntroduction to Runtime Systems
Introduction to Runtime Systems Towards Portability of Performance ST RM Static Optimizations Runtime Methods Team Storm Olivier Aumage Inria LaBRI, in cooperation with La Maison de la Simulation Contents
More informationAnalyzing the Performance of IWAVE on a Cluster using HPCToolkit
Analyzing the Performance of IWAVE on a Cluster using HPCToolkit John Mellor-Crummey and Laksono Adhianto Department of Computer Science Rice University {johnmc,laksono}@rice.edu TRIP Meeting March 30,
More informationHeterogeneous Computing and OpenCL
Heterogeneous Computing and OpenCL Hongsuk Yi (hsyi@kisti.re.kr) (Korea Institute of Science and Technology Information) Contents Overview of the Heterogeneous Computing Introduction to Intel Xeon Phi
More informationLecture 9: MIMD Architectures
Lecture 9: MIMD Architectures Introduction and classification Symmetric multiprocessors NUMA architecture Clusters Zebo Peng, IDA, LiTH 1 Introduction A set of general purpose processors is connected together.
More informationPresenting: Comparing the Power and Performance of Intel's SCC to State-of-the-Art CPUs and GPUs
Presenting: Comparing the Power and Performance of Intel's SCC to State-of-the-Art CPUs and GPUs A paper comparing modern architectures Joakim Skarding Christian Chavez Motivation Continue scaling of performance
More informationHybrid Implementation of 3D Kirchhoff Migration
Hybrid Implementation of 3D Kirchhoff Migration Max Grossman, Mauricio Araya-Polo, Gladys Gonzalez GTC, San Jose March 19, 2013 Agenda 1. Motivation 2. The Problem at Hand 3. Solution Strategy 4. GPU Implementation
More informationHPC with Multicore and GPUs
HPC with Multicore and GPUs Stan Tomov Electrical Engineering and Computer Science Department University of Tennessee, Knoxville COSC 594 Lecture Notes March 22, 2017 1/20 Outline Introduction - Hardware
More informationIntroduction II. Overview
Introduction II Overview Today we will introduce multicore hardware (we will introduce many-core hardware prior to learning OpenCL) We will also consider the relationship between computer hardware and
More informationHPC trends (Myths about) accelerator cards & more. June 24, Martin Schreiber,
HPC trends (Myths about) accelerator cards & more June 24, 2015 - Martin Schreiber, M.Schreiber@exeter.ac.uk Outline HPC & current architectures Performance: Programming models: OpenCL & OpenMP Some applications:
More informationChapter 6. Parallel Processors from Client to Cloud. Copyright 2014 Elsevier Inc. All rights reserved.
Chapter 6 Parallel Processors from Client to Cloud FIGURE 6.1 Hardware/software categorization and examples of application perspective on concurrency versus hardware perspective on parallelism. 2 FIGURE
More informationTutorial. Preparing for Stampede: Programming Heterogeneous Many-Core Supercomputers
Tutorial Preparing for Stampede: Programming Heterogeneous Many-Core Supercomputers Dan Stanzione, Lars Koesterke, Bill Barth, Kent Milfeld dan/lars/bbarth/milfeld@tacc.utexas.edu XSEDE 12 July 16, 2012
More informationExecution Strategy and Runtime Support for Regular and Irregular Applications on Emerging Parallel Architectures
Execution Strategy and Runtime Support for Regular and Irregular Applications on Emerging Parallel Architectures Xin Huo Advisor: Gagan Agrawal Motivation - Architecture Challenges on GPU architecture
More informationMulti/Many Core Programming Strategies
Multicore Challenge Conference 2012 UWE, Bristol Multi/Many Core Programming Strategies Greg Michaelson School of Mathematical & Computer Sciences Heriot-Watt University Multicore Challenge Conference
More informationParaFormance: An Advanced Refactoring Tool for Parallelising C++ Programs
n project Enterprise : An Advanced Refactoring Tool for Parallelising C++ Programs Chris Brown Craig Manson, Kenneth MacKenzie, Vladimir Janjic, Kevin Hammond University of St Andrews, Scotland @chrismarkbrown
More informationOpenMPSuperscalar: Task-Parallel Simulation and Visualization of Crowds with Several CPUs and GPUs
www.bsc.es OpenMPSuperscalar: Task-Parallel Simulation and Visualization of Crowds with Several CPUs and GPUs Hugo Pérez UPC-BSC Benjamin Hernandez Oak Ridge National Lab Isaac Rudomin BSC March 2015 OUTLINE
More information