8. Hardware-Aware Numerics. Approaching supercomputing...
|
|
|
- Kory Cummings
- 6 years ago
- Views:
Transcription
1 Approaching supercomputing... Numerisches Programmieren, Hans-Joachim Bungartz page 1 of 48
2 8.1. Hardware-Awareness Introduction Since numerical algorithms are ubiquitous, they have to run on a broad spectrum of processors or devices, resp.: commodity CPU (Intel, AMD,... ) special supercomputing CPU (vector processors,... ) special-purpose processors such as GPU (NVIDIA,... ) or the Cell Broadband Engine (in Sony s PlayStation) other devices: PDA, iphone,... While the classical concern of numerical algorithms lies on the algorithmic side (speed of convergence, complexity in terms of O(N k ), accuracy in terms of O(h k ), memory consumption), it has become obvious that this is not sufficient for performance, i. e. short run times implementational aspects gain more and more in importance: tailoring data structures exploiting pipelining exploiting memory hierarchies (the different cache levels, esp.) exploiting on-chip parallelism Numerisches Programmieren, Hans-Joachim Bungartz page 2 of 48
3 Of course, there needs to be a balance between code performance on the one side and code portability on the other side: hardware-conscious: increasing performance hardware-oblivious: increasing performance by aligning algorithm design to general architectural features, without taking into account specific details of the respective architecture in the algorithm design hardware-aware: comprises all measures that try to adapt algorithms to the underlying hardware, i.e. comprises hardware-conscious and hardware-oblivious Numerisches Programmieren, Hans-Joachim Bungartz page 3 of 48
4 Relevance Program a matrix-vector or a matrix-matrix product of increasing dimension: at some point, performance will decrease tremendously. Staying two to four orders of magnitude below the processor s peak performance is not a rare event, if an algorithm is coded without additional considerations. One problem is the so-called memory bottleneck or memory wall consider the average growth rates in the last years: CPU performance: 60% memory bandwidth: 23% memory latency: 5% Another hot topic arises from today s ubiquitous parallelism in present multi-core and upcoming many-core systems. Take a moment to think about possible parallelization strategies for the Jacobi or the Gauß-Seidel methods discussed in the chapter on iterative schemes. Tackling such problems is one focus of Scientific Computing. In this chapter, we will concentrate on one aspect: increasing cache-efficiency for matrix-matrix multiplication. Numerisches Programmieren, Hans-Joachim Bungartz page 4 of 48
5 8.2. Space-Filling Curves Introduction An unconventional strategy for cache-efficiency Origin of the idea: analysis and topology ( topological monsters ) Nice example of a construct from pure mathematics that gets practical relevance decades later Definition of a space-filling curve (SFC), for reasons of simplicity only in 2 D: Curve: image of a continuous mapping of the unit interval [0, 1] onto the unit square [0, 1] 2 Space-filling: curve covers the whole unit square (mapping is surjective) and, hence, covers an area greater than zero(!) f : [0, 1] =: I Q := [0, 1] 2, f surjective and continuous Prominent representatives: Hilbert s curve: 1891, the most famous space-filling curve Peano s curve: 1890, oldest space-filling curve Lebesgue s curve: quadtree principle, probably the most important SFC for computer science Numerisches Programmieren, Hans-Joachim Bungartz page 5 of 48
6 Hilbert s SFC The construction follows the geometric conception: if I can be mapped onto Q in the space-filling sense, then each of the four congruent subintervals of I can be mapped to one of the four quadrants of Q in the space-filling sense, too. Recursive application of this partitioning and allocation process preserving Neighborhood relations: neighboring subintervals in I are mapped onto neighboring subsquares of Q. Subset relations (inclusion): from I 1 I 2 follows f(i 1 ) f(i 2 ) Limit case: Hilbert s curve From the correspondence of nestings of intervals in I and nestings of squares in Q, we get pairs of points in I and of corresponding image points in Q. Of course, the iterative steps in this generation process are of practical relevance, not the limit case (the SFC) itself. Start with a generator (defines the order in which the subsquares are visited ) Apply generator in each subsquare (with appropriate similarity transformations) Connect the open ends Numerisches Programmieren, Hans-Joachim Bungartz page 6 of 48
7 Generation Processes with Hilbert s Generator Classical version of Hilbert: Variant of Moore: Modulo symmetry, these are the only two possibilities! Numerisches Programmieren, Hans-Joachim Bungartz page 7 of 48
8 Peano s SFC Ancestor of all SFCs Subdivision of I and Q into nine congruent subdomains Definition of a leitmotiv, again, defines the order of visit Now, there are 273 different (modulo symmetry) possibilities to recursively apply the generator preserving neighborhood and inclusion Serpentine type (left and center) and meander type (right) Numerisches Programmieren, Hans-Joachim Bungartz page 8 of 48
9 8.3. Matrix-Matrix Multiplication Relevance and Standard Algorithm Matrix-matrix multiplication is not a such frequently used building block of numerical algorithms as matrix-vector multiplication is. Nevertheless several appearances: Computational chemistry: computing changes of state in chemical systems Signal processing: performing some classes of transforms Standard sequential algorithm for two quadratic matrices A, B R M,M : for i=1 to n do for j=1 to n do c[i,j] := 0; for k=1 to n do c[i,j] := c[i,j]+a[i,k]*b[k,j]; That is: a sequence of M 2 scalar products of two vectors of length M For full matrices we get cubic complexity. Numerisches Programmieren, Hans-Joachim Bungartz page 9 of 48
10 Observation In a single iteration of the outer loop indexed by i, row i of matrix A and all rows of matrix B are read, while row i of matrix C is written. Consequence: once M reaches a certain size, B won t fit completely into the cache any more, and performance will fall dramatically (frequent cache misses and, hence, main memory accesses during each outer iteration step, i. e. row of A) Remedy: a recursive variant working with blocks of B only instead of the whole matrix B Numerisches Programmieren, Hans-Joachim Bungartz page 10 of 48
11 Recursive Block-Oriented Algorithm Subdivide both A and B into four smaller submatrices of consistent dimensions: ( ) ( ) A00 A A = 01 B00 B B = 01 A 10 A 11 B 10 B 11 The matrix product then reads ( ) A00 B C = 00 + A 01 B 10 A 00 B 01 + A 01 B 11 A 10 B 00 + A 11 B 10 A 10 B 01 + A 11 B 11 (compare the product of two 2 2-matrices) If the blocks of B are still too large for the cache, this subdivision step can be applied recursively to finally overcome the cache problem. Today, block-recursive approaches are widespread techniques which, by construction, leads to inherently good data access patterns and, thus, to good cache performance. This strategy is also important for parallel matrix-matrix algorithms. Numerisches Programmieren, Hans-Joachim Bungartz page 11 of 48
12 8.4. Peano-Based Matrix-Matrix Product Numerisches Programmieren, Hans-Joachim Bungartz page 12 of 48
13 1 1 1 Numerisches Programmieren, Hans-Joachim Bungartz page 13 of 48
14 Numerisches Programmieren, Hans-Joachim Bungartz page 14 of 48
15 Numerisches Programmieren, Hans-Joachim Bungartz page 15 of 48
16 Numerisches Programmieren, Hans-Joachim Bungartz page 16 of 48
17 Numerisches Programmieren, Hans-Joachim Bungartz page 17 of 48
18 Numerisches Programmieren, Hans-Joachim Bungartz page 18 of 48
19 Numerisches Programmieren, Hans-Joachim Bungartz page 19 of 48
20 Numerisches Programmieren, Hans-Joachim Bungartz page 20 of 48
21 Numerisches Programmieren, Hans-Joachim Bungartz page 21 of 48
22 Numerisches Programmieren, Hans-Joachim Bungartz page 22 of 48
23 Numerisches Programmieren, Hans-Joachim Bungartz page 23 of 48
24 Numerisches Programmieren, Hans-Joachim Bungartz page 24 of 48
25 Numerisches Programmieren, Hans-Joachim Bungartz page 25 of 48
26 Numerisches Programmieren, Hans-Joachim Bungartz page 26 of 48
27 Numerisches Programmieren, Hans-Joachim Bungartz page 27 of 48
28 Numerisches Programmieren, Hans-Joachim Bungartz page 28 of 48
29 Numerisches Programmieren, Hans-Joachim Bungartz page 29 of 48
30 Numerisches Programmieren, Hans-Joachim Bungartz page 30 of 48
31 Numerisches Programmieren, Hans-Joachim Bungartz page 31 of 48
32 Numerisches Programmieren, Hans-Joachim Bungartz page 32 of 48
33 Numerisches Programmieren, Hans-Joachim Bungartz page 33 of 48
34 Numerisches Programmieren, Hans-Joachim Bungartz page 34 of 48
35 Numerisches Programmieren, Hans-Joachim Bungartz page 35 of 48
36 Numerisches Programmieren, Hans-Joachim Bungartz page 36 of 48
37 Numerisches Programmieren, Hans-Joachim Bungartz page 37 of 48
38 Numerisches Programmieren, Hans-Joachim Bungartz page 38 of 48
39 Numerisches Programmieren, Hans-Joachim Bungartz page 39 of 48
40 Matrix Multiplication and 3D/2D Peano Traversals inherently cache efficient 3D-traversal of the block operations C[i,j] += A[i,k] * B[k,j] using a Peano curve projections of 3D curve to 2D-planes lead to 2D Peano curves 2D-planes correspond to the indices of A, B, and C: (i, k), (k, j), and (i, j) use Peano layout for matrices Goal: strictly local element access Numerisches Programmieren, Hans-Joachim Bungartz page 40 of 48
41 Block-Recursive Peano Element Order indexing according to iteration of a Peano curve P R P P Q S Q P R P Q S Q Q P R P Q S Q stopped on L1 blocks (size tuned to L1 cache 2 matrix blocks should fit) use column-major layout within L1-blocks Numerisches Programmieren, Hans-Joachim Bungartz page 41 of 48
42 Block-Recursive Multiplication multiplication of block matrices (cmp. 3 3-scheme): P A0 R A5 P A6 P B0 R B5 P B6 P C0 R C5 P C6 Q A1 S A4 Q A7 Q B1 S B4 Q B7 Q C1 S C4 Q C7 P A2 R A3 P A8 P B2 R B3 P B8 P C2 R C3 P C8 leads to eight different combinations (w.r.t. block numbering): P P P QR S RS R SQ Q P R R QP Q RQ P SS S all schemes similar to P P P (but reverse order) Numerisches Programmieren, Hans-Joachim Bungartz page 42 of 48
43 Access Pattern to the Matrix Blocks A B C Increment/Decrement access to elements O(k 3 ) operations on any block of k 2 elements Numerisches Programmieren, Hans-Joachim Bungartz page 43 of 48
44 Performance Memory Latency & Bandwidth Dual vs. Quad channel memory (Xeon, 2 quadcore) TifaMMy: MFlops/s Dual Ch. 8 Threads Quad Ch. 8 Threads Matrix Dimension GotoBLAS: MFlops/s Dual Ch. 8 Threads Quad Ch. 8 Threads Matrix Dimension Numerisches Programmieren, Hans-Joachim Bungartz page 44 of 48
45 Towards LU and ILU Decomposition 1. Sparse-Dense Matrix Operations: extension of the Peano approach to sparse matrices tree-oriented storage of dense and sparse matrices (L1 blocks zero, dense, or sparse) 2. Parallel LU Decomposition: block-oriented LU decomposition based on Peano curve shared-memory parallelisation with OpenMP 3 3. Towards ILU Decomposition: is it feasible at all? increase level of parallelism during parallelisation Numerisches Programmieren, Hans-Joachim Bungartz page 45 of 48
46 Extension for Sparse Matrices Data structure: 1. allow zero and dense blocks as L1 blocks 2. allow compressed sparse row (CSR) blocks as L1 blocks 3. adopt quadtree-like storage for matrices Algorithm: 1. keep block-recursive Peano scheme 2. skip operations involving zero blocks 3. implement L1 operations on dense and sparse blocks Numerisches Programmieren, Hans-Joachim Bungartz page 46 of 48
47 Sparse Peano Tree to Store Sparse Matrices Tree-structure sequentialised in Peano order: s[0] s[6] s[5] s[1] s[7] s[2] s[4] s[3] s[8] s[9]... dense block sparse block zero block s[0]s[1] s[2] s[3] s[4] s[5] s[6] s[7] s[8] s[9]... s[0] s[1] s[3] modified depth-first traversal (child information in parent node) Numerisches Programmieren, Hans-Joachim Bungartz page 47 of 48
48 Matrix Exponentials in Quantum Control quantum states modelled by evolution matrices: U (r) (t k ) = e i th(r) k e i th(r) k 1 e i th(r) 1 wanted: exponential function of sparse matrices H k : computed via Chebyshev polynomials requires sparse-dense matrix multiplication Numerisches Programmieren, Hans-Joachim Bungartz page 48 of 48
8. Hardware-Aware Numerics. Approaching supercomputing...
Approaching supercomputing... Numerisches Programmieren, Hans-Joachim Bungartz page 1 of 22 8.1. Hardware-Awareness Introduction Since numerical algorithms are ubiquitous, they have to run on a broad spectrum
Cache Oblivious Dense and Sparse Matrix Multiplication Based on Peano Curves
Cache Oblivious Dense and Sparse Matrix Multiplication Based on eano Curves Michael Bader and Alexander Heinecke Institut für Informatik, Technische Universitüt München, Germany Abstract. Cache oblivious
Efficient Storage and Processing of Adaptive Triangular Grids using Sierpinski Curves
Efficient Storage and Processing of Adaptive Triangular Grids using Sierpinski Curves Csaba Attila Vigh, Dr. Michael Bader Department of Informatics, TU München JASS 2006, course 2: Numerical Simulation:
HPC Algorithms and Applications
HPC Algorithms and Applications Dwarf #5 Structured Grids Michael Bader Winter 2012/2013 Dwarf #5 Structured Grids, Winter 2012/2013 1 Dwarf #5 Structured Grids 1. dense linear algebra 2. sparse linear
Algorithms for GIS csci3225
Algorithms for GIS csci3225 Laura Toma Bowdoin College COB multiplication, matrix layout and space-filling curves Matrix layout Matrix a is given in row-major order 8 row-major order 0 1 2 3 4 5 6 7 8
Space-Filling Curves An Introduction
Department of Informatics Technical University Munich Space-Filling Curves An Introduction Paper accompanying the presentation held on April nd 005 for the Joint Advanced Student School (JASS) in St. Petersburg
Principle Of Parallel Algorithm Design (cont.) Alexandre David B2-206
Principle Of Parallel Algorithm Design (cont.) Alexandre David B2-206 1 Today Characteristics of Tasks and Interactions (3.3). Mapping Techniques for Load Balancing (3.4). Methods for Containing Interaction
Behavioral Data Mining. Lecture 12 Machine Biology
Behavioral Data Mining Lecture 12 Machine Biology Outline CPU geography Mass storage Buses and Networks Main memory Design Principles Intel i7 close-up From Computer Architecture a Quantitative Approach
Efficient Algorithmic Approaches for Flow Simulations on Cartesian Grids
Efficient Algorithmic Approaches for Flow Simulations on Cartesian Grids M. Bader, H.-J. Bungartz, B. Gatzhammer, M. Mehl, T. Neckel, T. Weinzierl TUM Department of Informatics Chair of Scientific Computing
Space Filling Curves
Algorithms for GIS Space Filling Curves Laura Toma Bowdoin College A map from an interval to a square Space filling curves Space filling curves https://mathsbyagirl.wordpress.com/tag/curve/ A map from
On Level Scheduling for Incomplete LU Factorization Preconditioners on Accelerators
On Level Scheduling for Incomplete LU Factorization Preconditioners on Accelerators Karl Rupp, Barry Smith rupp@mcs.anl.gov Mathematics and Computer Science Division Argonne National Laboratory FEMTEC
Lecture 13: March 25
CISC 879 Software Support for Multicore Architectures Spring 2007 Lecture 13: March 25 Lecturer: John Cavazos Scribe: Ying Yu 13.1. Bryan Youse-Optimization of Sparse Matrix-Vector Multiplication on Emerging
Modern GPUs (Graphics Processing Units)
Modern GPUs (Graphics Processing Units) Powerful data parallel computation platform. High computation density, high memory bandwidth. Relatively low cost. NVIDIA GTX 580 512 cores 1.6 Tera FLOPs 1.5 GB
Module 18: "TLP on Chip: HT/SMT and CMP" Lecture 39: "Simultaneous Multithreading and Chip-multiprocessing" TLP on Chip: HT/SMT and CMP SMT
TLP on Chip: HT/SMT and CMP SMT Multi-threading Problems of SMT CMP Why CMP? Moore s law Power consumption? Clustered arch. ABCs of CMP Shared cache design Hierarchical MP file:///e /parallel_com_arch/lecture39/39_1.htm[6/13/2012
Algorithms for GIS: Space filling curves
Algorithms for GIS: Space filling curves Z-order visit quadrants recursively in this order: NW, NE, SW, SE Z-order visit quadrants recursively in this order: NW, NE, SW, SE Z-order visit quadrants recursively
AmgX 2.0: Scaling toward CORAL Joe Eaton, November 19, 2015
AmgX 2.0: Scaling toward CORAL Joe Eaton, November 19, 2015 Agenda Introduction to AmgX Current Capabilities Scaling V2.0 Roadmap for the future 2 AmgX Fast, scalable linear solvers, emphasis on iterative
Programming Models for Supercomputing in the Era of Multicore
Programming Models for Supercomputing in the Era of Multicore Marc Snir MULTI-CORE CHALLENGES 1 Moore s Law Reinterpreted Number of cores per chip doubles every two years, while clock speed decreases Need
Kartik Lakhotia, Rajgopal Kannan, Viktor Prasanna USENIX ATC 18
Accelerating PageRank using Partition-Centric Processing Kartik Lakhotia, Rajgopal Kannan, Viktor Prasanna USENIX ATC 18 Outline Introduction Partition-centric Processing Methodology Analytical Evaluation
Performance of Multicore LUP Decomposition
Performance of Multicore LUP Decomposition Nathan Beckmann Silas Boyd-Wickizer May 3, 00 ABSTRACT This paper evaluates the performance of four parallel LUP decomposition implementations. The implementations
Bandwidth Avoiding Stencil Computations
Bandwidth Avoiding Stencil Computations By Kaushik Datta, Sam Williams, Kathy Yelick, and Jim Demmel, and others Berkeley Benchmarking and Optimization Group UC Berkeley March 13, 2008 http://bebop.cs.berkeley.edu
Numerical Algorithms
Chapter 10 Slide 464 Numerical Algorithms Slide 465 Numerical Algorithms In textbook do: Matrix multiplication Solving a system of linear equations Slide 466 Matrices A Review An n m matrix Column a 0,0
Performance without Pain = Productivity Data Layout and Collective Communication in UPC
Performance without Pain = Productivity Data Layout and Collective Communication in UPC By Rajesh Nishtala (UC Berkeley), George Almási (IBM Watson Research Center), Călin Caşcaval (IBM Watson Research
GTC 2013: DEVELOPMENTS IN GPU-ACCELERATED SPARSE LINEAR ALGEBRA ALGORITHMS. Kyle Spagnoli. Research EM Photonics 3/20/2013
GTC 2013: DEVELOPMENTS IN GPU-ACCELERATED SPARSE LINEAR ALGEBRA ALGORITHMS Kyle Spagnoli Research Engineer @ EM Photonics 3/20/2013 INTRODUCTION» Sparse systems» Iterative solvers» High level benchmarks»
Introducing a Cache-Oblivious Blocking Approach for the Lattice Boltzmann Method
Introducing a Cache-Oblivious Blocking Approach for the Lattice Boltzmann Method G. Wellein, T. Zeiser, G. Hager HPC Services Regional Computing Center A. Nitsure, K. Iglberger, U. Rüde Chair for System
Fast BVH Construction on GPUs
Fast BVH Construction on GPUs Published in EUROGRAGHICS, (2009) C. Lauterbach, M. Garland, S. Sengupta, D. Luebke, D. Manocha University of North Carolina at Chapel Hill NVIDIA University of California
A GPU-based Approximate SVD Algorithm Blake Foster, Sridhar Mahadevan, Rui Wang
A GPU-based Approximate SVD Algorithm Blake Foster, Sridhar Mahadevan, Rui Wang University of Massachusetts Amherst Introduction Singular Value Decomposition (SVD) A: m n matrix (m n) U, V: orthogonal
Parallel Computing. Slides credit: M. Quinn book (chapter 3 slides), A Grama book (chapter 3 slides)
Parallel Computing 2012 Slides credit: M. Quinn book (chapter 3 slides), A Grama book (chapter 3 slides) Parallel Algorithm Design Outline Computational Model Design Methodology Partitioning Communication
Parallel FFT Program Optimizations on Heterogeneous Computers
Parallel FFT Program Optimizations on Heterogeneous Computers Shuo Chen, Xiaoming Li Department of Electrical and Computer Engineering University of Delaware, Newark, DE 19716 Outline Part I: A Hybrid
GPU-Accelerated Parallel Sparse LU Factorization Method for Fast Circuit Analysis
GPU-Accelerated Parallel Sparse LU Factorization Method for Fast Circuit Analysis Abstract: Lower upper (LU) factorization for sparse matrices is the most important computing step for circuit simulation
Complexity and Advanced Algorithms. Introduction to Parallel Algorithms
Complexity and Advanced Algorithms Introduction to Parallel Algorithms Why Parallel Computing? Save time, resources, memory,... Who is using it? Academia Industry Government Individuals? Two practical
Experiences with the Sparse Matrix-Vector Multiplication on a Many-core Processor
Experiences with the Sparse Matrix-Vector Multiplication on a Many-core Processor Juan C. Pichel Centro de Investigación en Tecnoloxías da Información (CITIUS) Universidade de Santiago de Compostela, Spain
Lecture 15: More Iterative Ideas
Lecture 15: More Iterative Ideas David Bindel 15 Mar 2010 Logistics HW 2 due! Some notes on HW 2. Where we are / where we re going More iterative ideas. Intro to HW 3. More HW 2 notes See solution code!
The Fast Multipole Method on NVIDIA GPUs and Multicore Processors
The Fast Multipole Method on NVIDIA GPUs and Multicore Processors Toru Takahashi, a Cris Cecka, b Eric Darve c a b c Department of Mechanical Science and Engineering, Nagoya University Institute for Applied
Accelerated ANSYS Fluent: Algebraic Multigrid on a GPU. Robert Strzodka NVAMG Project Lead
Accelerated ANSYS Fluent: Algebraic Multigrid on a GPU Robert Strzodka NVAMG Project Lead A Parallel Success Story in Five Steps 2 Step 1: Understand Application ANSYS Fluent Computational Fluid Dynamics
AMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 20: Sparse Linear Systems; Direct Methods vs. Iterative Methods Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 26
B. Tech. Project Second Stage Report on
B. Tech. Project Second Stage Report on GPU Based Active Contours Submitted by Sumit Shekhar (05007028) Under the guidance of Prof Subhasis Chaudhuri Table of Contents 1. Introduction... 1 1.1 Graphic
Challenges in large-scale graph processing on HPC platforms and the Graph500 benchmark. by Nkemdirim Dockery
Challenges in large-scale graph processing on HPC platforms and the Graph500 benchmark by Nkemdirim Dockery High Performance Computing Workloads Core-memory sized Floating point intensive Well-structured
Motivation for Parallelism. Motivation for Parallelism. ILP Example: Loop Unrolling. Types of Parallelism
Motivation for Parallelism Motivation for Parallelism The speed of an application is determined by more than just processor speed. speed Disk speed Network speed... Multiprocessors typically improve the
On the Comparative Performance of Parallel Algorithms on Small GPU/CUDA Clusters
1 On the Comparative Performance of Parallel Algorithms on Small GPU/CUDA Clusters N. P. Karunadasa & D. N. Ranasinghe University of Colombo School of Computing, Sri Lanka nishantha@opensource.lk, dnr@ucsc.cmb.ac.lk
Lecture 2. Memory locality optimizations Address space organization
Lecture 2 Memory locality optimizations Address space organization Announcements Office hours in EBU3B Room 3244 Mondays 3.00 to 4.00pm; Thurs 2:00pm-3:30pm Partners XSED Portal accounts Log in to Lilliput
Basic Communication Operations (Chapter 4)
Basic Communication Operations (Chapter 4) Vivek Sarkar Department of Computer Science Rice University vsarkar@cs.rice.edu COMP 422 Lecture 17 13 March 2008 Review of Midterm Exam Outline MPI Example Program:
Architecture-Conscious Database Systems
Architecture-Conscious Database Systems 2009 VLDB Summer School Shanghai Peter Boncz (CWI) Sources Thank You! l l l l Database Architectures for New Hardware VLDB 2004 tutorial, Anastassia Ailamaki Query
CS 475: Parallel Programming Introduction
CS 475: Parallel Programming Introduction Wim Bohm, Sanjay Rajopadhye Colorado State University Fall 2014 Course Organization n Let s make a tour of the course website. n Main pages Home, front page. Syllabus.
Solving Sparse Linear Systems. Forward and backward substitution for solving lower or upper triangular systems
AMSC 6 /CMSC 76 Advanced Linear Numerical Analysis Fall 7 Direct Solution of Sparse Linear Systems and Eigenproblems Dianne P. O Leary c 7 Solving Sparse Linear Systems Assumed background: Gauss elimination
Sequoia. Mattan Erez. The University of Texas at Austin
Sequoia Mattan Erez The University of Texas at Austin EE382N: Parallelism and Locality, Fall 2015 1 2 Emerging Themes Writing high-performance code amounts to Intelligently structuring algorithms [compiler
Report Seminar Algorithm Engineering
Report Seminar Algorithm Engineering G. S. Brodal, R. Fagerberg, K. Vinther: Engineering a Cache-Oblivious Sorting Algorithm Iftikhar Ahmad Chair of Algorithm and Complexity Department of Computer Science
Space Filling Curves and Hierarchical Basis. Klaus Speer
Space Filling Curves and Hierarchical Basis Klaus Speer Abstract Real world phenomena can be best described using differential equations. After linearisation we have to deal with huge linear systems of
Agenda Cache memory organization and operation Chapter 6 Performance impact of caches Cache Memories
Agenda Chapter 6 Cache Memories Cache memory organization and operation Performance impact of caches The memory mountain Rearranging loops to improve spatial locality Using blocking to improve temporal
Parallel Systems. Project topics
Parallel Systems Project topics 2016-2017 1. Scheduling Scheduling is a common problem which however is NP-complete, so that we are never sure about the optimality of the solution. Parallelisation is a
Multicore computer: Combines two or more processors (cores) on a single die. Also called a chip-multiprocessor.
CS 320 Ch. 18 Multicore Computers Multicore computer: Combines two or more processors (cores) on a single die. Also called a chip-multiprocessor. Definitions: Hyper-threading Intel's proprietary simultaneous
CUDA Memory Types All material not from online sources/textbook copyright Travis Desell, 2012
CUDA Memory Types All material not from online sources/textbook copyright Travis Desell, 2012 Overview 1. Memory Access Efficiency 2. CUDA Memory Types 3. Reducing Global Memory Traffic 4. Example: Matrix-Matrix
Graph Partitioning for High-Performance Scientific Simulations. Advanced Topics Spring 2008 Prof. Robert van Engelen
Graph Partitioning for High-Performance Scientific Simulations Advanced Topics Spring 2008 Prof. Robert van Engelen Overview Challenges for irregular meshes Modeling mesh-based computations as graphs Static
On the efficiency of the Accelerated Processing Unit for scientific computing
24 th High Performance Computing Symposium Pasadena, April 5 th 2016 On the efficiency of the Accelerated Processing Unit for scientific computing I. Said, P. Fortin, J.-L. Lamotte, R. Dolbeau, H. Calandra
Prof. Dr. Stefan Funken, Prof. Dr. Alexander Keller, Prof. Dr. Karsten Urban 11. Januar Scientific Computing Parallele Algorithmen
Prof. Dr. Stefan Funken, Prof. Dr. Alexander Keller, Prof. Dr. Karsten Urban 11. Januar 2007 Scientific Computing Parallele Algorithmen Page 2 Scientific Computing 11. Januar 2007 Funken / Keller / Urban
Algorithms for GIS:! Quadtrees
Algorithms for GIS: Quadtrees Quadtree A data structure that corresponds to a hierarchical subdivision of the plane Start with a square (containing inside input data) Divide into 4 equal squares (quadrants)
Datenbanksysteme II: Modern Hardware. Stefan Sprenger November 23, 2016
Datenbanksysteme II: Modern Hardware Stefan Sprenger November 23, 2016 Content of this Lecture Introduction to Modern Hardware CPUs, Cache Hierarchy Branch Prediction SIMD NUMA Cache-Sensitive Skip List
Optimizing Data Locality for Iterative Matrix Solvers on CUDA
Optimizing Data Locality for Iterative Matrix Solvers on CUDA Raymond Flagg, Jason Monk, Yifeng Zhu PhD., Bruce Segee PhD. Department of Electrical and Computer Engineering, University of Maine, Orono,
Today. Cache Memories. General Cache Concept. General Cache Organization (S, E, B) Cache Memories. Example Memory Hierarchy Smaller, faster,
Today Cache Memories CSci 2021: Machine Architecture and Organization November 7th-9th, 2016 Your instructor: Stephen McCamant Cache memory organization and operation Performance impact of caches The memory
WHY PARALLEL PROCESSING? (CE-401)
PARALLEL PROCESSING (CE-401) COURSE INFORMATION 2 + 1 credits (60 marks theory, 40 marks lab) Labs introduced for second time in PP history of SSUET Theory marks breakup: Midterm Exam: 15 marks Assignment:
Distributed-memory Algorithms for Dense Matrices, Vectors, and Arrays
Distributed-memory Algorithms for Dense Matrices, Vectors, and Arrays John Mellor-Crummey Department of Computer Science Rice University johnmc@rice.edu COMP 422/534 Lecture 19 25 October 2018 Topics for
High Performance Computing for PDE Towards Petascale Computing
High Performance Computing for PDE Towards Petascale Computing S. Turek, D. Göddeke with support by: Chr. Becker, S. Buijssen, M. Grajewski, H. Wobker Institut für Angewandte Mathematik, Univ. Dortmund
Recursive inverse factorization
Recursive inverse factorization Anton Artemov Division of Scientific Computing, Uppsala University anton.artemov@it.uu.se 06.09.2017 Context of work Research group on large scale electronic structure computations
Efficient Finite Element Geometric Multigrid Solvers for Unstructured Grids on GPUs
Efficient Finite Element Geometric Multigrid Solvers for Unstructured Grids on GPUs Markus Geveler, Dirk Ribbrock, Dominik Göddeke, Peter Zajac, Stefan Turek Institut für Angewandte Mathematik TU Dortmund,
Portability and Scalability of Sparse Tensor Decompositions on CPU/MIC/GPU Architectures
Photos placed in horizontal position with even amount of white space between photos and header Portability and Scalability of Sparse Tensor Decompositions on CPU/MIC/GPU Architectures Christopher Forster,
Warped parallel nearest neighbor searches using kd-trees
Warped parallel nearest neighbor searches using kd-trees Roman Sokolov, Andrei Tchouprakov D4D Technologies Kd-trees Binary space partitioning tree Used for nearest-neighbor search, range search Application:
Today Cache memory organization and operation Performance impact of caches
Cache Memories 1 Today Cache memory organization and operation Performance impact of caches The memory mountain Rearranging loops to improve spatial locality Using blocking to improve temporal locality
Lecture 3: Intro to parallel machines and models
Lecture 3: Intro to parallel machines and models David Bindel 1 Sep 2011 Logistics Remember: http://www.cs.cornell.edu/~bindel/class/cs5220-f11/ http://www.piazza.com/cornell/cs5220 Note: the entire class
Fast Multipole and Related Algorithms
Fast Multipole and Related Algorithms Ramani Duraiswami University of Maryland, College Park http://www.umiacs.umd.edu/~ramani Joint work with Nail A. Gumerov Efficiency by exploiting symmetry and A general
Benchmarking CPU Performance. Benchmarking CPU Performance
Cluster Computing Benchmarking CPU Performance Many benchmarks available MHz (cycle speed of processor) MIPS (million instructions per second) Peak FLOPS Whetstone Stresses unoptimized scalar performance,
Memory Systems IRAM. Principle of IRAM
Memory Systems 165 other devices of the module will be in the Standby state (which is the primary state of all RDRAM devices) or another state with low-power consumption. The RDRAM devices provide several
Parallel Implementations of Gaussian Elimination
s of Western Michigan University vasilije.perovic@wmich.edu January 27, 2012 CS 6260: in Parallel Linear systems of equations General form of a linear system of equations is given by a 11 x 1 + + a 1n
Cache memories are small, fast SRAM based memories managed automatically in hardware.
Cache Memories Cache memories are small, fast SRAM based memories managed automatically in hardware. Hold frequently accessed blocks of main memory CPU looks first for data in caches (e.g., L1, L2, and
Solving Dense Linear Systems on Graphics Processors
Solving Dense Linear Systems on Graphics Processors Sergio Barrachina Maribel Castillo Francisco Igual Rafael Mayo Enrique S. Quintana-Ortí High Performance Computing & Architectures Group Universidad
211: Computer Architecture Summer 2016
211: Computer Architecture Summer 2016 Liu Liu Topic: Assembly Programming Storage - Assembly Programming: Recap - Call-chain - Factorial - Storage: - RAM - Caching - Direct - Mapping Rutgers University
Lecture 25 of 41. Spatial Sorting: Binary Space Partitioning Quadtrees & Octrees
Spatial Sorting: Binary Space Partitioning Quadtrees & Octrees William H. Hsu Department of Computing and Information Sciences, KSU KSOL course pages: http://bit.ly/hgvxlh / http://bit.ly/evizre Public
Scientific Applications. Chao Sun
Large Scale Multiprocessors And Scientific Applications Zhou Li Chao Sun Contents Introduction Interprocessor Communication: The Critical Performance Issue Characteristics of Scientific Applications Synchronization:
Introduction: Modern computer architecture. The stored program computer and its inherent bottlenecks Multi- and manycore chips and nodes
Introduction: Modern computer architecture The stored program computer and its inherent bottlenecks Multi- and manycore chips and nodes Motivation: Multi-Cores where and why Introduction: Moore s law Intel
Matrix Multiplication
Matrix Multiplication CPS343 Parallel and High Performance Computing Spring 2013 CPS343 (Parallel and HPC) Matrix Multiplication Spring 2013 1 / 32 Outline 1 Matrix operations Importance Dense and sparse
Scheduling FFT Computation on SMP and Multicore Systems Ayaz Ali, Lennart Johnsson & Jaspal Subhlok
Scheduling FFT Computation on SMP and Multicore Systems Ayaz Ali, Lennart Johnsson & Jaspal Subhlok Texas Learning and Computation Center Department of Computer Science University of Houston Outline Motivation
Multi-GPU Scaling of Direct Sparse Linear System Solver for Finite-Difference Frequency-Domain Photonic Simulation
Multi-GPU Scaling of Direct Sparse Linear System Solver for Finite-Difference Frequency-Domain Photonic Simulation 1 Cheng-Han Du* I-Hsin Chung** Weichung Wang* * I n s t i t u t e o f A p p l i e d M
Top500 Supercomputer list
Top500 Supercomputer list Tends to represent parallel computers, so distributed systems such as SETI@Home are neglected. Does not consider storage or I/O issues Both custom designed machines and commodity
Matrix Multiplication
Matrix Multiplication CPS343 Parallel and High Performance Computing Spring 2018 CPS343 (Parallel and HPC) Matrix Multiplication Spring 2018 1 / 32 Outline 1 Matrix operations Importance Dense and sparse
How HPC Hardware and Software are Evolving Towards Exascale
How HPC Hardware and Software are Evolving Towards Exascale Kathy Yelick Associate Laboratory Director and NERSC Director Lawrence Berkeley National Laboratory EECS Professor, UC Berkeley NERSC Overview
Optimizing Molecular Dynamics
Optimizing Molecular Dynamics This chapter discusses performance tuning of parallel and distributed molecular dynamics (MD) simulations, which involves both: (1) intranode optimization within each node
THE AUSTRALIAN NATIONAL UNIVERSITY First Semester Examination June COMP3320/6464/HONS High Performance Scientific Computing
THE AUSTRALIAN NATIONAL UNIVERSITY First Semester Examination June 2014 COMP3320/6464/HONS High Performance Scientific Computing Study Period: 15 minutes Time Allowed: 3 hours Permitted Materials: Non-Programmable
Chapter 18 - Multicore Computers
Chapter 18 - Multicore Computers Luis Tarrataca luis.tarrataca@gmail.com CEFET-RJ Luis Tarrataca Chapter 18 - Multicore Computers 1 / 28 Table of Contents I 1 2 Where to focus your study Luis Tarrataca
10th August Part One: Introduction to Parallel Computing
Part One: Introduction to Parallel Computing 10th August 2007 Part 1 - Contents Reasons for parallel computing Goals and limitations Criteria for High Performance Computing Overview of parallel computer
(Sparse) Linear Solvers
(Sparse) Linear Solvers Ax = B Why? Many geometry processing applications boil down to: solve one or more linear systems Parameterization Editing Reconstruction Fairing Morphing 2 Don t you just invert
SCALING DGEMM TO MULTIPLE CAYMAN GPUS AND INTERLAGOS MANY-CORE CPUS FOR HPL
SCALING DGEMM TO MULTIPLE CAYMAN GPUS AND INTERLAGOS MANY-CORE CPUS FOR HPL Matthias Bach and David Rohr Frankfurt Institute for Advanced Studies Goethe University of Frankfurt I: INTRODUCTION 3 Scaling
Overview. CS 472 Concurrent & Parallel Programming University of Evansville
Overview CS 472 Concurrent & Parallel Programming University of Evansville Selection of slides from CIS 410/510 Introduction to Parallel Computing Department of Computer and Information Science, University
Memory Hierarchies && The New Bottleneck == Cache Conscious Data Access. Martin Grund
Memory Hierarchies && The New Bottleneck == Cache Conscious Data Access Martin Grund Agenda Key Question: What is the memory hierarchy and how to exploit it? What to take home How is computer memory organized.
Memory Bound Computing
Memory Bound Computing Francesc Alted Freelance Consultant & Trainer http://www.blosc.org/professional-services.html Advanced Scientific Programming in Python Reading, UK September, 2016 Goals Recognize
Algorithms and Architecture. William D. Gropp Mathematics and Computer Science
Algorithms and Architecture William D. Gropp Mathematics and Computer Science www.mcs.anl.gov/~gropp Algorithms What is an algorithm? A set of instructions to perform a task How do we evaluate an algorithm?
COMPUTATIONAL LINEAR ALGEBRA
COMPUTATIONAL LINEAR ALGEBRA Matrix Vector Multiplication Matrix matrix Multiplication Slides from UCSD and USB Directed Acyclic Graph Approach Jack Dongarra A new approach using Strassen`s algorithm Jim
Autotuning and Specialization: Speeding up Matrix Multiply for Small Matrices with Compiler Technology
Autotuning and Specialization: Speeding up Matrix Multiply for Small Matrices with Compiler Technology Jaewook Shin 1, Mary W. Hall 2, Jacqueline Chame 3, Chun Chen 2, Paul D. Hovland 1 1 ANL/MCS 2 University
Effect of memory latency
CACHE AWARENESS Effect of memory latency Consider a processor operating at 1 GHz (1 ns clock) connected to a DRAM with a latency of 100 ns. Assume that the processor has two ALU units and it is capable
COMPUTING ELEMENT EVOLUTION AND ITS IMPACT ON SIMULATION CODES
COMPUTING ELEMENT EVOLUTION AND ITS IMPACT ON SIMULATION CODES P(ND) 2-2 2014 Guillaume Colin de Verdière OCTOBER 14TH, 2014 P(ND)^2-2 PAGE 1 CEA, DAM, DIF, F-91297 Arpajon, France October 14th, 2014 Abstract:
TOOLS FOR IMPROVING CROSS-PLATFORM SOFTWARE DEVELOPMENT
TOOLS FOR IMPROVING CROSS-PLATFORM SOFTWARE DEVELOPMENT Eric Kelmelis 28 March 2018 OVERVIEW BACKGROUND Evolution of processing hardware CROSS-PLATFORM KERNEL DEVELOPMENT Write once, target multiple hardware
Cache-oblivious Programming
Cache-oblivious Programming Story so far We have studied cache optimizations for array programs Main transformations: loop interchange, loop tiling Loop tiling converts matrix computations into block matrix
Evaluation of sparse LU factorization and triangular solution on multicore architectures. X. Sherry Li
Evaluation of sparse LU factorization and triangular solution on multicore architectures X. Sherry Li Lawrence Berkeley National Laboratory ParLab, April 29, 28 Acknowledgement: John Shalf, LBNL Rich Vuduc,