CUDA programming model. N. Cardoso & P. Bicudo. Física Computacional (FC5)
|
|
- Roxanne Preston
- 5 years ago
- Views:
Transcription
1 CUDA programming model N. Cardoso & P. Bicudo Física Computacional (FC5) N. Cardoso & P. Bicudo CUDA programming model 1/23
2 Outline 1 CUDA qualifiers 2 CUDA Kernel Thread hierarchy Kernel, configuration and execution Exercise: Calculate the number of threads per block 3 Warps and Warp Scheduling 4 Synchronization N. Cardoso & P. Bicudo CUDA programming model 2/23
3 CUDA qualifiers We can write in the same file the Host code and the Device code The Host controls the Device! How to declare functions in CUDA that: run only in the Device and are called by the Host? run in Device and are called only by the Device? run only in the Host? N. Cardoso & P. Bicudo CUDA programming model 3/23
4 CUDA CUDA qualifiers global defines a kernel function must return void defines the code to be executed by all threads during the parallel execution call device defines a function that can be called inside the kernel or by other device function device and host can be used together, the compiler creates the code for both host and device When no qualifier is used then that function is a host function Executed in: Only called by: device void/float/int/... DeviceFunc() device device global void KernelFunc() device host* host void/float/int/... HostFunc() host host in Kepler and newer architectures, global functions can be called inside another global function, Dynamic Parallelism. N. Cardoso & P. Bicudo CUDA programming model 4/23
5 CUDA Kernel A kernel is a function executed on the GPU as an array of threads in parallel All threads execute the same code, can take different paths Each thread has an ID Select input/output data Control decisions Threads are grouped into blocks Blocks are grouped into a grid ernel Execution A kernel is executed as a grid of blocks of threads CUDA thread CUDA thread block CUDA kernel grid... CUDA core CUDA Streaming Multiprocessor CUDA-enabled GPU Each thread is executed by a core Each thread is executed by a core Each block is executed by one SM and does not migrate Each block is executed by one SM and does not migrate Several concurrent blocks can reside on one SM depending on the blocks memory requirements and the SM s memory resources Several concurrent blocks can reside on one SM depending on the blocks memory requirements and the SM s memory resources Each kernel is executed on one device Each kernel is executed on one device Multiple kernels can execute on a device at one time Multiple kernels can execute on a device at one time N. Cardoso & P. Bicudo CUDA programming model 5/23
6 CUDA Kernel Thread hierarchy code executed in the GPU (device); executed by a parallel array of threads; all threads run the same code hierarchical launches: threads are grouped in blocks; blocks are grouped in grids. CUDA has identification variables for threads, blocks and grids. Threads are logically grouped into thread blocks: Threads in the same block can cooperate cooperatively load/store blocks of memory that all will use through shared memory share results with each other or cooperate to produce a single result synchronize with each other ( syncthreads()) Threads in different blocks cannot cooperate Host Kernel 1 Device Grid 1 Grid 2 Kernel 2 Block (1, 1) (0,0,1) (1,0,1) (2,0,1) (3,0,1) Nuno Cardoso Block (0, 0) Block (0, 1) Block (1, 0) Block (1, 1) Thread Thread Thread Thread (0,0,0) (1,0,0) (2,0,0) (3,0,0) Thread Thread Thread Thread (0,1,0) (1,1,0) (2,1,0) (3,1,0) Courtesy: N. Cardoso & P. Bicudo CUDA programming model 6/23
7 CUDA Kernel Thread hierarchy IDs and dimensions: threads can have 3D IDs, unique for a single block (threadidx.x, threadidx.y, threadidx.z); maximum number of threads per block are architecture dependent. blocks can have 2D (for Fermi and below) or 3D (for Kepler and newer architectures) IDs, unique in the same grid (blockidx.x, blockidx.y, blockidx.z); Includes also the block dimensions, (blockdim.x, blockdim.y, blockdim.z) and grid dimensions (griddim.x, griddim.y, griddim.z). all dimensions are defined by the user/programmer in the kernel call; The list of these specifications supported by your GPU are available by running the./devicequery in the sample code directory Blocks can execute in any order, concurrently or sequentially This independence between blocks gives scalability: a kernel scales across any number of SMs Host Kernel 1 Device Grid 1 Grid 2 Kernel 2 Block (1, 1) (0,0,1) (1,0,1) (2,0,1) (3,0,1) Nuno Cardoso Block (0, 0) Block (0, 1) Block (1, 0) Block (1, 1) Thread Thread Thread Thread (0,0,0) (1,0,0) (2,0,0) (3,0,0) Thread Thread Thread Thread (0,1,0) (1,1,0) (2,1,0) (3,1,0) N. Cardoso & P. Bicudo CUDA programming model 7/23
8 CUDA Kernel Using 1D blocks and grids: global v o i d k e r n e l ( ) { i n t i d = b l o c k I d x. x blockdim. x + t h r e a d I d x. x ; (... ) } Using 2D blocks and grids: global v o i d k e r n e l ( ) { i n t i = b l o c k I d x. x blockdim. x + t h r e a d I d x. x ; i n t j = b l o c k I d x. y blockdim. y + t h r e a d I d x. y ; } i n t i d = i + j Nx ; (... ) Important: Start using 1D arrays from now on! a 2D array can be accessed as: i + j * Nx a 3D array can be accessed as: i + j * Nx + k * Nx * Ny pay attention to the faster and the slower indexes otherwise you lose performance in the previous example, i is the faster and k the slower! N. Cardoso & P. Bicudo CUDA programming model 8/23
9 CUDA Kernel Kernel, configuration and execution a kernel function can be called in the following way: dim3 DimGrid(100, 50); // 5000 thread blocks dim3 DimBlock(4, 8, 8); // 256 threads per block kernel<<< DimGrid, DimBlock >>>(...); dim3 can take 1, 2 or 3 parameters N. Cardoso & P. Bicudo CUDA programming model 9/23
10 CUDA Kernel Kernel, configuration and execution Calculating Y = ax + Y in parallel: global void kernel(int n, float a, float *x, float *y){ int i = blockidx.x * blockdim.x + threadidx.x; if(i<n) y[i] = a * x[i] + y[i]; } //call kernel function with 256 threads per block int nblocks = (n+255) / 256; //n is the array size kernel<<<nblocks,256>>> (n, 2.0, X, Y); N. Cardoso & P. Bicudo CUDA programming model 10/23
11 CUDA Kernel Execution configuration identifier in 1D: Thread device inline Blocks: float func(float Scalable x){ Cooperation (...) } global void kernel(float *input, float *output){ int threadid = blockidx.x * blockdim.x + threadidx.x; Threads float x = within input[threadid] a block cooperate via shared memory, atomic float y operations = func(x); and barrier synchronization output[threadid] = y; } Threads in different blocks cannot cooperate //call kernel function with 8 threads per block //problem size: N*8 = total number of threads kernel<<<n,8>>> (input, output); Divide monolithic thread array into multiple blocks threadid Thread Block 0 Thread Block 1 Thread Block N float x = input[threadid]; float y = func(x); output[threadid] = y; float x = input[threadid]; float y = func(x); output[threadid] = y; float x = input[threadid]; float y = func(x); output[threadid] = y; David Kirk/NVIDIA and Wen-mei W. Hwu, N. Cardoso & P. ECE Bicudo 498AL, University of Illinois, Urbana-Champaign CUDA programming model 11/23
12 What is the maximum number of threads per block? How many blocks can I run in a single SM? How is a thread block executed in a single SM? N. Cardoso & P. Bicudo CUDA programming model 12/23
13 32 CUDA Cores per SM (512 total) 8x peak FP64 performance 50% of peakad communication Fermi Architecture Example: 32 CUDA Cores per SM 32 fp32 ops/clock 16 fp64 ops/clock 32 int32 ops/clock 2 warp schedulers Up to 1536 threads concurrently Up to 1024 threads per block 4 special-function units 64KB shared mem + L1 cache 32K 32-bit registers Instruction Cache Scheduler Scheduler Dispatch Dispatch Register File Load/Store Units x 16 Special Func Units x 4 Interconnect Network 64K Configurable Cache/Shared Mem Uniform Cache N. Cardoso & P. Bicudo CUDA programming model 13/23
14 SM Multiprocessor SM, Streaming Multiprocessor GPU serves as a coprocessor to the CPU 32 CUDA Cores per SM (512 total) has its own device memory on the card SIMT (Single Instruction Multiple Thread) execution 8x peak FP64 performance Many threads execute 50% concurrently of peakad communication Same instruction Different data elements Hardware automatically handles divergence Threads run in groups of 32 (warp size) called warps Each warp is executed in a SIMD fashion (i.e. all threads within a warp must execute the same instruction at any given time). Problem: branch divergence Hardware multithreading Hardware resource allocation & thread scheduling Excess of threads to hide latency Any thread not waiting for something can run Context switching is (basically) free Hardware relies on threads to hide latency Instruction Cache Scheduler Scheduler Dispatch Dispatch Register File Load/Store Units x 16 Special Func Units x 4 Interconnect Network 64K Configurable Cache/Shared Mem Uniform Cache N. Cardoso & P. Bicudo CUDA programming model 14/23
15 Maximum number of blocks per SM: architecture dependent Fermi: 8 Kepler: 16 Maxwell: 32 Maximum number of threads per SM: architecture dependent Fermi: 1536 Kepler: 2048 Maxwell: 2048 Maximum number of threads per block: 1024 Block size must be multiple of the warp size (32) The warp size is currently 32 threads The warp size can change in future GPUs There isn t a best universal block size Needs to be tuned manually for each kernel Depends a lot of the resources used by each thread Start with 128 or 256 threads per block and tune in the end N. Cardoso & P. Bicudo CUDA programming model 15/23
16 Exercise: Calculate the number of threads per block Goal: Get the maximum number of threads in a SM. Limitations maximum number of threads per block: 512 maximum number of threads per SM: 1024 maximum number of blocks per SM: 8 block size: threads per block 8 64 = 512 threads 50% occupation of the SM block size: threads per block 1024/256 = 4 4 blocks per SM. block size: threads per block > 512 exceeds the SM capacity. Example: Tiled Matrix Multiplication d_p blockidx.y threadidx.y blockidx.x threadidx.x 19 blockdim.x blockdim.y assigned to a thread assigned to a thread block e : š A Å Ã A@ ž B ³ Goal: To have as many threads on an SM as possible ³ Limitations: Maximum Number of Threads per Block: 512 N. Cardoso & P. Bicudo CUDA programming Maximum Number of Threads per SM: 1024 Æ model 16/23 Maximum Number of Block per SM: 8
17 Warps and Warp Scheduling Each Thread Block divided in 32-thread "Warps" This is an implementation decision, not part of the CUDA programming model Warps are the basic scheduling unit in SM The SM implements a zero-overhead warp scheduling Warps whose next instruction has its operands ready for consumption are eligible for execution Eligible warps are selected for execution on a prioritized scheduling policy All threads in a warp execute the same instruction Prefer thread block sizes that result in mostly full warps Bad: kernel <<<N, 1>>> (... ) Okay: kernel <<<N / 32, 32>>>(... ) Better: kernel <<<N / 128, 128>>>(... ) Prefer to have enough threads per block to provide hardware with many warps to switch between This is how the GPU hides memory access latency N. Cardoso & P. Bicudo CUDA programming model 17/23
18 Warps and Warp Scheduling Exercise: If 3 blocks are processed by an SM and each Block has 256 threads, how many Warps are managed by the SM? Each Block is divided into 256/32 = 8 Warps There are 8 * 3 = 24 Warps At any point in time, only one of the 24 Warps will be selected for instruction fetch and execution. N. Cardoso & P. Bicudo CUDA programming model 18/23
19 Warps and Warp Scheduling anch Divergence in Warps Branch Divergence in Warps occurs when threads inside warps branches to different execution paths. rs when Example: threads e warps branches ifferent execution s. Branch Path A Path B 50% performance loss 50% performance loss 18 N. Cardoso & P. Bicudo CUDA programming model 19/23
20 Warps and Warp Scheduling Scoreboarding The SM schedules threads in groups of 32 parallel threads called warps. Each SM fea warp schedulers and two instruction dispatch units, allowing two warps to be issued a executed concurrently. Fermi s dual warp scheduler selects two warps, and issues on instruction from each warp to a group of sixteen cores, sixteen load/store units, or fou Because warps execute independently, Fermi s scheduler does not need to check for dependencies from within the instruction stream. Using this elegant model of dual-issu All register operands of all instructions in Fermi: 2 Warp schedulers (per SM) achieves near peak hardware performance. the Instruction Buffer are scoreboarded Instruction becomes ready after the needed values are deposited prevents hazards cleared Warp instructions Scheduling are eligible for issue Decoupled Memory/Processor pipelines À any thread Thread can warps continue are to issue scheduled to hide memory instructions access until delays scoreboarding prevents issueá Each warp runs until it is stalled due to memory access allowá Memory/Processor ops to proceed in shadow Upon of other a warp waiting stall, another warp is scheduled with near Memory/Processor zero overhead ops Á Number of The Warp original schedulers warp is will architecture be ready to be schedule once the memory content it asked for is ready dependent! Most instructions can be dual issued; two integer instructions, two floating instruction mix of integer, floating point, load, store, and SFU instructions can be issued concurre Double precision instructions do not support dual dispatch with any other operation. 64 KB Configurable Shared Memory and L1 Cache One of the key architectural innovations that greatly improved both the programmabili performance of GPU applications is on-chip shared memory. Shared memory enables within the same thread block to cooperate, facilitates extensive reuse of on-chip data, greatly reduces off-chip traffic. Shared memory is a key enabler for many high-perform CUDA applications. \ G80 and GT200 have 16 KB of shared memory per SM. In the Fermi architecture, each N. Cardoso & P. Bicudo 64 KB CUDA of on-chip programming memory model that can be configured as 48 KB of Shared memory with 20/2316
21 Thread Synchronization (within a block only) CUDA allows thread synchronization within the same block using v o i d syncthreads ( ) ; waits until all threads in the thread block have reached this point and all global and shared memory accesses made by these threads prior to syncthreads() are visible to all threads in the block. is used to coordinate communication between the threads of the same block. is allowed in conditional code but only if the conditional evaluates identically across the entire thread block, otherwise the code execution is likely to hang or produce unintended side effects. Devices of compute capability 2.x and higher support three variations of syncthreads(): int syncthreads_count(int predicate); is identical to syncthreads() with the additional feature that it evaluates predicate for all threads of the block and returns the number of threads for which predicate evaluates to non-zero. int syncthreads_and(int predicate); is identical to syncthreads() with the additional feature that it evaluates predicate for all threads of the block and returns non-zero if and only if predicate evaluates to non-zero for all of them. int syncthreads_or(int predicate); is identical to syncthreads() with the additional feature that it evaluates predicate for all threads of the block and returns non-zero if and only if predicate evaluates to non-zero for any of them. N. Cardoso & P. Bicudo CUDA programming model 21/23
22 Synchronization in the Host All kernel calls are asynchronous with the host: the control returns to the host (CPU) immediately after the kernel launch, the kernel is executed when all previous synchronous calls have ended and there are enough resources for it. synchronization barrier using cudadevicesynchronize() cudaerror_t cudadevicesynchronize(void) Blocks until the device has completed all preceding requested tasks. The only safe way to synchronize threads in different blocks is to terminate the kernel and start a new kernel for the activities after the synchronization point N. Cardoso & P. Bicudo CUDA programming model 22/23
23 Conclusion We must be aware of the restrictions imposed by hardware: threads/sm blocks/sm threads/blocks threads/warps The only safe way to synchronize threads in different blocks is to terminate the kernel and start a new kernel for the activities after the synchronization point The warp size is the most important number in CUDA Try to avoid or minimize branch divergence inside warps N. Cardoso & P. Bicudo CUDA programming model 23/23
Module 3: CUDA Execution Model -I. Objective
ECE 8823A GPU Architectures odule 3: CUDA Execution odel -I 1 Objective A more detailed look at kernel execution Data to thread assignment To understand the organization and scheduling of threads Resource
More informationCUDA PROGRAMMING MODEL. Carlo Nardone Sr. Solution Architect, NVIDIA EMEA
CUDA PROGRAMMING MODEL Carlo Nardone Sr. Solution Architect, NVIDIA EMEA CUDA: COMMON UNIFIED DEVICE ARCHITECTURE Parallel computing architecture and programming model GPU Computing Application Includes
More informationTesla Architecture, CUDA and Optimization Strategies
Tesla Architecture, CUDA and Optimization Strategies Lan Shi, Li Yi & Liyuan Zhang Hauptseminar: Multicore Architectures and Programming Page 1 Outline Tesla Architecture & CUDA CUDA Programming Optimization
More informationData Parallel Execution Model
CS/EE 217 GPU Architecture and Parallel Programming Lecture 3: Kernel-Based Data Parallel Execution Model David Kirk/NVIDIA and Wen-mei Hwu, 2007-2013 Objective To understand the organization and scheduling
More informationHPC COMPUTING WITH CUDA AND TESLA HARDWARE. Timothy Lanfear, NVIDIA
HPC COMPUTING WITH CUDA AND TESLA HARDWARE Timothy Lanfear, NVIDIA WHAT IS GPU COMPUTING? What is GPU Computing? x86 PCIe bus GPU Computing with CPU + GPU Heterogeneous Computing Low Latency or High Throughput?
More informationThreading Hardware in G80
ing Hardware in G80 1 Sources Slides by ECE 498 AL : Programming Massively Parallel Processors : Wen-Mei Hwu John Nickolls, NVIDIA 2 3D 3D API: API: OpenGL OpenGL or or Direct3D Direct3D GPU Command &
More informationLecture 3: Introduction to CUDA
CSCI-GA.3033-004 Graphics Processing Units (GPUs): Architecture and Programming Lecture 3: Introduction to CUDA Some slides here are adopted from: NVIDIA teaching kit Mohamed Zahran (aka Z) mzahran@cs.nyu.edu
More informationEE382N (20): Computer Architecture - Parallelism and Locality Spring 2015 Lecture 09 GPUs (II) Mattan Erez. The University of Texas at Austin
EE382 (20): Computer Architecture - ism and Locality Spring 2015 Lecture 09 GPUs (II) Mattan Erez The University of Texas at Austin 1 Recap 2 Streaming model 1. Use many slimmed down cores to run in parallel
More informationIntroduction to Parallel Computing with CUDA. Oswald Haan
Introduction to Parallel Computing with CUDA Oswald Haan ohaan@gwdg.de Schedule Introduction to Parallel Computing with CUDA Using CUDA CUDA Application Examples Using Multiple GPUs CUDA Application Libraries
More informationIntroduction to CUDA (1 of n*)
Agenda Introduction to CUDA (1 of n*) GPU architecture review CUDA First of two or three dedicated classes Joseph Kider University of Pennsylvania CIS 565 - Spring 2011 * Where n is 2 or 3 Acknowledgements
More informationModule 2: Introduction to CUDA C. Objective
ECE 8823A GPU Architectures Module 2: Introduction to CUDA C 1 Objective To understand the major elements of a CUDA program Introduce the basic constructs of the programming model Illustrate the preceding
More informationWhat is GPU? CS 590: High Performance Computing. GPU Architectures and CUDA Concepts/Terms
CS 590: High Performance Computing GPU Architectures and CUDA Concepts/Terms Fengguang Song Department of Computer & Information Science IUPUI What is GPU? Conventional GPUs are used to generate 2D, 3D
More informationEE382N (20): Computer Architecture - Parallelism and Locality Fall 2011 Lecture 18 GPUs (III)
EE382 (20): Computer Architecture - Parallelism and Locality Fall 2011 Lecture 18 GPUs (III) Mattan Erez The University of Texas at Austin EE382: Principles of Computer Architecture, Fall 2011 -- Lecture
More informationProgramming in CUDA. Malik M Khan
Programming in CUDA October 21, 2010 Malik M Khan Outline Reminder of CUDA Architecture Execution Model - Brief mention of control flow Heterogeneous Memory Hierarchy - Locality through data placement
More informationIntroduction to CUDA CME343 / ME May James Balfour [ NVIDIA Research
Introduction to CUDA CME343 / ME339 18 May 2011 James Balfour [ jbalfour@nvidia.com] NVIDIA Research CUDA Programing system for machines with GPUs Programming Language Compilers Runtime Environments Drivers
More informationLecture 2: CUDA Programming
CS 515 Programming Language and Compilers I Lecture 2: CUDA Programming Zheng (Eddy) Zhang Rutgers University Fall 2017, 9/12/2017 Review: Programming in CUDA Let s look at a sequential program in C first:
More informationJosef Pelikán, Jan Horáček CGG MFF UK Praha
GPGPU and CUDA 2012-2018 Josef Pelikán, Jan Horáček CGG MFF UK Praha pepca@cgg.mff.cuni.cz http://cgg.mff.cuni.cz/~pepca/ 1 / 41 Content advances in hardware multi-core vs. many-core general computing
More informationFundamental Optimizations in CUDA Peng Wang, Developer Technology, NVIDIA
Fundamental Optimizations in CUDA Peng Wang, Developer Technology, NVIDIA Optimization Overview GPU architecture Kernel optimization Memory optimization Latency optimization Instruction optimization CPU-GPU
More informationMattan Erez. The University of Texas at Austin
EE382V (17325): Principles in Computer Architecture Parallelism and Locality Fall 2007 Lecture 12 GPU Architecture (NVIDIA G80) Mattan Erez The University of Texas at Austin Outline 3D graphics recap and
More informationCUDA C Programming Mark Harris NVIDIA Corporation
CUDA C Programming Mark Harris NVIDIA Corporation Agenda Tesla GPU Computing CUDA Fermi What is GPU Computing? Introduction to Tesla CUDA Architecture Programming & Memory Models Programming Environment
More informationMaster Informatics Eng.
Advanced Architectures Master Informatics Eng. 2018/19 A.J.Proença Data Parallelism 3 (GPU/CUDA, Neural Nets,...) (most slides are borrowed) AJProença, Advanced Architectures, MiEI, UMinho, 2018/19 1 The
More informationParallel Computing. Lecture 19: CUDA - I
CSCI-UA.0480-003 Parallel Computing Lecture 19: CUDA - I Mohamed Zahran (aka Z) mzahran@cs.nyu.edu http://www.mzahran.com GPU w/ local DRAM (device) Behind CUDA CPU (host) Source: http://hothardware.com/reviews/intel-core-i5-and-i7-processors-and-p55-chipset/?page=4
More informationFundamental CUDA Optimization. NVIDIA Corporation
Fundamental CUDA Optimization NVIDIA Corporation Outline Fermi/Kepler Architecture Kernel optimizations Launch configuration Global memory throughput Shared memory access Instruction throughput / control
More informationIntroduction to CUDA Algoritmi e Calcolo Parallelo. Daniele Loiacono
Introduction to CUDA Algoritmi e Calcolo Parallelo References q This set of slides is mainly based on: " CUDA Technical Training, Dr. Antonino Tumeo, Pacific Northwest National Laboratory " Slide of Applied
More informationIntroduction to GPU programming. Introduction to GPU programming p. 1/17
Introduction to GPU programming Introduction to GPU programming p. 1/17 Introduction to GPU programming p. 2/17 Overview GPUs & computing Principles of CUDA programming One good reference: David B. Kirk
More informationGPU Programming. Lecture 2: CUDA C Basics. Miaoqing Huang University of Arkansas 1 / 34
1 / 34 GPU Programming Lecture 2: CUDA C Basics Miaoqing Huang University of Arkansas 2 / 34 Outline Evolvements of NVIDIA GPU CUDA Basic Detailed Steps Device Memories and Data Transfer Kernel Functions
More informationIntroduction to CUDA Algoritmi e Calcolo Parallelo. Daniele Loiacono
Introduction to CUDA Algoritmi e Calcolo Parallelo References This set of slides is mainly based on: CUDA Technical Training, Dr. Antonino Tumeo, Pacific Northwest National Laboratory Slide of Applied
More informationModule 2: Introduction to CUDA C
ECE 8823A GPU Architectures Module 2: Introduction to CUDA C 1 Objective To understand the major elements of a CUDA program Introduce the basic constructs of the programming model Illustrate the preceding
More informationLecture 2: Introduction to CUDA C
CS/EE 217 GPU Architecture and Programming Lecture 2: Introduction to CUDA C David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2013 1 CUDA /OpenCL Execution Model Integrated host+device app C program Serial or
More informationCUDA Programming Model
CUDA Xing Zeng, Dongyue Mou Introduction Example Pro & Contra Trend Introduction Example Pro & Contra Trend Introduction What is CUDA? - Compute Unified Device Architecture. - A powerful parallel programming
More informationCUDA Parallelism Model
GPU Teaching Kit Accelerated Computing CUDA Parallelism Model Kernel-Based SPMD Parallel Programming Multidimensional Kernel Configuration Color-to-Grayscale Image Processing Example Image Blur Example
More informationFundamental CUDA Optimization. NVIDIA Corporation
Fundamental CUDA Optimization NVIDIA Corporation Outline! Fermi Architecture! Kernel optimizations! Launch configuration! Global memory throughput! Shared memory access! Instruction throughput / control
More informationCOMP 605: Introduction to Parallel Computing Lecture : GPU Architecture
COMP 605: Introduction to Parallel Computing Lecture : GPU Architecture Mary Thomas Department of Computer Science Computational Science Research Center (CSRC) San Diego State University (SDSU) Posted:
More informationIntroduction to CUDA (2 of 2)
Announcements Introduction to CUDA (2 of 2) Patrick Cozzi University of Pennsylvania CIS 565 - Fall 2012 Open pull request for Project 0 Project 1 released. Due Sunday 09/30 Not due Tuesday, 09/25 Code
More information1/19/11. Administrative. L2: Hardware Execution Model and Overview. What is an Execution Model? Parallel programming model. Outline.
L2: Hardware Execution Model and Overview January 19, 2011 Administrative First assignment out, due Friday at 5PM Use handin on CADE machines to submit handin cs6963 lab1 The file
More informationLecture 6. Programming with Message Passing Message Passing Interface (MPI)
Lecture 6 Programming with Message Passing Message Passing Interface (MPI) Announcements 2011 Scott B. Baden / CSE 262 / Spring 2011 2 Finish CUDA Today s lecture Programming with message passing 2011
More informationGPU Fundamentals Jeff Larkin November 14, 2016
GPU Fundamentals Jeff Larkin , November 4, 206 Who Am I? 2002 B.S. Computer Science Furman University 2005 M.S. Computer Science UT Knoxville 2002 Graduate Teaching Assistant 2005 Graduate
More informationLecture 9. Thread Divergence Scheduling Instruction Level Parallelism
Lecture 9 Thread Divergence Scheduling Instruction Level Parallelism Announcements Tuesday s lecture on 2/11 will be moved to room 4140 from 6.30 PM to 7.50 PM Office hours cancelled on Thursday; make
More informationCS 179 Lecture 4. GPU Compute Architecture
CS 179 Lecture 4 GPU Compute Architecture 1 This is my first lecture ever Tell me if I m not speaking loud enough, going too fast/slow, etc. Also feel free to give me lecture feedback over email or at
More informationIntroduction to CUDA (1 of n*)
Administrivia Introduction to CUDA (1 of n*) Patrick Cozzi University of Pennsylvania CIS 565 - Spring 2011 Paper presentation due Wednesday, 02/23 Topics first come, first serve Assignment 4 handed today
More informationLecture 8: GPU Programming. CSE599G1: Spring 2017
Lecture 8: GPU Programming CSE599G1: Spring 2017 Announcements Project proposal due on Thursday (4/28) 5pm. Assignment 2 will be out today, due in two weeks. Implement GPU kernels and use cublas library
More informationGPU programming. Dr. Bernhard Kainz
GPU programming Dr. Bernhard Kainz Overview About myself Motivation GPU hardware and system architecture GPU programming languages GPU programming paradigms Pitfalls and best practice Reduction and tiling
More informationRegister file. A single large register file (ex. 16K registers) is partitioned among the threads of the dispatched blocks.
Sharing the resources of an SM Warp 0 Warp 1 Warp 47 Register file A single large register file (ex. 16K registers) is partitioned among the threads of the dispatched blocks Shared A single SRAM (ex. 16KB)
More informationME964 High Performance Computing for Engineering Applications
ME964 High Performance Computing for Engineering Applications Memory Issues in CUDA Execution Scheduling in CUDA February 23, 2012 Dan Negrut, 2012 ME964 UW-Madison Computers are useless. They can only
More informationCUDA Basics. July 6, 2016
Mitglied der Helmholtz-Gemeinschaft CUDA Basics July 6, 2016 CUDA Kernels Parallel portion of application: execute as a kernel Entire GPU executes kernel, many threads CUDA threads: Lightweight Fast switching
More informationLecture 15: Introduction to GPU programming. Lecture 15: Introduction to GPU programming p. 1
Lecture 15: Introduction to GPU programming Lecture 15: Introduction to GPU programming p. 1 Overview Hardware features of GPGPU Principles of GPU programming A good reference: David B. Kirk and Wen-mei
More informationCME 213 S PRING Eric Darve
CME 213 S PRING 2017 Eric Darve Review Secret behind GPU performance: simple cores but a large number of them; even more threads can exist live on the hardware (10k 20k threads live). Important performance
More informationCOSC 6374 Parallel Computations Introduction to CUDA
COSC 6374 Parallel Computations Introduction to CUDA Edgar Gabriel Fall 2014 Disclaimer Material for this lecture has been adopted based on various sources Matt Heavener, CS, State Univ. of NY at Buffalo
More informationLecture 5. Performance Programming with CUDA
Lecture 5 Performance Programming with CUDA Announcements 2011 Scott B. Baden / CSE 262 / Spring 2011 2 Today s lecture Matrix multiplication 2011 Scott B. Baden / CSE 262 / Spring 2011 3 Memory Hierarchy
More informationPractical Introduction to CUDA and GPU
Practical Introduction to CUDA and GPU Charlie Tang Centre for Theoretical Neuroscience October 9, 2009 Overview CUDA - stands for Compute Unified Device Architecture Introduced Nov. 2006, a parallel computing
More informationIntroduction to CUDA CIRC Summer School 2014
Introduction to CUDA CIRC Summer School 2014 Baowei Liu Center of Integrated Research Computing University of Rochester October 20, 2014 Introduction Overview What will you learn on this class? Start from
More informationGPU Programming Using CUDA. Samuli Laine NVIDIA Research
GPU Programming Using CUDA Samuli Laine NVIDIA Research Today GPU vs CPU Different architecture, different workloads Basics of CUDA Executing code on GPU Managing memory between CPU and GPU CUDA API Quick
More informationIntroduction to CUDA
Introduction to CUDA Oliver Meister November 7 th 2012 Tutorial Parallel Programming and High Performance Computing, November 7 th 2012 1 References D. Kirk, W. Hwu: Programming Massively Parallel Processors,
More informationDense Linear Algebra. HPC - Algorithms and Applications
Dense Linear Algebra HPC - Algorithms and Applications Alexander Pöppl Technical University of Munich Chair of Scientific Computing November 6 th 2017 Last Tutorial CUDA Architecture thread hierarchy:
More informationIntroduction to GPGPUs and to CUDA programming model
Introduction to GPGPUs and to CUDA programming model www.cineca.it Marzia Rivi m.rivi@cineca.it GPGPU architecture CUDA programming model CUDA efficient programming Debugging & profiling tools CUDA libraries
More informationIntroduction to Numerical General Purpose GPU Computing with NVIDIA CUDA. Part 1: Hardware design and programming model
Introduction to Numerical General Purpose GPU Computing with NVIDIA CUDA Part 1: Hardware design and programming model Dirk Ribbrock Faculty of Mathematics, TU dortmund 2016 Table of Contents Why parallel
More informationCUDA Architecture & Programming Model
CUDA Architecture & Programming Model Course on Multi-core Architectures & Programming Oliver Taubmann May 9, 2012 Outline Introduction Architecture Generation Fermi A Brief Look Back At Tesla What s New
More informationWarps and Reduction Algorithms
Warps and Reduction Algorithms 1 more on Thread Execution block partitioning into warps single-instruction, multiple-thread, and divergence 2 Parallel Reduction Algorithms computing the sum or the maximum
More informationCUDA Memory Types All material not from online sources/textbook copyright Travis Desell, 2012
CUDA Memory Types All material not from online sources/textbook copyright Travis Desell, 2012 Overview 1. Memory Access Efficiency 2. CUDA Memory Types 3. Reducing Global Memory Traffic 4. Example: Matrix-Matrix
More information1/26/09. Administrative. L4: Hardware Execution Model and Overview. Recall Execution Model. Outline. First assignment out, due Friday at 5PM
Administrative L4: Hardware Execution Model and Overview January 26, 2009 First assignment out, due Friday at 5PM Any questions? New mailing list: cs6963-discussion@list.eng.utah.edu Please use for all
More informationCUDA Workshop. High Performance GPU computing EXEBIT Karthikeyan
CUDA Workshop High Performance GPU computing EXEBIT- 2014 Karthikeyan CPU vs GPU CPU Very fast, serial, Low Latency GPU Slow, massively parallel, High Throughput Play Demonstration Compute Unified Device
More informationCOSC 6385 Computer Architecture. - Data Level Parallelism (II)
COSC 6385 Computer Architecture - Data Level Parallelism (II) Fall 2013 SIMD Instructions Originally developed for Multimedia applications Same operation executed for multiple data items Uses a fixed length
More informationCartoon parallel architectures; CPUs and GPUs
Cartoon parallel architectures; CPUs and GPUs CSE 6230, Fall 2014 Th Sep 11! Thanks to Jee Choi (a senior PhD student) for a big assist 1 2 3 4 5 6 7 8 9 10 11 12 13 14 ~ socket 14 ~ core 14 ~ HWMT+SIMD
More informationCUDA OPTIMIZATIONS ISC 2011 Tutorial
CUDA OPTIMIZATIONS ISC 2011 Tutorial Tim C. Schroeder, NVIDIA Corporation Outline Kernel optimizations Launch configuration Global memory throughput Shared memory access Instruction throughput / control
More informationSpring Prof. Hyesoon Kim
Spring 2011 Prof. Hyesoon Kim 2 Warp is the basic unit of execution A group of threads (e.g. 32 threads for the Tesla GPU architecture) Warp Execution Inst 1 Inst 2 Inst 3 Sources ready T T T T One warp
More informationMatrix Multiplication in CUDA. A case study
Matrix Multiplication in CUDA A case study 1 Matrix Multiplication: A Case Study Matrix multiplication illustrates many of the basic features of memory and thread management in CUDA Usage of thread/block
More informationGeneral Purpose GPU programming (GP-GPU) with Nvidia CUDA. Libby Shoop
General Purpose GPU programming (GP-GPU) with Nvidia CUDA Libby Shoop 3 What is (Historical) GPGPU? General Purpose computation using GPU and graphics API in applications other than 3D graphics GPU accelerates
More informationCUDA and GPU Performance Tuning Fundamentals: A hands-on introduction. Francesco Rossi University of Bologna and INFN
CUDA and GPU Performance Tuning Fundamentals: A hands-on introduction Francesco Rossi University of Bologna and INFN * Using this terminology since you ve already heard of SIMD and SPMD at this school
More informationE6895 Advanced Big Data Analytics Lecture 8: GPU Examples and GPU on ios devices
E6895 Advanced Big Data Analytics Lecture 8: GPU Examples and GPU on ios devices Ching-Yung Lin, Ph.D. Adjunct Professor, Dept. of Electrical Engineering and Computer Science IBM Chief Scientist, Graph
More informationGRAPHICS PROCESSING UNITS
GRAPHICS PROCESSING UNITS Slides by: Pedro Tomás Additional reading: Computer Architecture: A Quantitative Approach, 5th edition, Chapter 4, John L. Hennessy and David A. Patterson, Morgan Kaufmann, 2011
More informationSparse Linear Algebra in CUDA
Sparse Linear Algebra in CUDA HPC - Algorithms and Applications Alexander Pöppl Technical University of Munich Chair of Scientific Computing November 22 nd 2017 Table of Contents Homework - Worksheet 2
More informationPortland State University ECE 588/688. Graphics Processors
Portland State University ECE 588/688 Graphics Processors Copyright by Alaa Alameldeen 2018 Why Graphics Processors? Graphics programs have different characteristics from general purpose programs Highly
More informationCUDA Programming. Week 1. Basic Programming Concepts Materials are copied from the reference list
CUDA Programming Week 1. Basic Programming Concepts Materials are copied from the reference list G80/G92 Device SP: Streaming Processor (Thread Processors) SM: Streaming Multiprocessor 128 SP grouped into
More informationAn Introduction to GPGPU Pro g ra m m ing - CUDA Arc hitec ture
An Introduction to GPGPU Pro g ra m m ing - CUDA Arc hitec ture Rafia Inam Mälardalen Real-Time Research Centre Mälardalen University, Västerås, Sweden http://www.mrtc.mdh.se rafia.inam@mdh.se CONTENTS
More informationAdvanced CUDA Programming. Dr. Timo Stich
Advanced CUDA Programming Dr. Timo Stich (tstich@nvidia.com) Outline SIMT Architecture, Warps Kernel optimizations Global memory throughput Launch configuration Shared memory access Instruction throughput
More informationKernel optimizations Launch configuration Global memory throughput Shared memory access Instruction throughput / control flow
Fundamental Optimizations (GTC 2010) Paulius Micikevicius NVIDIA Outline Kernel optimizations Launch configuration Global memory throughput Shared memory access Instruction throughput / control flow Optimization
More informationCUDA. Schedule API. Language extensions. nvcc. Function type qualifiers (1) CUDA compiler to handle the standard C extensions.
Schedule CUDA Digging further into the programming manual Application Programming Interface (API) text only part, sorry Image utilities (simple CUDA examples) Performace considerations Matrix multiplication
More informationCUDA (Compute Unified Device Architecture)
CUDA (Compute Unified Device Architecture) Mike Bailey History of GPU Performance vs. CPU Performance GFLOPS Source: NVIDIA G80 = GeForce 8800 GTX G71 = GeForce 7900 GTX G70 = GeForce 7800 GTX NV40 = GeForce
More informationOverview. Lecture 1: an introduction to CUDA. Hardware view. Hardware view. hardware view software view CUDA programming
Overview Lecture 1: an introduction to CUDA Mike Giles mike.giles@maths.ox.ac.uk hardware view software view Oxford University Mathematical Institute Oxford e-research Centre Lecture 1 p. 1 Lecture 1 p.
More information2/2/11. Administrative. L6: Memory Hierarchy Optimization IV, Bandwidth Optimization. Project Proposal (due 3/9) Faculty Project Suggestions
Administrative L6: Memory Hierarchy Optimization IV, Bandwidth Optimization Next assignment available Goals of assignment: simple memory hierarchy management block-thread decomposition tradeoff Due Tuesday,
More informationCS377P Programming for Performance GPU Programming - II
CS377P Programming for Performance GPU Programming - II Sreepathi Pai UTCS November 11, 2015 Outline 1 GPU Occupancy 2 Divergence 3 Costs 4 Cooperation to reduce costs 5 Scheduling Regular Work Outline
More informationParallel Programming Principle and Practice. Lecture 9 Introduction to GPGPUs and CUDA Programming Model
Parallel Programming Principle and Practice Lecture 9 Introduction to GPGPUs and CUDA Programming Model Outline Introduction to GPGPUs and Cuda Programming Model The Cuda Thread Hierarchy / Memory Hierarchy
More informationME964 High Performance Computing for Engineering Applications
ME964 High Performance Computing for Engineering Applications Execution Scheduling in CUDA Revisiting Memory Issues in CUDA February 17, 2011 Dan Negrut, 2011 ME964 UW-Madison Computers are useless. They
More informationLecture 9. Outline. CUDA : a General-Purpose Parallel Computing Architecture. CUDA Device and Threads CUDA. CUDA Architecture CUDA (I)
Lecture 9 CUDA CUDA (I) Compute Unified Device Architecture 1 2 Outline CUDA Architecture CUDA Architecture CUDA programming model CUDA-C 3 4 CUDA : a General-Purpose Parallel Computing Architecture CUDA
More informationCUDA Lecture 2. Manfred Liebmann. Technische Universität München Chair of Optimal Control Center for Mathematical Sciences, M17
CUDA Lecture 2 Manfred Liebmann Technische Universität München Chair of Optimal Control Center for Mathematical Sciences, M17 manfred.liebmann@tum.de December 15, 2015 CUDA Programming Fundamentals CUDA
More informationGPU Programming. Performance Considerations. Miaoqing Huang University of Arkansas Fall / 60
1 / 60 GPU Programming Performance Considerations Miaoqing Huang University of Arkansas Fall 2013 2 / 60 Outline Control Flow Divergence Memory Coalescing Shared Memory Bank Conflicts Occupancy Loop Unrolling
More informationCE 431 Parallel Computer Architecture Spring Graphics Processor Units (GPU) Architecture
CE 431 Parallel Computer Architecture Spring 2017 Graphics Processor Units (GPU) Architecture Nikos Bellas Computer and Communications Engineering Department University of Thessaly Some slides borrowed
More informationHIGH-PERFORMANCE COMPUTING WITH CUDA AND TESLA GPUS
HIGH-PERFORMANCE COMPUTING WITH CUDA AND TESLA GPUS Timothy Lanfear, NVIDIA WHAT IS GPU COMPUTING? What is GPU Computing? x86 PCIe bus GPU Computing with CPU + GPU Heterogeneous Computing Low Latency or
More informationCS/EE 217 GPU Architecture and Parallel Programming. Lecture 10. Reduction Trees
CS/EE 217 GPU Architecture and Parallel Programming Lecture 10 Reduction Trees David Kirk/NVIDIA and Wen-mei W. Hwu University of Illinois, 2007-2012 1 Objective To master Reduction Trees, arguably the
More informationCSE 160 Lecture 24. Graphical Processing Units
CSE 160 Lecture 24 Graphical Processing Units Announcements Next week we meet in 1202 on Monday 3/11 only On Weds 3/13 we have a 2 hour session Usual class time at the Rady school final exam review SDSC
More informationCS 179: GPU Computing. Recitation 2: Synchronization, Shared memory, Matrix Transpose
CS 179: GPU Computing Recitation 2: Synchronization, Shared memory, Matrix Transpose Synchronization Ideal case for parallelism: no resources shared between threads no communication between threads Many
More informationCS427 Multicore Architecture and Parallel Computing
CS427 Multicore Architecture and Parallel Computing Lecture 6 GPU Architecture Li Jiang 2014/10/9 1 GPU Scaling A quiet revolution and potential build-up Calculation: 936 GFLOPS vs. 102 GFLOPS Memory Bandwidth:
More informationProgramming with CUDA, WS09
Programming with CUDA and Parallel Algorithms Waqar Saleem Jens Müller Lecture 3 Thursday, 29 Nov, 2009 Recap Motivational videos Example kernel Thread IDs Memory overhead CUDA hardware and programming
More informationGPGPU. Lecture 2: CUDA
GPGPU Lecture 2: CUDA GPU is fast Previous GPGPU Constraints Dealing with graphics API Working with the corner cases of the graphics API Addressing modes Limited texture size/dimension Shader capabilities
More informationCOSC 462 Parallel Programming
November 22, 2017 1/12 COSC 462 Parallel Programming CUDA Beyond Basics Piotr Luszczek Mixing Blocks and Threads int N = 100, SN = N * sizeof(double); global void sum(double *a, double *b, double *c) {
More informationReductions and Low-Level Performance Considerations CME343 / ME May David Tarjan NVIDIA Research
Reductions and Low-Level Performance Considerations CME343 / ME339 27 May 2011 David Tarjan [dtarjan@nvidia.com] NVIDIA Research REDUCTIONS Reduction! Reduce vector to a single value! Via an associative
More informationCUDA Experiences: Over-Optimization and Future HPC
CUDA Experiences: Over-Optimization and Future HPC Carl Pearson 1, Simon Garcia De Gonzalo 2 Ph.D. candidates, Electrical and Computer Engineering 1 / Computer Science 2, University of Illinois Urbana-Champaign
More informationECE 408 / CS 483 Final Exam, Fall 2014
ECE 408 / CS 483 Final Exam, Fall 2014 Thursday 18 December 2014 8:00 to 11:00 Central Standard Time You may use any notes, books, papers, or other reference materials. In the interest of fair access across
More informationLecture 7. Using Shared Memory Performance programming and the memory hierarchy
Lecture 7 Using Shared Memory Performance programming and the memory hierarchy Announcements Scott B. Baden /CSE 260/ Winter 2014 2 Assignment #1 Blocking for cache will boost performance but a lot more
More informationComputer Architecture
Jens Teubner Computer Architecture Summer 2017 1 Computer Architecture Jens Teubner, TU Dortmund jens.teubner@cs.tu-dortmund.de Summer 2017 Jens Teubner Computer Architecture Summer 2017 34 Part II Graphics
More information