LBANN: Livermore Big Ar.ficial Neural Network HPC Toolkit
|
|
|
- Bryan Black
- 5 years ago
- Views:
Transcription
1 LBANN: Livermore Big Ar.ficial Neural Network HPC Toolkit MLHPC 2015 Nov. 15, 2015 Brian Van Essen, Hyojin Kim, Roger Pearce, Kofi Boakye, Barry Chen Center for Applied ScienDfic CompuDng (CASC) + ComputaDonal Engineering This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA Lawrence Livermore National Security, LLC
2 Na.onal Security, Science, and Economic Compe..veness Applica.ons are Genera.ng Ever- Growing Collec.ons of Data Advanced Manufacturing Aerial PlaHorms Massive Open Multimedia Data Pace of collection far exceeds human inspection ability We have Big Data but Small Labels Neural Interfaces Science Facili.es 2
3 Big Data Small Labels Focus on training with unsupervised feature extracdon Stacked auto- encoders Fine- tune with small labels Moving beyond strict image processing Biological data sets Sensors from large sciendfic instruments (e.g. NIF) Incorporate imagery with addidonal sensor modalides (e.g. addidve manufacturing) Preliminary focus on large, fully connected dense layers Extend to unrolled RNN Adding support for convoludonal kernels OpDmize for data- intensive HPC systems Distributed memory algorithm Low latency interconnect Node- local State- of- the art distributed linear algebra library 3
4 Extrac.ng Parallelism Model Parallelism (train a single model faster) Distributed algorithm across muldple HPC nodes Future work will extend this to include a[ached GPU accelerators Data Parallelism (process data faster) Larger mini- batches reduce synchronizadon steps Leverage node- local for data staging Overlap communicadon with computadon Future work: GPU- offload Train muldple models concurrently Node- local data amplificadon 4
5 Computa.onal Horsepower is Required for the Deep Learning Andrew Ng s Deep Learning Rocket Analogy: Powerful engine: Use large Low Bias models Rocket fuel: Minimize Variance with vast training data HPC resources enable the training of massive- scale Deep Learning networks Vast collecdons of data fuel the Deep Learning engine 5
6 Distribu.ng DNN across HPC nodes Each layer of model is distributed across nodes Distributed matrix library (Elemental) provides dense matrix operadons Rank 0 - N 0 Rank 1 - N 1 Rank 2 - N 2 Rank 3 - N 3 Model M 0 - Layer H 0 Model M 0 - Input Layer Input data is staged into node- local Each node stages a separate mini- batch mini-batch 0 mini-batch 1 mini-batch 2 mini-batch 3 Input from Lustre 6
7 Distribu.ng data AcDve mini- batch is replicated from source node to each MPI rank First layer muldplies distributed matrix with replicated input data Rank 0 - N 0 Rank 1 - N 1 Rank 2 - N 2 Rank 3 - N 3 Rank 0 - N 0 Rank 1 - N 1 Rank 2 - N 2 Rank 3 - N 3 Model M 0 - Layer H 0 Model M 0 - Layer H 0 Model M 0 - Input Layer Model M 0 - Input Layer 7
8 Experimental setup & Learning Task LLNL Catalyst HPC system (324 nodes) 24 Xeon EP X5660 cores, 128 GB DRAM, and 800GB of node- local Aggregate bandwidth of GB/s to a Lustre parallel file system 48 Hyper- Threaded cores per node ILSVRC2012 data set Image size: = 196,608 Neural network topology 197K X 197K, X is the number of neurons in a fully connected hidden layer Network sizes: 50K, 100K, 400K neurons Matrix sizes: 9.8B, 19.7B, 78.6B parameters Weight matrix sizes: 73GB, 147GB, 293GB (double FP) Sokware stack (C++) Elemental distributed linear algebra library MPI communicadon Intel muld- threaded BLAS library 8
9 Reconstruc8on"error"(sum."squares"error)"of"LBANN"autoencoder"with"ImageNet"! Training an auto- encoder 25000" 10K"neurons" 50K"neurons" 20000" 400K"neurons" Visualizing auto- encoder learning ReconstrucDon cost ReconstrucDng training image aker 100 and 200 epochs 15000" 10000" 5000" 0" 0" 10" 20" 30" 40" 50" 60" 70" 80" 90" 100" 110" 120" 130" 140" 150" 160" 170" 180" 190" Epochs" 10K neurons epochs 50K neurons epochs 400K neurons epochs Original image 9
10 Strong Scaling: Time per unit work (mini- batch) # nodes versus mini- batch training Dme Processing muldple images per step Reduces number of synchronizadons per epoch Computes a be[er gradient Balancing # of steps versus quality of step 50" 40" Strong"scaling"9"Training";me"of"each"mini"batch"(in"sec)" Mini"batch"size"="8" Mini"batch"size"="32" Mini"batch"size"="128" Mini"batch"size"="512" Mini"batch"size"="2048" Mini"batch"size"="16" Mini"batch"size"="64" Mini"batch"size"="256" Mini"batch"size"="1024" Test 50K neurons nodes, 12 ranks per node mini- batch sizes from images 30" 20" Large mini- batches benefit greatly from addidonal nodes Good scaling up to 64 nodes Smaller mini- batches have limited improvement beyond 16 or 32 nodes Insufficient work to effecdvely amordze communicadon overheads 10" 0" 8" 16" 32" 64" 128" #"Nodes" 10
11 Strong Scaling: Total.me for fixed amount of work Mini- batch size versus wall clock Dme Fixed number of epochs Test 50K neurons nodes 12 ranks per node mini- batch sizes from images Good scaling with smaller node counts Diminishing returns for MB > " 40000" 35000" 30000" 25000" 20000" 15000" 10000" Num."Nodes"="16" Num."Nodes"="32" Num."Nodes"="64" Num."Nodes"="128" On larger node counts problem size is too small to leverage resources Too li[le work per node to offset communicadon overhead Diminishing returns aker 32 MB > " 0" 8" 16" 32" 64" 128" 256" 512" 1024" 2048" Mini-batch"size" 11
12 Weak Scaling: Increasing problem size and compute resources 22# Weak#scaling#6#Training#9me#of#each#mini#batch#(in#sec)# 20# 18# 16# 14# 8# 16# Nodes# 32# 64# Scaling the number of neurons from 50K to 400K 8 nodes to 64 nodes, respecdvely Mini- batch size is 256 images. Processing Dme of each mini- batch if fairly constant as problem size and available resources increase ~10% variadon in MB processing Dme Scaling up model sizes: matrices become more rectangular as # neurons increases 2D parddoning scheme for data distribudon 12
13 Tuning Elemental library algorithms Test 400K neurons 256 image mini- batch 23.25& 22.75& 22.25& 21.75& 21.25& 20.75& 20.25& 19.75& 19.25& 18.75& 18.25& Data is distributed element- wise Exploring new block distributed implementadon in v0.86- git Training&;me&of&each&mini&batch&(in&sec)& 32& 64& 128& 256& 512& 1024& 2048& 4096& 8192& 16384& Block&Size& Algorithmic block size affects operator implementadon Performance levels out once a sufficiently large block size is reached for local BLAS libraries 19% performance difference between block size of 32 and
14 Load- balancing distributed algorithm versus node- local math lib. 19# Training#8me#of#each#mini#batch#(in#sec)# 17# 15# Test 400K neurons 128 image mini- batch 13# 6# 9# Tasks#per#Node# 12# 18# 24# Balancing # of tasks versus # of threads per task Tuning the available resources to Intel BLAS library BLAS library uses free cores for thread- parallel math operadons 48 HyperThreaded cores per node # of tasks should evenly divide # cores 8 threads per task provided peak performance 18 tasks per node is 19% worse than average training Dme 14
15 Local data staging & Parallelizing I/O 4500" 4000" 3500" Parallel"I/O"8"Transfer+Load">me"(in"sec)" Uncached"node8local"" Cached"Lustre"PFS" Test 50K neurons 128 image mini- batch 32 nodes 3000" 2500" 2000" 1500" 1000" Stage data to node- local Avoids addidonal memory pressure ~12.9x faster than PFS with 128 I/O streams 500" 0" 1" 2" 4" 8" 16" 32" 64" 128" #"of"parallel"i/o"streams" Includes 18.56s overhead for copying and untar ing the data from the Lustre PFS Dovetails into future data augmentadon techniques Sufficient I/O parallelism significantly amordze data movement 15
16 Summary: LBANN is a work in progress LBANN toolkit is opdmized for: Unsupervised feature extracdon Data- intensive High Performance CompuDng systems Large, distributed neural network models Elemental library provides scalable, distributed linear algebra library Next Steps: ConvoluDonal, local recepdve fields Integrate GPU accelerators (including muldple per node) Open source release Explore training muldple models in parallel 16
17
LBANN: Livermore Big Artificial Neural Network HPC Toolkit
LBANN: Livermore Big Artificial Neural Network HPC Toolkit Brian Van Essen, Hyojin Kim, Roger Pearce, Kofi Boakye, Barry Chen Lawrence Livermore National Laboratory MLHPC 2015 Presented by: Sourav Chakraborty
Got Burst Buffer. Now What? Early experiences, exciting future possibilities, and what we need from the system to make it work
Got Burst Buffer. Now What? Early experiences, exciting future possibilities, and what we need from the system to make it work The Salishan Conference on High-Speed Computing April 26, 2016 Adam Moody
Graphite IntroducDon and Overview. Goals, Architecture, and Performance
Graphite IntroducDon and Overview Goals, Architecture, and Performance 4 The Future of MulDcore #Cores 128 1000 cores? CompuDng has moved aggressively to muldcore 64 32 MIT Raw Intel SSC Up to 72 cores
GPU ACCELERATION OF WSMP (WATSON SPARSE MATRIX PACKAGE)
GPU ACCELERATION OF WSMP (WATSON SPARSE MATRIX PACKAGE) NATALIA GIMELSHEIN ANSHUL GUPTA STEVE RENNICH SEID KORIC NVIDIA IBM NVIDIA NCSA WATSON SPARSE MATRIX PACKAGE (WSMP) Cholesky, LDL T, LU factorization
Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability
Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability Janis Keuper Itwm.fraunhofer.de/ml Competence Center High Performance Computing Fraunhofer ITWM, Kaiserslautern,
Deep Learning with Intel DAAL
Deep Learning with Intel DAAL on Knights Landing Processor David Ojika dave.n.ojika@cern.ch March 22, 2017 Outline Introduction and Motivation Intel Knights Landing Processor Intel Data Analytics and Acceleration
Allevia'ng memory bandwidth pressure with wavefront temporal blocking and diamond 'ling Tareq Malas* Georg Hager Gerhard Wellein David Keyes*
Allevia'ng memory bandwidth pressure with wavefront temporal blocking and diamond 'ling Tareq Malas* Georg Hager Gerhard Wellein David Keyes* Erlangen Regional Compu0ng Center, Germany *King Abdullah Univ.
Noise Injection Techniques to Expose Subtle and Unintended Message Races
Noise Injection Techniques to Expose Subtle and Unintended Message Races PPoPP2017 February 6th, 2017 Kento Sato, Dong H. Ahn, Ignacio Laguna, Gregory L. Lee, Martin Schulz and Christopher M. Chambreau
Persistent RNNs. (stashing recurrent weights on-chip) Gregory Diamos. April 7, Baidu SVAIL
(stashing recurrent weights on-chip) Baidu SVAIL April 7, 2016 SVAIL Think hard AI. Goal Develop hard AI technologies that impact 100 million users. Deep Learning at SVAIL 100 GFLOP/s 1 laptop 6 TFLOP/s
QR Decomposition on GPUs
QR Decomposition QR Algorithms Block Householder QR Andrew Kerr* 1 Dan Campbell 1 Mark Richards 2 1 Georgia Tech Research Institute 2 School of Electrical and Computer Engineering Georgia Institute of
High Performance Computing
High Performance Computing 9th Lecture 2016/10/28 YUKI ITO 1 Selected Paper: vdnn: Virtualized Deep Neural Networks for Scalable, MemoryEfficient Neural Network Design Minsoo Rhu, Natalia Gimelshein, Jason
SDA: Software-Defined Accelerator for Large- Scale DNN Systems
SDA: Software-Defined Accelerator for Large- Scale DNN Systems Jian Ouyang, 1 Shiding Lin, 1 Wei Qi, Yong Wang, Bo Yu, Song Jiang, 2 1 Baidu, Inc. 2 Wayne State University Introduction of Baidu A dominant
High-Performance Data Loading and Augmentation for Deep Neural Network Training
High-Performance Data Loading and Augmentation for Deep Neural Network Training Trevor Gale tgale@ece.neu.edu Steven Eliuk steven.eliuk@gmail.com Cameron Upright c.upright@samsung.com Roadmap 1. The General-Purpose
LS-DYNA Best-Practices: Networking, MPI and Parallel File System Effect on LS-DYNA Performance
11 th International LS-DYNA Users Conference Computing Technology LS-DYNA Best-Practices: Networking, MPI and Parallel File System Effect on LS-DYNA Performance Gilad Shainer 1, Tong Liu 2, Jeff Layton
Introduction to CUDA Algoritmi e Calcolo Parallelo. Daniele Loiacono
Introduction to CUDA Algoritmi e Calcolo Parallelo References q This set of slides is mainly based on: " CUDA Technical Training, Dr. Antonino Tumeo, Pacific Northwest National Laboratory " Slide of Applied
Leveraging Flash in HPC Systems
Leveraging Flash in HPC Systems IEEE MSST June 3, 2015 This work was performed under the auspices of the U.S. Department of Energy by under Contract DE-AC52-07NA27344. Lawrence Livermore National Security,
Characterization and Benchmarking of Deep Learning. Natalia Vassilieva, PhD Sr. Research Manager
Characterization and Benchmarking of Deep Learning Natalia Vassilieva, PhD Sr. Research Manager Deep learning applications Vision Speech Text Other Search & information extraction Security/Video surveillance
SDA: Software-Defined Accelerator for Large- Scale DNN Systems
SDA: Software-Defined Accelerator for Large- Scale DNN Systems Jian Ouyang, 1 Shiding Lin, 1 Wei Qi, 1 Yong Wang, 1 Bo Yu, 1 Song Jiang, 2 1 Baidu, Inc. 2 Wayne State University Introduction of Baidu A
Introduction to CUDA Algoritmi e Calcolo Parallelo. Daniele Loiacono
Introduction to CUDA Algoritmi e Calcolo Parallelo References This set of slides is mainly based on: CUDA Technical Training, Dr. Antonino Tumeo, Pacific Northwest National Laboratory Slide of Applied
High Performance Computing. Introduction to Parallel Computing
High Performance Computing Introduction to Parallel Computing Acknowledgements Content of the following presentation is borrowed from The Lawrence Livermore National Laboratory https://hpc.llnl.gov/training/tutorials
DELL EMC ISILON F800 AND H600 I/O PERFORMANCE
DELL EMC ISILON F800 AND H600 I/O PERFORMANCE ABSTRACT This white paper provides F800 and H600 performance data. It is intended for performance-minded administrators of large compute clusters that access
Trends in HPC (hardware complexity and software challenges)
Trends in HPC (hardware complexity and software challenges) Mike Giles Oxford e-research Centre Mathematical Institute MIT seminar March 13th, 2013 Mike Giles (Oxford) HPC Trends March 13th, 2013 1 / 18
KNL tools. Dr. Fabio Baruffa
KNL tools Dr. Fabio Baruffa fabio.baruffa@lrz.de 2 Which tool do I use? A roadmap to optimization We will focus on tools developed by Intel, available to users of the LRZ systems. Again, we will skip the
MICROWAY S NVIDIA TESLA V100 GPU SOLUTIONS GUIDE
MICROWAY S NVIDIA TESLA V100 GPU SOLUTIONS GUIDE LEVERAGE OUR EXPERTISE sales@microway.com http://microway.com/tesla NUMBERSMASHER TESLA 4-GPU SERVER/WORKSTATION Flexible form factor 4 PCI-E GPUs + 3 additional
Energy Efficient K-Means Clustering for an Intel Hybrid Multi-Chip Package
High Performance Machine Learning Workshop Energy Efficient K-Means Clustering for an Intel Hybrid Multi-Chip Package Matheus Souza, Lucas Maciel, Pedro Penna, Henrique Freitas 24/09/2018 Agenda Introduction
Single-Points of Performance
Single-Points of Performance Mellanox Technologies Inc. 29 Stender Way, Santa Clara, CA 9554 Tel: 48-97-34 Fax: 48-97-343 http://www.mellanox.com High-performance computations are rapidly becoming a critical
Best Practices for Setting BIOS Parameters for Performance
White Paper Best Practices for Setting BIOS Parameters for Performance Cisco UCS E5-based M3 Servers May 2013 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page
HPE Deep Learning Cookbook: Recipes to Run Deep Learning Workloads. Natalia Vassilieva, Sergey Serebryakov
HPE Deep Learning Cookbook: Recipes to Run Deep Learning Workloads Natalia Vassilieva, Sergey Serebryakov Deep learning ecosystem today Software Hardware 2 HPE s portfolio for deep learning Government,
CO405H. Department of Compu:ng Imperial College London. Computing in Space with OpenSPL Topic 15: Porting CPU Software to DFEs
CO405H Computing in Space with OpenSPL Topic 15: Porting CPU Software to DFEs Oskar Mencer Georgi Gaydadjiev Department of Compu:ng Imperial College London h#p://www.doc.ic.ac.uk/~oskar/ h#p://www.doc.ic.ac.uk/~georgig/
HPC Architectures. Types of resource currently in use
HPC Architectures Types of resource currently in use Reusing this material This work is licensed under a Creative Commons Attribution- NonCommercial-ShareAlike 4.0 International License. http://creativecommons.org/licenses/by-nc-sa/4.0/deed.en_us
Outline. Motivation Parallel k-means Clustering Intel Computing Architectures Baseline Performance Performance Optimizations Future Trends
Collaborators: Richard T. Mills, Argonne National Laboratory Sarat Sreepathi, Oak Ridge National Laboratory Forrest M. Hoffman, Oak Ridge National Laboratory Jitendra Kumar, Oak Ridge National Laboratory
Harp-DAAL for High Performance Big Data Computing
Harp-DAAL for High Performance Big Data Computing Large-scale data analytics is revolutionizing many business and scientific domains. Easy-touse scalable parallel techniques are necessary to process big
Profiling DNN Workloads on a Volta-based DGX-1 System
Profiling DNN Workloads on a Volta-based DGX-1 System Saiful A. Mojumder 1, Marcia S Louis 1, Yifan Sun 2, Amir Kavyan Ziabari 3, José L. Abellán 4, John Kim 5, David Kaeli 2, Ajay Joshi 1 1 ECE Department,
ABySS Performance Benchmark and Profiling. May 2010
ABySS Performance Benchmark and Profiling May 2010 Note The following research was performed under the HPC Advisory Council activities Participating vendors: AMD, Dell, Mellanox Compute resource - HPC
Economic Viability of Hardware Overprovisioning in Power- Constrained High Performance Compu>ng
Economic Viability of Hardware Overprovisioning in Power- Constrained High Performance Compu>ng Energy Efficient Supercompu1ng, SC 16 November 14, 2016 This work was performed under the auspices of the U.S.
Performance analysis tools: Intel VTuneTM Amplifier and Advisor. Dr. Luigi Iapichino
Performance analysis tools: Intel VTuneTM Amplifier and Advisor Dr. Luigi Iapichino luigi.iapichino@lrz.de Which tool do I use in my project? A roadmap to optimisation After having considered the MPI layer,
Intel profiling tools and roofline model. Dr. Luigi Iapichino
Intel profiling tools and roofline model Dr. Luigi Iapichino luigi.iapichino@lrz.de Which tool do I use in my project? A roadmap to optimization (and to the next hour) We will focus on tools developed
EARLY EVALUATION OF THE CRAY XC40 SYSTEM THETA
EARLY EVALUATION OF THE CRAY XC40 SYSTEM THETA SUDHEER CHUNDURI, SCOTT PARKER, KEVIN HARMS, VITALI MOROZOV, CHRIS KNIGHT, KALYAN KUMARAN Performance Engineering Group Argonne Leadership Computing Facility
Exploiting GPU Caches in Sparse Matrix Vector Multiplication. Yusuke Nagasaka Tokyo Institute of Technology
Exploiting GPU Caches in Sparse Matrix Vector Multiplication Yusuke Nagasaka Tokyo Institute of Technology Sparse Matrix Generated by FEM, being as the graph data Often require solving sparse linear equation
Entropy-Aware I/O Pipelining for Large-Scale Deep Learning on HPC Systems
Entropy-Aware I/O Pipelining for Large-Scale Deep Learning on HPC Systems Yue Zhu Fahim Chowdhury Huansong Fu Adam Moody Kathryn Mohror Kento Sato Weikuan Yu Florida State University Lawrence Livermore
Can FPGAs beat GPUs in accelerating next-generation Deep Neural Networks? Discussion of the FPGA 17 paper by Intel Corp. (Nurvitadhi et al.
Can FPGAs beat GPUs in accelerating next-generation Deep Neural Networks? Discussion of the FPGA 17 paper by Intel Corp. (Nurvitadhi et al.) Andreas Kurth 2017-12-05 1 In short: The situation Image credit:
World s most advanced data center accelerator for PCIe-based servers
NVIDIA TESLA P100 GPU ACCELERATOR World s most advanced data center accelerator for PCIe-based servers HPC data centers need to support the ever-growing demands of scientists and researchers while staying
Approaches to acceleration: GPUs vs Intel MIC. Fabio AFFINITO SCAI department
Approaches to acceleration: GPUs vs Intel MIC Fabio AFFINITO SCAI department Single core Multi core Many core GPU Intel MIC 61 cores 512bit-SIMD units from http://www.karlrupp.net/ from http://www.karlrupp.net/
Progress on Efficient Integration of Lustre* and Hadoop/YARN
Progress on Efficient Integration of Lustre* and Hadoop/YARN Weikuan Yu Robin Goldstone Omkar Kulkarni Bryon Neitzel * Some name and brands may be claimed as the property of others. MapReduce l l l l A
Exploiting InfiniBand and GPUDirect Technology for High Performance Collectives on GPU Clusters
Exploiting InfiniBand and Direct Technology for High Performance Collectives on Clusters Ching-Hsiang Chu chu.368@osu.edu Department of Computer Science and Engineering The Ohio State University OSU Booth
Toward portable I/O performance by leveraging system abstractions of deep memory and interconnect hierarchies
Toward portable I/O performance by leveraging system abstractions of deep memory and interconnect hierarchies François Tessier, Venkatram Vishwanath, Paul Gressier Argonne National Laboratory, USA Wednesday
Parallel Combinatorial BLAS and Applications in Graph Computations
Parallel Combinatorial BLAS and Applications in Graph Computations Aydın Buluç John R. Gilbert University of California, Santa Barbara SIAM ANNUAL MEETING 2009 July 8, 2009 1 Primitives for Graph Computations
Accelera'on A+acks on PBKDF2
Accelera'on A+acks on PBKDF2 Or, what is inside the black- box of oclhashcat? Andrew Ruddick, UK Dr. Jeff Yan, Lancaster University, UK andrew.ruddick@hotmail.co.uk, jeff.yan@lancaster.ac.uk What is PBKDF2?
Smart Trading with Cray Systems: Making Smarter Models + Better Decisions in Algorithmic Trading
Smart Trading with Cray Systems: Making Smarter Models + Better Decisions in Algorithmic Trading Smart Trading with Cray Systems Agenda: Cray Overview Market Trends & Challenges Mitigating Risk with Deeper
Key Technologies for 100 PFLOPS. Copyright 2014 FUJITSU LIMITED
Key Technologies for 100 PFLOPS How to keep the HPC-tree growing Molecular dynamics Computational materials Drug discovery Life-science Quantum chemistry Eigenvalue problem FFT Subatomic particle phys.
Performance and Energy Usage of Workloads on KNL and Haswell Architectures
Performance and Energy Usage of Workloads on KNL and Haswell Architectures Tyler Allen 1 Christopher Daley 2 Doug Doerfler 2 Brian Austin 2 Nicholas Wright 2 1 Clemson University 2 National Energy Research
Snowpack: Efficient Parameter Choice for GPU Kernels via Static Analysis and Statistical Prediction ScalA 17, Denver, CO, USA, November 13, 2017
Snowpack: Efficient Parameter Choice for GPU Kernels via Static Analysis and Statistical Prediction ScalA 17, Denver, CO, USA, November 13, 2017 Ignacio Laguna Ranvijay Singh, Paul Wood, Ravi Gupta, Saurabh
CafeGPI. Single-Sided Communication for Scalable Deep Learning
CafeGPI Single-Sided Communication for Scalable Deep Learning Janis Keuper itwm.fraunhofer.de/ml Competence Center High Performance Computing Fraunhofer ITWM, Kaiserslautern, Germany Deep Neural Networks
GPU Accelerated Machine Learning for Bond Price Prediction
GPU Accelerated Machine Learning for Bond Price Prediction Venkat Bala Rafael Nicolas Fermin Cota Motivation Primary Goals Demonstrate potential benefits of using GPUs over CPUs for machine learning Exploit
TUNING CUDA APPLICATIONS FOR MAXWELL
TUNING CUDA APPLICATIONS FOR MAXWELL DA-07173-001_v7.0 March 2015 Application Note TABLE OF CONTENTS Chapter 1. Maxwell Tuning Guide... 1 1.1. NVIDIA Maxwell Compute Architecture... 1 1.2. CUDA Best Practices...2
Technologies for High Performance Data Analytics
Technologies for High Performance Data Analytics Dr. Jens Krüger Fraunhofer ITWM 1 Fraunhofer ITWM n Institute for Industrial Mathematics n Located in Kaiserslautern, Germany n Staff: ~ 240 employees +
Performance Tools for Technical Computing
Christian Terboven terboven@rz.rwth-aachen.de Center for Computing and Communication RWTH Aachen University Intel Software Conference 2010 April 13th, Barcelona, Spain Agenda o Motivation and Methodology
Master Informatics Eng.
Advanced Architectures Master Informatics Eng. 207/8 A.J.Proença The Roofline Performance Model (most slides are borrowed) AJProença, Advanced Architectures, MiEI, UMinho, 207/8 AJProença, Advanced Architectures,
FastForward I/O and Storage: ACG 8.6 Demonstration
FastForward I/O and Storage: ACG 8.6 Demonstration Kyle Ambert, Jaewook Yu, Arnab Paul Intel Labs June, 2014 NOTICE: THIS MANUSCRIPT HAS BEEN AUTHORED BY INTEL UNDER ITS SUBCONTRACT WITH LAWRENCE LIVERMORE
Introduction to Xeon Phi. Bill Barth January 11, 2013
Introduction to Xeon Phi Bill Barth January 11, 2013 What is it? Co-processor PCI Express card Stripped down Linux operating system Dense, simplified processor Many power-hungry operations removed Wider
CS8803SC Software and Hardware Cooperative Computing GPGPU. Prof. Hyesoon Kim School of Computer Science Georgia Institute of Technology
CS8803SC Software and Hardware Cooperative Computing GPGPU Prof. Hyesoon Kim School of Computer Science Georgia Institute of Technology Why GPU? A quiet revolution and potential build-up Calculation: 367
MAGMA. Matrix Algebra on GPU and Multicore Architectures
MAGMA Matrix Algebra on GPU and Multicore Architectures Innovative Computing Laboratory Electrical Engineering and Computer Science University of Tennessee Piotr Luszczek (presenter) web.eecs.utk.edu/~luszczek/conf/
Accelerating String Matching Algorithms on Multicore Processors Cheng-Hung Lin
Accelerating String Matching Algorithms on Multicore Processors Cheng-Hung Lin Department of Electrical Engineering, National Taiwan Normal University, Taipei, Taiwan Abstract String matching is the most
Mapping MPI+X Applications to Multi-GPU Architectures
Mapping MPI+X Applications to Multi-GPU Architectures A Performance-Portable Approach Edgar A. León Computer Scientist San Jose, CA March 28, 2018 GPU Technology Conference This work was performed under
IHK/McKernel: A Lightweight Multi-kernel Operating System for Extreme-Scale Supercomputing
: A Lightweight Multi-kernel Operating System for Extreme-Scale Supercomputing Balazs Gerofi Exascale System Software Team, RIKEN Center for Computational Science 218/Nov/15 SC 18 Intel Extreme Computing
GPU ACCELERATED DATABASE MANAGEMENT SYSTEMS
CIS 601 - Graduate Seminar Presentation 1 GPU ACCELERATED DATABASE MANAGEMENT SYSTEMS PRESENTED BY HARINATH AMASA CSU ID: 2697292 What we will talk about.. Current problems GPU What are GPU Databases GPU
Oak Ridge National Laboratory Computing and Computational Sciences
Oak Ridge National Laboratory Computing and Computational Sciences OFA Update by ORNL Presented by: Pavel Shamis (Pasha) OFA Workshop Mar 17, 2015 Acknowledgments Bernholdt David E. Hill Jason J. Leverman
W H I T E P A P E R U n l o c k i n g t h e P o w e r o f F l a s h w i t h t h e M C x - E n a b l e d N e x t - G e n e r a t i o n V N X
Global Headquarters: 5 Speen Street Framingham, MA 01701 USA P.508.872.8200 F.508.935.4015 www.idc.com W H I T E P A P E R U n l o c k i n g t h e P o w e r o f F l a s h w i t h t h e M C x - E n a b
OpenACC2 vs.openmp4. James Lin 1,2 and Satoshi Matsuoka 2
2014@San Jose Shanghai Jiao Tong University Tokyo Institute of Technology OpenACC2 vs.openmp4 he Strong, the Weak, and the Missing to Develop Performance Portable Applica>ons on GPU and Xeon Phi James
Oncilla - a Managed GAS Runtime for Accelerating Data Warehousing Queries
Oncilla - a Managed GAS Runtime for Accelerating Data Warehousing Queries Jeffrey Young, Alex Merritt, Se Hoon Shon Advisor: Sudhakar Yalamanchili 4/16/13 Sponsors: Intel, NVIDIA, NSF 2 The Problem Big
Advances of parallel computing. Kirill Bogachev May 2016
Advances of parallel computing Kirill Bogachev May 2016 Demands in Simulations Field development relies more and more on static and dynamic modeling of the reservoirs that has come a long way from being
TUNING CUDA APPLICATIONS FOR MAXWELL
TUNING CUDA APPLICATIONS FOR MAXWELL DA-07173-001_v6.5 August 2014 Application Note TABLE OF CONTENTS Chapter 1. Maxwell Tuning Guide... 1 1.1. NVIDIA Maxwell Compute Architecture... 1 1.2. CUDA Best Practices...2
Performance Analysis of BLAS Libraries in SuperLU_DIST for SuperLU_MCDT (Multi Core Distributed) Development
Available online at www.prace-ri.eu Partnership for Advanced Computing in Europe Performance Analysis of BLAS Libraries in SuperLU_DIST for SuperLU_MCDT (Multi Core Distributed) Development M. Serdar Celebi
An Advanced Graph Processor Prototype
An Advanced Graph Processor Prototype Vitaliy Gleyzer GraphEx 2016 DISTRIBUTION STATEMENT A. Approved for public release: distribution unlimited. This material is based upon work supported by the Assistant
Big Data Analytics Performance for Large Out-Of- Core Matrix Solvers on Advanced Hybrid Architectures
Procedia Computer Science Volume 51, 2015, Pages 2774 2778 ICCS 2015 International Conference On Computational Science Big Data Analytics Performance for Large Out-Of- Core Matrix Solvers on Advanced Hybrid
Dynamic Fine Grain Scheduling of Pipeline Parallelism. Presented by: Ram Manohar Oruganti and Michael TeWinkle
Dynamic Fine Grain Scheduling of Pipeline Parallelism Presented by: Ram Manohar Oruganti and Michael TeWinkle Overview Introduction Motivation Scheduling Approaches GRAMPS scheduling method Evaluation
Introduction to Parallel and Distributed Computing. Linh B. Ngo CPSC 3620
Introduction to Parallel and Distributed Computing Linh B. Ngo CPSC 3620 Overview: What is Parallel Computing To be run using multiple processors A problem is broken into discrete parts that can be solved
The Future of High Performance Computing
The Future of High Performance Computing Randal E. Bryant Carnegie Mellon University http://www.cs.cmu.edu/~bryant Comparing Two Large-Scale Systems Oakridge Titan Google Data Center 2 Monolithic supercomputer
Center for Scalable Application Development Software (CScADS): Automatic Performance Tuning Workshop
Center for Scalable Application Development Software (CScADS): Automatic Performance Tuning Workshop http://cscads.rice.edu/ Discussion and Feedback CScADS Autotuning 07 Top Priority Questions for Discussion
Implementing Long-term Recurrent Convolutional Network Using HLS on POWER System
Implementing Long-term Recurrent Convolutional Network Using HLS on POWER System Xiaofan Zhang1, Mohamed El Hadedy1, Wen-mei Hwu1, Nam Sung Kim1, Jinjun Xiong2, Deming Chen1 1 University of Illinois Urbana-Champaign
MPI Optimizations via MXM and FCA for Maximum Performance on LS-DYNA
MPI Optimizations via MXM and FCA for Maximum Performance on LS-DYNA Gilad Shainer 1, Tong Liu 1, Pak Lui 1, Todd Wilde 1 1 Mellanox Technologies Abstract From concept to engineering, and from design to
Placement de processus (MPI) sur architecture multi-cœur NUMA
Placement de processus (MPI) sur architecture multi-cœur NUMA Emmanuel Jeannot, Guillaume Mercier LaBRI/INRIA Bordeaux Sud-Ouest/ENSEIRB Runtime Team Lyon, journées groupe de calcul, november 2010 Emmanuel.Jeannot@inria.fr
Evaluation of Parallel I/O Performance and Energy with Frequency Scaling on Cray XC30 Suren Byna and Brian Austin
Evaluation of Parallel I/O Performance and Energy with Frequency Scaling on Cray XC30 Suren Byna and Brian Austin Lawrence Berkeley National Laboratory Energy efficiency at Exascale A design goal for future
Atos announces the Bull sequana X1000 the first exascale-class supercomputer. Jakub Venc
Atos announces the Bull sequana X1000 the first exascale-class supercomputer Jakub Venc The world is changing The world is changing Digital simulation will be the key contributor to overcome 21 st century
Performance Analysis of Memory Transfers and GEMM Subroutines on NVIDIA TESLA GPU Cluster
Performance Analysis of Memory Transfers and GEMM Subroutines on NVIDIA TESLA GPU Cluster Veerendra Allada, Troy Benjegerdes Electrical and Computer Engineering, Ames Laboratory Iowa State University &
CISC 879 Software Support for Multicore Architectures Spring Student Presentation 6: April 8. Presenter: Pujan Kafle, Deephan Mohan
CISC 879 Software Support for Multicore Architectures Spring 2008 Student Presentation 6: April 8 Presenter: Pujan Kafle, Deephan Mohan Scribe: Kanik Sem The following two papers were presented: A Synchronous
Accelerating MPI Message Matching and Reduction Collectives For Multi-/Many-core Architectures Mohammadreza Bayatpour, Hari Subramoni, D. K.
Accelerating MPI Message Matching and Reduction Collectives For Multi-/Many-core Architectures Mohammadreza Bayatpour, Hari Subramoni, D. K. Panda Department of Computer Science and Engineering The Ohio
Designing High-Performance MPI Collectives in MVAPICH2 for HPC and Deep Learning
5th ANNUAL WORKSHOP 209 Designing High-Performance MPI Collectives in MVAPICH2 for HPC and Deep Learning Hari Subramoni Dhabaleswar K. (DK) Panda The Ohio State University The Ohio State University E-mail:
CSense: A Stream- Processing Toolkit for Robust and High- Rate of Mobile Sensing ApplicaDons
CSense: A Stream- Processing Toolkit for Robust and High- Rate of Mobile Sensing ApplicaDons IPSN 2014 Farley Lai, Syed Shabih Hasan, AusDn Laugesen, Octav Chipara Department of Computer Science University
Turing Architecture and CUDA 10 New Features. Minseok Lee, Developer Technology Engineer, NVIDIA
Turing Architecture and CUDA 10 New Features Minseok Lee, Developer Technology Engineer, NVIDIA Turing Architecture New SM Architecture Multi-Precision Tensor Core RT Core Turing MPS Inference Accelerated,
Deep Neural Networks for Market Prediction on the Xeon Phi *
Deep Neural Networks for Market Prediction on the Xeon Phi * Matthew Dixon 1, Diego Klabjan 2 and * The support of Intel Corporation is gratefully acknowledged Jin Hoon Bang 3 1 Department of Finance,
Toward Automated Application Profiling on Cray Systems
Toward Automated Application Profiling on Cray Systems Charlene Yang, Brian Friesen, Thorsten Kurth, Brandon Cook NERSC at LBNL Samuel Williams CRD at LBNL I have a dream.. M.L.K. Collect performance data:
CUDA. Matthew Joyner, Jeremy Williams
CUDA Matthew Joyner, Jeremy Williams Agenda What is CUDA? CUDA GPU Architecture CPU/GPU Communication Coding in CUDA Use cases of CUDA Comparison to OpenCL What is CUDA? What is CUDA? CUDA is a parallel
CSCI 402: Computer Architectures. Parallel Processors (2) Fengguang Song Department of Computer & Information Science IUPUI.
CSCI 402: Computer Architectures Parallel Processors (2) Fengguang Song Department of Computer & Information Science IUPUI 6.6 - End Today s Contents GPU Cluster and its network topology The Roofline performance
Chelsio Communications. Meeting Today s Datacenter Challenges. Produced by Tabor Custom Publishing in conjunction with: CUSTOM PUBLISHING
Meeting Today s Datacenter Challenges Produced by Tabor Custom Publishing in conjunction with: 1 Introduction In this era of Big Data, today s HPC systems are faced with unprecedented growth in the complexity
Intelligent Hybrid Flash Management
Intelligent Hybrid Flash Management Jérôme Gaysse Senior Technology&Market Analyst jerome.gaysse@silinnov-consulting.com Flash Memory Summit 2018 Santa Clara, CA 1 Research context Analysis of system &
Performance comparison between a massive SMP machine and clusters
Performance comparison between a massive SMP machine and clusters Martin Scarcia, Stefano Alberto Russo Sissa/eLab joint Democritos/Sissa Laboratory for e-science Via Beirut 2/4 34151 Trieste, Italy Stefano
NAMD Performance Benchmark and Profiling. January 2015
NAMD Performance Benchmark and Profiling January 2015 2 Note The following research was performed under the HPC Advisory Council activities Participating vendors: Intel, Dell, Mellanox Compute resource
Recurrent Neural Networks. Deep neural networks have enabled major advances in machine learning and AI. Convolutional Neural Networks
Deep neural networks have enabled major advances in machine learning and AI Computer vision Language translation Speech recognition Question answering And more Problem: DNNs are challenging to serve and
NVIDIA TESLA V100 GPU ARCHITECTURE THE WORLD S MOST ADVANCED DATA CENTER GPU
NVIDIA TESLA V100 GPU ARCHITECTURE THE WORLD S MOST ADVANCED DATA CENTER GPU WP-08608-001_v1.1 August 2017 WP-08608-001_v1.1 TABLE OF CONTENTS Introduction to the NVIDIA Tesla V100 GPU Architecture...