Deep Learning Inferencing on IBM Cloud with NVIDIA TensorRT
|
|
- Catherine King
- 5 years ago
- Views:
Transcription
1 Deep Learning Inferencing on IBM Cloud with NVIDIA TensorRT Khoa Huynh Senior Technical Staff Member (STSM), IBM Larry Brown Senior Software Engineer, IBM
2 Agenda Introduction Inferencing with PyCaffe TensorRT Overview TensorRT Implementation Performance Results Conclusions Q & A 2
3 Introduction AI especially deep learning has seen rapid advancement in recent years Initial focus on image processing, expanding to natural language, different neural network models, recurrent networks, and DL frameworks Much attention to developing networks and training models Very compute intensive, GPUs nearly a necessity As DL becomes mainstream focus is shifting to inferencing (use of the trained network) Inferencing cloud service could handle requests from multiple users One request at a time or collect a batch of requests to inference at once But only wait a short time to fill a batch or latency is affected Or one user might submit a larger number of images to classify at once Inferencing cloud service needs quick response - latency seen by the user to handle large volume overall throughput 3
4 IBM Cloud GPU Offerings Bare Metal Servers Nvidia M60 GPUs (monthly & hourly) Nvidia K80 GPU PCIe cards (monthly & hourly) Nvidia P100 GPUs (monthly) Nvidia V100 GPUs (monthly) Virtual Servers Nvidia P100 GPUs (monthly & hourly) Nvidia V100 (coming soon monthly & hourly) Deep Learning as a Service (DLaaS) Part of Watson Machine Learning (WML) Focused on deep learning training Allows user to run training jobs on a cluster of GPU-enabled machines using various frameworks PowerAI Available in 2Q2018 with PowerAI R5 Delivered through IBM Cloud Catalog & supported by IBM Trusted Partner Nimbix On-demand cloud provisioning Containerized Native Distributed Deep Learning (DDL) and Large Model Support (LMS) 4
5 Inferencing with PyCaffe Given a trained model want to use it to classify images. Study performance of various GPUs, FPGAs, etc. Can use C++, but more familiar with Python so that was the language of choice. Unlike training, a single GPU is used. Use multiple threads, processes, or services if more volume needs to take advantage of more GPUs. root@v100:~/infer_caffe# python infer_caffe.py -h usage: infer_caffe.py [-h] -m MODEL -w WEIGHTS -l LMDB [-b BATCH] [-i ITERATIONS] [-c CAFFEROOT] [--blobname BLOBNAME] [--labels LABELS] [--meanimage MEANIMAGE] [--debug] [--gpu] [--csvfile CSVFILE] [--quiet] Use a trained Caffe model to classify images from a LMDB database. 5
6 infer_caffe Sample Output python infer_caffe.py -m ~/model_zoo/caffe/vgg16/pretrained/vgg_ilsvrc_16_layers_deploy.prototxt -w ~/model_zoo/caffe/vgg16/pretrained/vgg_ilsvrc_16_layers.caffemodel -l /datasets/x86_lmdb/lmdb/ilsvrc12_val_lmdb/ -c /opt/nvidia/caffe-0.16/caffe -b 1 -i 5 --gpu --csvfile./pycaffe.csv --quiet Final Stats (times in seconds) Date: 03/08/2018 Time: 09:05:54 Host: V100 Iterations: 5 Batch size: 1 Data type: NA Total run time: Stats for all iterations Total predictions: 5 Correct top 1 predictions: 5 Correct top 5 predictions: 5 Top 1 accuracy: % Top 5 accuracy: % Inference time -- Total: Mean: Min: Max: Range: STD: Median: Inference time/prediction: Images/sec:
7 Program Flow Parse command line # Create the neural network. net = caffe.net(model_def, # defines the structure of the model model_weights, # contains the trained weights caffe.test) # use test mode (e.g., don't perform dropout) for each iteration read a batch of images from LMDB # Call the network. out = net.forward() # Time only this step. output statistics 7
8 TensorRT Overview Speeds up inferencing by Merging layers and tensors to reduce size of network and execute in a single kernel. Selects the best specialized kernel for the target hardware based on layer parameters and measured performance. Stages Build: optimize the network (layers, weights, labels) to produce a runtime plan or engine. Optimization can take some time so the resulting engine can be serialized to a file. Deploy: run the engine with given input data to get the resulting predictions. Supports Python and C++. TensorRT Lite is a simplified interface for Python (not used here). Can create the TRT network yourself or use TRT utility to import and convert framework model into TRT form. Caffe and UFF (Universal Framework Format) compatible frameworks such as TensorFlow. 8
9 Reduced Precision Inferencing Model trained in FLOAT (FP32). TensorRT inferencing can use FLOAT, HALF, or INT8 (as supported by the GPU). Increase speed with no or little loss of accuracy. HALF could be some reduction of accuracy, not noticeable in general. INT8 a small reduction of accuracy. INT8 requires calibration files. Uses sample runs of data through the net to determine the range of FLOAT values encountered. Maps that range to INT8 s smaller range. Caffe patch available to easily generate these calibration files during a short training run when environment variable TENSORRT_INT8_BATCH_DIRECTORY is set. NVIDIA suggests For ImageNet networks, around 500 calibration images is adequate. 9
10 TensorRT Implementation Re-implementation of caffe_infer.py using TRT instead of pycaffe. Shares code for getting images, collecting stats, overall flow. python infer_caffe_trt.py -h usage: infer_caffe_trt.py [-h] -m MODEL -w WEIGHTS -l LMDB [-b BATCH] [-i ITERATIONS] [-c CAFFEROOT] [--imageshape IMAGESHAPE] [--max_batch MAX_BATCH] [--outputlayer OUTPUTLAYER] [--outputsize OUTPUTSIZE] [--dtype {FLOAT,HALF,INT8}] [--labels LABELS] [--meanimage MEANIMAGE] [--csvfile CSVFILE] [--calbatchdir CALBATCHDIR] [--firstcalbatch FIRSTCALBATCH] [--numcalbatches NUMCALBATCHES] [--debug] [--quiet] Uses NVidia TensorRT to optimize and run inference on a trained Caffe model performing an image recognition task and prints performance and accuracy results. 10
11 TRT Program Flow # Create the engine using the TRT utilities for Caffe. # Use the caffe model converter utility in tensorrt.utils. # We provide it a logger, a path to the model prototxt, the model file, the max batch size, # the max workspace size, the output layer(s) and the data type of the weights. engine_dtype = trt.infer.datatype[dtype] calibrator = None if engine_dtype == trt.infer.datatype.int8: calibrator = infer_utils.calibrator.calibrator(cal_batch_dir, first_cal_batch, num_cal_batches, debug) engine = trt.utils.caffe_to_trt_engine(trt_logger, model, weights, max_batch, 1 << 25, [output_layer], engine_dtype, calibrator=calibrator) # Allocate memory on the GPU with PyCUDA and register it with the engine. # The size of the allocations is the size of the input and expected output * the batch size d_input = cuda.mem_alloc(batch_size * image_shape[0] * image_shape[1] * 3 * np.dtype(np.float32).itemsize) d_output = cuda.mem_alloc(batch_size * output.size * output.dtype.itemsize) # The engine needs bindings provided as pointers to the GPU memory. # PyCUDA lets us do this for memory allocations by casting those allocations to ints bindings = [int(d_input), int(d_output)] # Create a cuda stream to run inference in. stream = cuda.stream() 11
12 TRT Program Flow # Time moving the data to the GPU, running the network, and getting the results back to the host as part of the # inference operation for this iteration. stats.begin_iteration() cuda.memcpy_htod_async(d_input, batchin, stream) # execute model context.enqueue(batch_size, bindings, stream.handle, None) # transfer predictions back cuda.memcpy_dtoh_async(output, d_output, stream) # syncronize threads stream.synchronize() 12
13 0.5 V100 Caffe VGGNet16 TensorRT 3.01 Inference Latency PyCaffe FLOAT HALF INT Average Latency/Batch (sec) Batch Size 13
14 2500 V100 Caffe VGGNet16 TensorRT 3.01 Inference Throughput PyCaffe FLOAT HALF INT Images/Second Batch Size 14
15 100 V100 Caffe VGGNet16 TensorRT Inference Accuracy PyCaffe FLOAT HALF INT Accuracy Per Cent Top 1 Top 5 15
16 90 V100 Caffe VGGNet16 TensortRT INT8 Inference Accuracy Top 1 Top Accuracy Per Cent Number of Calibration Images 16
17 Deep-Learning Model Inferencing Image Classification with VGG-16 on Caffe (Single Precision) Number of Images Processed Per Second (Inferening) Higher is better x Nvidia K80 GPU 1 x Nvidia P4 GPU 1 x Nvidia P100 GPU 1 x Nvidia V100 GPU 1 x Intel DLIA 1 x Nvidia K80 GPU SL Bare-Metal (Dual Xeon E5-2690v4) PyCaffe TensorRT TensorRT (HALF2) TensorRT (INT8) Intel Bare-Metal (Dual Xeon E5-2650v3) SL Bare-Metal (Dual Xeon E5-2690v4) SL Bare-Metal (Dual Xeon E5-2690v4) Intel Bare-Metal (Dual Xeon E5-2690v4) AWS p2.16xlarge (Dual Xeon E5-2686v4) Notes: Nvidia TensorRT with half-precision support improves DL inferencing performance by 5X on a V100 GPU The Nvidia P4 GPU is comparable to many FPGAs in terms of power consumption (30-70W) 17
18 Deep-Learning Model Inferencing Latency Per Iteration (ms) Deep Learning Model Inferencing - Image Classification VGG-16 Neural Net on Caffe Framework with TensorRT Lower is better (Single Precision Except Where Noted Otherwise) Batch Size K80 GPU P100 GPU P100 GPU w/ HALF2 V100 GPU V100 GPU w/ HALF2 Notes: The P100 and V100 GPUs deliver much better latencies and handles much larger batch sizes than the K80 GPU The Nvidia P4 GPU is comparable to many FPGAs in terms of power consumption (30-70W) Nvidia P100 GPU delivers 5X inferencing performance, and could handle much larger batch sizes, than previous-generation K80 GPU Nvidia P4 GPU could deliver up to 50% inferencing performance at less than 25% power consumption of a P100 GPU making the P4 a very cost-effective DL inferencing engine for a cloud platform Nvidia P4 GPU Intel DLIA Deep-Learning Model Inferencing (Image Classification) Nvidia P4 GPU vs. FPGA for DL VGG-16 Neural Net on Caffe Framework (Single Precision) Higher is better Classification Power Efficiency (Images/Second/Watt)
19 TensorRT Impressions Python interface was very important Utility to convert Caffe model to TRT network saved much work Building the network with native TRT calls is much more advanced Allows flexibility and customization for those who need it INT8 Calibration was challenging Doc was not complete Found a good blog that referenced classes not in the doc Need to write a Calibrator implementation by extending a class class Calibrator(trt.infer.Int8EntropyCalibrator): # from the doc did not work class Calibrator(trt.infer.EntropyCalibrator): # from the blog did work Overall for Caffe our experience was good. Other frameworks might require more use of lower TRT calls. 19
20 Conclusions V100 GPU is faster than anything else Nvidia TensorRT with half-precision support improves DL inferencing performance by 5X on a V100 GPU TensorRT is much faster than pycaffe HALF and INT8 are significantly faster than FLOAT with no or little loss of accuracy INT8 slightly better than HALF for batch size 1 Latency 1.8 vs 2.0 msec Throughput 558 vs 505 imagers/sec HALF has better throughput and latency at larger batch sizes Surprising that INT8 accuracy when calibration used only 1 image performed so well Suggest to follow guidance of using more images as calibration is not that slow and need only be done once Experiment with your own network and data as your results could vary 20
21 Thank You Khoa Huynh Senior Technical Staff Member (STSM), IBM Larry Brown Senior Software Engineer, IBM 21
S8765 Performance Optimization for Deep- Learning on the Latest POWER Systems
S8765 Performance Optimization for Deep- Learning on the Latest POWER Systems Khoa Huynh Senior Technical Staff Member (STSM), IBM Jonathan Samn Software Engineer, IBM Evolving from compute systems to
More informationEFFICIENT INFERENCE WITH TENSORRT. Han Vanholder
EFFICIENT INFERENCE WITH TENSORRT Han Vanholder AI INFERENCING IS EXPLODING 2 Trillion Messages Per Day On LinkedIn 500M Daily active users of iflytek 140 Billion Words Per Day Translated by Google 60
More informationInference Optimization Using TensorRT with Use Cases. Jack Han / 한재근 Solutions Architect NVIDIA
Inference Optimization Using TensorRT with Use Cases Jack Han / 한재근 Solutions Architect NVIDIA Search Image NLP Maps TensorRT 4 Adoption Use Cases Speech Video AI Inference is exploding 1 Billion Videos
More informationNVIDIA FOR DEEP LEARNING. Bill Veenhuis
NVIDIA FOR DEEP LEARNING Bill Veenhuis bveenhuis@nvidia.com Nvidia is the world s leading ai platform ONE ARCHITECTURE CUDA 2 GPU: Perfect Companion for Accelerating Apps & A.I. CPU GPU 3 Intro to AI AGENDA
More informationXilinx ML Suite Overview
Xilinx ML Suite Overview Yao Fu System Architect Data Center Acceleration Xilinx Accelerated Computing Workloads Machine Learning Inference Image classification and object detection Video Streaming Frame
More informationNVIDIA GPU CLOUD DEEP LEARNING FRAMEWORKS
TECHNICAL OVERVIEW NVIDIA GPU CLOUD DEEP LEARNING FRAMEWORKS A Guide to the Optimized Framework Containers on NVIDIA GPU Cloud Introduction Artificial intelligence is helping to solve some of the most
More informationDEEP NEURAL NETWORKS CHANGING THE AUTONOMOUS VEHICLE LANDSCAPE. Dennis Lui August 2017
DEEP NEURAL NETWORKS CHANGING THE AUTONOMOUS VEHICLE LANDSCAPE Dennis Lui August 2017 THE RISE OF GPU COMPUTING APPLICATIONS 10 7 10 6 GPU-Computing perf 1.5X per year 1000X by 2025 ALGORITHMS 10 5 1.1X
More informationEmbedded GPGPU and Deep Learning for Industrial Market
Embedded GPGPU and Deep Learning for Industrial Market Author: Dan Mor GPGPU and HPEC Product Line Manager September 2018 Table of Contents 1. INTRODUCTION... 3 2. DIFFICULTIES IN CURRENT EMBEDDED INDUSTRIAL
More informationBeyond Training The next steps of Machine Learning. Chris /in/chrisparsonsdev
Beyond Training The next steps of Machine Learning Chris Parsons chrisparsons@uk.ibm.com @chrisparsonsdev /in/chrisparsonsdev What is this talk? Part 1 What is Machine Learning? AI Infrastructure PowerAI
More informationMIOVISION DEEP LEARNING TRAFFIC ANALYTICS SYSTEM FOR REAL-WORLD DEPLOYMENT. Kurtis McBride CEO, Miovision
MIOVISION DEEP LEARNING TRAFFIC ANALYTICS SYSTEM FOR REAL-WORLD DEPLOYMENT Kurtis McBride CEO, Miovision ABOUT MIOVISION COMPANY Founded in 2005 40% growth, year over year Offices in Kitchener, Canada
More informationS8822 OPTIMIZING NMT WITH TENSORRT Micah Villmow Senior TensorRT Software Engineer
S8822 OPTIMIZING NMT WITH TENSORRT Micah Villmow Senior TensorRT Software Engineer 2 100 倍以上速く 本当に可能ですか? 2 DOUGLAS ADAMS BABEL FISH Neural Machine Translation Unit 3 4 OVER 100X FASTER, IS IT REALLY POSSIBLE?
More informationDeploying Deep Learning Networks to Embedded GPUs and CPUs
Deploying Deep Learning Networks to Embedded GPUs and CPUs Rishu Gupta, PhD Senior Application Engineer, Computer Vision 2015 The MathWorks, Inc. 1 MATLAB Deep Learning Framework Access Data Design + Train
More informationWorld s most advanced data center accelerator for PCIe-based servers
NVIDIA TESLA P100 GPU ACCELERATOR World s most advanced data center accelerator for PCIe-based servers HPC data centers need to support the ever-growing demands of scientists and researchers while staying
More informationNVIDIA DGX SYSTEMS PURPOSE-BUILT FOR AI
NVIDIA DGX SYSTEMS PURPOSE-BUILT FOR AI Overview Unparalleled Value Product Portfolio Software Platform From Desk to Data Center to Cloud Summary AI researchers depend on computing performance to gain
More informationA performance comparison of Deep Learning frameworks on KNL
A performance comparison of Deep Learning frameworks on KNL R. Zanella, G. Fiameni, M. Rorro Middleware, Data Management - SCAI - CINECA IXPUG Bologna, March 5, 2018 Table of Contents 1. Problem description
More informationNVIDIA DEEP LEARNING INSTITUTE
NVIDIA DEEP LEARNING INSTITUTE TRAINING CATALOG Valid Through July 31, 2018 INTRODUCTION The NVIDIA Deep Learning Institute (DLI) trains developers, data scientists, and researchers on how to use artificial
More informationS INSIDE NVIDIA GPU CLOUD DEEP LEARNING FRAMEWORK CONTAINERS
S8497 - INSIDE NVIDIA GPU CLOUD DEEP LEARNING FRAMEWORK CONTAINERS Chris Lamb CUDA and NGC Engineering, NVIDIA John Barco NGC Product Management, NVIDIA NVIDIA GPU Cloud (NGC) overview AGENDA Using NGC
More informationGPU Coder: Automatic CUDA and TensorRT code generation from MATLAB
GPU Coder: Automatic CUDA and TensorRT code generation from MATLAB Ram Kokku 2018 The MathWorks, Inc. 1 GPUs and CUDA programming faster Performance CUDA OpenCL C/C++ GPU Coder MATLAB Python Ease of programming
More informationDeep Learning Inference on Openshift with GPUs
Deep Learning Inference on Openshift with GPUs OpenShift Commons, Seattle, Dec 10 2018 Tripti Singhal Product Manager, NVIDIA Deep Learning Software Tushar Katarki Product Manager, AI on OpenShift AGENDA
More informationScalable Distributed Training with Parameter Hub: a whirlwind tour
Scalable Distributed Training with Parameter Hub: a whirlwind tour TVM Stack Optimization High-Level Differentiable IR Tensor Expression IR AutoTVM LLVM, CUDA, Metal VTA AutoVTA Edge FPGA Cloud FPGA ASIC
More informationNVIDIA DLI HANDS-ON TRAINING COURSE CATALOG
NVIDIA DLI HANDS-ON TRAINING COURSE CATALOG Valid Through July 31, 2018 INTRODUCTION The NVIDIA Deep Learning Institute (DLI) trains developers, data scientists, and researchers on how to use artificial
More informationShrinath Shanbhag Senior Software Engineer Microsoft Corporation
Accelerating GPU inferencing with DirectML and DirectX 12 Shrinath Shanbhag Senior Software Engineer Microsoft Corporation Machine Learning Machine learning has become immensely popular over the last decade
More informationDeep learning in MATLAB From Concept to CUDA Code
Deep learning in MATLAB From Concept to CUDA Code Roy Fahn Applications Engineer Systematics royf@systematics.co.il 03-7660111 Ram Kokku Principal Engineer MathWorks ram.kokku@mathworks.com 2017 The MathWorks,
More informationPouya Kousha Fall 2018 CSE 5194 Prof. DK Panda
Pouya Kousha Fall 2018 CSE 5194 Prof. DK Panda 1 Motivation And Intro Programming Model Spark Data Transformation Model Construction Model Training Model Inference Execution Model Data Parallel Training
More informationA NEW COMPUTING ERA JENSEN HUANG, FOUNDER & CEO GTC CHINA 2017
A NEW COMPUTING ERA JENSEN HUANG, FOUNDER & CEO GTC CHINA 2017 TWO FORCES DRIVING THE FUTURE OF COMPUTING 10 7 Transistors (thousands) 10 6 10 5 1.1X per year 10 4 10 3 10 2 1.5X per year Single-threaded
More informationTESLA V100 PERFORMANCE GUIDE. Life Sciences Applications
TESLA V100 PERFORMANCE GUIDE Life Sciences Applications NOVEMBER 2017 TESLA V100 PERFORMANCE GUIDE Modern high performance computing (HPC) data centers are key to solving some of the world s most important
More informationIBM Deep Learning Solutions
IBM Deep Learning Solutions Reference Architecture for Deep Learning on POWER8, P100, and NVLink October, 2016 How do you teach a computer to Perceive? 2 Deep Learning: teaching Siri to recognize a bicycle
More informationDemystifying Deep Learning
Demystifying Deep Learning Mandar Gujrathi Mandar.Gujrathi@mathworks.com.au 2015 The MathWorks, Inc. 1 2 Deep Learning Applications Voice assistants (speech to text) Teaching character to beat video game
More informationTENSORRT 4.0 RELEASE CANDIDATE (RC)
TENSORRT 4.0 RELEASE CANDIDATE (RC) DU-08731-001_v4.0 RC March 2018 Installation Guide TABLE OF CONTENTS Chapter 1. Overview... 1 Chapter 2. Getting Started... 2 Chapter 3. Downloading TensorRT...3 Chapter
More informationIn partnership with. VelocityAI REFERENCE ARCHITECTURE WHITE PAPER
In partnership with VelocityAI REFERENCE JULY // 2018 Contents Introduction 01 Challenges with Existing AI/ML/DL Solutions 01 Accelerate AI/ML/DL Workloads with Vexata VelocityAI 02 VelocityAI Reference
More informationTENSORRT 3.0. DU _v3.0 February Installation Guide
TENSORRT 3.0 DU-08731-001_v3.0 February 2018 Installation Guide TABLE OF CONTENTS Chapter 1. Overview... 1 Chapter 2. Getting Started... 2 Chapter 3. Downloading TensorRT...4 Chapter 4. Installing TensorRT...
More informationDemystifying Deep Learning
Demystifying Deep Learning Let the computers do the hard work Jérémy Huard 2015 The MathWorks, Inc. 1 2 Why MATLAB for Deep Learning? MATLAB is Productive MATLAB is Fast MATLAB Integrates with Open Source
More informationCERN openlab & IBM Research Workshop Trip Report
CERN openlab & IBM Research Workshop Trip Report Jakob Blomer, Javier Cervantes, Pere Mato, Radu Popescu 2018-12-03 Workshop Organization 1 full day at IBM Research Zürich ~25 participants from CERN ~10
More informationHigh Performance Computing
High Performance Computing 9th Lecture 2016/10/28 YUKI ITO 1 Selected Paper: vdnn: Virtualized Deep Neural Networks for Scalable, MemoryEfficient Neural Network Design Minsoo Rhu, Natalia Gimelshein, Jason
More informationSDA: Software-Defined Accelerator for Large- Scale DNN Systems
SDA: Software-Defined Accelerator for Large- Scale DNN Systems Jian Ouyang, 1 Shiding Lin, 1 Wei Qi, Yong Wang, Bo Yu, Song Jiang, 2 1 Baidu, Inc. 2 Wayne State University Introduction of Baidu A dominant
More informationMachine Learning In A Snap. Thomas Parnell Research Staff Member IBM Research - Zurich
Machine Learning In A Snap Thomas Parnell Research Staff Member IBM Research - Zurich What are GLMs? Ridge Regression Support Vector Machines Regression Generalized Linear Models Classification Lasso Regression
More informationOnto Petaflops with Kubernetes
Onto Petaflops with Kubernetes Vishnu Kannan Google Inc. vishh@google.com Key Takeaways Kubernetes can manage hardware accelerators at Scale Kubernetes provides a playground for ML ML journey with Kubernetes
More informationDeep Learning with Tensorflow AlexNet
Machine Learning and Computer Vision Group Deep Learning with Tensorflow http://cvml.ist.ac.at/courses/dlwt_w17/ AlexNet Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton, "Imagenet classification
More informationRevolutionizing the Datacenter
Power-Efficient Machine Learning using FPGAs on POWER Systems Ralph Wittig, Distinguished Engineer Office of the CTO, Xilinx Revolutionizing the Datacenter Join the Conversation #OpenPOWERSummit Top-5
More informationNVIDIA DEEP LEARNING PLATFORM
TECHNICAL OVERVIEW NVIDIA DEEP LEARNING PLATFORM Giant Leaps in Performance and Efficiency for AI Services, From the Data Center to the Network s Edge Introduction Artificial intelligence (AI), the dream
More informationEfficient Communication Library for Large-Scale Deep Learning
IBM Research AI Efficient Communication Library for Large-Scale Deep Learning Mar 26, 2018 Minsik Cho (minsikcho@us.ibm.com) Deep Learning changing Our Life Automotive/transportation Security/public safety
More informationGPU FOR DEEP LEARNING. 周国峰 Wuhan University 2017/10/13
GPU FOR DEEP LEARNING chandlerz@nvidia.com 周国峰 Wuhan University 2017/10/13 Why Deep Learning Boost Today? Nvidia SDK for Deep Learning? Agenda CUDA 8.0 cudnn TensorRT (GIE) NCCL DIGITS 2 Why Deep Learning
More informationIntroduction to Deep Learning in Signal Processing & Communications with MATLAB
Introduction to Deep Learning in Signal Processing & Communications with MATLAB Dr. Amod Anandkumar Pallavi Kar Application Engineering Group, Mathworks India 2019 The MathWorks, Inc. 1 Different Types
More informationHPE Deep Learning Cookbook: Recipes to Run Deep Learning Workloads. Natalia Vassilieva, Sergey Serebryakov
HPE Deep Learning Cookbook: Recipes to Run Deep Learning Workloads Natalia Vassilieva, Sergey Serebryakov Deep learning ecosystem today Software Hardware 2 HPE s portfolio for deep learning Government,
More informationTENSORRT. RN _v01 January Release Notes
TENSORRT RN-08624-030_v01 January 2018 Release Notes TABLE OF CONTENTS Chapter Chapter Chapter Chapter 1. 2. 3. 4. Overview...1 Release 3.0.2... 2 Release 3.0.1... 4 Release 2.1... 10 RN-08624-030_v01
More informationRecurrent Neural Networks. Deep neural networks have enabled major advances in machine learning and AI. Convolutional Neural Networks
Deep neural networks have enabled major advances in machine learning and AI Computer vision Language translation Speech recognition Question answering And more Problem: DNNs are challenging to serve and
More informationSupporting GPUs in Docker Containers on Apache Mesos
Supporting GPUs in Docker Containers on Apache Mesos MesosCon Europe - 2016 Kevin Klues Senior Software Engineer Mesosphere Yubo Li Staff Researcher IBM Research China Kevin Klues Yubo Li Kevin Klues is
More informationAccelerating Binarized Convolutional Neural Networks with Software-Programmable FPGAs
Accelerating Binarized Convolutional Neural Networks with Software-Programmable FPGAs Ritchie Zhao 1, Weinan Song 2, Wentao Zhang 2, Tianwei Xing 3, Jeng-Hau Lin 4, Mani Srivastava 3, Rajesh Gupta 4, Zhiru
More informationSDA: Software-Defined Accelerator for Large- Scale DNN Systems
SDA: Software-Defined Accelerator for Large- Scale DNN Systems Jian Ouyang, 1 Shiding Lin, 1 Wei Qi, 1 Yong Wang, 1 Bo Yu, 1 Song Jiang, 2 1 Baidu, Inc. 2 Wayne State University Introduction of Baidu A
More informationDeep Learning mit PowerAI - Ein Überblick
Stephen Lutz Deep Learning mit PowerAI - Open Group Master Certified IT Specialist Technical Sales IBM Cognitive Infrastructure IBM Germany Ein Überblick Stephen.Lutz@de.ibm.com What s that? and what s
More informationNVIDIA AI BRAIN OF SELF DRIVING AND HD MAPPING. September 13, 2016
NVIDIA AI BRAIN OF SELF DRIVING AND HD MAPPING September 13, 2016 AI FOR AUTONOMOUS DRIVING MAPPING KALDI LOCALIZATION DRIVENET Training on DGX-1 NVIDIA DGX-1 NVIDIA DRIVE PX 2 Driving with DriveWorks
More informationNvidia Jetson TX2 and its Software Toolset. João Fernandes 2017/2018
Nvidia Jetson TX2 and its Software Toolset João Fernandes 2017/2018 In this presentation Nvidia Jetson TX2: Hardware Nvidia Jetson TX2: Software Machine Learning: Neural Networks Convolutional Neural Networks
More informationNVIDIA PLATFORM FOR AI
NVIDIA PLATFORM FOR AI João Paulo Navarro, Solutions Architect - Linkedin i am ai HTTPS://WWW.YOUTUBE.COM/WATCH?V=GIZ7KYRWZGQ 2 NVIDIA Gaming VR AI & HPC Self-Driving Cars GPU Computing 3 GPU COMPUTING
More informationOpenACC Course. Office Hour #2 Q&A
OpenACC Course Office Hour #2 Q&A Q1: How many threads does each GPU core have? A: GPU cores execute arithmetic instructions. Each core can execute one single precision floating point instruction per cycle
More informationDeep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations, and Hardware Implications
Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations, and Hardware Implications Jongsoo Park Facebook AI System SW/HW Co-design Team Sep-21 2018 Team Introduction
More informationAdvancing State-of-the-Art of Autonomous Vehicles and Robotics Research using AWS GPU Instances
Advancing State-of-the-Art of Autonomous Vehicles and Robotics Research using AWS GPU Instances Adrien Gaidon - Machine Learning Lead, Toyota Research Institute Mike Garrison - Senior Systems Engineer,
More informationDeep Learning Frameworks with Spark and GPUs
Deep Learning Frameworks with Spark and GPUs Abstract Spark is a powerful, scalable, real-time data analytics engine that is fast becoming the de facto hub for data science and big data. However, in parallel,
More informationDGX UPDATE. Customer Presentation Deck May 8, 2017
DGX UPDATE Customer Presentation Deck May 8, 2017 NVIDIA DGX-1: The World s Fastest AI Supercomputer FASTEST PATH TO DEEP LEARNING EFFORTLESS PRODUCTIVITY REVOLUTIONARY AI PERFORMANCE Fully-integrated
More informationThe OpenVX Computer Vision and Neural Network Inference
The OpenVX Computer and Neural Network Inference Standard for Portable, Efficient Code Radhakrishna Giduthuri Editor, OpenVX Khronos Group radha.giduthuri@amd.com @RadhaGiduthuri Copyright 2018 Khronos
More informationTHE NVIDIA DEEP LEARNING ACCELERATOR
THE NVIDIA DEEP LEARNING ACCELERATOR INTRODUCTION NVDLA NVIDIA Deep Learning Accelerator Developed as part of Xavier NVIDIA s SOC for autonomous driving applications Optimized for Convolutional Neural
More informationLecture 12: Model Serving. CSE599W: Spring 2018
Lecture 12: Model Serving CSE599W: Spring 2018 Deep Learning Applications That drink will get you to 2800 calories for today I last saw your keys in the store room Remind Tom of the party You re on page
More informationMACHINE LEARNING WITH NVIDIA AND IBM POWER AI
MACHINE LEARNING WITH NVIDIA AND IBM POWER AI July 2017 Joerg Krall Sr. Business Ddevelopment Manager MFG EMEA jkrall@nvidia.com A NEW ERA OF COMPUTING AI & IOT Deep Learning, GPU 100s of billions of devices
More information2015 The MathWorks, Inc. 1
2015 The MathWorks, Inc. 1 개발에서구현까지 MATLAB 환경에서의딥러닝 김종남 Application Engineer 2015 The MathWorks, Inc. 2 3 Why MATLAB for Deep Learning? MATLAB is Productive MATLAB is Fast MATLAB Integrates with Open Source
More informationAllGoVision Achieves high Performance Optimization for its ANPR Solution with OpenVINO TM Toolkit
AllGoVision Achieves high Performance Optimization for its ANPR Solution with OpenVINO TM Toolkit Version.9 Migrating to OpenVINO framework help its deep learning Automatic Number Plate Recognition (ANPR)
More informationDeep learning prevalence. first neuroscience department. Spiking Neuron Operant conditioning First 1 Billion transistor processor
WELCOME TO Operant conditioning 1938 Spiking Neuron 1952 first neuroscience department 1964 Deep learning prevalence mid 2000s The Turing Machine 1936 Transistor 1947 First computer science department
More informationPOINT CLOUD DEEP LEARNING
POINT CLOUD DEEP LEARNING Innfarn Yoo, 3/29/28 / 57 Introduction AGENDA Previous Work Method Result Conclusion 2 / 57 INTRODUCTION 3 / 57 2D OBJECT CLASSIFICATION Deep Learning for 2D Object Classification
More informationCisco UCS C480 ML M5 Rack Server Performance Characterization
White Paper Cisco UCS C480 ML M5 Rack Server Performance Characterization The Cisco UCS C480 ML M5 Rack Server platform is designed for artificial intelligence and machine-learning workloads. 2018 Cisco
More informationOptimizing Efficiency of Deep Learning Workloads through GPU Virtualization
Optimizing Efficiency of Deep Learning Workloads through GPU Virtualization Presenters: Tim Kaldewey Performance Architect, Watson Group Michael Gschwind Chief Engineer ML & DL, Systems Group David K.
More informationTensorFlow: A System for Learning-Scale Machine Learning. Google Brain
TensorFlow: A System for Learning-Scale Machine Learning Google Brain The Problem Machine learning is everywhere This is in large part due to: 1. Invention of more sophisticated machine learning models
More informationTuring Architecture and CUDA 10 New Features. Minseok Lee, Developer Technology Engineer, NVIDIA
Turing Architecture and CUDA 10 New Features Minseok Lee, Developer Technology Engineer, NVIDIA Turing Architecture New SM Architecture Multi-Precision Tensor Core RT Core Turing MPS Inference Accelerated,
More informationOPTIMIZED GPU KERNELS FOR DEEP LEARNING. Amir Khosrowshahi
OPTIMIZED GPU KERNELS FOR DEEP LEARNING Amir Khosrowshahi GTC 17 Mar 2015 Outline About nervana Optimizing deep learning at assembler level Limited precision for deep learning neon benchmarks 2 About nervana
More informationTENSORRT. SWE-SWDOCTRT-001-RELN_vTensorRT October Release Notes
TENSORRT SWE-SWDOCTRT-001-RELN_v 5.0.3 October 2018 Release Notes TABLE OF CONTENTS Chapter 1. Overview...1 Chapter 2. Release 5.x.x... 2 2.1. Release 5.0.3... 2 2.2. Release 5.0.2... 3 2.3. Release 5.0.1
More informationHigh-Performance Data Loading and Augmentation for Deep Neural Network Training
High-Performance Data Loading and Augmentation for Deep Neural Network Training Trevor Gale tgale@ece.neu.edu Steven Eliuk steven.eliuk@gmail.com Cameron Upright c.upright@samsung.com Roadmap 1. The General-Purpose
More informationX10 specific Optimization of CPU GPU Data transfer with Pinned Memory Management
X10 specific Optimization of CPU GPU Data transfer with Pinned Memory Management Hideyuki Shamoto, Tatsuhiro Chiba, Mikio Takeuchi Tokyo Institute of Technology IBM Research Tokyo Programming for large
More informationGPU Programming Using NVIDIA CUDA
GPU Programming Using NVIDIA CUDA Siddhante Nangla 1, Professor Chetna Achar 2 1, 2 MET s Institute of Computer Science, Bandra Mumbai University Abstract: GPGPU or General-Purpose Computing on Graphics
More informationCharacterization and Benchmarking of Deep Learning. Natalia Vassilieva, PhD Sr. Research Manager
Characterization and Benchmarking of Deep Learning Natalia Vassilieva, PhD Sr. Research Manager Deep learning applications Vision Speech Text Other Search & information extraction Security/Video surveillance
More informationTraining Neural Networks with Mixed Precision MICHAEL CARILLI CHRISTIAN SAROFEEN MICHAEL RUBERRY BEN BARSDELL
Training Neural Networks with Mixed Precision MICHAEL CARILLI CHRISTIAN SAROFEEN MICHAEL RUBERRY BEN BARSDELL 1 THIS TALK Using mixed precision and Volta your networks can be: 1. 2-4x faster 2. half the
More informationDeep Learning on Modern Architectures. Keren Zhou 4/17/2017
Deep Learning on Modern Architectures Keren Zhou 4/17/2017 HPC Software Stack Application Algorithm Data Layout CPU GPU MIC Others HPC Software Stack Deep Learning Algorithm Data Layout CPU GPU MIC Others
More informationS WHAT THE PROFILER IS TELLING YOU: OPTIMIZING GPU KERNELS. Jakob Progsch, Mathias Wagner GTC 2018
S8630 - WHAT THE PROFILER IS TELLING YOU: OPTIMIZING GPU KERNELS Jakob Progsch, Mathias Wagner GTC 2018 1. Know your hardware BEFORE YOU START What are the target machines, how many nodes? Machine-specific
More informationPOWERING THE AI REVOLUTION JENSEN HUANG, FOUNDER & CEO GTC 2017
POWERING THE AI REVOLUTION JENSEN HUANG, FOUNDER & CEO GTC 2017 LIFE AFTER MOORE S LAW 10 7 40 Years of Microprocessor Trend Data 10 6 10 5 Transistors (thousands) 1.1X per year 10 4 10 3 1.5X per year
More informationDeep Learning Accelerators
Deep Learning Accelerators Abhishek Srivastava (as29) Samarth Kulshreshtha (samarth5) University of Illinois, Urbana-Champaign Submitted as a requirement for CS 433 graduate student project Outline Introduction
More informationMachine Learning on VMware vsphere with NVIDIA GPUs
Machine Learning on VMware vsphere with NVIDIA GPUs Uday Kurkure, Hari Sivaraman, Lan Vu GPU Technology Conference 2017 2016 VMware Inc. All rights reserved. Gartner Hype Cycle for Emerging Technology
More informationDemocratizing Machine Learning on Kubernetes
Democratizing Machine Learning on Kubernetes Joy Qiao, Senior Solution Architect - AI and Research Group, Microsoft Lachlan Evenson - Principal Program Manager AKS/ACS, Microsoft Who are we? The Data Scientist
More informationAn introduction to Machine Learning silicon
An introduction to Machine Learning silicon November 28 2017 Insight for Technology Investors AI/ML terminology Artificial Intelligence Machine Learning Deep Learning Algorithms: CNNs, RNNs, etc. Additional
More informationCafeGPI. Single-Sided Communication for Scalable Deep Learning
CafeGPI Single-Sided Communication for Scalable Deep Learning Janis Keuper itwm.fraunhofer.de/ml Competence Center High Performance Computing Fraunhofer ITWM, Kaiserslautern, Germany Deep Neural Networks
More informationDNNBuilder: an Automated Tool for Building High-Performance DNN Hardware Accelerators for FPGAs
IBM Research AI Systems Day DNNBuilder: an Automated Tool for Building High-Performance DNN Hardware Accelerators for FPGAs Xiaofan Zhang 1, Junsong Wang 2, Chao Zhu 2, Yonghua Lin 2, Jinjun Xiong 3, Wen-mei
More informationCaffe tutorial. Seong Joon Oh
Caffe tutorial Seong Joon Oh What is Caffe? Convolution Architecture For Feature Extraction (CAFFE) Open framework, models, and examples for deep learning 600+ citations, 100+ contributors, 7,000+ stars,
More informationOPERATIONALIZING MACHINE LEARNING USING GPU ACCELERATED, IN-DATABASE ANALYTICS
OPERATIONALIZING MACHINE LEARNING USING GPU ACCELERATED, IN-DATABASE ANALYTICS 1 Why GPUs? A Tale of Numbers 100x Performance Increase Infrastructure Cost Savings Performance 100x gains over traditional
More informationTESLA P100 PERFORMANCE GUIDE. HPC and Deep Learning Applications
TESLA P PERFORMANCE GUIDE HPC and Deep Learning Applications MAY 217 TESLA P PERFORMANCE GUIDE Modern high performance computing (HPC) data centers are key to solving some of the world s most important
More informationAutomatic Code Generation TVM Stack
Automatic Code Generation TVM Stack CSE 599W Spring TVM stack is an active project by saml.cs.washington.edu and many partners in the open source community The Gap between Framework and Hardware Frameworks
More informationTENSORRT. RN _v01 June Release Notes
TENSORRT RN-08624-030_v01 June 2018 Release Notes TABLE OF CONTENTS Chapter Chapter Chapter Chapter Chapter Chapter 1. 2. 3. 4. 5. 6. Overview...1 Release 4.0.1... 2 Release 3.0.4... 6 Release 3.0.2...
More informationCloud Computing with FPGA-based NVMe SSDs
Cloud Computing with FPGA-based NVMe SSDs Bharadwaj Pudipeddi, CTO NVXL Santa Clara, CA 1 Choice of NVMe Controllers ASIC NVMe: Fully off-loaded, consistent performance, M.2 or U.2 form factor ASIC OpenChannel:
More informationS8901 Quadro for AI, VR and Simulation
S8901 Quadro for AI, VR and Simulation Carl Flygare, PNY Quadro Product Marketing Manager Allen Bourgoyne, NVIDIA Senior Product Marketing Manager The question of whether a computer can think is no more
More informationExploiting the OpenPOWER Platform for Big Data Analytics and Cognitive. Rajesh Bordawekar and Ruchir Puri IBM T. J. Watson Research Center
Exploiting the OpenPOWER Platform for Big Data Analytics and Cognitive Rajesh Bordawekar and Ruchir Puri IBM T. J. Watson Research Center 3/17/2015 2014 IBM Corporation Outline IBM OpenPower Platform Accelerating
More informationA High-Performing Cloud Begins with a Strong Foundation. A solution guide for IBM Cloud bare metal servers
A High-Performing Cloud Begins with a Strong Foundation A solution guide for IBM Cloud bare metal servers 02 IBM Cloud Bare Metal Servers Bare metal and the bottom line Today s workloads are dynamic and
More informationFast Hardware For AI
Fast Hardware For AI Karl Freund karl@moorinsightsstrategy.com Sr. Analyst, AI and HPC Moor Insights & Strategy Follow my blogs covering Machine Learning Hardware on Forbes: http://www.forbes.com/sites/moorinsights
More informationResearch Faculty Summit Systems Fueling future disruptions
Research Faculty Summit 2018 Systems Fueling future disruptions Wolong: A Back-end Optimizer for Deep Learning Computation Jilong Xue Researcher, Microsoft Research Asia System Challenge in Deep Learning
More informationAI-accelerated HPC Hardware Infrastructure. Francis Lam Huawei Technologies
AI-accelerated HPC Hardware Infrastructure Francis Lam Huawei Technologies Contents Huawei HPC Momentum Boundless Computing AI accelerating HPC HPC accelerating AI www.huawei.com Huawei Confidential 2
More informationSmall is the New Big: Data Analytics on the Edge
Small is the New Big: Data Analytics on the Edge An overview of processors and algorithms for deep learning techniques on the edge Dr. Abhay Samant VP Engineering, Hiller Measurements Adjunct Faculty,
More informationSVM multiclass classification in 10 steps 17/32
SVM multiclass classification in 10 steps import numpy as np # load digits dataset from sklearn import datasets digits = datasets. load_digits () # define training set size n_samples = len ( digits. images
More information