Usable while performant: the challenges building. Soumith Chintala
|
|
|
- Angel Park
- 5 years ago
- Views:
Transcription
1
2 Usable while performant: the challenges building Soumith Chintala
3 Problem Statement Deep Learning Workloads
4 Problem Statement Deep Learning Workloads for epoch in range(max_epochs): for data, target in enumerate(training_data): output = model(data) loss = F.nll_loss(output, target) loss.backward() optimizer.step()
5 Problem Statement Deep Learning Workloads N samples, each of some shape D for epoch in range(max_epochs): for data, target in enumerate(training_data): output = model(data) loss = F.nll_loss(output, target) loss.backward() optimizer.step()
6 Problem Statement Deep Learning Workloads mini-batch of M samples (M << N), each of shape D for epoch in range(max_epochs): for data, target in enumerate(training_data): output = model(data) loss = F.nll_loss(output, target) loss.backward() optimizer.step()
7 Problem Statement Deep Learning Workloads neural network with weights for epoch in range(max_epochs): for data, target in enumerate(training_data): output = model(data) loss = F.nll_loss(output, target) loss.backward() optimizer.step()
8 Problem Statement Deep Learning Workloads backpropagation: compute derivatives wrt loss, using chain rule for epoch in range(max_epochs): for data, target in enumerate(training_data): output = model(data) loss = F.nll_loss(output, target) loss.backward() optimizer.step()
9 Problem Statement Deep Learning Workloads update weights using the computed gradients for epoch in range(max_epochs): for data, target in enumerate(training_data): output = model(data) loss = F.nll_loss(output, target) loss.backward() optimizer.step()
10 Problem Statement Deep Learning Workloads for epoch in range(max_epochs): for data, target in enumerate(training_data): output = model(data) loss = F.nll_loss(output, target) loss.backward() optimizer.step()
11 Problem Statement Deep Learning Workloads neural network with weights for epoch in range(max_epochs): for data, target in enumerate(training_data): output = model(data) loss = F.nll_loss(output, target) loss.backward() optimizer.step()
12 Types of typical operators Convolution Figure by Vincent Dumolin:
13 Types of typical operators Convolution Figure by Vincent Dumolin:
14 Types of typical operators Convolution for oc in output_channel: for ic in input_channel: for h in output_height: for w in output_width: for kh in kernel_height: for kw in kernel_width: output_pixel += input_pixel * kernel_value Figure by Vincent Dumolin:
15 Types of typical operators Figure by Vincent Dumolin:
16 Types of typical operators Matrix Multiply Figure by Wikipedia:
17 Types of typical operators Pointwise operations for (i=0; i < data_length; i++) { } output[i] = input1[i] + input2[i] Figure by Wikipedia:
18 Types of typical operators Reduction operations double sum = 0.0; for (i=0; i < data_length; i++) { sum += input[i]; } Figure by Wikipedia:
19 Chained Together Input Output Figure by Wikipedia:
20 Chained Together Input Output Figure by Wikipedia:
21 Chained Together Figure by Wikipedia: Input Output "deep"
22 Chained Together Figure by Wikipedia: Input Output "deep" recurrent
23 Trained with Gradient Descent Figure by Wikipedia: Input Output "deep" recurrent
24 Problem Statement Deep Learning Workloads an easy way to see recurrence for epoch in range(max_epochs): for data, target in enumerate(training_data): output = model(data) loss = F.nll_loss(output, target) loss.backward() optimizer.step()
25 Problem Statement Deep Learning Workloads an easy way to see recurrence for epoch in range(max_epochs): for data, target in enumerate(training_data): output, hidden = [], zeros() for t in data.size(0): out, hidden = model(data[t], hidden) output.append(out) loss = F.nll_loss(output, target) loss.backward() optimizer.step()
26 Problem Statement Deep Learning Workloads - Vision models - model is very deep, straight-line chain with no recurrence - lots of convolutions - typically run on GPUs
27 Problem Statement Deep Learning Workloads - Vision models - model is very deep, straight-line chain with no recurrence - lots of convolutions - typically run on GPUs - NLP models - LSTM-RNN - model is 1 to 4 "layers" deep - two matmuls across space and time along with pointwise ops -typically run on CPUs if small, GPUs if large
28 Deep Learning Frameworks Make this easy to program for epoch in range(max_epochs): for data, target in enumerate(training_data): output = model(data) loss = F.nll_loss(output, target) loss.backward() optimizer.step()
29 Pre-PyTorch meta programming meta programming imperative
30 Caffe
31 Caffe define protobuf, run via command-line utility
32 Caffe define protobuf, run via command-line utility small, efficient library. Could do convents well.
33 Theano
34 Theano meta-program Theano VM via Python API
35 Theano meta-program Theano VM via Python API whole program optimizations, graph fusion
36 Theano meta-program Theano VM via Python API whole program optimizations, graph fusion graphs took minutes to hours to compile and start
37 Torch-7
38 Torch-7 imperative programming in Lua
39 Torch-7 imperative programming in Lua tied closely to underlying C89 implementations
40 Torch-7 imperative programming in Lua tied closely to underlying C89 implementations Lua lacked good tooling and ecosystem
41 What is PyTorch? automatic differentiation Ndarray library gradient based Utilities engine with GPU support optimization package (data loading, etc.) Deep Learning Numpy-alternative Reinforcement Learning
42 ndarray library np.ndarray <-> torch.tensor 200+ operations, similar to numpy very fast acceleration on NVIDIA GPUs
43 ndarray library Numpy PyTorch
44 ndarray / Tensor library
45 ndarray / Tensor library
46 ndarray / Tensor library
47 ndarray / Tensor library
48 NumPy bridge
49 NumPy bridge Zero memory-copy very efficient
50 NumPy bridge
51 NumPy bridge
52 Seamless GPU Tensors
53 automatic differentiation engine for deep learning and reinforcement learning
54 PyTorch Autograd W_h = torch.randn(20, 20, requires_grad=true) W_x = torch.randn(20, 10, requires_grad=true) x = torch.randn(1, 10) prev_h = torch.randn(1, 20)
55 PyTorch Autograd W_h = torch.randn(20, 20, requires_grad=true) W_x = torch.randn(20, 10, requires_grad=true) x = torch.randn(1, 10) prev_h = torch.randn(1, 20) MM MM i2h = torch.mm(w_x, x.t()) h2h = torch.mm(w_h, prev_h.t())
56 PyTorch Autograd W_h = torch.randn(20, 20, requires_grad=true) W_x = torch.randn(20, 10, requires_grad=true) x = torch.randn(1, 10) prev_h = torch.randn(1, 20) MM MM i2h = torch.mm(w_x, x.t()) h2h = torch.mm(w_h, prev_h.t()) next_h = i2h + h2h
57 PyTorch Autograd W_h = torch.randn(20, 20, requires_grad=true) W_x = torch.randn(20, 10, requires_grad=true) x = torch.randn(1, 10) prev_h = torch.randn(1, 20) MM MM i2h = torch.mm(w_x, x.t()) h2h = torch.mm(w_h, prev_h.t()) next_h = i2h + h2h Add
58 PyTorch Autograd W_h = torch.randn(20, 20, requires_grad=true) W_x = torch.randn(20, 10, requires_grad=true) x = torch.randn(1, 10) prev_h = torch.randn(1, 20) MM MM i2h = torch.mm(w_x, x.t()) h2h = torch.mm(w_h, prev_h.t()) next_h = i2h + h2h next_h = next_h.tanh() Add Tanh
59 PyTorch Autograd W_h = torch.randn(20, 20, requires_grad=true) W_x = torch.randn(20, 10, requires_grad=true) x = torch.randn(1, 10) prev_h = torch.randn(1, 20) MM MM i2h = torch.mm(w_x, x.t()) h2h = torch.mm(w_h, prev_h.t()) next_h = i2h + h2h next_h = next_h.tanh() Add Tanh next_h.backward(torch.ones(1, 20))
60 Neural Networks
61 Neural Networks
62 Neural Networks
63 Optimization package SGD, Adagrad, RMSProp, LBFGS, etc.
64 Bootstrapping Writing Building models Implementing Checkpointing Dataset loaders Training loop models Python + PyTorch - an environment to do all of this Interfacing with Dealing with Building Building optimizers environments GPUs Baselines
65 Bootstrapping Writing Building models Implementing Checkpointing Dataset loaders Training loop models bootstrapping Python + PyTorch the Python - an environment tooling stack to do for all good of this UX Interfacing with Dealing with Building Building optimizers environments GPUs Baselines
66 Python is slow, interpreted Global interpreter-lock application logic is order of magnitude slower than C++ moved autograd engine to C++ moved everything to ATen - Side-effect, a clean C++ API
Adam Paszke, Sam Gross, Soumith Chintala, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia
Adam Paszke, Sam Gross, Soumith Chintala, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Alban Desmaison, Andreas Kopf, Edward Yang, Zach Devito,
Frameworks in Python for Numeric Computation / ML
Frameworks in Python for Numeric Computation / ML Why use a framework? Why not use the built-in data structures? Why not write our own matrix multiplication function? Frameworks are needed not only because
NVIDIA GPU CLOUD DEEP LEARNING FRAMEWORKS
TECHNICAL OVERVIEW NVIDIA GPU CLOUD DEEP LEARNING FRAMEWORKS A Guide to the Optimized Framework Containers on NVIDIA GPU Cloud Introduction Artificial intelligence is helping to solve some of the most
TensorFlow: A System for Learning-Scale Machine Learning. Google Brain
TensorFlow: A System for Learning-Scale Machine Learning Google Brain The Problem Machine learning is everywhere This is in large part due to: 1. Invention of more sophisticated machine learning models
DEEP LEARNING IN PYTHON. The need for optimization
DEEP LEARNING IN PYTHON The need for optimization A baseline neural network Input 2 Hidden Layer 5 2 Output - 9-3 Actual Value of Target: 3 Error: Actual - Predicted = 4 A baseline neural network Input
Tutorial on Keras CAP ADVANCED COMPUTER VISION SPRING 2018 KISHAN S ATHREY
Tutorial on Keras CAP 6412 - ADVANCED COMPUTER VISION SPRING 2018 KISHAN S ATHREY Deep learning packages TensorFlow Google PyTorch Facebook AI research Keras Francois Chollet (now at Google) Chainer Company
Keras: Handwritten Digit Recognition using MNIST Dataset
Keras: Handwritten Digit Recognition using MNIST Dataset IIT PATNA January 31, 2018 1 / 30 OUTLINE 1 Keras: Introduction 2 Installing Keras 3 Keras: Building, Testing, Improving A Simple Network 2 / 30
Deep Learning with Torch
Deep Learning with Torch The good, the bad, the ugly since 2002 Jimmy Ba jimmy@psi.utoronto.ca What is Torch? Year 2012 Google Answer: Torch7 provides a Matlab-like environment for state-of-the-art machine
A Quick Guide on Training a neural network using Keras.
A Quick Guide on Training a neural network using Keras. TensorFlow and Keras Keras Open source High level, less flexible Easy to learn Perfect for quick implementations Starts by François Chollet from
Natural Language Processing with Deep Learning CS224N/Ling284. Christopher Manning Lecture 4: Backpropagation and computation graphs
Natural Language Processing with Deep Learning CS4N/Ling84 Christopher Manning Lecture 4: Backpropagation and computation graphs Lecture Plan Lecture 4: Backpropagation and computation graphs 1. Matrix
MIXED PRECISION TRAINING OF NEURAL NETWORKS. Carl Case, Senior Architect, NVIDIA
MIXED PRECISION TRAINING OF NEURAL NETWORKS Carl Case, Senior Architect, NVIDIA OUTLINE 1. What is mixed precision training with FP16? 2. Considerations and methodology for mixed precision training 3.
MoonRiver: Deep Neural Network in C++
MoonRiver: Deep Neural Network in C++ Chung-Yi Weng Computer Science & Engineering University of Washington chungyi@cs.washington.edu Abstract Artificial intelligence resurges with its dramatic improvement
Gradient Descent Optimization Algorithms for Deep Learning Batch gradient descent Stochastic gradient descent Mini-batch gradient descent
Gradient Descent Optimization Algorithms for Deep Learning Batch gradient descent Stochastic gradient descent Mini-batch gradient descent Slide credit: http://sebastianruder.com/optimizing-gradient-descent/index.html#batchgradientdescent
Deep Learning Benchmarks Mumtaz Vauhkonen, Quaizar Vohra, Saurabh Madaan Collaboration with Adam Coates, Stanford Unviersity
Deep Learning Benchmarks Mumtaz Vauhkonen, Quaizar Vohra, Saurabh Madaan Collaboration with Adam Coates, Stanford Unviersity Abstract: This project aims at creating a benchmark for Deep Learning (DL) algorithms
autograd tutorial Paul Vicol, Slides Based on Ryan Adams January 30, 2017 CSC 321, University of Toronto
autograd tutorial Paul Vicol, Slides Based on Ryan Adams January 30, 2017 CSC 321, University of Toronto 1 tutorial outline 1. Automatic Differentiation 2. Introduction to Autograd 3. IPython Notebook
ImageNet Classification with Deep Convolutional Neural Networks
ImageNet Classification with Deep Convolutional Neural Networks Alex Krizhevsky Ilya Sutskever Geoffrey Hinton University of Toronto Canada Paper with same name to appear in NIPS 2012 Main idea Architecture
RNN LSTM and Deep Learning Libraries
RNN LSTM and Deep Learning Libraries UDRC Summer School Muhammad Awais m.a.rana@surrey.ac.uk Outline Recurrent Neural Network Application of RNN LSTM Caffe Torch Theano TensorFlow Flexibility of Recurrent
Application of Deep Learning Techniques in Satellite Telemetry Analysis.
Application of Deep Learning Techniques in Satellite Telemetry Analysis. Greg Adamski, Member of Technical Staff L3 Technologies Telemetry and RF Products Julian Spencer Jones, Spacecraft Engineer Telenor
NVIDIA DLI HANDS-ON TRAINING COURSE CATALOG
NVIDIA DLI HANDS-ON TRAINING COURSE CATALOG Valid Through July 31, 2018 INTRODUCTION The NVIDIA Deep Learning Institute (DLI) trains developers, data scientists, and researchers on how to use artificial
MIXED PRECISION TRAINING: THEORY AND PRACTICE Paulius Micikevicius
MIXED PRECISION TRAINING: THEORY AND PRACTICE Paulius Micikevicius What is Mixed Precision Training? Reduced precision tensor math with FP32 accumulation, FP16 storage Successfully used to train a variety
Machine Learning With Python. Bin Chen Nov. 7, 2017 Research Computing Center
Machine Learning With Python Bin Chen Nov. 7, 2017 Research Computing Center Outline Introduction to Machine Learning (ML) Introduction to Neural Network (NN) Introduction to Deep Learning NN Introduction
Neural Network Neurons
Neural Networks Neural Network Neurons 1 Receives n inputs (plus a bias term) Multiplies each input by its weight Applies activation function to the sum of results Outputs result Activation Functions Given
Research Faculty Summit Systems Fueling future disruptions
Research Faculty Summit 2018 Systems Fueling future disruptions Wolong: A Back-end Optimizer for Deep Learning Computation Jilong Xue Researcher, Microsoft Research Asia System Challenge in Deep Learning
Index. Umberto Michelucci 2018 U. Michelucci, Applied Deep Learning,
A Acquisition function, 298, 301 Adam optimizer, 175 178 Anaconda navigator conda command, 3 Create button, 5 download and install, 1 installing packages, 8 Jupyter Notebook, 11 13 left navigation pane,
Convolutional Networks for Text
CS11-747 Neural Networks for NLP Convolutional Networks for Text Graham Neubig Site https://phontron.com/class/nn4nlp2017/ An Example Prediction Problem: Sentence Classification I hate this movie very
Deep neural networks
Deep neural networks Outline What s new in ANNs in the last 5-10 years? Deeper networks, more data, and faster training Scalability and use of GPUs Symbolic differentiation reverse-mode automatic differentiation
Training Deep Neural Networks (in parallel)
Lecture 9: Training Deep Neural Networks (in parallel) Visual Computing Systems How would you describe this professor? Easy? Mean? Boring? Nerdy? Professor classification task Classifies professors as
Deep Neural Networks Optimization
Deep Neural Networks Optimization Creative Commons (cc) by Akritasa http://arxiv.org/pdf/1406.2572.pdf Slides from Geoffrey Hinton CSC411/2515: Machine Learning and Data Mining, Winter 2018 Michael Guerzhoy
CS281 Section 3: Practical Optimization
CS281 Section 3: Practical Optimization David Duvenaud and Dougal Maclaurin Most parameter estimation problems in machine learning cannot be solved in closed form, so we often have to resort to numerical
Neural Network Exchange Format
Copyright Khronos Group 2017 - Page 1 Neural Network Exchange Format Deploying Trained Networks to Inference Engines Viktor Gyenes, specification editor Copyright Khronos Group 2017 - Page 2 Outlook The
Characterization and Benchmarking of Deep Learning. Natalia Vassilieva, PhD Sr. Research Manager
Characterization and Benchmarking of Deep Learning Natalia Vassilieva, PhD Sr. Research Manager Deep learning applications Vision Speech Text Other Search & information extraction Security/Video surveillance
Lecture 2 Notes. Outline. Neural Networks. The Big Idea. Architecture. Instructors: Parth Shah, Riju Pahwa
Instructors: Parth Shah, Riju Pahwa Lecture 2 Notes Outline 1. Neural Networks The Big Idea Architecture SGD and Backpropagation 2. Convolutional Neural Networks Intuition Architecture 3. Recurrent Neural
Day 1 Lecture 6. Software Frameworks for Deep Learning
Day 1 Lecture 6 Software Frameworks for Deep Learning Packages Caffe Theano NVIDIA Digits Lasagne Keras Blocks Torch TensorFlow MxNet MatConvNet Nervana Neon Leaf Caffe Deep learning framework from Berkeley
Optimizing CNN Inference on CPUs
Optimizing CNN Inference on CPUs Yizhi Liu, Yao Wang, Yida Wang With others in AWS AI Agenda Deep learning inference optimization Optimization on Intel CPUs Evaluation Make DL inference easier and faster
Machine Learning Workshop
Machine Learning Workshop {Presenters} Feb. 20th, 2018 Theory of Neural Networks Architecture and Types of Layers: Fully Connected (FC) Convolutional Neural Network (CNN) Pooling Drop out Residual Recurrent
Parallel Deep Network Training
Lecture 19: Parallel Deep Network Training Parallel Computer Architecture and Programming How would you describe this professor? Easy? Mean? Boring? Nerdy? Professor classification task Classifies professors
SEMANTIC COMPUTING. Lecture 9: Deep Learning: Recurrent Neural Networks (RNNs) TU Dresden, 21 December 2018
SEMANTIC COMPUTING Lecture 9: Deep Learning: Recurrent Neural Networks (RNNs) Dagmar Gromann International Center For Computational Logic TU Dresden, 21 December 2018 Overview Handling Overfitting Recurrent
CS489/698: Intro to ML
CS489/698: Intro to ML Lecture 14: Training of Deep NNs Instructor: Sun Sun 1 Outline Activation functions Regularization Gradient-based optimization 2 Examples of activation functions 3 5/28/18 Sun Sun
Recurrent Neural Networks and Transfer Learning for Action Recognition
Recurrent Neural Networks and Transfer Learning for Action Recognition Andrew Giel Stanford University agiel@stanford.edu Ryan Diaz Stanford University ryandiaz@stanford.edu Abstract We have taken on the
Deep Learning Frameworks. COSC 7336: Advanced Natural Language Processing Fall 2017
Deep Learning Frameworks COSC 7336: Advanced Natural Language Processing Fall 2017 Today s lecture Deep learning software overview TensorFlow Keras Practical Graphical Processing Unit (GPU) From graphical
CNN Basics. Chongruo Wu
CNN Basics Chongruo Wu Overview 1. 2. 3. Forward: compute the output of each layer Back propagation: compute gradient Updating: update the parameters with computed gradient Agenda 1. Forward Conv, Fully
Encoding RNNs, 48 End of sentence (EOS) token, 207 Exploding gradient, 131 Exponential function, 42 Exponential Linear Unit (ELU), 44
A Activation potential, 40 Annotated corpus add padding, 162 check versions, 158 create checkpoints, 164, 166 create input, 160 create train and validation datasets, 163 dropout, 163 DRUG-AE.rel file,
Scaling Distributed Machine Learning
Scaling Distributed Machine Learning with System and Algorithm Co-design Mu Li Thesis Defense CSD, CMU Feb 2nd, 2017 nx min w f i (w) Distributed systems i=1 Large scale optimization methods Large-scale
Deep Learning. Practical introduction with Keras JORDI TORRES 27/05/2018. Chapter 3 JORDI TORRES
Deep Learning Practical introduction with Keras Chapter 3 27/05/2018 Neuron A neural network is formed by neurons connected to each other; in turn, each connection of one neural network is associated
Unified Deep Learning with CPU, GPU, and FPGA Technologies
Unified Deep Learning with CPU, GPU, and FPGA Technologies Allen Rush 1, Ashish Sirasao 2, Mike Ignatowski 1 1: Advanced Micro Devices, Inc., 2: Xilinx, Inc. Abstract Deep learning and complex machine
Keras: Handwritten Digit Recognition using MNIST Dataset
Keras: Handwritten Digit Recognition using MNIST Dataset IIT PATNA February 9, 2017 1 / 24 OUTLINE 1 Introduction Keras: Deep Learning library for Theano and TensorFlow 2 Installing Keras Installation
DEEP LEARNING REVIEW. Yann LeCun, Yoshua Bengio & Geoffrey Hinton Nature Presented by Divya Chitimalla
DEEP LEARNING REVIEW Yann LeCun, Yoshua Bengio & Geoffrey Hinton Nature 2015 -Presented by Divya Chitimalla What is deep learning Deep learning allows computational models that are composed of multiple
NVIDIA DEEP LEARNING INSTITUTE
NVIDIA DEEP LEARNING INSTITUTE TRAINING CATALOG Valid Through July 31, 2018 INTRODUCTION The NVIDIA Deep Learning Institute (DLI) trains developers, data scientists, and researchers on how to use artificial
A Torch Library for Action Recognition and Detection Using CNNs and LSTMs
A Torch Library for Action Recognition and Detection Using CNNs and LSTMs Gary Thung and Helen Jiang Stanford University {gthung, helennn}@stanford.edu Abstract It is very common now to see deep neural
Hardware and Software. Fei-Fei Li & Justin Johnson & Serena Yeung. Lecture 6-1
Lecture 6: Hardware and Software Lecture 6-1 Administrative Assignment 1 was due yesterday. Assignment 2 is out, due Wed May 1. Project proposal due Wed April 24. Project-only office hours leading up to
The Anatomy of Deep Learning Frameworks* *Everything you wanted to know about DL Frameworks but were afraid to ask
The Anatomy of Deep Learning Frameworks* *Everything you wanted to know about DL Frameworks but were afraid to ask $whoami Master s Student in CS @ ETH Zurich, bachelors from BITS Pilani Contributor to
Getting started with Caffe. Jon Barker, Solutions Architect
Getting started with Caffe Jon Barker, Solutions Architect Caffe tour Overview Agenda Example applications Setup Performance Hands-on lab preview 2 A tour of Caffe 3 What is Caffe? An open framework for
Machine Learning on VMware vsphere with NVIDIA GPUs
Machine Learning on VMware vsphere with NVIDIA GPUs Uday Kurkure, Hari Sivaraman, Lan Vu GPU Technology Conference 2017 2016 VMware Inc. All rights reserved. Gartner Hype Cycle for Emerging Technology
Code Mania Artificial Intelligence: a. Module - 1: Introduction to Artificial intelligence and Python:
Code Mania 2019 Artificial Intelligence: a. Module - 1: Introduction to Artificial intelligence and Python: 1. Introduction to Artificial Intelligence 2. Introduction to python programming and Environment
Deep Learning Applications
October 20, 2017 Overview Supervised Learning Feedforward neural network Convolution neural network Recurrent neural network Recursive neural network (Recursive neural tensor network) Unsupervised Learning
Deep Learning Compiler
Deep Learning Compiler AWS AI Acknowledgement Amazon Sagemaker Neo Enables developers to train machine learning models once and run them anywhere in the cloud and at the edge Hardware targets Intel CPU,
Matrix Computations and " Neural Networks in Spark
Matrix Computations and " Neural Networks in Spark Reza Zadeh Paper: http://arxiv.org/abs/1509.02256 Joint work with many folks on paper. @Reza_Zadeh http://reza-zadeh.com Training Neural Networks Datasets
Machine Learning 13. week
Machine Learning 13. week Deep Learning Convolutional Neural Network Recurrent Neural Network 1 Why Deep Learning is so Popular? 1. Increase in the amount of data Thanks to the Internet, huge amount of
An Introduction to NNs using Keras
An Introduction to NNs using Keras Michela Paganini michela.paganini@cern.ch Yale University 1 Keras Modular, powerful and intuitive Deep Learning python library built on Theano and TensorFlow Minimalist,
EECS 496 Statistical Language Models. Winter 2018
EECS 496 Statistical Language Models Winter 2018 Introductions Professor: Doug Downey Course web site: www.cs.northwestern.edu/~ddowney/courses/496_winter2018 (linked off prof. home page) Logistics Grading
Training Neural Networks with Mixed Precision MICHAEL CARILLI CHRISTIAN SAROFEEN MICHAEL RUBERRY BEN BARSDELL
Training Neural Networks with Mixed Precision MICHAEL CARILLI CHRISTIAN SAROFEEN MICHAEL RUBERRY BEN BARSDELL 1 THIS TALK Using mixed precision and Volta your networks can be: 1. 2-4x faster 2. half the
PLT: Inception (cuz there are so many layers)
PLT: Inception (cuz there are so many layers) By: Andrew Aday, (aza2112) Amol Kapoor (ajk2227), Jonathan Zhang (jz2814) Proposal Abstract Overview of domain Purpose Language Outline Types Operators Syntax
LSTM: An Image Classification Model Based on Fashion-MNIST Dataset
LSTM: An Image Classification Model Based on Fashion-MNIST Dataset Kexin Zhang, Research School of Computer Science, Australian National University Kexin Zhang, U6342657@anu.edu.au Abstract. The application
Operator Vectorization Library A TensorFlow Plugin
Operator Vectorization Library A TensorFlow Plugin Matthew Pickett, Karen Brems, Florian Raudies Hewlett Packard Labs HPE-2016-94 Keyword(s): Machine learning; GPU acceleration; TensorFlow Abstract: TensorFlow
Deep neural networks II
Deep neural networks II May 31 st, 2018 Yong Jae Lee UC Davis Many slides from Rob Fergus, Svetlana Lazebnik, Jia-Bin Huang, Derek Hoiem, Adriana Kovashka, Why (convolutional) neural networks? State of
LSTM and its variants for visual recognition. Xiaodan Liang Sun Yat-sen University
LSTM and its variants for visual recognition Xiaodan Liang xdliang328@gmail.com Sun Yat-sen University Outline Context Modelling with CNN LSTM and its Variants LSTM Architecture Variants Application in
Deep Learning for Computer Vision II
IIIT Hyderabad Deep Learning for Computer Vision II C. V. Jawahar Paradigm Shift Feature Extraction (SIFT, HoG, ) Part Models / Encoding Classifier Sparrow Feature Learning Classifier Sparrow L 1 L 2 L
Sentiment Classification of Food Reviews
Sentiment Classification of Food Reviews Hua Feng Department of Electrical Engineering Stanford University Stanford, CA 94305 fengh15@stanford.edu Ruixi Lin Department of Electrical Engineering Stanford
Deep Learning for Visual Computing Prof. Debdoot Sheet Department of Electrical Engineering Indian Institute of Technology, Kharagpur
Deep Learning for Visual Computing Prof. Debdoot Sheet Department of Electrical Engineering Indian Institute of Technology, Kharagpur Lecture - 05 Classification with Perceptron Model So, welcome to today
COMP 551 Applied Machine Learning Lecture 14: Neural Networks
COMP 551 Applied Machine Learning Lecture 14: Neural Networks Instructor: (jpineau@cs.mcgill.ca) Class web page: www.cs.mcgill.ca/~jpineau/comp551 Unless otherwise noted, all material posted for this course
Tutorial on Deep Learning with Theano and Lasagne. Jan Schlüter Sander Dieleman EMBL-EBI
Tutorial on Deep Learning with Theano and Lasagne Jan Schlüter Sander Dieleman EMBL-EBI 2016-04-21 Tutorial Outline Foundations What are Artificial Neural Networks? Practical part: Theano and Lasagne Learning
Deep Learning with R. Francesca Lazzeri Data Scientist II - Microsoft, AI Research
with R Francesca Lazzeri - @frlazzeri Data Scientist II - Microsoft, AI Research Agenda with R What is Demo Better understanding of R DL tools Fundamental concepts in Forward Propagation Algorithm Activation
Matchbox. automatic batching for imperative deep learning NVIDIA GTC, 2018/3/28. James Bradbury
Matchbox automatic batching for imperative deep learning James Bradbury NVIDIA GTC, 2018/3/28 Roadmap Imperative deep learning How manual batching works Other tools for automatic batching The Matchbox
OPTIMIZED GPU KERNELS FOR DEEP LEARNING. Amir Khosrowshahi
OPTIMIZED GPU KERNELS FOR DEEP LEARNING Amir Khosrowshahi GTC 17 Mar 2015 Outline About nervana Optimizing deep learning at assembler level Limited precision for deep learning neon benchmarks 2 About nervana
CSC 578 Neural Networks and Deep Learning
CSC 578 Neural Networks and Deep Learning Fall 2018/19 7. Recurrent Neural Networks (Some figures adapted from NNDL book) 1 Recurrent Neural Networks 1. Recurrent Neural Networks (RNNs) 2. RNN Training
A Simple (?) Exercise: Predicting the Next Word
CS11-747 Neural Networks for NLP A Simple (?) Exercise: Predicting the Next Word Graham Neubig Site https://phontron.com/class/nn4nlp2017/ Are These Sentences OK? Jane went to the store. store to Jane
Parallel Deep Network Training
Lecture 26: Parallel Deep Network Training Parallel Computer Architecture and Programming CMU 15-418/15-618, Spring 2016 Tunes Speech Debelle Finish This Album (Speech Therapy) Eat your veggies and study
National College of Ireland Project Submission Sheet 2015/2016 School of Computing
National College of Ireland Project Submission Sheet 2015/2016 School of Computing Student Name: Sean McNally Student ID: 15021581 Programme: MSc Data Analytics Year: 2015-2016 Module: Supervisor: Configuration
Caffe tutorial. Seong Joon Oh
Caffe tutorial Seong Joon Oh What is Caffe? Convolution Architecture For Feature Extraction (CAFFE) Open framework, models, and examples for deep learning 600+ citations, 100+ contributors, 7,000+ stars,
Frameworks. Advanced Machine Learning for NLP Jordan Boyd-Graber INTRODUCTION. Slides adapted from Chris Dyer, Yoav Goldberg, Graham Neubig
Frameworks Advanced Machine Learning for NLP Jordan Boyd-Graber INTRODUCTION Slides adapted from Chris Dyer, Yoav Goldberg, Graham Neubig Advanced Machine Learning for NLP Boyd-Graber Frameworks 1 of 10
Perceptron: This is convolution!
Perceptron: This is convolution! v v v Shared weights v Filter = local perceptron. Also called kernel. By pooling responses at different locations, we gain robustness to the exact spatial location of image
Recurrent Neural Networks. Deep neural networks have enabled major advances in machine learning and AI. Convolutional Neural Networks
Deep neural networks have enabled major advances in machine learning and AI Computer vision Language translation Speech recognition Question answering And more Problem: DNNs are challenging to serve and
Asynchronous Parallel Learning for Neural Networks and Structured Models with Dense Features
Asynchronous Parallel Learning for Neural Networks and Structured Models with Dense Features Xu SUN ( 孙栩 ) Peking University xusun@pku.edu.cn Motivation Neural networks -> Good Performance CNN, RNN, LSTM
Convolutional Sequence to Sequence Learning. Denis Yarats with Jonas Gehring, Michael Auli, David Grangier, Yann Dauphin Facebook AI Research
Convolutional Sequence to Sequence Learning Denis Yarats with Jonas Gehring, Michael Auli, David Grangier, Yann Dauphin Facebook AI Research Sequence generation Need to model a conditional distribution
Modeling Sequences Conditioned on Context with RNNs
Modeling Sequences Conditioned on Context with RNNs Sargur Srihari srihari@buffalo.edu This is part of lecture slides on Deep Learning: http://www.cedar.buffalo.edu/~srihari/cse676 1 10. Topics in Sequence
Deep Learning with Tensorflow AlexNet
Machine Learning and Computer Vision Group Deep Learning with Tensorflow http://cvml.ist.ac.at/courses/dlwt_w17/ AlexNet Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton, "Imagenet classification
Administrative. Assignment 1 due Wednesday April 18, 11:59pm
Lecture 4-1 Administrative Assignment 1 due Wednesday April 18, 11:59pm Lecture 4-2 Administrative All office hours this week will use queuestatus Lecture 4-3 Where we are... scores function SVM loss data
Sequence Modeling: Recurrent and Recursive Nets. By Pyry Takala 14 Oct 2015
Sequence Modeling: Recurrent and Recursive Nets By Pyry Takala 14 Oct 2015 Agenda Why Recurrent neural networks? Anatomy and basic training of an RNN (10.2, 10.2.1) Properties of RNNs (10.2.2, 8.2.6) Using
Introducing CURRENNT: The Munich Open-Source CUDA RecurREnt Neural Network Toolkit
Journal of Machine Learning Research 6 205) 547-55 Submitted 7/3; Published 3/5 Introducing CURRENNT: The Munich Open-Source CUDA RecurREnt Neural Network Toolkit Felix Weninger weninger@tum.de Johannes
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung Chung, Caglar Gulcehre, KyungHyun Cho and Yoshua Bengio Presenter: Yu-Wei Lin Background: Recurrent Neural
Can Active Memory Replace Attention?
Google Brain NIPS 2016 Presenter: Chao Jiang NIPS 2016 Presenter: Chao Jiang 1 / Outline 1 Introduction 2 Active Memory 3 Step by Step to Neural GPU 4 Another two steps: 1. the Markovian Neural GPU 5 Another
DEEP NEURAL NETWORKS AND GPUS. Julie Bernauer
DEEP NEURAL NETWORKS AND GPUS Julie Bernauer GPU Computing GPU Computing Run Computations on GPUs x86 CUDA Framework to Program NVIDIA GPUs A simple sum of two vectors (arrays) in C void vector_add(int
LBANN: Livermore Big Artificial Neural Network HPC Toolkit
LBANN: Livermore Big Artificial Neural Network HPC Toolkit Brian Van Essen, Hyojin Kim, Roger Pearce, Kofi Boakye, Barry Chen Lawrence Livermore National Laboratory MLHPC 2015 Presented by: Sourav Chakraborty
Mocha.jl. Deep Learning in Julia. Chiyuan Zhang CSAIL, MIT
Mocha.jl Deep Learning in Julia Chiyuan Zhang (@pluskid) CSAIL, MIT Deep Learning Learning with multi-layer (3~30) neural networks, on a huge training set. State-of-the-art on many AI tasks Computer Vision:
Neural Networks. CE-725: Statistical Pattern Recognition Sharif University of Technology Spring Soleymani
Neural Networks CE-725: Statistical Pattern Recognition Sharif University of Technology Spring 2013 Soleymani Outline Biological and artificial neural networks Feed-forward neural networks Single layer
Profiling DNN Workloads on a Volta-based DGX-1 System
Profiling DNN Workloads on a Volta-based DGX-1 System Saiful A. Mojumder 1, Marcia S Louis 1, Yifan Sun 2, Amir Kavyan Ziabari 3, José L. Abellán 4, John Kim 5, David Kaeli 2, Ajay Joshi 1 1 ECE Department,
Lecture 20: Neural Networks for NLP. Zubin Pahuja
Lecture 20: Neural Networks for NLP Zubin Pahuja zpahuja2@illinois.edu courses.engr.illinois.edu/cs447 CS447: Natural Language Processing 1 Today s Lecture Feed-forward neural networks as classifiers simple
Advanced Video Analysis & Imaging
Advanced Video Analysis & Imaging (5LSH0), Module 09B Machine Learning with Convolutional Neural Networks (CNNs) - Workout Farhad G. Zanjani, Clint Sebastian, Egor Bondarev, Peter H.N. de With ( p.h.n.de.with@tue.nl
Deep Learning on Modern Architectures. Keren Zhou 4/17/2017
Deep Learning on Modern Architectures Keren Zhou 4/17/2017 HPC Software Stack Application Algorithm Data Layout CPU GPU MIC Others HPC Software Stack Deep Learning Algorithm Data Layout CPU GPU MIC Others
TENSORRT. RN _v01 January Release Notes
TENSORRT RN-08624-030_v01 January 2018 Release Notes TABLE OF CONTENTS Chapter Chapter Chapter Chapter 1. 2. 3. 4. Overview...1 Release 3.0.2... 2 Release 3.0.1... 4 Release 2.1... 10 RN-08624-030_v01