Large-Scale Evolution of Image Classifiers
|
|
|
- Emerald Pitts
- 6 years ago
- Views:
Transcription
1 Large-Scale Evolution of Image Classifiers Esteban Real 1 Sherry Moore 1 Andrew Selle 1 Saurabh Saxena 1 Yutaka Leon Suematsu 2 Jie Tan 1 Quoc V.Le 1 Alexey Kurakin 1 1 Google Brain 2 Google Research ICML, 2017 Presenter: Tianlu Wang ICML, 2017Presenter: Tianlu Wang 1 / Esteban Real, Sherry Moore, Andrew Selle, Saurabh Large-Scale Saxena, Evolution Yutaka Leon of Image Suematsu, Classifiers Jie Tan, Quoc V.Le, Alexey Kurakin (Google Brain 26
2 Outline 1 Introduction Motivation Backgrounds 2 Related Work Neuro-evolution Non-evolutionary 3 Methods Algorithm Overview Encoding and Mutations More Details 4 Results Progress of experiments Comparisons Meta-parameters 5 Summary ICML, 2017Presenter: Tianlu Wang 2 / Esteban Real, Sherry Moore, Andrew Selle, Saurabh Large-Scale Saxena, Evolution Yutaka Leon of Image Suematsu, Classifiers Jie Tan, Quoc V.Le, Alexey Kurakin (Google Brain 26
3 Outline 1 Introduction Motivation Backgrounds 2 Related Work Neuro-evolution Non-evolutionary 3 Methods Algorithm Overview Encoding and Mutations More Details 4 Results Progress of experiments Comparisons Meta-parameters 5 Summary ICML, 2017Presenter: Tianlu Wang 3 / Esteban Real, Sherry Moore, Andrew Selle, Saurabh Large-Scale Saxena, Evolution Yutaka Leon of Image Suematsu, Classifiers Jie Tan, Quoc V.Le, Alexey Kurakin (Google Brain 26
4 Motivation AlexNet, GoogleNet, VGG, ResNet... Designing neural network architectures can be challenging Discover network architectures automatically ICML, 2017Presenter: Tianlu Wang 4 / Esteban Real, Sherry Moore, Andrew Selle, Saurabh Large-Scale Saxena, Evolution Yutaka Leon of Image Suematsu, Classifiers Jie Tan, Quoc V.Le, Alexey Kurakin (Google Brain 26
5 Outline 1 Introduction Motivation Backgrounds 2 Related Work Neuro-evolution Non-evolutionary 3 Methods Algorithm Overview Encoding and Mutations More Details 4 Results Progress of experiments Comparisons Meta-parameters 5 Summary ICML, 2017Presenter: Tianlu Wang 5 / Esteban Real, Sherry Moore, Andrew Selle, Saurabh Large-Scale Saxena, Evolution Yutaka Leon of Image Suematsu, Classifiers Jie Tan, Quoc V.Le, Alexey Kurakin (Google Brain 26
6 Backgrounds Achievements: evolution algorithm outputs a fully-trained model with no human participation Drawbacks: significant computation Image classification, CIFAR-10, CIFAR-100 ICML, 2017Presenter: Tianlu Wang 6 / Esteban Real, Sherry Moore, Andrew Selle, Saurabh Large-Scale Saxena, Evolution Yutaka Leon of Image Suematsu, Classifiers Jie Tan, Quoc V.Le, Alexey Kurakin (Google Brain 26
7 Outline 1 Introduction Motivation Backgrounds 2 Related Work Neuro-evolution Non-evolutionary 3 Methods Algorithm Overview Encoding and Mutations More Details 4 Results Progress of experiments Comparisons Meta-parameters 5 Summary ICML, 2017Presenter: Tianlu Wang 7 / Esteban Real, Sherry Moore, Andrew Selle, Saurabh Large-Scale Saxena, Evolution Yutaka Leon of Image Suematsu, Classifiers Jie Tan, Quoc V.Le, Alexey Kurakin (Google Brain 26
8 Neuro-evolution Weight evolution: back propagation Weight and architecture: NEAT algorithm (node and connection) ICML, 2017Presenter: Tianlu Wang 8 / Esteban Real, Sherry Moore, Andrew Selle, Saurabh Large-Scale Saxena, Evolution Yutaka Leon of Image Suematsu, Classifiers Jie Tan, Quoc V.Le, Alexey Kurakin (Google Brain 26
9 Outline 1 Introduction Motivation Backgrounds 2 Related Work Neuro-evolution Non-evolutionary 3 Methods Algorithm Overview Encoding and Mutations More Details 4 Results Progress of experiments Comparisons Meta-parameters 5 Summary ICML, 2017Presenter: Tianlu Wang 9 / Esteban Real, Sherry Moore, Andrew Selle, Saurabh Large-Scale Saxena, Evolution Yutaka Leon of Image Suematsu, Classifiers Jie Tan, Quoc V.Le, Alexey Kurakin (Google Brain 26
10 Non-evolutionary Bayesian optimization Reinforcement learning Q-learning ICML, 2017Presenter: Tianlu Wang 10
11 Outline 1 Introduction Motivation Backgrounds 2 Related Work Neuro-evolution Non-evolutionary 3 Methods Algorithm Overview Encoding and Mutations More Details 4 Results Progress of experiments Comparisons Meta-parameters 5 Summary ICML, 2017Presenter: Tianlu Wang 11
12 Algorithm Overview Input: a population of models, each model is a trained single-layer nonconvolutional model with learning rate = 0.1 Measurement: accuracy on validation dataset ICML, 2017Presenter: Tianlu Wang 12
13 Algorithm Overview Input: a population of models, each model is a trained single-layer nonconvolutional model with learning rate = 0.1 Measurement: accuracy on validation dataset model2 model1 model3 model X model Y worker mutation Validation dataset model X model Y ICML, 2017Presenter: Tianlu Wang 12 steban Real, Sherry Moore, Andrew Selle, Saurabh Large-Scale Saxena, Evolution Yutaka Leon of Image Suematsu, Classifiers Jie Tan, Quoc V.Le, Alexey Kurakin (Google/ Brain 26
14 Algorithm Overview Input: a population of models, each model is a trained single-layer nonconvolutional model with learning rate = 0.1 Measurement: accuracy on validation dataset model2 model1 model3 model X model Y worker mutation Validation dataset model X model Y When to stop? ICML, 2017Presenter: Tianlu Wang 12
15 Outline 1 Introduction Motivation Backgrounds 2 Related Work Neuro-evolution Non-evolutionary 3 Methods Algorithm Overview Encoding and Mutations More Details 4 Results Progress of experiments Comparisons Meta-parameters 5 Summary ICML, 2017Presenter: Tianlu Wang 13
16 Model Encoding Individual model is encoded as a graph: Vertices rank-3 tensor(image width * image height * channels) activations(batch normalization with ReLU or plain linear layer) Edges Identity connections Convolutions ICML, 2017Presenter: Tianlu Wang 14
17 Model Encoding Individual model is encoded as a graph: Vertices rank-3 tensor(image width * image height * channels) activations(batch normalization with ReLU or plain linear layer) Edges Identity connections Convolutions Inconsistent input: pick and keep primary one reshape(interpolation/truncation/padding) non-primary ones ICML, 2017Presenter: Tianlu Wang 14
18 Mutations The worker picks a mutation at random from a set: ALTER-LEARNING-RATE IDENTITY (effectively means keep training) RESET-WEIGHTS INSERT/REMOVE CONVOLUTION ALTER-STRIDE ALTER-NUMBER-OF-CHANNELS FILTER-SIZE INSERT-ONE-TO-ONE INSERT/REMOVE SKIP ICML, 2017Presenter: Tianlu Wang 15
19 Outline 1 Introduction Motivation Backgrounds 2 Related Work Neuro-evolution Non-evolutionary 3 Methods Algorithm Overview Encoding and Mutations More Details 4 Results Progress of experiments Comparisons Meta-parameters 5 Summary ICML, 2017Presenter: Tianlu Wang 16
20 More Details Poor initial conditions(12th silde) 45,000 training; 5,000 validation; test SGD with momentum of 0.9, batch size 50, weight decay Computation cost: floating-point operations Inherit parameters weights whenever possible ICML, 2017Presenter: Tianlu Wang 17
21 Outline 1 Introduction Motivation Backgrounds 2 Related Work Neuro-evolution Non-evolutionary 3 Methods Algorithm Overview Encoding and Mutations More Details 4 Results Progress of experiments Comparisons Meta-parameters 5 Summary ICML, 2017Presenter: Tianlu Wang 18
22 Progress of an evolution experiment ICML, 2017Presenter: Tianlu Wang 19
23 Repeatability of results and controls ICML, 2017Presenter: Tianlu Wang 20
24 Outline 1 Introduction Motivation Backgrounds 2 Related Work Neuro-evolution Non-evolutionary 3 Methods Algorithm Overview Encoding and Mutations More Details 4 Results Progress of experiments Comparisons Meta-parameters 5 Summary ICML, 2017Presenter: Tianlu Wang 21
25 Compared to hand-designed networks ICML, 2017Presenter: Tianlu Wang 22
26 Compared to auto-discovered networks ICML, 2017Presenter: Tianlu Wang 23
27 Outline 1 Introduction Motivation Backgrounds 2 Related Work Neuro-evolution Non-evolutionary 3 Methods Algorithm Overview Encoding and Mutations More Details 4 Results Progress of experiments Comparisons Meta-parameters 5 Summary ICML, 2017Presenter: Tianlu Wang 24
28 Improve the method Large population size More training steps Increase mutation rate Reset all weights ICML, 2017Presenter: Tianlu Wang 25
29 Summary Neuro-evolution starts from trivial initial conditions and yields fully trained models Construct large, accurate networks for two challenging and popular image classification benchmarks Large search space and high computation cost ICML, 2017Presenter: Tianlu Wang 26
COMP9444 Neural Networks and Deep Learning 7. Image Processing. COMP9444 c Alan Blair, 2017
COMP9444 Neural Networks and Deep Learning 7. Image Processing COMP9444 17s2 Image Processing 1 Outline Image Datasets and Tasks Convolution in Detail AlexNet Weight Initialization Batch Normalization
Machine Learning. Deep Learning. Eric Xing (and Pengtao Xie) , Fall Lecture 8, October 6, Eric CMU,
Machine Learning 10-701, Fall 2015 Deep Learning Eric Xing (and Pengtao Xie) Lecture 8, October 6, 2015 Eric Xing @ CMU, 2015 1 A perennial challenge in computer vision: feature engineering SIFT Spin image
Convolutional Neural Networks: Applications and a short timeline. 7th Deep Learning Meetup Kornel Kis Vienna,
Convolutional Neural Networks: Applications and a short timeline 7th Deep Learning Meetup Kornel Kis Vienna, 1.12.2016. Introduction Currently a master student Master thesis at BME SmartLab Started deep
CNNS FROM THE BASICS TO RECENT ADVANCES. Dmytro Mishkin Center for Machine Perception Czech Technical University in Prague
CNNS FROM THE BASICS TO RECENT ADVANCES Dmytro Mishkin Center for Machine Perception Czech Technical University in Prague ducha.aiki@gmail.com OUTLINE Short review of the CNN design Architecture progress
Fei-Fei Li & Justin Johnson & Serena Yeung
Lecture 9-1 Administrative A2 due Wed May 2 Midterm: In-class Tue May 8. Covers material through Lecture 10 (Thu May 3). Sample midterm released on piazza. Midterm review session: Fri May 4 discussion
CNN Basics. Chongruo Wu
CNN Basics Chongruo Wu Overview 1. 2. 3. Forward: compute the output of each layer Back propagation: compute gradient Updating: update the parameters with computed gradient Agenda 1. Forward Conv, Fully
CS 179 Lecture 16. Logistic Regression & Parallel SGD
CS 179 Lecture 16 Logistic Regression & Parallel SGD 1 Outline logistic regression (stochastic) gradient descent parallelizing SGD for neural nets (with emphasis on Google s distributed neural net implementation)
Progressive Neural Architecture Search
Progressive Neural Architecture Search Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, Kevin Murphy 09/10/2018 @ECCV 1 Outline Introduction
Deep Learning Explained Module 4: Convolution Neural Networks (CNN or Conv Nets)
Deep Learning Explained Module 4: Convolution Neural Networks (CNN or Conv Nets) Sayan D. Pathak, Ph.D., Principal ML Scientist, Microsoft Roland Fernandez, Senior Researcher, Microsoft Module Outline
CENG 783. Special topics in. Deep Learning. AlchemyAPI. Week 11. Sinan Kalkan
CENG 783 Special topics in Deep Learning AlchemyAPI Week 11 Sinan Kalkan TRAINING A CNN Fig: http://www.robots.ox.ac.uk/~vgg/practicals/cnn/ Feed-forward pass Note that this is written in terms of the
Fuzzy Set Theory in Computer Vision: Example 3
Fuzzy Set Theory in Computer Vision: Example 3 Derek T. Anderson and James M. Keller FUZZ-IEEE, July 2017 Overview Purpose of these slides are to make you aware of a few of the different CNN architectures
Convolutional Neural Networks
NPFL114, Lecture 4 Convolutional Neural Networks Milan Straka March 25, 2019 Charles University in Prague Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics unless otherwise
Inception and Residual Networks. Hantao Zhang. Deep Learning with Python.
Inception and Residual Networks Hantao Zhang Deep Learning with Python https://en.wikipedia.org/wiki/residual_neural_network Deep Neural Network Progress from Large Scale Visual Recognition Challenge (ILSVRC)
arxiv: v1 [cs.cv] 1 Jul 2018
![arxiv: v1 [cs.cv] 1 Jul 2018](/thumbs/95/123671304.jpg "arxiv: v1 [cs.cv] 1 Jul 2018")
Autonomous Deep Learning: A Genetic DCNN Designer for Image Classification Benteng Ma Yong Xia* School of Computer Science, Northwestern Polytechnical University Xian 710072, China yxia@nwpu.edu.cn arxiv:1807.00284v1
CAP 6412 Advanced Computer Vision
CAP 6412 Advanced Computer Vision http://www.cs.ucf.edu/~bgong/cap6412.html Boqing Gong April 21st, 2016 Today Administrivia Free parameters in an approach, model, or algorithm? Egocentric videos by Aisha
Deep Learning. Visualizing and Understanding Convolutional Networks. Christopher Funk. Pennsylvania State University.
Visualizing and Understanding Convolutional Networks Christopher Pennsylvania State University February 23, 2015 Some Slide Information taken from Pierre Sermanet (Google) presentation on and Computer
MoonRiver: Deep Neural Network in C++
MoonRiver: Deep Neural Network in C++ Chung-Yi Weng Computer Science & Engineering University of Washington chungyi@cs.washington.edu Abstract Artificial intelligence resurges with its dramatic improvement
Convolutional Neural Networks and Supervised Learning
Convolutional Neural Networks and Supervised Learning Eilif Solberg August 30, 2018 Outline Convolutional Architectures Convolutional neural networks Training Loss Optimization Regularization Hyperparameter
Tutorial on Keras CAP ADVANCED COMPUTER VISION SPRING 2018 KISHAN S ATHREY
Tutorial on Keras CAP 6412 - ADVANCED COMPUTER VISION SPRING 2018 KISHAN S ATHREY Deep learning packages TensorFlow Google PyTorch Facebook AI research Keras Francois Chollet (now at Google) Chainer Company
arxiv: v1 [cs.cv] 17 Jan 2019
![arxiv: v1 [cs.cv] 17 Jan 2019](/thumbs/93/114449085.jpg "arxiv: v1 [cs.cv] 17 Jan 2019")
EAT-NAS: Elastic Architecture Transfer for Accelerating Large-scale Neural Architecture Search arxiv:1901.05884v1 [cs.cv] 17 Jan 2019 Jiemin Fang 1, Yukang Chen 3, Xinbang Zhang 3, Qian Zhang 2 Chang Huang
Advanced Video Analysis & Imaging
Advanced Video Analysis & Imaging (5LSH0), Module 09B Machine Learning with Convolutional Neural Networks (CNNs) - Workout Farhad G. Zanjani, Clint Sebastian, Egor Bondarev, Peter H.N. de With ( p.h.n.de.with@tue.nl
Deep Learning in Visual Recognition. Thanks Da Zhang for the slides
Deep Learning in Visual Recognition Thanks Da Zhang for the slides Deep Learning is Everywhere 2 Roadmap Introduction Convolutional Neural Network Application Image Classification Object Detection Object
Deep Learning for Computer Vision II
IIIT Hyderabad Deep Learning for Computer Vision II C. V. Jawahar Paradigm Shift Feature Extraction (SIFT, HoG, ) Part Models / Encoding Classifier Sparrow Feature Learning Classifier Sparrow L 1 L 2 L
Deep learning for dense per-pixel prediction. Chunhua Shen The University of Adelaide, Australia
Deep learning for dense per-pixel prediction Chunhua Shen The University of Adelaide, Australia Image understanding Classification error Convolution Neural Networks 0.3 0.2 0.1 Image Classification [Krizhevsky
Inception Network Overview. David White CS793
Inception Network Overview David White CS793 So, Leonardo DiCaprio dreams about dreaming... https://m.media-amazon.com/images/m/mv5bmjaxmzy3njcxnf5bml5banbnxkftztcwnti5otm0mw@@._v1_sy1000_cr0,0,675,1 000_AL_.jpg
Characterization and Benchmarking of Deep Learning. Natalia Vassilieva, PhD Sr. Research Manager
Characterization and Benchmarking of Deep Learning Natalia Vassilieva, PhD Sr. Research Manager Deep learning applications Vision Speech Text Other Search & information extraction Security/Video surveillance
Deep Neural Network Hyperparameter Optimization with Genetic Algorithms
Deep Neural Network Hyperparameter Optimization with Genetic Algorithms EvoDevo A Genetic Algorithm Framework Aaron Vose, Jacob Balma, Geert Wenes, and Rangan Sukumar Cray Inc. October 2017 Presenter Vose,
arxiv: v2 [cs.lg] 21 Nov 2017
![arxiv: v2 [cs.lg] 21 Nov 2017](/thumbs/79/79826195.jpg "arxiv: v2 [cs.lg] 21 Nov 2017")
Efficient Architecture Search by Network Transformation Han Cai 1, Tianyao Chen 1, Weinan Zhang 1, Yong Yu 1, Jun Wang 2 1 Shanghai Jiao Tong University, 2 University College London {hcai,tychen,wnzhang,yyu}@apex.sjtu.edu.cn,
Elastic Neural Networks for Classification
Elastic Neural Networks for Classification Yi Zhou 1, Yue Bai 1, Shuvra S. Bhattacharyya 1, 2 and Heikki Huttunen 1 1 Tampere University of Technology, Finland, 2 University of Maryland, USA arxiv:1810.00589v3
Supplementary material for Analyzing Filters Toward Efficient ConvNet
Supplementary material for Analyzing Filters Toward Efficient Net Takumi Kobayashi National Institute of Advanced Industrial Science and Technology, Japan takumi.kobayashi@aist.go.jp A. Orthonormal Steerable
arxiv: v2 [cs.ne] 11 Aug 2017
![arxiv: v2 [cs.ne] 11 Aug 2017](/thumbs/76/73550421.jpg "arxiv: v2 [cs.ne] 11 Aug 2017")
A Genetic Programming Approach to Designing Convolutional Neural Network Architectures Masanori Suganuma Yokohama National University 79-7 Tokiwadai Hodogaya-ku Yokohama, Japan 240-8501 suganuma-masanori-hf@ynu.jp
Know your data - many types of networks
Architectures Know your data - many types of networks Fixed length representation Variable length representation Online video sequences, or samples of different sizes Images Specific architectures for
Neural Optimizer Search with Reinforcement Learning
Irwan Bello 1 Barret Zoph 1 Vijay Vasudevan 1 Quoc V. Le 1 1 Google Brain ICLR, 2017/ Presenter: Anant Kharkar Outline 1 Introduction Motivation Approach 2 Methods Domain-Specific Language Controller RNN
Deep Learning Workshop. Nov. 20, 2015 Andrew Fishberg, Rowan Zellers
Deep Learning Workshop Nov. 20, 2015 Andrew Fishberg, Rowan Zellers Why deep learning? The ImageNet Challenge Goal: image classification with 1000 categories Top 5 error rate of 15%. Krizhevsky, Alex,
HENet: A Highly Efficient Convolutional Neural. Networks Optimized for Accuracy, Speed and Storage
HENet: A Highly Efficient Convolutional Neural Networks Optimized for Accuracy, Speed and Storage Qiuyu Zhu Shanghai University zhuqiuyu@staff.shu.edu.cn Ruixin Zhang Shanghai University chriszhang96@shu.edu.cn
High Performance Computing
High Performance Computing 9th Lecture 2016/10/28 YUKI ITO 1 Selected Paper: vdnn: Virtualized Deep Neural Networks for Scalable, MemoryEfficient Neural Network Design Minsoo Rhu, Natalia Gimelshein, Jason
Deep Learning with Intel DAAL
Deep Learning with Intel DAAL on Knights Landing Processor David Ojika dave.n.ojika@cern.ch March 22, 2017 Outline Introduction and Motivation Intel Knights Landing Processor Intel Data Analytics and Acceleration
Learning to Segment Object Candidates
Learning to Segment Object Candidates Pedro Pinheiro, Ronan Collobert and Piotr Dollar Presented by - Sivaraman, Kalpathy Sitaraman, M.S. in Computer Science, University of Virginia Facebook Artificial
Tuning the Layers of Neural Networks for Robust Generalization
208 Int'l Conf. Data Science ICDATA'18 Tuning the Layers of Neural Networks for Robust Generalization C. P. Chiu, and K. Y. Michael Wong Department of Physics Hong Kong University of Science and Technology
Evolving Multitask Neural Network Structure
Evolving Multitask Neural Network Structure Risto Miikkulainen The University of Texas at Austin and Sentient Technologies, Inc. What is Metalearning? In this talk: Discovering effective NN structure...
Study of Residual Networks for Image Recognition
Study of Residual Networks for Image Recognition Mohammad Sadegh Ebrahimi Stanford University sadegh@stanford.edu Hossein Karkeh Abadi Stanford University hosseink@stanford.edu Abstract Deep neural networks
arxiv: v1 [cs.lg] 17 Jan 2019
![arxiv: v1 [cs.lg] 17 Jan 2019](/thumbs/88/115116440.jpg "arxiv: v1 [cs.lg] 17 Jan 2019")
NEUNETS: AN AUTOMATED SYNTHESIS ENGINE FOR NEURAL NETWORK DESIGN Atin Sood,,1 Benjamin Elder,,2 Benjamin Herta,,4 Chao Xue,,6 Costas Bekas,,3 A. Cristiano I. Malossi,,3 Debashish Saha,,4 Florian Scheidegger,,3
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. By Joa õ Carreira and Andrew Zisserman Presenter: Zhisheng Huang 03/02/2018
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset By Joa õ Carreira and Andrew Zisserman Presenter: Zhisheng Huang 03/02/2018 Outline: Introduction Action classification architectures
Convolutional Neural Networks
Lecturer: Barnabas Poczos Introduction to Machine Learning (Lecture Notes) Convolutional Neural Networks Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publications.
Intro to Deep Learning. Slides Credit: Andrej Karapathy, Derek Hoiem, Marc Aurelio, Yann LeCunn
Intro to Deep Learning Slides Credit: Andrej Karapathy, Derek Hoiem, Marc Aurelio, Yann LeCunn Why this class? Deep Features Have been able to harness the big data in the most efficient and effective
YOLO9000: Better, Faster, Stronger
YOLO9000: Better, Faster, Stronger Date: January 24, 2018 Prepared by Haris Khan (University of Toronto) Haris Khan CSC2548: Machine Learning in Computer Vision 1 Overview 1. Motivation for one-shot object
2. Blackbox hyperparameter optimization and AutoML
AutoML 2017. Automatic Selection, Configuration & Composition of ML Algorithms. at ECML PKDD 2017, Skopje. 2. Blackbox hyperparameter optimization and AutoML Pavel Brazdil, Frank Hutter, Holger Hoos, Joaquin
Deep Learning and Its Applications
Convolutional Neural Network and Its Application in Image Recognition Oct 28, 2016 Outline 1 A Motivating Example 2 The Convolutional Neural Network (CNN) Model 3 Training the CNN Model 4 Issues and Recent
Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability
Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability Janis Keuper Itwm.fraunhofer.de/ml Competence Center High Performance Computing Fraunhofer ITWM, Kaiserslautern,
Deep Learning with Tensorflow AlexNet
Machine Learning and Computer Vision Group Deep Learning with Tensorflow http://cvml.ist.ac.at/courses/dlwt_w17/ AlexNet Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton, "Imagenet classification
DEEP NEURAL NETWORKS AND GPUS. Julie Bernauer
DEEP NEURAL NETWORKS AND GPUS Julie Bernauer GPU Computing GPU Computing Run Computations on GPUs x86 CUDA Framework to Program NVIDIA GPUs A simple sum of two vectors (arrays) in C void vector_add(int
Index. Springer Nature Switzerland AG 2019 B. Moons et al., Embedded Deep Learning,
Index A Algorithmic noise tolerance (ANT), 93 94 Application specific instruction set processors (ASIPs), 115 116 Approximate computing application level, 95 circuits-levels, 93 94 DAS and DVAS, 107 110
Como funciona o Deep Learning
Como funciona o Deep Learning Moacir Ponti (com ajuda de Gabriel Paranhos da Costa) ICMC, Universidade de São Paulo Contact: www.icmc.usp.br/~moacir moacir@icmc.usp.br Uberlandia-MG/Brazil October, 2017
Lecture 37: ConvNets (Cont d) and Training
Lecture 37: ConvNets (Cont d) and Training CS 4670/5670 Sean Bell [http://bbabenko.tumblr.com/post/83319141207/convolutional-learnings-things-i-learned-by] (Unrelated) Dog vs Food [Karen Zack, @teenybiscuit]
Evolving Deep Convolutional Neural Networks for Image Classification
Evolving Deep onvolutional Neural Networks for Image lassification Yanan Sun, Bing Xue and Mengjie Zhang School of Engineering and omputer Science, Victoria University of Wellington PO Box 6, Wellington
Deep Learning For Video Classification. Presented by Natalie Carlebach & Gil Sharon
Deep Learning For Video Classification Presented by Natalie Carlebach & Gil Sharon Overview Of Presentation Motivation Challenges of video classification Common datasets 4 different methods presented in
Perceptron: This is convolution!
Perceptron: This is convolution! v v v Shared weights v Filter = local perceptron. Also called kernel. By pooling responses at different locations, we gain robustness to the exact spatial location of image
Deep Convolutional Neural Networks. Nov. 20th, 2015 Bruce Draper
Deep Convolutional Neural Networks Nov. 20th, 2015 Bruce Draper Background: Fully-connected single layer neural networks Feed-forward classification Trained through back-propagation Example Computer Vision
CNN optimization. Rassadin A
CNN optimization Rassadin A. 01.2017-02.2017 What to optimize? Training stage time consumption (CPU / GPU) Inference stage time consumption (CPU / GPU) Training stage memory consumption Inference stage
Deep Residual Learning
Deep Residual Learning MSRA @ ILSVRC & COCO 2015 competitions Kaiming He with Xiangyu Zhang, Shaoqing Ren, Jifeng Dai, & Jian Sun Microsoft Research Asia (MSRA) MSRA @ ILSVRC & COCO 2015 Competitions 1st
CSE 559A: Computer Vision
CSE 559A: Computer Vision Fall 2018: T-R: 11:30-1pm @ Lopata 101 Instructor: Ayan Chakrabarti (ayan@wustl.edu). Course Staff: Zhihao Xia, Charlie Wu, Han Liu http://www.cse.wustl.edu/~ayan/courses/cse559a/
Generative Modeling with Convolutional Neural Networks. Denis Dus Data Scientist at InData Labs
Generative Modeling with Convolutional Neural Networks Denis Dus Data Scientist at InData Labs What we will discuss 1. 2. 3. 4. Discriminative vs Generative modeling Convolutional Neural Networks How to
Using CODEQ to Train Feed-forward Neural Networks
Using CODEQ to Train Feed-forward Neural Networks Mahamed G. H. Omran 1 and Faisal al-adwani 2 1 Department of Computer Science, Gulf University for Science and Technology, Kuwait, Kuwait omran.m@gust.edu.kw
A Novel Weight-Shared Multi-Stage Network Architecture of CNNs for Scale Invariance
JOURNAL OF L A TEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 1 A Novel Weight-Shared Multi-Stage Network Architecture of CNNs for Scale Invariance Ryo Takahashi, Takashi Matsubara, Member, IEEE, and Kuniaki
INTRODUCTION TO DEEP LEARNING
INTRODUCTION TO DEEP LEARNING CONTENTS Introduction to deep learning Contents 1. Examples 2. Machine learning 3. Neural networks 4. Deep learning 5. Convolutional neural networks 6. Conclusion 7. Additional
Machine Learning. MGS Lecture 3: Deep Learning
Dr Michel F. Valstar http://cs.nott.ac.uk/~mfv/ Machine Learning MGS Lecture 3: Deep Learning Dr Michel F. Valstar http://cs.nott.ac.uk/~mfv/ WHAT IS DEEP LEARNING? Shallow network: Only one hidden layer
RGBd Image Semantic Labelling for Urban Driving Scenes via a DCNN
RGBd Image Semantic Labelling for Urban Driving Scenes via a DCNN Jason Bolito, Research School of Computer Science, ANU Supervisors: Yiran Zhong & Hongdong Li 2 Outline 1. Motivation and Background 2.
Brainchip OCTOBER
Brainchip OCTOBER 2017 1 Agenda Neuromorphic computing background Akida Neuromorphic System-on-Chip (NSoC) Brainchip OCTOBER 2017 2 Neuromorphic Computing Background Brainchip OCTOBER 2017 3 A Brief History
Encoder-Decoder Networks for Semantic Segmentation. Sachin Mehta
Encoder-Decoder Networks for Semantic Segmentation Sachin Mehta Outline > Overview of Semantic Segmentation > Encoder-Decoder Networks > Results What is Semantic Segmentation? Input: RGB Image Output:
CONVOLUTIONAL NEURAL NETWORK OPTIMIZATION USING GENETIC ALGORITHMS
CONVOLUTIONAL NEURAL NETWORK OPTIMIZATION USING GENETIC ALGORITHMS Thesis Submitted to The School of Engineering of the UNIVERSITY OF DAYTON In Partial Fulfillment of the Requirements for The Degree of
Capsule Networks. Eric Mintun
Capsule Networks Eric Mintun Motivation An improvement* to regular Convolutional Neural Networks. Two goals: Replace max-pooling operation with something more intuitive. Keep more info about an activated
DECISION TREES & RANDOM FORESTS X CONVOLUTIONAL NEURAL NETWORKS
DECISION TREES & RANDOM FORESTS X CONVOLUTIONAL NEURAL NETWORKS Deep Neural Decision Forests Microsoft Research Cambridge UK, ICCV 2015 Decision Forests, Convolutional Networks and the Models in-between
Dynamic Routing Between Capsules
Report Explainable Machine Learning Dynamic Routing Between Capsules Author: Michael Dorkenwald Supervisor: Dr. Ullrich Köthe 28. Juni 2018 Inhaltsverzeichnis 1 Introduction 2 2 Motivation 2 3 CapusleNet
SEMANTIC COMPUTING. Lecture 8: Introduction to Deep Learning. TU Dresden, 7 December Dagmar Gromann International Center For Computational Logic
SEMANTIC COMPUTING Lecture 8: Introduction to Deep Learning Dagmar Gromann International Center For Computational Logic TU Dresden, 7 December 2018 Overview Introduction Deep Learning General Neural Networks
Facial Expression Classification with Random Filters Feature Extraction
Facial Expression Classification with Random Filters Feature Extraction Mengye Ren Facial Monkey mren@cs.toronto.edu Zhi Hao Luo It s Me lzh@cs.toronto.edu I. ABSTRACT In our work, we attempted to tackle
Machine Learning With Python. Bin Chen Nov. 7, 2017 Research Computing Center
Machine Learning With Python Bin Chen Nov. 7, 2017 Research Computing Center Outline Introduction to Machine Learning (ML) Introduction to Neural Network (NN) Introduction to Deep Learning NN Introduction
Deep Learning for Embedded Security Evaluation
Deep Learning for Embedded Security Evaluation Emmanuel Prouff 1 1 Laboratoire de Sécurité des Composants, ANSSI, France April 2018, CISCO April 2018, CISCO E. Prouff 1/22 Contents 1. Context and Motivation
Multi-Glance Attention Models For Image Classification
Multi-Glance Attention Models For Image Classification Chinmay Duvedi Stanford University Stanford, CA cduvedi@stanford.edu Pararth Shah Stanford University Stanford, CA pararth@stanford.edu Abstract We
DL Tutorial. Xudong Cao
DL Tutorial Xudong Cao Historical Line 1960s Perceptron 1980s MLP BP algorithm 2006 RBM unsupervised learning 2012 AlexNet ImageNet Comp. 2014 GoogleNet VGGNet ImageNet Comp. Rule based AI algorithm Game
Deconvolutions in Convolutional Neural Networks
Overview Deconvolutions in Convolutional Neural Networks Bohyung Han bhhan@postech.ac.kr Computer Vision Lab. Convolutional Neural Networks (CNNs) Deconvolutions in CNNs Applications Network visualization
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution and Fully Connected CRFs
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution and Fully Connected CRFs Zhipeng Yan, Moyuan Huang, Hao Jiang 5/1/2017 1 Outline Background semantic segmentation Objective,
Binary Convolutional Neural Network on RRAM
Binary Convolutional Neural Network on RRAM Tianqi Tang, Lixue Xia, Boxun Li, Yu Wang, Huazhong Yang Dept. of E.E, Tsinghua National Laboratory for Information Science and Technology (TNList) Tsinghua
Lecture note 7: Playing with convolutions in TensorFlow
Lecture note 7: Playing with convolutions in TensorFlow CS 20SI: TensorFlow for Deep Learning Research (cs20si.stanford.edu) Prepared by Chip Huyen ( huyenn@stanford.edu ) This lecture note is an unfinished
Progressive Neural Architecture Search
Progressive Neural Architecture Search Chenxi Liu 1, Barret Zoph 2, Maxim Neumann 2, Jonathon Shlens 2, Wei Hua 2, Li-Jia Li 2, Li Fei-Fei 2,3, Alan Yuille 1, Jonathan Huang 2, and Kevin Murphy 2 1 Johns
NEUNETS: AN AUTOMATED SYNTHESIS ENGINE FOR NEURAL NETWORK DESIGN. IBM Watson AI Platform and IBM Research AI
NEUNETS: AN AUTOMATED SYNTHESIS ENGINE FOR NEURAL NETWORK DESIGN Atin Sood,,1 Benjamin Elder,,2 Benjamin Herta,,4 Chao Xue,,6 Costas Bekas,,3 A. Cristiano I. Malossi,,3 Debashish Saha,,4 Florian Scheidegger,,3
SIMPLE AND EFFICIENT ARCHITECTURE SEARCH FOR CONVOLUTIONAL NEURAL NETWORKS
SIMPLE AND EFFICIENT ARCHITECTURE SEARCH FOR CONVOLUTIONAL NEURAL NETWORKS Anonymous authors Paper under double-blind review ABSTRACT Neural networks have recently had a lot of success for many tasks.
Convolutional Neural Networks. Computer Vision Jia-Bin Huang, Virginia Tech
Convolutional Neural Networks Computer Vision Jia-Bin Huang, Virginia Tech Today s class Overview Convolutional Neural Network (CNN) Training CNN Understanding and Visualizing CNN Image Categorization:
arxiv: v2 [cs.cv] 24 Mar 2018
![arxiv: v2 [cs.cv] 24 Mar 2018](/thumbs/80/80480588.jpg "arxiv: v2 [cs.cv] 24 Mar 2018")
arxiv:1712.00559v2 [cs.cv] 24 Mar 2018 Progressive Neural Architecture Search Chenxi Liu 1,, Barret Zoph 2, Maxim Neumann 3, Jonathon Shlens 2, Wei Hua 3, Li-Jia Li 3, Li Fei-Fei 3,4, Alan Yuille 1, Jonathan
CafeGPI. Single-Sided Communication for Scalable Deep Learning
CafeGPI Single-Sided Communication for Scalable Deep Learning Janis Keuper itwm.fraunhofer.de/ml Competence Center High Performance Computing Fraunhofer ITWM, Kaiserslautern, Germany Deep Neural Networks
Fusion of Mini-Deep Nets
Rochester Institute of Technology RIT Scholar Works Theses Thesis/Dissertation Collections 5-2016 Fusion of Mini-Deep Nets Sai Prasad Nooka spn8235@rit.edu Follow this and additional works at: http://scholarworks.rit.edu/theses
Device Placement Optimization with Reinforcement Learning
Device Placement Optimization with Reinforcement Learning Azalia Mirhoseini, Anna Goldie, Hieu Pham, Benoit Steiner, Mohammad Norouzi, Naveen Kumar, Rasmus Larsen, Yuefeng Zhou, Quoc Le, Samy Bengio, and
arxiv: v2 [cs.lg] 3 Dec 2018
![arxiv: v2 [cs.lg] 3 Dec 2018](/thumbs/94/118256495.jpg "arxiv: v2 [cs.lg] 3 Dec 2018")
MONAS: Multi-Objective Neural Architecture Search Chi-Hung Hsu, 1 Shu-Huan Chang, 1 Jhao-Hong Liang, 1 Hsin-Ping Chou, 1 Chun-Hao Liu, 1 Shih-Chieh Chang, 1 Jia-Yu Pan, 2 Yu-Ting Chen, 2 Wei Wei, 2 Da-Cheng
CrescendoNet: A New Deep Convolutional Neural Network with Ensemble Behavior
CrescendoNet: A New Deep Convolutional Neural Network with Ensemble Behavior Xiang Zhang, Nishant Vishwamitra, Hongxin Hu, Feng Luo School of Computing Clemson University xzhang7@clemson.edu Abstract We
arxiv: v1 [cs.cv] 26 Nov 2018
![arxiv: v1 [cs.cv] 26 Nov 2018](/thumbs/91/105110741.jpg "arxiv: v1 [cs.cv] 26 Nov 2018")
Evolving Space-Time Neural Architectures for Videos AJ Piergiovanni, Anelia Angelova, Alexander Toshev, Michael S. Ryoo Google Brain {ajpiergi,anelia,toshev,mryoo}@google.com arxiv:8.066v [cs.cv] 26 Nov
Using Genetic Algorithms to Solve the Box Stacking Problem
Using Genetic Algorithms to Solve the Box Stacking Problem Jenniffer Estrada, Kris Lee, Ryan Edgar October 7th, 2010 Abstract The box stacking or strip stacking problem is exceedingly difficult to solve
Training Deep Neural Networks (in parallel)
Lecture 9: Training Deep Neural Networks (in parallel) Visual Computing Systems How would you describe this professor? Easy? Mean? Boring? Nerdy? Professor classification task Classifies professors as
Channel Locality Block: A Variant of Squeeze-and-Excitation
Channel Locality Block: A Variant of Squeeze-and-Excitation 1 st Huayu Li Northern Arizona University Flagstaff, United State Northern Arizona University hl459@nau.edu arxiv:1901.01493v1 [cs.lg] 6 Jan
arxiv: v2 [cs.cv] 30 Oct 2018
![arxiv: v2 [cs.cv] 30 Oct 2018](/thumbs/94/118508906.jpg "arxiv: v2 [cs.cv] 30 Oct 2018")
Adversarial Noise Layer: Regularize Neural Network By Adding Noise Zhonghui You, Jinmian Ye, Kunming Li, Zenglin Xu, Ping Wang School of Electronics Engineering and Computer Science, Peking University
Deep Learning. Roland Olsson
Deep Learning Roland Olsson The first deep learning success Classifying handwritten digits Published test error rates without preprocessing for the MNIST dataset 12% for linear discriminants 3.3% for 40
Deep Learning for Computer Vision with MATLAB By Jon Cherrie
Deep Learning for Computer Vision with MATLAB By Jon Cherrie 2015 The MathWorks, Inc. 1 Deep learning is getting a lot of attention "Dahl and his colleagues won $22,000 with a deeplearning system. 'We