Outline GF-RNN ReNet. Outline

Size: px
Start display at page:

Download "Outline GF-RNN ReNet. Outline"

Transcription

1 Outline Gated Feedback Recurrent Neural Networks. arxiv1502. Introduction: RNN & Gated RNN Gated Feedback Recurrent Neural Networks (GF-RNN) Experiments: Character-level Language Modeling & Python Program Evaluation ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks. arxiv1505. Introduction ReNet: 4 RNNs that sweep over lower-layer features in 4 directions Experiments: MNIST & CIFAR-10 & Street View House Numbers LU Yangyang luyy11@pku.edu.cn May KERE Seminar

2 Authors Gated Feedback Recurrent Neural Networks arxiv.org. 9 Feb Feb Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, Yoshua Bengio (University of Montreal) ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks arxiv.org. 3 May Francesco Visin, Kyle Kastner, Kyunghyun Cho, Matteo Matteucci, Aaron Courville, Yoshua Bengio (University of Montreal)

3 Outline Gated Feedback Recurrent Neural Networks. arxiv1502. Introduction: RNN & Gated RNN Gated Feedback Recurrent Neural Networks (GF-RNN) Experiments: Character-level Language Modeling & Python Program Evaluation ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks. arxiv1505.

4 Recurrent Neural Networks (RNN) FOR Sequence Modeling Can process a sequence of arbitrary length Recursively applying a transition function to its internal hidden state for each symbol of the input sequence Theoretically capture any long-term dependency in an input sequence Difficult to train an RNN to actually do so h t = f(x t, h t 1 ) = φ(wx t + Uh t 1 ) p(x 1, x 2,..., x T ) = p(x 1 )p(x 2 x 1 )...p(x T x 1,..., x T 1 ) p(x t+1 x 1,..., x t) = g(h t) Figure: A single-layer RNN

5 Gated Recurrent Neural Networks 1 LSTM & GRU A LSTM Unit: h j t = oj t tanh(cj t ) A GRU Unit: c j t = fj t cj t 1 + ij t cj t c j t = tanh(wcx t + Uch t 1 )j f j t = σ(w f x t + U f h t 1 + V f c t )j i j t = σ(w i x t + U i h t 1 + V i c t )j h j t = (1 zj t )hj t 1 + zj t h j t z j t = σ(wz x t + Uz h t 1 )j h j t = tanh(w x t + U(r t h t 1 ))j r j t = σ(wr x t + Ur h t 1 )j o j t = σ(wox t + Uoh t 1 + Voc t )j 1 Chung, J.,et al. Empirical evaluation of gated recurrent neural networks on sequence modeling. arxiv 14.

6 Gated Recurrent Neural Networks Modifying the RNN architecture Using a gated activation function: - the long short-term memory unit (LSTM): a memory cell, an input gate, a forget gate, and an output gate - the gated recurrent unit (GRU): a reset gate and an update gate Can contain both fast changing and slow changing components - stacked multiple levels of recurrent layers - partitioned and grouped hidden units to allow feedback information at multiple timescales Achieved promising results in both classification and generation tasks Gated-feedback RNN (GF-RNN): learning multiple adaptive timescales

7 GF-RNN: Overview Figure: A Clockwork RNN A sequence often consists of both slow-moving and fast-moving components. - slow-moving: long-term dependencies - fast-moving: short-term dependencies El Hihi & Bengio (1995): an RNN can capture these dependencies of different timescales more easily and efficiently when the hidden units of the RNN is explicitly partitioned into groups that correspond to different timescales. The clockwork RNN (CW-RNN) (Koutnik et al., 2014): updating the i-th module only when t mod 2 i 1 = 0 to generalize the CW-RNN by allowing the model to adaptively adjust the connectivity pattern between the hidden layers in the consecutive time-steps

8 GF-RNN: Overview (cont.) Partition the hidden units into multiple modules: each module corresponds to a different layer in a stack of recurrent layers Compared to CW-RNN: do not set an explicit rate for each module each module: hierarchically stacked different timescales Each module is fully connected to all the other modules across the stack and itself. The global reset gate: gated the recurrent connection between two modules based on the current input and the previous states of the hidden layers

9 GF-RNN: The global reset gate h i t : the hidden unit on the i-th layer at time-step t wg i j, u i j g : weights for inputs and hidden states of all layers at time-step t 1 g i j : control the signal from the i-th layer h i t 1 to the j-th layer hj t based on the input and the previous hidden units

10 GF-RNN: The global reset gate h i t : the hidden unit on the i-th layer at time-step t wg i j, u i j g : weights for inputs and hidden states of all layers at time-step t 1 g i j : control the signal from the i-th layer h i t 1 to the j-th layer hj t based on the input and the previous hidden units Information flows: stacked RNN & GF-RNN: lower layers upper layers GF-RNN: lower layers upper layers (finer timescale coarser timescale) A gated-feedback RNN: A fully-connected recurrent transition and global reset gates

11 GF-RNN: Different Units of Practical Implementation tanh Units LSTM: LSTM Units & GRU Units: only use the global reset gates when computing the new state h j t = oj t tanh(cj t ) GRU: c j t = fj t cj t 1 + ij t cj t h j t = (1 zj t )hj t 1 + zj t h j t f j t = σ(w f x t + U f h t 1 + V f c t )j i j t = σ(w i x t + U i h t 1 + V i c t )j z j t = σ(wz x t + Uz h t 1 )j r j t = σ(wr x t + Ur h t 1 )j o j t = σ(wox t + Uoh t 1 + Voc t )j

12 Experiment Tasks BOTH: representative examples of discrete sequence modeling Objective Function: to minimize the negative log-likelihood of training sequences Character-level language modeling: English Wikipedia: 100MB characters Contents: Latin alphabets, non-latin alphabets, XML markups and special characters Vocabulary: 205 characters (one token for unknown character) Train/CV/Test: 90MB/5MB/5MB Preformance measure: the average number of bits-per-character (BPC, E[ log 2 P (x t+1 h t)]) Pythong Program Evaluation: Goal:to generate or predict a correct return value of a given Python script Input: python scripts (include addition, multiplication, subtraction, for-loop, variable assignment, logical comparison and if-else statement) Output: predicted value of the given Python script Input/Output: 41/31 symbols

13 Examples for Python Program Evaluation 2 2 Zaremba, Wojciech and Sutskever, Ilya. Learning to execute. arxiv preprint arxiv: , 2014.

14 Experiments: Character-level Language Modeling The sizes of models: Tuning parameters: RMSProp and momentum Test set BPC of models trained on the Hutter dataset for a 100 epochs:

15 Experiments: Character-level Language Modeling (cont.) Text Generation based on character-level language modeling: Given the seed at the left-most column (bold-faced font), the models predict next characters. Tabs, spaces and new-line characters are also generated by the models.

16 Experiments: Python Program Evaluation Using an RNN encoder-decoder approach: Python scripts ENCODER (50 timesteps) h t DECODER character-level results

17 Outline Gated Feedback Recurrent Neural Networks. arxiv1502. ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks. arxiv1505. Introduction ReNet: 4 RNNs that sweep over lower-layer features in 4 directions Experiments: MNIST & CIFAR-10 & Street View House Numbers

18 Introduction Object Recognition: Convolutional Neural Networks (CNN): LeNet-5 - based on local context window Recurrent Neural Networks: (Graves and Schmidhuber, 2009): a multi-dimensional RNN ReNet: purely uni-dimensional RNNs: replace each convolutional layer (conv. + pooling) in the CNN 4 RNNs that sweep over lower-layer features in 4 directions:,,, - each feature activation: at the specific location with respect to the whole image

19 A one-layer ReNet The Input Image: x R w h c (width, height, feature dimensionality) Give a patch size - w p h p: split the input image x into a set of I J (nonoverlapping) patches X = {x ij }, x ij R wp hp c 1. Sweep the image vertically with 2 RNNs(, ): Each RNN takes as an input one (flattened) patch at a time and updates its hidden state, working along each column j of the split input image X. 2. Concatenate the intermediate hidden states zi,j F, zr i,j at each location (i, j) to get a composite feature map v = {z i,j } j=1,...,j i=1,...,i, z ij R 1 hp 2d (d: the number of recurrent units) 3. Sweep V horizonally with 2 RNNs(, ) in a similar manner. The resulting feature map H = z i,j, z ij R1 1 2d : the features of the original image patch x i,j in the context of the whole image The deep ReNet: stacked multiple φ s (φ: the function from X to H)

20 Differences between LeNet and ReNet BOTH: apply the same set of filters to patches of the input image or the feature map from the layer below LeNet: Patches: overlap Many levels of convolution+pooling layers: to detect redundant features from different parts of the image Using max-pooling to achieve local translation invariance highly parallelizable due to the independence of computing activations at each layer

21 Differences between LeNet and ReNet ReNet: Patches: not overlap The lateral connections: help extract a more compact feature representation of the input image at each layer The lateral connection in ReNet can emulate the local competition among features induced by the max-pooling in LeNet not easily parallelizable, due to the sequential nature of RNN

22 Datasets MNIST: - 70K handwritten digits from 0 to pixels, each pixel: grayscale in [0,255] - Train/Dev/Test: 50K/10K/10K CIFAR-10: - 60K images: a curated subset of the 80M tiny images dataset - 10 categories: airplane, automobile, bird, cat, deer, dog, frog, horse, ship and truck pixels, each pixel: 3 color channels (red-green-blue) - Train/Dev/Test: 40K/10K/10K Street View House Numbers (SVHN): - cropped images representing house numbers captured by Google StreetView vehicles as a part of the Google Maps mapping process - These images consist of digits 0 through 9 with values in the range of [0, 255] pixels, each pixel: grayscale in [0,255] - Train/Dev/Test: 543,949/60,439/26,032 Data Augmentation: Flippling: flipped each sample horizontally with 25% chance, flipped it vertically with 25% chance Shifting: - 2 pixels (25% chance), 2 pixels (25% chance) - 2 pixels (25% chance), 2 pixels (25% chance)

23 Unit Implementation: - GRU: MNIST, CIFAR-10 - LSTM: SVHN Model Architectures General Architecture: - N RE : the number of ReNet layers - d RE : the feature dimensionality - N F C : the number of fully-connected layers - d F C : the feature dimensionality of fully-connected layers - f F C : types of hidden units

24 Experiment Results

25 Choice of Recurrent Units Discussion ReNet preforms well independently of the specific implementation of the recurrent units (either LSTM or GRU). Gated recurrent units, either the GRU or the LSTM, outperform a usual sigmoidal unit (affine transformation followed by an elementwise sigmoid function.) Analysis of the Trained ReNet ReNet performs comparably to deep convolutional neural networks which are the de facto standard for object recognition. ReNet does not outperform state-of-the-art convolutional neural networks on any of the three benchmark datasets. Authors expect that the internal behavior of ReNet will differ from that of LeNet significantly, which need further investigation. Computationally Efficient Implementation ReNet is less parallelizable due to the sequential nature of RNN. ReNet allows the forward and backward RNNs to be run independently from each other, which allows for parallel computation.

26 Summary Gated Feedback Recurrent Neural Networks. arxiv1502. Stacked RNN + Fully-connected recurrent transition + Global reset gates Using 3 kinds of implementation units: tanh, LSTM, GRU Experiments: Character-level Text Generation - English Wikipedia - Better than conventional stacked RNN Experiments: Python Program Evaluation - Predict return values of Python scripts - Using a RNN Encoder-Decoder approach ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks. arxiv1505. A one-layer ReNet: 4 RNNs in 4 directions to sweep the input image Stacked ReNet layers Deep Architectures Using 2 kinds of implementation units: LSTM, GRU Experiments: MNIST & CIFAR-10 & Street View House Numbers - Using flipping and shifting as data augmentation - ReNet performs comparably to deep CNN, but does not outperform state-ofthe-art CNN on any of the three tasks.

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung Chung, Caglar Gulcehre, KyungHyun Cho and Yoshua Bengio Presenter: Yu-Wei Lin Background: Recurrent Neural

More information

Pixel-level Generative Model

Pixel-level Generative Model Pixel-level Generative Model Generative Image Modeling Using Spatial LSTMs (2015NIPS) L. Theis and M. Bethge University of Tübingen, Germany Pixel Recurrent Neural Networks (2016ICML) A. van den Oord,

More information

LSTM and its variants for visual recognition. Xiaodan Liang Sun Yat-sen University

LSTM and its variants for visual recognition. Xiaodan Liang Sun Yat-sen University LSTM and its variants for visual recognition Xiaodan Liang xdliang328@gmail.com Sun Yat-sen University Outline Context Modelling with CNN LSTM and its Variants LSTM Architecture Variants Application in

More information

Recurrent Neural Networks

Recurrent Neural Networks Recurrent Neural Networks Javier Béjar Deep Learning 2018/2019 Fall Master in Artificial Intelligence (FIB-UPC) Introduction Sequential data Many problems are described by sequences Time series Video/audio

More information

LSTM with Working Memory

LSTM with Working Memory LSTM with Working Memory Andrew Pulver Department of Computer Science University at Albany Email: apulver@albany.edu Siwei Lyu Department of Computer Science University at Albany Email: slyu@albany.edu

More information

Multi-Glance Attention Models For Image Classification

Multi-Glance Attention Models For Image Classification Multi-Glance Attention Models For Image Classification Chinmay Duvedi Stanford University Stanford, CA cduvedi@stanford.edu Pararth Shah Stanford University Stanford, CA pararth@stanford.edu Abstract We

More information

Deep Learning in Visual Recognition. Thanks Da Zhang for the slides

Deep Learning in Visual Recognition. Thanks Da Zhang for the slides Deep Learning in Visual Recognition Thanks Da Zhang for the slides Deep Learning is Everywhere 2 Roadmap Introduction Convolutional Neural Network Application Image Classification Object Detection Object

More information

Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks

Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks Rahul Dey and Fathi M. Salem Circuits, Systems, and Neural Networks (CSANN) LAB Department of Electrical and Computer Engineering Michigan State

More information

Empirical Evaluation of RNN Architectures on Sentence Classification Task

Empirical Evaluation of RNN Architectures on Sentence Classification Task Empirical Evaluation of RNN Architectures on Sentence Classification Task Lei Shen, Junlin Zhang Chanjet Information Technology lorashen@126.com, zhangjlh@chanjet.com Abstract. Recurrent Neural Networks

More information

CS489/698: Intro to ML

CS489/698: Intro to ML CS489/698: Intro to ML Lecture 14: Training of Deep NNs Instructor: Sun Sun 1 Outline Activation functions Regularization Gradient-based optimization 2 Examples of activation functions 3 5/28/18 Sun Sun

More information

Machine Learning 13. week

Machine Learning 13. week Machine Learning 13. week Deep Learning Convolutional Neural Network Recurrent Neural Network 1 Why Deep Learning is so Popular? 1. Increase in the amount of data Thanks to the Internet, huge amount of

More information

MoonRiver: Deep Neural Network in C++

MoonRiver: Deep Neural Network in C++ MoonRiver: Deep Neural Network in C++ Chung-Yi Weng Computer Science & Engineering University of Washington chungyi@cs.washington.edu Abstract Artificial intelligence resurges with its dramatic improvement

More information

DEEP LEARNING REVIEW. Yann LeCun, Yoshua Bengio & Geoffrey Hinton Nature Presented by Divya Chitimalla

DEEP LEARNING REVIEW. Yann LeCun, Yoshua Bengio & Geoffrey Hinton Nature Presented by Divya Chitimalla DEEP LEARNING REVIEW Yann LeCun, Yoshua Bengio & Geoffrey Hinton Nature 2015 -Presented by Divya Chitimalla What is deep learning Deep learning allows computational models that are composed of multiple

More information

From Maxout to Channel-Out: Encoding Information on Sparse Pathways

From Maxout to Channel-Out: Encoding Information on Sparse Pathways From Maxout to Channel-Out: Encoding Information on Sparse Pathways Qi Wang and Joseph JaJa Department of Electrical and Computer Engineering and, University of Maryland Institute of Advanced Computer

More information

Deep Learning with Tensorflow AlexNet

Deep Learning with Tensorflow   AlexNet Machine Learning and Computer Vision Group Deep Learning with Tensorflow http://cvml.ist.ac.at/courses/dlwt_w17/ AlexNet Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton, "Imagenet classification

More information

Deep Learning Workshop. Nov. 20, 2015 Andrew Fishberg, Rowan Zellers

Deep Learning Workshop. Nov. 20, 2015 Andrew Fishberg, Rowan Zellers Deep Learning Workshop Nov. 20, 2015 Andrew Fishberg, Rowan Zellers Why deep learning? The ImageNet Challenge Goal: image classification with 1000 categories Top 5 error rate of 15%. Krizhevsky, Alex,

More information

Machine Learning. Deep Learning. Eric Xing (and Pengtao Xie) , Fall Lecture 8, October 6, Eric CMU,

Machine Learning. Deep Learning. Eric Xing (and Pengtao Xie) , Fall Lecture 8, October 6, Eric CMU, Machine Learning 10-701, Fall 2015 Deep Learning Eric Xing (and Pengtao Xie) Lecture 8, October 6, 2015 Eric Xing @ CMU, 2015 1 A perennial challenge in computer vision: feature engineering SIFT Spin image

More information

Deep Neural Network Hyperparameter Optimization with Genetic Algorithms

Deep Neural Network Hyperparameter Optimization with Genetic Algorithms Deep Neural Network Hyperparameter Optimization with Genetic Algorithms EvoDevo A Genetic Algorithm Framework Aaron Vose, Jacob Balma, Geert Wenes, and Rangan Sukumar Cray Inc. October 2017 Presenter Vose,

More information

An Empirical Evaluation of Deep Architectures on Problems with Many Factors of Variation

An Empirical Evaluation of Deep Architectures on Problems with Many Factors of Variation An Empirical Evaluation of Deep Architectures on Problems with Many Factors of Variation Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio Université de Montréal 13/06/2007

More information

LSTM: An Image Classification Model Based on Fashion-MNIST Dataset

LSTM: An Image Classification Model Based on Fashion-MNIST Dataset LSTM: An Image Classification Model Based on Fashion-MNIST Dataset Kexin Zhang, Research School of Computer Science, Australian National University Kexin Zhang, U6342657@anu.edu.au Abstract. The application

More information

arxiv: v2 [cs.cv] 14 May 2018

arxiv: v2 [cs.cv] 14 May 2018 ContextVP: Fully Context-Aware Video Prediction Wonmin Byeon 1234, Qin Wang 1, Rupesh Kumar Srivastava 3, and Petros Koumoutsakos 1 arxiv:1710.08518v2 [cs.cv] 14 May 2018 Abstract Video prediction models

More information

Rotation Invariance Neural Network

Rotation Invariance Neural Network Rotation Invariance Neural Network Shiyuan Li Abstract Rotation invariance and translate invariance have great values in image recognition. In this paper, we bring a new architecture in convolutional neural

More information

Machine Learning With Python. Bin Chen Nov. 7, 2017 Research Computing Center

Machine Learning With Python. Bin Chen Nov. 7, 2017 Research Computing Center Machine Learning With Python Bin Chen Nov. 7, 2017 Research Computing Center Outline Introduction to Machine Learning (ML) Introduction to Neural Network (NN) Introduction to Deep Learning NN Introduction

More information

Overall Description. Goal: to improve spatial invariance to the input data. Translation, Rotation, Scale, Clutter, Elastic

Overall Description. Goal: to improve spatial invariance to the input data. Translation, Rotation, Scale, Clutter, Elastic Philippe Giguère Overall Description Goal: to improve spatial invariance to the input data Translation, Rotation, Scale, Clutter, Elastic How: add a learnable module which explicitly manipulate spatially

More information

CSC 578 Neural Networks and Deep Learning

CSC 578 Neural Networks and Deep Learning CSC 578 Neural Networks and Deep Learning Fall 2018/19 7. Recurrent Neural Networks (Some figures adapted from NNDL book) 1 Recurrent Neural Networks 1. Recurrent Neural Networks (RNNs) 2. RNN Training

More information

Recurrent Neural Nets II

Recurrent Neural Nets II Recurrent Neural Nets II Steven Spielberg Pon Kumar, Tingke (Kevin) Shen Machine Learning Reading Group, Fall 2016 9 November, 2016 Outline 1 Introduction 2 Problem Formulations with RNNs 3 LSTM for Optimization

More information

COMP9444 Neural Networks and Deep Learning 7. Image Processing. COMP9444 c Alan Blair, 2017

COMP9444 Neural Networks and Deep Learning 7. Image Processing. COMP9444 c Alan Blair, 2017 COMP9444 Neural Networks and Deep Learning 7. Image Processing COMP9444 17s2 Image Processing 1 Outline Image Datasets and Tasks Convolution in Detail AlexNet Weight Initialization Batch Normalization

More information

Sentiment Classification of Food Reviews

Sentiment Classification of Food Reviews Sentiment Classification of Food Reviews Hua Feng Department of Electrical Engineering Stanford University Stanford, CA 94305 fengh15@stanford.edu Ruixi Lin Department of Electrical Engineering Stanford

More information

Layerwise Interweaving Convolutional LSTM

Layerwise Interweaving Convolutional LSTM Layerwise Interweaving Convolutional LSTM Tiehang Duan and Sargur N. Srihari Department of Computer Science and Engineering The State University of New York at Buffalo Buffalo, NY 14260, United States

More information

arxiv: v3 [cs.lg] 30 Dec 2016

arxiv: v3 [cs.lg] 30 Dec 2016 Video Ladder Networks Francesco Cricri Nokia Technologies francesco.cricri@nokia.com Xingyang Ni Tampere University of Technology xingyang.ni@tut.fi arxiv:1612.01756v3 [cs.lg] 30 Dec 2016 Mikko Honkala

More information

Hierarchical Multiscale Gated Recurrent Unit

Hierarchical Multiscale Gated Recurrent Unit Hierarchical Multiscale Gated Recurrent Unit Alex Pine alex.pine@nyu.edu Sean D Rosario seandrosario@nyu.edu Abstract The hierarchical multiscale long-short term memory recurrent neural network (HM- LSTM)

More information

arxiv: v1 [cs.ne] 6 Jul 2015

arxiv: v1 [cs.ne] 6 Jul 2015 Grid Long Short-Term Memory Nal Kalchbrenner Ivo Danihelka Alex Graves Google DeepMind arxiv:1507.01526v1 [cs.ne] 6 Jul 2015 Abstract This paper introduces Grid Long Short-Term Memory, a network of LSTM

More information

Show, Discriminate, and Tell: A Discriminatory Image Captioning Model with Deep Neural Networks

Show, Discriminate, and Tell: A Discriminatory Image Captioning Model with Deep Neural Networks Show, Discriminate, and Tell: A Discriminatory Image Captioning Model with Deep Neural Networks Zelun Luo Department of Computer Science Stanford University zelunluo@stanford.edu Te-Lin Wu Department of

More information

Introduction to Neural Networks

Introduction to Neural Networks Introduction to Neural Networks Jakob Verbeek 2017-2018 Biological motivation Neuron is basic computational unit of the brain about 10^11 neurons in human brain Simplified neuron model as linear threshold

More information

Deep Learning for Computer Vision II

Deep Learning for Computer Vision II IIIT Hyderabad Deep Learning for Computer Vision II C. V. Jawahar Paradigm Shift Feature Extraction (SIFT, HoG, ) Part Models / Encoding Classifier Sparrow Feature Learning Classifier Sparrow L 1 L 2 L

More information

Deep Learning and Its Applications

Deep Learning and Its Applications Convolutional Neural Network and Its Application in Image Recognition Oct 28, 2016 Outline 1 A Motivating Example 2 The Convolutional Neural Network (CNN) Model 3 Training the CNN Model 4 Issues and Recent

More information

Image Captioning with Object Detection and Localization

Image Captioning with Object Detection and Localization Image Captioning with Object Detection and Localization Zhongliang Yang, Yu-Jin Zhang, Sadaqat ur Rehman, Yongfeng Huang, Department of Electronic Engineering, Tsinghua University, Beijing 100084, China

More information

Unsupervised Feature Learning for Optical Character Recognition

Unsupervised Feature Learning for Optical Character Recognition Unsupervised Feature Learning for Optical Character Recognition Devendra K Sahu and C. V. Jawahar Center for Visual Information Technology, IIIT Hyderabad, India. Abstract Most of the popular optical character

More information

Show, Discriminate, and Tell: A Discriminatory Image Captioning Model with Deep Neural Networks

Show, Discriminate, and Tell: A Discriminatory Image Captioning Model with Deep Neural Networks Show, Discriminate, and Tell: A Discriminatory Image Captioning Model with Deep Neural Networks Boya Peng Department of Computer Science Stanford University boya@stanford.edu Zelun Luo Department of Computer

More information

CAP 6412 Advanced Computer Vision

CAP 6412 Advanced Computer Vision CAP 6412 Advanced Computer Vision http://www.cs.ucf.edu/~bgong/cap6412.html Boqing Gong Feb 04, 2016 Today Administrivia Attention Modeling in Image Captioning, by Karan Neural networks & Backpropagation

More information

Code Mania Artificial Intelligence: a. Module - 1: Introduction to Artificial intelligence and Python:

Code Mania Artificial Intelligence: a. Module - 1: Introduction to Artificial intelligence and Python: Code Mania 2019 Artificial Intelligence: a. Module - 1: Introduction to Artificial intelligence and Python: 1. Introduction to Artificial Intelligence 2. Introduction to python programming and Environment

More information

Cambridge Interview Technical Talk

Cambridge Interview Technical Talk Cambridge Interview Technical Talk February 2, 2010 Table of contents Causal Learning 1 Causal Learning Conclusion 2 3 Motivation Recursive Segmentation Learning Causal Learning Conclusion Causal learning

More information

Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks

Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks Alberto Montes al.montes.gomez@gmail.com Santiago Pascual TALP Research Center santiago.pascual@tsc.upc.edu Amaia Salvador

More information

Capsule Networks. Eric Mintun

Capsule Networks. Eric Mintun Capsule Networks Eric Mintun Motivation An improvement* to regular Convolutional Neural Networks. Two goals: Replace max-pooling operation with something more intuitive. Keep more info about an activated

More information

JOINT INTENT DETECTION AND SLOT FILLING USING CONVOLUTIONAL NEURAL NETWORKS. Puyang Xu, Ruhi Sarikaya. Microsoft Corporation

JOINT INTENT DETECTION AND SLOT FILLING USING CONVOLUTIONAL NEURAL NETWORKS. Puyang Xu, Ruhi Sarikaya. Microsoft Corporation JOINT INTENT DETECTION AND SLOT FILLING USING CONVOLUTIONAL NEURAL NETWORKS Puyang Xu, Ruhi Sarikaya Microsoft Corporation ABSTRACT We describe a joint model for intent detection and slot filling based

More information

Low-Power Neural Processor for Embedded Human and Face detection

Low-Power Neural Processor for Embedded Human and Face detection Low-Power Neural Processor for Embedded Human and Face detection Olivier Brousse 1, Olivier Boisard 1, Michel Paindavoine 1,2, Jean-Marc Philippe, Alexandre Carbon (1) GlobalSensing Technologies (GST)

More information

Combining Neural Networks and Log-linear Models to Improve Relation Extraction

Combining Neural Networks and Log-linear Models to Improve Relation Extraction Combining Neural Networks and Log-linear Models to Improve Relation Extraction Thien Huu Nguyen and Ralph Grishman Computer Science Department, New York University {thien,grishman}@cs.nyu.edu Outline Relation

More information

Weighted Convolutional Neural Network. Ensemble.

Weighted Convolutional Neural Network. Ensemble. Weighted Convolutional Neural Network Ensemble Xavier Frazão and Luís A. Alexandre Dept. of Informatics, Univ. Beira Interior and Instituto de Telecomunicações Covilhã, Portugal xavierfrazao@gmail.com

More information

Deep Learning. Deep Learning. Practical Application Automatically Adding Sounds To Silent Movies

Deep Learning. Deep Learning. Practical Application Automatically Adding Sounds To Silent Movies http://blog.csdn.net/zouxy09/article/details/8775360 Automatic Colorization of Black and White Images Automatically Adding Sounds To Silent Movies Traditionally this was done by hand with human effort

More information

Dynamic Routing Between Capsules. Yiting Ethan Li, Haakon Hukkelaas, and Kaushik Ram Ramasamy

Dynamic Routing Between Capsules. Yiting Ethan Li, Haakon Hukkelaas, and Kaushik Ram Ramasamy Dynamic Routing Between Capsules Yiting Ethan Li, Haakon Hukkelaas, and Kaushik Ram Ramasamy Problems & Results Object classification in images without losing information about important parts of the picture.

More information

Convolutional Neural Networks

Convolutional Neural Networks Lecturer: Barnabas Poczos Introduction to Machine Learning (Lecture Notes) Convolutional Neural Networks Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publications.

More information

Recurrent Convolutional Neural Networks for Scene Labeling

Recurrent Convolutional Neural Networks for Scene Labeling Recurrent Convolutional Neural Networks for Scene Labeling Pedro O. Pinheiro, Ronan Collobert Reviewed by Yizhe Zhang August 14, 2015 Scene labeling task Scene labeling: assign a class label to each pixel

More information

LSTM for Language Translation and Image Captioning. Tel Aviv University Deep Learning Seminar Oran Gafni & Noa Yedidia

LSTM for Language Translation and Image Captioning. Tel Aviv University Deep Learning Seminar Oran Gafni & Noa Yedidia 1 LSTM for Language Translation and Image Captioning Tel Aviv University Deep Learning Seminar Oran Gafni & Noa Yedidia 2 Part I LSTM for Language Translation Motivation Background (RNNs, LSTMs) Model

More information

Hierarchical Video Frame Sequence Representation with Deep Convolutional Graph Network

Hierarchical Video Frame Sequence Representation with Deep Convolutional Graph Network Hierarchical Video Frame Sequence Representation with Deep Convolutional Graph Network Feng Mao [0000 0001 6171 3168], Xiang Wu [0000 0003 2698 2156], Hui Xue, and Rong Zhang Alibaba Group, Hangzhou, China

More information

Deep Neural Networks:

Deep Neural Networks: Deep Neural Networks: Part II Convolutional Neural Network (CNN) Yuan-Kai Wang, 2016 Web site of this course: http://pattern-recognition.weebly.com source: CNN for ImageClassification, by S. Lazebnik,

More information

Keras: Handwritten Digit Recognition using MNIST Dataset

Keras: Handwritten Digit Recognition using MNIST Dataset Keras: Handwritten Digit Recognition using MNIST Dataset IIT PATNA February 9, 2017 1 / 24 OUTLINE 1 Introduction Keras: Deep Learning library for Theano and TensorFlow 2 Installing Keras Installation

More information

Accelerating Convolutional Neural Nets. Yunming Zhang

Accelerating Convolutional Neural Nets. Yunming Zhang Accelerating Convolutional Neural Nets Yunming Zhang Focus Convolutional Neural Nets is the state of the art in classifying the images The models take days to train Difficult for the programmers to tune

More information

Index. Umberto Michelucci 2018 U. Michelucci, Applied Deep Learning,

Index. Umberto Michelucci 2018 U. Michelucci, Applied Deep Learning, A Acquisition function, 298, 301 Adam optimizer, 175 178 Anaconda navigator conda command, 3 Create button, 5 download and install, 1 installing packages, 8 Jupyter Notebook, 11 13 left navigation pane,

More information

All You Want To Know About CNNs. Yukun Zhu

All You Want To Know About CNNs. Yukun Zhu All You Want To Know About CNNs Yukun Zhu Deep Learning Deep Learning Image from http://imgur.com/ Deep Learning Image from http://imgur.com/ Deep Learning Image from http://imgur.com/ Deep Learning Image

More information

Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. By Joa õ Carreira and Andrew Zisserman Presenter: Zhisheng Huang 03/02/2018

Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. By Joa õ Carreira and Andrew Zisserman Presenter: Zhisheng Huang 03/02/2018 Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset By Joa õ Carreira and Andrew Zisserman Presenter: Zhisheng Huang 03/02/2018 Outline: Introduction Action classification architectures

More information

Sequence Modeling: Recurrent and Recursive Nets. By Pyry Takala 14 Oct 2015

Sequence Modeling: Recurrent and Recursive Nets. By Pyry Takala 14 Oct 2015 Sequence Modeling: Recurrent and Recursive Nets By Pyry Takala 14 Oct 2015 Agenda Why Recurrent neural networks? Anatomy and basic training of an RNN (10.2, 10.2.1) Properties of RNNs (10.2.2, 8.2.6) Using

More information

27: Hybrid Graphical Models and Neural Networks

27: Hybrid Graphical Models and Neural Networks 10-708: Probabilistic Graphical Models 10-708 Spring 2016 27: Hybrid Graphical Models and Neural Networks Lecturer: Matt Gormley Scribes: Jakob Bauer Otilia Stretcu Rohan Varma 1 Motivation We first look

More information

Artificial Intelligence Introduction Handwriting Recognition Kadir Eren Unal ( ), Jakob Heyder ( )

Artificial Intelligence Introduction Handwriting Recognition Kadir Eren Unal ( ), Jakob Heyder ( ) Structure: 1. Introduction 2. Problem 3. Neural network approach a. Architecture b. Phases of CNN c. Results 4. HTM approach a. Architecture b. Setup c. Results 5. Conclusion 1.) Introduction Artificial

More information

Convolution Neural Networks for Chinese Handwriting Recognition

Convolution Neural Networks for Chinese Handwriting Recognition Convolution Neural Networks for Chinese Handwriting Recognition Xu Chen Stanford University 450 Serra Mall, Stanford, CA 94305 xchen91@stanford.edu Abstract Convolutional neural networks have been proven

More information

Tutorial on Keras CAP ADVANCED COMPUTER VISION SPRING 2018 KISHAN S ATHREY

Tutorial on Keras CAP ADVANCED COMPUTER VISION SPRING 2018 KISHAN S ATHREY Tutorial on Keras CAP 6412 - ADVANCED COMPUTER VISION SPRING 2018 KISHAN S ATHREY Deep learning packages TensorFlow Google PyTorch Facebook AI research Keras Francois Chollet (now at Google) Chainer Company

More information

Deep Learning With Noise

Deep Learning With Noise Deep Learning With Noise Yixin Luo Computer Science Department Carnegie Mellon University yixinluo@cs.cmu.edu Fan Yang Department of Mathematical Sciences Carnegie Mellon University fanyang1@andrew.cmu.edu

More information

CS231N Section. Video Understanding 6/1/2018

CS231N Section. Video Understanding 6/1/2018 CS231N Section Video Understanding 6/1/2018 Outline Background / Motivation / History Video Datasets Models Pre-deep learning CNN + RNN 3D convolution Two-stream What we ve seen in class so far... Image

More information

Bayesian model ensembling using meta-trained recurrent neural networks

Bayesian model ensembling using meta-trained recurrent neural networks Bayesian model ensembling using meta-trained recurrent neural networks Luca Ambrogioni l.ambrogioni@donders.ru.nl Umut Güçlü u.guclu@donders.ru.nl Yağmur Güçlütürk y.gucluturk@donders.ru.nl Julia Berezutskaya

More information

Deep Learning Applications

Deep Learning Applications October 20, 2017 Overview Supervised Learning Feedforward neural network Convolution neural network Recurrent neural network Recursive neural network (Recursive neural tensor network) Unsupervised Learning

More information

A Comparison of Sequence-Trained Deep Neural Networks and Recurrent Neural Networks Optical Modeling For Handwriting Recognition

A Comparison of Sequence-Trained Deep Neural Networks and Recurrent Neural Networks Optical Modeling For Handwriting Recognition A Comparison of Sequence-Trained Deep Neural Networks and Recurrent Neural Networks Optical Modeling For Handwriting Recognition Théodore Bluche, Hermann Ney, Christopher Kermorvant SLSP 14, Grenoble October

More information

Seminars in Artifiial Intelligenie and Robotiis

Seminars in Artifiial Intelligenie and Robotiis Seminars in Artifiial Intelligenie and Robotiis Computer Vision for Intelligent Robotiis Basiis and hints on CNNs Alberto Pretto What is a neural network? We start from the frst type of artifcal neuron,

More information

Profiling the Performance of Binarized Neural Networks. Daniel Lerner, Jared Pierce, Blake Wetherton, Jialiang Zhang

Profiling the Performance of Binarized Neural Networks. Daniel Lerner, Jared Pierce, Blake Wetherton, Jialiang Zhang Profiling the Performance of Binarized Neural Networks Daniel Lerner, Jared Pierce, Blake Wetherton, Jialiang Zhang 1 Outline Project Significance Prior Work Research Objectives Hypotheses Testing Framework

More information

Learning Discrete Representations via Information Maximizing Self-Augmented Training

Learning Discrete Representations via Information Maximizing Self-Augmented Training A. Relation to Denoising and Contractive Auto-encoders Our method is related to denoising auto-encoders (Vincent et al., 2008). Auto-encoders maximize a lower bound of mutual information (Cover & Thomas,

More information

Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction

Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction by Noh, Hyeonwoo, Paul Hongsuck Seo, and Bohyung Han.[1] Presented : Badri Patro 1 1 Computer Vision Reading

More information

Deep Learning For Video Classification. Presented by Natalie Carlebach & Gil Sharon

Deep Learning For Video Classification. Presented by Natalie Carlebach & Gil Sharon Deep Learning For Video Classification Presented by Natalie Carlebach & Gil Sharon Overview Of Presentation Motivation Challenges of video classification Common datasets 4 different methods presented in

More information

arxiv: v1 [cs.lg] 16 Jan 2013

arxiv: v1 [cs.lg] 16 Jan 2013 Stochastic Pooling for Regularization of Deep Convolutional Neural Networks arxiv:131.3557v1 [cs.lg] 16 Jan 213 Matthew D. Zeiler Department of Computer Science Courant Institute, New York University zeiler@cs.nyu.edu

More information

arxiv: v1 [cs.cv] 16 Mar 2018

arxiv: v1 [cs.cv] 16 Mar 2018 Semantic Adversarial Examples Hossein Hosseini Radha Poovendran Network Security Lab (NSL) Department of Electrical Engineering, University of Washington, Seattle, WA arxiv:1804.00499v1 [cs.cv] 16 Mar

More information

EVALUATION OF RECURRENT NEURAL NETWORKS FOR CROP RECOGNITION FROM MULTITEMPORAL REMOTE SENSING IMAGES ABSTRACT

EVALUATION OF RECURRENT NEURAL NETWORKS FOR CROP RECOGNITION FROM MULTITEMPORAL REMOTE SENSING IMAGES ABSTRACT S B C Anais do XXVII Congresso Brasileiro de Cartografia e XXVI Exposicarta 6 a 9 de novembro de 2017, SBC, Rio de Janeiro - RJ, p. 800-804 EVALUATION OF RECURRENT NEURAL NETWORKS FOR CROP RECOGNITION

More information

Face Recognition A Deep Learning Approach

Face Recognition A Deep Learning Approach Face Recognition A Deep Learning Approach Lihi Shiloh Tal Perl Deep Learning Seminar 2 Outline What about Cat recognition? Classical face recognition Modern face recognition DeepFace FaceNet Comparison

More information

Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks Supplementary Material

Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks Supplementary Material Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks Supplementary Material Emily Denton Dept. of Computer Science Courant Institute New York University Soumith Chintala Arthur

More information

DCU-UvA Multimodal MT System Report

DCU-UvA Multimodal MT System Report DCU-UvA Multimodal MT System Report Iacer Calixto ADAPT Centre School of Computing Dublin City University Dublin, Ireland iacer.calixto@adaptcentre.ie Desmond Elliott ILLC University of Amsterdam Science

More information

arxiv: v1 [cs.cv] 4 Feb 2018

arxiv: v1 [cs.cv] 4 Feb 2018 End2You The Imperial Toolkit for Multimodal Profiling by End-to-End Learning arxiv:1802.01115v1 [cs.cv] 4 Feb 2018 Panagiotis Tzirakis Stefanos Zafeiriou Björn W. Schuller Department of Computing Imperial

More information

Deep Learning. Visualizing and Understanding Convolutional Networks. Christopher Funk. Pennsylvania State University.

Deep Learning. Visualizing and Understanding Convolutional Networks. Christopher Funk. Pennsylvania State University. Visualizing and Understanding Convolutional Networks Christopher Pennsylvania State University February 23, 2015 Some Slide Information taken from Pierre Sermanet (Google) presentation on and Computer

More information

HENet: A Highly Efficient Convolutional Neural. Networks Optimized for Accuracy, Speed and Storage

HENet: A Highly Efficient Convolutional Neural. Networks Optimized for Accuracy, Speed and Storage HENet: A Highly Efficient Convolutional Neural Networks Optimized for Accuracy, Speed and Storage Qiuyu Zhu Shanghai University zhuqiuyu@staff.shu.edu.cn Ruixin Zhang Shanghai University chriszhang96@shu.edu.cn

More information

Convolutional Neural Networks. CSC 4510/9010 Andrew Keenan

Convolutional Neural Networks. CSC 4510/9010 Andrew Keenan Convolutional Neural Networks CSC 4510/9010 Andrew Keenan Neural network to Convolutional Neural Network 0.4 0.7 0.3 Neural Network Basics 0.4 0.7 0.3 Individual Neuron Network Backpropagation 1. 2. 3.

More information

Visual object classification by sparse convolutional neural networks

Visual object classification by sparse convolutional neural networks Visual object classification by sparse convolutional neural networks Alexander Gepperth 1 1- Ruhr-Universität Bochum - Institute for Neural Dynamics Universitätsstraße 150, 44801 Bochum - Germany Abstract.

More information

Object Detection Lecture Introduction to deep learning (CNN) Idar Dyrdal

Object Detection Lecture Introduction to deep learning (CNN) Idar Dyrdal Object Detection Lecture 10.3 - Introduction to deep learning (CNN) Idar Dyrdal Deep Learning Labels Computational models composed of multiple processing layers (non-linear transformations) Used to learn

More information

Know your data - many types of networks

Know your data - many types of networks Architectures Know your data - many types of networks Fixed length representation Variable length representation Online video sequences, or samples of different sizes Images Specific architectures for

More information

ABC-CNN: Attention Based CNN for Visual Question Answering

ABC-CNN: Attention Based CNN for Visual Question Answering ABC-CNN: Attention Based CNN for Visual Question Answering CIS 601 PRESENTED BY: MAYUR RUMALWALA GUIDED BY: DR. SUNNIE CHUNG AGENDA Ø Introduction Ø Understanding CNN Ø Framework of ABC-CNN Ø Datasets

More information

Ryerson University CP8208. Soft Computing and Machine Intelligence. Naive Road-Detection using CNNS. Authors: Sarah Asiri - Domenic Curro

Ryerson University CP8208. Soft Computing and Machine Intelligence. Naive Road-Detection using CNNS. Authors: Sarah Asiri - Domenic Curro Ryerson University CP8208 Soft Computing and Machine Intelligence Naive Road-Detection using CNNS Authors: Sarah Asiri - Domenic Curro April 24 2016 Contents 1 Abstract 2 2 Introduction 2 3 Motivation

More information

On the Efficiency of Recurrent Neural Network Optimization Algorithms

On the Efficiency of Recurrent Neural Network Optimization Algorithms On the Efficiency of Recurrent Neural Network Optimization Algorithms Ben Krause, Liang Lu, Iain Murray, Steve Renals University of Edinburgh Department of Informatics s17005@sms.ed.ac.uk, llu@staffmail.ed.ac.uk,

More information

Recurrent Neural Networks and Transfer Learning for Action Recognition

Recurrent Neural Networks and Transfer Learning for Action Recognition Recurrent Neural Networks and Transfer Learning for Action Recognition Andrew Giel Stanford University agiel@stanford.edu Ryan Diaz Stanford University ryandiaz@stanford.edu Abstract We have taken on the

More information

6. Convolutional Neural Networks

6. Convolutional Neural Networks 6. Convolutional Neural Networks CS 519 Deep Learning, Winter 2017 Fuxin Li With materials from Zsolt Kira Quiz coming up Next Thursday (2/2) 20 minutes Topics: Optimization Basic neural networks No Convolutional

More information

Application of Deep Learning Techniques in Satellite Telemetry Analysis.

Application of Deep Learning Techniques in Satellite Telemetry Analysis. Application of Deep Learning Techniques in Satellite Telemetry Analysis. Greg Adamski, Member of Technical Staff L3 Technologies Telemetry and RF Products Julian Spencer Jones, Spacecraft Engineer Telenor

More information

Host based Intrusion Detection System with Combined CNN/RNN Model

Host based Intrusion Detection System with Combined CNN/RNN Model Host based Intrusion Detection System with Combined CNN/RNN Model Ashima Chawla, Brian Lee, Sheila Fallon and Paul Jacob Athlone Institute of Technology, Ireland a.chawla@research.ait.ie, {blee, sheilafallon,

More information

UTS submission to Google YouTube-8M Challenge 2017

UTS submission to Google YouTube-8M Challenge 2017 UTS submission to Google YouTube-8M Challenge 2017 Linchao Zhu Yanbin Liu Yi Yang University of Technology Sydney {zhulinchao7, csyanbin, yee.i.yang}@gmail.com Abstract In this paper, we present our solution

More information

Domain-Aware Sentiment Classification with GRUs and CNNs

Domain-Aware Sentiment Classification with GRUs and CNNs Domain-Aware Sentiment Classification with GRUs and CNNs Guangyuan Piao 1(B) and John G. Breslin 2 1 Insight Centre for Data Analytics, Data Science Institute, National University of Ireland Galway, Galway,

More information

YOLO9000: Better, Faster, Stronger

YOLO9000: Better, Faster, Stronger YOLO9000: Better, Faster, Stronger Date: January 24, 2018 Prepared by Haris Khan (University of Toronto) Haris Khan CSC2548: Machine Learning in Computer Vision 1 Overview 1. Motivation for one-shot object

More information

arxiv: v1 [cs.cl] 1 Feb 2016

arxiv: v1 [cs.cl] 1 Feb 2016 Efficient Character-level Document Classification by Combining Convolution and Recurrent Layers arxiv:1602.00367v1 [cs.cl] 1 Feb 2016 Yijun Xiao Center for Data Sciences, New York University ryjxiao@nyu.edu

More information

Plan, Attend, Generate: Planning for Sequence-to-Sequence Models

Plan, Attend, Generate: Planning for Sequence-to-Sequence Models Plan, Attend, Generate: Planning for Sequence-to-Sequence Models Francis Dutil, Caglar Gulcehre, Adam Trischler, Yoshua Bengio Presented by Xinyuan Zhang April 26, 2018 April 26, 2018 1 / 11 Introduction

More information