Transfer Learning. Style Transfer in Deep Learning

Size: px
Start display at page:

Download "Transfer Learning. Style Transfer in Deep Learning"

Transcription

1 Transfer Learning & Style Transfer in Deep Learning 4-DEC-2016 Gal Barzilai, Ram Machlev Deep Learning Seminar School of Electrical Engineering Tel Aviv University

2 Part 1: Transfer Learning in Deep Learning 6-OCT-2013 (976 cited) Yangqing Jia, author of Caffe and DeCAF.

3 One of The Main Problems in Deep Learning Approaches: with limited training data, fully-supervised deep architectures generally overfit many visual recognition challenges have tasks with few training examples 3

4 Transfer Learning Concept learning the features on large-scale data in a supervised setting, then transferring them to different tasks with different labels. Input A (for example: cars) Task A Transfer Input B (for example: trucks) Layer n AnB: Frozen Weights New Learned Weights Back-propagation Task B 4

5 Accuracy experiments in : J Yosinski Input A- 500 Classes from ImageNet Layer 3 Task A Transfer Input B- 500 Classes from ImageNet A3B: Frozen Weights Back-propagation Task B 5

6 DeCAF Approach- Deep convolutional representations are learned on a set of related problems but applied to new tasks which have too few training examples to learn a full deep representation. The model can be considered as : deep architecture for transfer learning based on a supervised pre-training phase. Or simply as- convolutional network weights learned on a set of pre-defined object recognition tasks. 6

7 Adopted Network Deep CNN architecture proposed by Krizhevsky [Krizhevsky NIPS 2012]. 5 convolutional layers (with pooling and ReLU) 3 fully-connected layers won ImageNet Large Scale Visual recognition Challenge (ILSVRC) 2012 (10,000,000 labeled images depicting 10,000+ object categories) as training top-1 validation error rate of 40.7% follow architecture and training protocol with two differences input 256 x 256 images rather than 224 x 224 images no data augmentation trick 7

8 DeCAF1 DeCAF2 DeCAF3 DeCAF4 DeCAF5 DeCAF6 DeCAF Activations of The nth Hidden Layer of The Deep Convolutional Neural Network As a Feature DeCAFn. 8

9 Feature Generalization and Visualization Features that were compared : ImageNet Training Database (ILVRC-2012) GIST Features GIST features LLC features Feature extraction Krizhevsky s net DeCAFn LLC Features Known Known features features extraction extraction approach approach (Locality-constrained Linear Coding. J Wang, 2010) t-sne map t-sne map (a low dimensional representation of the scene, not require any form of segmentation, Oliva A and Torralba 2001) visualize features in the following way: run t- SNE algorithm - a 2-dimensional embedding of the highdimensional feature space. plot features as points colored depending on their semantic 9 category.

10 t-sne feature visualizations on the ILSVRC-2012 validation set. (after trained on ILSVRC-2012 training set, prevent overfitting) LLC FEATURES GIST FEATURES GIST or LLC fail to capture the semantic difference between images 11

11 t-sne feature visualizations on the ILSVRC-2012 validation set. (after trained on ILSVRC-2012 training set, prevent overfitting) DeCAF1 DeCAF6 DeCAF 1 FEATURES first layers learn low-level features DeCAF 6 FEATURES latter layers learn semantic or high level features. 12

12 DeCAF 6 features trained on ILSVRC-2012 generalized to SUN-397 SUN-397: Large-scale scene recognition from abbey to zoo. (899 categories and 130,519 images) Different semantic categories 13

13 Experiments all the network s hidden layer weights are frozen to those learned on the ILSVRC-2012 dataset. New dataset Krizhevsky s net After train on ILVRC linear Classifier Frozen weights Activation feature Of new dataset Train New Task Results on multiple datasets to evaluate the strength of DeCAF for Object recognition Domain adaptation subcategory recognition scene recognition each task differ somewhat from that for which the architecture was trained. 14

14 Experiment: Object Recognition Caltech Pictures of objects belonging to 101 categories. About 40 to 800 images per category. Most categories have about 50 images. Evaluating linear classifier performance on DeCAF6 and DeCAF7. using dropout Compared also with the two-layers convolutional network of Jarret et al (2009) 15

15 Experiment: Domain Adaptation Office dataset (Saenko et al., 2010), which has 3 domains (31 categories in each domain): Amazon: images taken from amazon.com Webcam & Dslr: images taken in office environment using a webcam or digital SLR camera Domain shift: Source -> target Domain shift: Source -> target Trained Liner classifiers Adaptive methods ecent deep domain adaptation

16 Experiments Domain Adaptation DeCAF robust to resolution changes DeCAF provides better category clustering than SURF DeCAF clusters same category instances across domains SURF FEATURES DeCAF 6 FEATURES 17

17 Experiment: Subcategory Recognition Fine grained recognition involves recognizing subclasses of the same object class such as different bird species, dog breeds, flower types, etc. Caltech-UCSD birds dataset ( ~6000 photos of 200 bird species) - First,adopt ImageNet-like pipeline, DeCAF6 and a multi-class logistic regression ( as previous experiments) - Second, adopt deformable part descriptors (DPD) method [Zhang et al., 2013] 18

18 Experiment: Subcategory Recognition (only) (applied DeCAF in the same pre-trained DPM model and part predictions and used the same pooling weights). 19

19 Experiment: Scene Recognition Goal: classify the scene of the entire image SUN-397 large-scale scene recognition database (899 categories and 130,519 images) Outperforms Xiao ed al. (2010), the current state-of-the-art method 20

20 DeCAF demonstrate: Discussion Achieve high classification accuracy on tasks with sparse labeled data using simple linear classifiers. outperforming current state-of-the-art approaches based on sophisticated multi-kernel learning techniques with traditional hand-engineered features. the features tend to cluster images into interesting semantic categories on which the network was never explicitly trained. can substantially improve the performance of a wide variety of existing methods across a spectrum of visual recognition tasks 21

21 An Open-Source Convolutional Model Caffe ( at first it was called decaf) is a deep learning Python framework made with expression, speed, and modularity in mind. It is developed by the Berkeley Vision and Learning Center (BVLC) and by community contributors. Yangqing Jia created the project during his PhD at UC Berkeley. The framework allows one to easily train networks consisting of various layer types and to execute pre-trained networks efficiently without being restricted to a GPU. able to process about 40 images per second with an 8-core commodity machine when the CNN model is executed in a minibatch mode. In addition, they have released the network parameters used in their experiments to allow for out-of-the-box feature extraction without the need to re-train the large network 22

22 Image Style Transfer Using Convolutional Neural Networks 4-DEC-2016 Gal Barzilai, Ram Machlev Deep Learning Seminar School of Electrical Engineering Tel Aviv University

23 Part 2: Style Transfer

24 Texture Transfer - Review Transferring the style from one image onto another can be consider a problem of texture transfer. Our goal synthesize a texture from a source image while constraining the texture synthesis in order to preserve the semantic content of a target image. 25

25 Several Examples 26

26 Texture Transfer Former Approaches Large range of powerful non-parametric algorithms can synthesize photorealistic natural textures by resampling the pixels of a given source texture. For example Texture synthesis by non-parametric sampling These algorithms suffer from the limitation that they use only low-level image features of the target image to inform the texture transfer. There is a need for an algorithm that uses the high level image features for style transfer, and this article addresses this issue. 27

27 Texture Transfer Deep Learning Approach The article proposes a novel algorithm A neural Algorithm of Artistic Style (also suggested by the authors in an article with the same name). Many available implementations on Github (for example: ) The algorithm uses a CNN network that was trained for object recognition and localization (the chosen network was a VGG network). 28

28 Image Representations in CNN The number of different filters increase along the processing hierarchy. The size of filtered images is reduced by down-sampling mechanism, in our network average-pooling, leading to a decrease in the total number of units per layer of the network.

29 CNN Network VGG The VGG network was designed by the Visual Geometry Group in Oxford. Article: Very Deep Convolutional Networks for Large-Scale Image Recognition Developed a 16 layer and 19 layer models. Network parameters: normalized version, 16 convolutional, 5 pooling layers of the 19 VGG network. The normalization is of the weights such that the mean activation of each convolutional filter over images and positions is equal to one. No Fully connected layers. Average pooling instead of max pooling (gave better results, no theoretical explanation was provided. 30

30 Style Transfer Algorithm

31 Content Representation - Notations A layer with N l distinct filters has N l feature maps of size M l, Where M l is the height times the width of the feature maps. The responses in layer l can be stored in a matrix F l R N l M l, Where F ij l is the activation of the i th filter at position j in layer l. p is the original content image, x is the image generated (initialized from white noise). P l and F l are their respective feature representations in layer l. 32

32 Style Transfer Algorithm

33 Content Representation - Calculations Squared-error loss between two feature representations The derivative of the loss with respect to the activation in layer l The gradient with respect to the image x can be computed using standard error back-propagation. 34

34 Style Transfer Algorithm L total = L content Content Representation, ignoring style effect

35 Style Representation 1 To obtain a representation of the style of an input image, the authors used a feature space designed to capture texture information (the authors published it Texture Synthesis Using Convolutional Neural Networks. The feature space can be built on top of the filter responses in any layer of the network. It consists of the correlations between the different filter responses. The feature correlations are given by the gram matrix G l R N l N l l where G ij is the inner product between the vectorised feature maps i and j in layer l. 36

36 Style Representation 2 We include the feature correlations of multiple layers, and gain a stationary, multi-scale representation of the input image which captures its texture. We can visualize the information captured by the style feature spaces built on different layers of the network by constructing an image that matches the style representation of the style image. This is done by using gradient decent from a white noise image to minimize the mean-square distance between the entries of the Gram matrices from the style image and the gram matrices of the image to be generated. 37

37 Style Representation 3 a is the original style image, x is the image generated (initialized from white noise). A l and G l are their respective style representation in layer l. The contribution of each layer l to the total loss is: The total style loss is 38

38 Style Transfer Algorithm

39 Style Representation 4 w l are weighting factors of the contribution of each layer to the total loss. The derivative of E l with resect to the activations in layer l can be computed as: The gradients of E l with respect to the pixel values of x can be computed using standard error back-propagation. 40

40 Style Transfer Algorithm L total = L style Style Representation, ignoring content effect

41 Style Transfer Now we want to find a compromise between the style of the style picture and the content of the content image. We will now jointly minimize the distance of the feature representations of a white noise from the content representation of the photograph in one layer (a high one) and the style representation of the painting defined on a number of layer of the CNN. The loss function to be minimized is α and β are weighting factors. 42

42 Style Transfer Algorithm

43 Style Transfer Implementation Consideration The optimization strategy is L-BFGS which the authors found best for image synthesis. This is a limited memory BFGS, BFGS is an iterative method for solve unconstrained nonlinear optimization problems. The style image was resized to the size of the content image in order to extract image information on comparable size. 44

44 Style Transfer Algorithm

45 The Main Result The representation of content and style in CNN are well separable. Therefore we can manipulate both representations independently to produce new, perceptually meaningful images. 46

46 Trade-Off Between Content And Style Matching The higher the ratio of α the content of the β picture resembles to the content of the content image and less to the style of the style image 47

47 The Effect of Matching The Content Representation in Different Layers of The Network On the lower layer of the network (conv2_2) the texture of the painting is blended over the photograph. On the higher layer of the network (conv4_2) the new picture looks as if the content of the original picture was preserved in the style of the painting. Therefore, usually the more appealing image are creating from matching the style from the higher layers. Both images were produced with a ratio α β =

48 Initialization of Gradient Decent The initial guess changes the output image! Initializing from a predefined image leads to one image (neglecting the stochasticity of the gradient-decent)! Image A initialized from content image. Image B initialized from style image. The last four images were initialized from white noise. Initializing from white noise gives us an infinite number of potential output images. Small Bias toward initial guess (A or B). 49

49 Photorealistic Style Transfer Style New York. Content London. The photo-realism doesn t fully preserved. 50

50 Discussion Slide 1 In this article it was demonstrated how to use feature representations from CNN to transfer image style between arbitrary images. Limitations resolution of the synthesized images the speed of generating an image depends linearly in the number of pixels (both for the optimization problem and the number of units in the CNN). In this article a 512x512 pixels images were generated with an Nvidia K40 GPU and it could take an hour. The meaning of this limitation is that the algorithm can t be used for online and interactive applications. 51

51 Discussion Slide 2 Synthesized images are sometimes subject to some lowlevel noise. This is less problematic for artistic style transfer, and more relevant when both content and style images are photographs, because the photorealism of the image is affected. The authors say the noise resembles the filters of units in the network, and suggest to develop a de-noising technique to post process the image after the optimization. The separation of image content from style is not a well defined problem. This is because it s hard to define what is style in an image. 52

52 Questions 53

DeCAF: a Deep Convolutional Activation Feature for Generic Visual Recognition

DeCAF: a Deep Convolutional Activation Feature for Generic Visual Recognition DeCAF: a Deep Convolutional Activation Feature for Generic Visual Recognition ECS 289G 10/06/2016 Authors: Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng and Trevor Darrell

More information

A Neural Algorithm of Artistic Style. Leon A. Gatys, Alexander S. Ecker, Matthias Bethge

A Neural Algorithm of Artistic Style. Leon A. Gatys, Alexander S. Ecker, Matthias Bethge A Neural Algorithm of Artistic Style Leon A. Gatys, Alexander S. Ecker, Matthias Bethge Presented by Shishir Mathur (1 Sept 2016) What is this paper This is the research paper behind Prisma It creates

More information

ImageNet Classification with Deep Convolutional Neural Networks

ImageNet Classification with Deep Convolutional Neural Networks ImageNet Classification with Deep Convolutional Neural Networks Alex Krizhevsky Ilya Sutskever Geoffrey Hinton University of Toronto Canada Paper with same name to appear in NIPS 2012 Main idea Architecture

More information

Deep Learning with Tensorflow AlexNet

Deep Learning with Tensorflow   AlexNet Machine Learning and Computer Vision Group Deep Learning with Tensorflow http://cvml.ist.ac.at/courses/dlwt_w17/ AlexNet Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton, "Imagenet classification

More information

Machine Learning. Deep Learning. Eric Xing (and Pengtao Xie) , Fall Lecture 8, October 6, Eric CMU,

Machine Learning. Deep Learning. Eric Xing (and Pengtao Xie) , Fall Lecture 8, October 6, Eric CMU, Machine Learning 10-701, Fall 2015 Deep Learning Eric Xing (and Pengtao Xie) Lecture 8, October 6, 2015 Eric Xing @ CMU, 2015 1 A perennial challenge in computer vision: feature engineering SIFT Spin image

More information

Deep Learning for Computer Vision with MATLAB By Jon Cherrie

Deep Learning for Computer Vision with MATLAB By Jon Cherrie Deep Learning for Computer Vision with MATLAB By Jon Cherrie 2015 The MathWorks, Inc. 1 Deep learning is getting a lot of attention "Dahl and his colleagues won $22,000 with a deeplearning system. 'We

More information

A FRAMEWORK OF EXTRACTING MULTI-SCALE FEATURES USING MULTIPLE CONVOLUTIONAL NEURAL NETWORKS. Kuan-Chuan Peng and Tsuhan Chen

A FRAMEWORK OF EXTRACTING MULTI-SCALE FEATURES USING MULTIPLE CONVOLUTIONAL NEURAL NETWORKS. Kuan-Chuan Peng and Tsuhan Chen A FRAMEWORK OF EXTRACTING MULTI-SCALE FEATURES USING MULTIPLE CONVOLUTIONAL NEURAL NETWORKS Kuan-Chuan Peng and Tsuhan Chen School of Electrical and Computer Engineering, Cornell University, Ithaca, NY

More information

CSE 559A: Computer Vision

CSE 559A: Computer Vision CSE 559A: Computer Vision Fall 2018: T-R: 11:30-1pm @ Lopata 101 Instructor: Ayan Chakrabarti (ayan@wustl.edu). Course Staff: Zhihao Xia, Charlie Wu, Han Liu http://www.cse.wustl.edu/~ayan/courses/cse559a/

More information

COMP9444 Neural Networks and Deep Learning 7. Image Processing. COMP9444 c Alan Blair, 2017

COMP9444 Neural Networks and Deep Learning 7. Image Processing. COMP9444 c Alan Blair, 2017 COMP9444 Neural Networks and Deep Learning 7. Image Processing COMP9444 17s2 Image Processing 1 Outline Image Datasets and Tasks Convolution in Detail AlexNet Weight Initialization Batch Normalization

More information

Towards Large-Scale Semantic Representations for Actionable Exploitation. Prof. Trevor Darrell UC Berkeley

Towards Large-Scale Semantic Representations for Actionable Exploitation. Prof. Trevor Darrell UC Berkeley Towards Large-Scale Semantic Representations for Actionable Exploitation Prof. Trevor Darrell UC Berkeley traditional surveillance sensor emerging crowd sensor Desired capabilities: spatio-temporal reconstruction

More information

Inception and Residual Networks. Hantao Zhang. Deep Learning with Python.

Inception and Residual Networks. Hantao Zhang. Deep Learning with Python. Inception and Residual Networks Hantao Zhang Deep Learning with Python https://en.wikipedia.org/wiki/residual_neural_network Deep Neural Network Progress from Large Scale Visual Recognition Challenge (ILSVRC)

More information

Part Localization by Exploiting Deep Convolutional Networks

Part Localization by Exploiting Deep Convolutional Networks Part Localization by Exploiting Deep Convolutional Networks Marcel Simon, Erik Rodner, and Joachim Denzler Computer Vision Group, Friedrich Schiller University of Jena, Germany www.inf-cv.uni-jena.de Abstract.

More information

Convolutional Neural Networks. Computer Vision Jia-Bin Huang, Virginia Tech

Convolutional Neural Networks. Computer Vision Jia-Bin Huang, Virginia Tech Convolutional Neural Networks Computer Vision Jia-Bin Huang, Virginia Tech Today s class Overview Convolutional Neural Network (CNN) Training CNN Understanding and Visualizing CNN Image Categorization:

More information

Using Machine Learning for Classification of Cancer Cells

Using Machine Learning for Classification of Cancer Cells Using Machine Learning for Classification of Cancer Cells Camille Biscarrat University of California, Berkeley I Introduction Cell screening is a commonly used technique in the development of new drugs.

More information

All You Want To Know About CNNs. Yukun Zhu

All You Want To Know About CNNs. Yukun Zhu All You Want To Know About CNNs Yukun Zhu Deep Learning Deep Learning Image from http://imgur.com/ Deep Learning Image from http://imgur.com/ Deep Learning Image from http://imgur.com/ Deep Learning Image

More information

Image Transformation via Neural Network Inversion

Image Transformation via Neural Network Inversion Image Transformation via Neural Network Inversion Asha Anoosheh Rishi Kapadia Jared Rulison Abstract While prior experiments have shown it is possible to approximately reconstruct inputs to a neural net

More information

Deep Neural Networks:

Deep Neural Networks: Deep Neural Networks: Part II Convolutional Neural Network (CNN) Yuan-Kai Wang, 2016 Web site of this course: http://pattern-recognition.weebly.com source: CNN for ImageClassification, by S. Lazebnik,

More information

COMP 551 Applied Machine Learning Lecture 16: Deep Learning

COMP 551 Applied Machine Learning Lecture 16: Deep Learning COMP 551 Applied Machine Learning Lecture 16: Deep Learning Instructor: Ryan Lowe (ryan.lowe@cs.mcgill.ca) Slides mostly by: Class web page: www.cs.mcgill.ca/~hvanho2/comp551 Unless otherwise noted, all

More information

Deep Learning for Computer Vision II

Deep Learning for Computer Vision II IIIT Hyderabad Deep Learning for Computer Vision II C. V. Jawahar Paradigm Shift Feature Extraction (SIFT, HoG, ) Part Models / Encoding Classifier Sparrow Feature Learning Classifier Sparrow L 1 L 2 L

More information

Deep Learning. Visualizing and Understanding Convolutional Networks. Christopher Funk. Pennsylvania State University.

Deep Learning. Visualizing and Understanding Convolutional Networks. Christopher Funk. Pennsylvania State University. Visualizing and Understanding Convolutional Networks Christopher Pennsylvania State University February 23, 2015 Some Slide Information taken from Pierre Sermanet (Google) presentation on and Computer

More information

A Neural Algorithm of Artistic Style. Leon A. Gatys, Alexander S. Ecker, Mattthias Bethge Presented by Weidi Xie (1st Oct 2015 )

A Neural Algorithm of Artistic Style. Leon A. Gatys, Alexander S. Ecker, Mattthias Bethge Presented by Weidi Xie (1st Oct 2015 ) A Neural Algorithm of Artistic Style Leon A. Gatys, Alexander S. Ecker, Mattthias Bethge Presented by Weidi Xie (1st Oct 2015 ) What does the paper do? 2 Create artistic images of high perceptual quality.

More information

arxiv: v1 [cs.cv] 22 Feb 2017

arxiv: v1 [cs.cv] 22 Feb 2017 Synthesising Dynamic Textures using Convolutional Neural Networks arxiv:1702.07006v1 [cs.cv] 22 Feb 2017 Christina M. Funke, 1, 2, 3, Leon A. Gatys, 1, 2, 4, Alexander S. Ecker 1, 2, 5 1, 2, 3, 6 and Matthias

More information

Scalable and Modularized RTL Compilation of Convolutional Neural Networks onto FPGA

Scalable and Modularized RTL Compilation of Convolutional Neural Networks onto FPGA Scalable and Modularized RTL Compilation of Convolutional Neural Networks onto FPGA Yufei Ma, Naveen Suda, Yu Cao, Jae-sun Seo, Sarma Vrudhula School of Electrical, Computer and Energy Engineering School

More information

Convolutional Neural Networks

Convolutional Neural Networks Lecturer: Barnabas Poczos Introduction to Machine Learning (Lecture Notes) Convolutional Neural Networks Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publications.

More information

Rich feature hierarchies for accurate object detection and semantic segmentation

Rich feature hierarchies for accurate object detection and semantic segmentation Rich feature hierarchies for accurate object detection and semantic segmentation Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik Presented by Pandian Raju and Jialin Wu Last class SGD for Document

More information

Convolutional Neural Networks + Neural Style Transfer. Justin Johnson 2/1/2017

Convolutional Neural Networks + Neural Style Transfer. Justin Johnson 2/1/2017 Convolutional Neural Networks + Neural Style Transfer Justin Johnson 2/1/2017 Outline Convolutional Neural Networks Convolution Pooling Feature Visualization Neural Style Transfer Feature Inversion Texture

More information

Kaggle Data Science Bowl 2017 Technical Report

Kaggle Data Science Bowl 2017 Technical Report Kaggle Data Science Bowl 2017 Technical Report qfpxfd Team May 11, 2017 1 Team Members Table 1: Team members Name E-Mail University Jia Ding dingjia@pku.edu.cn Peking University, Beijing, China Aoxue Li

More information

Spatial Localization and Detection. Lecture 8-1

Spatial Localization and Detection. Lecture 8-1 Lecture 8: Spatial Localization and Detection Lecture 8-1 Administrative - Project Proposals were due on Saturday Homework 2 due Friday 2/5 Homework 1 grades out this week Midterm will be in-class on Wednesday

More information

CNN Basics. Chongruo Wu

CNN Basics. Chongruo Wu CNN Basics Chongruo Wu Overview 1. 2. 3. Forward: compute the output of each layer Back propagation: compute gradient Updating: update the parameters with computed gradient Agenda 1. Forward Conv, Fully

More information

CS 2750: Machine Learning. Neural Networks. Prof. Adriana Kovashka University of Pittsburgh April 13, 2016

CS 2750: Machine Learning. Neural Networks. Prof. Adriana Kovashka University of Pittsburgh April 13, 2016 CS 2750: Machine Learning Neural Networks Prof. Adriana Kovashka University of Pittsburgh April 13, 2016 Plan for today Neural network definition and examples Training neural networks (backprop) Convolutional

More information

Deep Tracking: Biologically Inspired Tracking with Deep Convolutional Networks

Deep Tracking: Biologically Inspired Tracking with Deep Convolutional Networks Deep Tracking: Biologically Inspired Tracking with Deep Convolutional Networks Si Chen The George Washington University sichen@gwmail.gwu.edu Meera Hahn Emory University mhahn7@emory.edu Mentor: Afshin

More information

Return of the Devil in the Details: Delving Deep into Convolutional Nets

Return of the Devil in the Details: Delving Deep into Convolutional Nets Return of the Devil in the Details: Delving Deep into Convolutional Nets Ken Chatfield - Karen Simonyan - Andrea Vedaldi - Andrew Zisserman University of Oxford The Devil is still in the Details 2011 2014

More information

Perceptron: This is convolution!

Perceptron: This is convolution! Perceptron: This is convolution! v v v Shared weights v Filter = local perceptron. Also called kernel. By pooling responses at different locations, we gain robustness to the exact spatial location of image

More information

Machine Learning 13. week

Machine Learning 13. week Machine Learning 13. week Deep Learning Convolutional Neural Network Recurrent Neural Network 1 Why Deep Learning is so Popular? 1. Increase in the amount of data Thanks to the Internet, huge amount of

More information

Deep Learning in Visual Recognition. Thanks Da Zhang for the slides

Deep Learning in Visual Recognition. Thanks Da Zhang for the slides Deep Learning in Visual Recognition Thanks Da Zhang for the slides Deep Learning is Everywhere 2 Roadmap Introduction Convolutional Neural Network Application Image Classification Object Detection Object

More information

Computer Vision Lecture 16

Computer Vision Lecture 16 Computer Vision Lecture 16 Deep Learning for Object Categorization 14.01.2016 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de leibe@vision.rwth-aachen.de Announcements Seminar registration period

More information

Content-Based Image Recovery

Content-Based Image Recovery Content-Based Image Recovery Hong-Yu Zhou and Jianxin Wu National Key Laboratory for Novel Software Technology Nanjing University, China zhouhy@lamda.nju.edu.cn wujx2001@nju.edu.cn Abstract. We propose

More information

Classification of objects from Video Data (Group 30)

Classification of objects from Video Data (Group 30) Classification of objects from Video Data (Group 30) Sheallika Singh 12665 Vibhuti Mahajan 12792 Aahitagni Mukherjee 12001 M Arvind 12385 1 Motivation Video surveillance has been employed for a long time

More information

Photo-realistic Renderings for Machines Seong-heum Kim

Photo-realistic Renderings for Machines Seong-heum Kim Photo-realistic Renderings for Machines 20105034 Seong-heum Kim CS580 Student Presentations 2016.04.28 Photo-realistic Renderings for Machines Scene radiances Model descriptions (Light, Shape, Material,

More information

3D model classification using convolutional neural network

3D model classification using convolutional neural network 3D model classification using convolutional neural network JunYoung Gwak Stanford jgwak@cs.stanford.edu Abstract Our goal is to classify 3D models directly using convolutional neural network. Most of existing

More information

CS 229 Final Report: Artistic Style Transfer for Face Portraits

CS 229 Final Report: Artistic Style Transfer for Face Portraits CS 229 Final Report: Artistic Style Transfer for Face Portraits Daniel Hsu, Marcus Pan, Chen Zhu {dwhsu, mpanj, chen0908}@stanford.edu Dec 16, 2016 1 Introduction The goal of our project is to learn the

More information

Restricted Boltzmann Machines. Shallow vs. deep networks. Stacked RBMs. Boltzmann Machine learning: Unsupervised version

Restricted Boltzmann Machines. Shallow vs. deep networks. Stacked RBMs. Boltzmann Machine learning: Unsupervised version Shallow vs. deep networks Restricted Boltzmann Machines Shallow: one hidden layer Features can be learned more-or-less independently Arbitrary function approximator (with enough hidden units) Deep: two

More information

A Deep Learning Framework for Authorship Classification of Paintings

A Deep Learning Framework for Authorship Classification of Paintings A Deep Learning Framework for Authorship Classification of Paintings Kai-Lung Hua ( 花凱龍 ) Dept. of Computer Science and Information Engineering National Taiwan University of Science and Technology Taipei,

More information

YOLO9000: Better, Faster, Stronger

YOLO9000: Better, Faster, Stronger YOLO9000: Better, Faster, Stronger Date: January 24, 2018 Prepared by Haris Khan (University of Toronto) Haris Khan CSC2548: Machine Learning in Computer Vision 1 Overview 1. Motivation for one-shot object

More information

Object detection using Region Proposals (RCNN) Ernest Cheung COMP Presentation

Object detection using Region Proposals (RCNN) Ernest Cheung COMP Presentation Object detection using Region Proposals (RCNN) Ernest Cheung COMP790-125 Presentation 1 2 Problem to solve Object detection Input: Image Output: Bounding box of the object 3 Object detection using CNN

More information

Neural Networks. Single-layer neural network. CSE 446: Machine Learning Emily Fox University of Washington March 10, /10/2017

Neural Networks. Single-layer neural network. CSE 446: Machine Learning Emily Fox University of Washington March 10, /10/2017 3/0/207 Neural Networks Emily Fox University of Washington March 0, 207 Slides adapted from Ali Farhadi (via Carlos Guestrin and Luke Zettlemoyer) Single-layer neural network 3/0/207 Perceptron as a neural

More information

An Exploration of Computer Vision Techniques for Bird Species Classification

An Exploration of Computer Vision Techniques for Bird Species Classification An Exploration of Computer Vision Techniques for Bird Species Classification Anne L. Alter, Karen M. Wang December 15, 2017 Abstract Bird classification, a fine-grained categorization task, is a complex

More information

CS 1674: Intro to Computer Vision. Neural Networks. Prof. Adriana Kovashka University of Pittsburgh November 16, 2016

CS 1674: Intro to Computer Vision. Neural Networks. Prof. Adriana Kovashka University of Pittsburgh November 16, 2016 CS 1674: Intro to Computer Vision Neural Networks Prof. Adriana Kovashka University of Pittsburgh November 16, 2016 Announcements Please watch the videos I sent you, if you haven t yet (that s your reading)

More information

Deep Learning for Visual Manipulation and Synthesis

Deep Learning for Visual Manipulation and Synthesis Deep Learning for Visual Manipulation and Synthesis Jun-Yan Zhu 朱俊彦 UC Berkeley 2017/01/11 @ VALSE What is visual manipulation? Image Editing Program input photo User Input result Desired output: stay

More information

Caffe tutorial. Seong Joon Oh

Caffe tutorial. Seong Joon Oh Caffe tutorial Seong Joon Oh What is Caffe? Convolution Architecture For Feature Extraction (CAFFE) Open framework, models, and examples for deep learning 600+ citations, 100+ contributors, 7,000+ stars,

More information

Caffe2C: A Framework for Easy Implementation of CNN-based Mobile Applications

Caffe2C: A Framework for Easy Implementation of CNN-based Mobile Applications Caffe2C: A Framework for Easy Implementation of CNN-based Mobile Applications Ryosuke Tanno and Keiji Yanai Department of Informatics, The University of Electro-Communications, Tokyo 1. INTRODUCTION Deep

More information

Know your data - many types of networks

Know your data - many types of networks Architectures Know your data - many types of networks Fixed length representation Variable length representation Online video sequences, or samples of different sizes Images Specific architectures for

More information

Tutorial on Keras CAP ADVANCED COMPUTER VISION SPRING 2018 KISHAN S ATHREY

Tutorial on Keras CAP ADVANCED COMPUTER VISION SPRING 2018 KISHAN S ATHREY Tutorial on Keras CAP 6412 - ADVANCED COMPUTER VISION SPRING 2018 KISHAN S ATHREY Deep learning packages TensorFlow Google PyTorch Facebook AI research Keras Francois Chollet (now at Google) Chainer Company

More information

MACHINE LEARNING CLASSIFIERS ADVANTAGES AND CHALLENGES OF SELECTED METHODS

MACHINE LEARNING CLASSIFIERS ADVANTAGES AND CHALLENGES OF SELECTED METHODS MACHINE LEARNING CLASSIFIERS ADVANTAGES AND CHALLENGES OF SELECTED METHODS FRANK ORBEN, TECHNICAL SUPPORT / DEVELOPER IMAGE PROCESSING, STEMMER IMAGING OUTLINE Introduction Task: Classification Theory

More information

Semantic Segmentation

Semantic Segmentation Semantic Segmentation UCLA:https://goo.gl/images/I0VTi2 OUTLINE Semantic Segmentation Why? Paper to talk about: Fully Convolutional Networks for Semantic Segmentation. J. Long, E. Shelhamer, and T. Darrell,

More information

INTRODUCTION TO DEEP LEARNING

INTRODUCTION TO DEEP LEARNING INTRODUCTION TO DEEP LEARNING CONTENTS Introduction to deep learning Contents 1. Examples 2. Machine learning 3. Neural networks 4. Deep learning 5. Convolutional neural networks 6. Conclusion 7. Additional

More information

Convolutional Neural Networks

Convolutional Neural Networks NPFL114, Lecture 4 Convolutional Neural Networks Milan Straka March 25, 2019 Charles University in Prague Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics unless otherwise

More information

Rotation Invariance Neural Network

Rotation Invariance Neural Network Rotation Invariance Neural Network Shiyuan Li Abstract Rotation invariance and translate invariance have great values in image recognition. In this paper, we bring a new architecture in convolutional neural

More information

A performance comparison of Deep Learning frameworks on KNL

A performance comparison of Deep Learning frameworks on KNL A performance comparison of Deep Learning frameworks on KNL R. Zanella, G. Fiameni, M. Rorro Middleware, Data Management - SCAI - CINECA IXPUG Bologna, March 5, 2018 Table of Contents 1. Problem description

More information

Recurrent Neural Networks and Transfer Learning for Action Recognition

Recurrent Neural Networks and Transfer Learning for Action Recognition Recurrent Neural Networks and Transfer Learning for Action Recognition Andrew Giel Stanford University agiel@stanford.edu Ryan Diaz Stanford University ryandiaz@stanford.edu Abstract We have taken on the

More information

Real-time Object Detection CS 229 Course Project

Real-time Object Detection CS 229 Course Project Real-time Object Detection CS 229 Course Project Zibo Gong 1, Tianchang He 1, and Ziyi Yang 1 1 Department of Electrical Engineering, Stanford University December 17, 2016 Abstract Objection detection

More information

Deep Convolutional Neural Networks. Nov. 20th, 2015 Bruce Draper

Deep Convolutional Neural Networks. Nov. 20th, 2015 Bruce Draper Deep Convolutional Neural Networks Nov. 20th, 2015 Bruce Draper Background: Fully-connected single layer neural networks Feed-forward classification Trained through back-propagation Example Computer Vision

More information

Safety verification for deep neural networks

Safety verification for deep neural networks Safety verification for deep neural networks Marta Kwiatkowska Department of Computer Science, University of Oxford UC Berkeley, 8 th November 2016 Setting the scene Deep neural networks have achieved

More information

Proceedings of the International MultiConference of Engineers and Computer Scientists 2018 Vol I IMECS 2018, March 14-16, 2018, Hong Kong

Proceedings of the International MultiConference of Engineers and Computer Scientists 2018 Vol I IMECS 2018, March 14-16, 2018, Hong Kong , March 14-16, 2018, Hong Kong , March 14-16, 2018, Hong Kong , March 14-16, 2018, Hong Kong , March 14-16, 2018, Hong Kong TABLE I CLASSIFICATION ACCURACY OF DIFFERENT PRE-TRAINED MODELS ON THE TEST DATA

More information

Automatic detection of books based on Faster R-CNN

Automatic detection of books based on Faster R-CNN Automatic detection of books based on Faster R-CNN Beibei Zhu, Xiaoyu Wu, Lei Yang, Yinghua Shen School of Information Engineering, Communication University of China Beijing, China e-mail: zhubeibei@cuc.edu.cn,

More information

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun Presented by Tushar Bansal Objective 1. Get bounding box for all objects

More information

Regionlet Object Detector with Hand-crafted and CNN Feature

Regionlet Object Detector with Hand-crafted and CNN Feature Regionlet Object Detector with Hand-crafted and CNN Feature Xiaoyu Wang Research Xiaoyu Wang Research Ming Yang Horizon Robotics Shenghuo Zhu Alibaba Group Yuanqing Lin Baidu Overview of this section Regionlet

More information

Neural style transfer

Neural style transfer 1/32 Neural style transfer Victor Kitov v.v.kitov@yandex.ru 2/32 Neural style transfer Input: content image, style image. Style transfer - application of artistic style from style image to content image.

More information

CS230: Lecture 3 Various Deep Learning Topics

CS230: Lecture 3 Various Deep Learning Topics CS230: Lecture 3 Various Deep Learning Topics Kian Katanforoosh, Andrew Ng Today s outline We will learn how to: - Analyse a problem from a deep learning approach - Choose an architecture - Choose a loss

More information

REGION AVERAGE POOLING FOR CONTEXT-AWARE OBJECT DETECTION

REGION AVERAGE POOLING FOR CONTEXT-AWARE OBJECT DETECTION REGION AVERAGE POOLING FOR CONTEXT-AWARE OBJECT DETECTION Kingsley Kuan 1, Gaurav Manek 1, Jie Lin 1, Yuan Fang 1, Vijay Chandrasekhar 1,2 Institute for Infocomm Research, A*STAR, Singapore 1 Nanyang Technological

More information

arxiv: v1 [cs.cv] 6 Jul 2016

arxiv: v1 [cs.cv] 6 Jul 2016 arxiv:607.079v [cs.cv] 6 Jul 206 Deep CORAL: Correlation Alignment for Deep Domain Adaptation Baochen Sun and Kate Saenko University of Massachusetts Lowell, Boston University Abstract. Deep neural networks

More information

Object Detection Based on Deep Learning

Object Detection Based on Deep Learning Object Detection Based on Deep Learning Yurii Pashchenko AI Ukraine 2016, Kharkiv, 2016 Image classification (mostly what you ve seen) http://tutorial.caffe.berkeleyvision.org/caffe-cvpr15-detection.pdf

More information

Unsupervised Domain Adaptation by Backpropagation. Chih-Hui Ho, Xingyu Gu, Yuan Qi

Unsupervised Domain Adaptation by Backpropagation. Chih-Hui Ho, Xingyu Gu, Yuan Qi Unsupervised Domain Adaptation by Backpropagation Chih-Hui Ho, Xingyu Gu, Yuan Qi Problems Deep network: requires massive labeled training data. Labeled data: Available sometimes: Image recognition Speech

More information

Unsupervised learning in Vision

Unsupervised learning in Vision Chapter 7 Unsupervised learning in Vision The fields of Computer Vision and Machine Learning complement each other in a very natural way: the aim of the former is to extract useful information from visual

More information

CENG 783. Special topics in. Deep Learning. AlchemyAPI. Week 11. Sinan Kalkan

CENG 783. Special topics in. Deep Learning. AlchemyAPI. Week 11. Sinan Kalkan CENG 783 Special topics in Deep Learning AlchemyAPI Week 11 Sinan Kalkan TRAINING A CNN Fig: http://www.robots.ox.ac.uk/~vgg/practicals/cnn/ Feed-forward pass Note that this is written in terms of the

More information

Deep Face Recognition. Nathan Sun

Deep Face Recognition. Nathan Sun Deep Face Recognition Nathan Sun Why Facial Recognition? Picture ID or video tracking Higher Security for Facial Recognition Software Immensely useful to police in tracking suspects Your face will be an

More information

Apparel Classifier and Recommender using Deep Learning

Apparel Classifier and Recommender using Deep Learning Apparel Classifier and Recommender using Deep Learning Live Demo at: http://saurabhg.me/projects/tag-that-apparel Saurabh Gupta sag043@ucsd.edu Siddhartha Agarwal siagarwa@ucsd.edu Apoorve Dave a1dave@ucsd.edu

More information

Rich feature hierarchies for accurate object detection and semant

Rich feature hierarchies for accurate object detection and semant Rich feature hierarchies for accurate object detection and semantic segmentation Speaker: Yucong Shen 4/5/2018 Develop of Object Detection 1 DPM (Deformable parts models) 2 R-CNN 3 Fast R-CNN 4 Faster

More information

PARTIAL STYLE TRANSFER USING WEAKLY SUPERVISED SEMANTIC SEGMENTATION. Shin Matsuo Wataru Shimoda Keiji Yanai

PARTIAL STYLE TRANSFER USING WEAKLY SUPERVISED SEMANTIC SEGMENTATION. Shin Matsuo Wataru Shimoda Keiji Yanai PARTIAL STYLE TRANSFER USING WEAKLY SUPERVISED SEMANTIC SEGMENTATION Shin Matsuo Wataru Shimoda Keiji Yanai Department of Informatics, The University of Electro-Communications, Tokyo 1-5-1 Chofugaoka,

More information

11. Neural Network Regularization

11. Neural Network Regularization 11. Neural Network Regularization CS 519 Deep Learning, Winter 2016 Fuxin Li With materials from Andrej Karpathy, Zsolt Kira Preventing overfitting Approach 1: Get more data! Always best if possible! If

More information

Transferrable Representations for Visual Recognition

Transferrable Representations for Visual Recognition Transferrable Representations for Visual Recognition Jeffrey Donahue Electrical Engineering and Computer Sciences University of California at Berkeley Technical Report No. UCB/EECS-2017-106 http://www2.eecs.berkeley.edu/pubs/techrpts/2017/eecs-2017-106.html

More information

Real-Time Depth Estimation from 2D Images

Real-Time Depth Estimation from 2D Images Real-Time Depth Estimation from 2D Images Jack Zhu Ralph Ma jackzhu@stanford.edu ralphma@stanford.edu. Abstract ages. We explore the differences in training on an untrained network, and on a network pre-trained

More information

Deep Learning Basic Lecture - Complex Systems & Artificial Intelligence 2017/18 (VO) Asan Agibetov, PhD.

Deep Learning Basic Lecture - Complex Systems & Artificial Intelligence 2017/18 (VO) Asan Agibetov, PhD. Deep Learning 861.061 Basic Lecture - Complex Systems & Artificial Intelligence 2017/18 (VO) Asan Agibetov, PhD asan.agibetov@meduniwien.ac.at Medical University of Vienna Center for Medical Statistics,

More information

Deep Learning. Vladimir Golkov Technical University of Munich Computer Vision Group

Deep Learning. Vladimir Golkov Technical University of Munich Computer Vision Group Deep Learning Vladimir Golkov Technical University of Munich Computer Vision Group 1D Input, 1D Output target input 2 2D Input, 1D Output: Data Distribution Complexity Imagine many dimensions (data occupies

More information

CNNS FROM THE BASICS TO RECENT ADVANCES. Dmytro Mishkin Center for Machine Perception Czech Technical University in Prague

CNNS FROM THE BASICS TO RECENT ADVANCES. Dmytro Mishkin Center for Machine Perception Czech Technical University in Prague CNNS FROM THE BASICS TO RECENT ADVANCES Dmytro Mishkin Center for Machine Perception Czech Technical University in Prague ducha.aiki@gmail.com OUTLINE Short review of the CNN design Architecture progress

More information

Study of Residual Networks for Image Recognition

Study of Residual Networks for Image Recognition Study of Residual Networks for Image Recognition Mohammad Sadegh Ebrahimi Stanford University sadegh@stanford.edu Hossein Karkeh Abadi Stanford University hosseink@stanford.edu Abstract Deep neural networks

More information

GAN Related Works. CVPR 2018 & Selective Works in ICML and NIPS. Zhifei Zhang

GAN Related Works. CVPR 2018 & Selective Works in ICML and NIPS. Zhifei Zhang GAN Related Works CVPR 2018 & Selective Works in ICML and NIPS Zhifei Zhang Generative Adversarial Networks (GANs) 9/12/2018 2 Generative Adversarial Networks (GANs) Feedforward Backpropagation Real? z

More information

DL Tutorial. Xudong Cao

DL Tutorial. Xudong Cao DL Tutorial Xudong Cao Historical Line 1960s Perceptron 1980s MLP BP algorithm 2006 RBM unsupervised learning 2012 AlexNet ImageNet Comp. 2014 GoogleNet VGGNet ImageNet Comp. Rule based AI algorithm Game

More information

Depth Estimation from a Single Image Using a Deep Neural Network Milestone Report

Depth Estimation from a Single Image Using a Deep Neural Network Milestone Report Figure 1: The architecture of the convolutional network. Input: a single view image; Output: a depth map. 3 Related Work In [4] they used depth maps of indoor scenes produced by a Microsoft Kinect to successfully

More information

Overall Description. Goal: to improve spatial invariance to the input data. Translation, Rotation, Scale, Clutter, Elastic

Overall Description. Goal: to improve spatial invariance to the input data. Translation, Rotation, Scale, Clutter, Elastic Philippe Giguère Overall Description Goal: to improve spatial invariance to the input data Translation, Rotation, Scale, Clutter, Elastic How: add a learnable module which explicitly manipulate spatially

More information

Classifying Depositional Environments in Satellite Images

Classifying Depositional Environments in Satellite Images Classifying Depositional Environments in Satellite Images Alex Miltenberger and Rayan Kanfar Department of Geophysics School of Earth, Energy, and Environmental Sciences Stanford University 1 Introduction

More information

Structured Prediction using Convolutional Neural Networks

Structured Prediction using Convolutional Neural Networks Overview Structured Prediction using Convolutional Neural Networks Bohyung Han bhhan@postech.ac.kr Computer Vision Lab. Convolutional Neural Networks (CNNs) Structured predictions for low level computer

More information

Fully Convolutional Networks for Semantic Segmentation

Fully Convolutional Networks for Semantic Segmentation Fully Convolutional Networks for Semantic Segmentation Jonathan Long* Evan Shelhamer* Trevor Darrell UC Berkeley Chaim Ginzburg for Deep Learning seminar 1 Semantic Segmentation Define a pixel-wise labeling

More information

Supplementary material for the paper Are Sparse Representations Really Relevant for Image Classification?

Supplementary material for the paper Are Sparse Representations Really Relevant for Image Classification? Supplementary material for the paper Are Sparse Representations Really Relevant for Image Classification? Roberto Rigamonti, Matthew A. Brown, Vincent Lepetit CVLab, EPFL Lausanne, Switzerland firstname.lastname@epfl.ch

More information

INF 5860 Machine learning for image classification. Lecture 11: Visualization Anne Solberg April 4, 2018

INF 5860 Machine learning for image classification. Lecture 11: Visualization Anne Solberg April 4, 2018 INF 5860 Machine learning for image classification Lecture 11: Visualization Anne Solberg April 4, 2018 Reading material The lecture is based on papers: Deep Dream: https://research.googleblog.com/2015/06/inceptionism-goingdeeper-into-neural.html

More information

In Defense of Fully Connected Layers in Visual Representation Transfer

In Defense of Fully Connected Layers in Visual Representation Transfer In Defense of Fully Connected Layers in Visual Representation Transfer Chen-Lin Zhang, Jian-Hao Luo, Xiu-Shen Wei, Jianxin Wu National Key Laboratory for Novel Software Technology, Nanjing University,

More information

SEMANTIC COMPUTING. Lecture 8: Introduction to Deep Learning. TU Dresden, 7 December Dagmar Gromann International Center For Computational Logic

SEMANTIC COMPUTING. Lecture 8: Introduction to Deep Learning. TU Dresden, 7 December Dagmar Gromann International Center For Computational Logic SEMANTIC COMPUTING Lecture 8: Introduction to Deep Learning Dagmar Gromann International Center For Computational Logic TU Dresden, 7 December 2018 Overview Introduction Deep Learning General Neural Networks

More information

CMU Lecture 18: Deep learning and Vision: Convolutional neural networks. Teacher: Gianni A. Di Caro

CMU Lecture 18: Deep learning and Vision: Convolutional neural networks. Teacher: Gianni A. Di Caro CMU 15-781 Lecture 18: Deep learning and Vision: Convolutional neural networks Teacher: Gianni A. Di Caro DEEP, SHALLOW, CONNECTED, SPARSE? Fully connected multi-layer feed-forward perceptrons: More powerful

More information

ASCII Art Synthesis with Convolutional Networks

ASCII Art Synthesis with Convolutional Networks ASCII Art Synthesis with Convolutional Networks Osamu Akiyama Faculty of Medicine, Osaka University oakiyama1986@gmail.com 1 Introduction ASCII art is a type of graphic art that presents a picture with

More information

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications Learning Visual Semantics: Models, Massive Computation, and Innovative Applications Part II: Visual Features and Representations Liangliang Cao, IBM Watson Research Center Evolvement of Visual Features

More information