Lab meeting (Paper review session) Stacked Generative Adversarial Networks

Size: px
Start display at page:

Download "Lab meeting (Paper review session) Stacked Generative Adversarial Networks"

Transcription

1 Lab meeting (Paper review session) Stacked Generative Adversarial Networks Saehoon Kim (Ph. D. candidate) Machine Learning Group

2 Papers to be covered Stacked Generative Adversarial Networks X. Huang (Cornell), Y. Li (Cornell), O. Poursaeed (Cornell), J. Hopcroft (Cornell), S. Belongie (Cornell) arxiv: v1 [cs.cv] 13 Dec 2016 StackGAN: Text to Photo-realistic Synthesis with Stacked Generative Adversarial Networks H. Zhang (Rutgers Univ.) et al. arxiv: v1 [cs.cv] 10 Dec

3 Generative Adversarial Networks The generator and discriminator play the following twoplayer minimax game with value function The generator maps a latent space to a data space The discriminator represents the probability that the X came from the data rather than 2

4 An Theoretical Analysis of GANs [1] [1] Generative Adversarial Nets, I. J. Goodfellow, et al, NIPS 14 3

5 Practical Learning Techniques [1] To train the generative network, the objective function is slightly twisted (no theoretical analysis is guaranteed) [1] Improved techniques to train GANs, T. Salimans, et al, NIPS 16 4

6 Inception Score We apply the Inception model (GoogLeNet) to get the conditional label distribution We expect that a well-generated image has a conditional label distribution with low entropy We expect that the model to generate varied images the marginal with high entropy The following score is very natural to assess the quality of generative models 5

7 Deep Convolutional GANs (DCGANs) [1] 100-(4x4x1024) projection matrix Transposed convolution (a.k.a. deconvolution) [1] Unsupervised representation with deep convolutional GANs, A. Radford et al, ICLR 16 6

8 Transposed convolution [1] 7

9 Stacked Generative Adversarial Networks X. Huang, Y. Li, O. Poursaeed, J. Hopcroft, S. Belongie (Cornell University) In this paper we aim to leverage the powerful bottom-up discriminative representations to guide a top-down generative model. We propose a novel generative model named Stacked Generative Adversarial Networks (SGAN), which is trained to invert the hierarchical representations of a discriminative bottom-up deep network. Our model consists of a top-down stack of GANs, each trained to generate plausible lower-level representations, conditioned on higher level representations. A representation discriminator is introduced at each feature hierarchy to encourage the representation manifold of the generator to align with that of the bottom-up discriminative network, providing intermediate supervision. In addition, we introduce a conditional loss that encourages the use of conditional information from the layer above, and a novel entropy loss that maximizes a variational lower bound on the conditional entropy of generator outputs. To the best of our knowledge, the entropy loss is the first attempt to tackle the conditional model collapse problem that is common in conditional GANs. We first train each GAN of the stack independently, and then we train the stack end-to-end. Unlike the original GAN that uses a single noise vector to represent all the variations, our SGAN decomposes variations into multiple levels and gradually resolves uncertainties in the top-down generative process. 8

10 Hierarchical image generation Lower-level representation conditioned on higher-level representation 9

11 Stacked Generative Adversarial Network (SGAN) [Pre-trained Encoder] Convolution Pooling Convolution Pooling Fullyconnected input conv1 pool1 conv2 pool2 fc3 fc4 10

12 Stacked Generative Adversarial Network (SGAN) [Stacked Generators] Our goal is to train a top-down generator G that inverts E G consists of a top-down stack of generators G i which is trained to invert a bottom-up mapping E i The definition of each generator is defined as follows: 11

13 An overview of image generation New images can be sampled from SGAN by feeding random noise to each generator This is different from DCGAN, because multiple noise variables are considered to generate the single image Each generator can be designed by transposed convolution operators 12

14 An overview of SGAN 13

15 Training Discriminator [Standard Loss] A discriminator D i distinguishes generated representation h i from real representations h i The loss for the discriminator is defined as 14

16 Training Generator (1/3) [Adversarial Loss] They first train each GAN independently by using adv,indep L Gi They train them jointly in an end-to-end manner by using adv,joint L Gi 15

17 Training Generator (2/3) [Conditional Loss] They regularize the generator by feeding the generated lower-level representations back to the encoder They enforce the recovered representations to be similar to the original representations 16

18 Training Generator (3/3) [Entropy Loss] They encourage the generated representation h i to be sufficiently diverse when conditioned on h i+1 The conditional entropy H( h i h i+1 ) should be as high as possible They propose to maximize a variational lower bound on the conditional entropy 17

19 Experiments [Encoder] They use a small CNN as the encoder: conv1-pool1- conv2-pool2-fc3-fc4-softmax [Generator] The top GAN G 1 generates fc3 features from some random noise z 1, conditioned on label y The bottom GAN G 0 generates images from some random noise z 0, conditioned on fc3 features from GAN G 1 18

20 SVHN Results 19

21 CIFAR Results 20

22 Inception Scores (CIFAR-10) 21

23 Stack GAN: Text to Photo-realist Image Synthesis H. Zhang (Rutgers), T. Xu (Lehigh Univ.), H. Li (CUHK), S. Zhang (UNC), X. Huang (Lehigh Univ.), X. Wang (CUHK), D. Metaxas (Rutgers) In this paper, we propose stacked Generative Adversarial Networks (StackGAN) to generate photo-realistic images conditioned on text descriptions. The Stage-I GAN sketches the primitive shape and basic colors of the object based on the given text description, yielding Stage-I low resolution images. The Stage-II GAN takes Stage-I results and text descriptions as inputs, and generates high resolution images with photorealistic details. The Stage-II GAN is able to rectify defects and add compelling details with the refinement process. Samples generated by StackGAN are more plausible than those generated by existing approaches. Importantly, our StackGAN for the first time generates realistic images conditioned on only text descriptions, while state-of-the-art methods can generate at most images. To demonstrate the effectiveness of the proposed StackGAN, extensive experiments are conducted on CUB and Oxford-102 datasets. 22

24 Motivating Examples 23

25 The architecture of StackGAN 24

26 Stage-I GAN [Model Architecture] Transposed Conv. Feedforward NN LSTM or CNN with word embedding Dimension Reduction & Reshaped 25

27 Stage-II GAN [Model Architecture] Same with Stage-I generator Standard CNN Transposed Conv. Same with Stage-I discriminator 26

28 Examples 27

29 Comparison between Stage I and II 28

Progress on Generative Adversarial Networks

Progress on Generative Adversarial Networks Progress on Generative Adversarial Networks Wangmeng Zuo Vision Perception and Cognition Centre Harbin Institute of Technology Content Image generation: problem formulation Three issues about GAN Discriminate

More information

GAN Related Works. CVPR 2018 & Selective Works in ICML and NIPS. Zhifei Zhang

GAN Related Works. CVPR 2018 & Selective Works in ICML and NIPS. Zhifei Zhang GAN Related Works CVPR 2018 & Selective Works in ICML and NIPS Zhifei Zhang Generative Adversarial Networks (GANs) 9/12/2018 2 Generative Adversarial Networks (GANs) Feedforward Backpropagation Real? z

More information

GAN Frontiers/Related Methods

GAN Frontiers/Related Methods GAN Frontiers/Related Methods Improving GAN Training Improved Techniques for Training GANs (Salimans, et. al 2016) CSC 2541 (07/10/2016) Robin Swanson (robin@cs.toronto.edu) Training GANs is Difficult

More information

arxiv: v1 [cs.cv] 5 Jul 2017

arxiv: v1 [cs.cv] 5 Jul 2017 AlignGAN: Learning to Align Cross- Images with Conditional Generative Adversarial Networks Xudong Mao Department of Computer Science City University of Hong Kong xudonmao@gmail.com Qing Li Department of

More information

Alternatives to Direct Supervision

Alternatives to Direct Supervision CreativeAI: Deep Learning for Graphics Alternatives to Direct Supervision Niloy Mitra Iasonas Kokkinos Paul Guerrero Nils Thuerey Tobias Ritschel UCL UCL UCL TUM UCL Timetable Theory and Basics State of

More information

(University Improving of Montreal) Generative Adversarial Networks with Denoising Feature Matching / 17

(University Improving of Montreal) Generative Adversarial Networks with Denoising Feature Matching / 17 Improving Generative Adversarial Networks with Denoising Feature Matching David Warde-Farley 1 Yoshua Bengio 1 1 University of Montreal, ICLR,2017 Presenter: Bargav Jayaraman Outline 1 Introduction 2 Background

More information

Generative Modeling with Convolutional Neural Networks. Denis Dus Data Scientist at InData Labs

Generative Modeling with Convolutional Neural Networks. Denis Dus Data Scientist at InData Labs Generative Modeling with Convolutional Neural Networks Denis Dus Data Scientist at InData Labs What we will discuss 1. 2. 3. 4. Discriminative vs Generative modeling Convolutional Neural Networks How to

More information

Unsupervised Learning

Unsupervised Learning Deep Learning for Graphics Unsupervised Learning Niloy Mitra Iasonas Kokkinos Paul Guerrero Vladimir Kim Kostas Rematas Tobias Ritschel UCL UCL/Facebook UCL Adobe Research U Washington UCL Timetable Niloy

More information

arxiv: v1 [cs.cv] 8 Jan 2019

arxiv: v1 [cs.cv] 8 Jan 2019 GILT: Generating Images from Long Text Ori Bar El, Ori Licht, Netanel Yosephian Tel-Aviv University {oribarel, oril, yosephian}@mail.tau.ac.il arxiv:1901.02404v1 [cs.cv] 8 Jan 2019 Abstract Creating an

More information

Introduction to Generative Adversarial Networks

Introduction to Generative Adversarial Networks Introduction to Generative Adversarial Networks Luke de Oliveira Vai Technologies Lawrence Berkeley National Laboratory @lukede0 @lukedeo lukedeo@vaitech.io https://ldo.io 1 Outline Why Generative Modeling?

More information

Generative Adversarial Text to Image Synthesis

Generative Adversarial Text to Image Synthesis Generative Adversarial Text to Image Synthesis Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee Presented by: Jingyao Zhan Contents Introduction Related Work Method

More information

Controllable Generative Adversarial Network

Controllable Generative Adversarial Network Controllable Generative Adversarial Network arxiv:1708.00598v2 [cs.lg] 12 Sep 2017 Minhyeok Lee 1 and Junhee Seok 1 1 School of Electrical Engineering, Korea University, 145 Anam-ro, Seongbuk-gu, Seoul,

More information

arxiv: v1 [cs.cv] 7 Mar 2018

arxiv: v1 [cs.cv] 7 Mar 2018 Accepted as a conference paper at the European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN) 2018 Inferencing Based on Unsupervised Learning of Disentangled

More information

arxiv: v1 [cs.cv] 17 Nov 2016

arxiv: v1 [cs.cv] 17 Nov 2016 Inverting The Generator Of A Generative Adversarial Network arxiv:1611.05644v1 [cs.cv] 17 Nov 2016 Antonia Creswell BICV Group Bioengineering Imperial College London ac2211@ic.ac.uk Abstract Anil Anthony

More information

Autoencoder. Representation learning (related to dictionary learning) Both the input and the output are x

Autoencoder. Representation learning (related to dictionary learning) Both the input and the output are x Deep Learning 4 Autoencoder, Attention (spatial transformer), Multi-modal learning, Neural Turing Machine, Memory Networks, Generative Adversarial Net Jian Li IIIS, Tsinghua Autoencoder Autoencoder Unsupervised

More information

GENERATIVE ADVERSARIAL NETWORKS (GAN) Presented by Omer Stein and Moran Rubin

GENERATIVE ADVERSARIAL NETWORKS (GAN) Presented by Omer Stein and Moran Rubin GENERATIVE ADVERSARIAL NETWORKS (GAN) Presented by Omer Stein and Moran Rubin GENERATIVE MODEL Given a training dataset, x, try to estimate the distribution, Pdata(x) Explicitly or Implicitly (GAN) Explicitly

More information

CS230: Lecture 4 Attacking Networks with Adversarial Examples - Generative Adversarial Networks

CS230: Lecture 4 Attacking Networks with Adversarial Examples - Generative Adversarial Networks Go to www.menti.com and use the code 91 41 37 CS230: Lecture 4 Attacking Networks with Adversarial Examples - Generative Adversarial Networks Kian Katanforoosh Today s outline I. Attacking NNs with Adversarial

More information

Introduction to Generative Adversarial Networks

Introduction to Generative Adversarial Networks Introduction to Generative Adversarial Networks Ian Goodfellow, OpenAI Research Scientist NIPS 2016 Workshop on Adversarial Training Barcelona, 2016-12-9 Adversarial Training A phrase whose usage is in

More information

arxiv: v1 [cs.cv] 6 Sep 2018

arxiv: v1 [cs.cv] 6 Sep 2018 arxiv:1809.01890v1 [cs.cv] 6 Sep 2018 Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto

More information

StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks

StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks 1 StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Senior Member, IEEE, Xiaogang Wang, Member, IEEE, Xiaolei Huang, Member,

More information

SYNTHESIS OF IMAGES BY TWO-STAGE GENERATIVE ADVERSARIAL NETWORKS. Qiang Huang, Philip J.B. Jackson, Mark D. Plumbley, Wenwu Wang

SYNTHESIS OF IMAGES BY TWO-STAGE GENERATIVE ADVERSARIAL NETWORKS. Qiang Huang, Philip J.B. Jackson, Mark D. Plumbley, Wenwu Wang SYNTHESIS OF IMAGES BY TWO-STAGE GENERATIVE ADVERSARIAL NETWORKS Qiang Huang, Philip J.B. Jackson, Mark D. Plumbley, Wenwu Wang Centre for Vision, Speech and Signal Processing University of Surrey, Guildford,

More information

Generative Adversarial Networks (GANs) Ian Goodfellow, Research Scientist MLSLP Keynote, San Francisco

Generative Adversarial Networks (GANs) Ian Goodfellow, Research Scientist MLSLP Keynote, San Francisco Generative Adversarial Networks (GANs) Ian Goodfellow, Research Scientist MLSLP Keynote, San Francisco 2016-09-13 Generative Modeling Density estimation Sample generation Training examples Model samples

More information

GANs for Exploiting Unlabeled Data. Presented by: Uriya Pesso Nimrod Gilboa Markevich

GANs for Exploiting Unlabeled Data. Presented by: Uriya Pesso Nimrod Gilboa Markevich GANs for Exploiting Unlabeled Data Improved Techniques for Training GANs Learning from Simulated and Unsupervised Images through Adversarial Training Presented by: Uriya Pesso Nimrod Gilboa Markevich [

More information

Introduction to GAN. Generative Adversarial Networks. Junheng(Jeff) Hao

Introduction to GAN. Generative Adversarial Networks. Junheng(Jeff) Hao Introduction to GAN Generative Adversarial Networks Junheng(Jeff) Hao Adversarial Training is the coolest thing since sliced bread. -- Yann LeCun Roadmap 1. Generative Modeling 2. GAN 101: What is GAN?

More information

Learning Photographic Image Synthesis With Cascaded Refinement Networks. Jonathan Louie Huy Doan Siavash Motalebi

Learning Photographic Image Synthesis With Cascaded Refinement Networks. Jonathan Louie Huy Doan Siavash Motalebi Learning Photographic Image Synthesis With Cascaded Refinement Networks Jonathan Louie Huy Doan Siavash Motalebi Introduction and Background Intro We are researching and re-implementing Photographic Image

More information

Convolutional Neural Networks + Neural Style Transfer. Justin Johnson 2/1/2017

Convolutional Neural Networks + Neural Style Transfer. Justin Johnson 2/1/2017 Convolutional Neural Networks + Neural Style Transfer Justin Johnson 2/1/2017 Outline Convolutional Neural Networks Convolution Pooling Feature Visualization Neural Style Transfer Feature Inversion Texture

More information

Inverting The Generator Of A Generative Adversarial Network

Inverting The Generator Of A Generative Adversarial Network 1 Inverting The Generator Of A Generative Adversarial Network Antonia Creswell and Anil A Bharath, Imperial College London arxiv:1802.05701v1 [cs.cv] 15 Feb 2018 Abstract Generative adversarial networks

More information

An Empirical Study of Generative Adversarial Networks for Computer Vision Tasks

An Empirical Study of Generative Adversarial Networks for Computer Vision Tasks An Empirical Study of Generative Adversarial Networks for Computer Vision Tasks Report for Undergraduate Project - CS396A Vinayak Tantia (Roll No: 14805) Guide: Prof Gaurav Sharma CSE, IIT Kanpur, India

More information

Implicit generative models: dual vs. primal approaches

Implicit generative models: dual vs. primal approaches Implicit generative models: dual vs. primal approaches Ilya Tolstikhin MPI for Intelligent Systems ilya@tue.mpg.de Machine Learning Summer School 2017 Tübingen, Germany Contents 1. Unsupervised generative

More information

One Network to Solve Them All Solving Linear Inverse Problems using Deep Projection Models

One Network to Solve Them All Solving Linear Inverse Problems using Deep Projection Models One Network to Solve Them All Solving Linear Inverse Problems using Deep Projection Models [Supplemental Materials] 1. Network Architecture b ref b ref +1 We now describe the architecture of the networks

More information

Generative Adversarial Network

Generative Adversarial Network Generative Adversarial Network Many slides from NIPS 2014 Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio Generative adversarial

More information

Supplementary Material: Unsupervised Domain Adaptation for Face Recognition in Unlabeled Videos

Supplementary Material: Unsupervised Domain Adaptation for Face Recognition in Unlabeled Videos Supplementary Material: Unsupervised Domain Adaptation for Face Recognition in Unlabeled Videos Kihyuk Sohn 1 Sifei Liu 2 Guangyu Zhong 3 Xiang Yu 1 Ming-Hsuan Yang 2 Manmohan Chandraker 1,4 1 NEC Labs

More information

Deep Learning for Visual Manipulation and Synthesis

Deep Learning for Visual Manipulation and Synthesis Deep Learning for Visual Manipulation and Synthesis Jun-Yan Zhu 朱俊彦 UC Berkeley 2017/01/11 @ VALSE What is visual manipulation? Image Editing Program input photo User Input result Desired output: stay

More information

arxiv: v4 [cs.lg] 1 May 2018

arxiv: v4 [cs.lg] 1 May 2018 Controllable Generative Adversarial Network arxiv:1708.00598v4 [cs.lg] 1 May 2018 Minhyeok Lee School of Electrical Engineering Korea University Seoul, Korea 02841 suam6409@korea.ac.kr Abstract Junhee

More information

Autoencoders. Stephen Scott. Introduction. Basic Idea. Stacked AE. Denoising AE. Sparse AE. Contractive AE. Variational AE GAN.

Autoencoders. Stephen Scott. Introduction. Basic Idea. Stacked AE. Denoising AE. Sparse AE. Contractive AE. Variational AE GAN. Stacked Denoising Sparse Variational (Adapted from Paul Quint and Ian Goodfellow) Stacked Denoising Sparse Variational Autoencoding is training a network to replicate its input to its output Applications:

More information

Defense Data Generation in Distributed Deep Learning System Se-Yoon Oh / ADD-IDAR

Defense Data Generation in Distributed Deep Learning System Se-Yoon Oh / ADD-IDAR Defense Data Generation in Distributed Deep Learning System Se-Yoon Oh / 2017. 10. 31 syoh@add.re.kr Page 1/36 Overview 1. Introduction 2. Data Generation Synthesis 3. Distributed Deep Learning 4. Conclusions

More information

Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform. Xintao Wang Ke Yu Chao Dong Chen Change Loy

Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform. Xintao Wang Ke Yu Chao Dong Chen Change Loy Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform Xintao Wang Ke Yu Chao Dong Chen Change Loy Problem enlarge 4 times Low-resolution image High-resolution image Previous

More information

arxiv: v1 [cs.cv] 16 Jul 2017

arxiv: v1 [cs.cv] 16 Jul 2017 enerative adversarial network based on resnet for conditional image restoration Paper: jc*-**-**-****: enerative Adversarial Network based on Resnet for Conditional Image Restoration Meng Wang, Huafeng

More information

Adversarially Learned Inference

Adversarially Learned Inference Institut des algorithmes d apprentissage de Montréal Adversarially Learned Inference Aaron Courville CIFAR Fellow Université de Montréal Joint work with: Vincent Dumoulin, Ishmael Belghazi, Olivier Mastropietro,

More information

Learning to generate with adversarial networks

Learning to generate with adversarial networks Learning to generate with adversarial networks Gilles Louppe June 27, 2016 Problem statement Assume training samples D = {x x p data, x X } ; We want a generative model p model that can draw new samples

More information

Mode Regularized Generative Adversarial Networks

Mode Regularized Generative Adversarial Networks Mode Regularized Generative Adversarial Networks Tong Che 1 Yanran Li 2 Athul Paul Jacob 3 Yoshua Bengio 1 Wenjie Li 2 1 Montreal Institute for Learning Algorithms, Universite de Montreal, Montreal, Canada

More information

Deep Generative Models and a Probabilistic Programming Library

Deep Generative Models and a Probabilistic Programming Library Deep Generative Models and a Probabilistic Programming Library Discriminative (Deep) Learning Learn a (differentiable) function mapping from input to output x f(x; θ) y Gradient back-propagation Generative

More information

arxiv: v2 [cs.cv] 26 Mar 2017

arxiv: v2 [cs.cv] 26 Mar 2017 TAC-GAN Text Conditioned Auxiliary Classifier Generative Adversarial Network arxiv:1703.06412v2 [cs.cv] 26 ar 2017 Ayushman Dash 1 John Gamboa 1 Sheraz Ahmed 3 arcus Liwicki 14 uhammad Zeshan Afzal 12

More information

Lecture 19: Generative Adversarial Networks

Lecture 19: Generative Adversarial Networks Lecture 19: Generative Adversarial Networks Roger Grosse 1 Introduction Generative modeling is a type of machine learning where the aim is to model the distribution that a given set of data (e.g. images,

More information

arxiv: v1 [cs.cv] 1 Aug 2017

arxiv: v1 [cs.cv] 1 Aug 2017 Deep Generative Adversarial Neural Networks for Realistic Prostate Lesion MRI Synthesis Andy Kitchen a, Jarrel Seah b a,* Independent Researcher b STAT Innovations Pty. Ltd., PO Box 274, Ashburton VIC

More information

Stacking VAE and GAN for Context-aware Text-to-Image Generation

Stacking VAE and GAN for Context-aware Text-to-Image Generation 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM) Stacking VAE and GAN for Context-aware Text-to-Image Generation Chenrui Zhang and Yuxin Peng* Institute of Computer Science and

More information

CNN for Low Level Image Processing. Huanjing Yue

CNN for Low Level Image Processing. Huanjing Yue CNN for Low Level Image Processing Huanjing Yue 2017.11 1 Deep Learning for Image Restoration General formulation: min Θ L( x, x) s. t. x = F(y; Θ) Loss function Parameters to be learned Key issues The

More information

Generative Adversarial Networks (GANs)

Generative Adversarial Networks (GANs) Generative Adversarial Networks (GANs) Hossein Azizpour Most of the slides are courtesy of Dr. Ian Goodfellow (Research Scientist at OpenAI) and from his presentation at NIPS 2016 tutorial Note. I am generally

More information

TGANv2: Efficient Training of Large Models for Video Generation with Multiple Subsampling Layers

TGANv2: Efficient Training of Large Models for Video Generation with Multiple Subsampling Layers TGANv2: Efficient Training of Large Models for Video Generation with Multiple Subsampling Layers Masaki Saito Shunta Saito Preferred Networks, Inc. {msaito, shunta}@preferred.jp arxiv:1811.09245v1 [cs.cv]

More information

Introduction to GAN. Generative Adversarial Networks. Junheng(Jeff) Hao

Introduction to GAN. Generative Adversarial Networks. Junheng(Jeff) Hao Introduction to GAN Generative Adversarial Networks Junheng(Jeff) Hao Adversarial Training is the coolest thing since sliced bread. -- Yann LeCun Roadmap 1. Generative Modeling 2. GAN 101: What is GAN?

More information

Human Pose Estimation with Deep Learning. Wei Yang

Human Pose Estimation with Deep Learning. Wei Yang Human Pose Estimation with Deep Learning Wei Yang Applications Understand Activities Family Robots American Heist (2014) - The Bank Robbery Scene 2 What do we need to know to recognize a crime scene? 3

More information

arxiv: v1 [cs.cv] 20 Sep 2018

arxiv: v1 [cs.cv] 20 Sep 2018 C4Synth: Cross-Caption Cycle-Consistent Text-to-Image Synthesis K J Joseph Arghya Pal Sailaja Rajanala Vineeth N Balasubramanian IIT Hyderabad, India arxiv:1809.10238v1 [cs.cv] 20 Sep 2018 cs17m18p100001@iith.ac.in

More information

Lecture 3 GANs and Their Applications in Image Generation

Lecture 3 GANs and Their Applications in Image Generation Lecture 3 GANs and Their Applications in Image Generation Lin ZHANG, PhD School of Software Engineering Tongji University Fall 2017 Outline Introduction Theoretical Part Application Part Existing Implementations

More information

Machine Learning. Deep Learning. Eric Xing (and Pengtao Xie) , Fall Lecture 8, October 6, Eric CMU,

Machine Learning. Deep Learning. Eric Xing (and Pengtao Xie) , Fall Lecture 8, October 6, Eric CMU, Machine Learning 10-701, Fall 2015 Deep Learning Eric Xing (and Pengtao Xie) Lecture 8, October 6, 2015 Eric Xing @ CMU, 2015 1 A perennial challenge in computer vision: feature engineering SIFT Spin image

More information

arxiv: v1 [cs.ne] 11 Jun 2018

arxiv: v1 [cs.ne] 11 Jun 2018 Generative Adversarial Network Architectures For Image Synthesis Using Capsule Networks arxiv:1806.03796v1 [cs.ne] 11 Jun 2018 Yash Upadhyay University of Minnesota, Twin Cities Minneapolis, MN, 55414

More information

Auto-encoder with Adversarially Regularized Latent Variables

Auto-encoder with Adversarially Regularized Latent Variables Information Engineering Express International Institute of Applied Informatics 2017, Vol.3, No.3, P.11 20 Auto-encoder with Adversarially Regularized Latent Variables for Semi-Supervised Learning Ryosuke

More information

Tempered Adversarial Networks

Tempered Adversarial Networks Mehdi S. M. Sajjadi 1 2 Giambattista Parascandolo 1 2 Arash Mehrjou 1 Bernhard Schölkopf 1 Abstract Generative adversarial networks (GANs) have been shown to produce realistic samples from high-dimensional

More information

DOMAIN-ADAPTIVE GENERATIVE ADVERSARIAL NETWORKS FOR SKETCH-TO-PHOTO INVERSION

DOMAIN-ADAPTIVE GENERATIVE ADVERSARIAL NETWORKS FOR SKETCH-TO-PHOTO INVERSION DOMAIN-ADAPTIVE GENERATIVE ADVERSARIAL NETWORKS FOR SKETCH-TO-PHOTO INVERSION Yen-Cheng Liu 1, Wei-Chen Chiu 2, Sheng-De Wang 1, and Yu-Chiang Frank Wang 1 1 Graduate Institute of Electrical Engineering,

More information

Stacked Generative Adversarial Networks

Stacked Generative Adversarial Networks Stacked Generative Adversarial Networks Xun Huang 1 Yixuan Li 2 Omid Poursaeed 2 John Hopcroft 1 Serge Belongie 1,3 1 Department of Computer Science, Cornell University 2 School of Electrical and Computer

More information

Image Restoration with Deep Generative Models

Image Restoration with Deep Generative Models Image Restoration with Deep Generative Models Raymond A. Yeh *, Teck-Yian Lim *, Chen Chen, Alexander G. Schwing, Mark Hasegawa-Johnson, Minh N. Do Department of Electrical and Computer Engineering, University

More information

DOMAIN-ADAPTIVE GENERATIVE ADVERSARIAL NETWORKS FOR SKETCH-TO-PHOTO INVERSION

DOMAIN-ADAPTIVE GENERATIVE ADVERSARIAL NETWORKS FOR SKETCH-TO-PHOTO INVERSION 2017 IEEE INTERNATIONAL WORKSHOP ON MACHINE LEARNING FOR SIGNAL PROCESSING, SEPT. 25 28, 2017, TOKYO, JAPAN DOMAIN-ADAPTIVE GENERATIVE ADVERSARIAL NETWORKS FOR SKETCH-TO-PHOTO INVERSION Yen-Cheng Liu 1,

More information

Machine Learning 13. week

Machine Learning 13. week Machine Learning 13. week Deep Learning Convolutional Neural Network Recurrent Neural Network 1 Why Deep Learning is so Popular? 1. Increase in the amount of data Thanks to the Internet, huge amount of

More information

Deep Fakes using Generative Adversarial Networks (GAN)

Deep Fakes using Generative Adversarial Networks (GAN) Deep Fakes using Generative Adversarial Networks (GAN) Tianxiang Shen UCSD La Jolla, USA tis038@eng.ucsd.edu Ruixian Liu UCSD La Jolla, USA rul188@eng.ucsd.edu Ju Bai UCSD La Jolla, USA jub010@eng.ucsd.edu

More information

Deep Learning For Video Classification. Presented by Natalie Carlebach & Gil Sharon

Deep Learning For Video Classification. Presented by Natalie Carlebach & Gil Sharon Deep Learning For Video Classification Presented by Natalie Carlebach & Gil Sharon Overview Of Presentation Motivation Challenges of video classification Common datasets 4 different methods presented in

More information

Data Set Extension with Generative Adversarial Nets

Data Set Extension with Generative Adversarial Nets Department of Artificial Intelligence University of Groningen, The Netherlands Data Set Extension with Generative Adversarial Nets Master s Thesis Luuk Boulogne S2366681 Primary supervisor: Secondary supervisor:

More information

LEARNING TO GENERATE CHAIRS WITH CONVOLUTIONAL NEURAL NETWORKS

LEARNING TO GENERATE CHAIRS WITH CONVOLUTIONAL NEURAL NETWORKS LEARNING TO GENERATE CHAIRS WITH CONVOLUTIONAL NEURAL NETWORKS Alexey Dosovitskiy, Jost Tobias Springenberg and Thomas Brox University of Freiburg Presented by: Shreyansh Daftry Visual Learning and Recognition

More information

Show, Discriminate, and Tell: A Discriminatory Image Captioning Model with Deep Neural Networks

Show, Discriminate, and Tell: A Discriminatory Image Captioning Model with Deep Neural Networks Show, Discriminate, and Tell: A Discriminatory Image Captioning Model with Deep Neural Networks Zelun Luo Department of Computer Science Stanford University zelunluo@stanford.edu Te-Lin Wu Department of

More information

Variational Autoencoders. Sargur N. Srihari

Variational Autoencoders. Sargur N. Srihari Variational Autoencoders Sargur N. srihari@cedar.buffalo.edu Topics 1. Generative Model 2. Standard Autoencoder 3. Variational autoencoders (VAE) 2 Generative Model A variational autoencoder (VAE) is a

More information

Paired 3D Model Generation with Conditional Generative Adversarial Networks

Paired 3D Model Generation with Conditional Generative Adversarial Networks Accepted to 3D Reconstruction in the Wild Workshop European Conference on Computer Vision (ECCV) 2018 Paired 3D Model Generation with Conditional Generative Adversarial Networks Cihan Öngün Alptekin Temizel

More information

SiftingGAN: Generating and Sifting Labeled Samples to Improve the Remote Sensing Image Scene Classification Baseline in vitro

SiftingGAN: Generating and Sifting Labeled Samples to Improve the Remote Sensing Image Scene Classification Baseline in vitro 1 SiftingGAN: Generating and Sifting Labeled Samples to Improve the Remote Sensing Image Scene Classification Baseline in vitro Dongao Ma, Ping Tang, and Lijun Zhao arxiv:1809.04985v4 [cs.cv] 30 Nov 2018

More information

Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks Supplementary Material

Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks Supplementary Material Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks Supplementary Material Emily Denton Dept. of Computer Science Courant Institute New York University Soumith Chintala Arthur

More information

Autoencoding Beyond Pixels Using a Learned Similarity Metric

Autoencoding Beyond Pixels Using a Learned Similarity Metric Autoencoding Beyond Pixels Using a Learned Similarity Metric International Conference on Machine Learning, 2016 Anders Boesen Lindbo Larsen, Hugo Larochelle, Søren Kaae Sønderby, Ole Winther Technical

More information

INF 5860 Machine learning for image classification. Lecture 11: Visualization Anne Solberg April 4, 2018

INF 5860 Machine learning for image classification. Lecture 11: Visualization Anne Solberg April 4, 2018 INF 5860 Machine learning for image classification Lecture 11: Visualization Anne Solberg April 4, 2018 Reading material The lecture is based on papers: Deep Dream: https://research.googleblog.com/2015/06/inceptionism-goingdeeper-into-neural.html

More information

AttentionNet for Accurate Localization and Detection of Objects. (To appear in ICCV 2015)

AttentionNet for Accurate Localization and Detection of Objects. (To appear in ICCV 2015) AttentionNet for Accurate Localization and Detection of Objects. (To appear in ICCV 2015) Donggeun Yoo, Sunggyun Park, Joon-Young Lee, Anthony Paek, In So Kweon. State-of-the-art frameworks for object

More information

Restricted Boltzmann Machines. Shallow vs. deep networks. Stacked RBMs. Boltzmann Machine learning: Unsupervised version

Restricted Boltzmann Machines. Shallow vs. deep networks. Stacked RBMs. Boltzmann Machine learning: Unsupervised version Shallow vs. deep networks Restricted Boltzmann Machines Shallow: one hidden layer Features can be learned more-or-less independently Arbitrary function approximator (with enough hidden units) Deep: two

More information

Know your data - many types of networks

Know your data - many types of networks Architectures Know your data - many types of networks Fixed length representation Variable length representation Online video sequences, or samples of different sizes Images Specific architectures for

More information

Overall Description. Goal: to improve spatial invariance to the input data. Translation, Rotation, Scale, Clutter, Elastic

Overall Description. Goal: to improve spatial invariance to the input data. Translation, Rotation, Scale, Clutter, Elastic Philippe Giguère Overall Description Goal: to improve spatial invariance to the input data Translation, Rotation, Scale, Clutter, Elastic How: add a learnable module which explicitly manipulate spatially

More information

Index. Springer Nature Switzerland AG 2019 B. Moons et al., Embedded Deep Learning,

Index. Springer Nature Switzerland AG 2019 B. Moons et al., Embedded Deep Learning, Index A Algorithmic noise tolerance (ANT), 93 94 Application specific instruction set processors (ASIPs), 115 116 Approximate computing application level, 95 circuits-levels, 93 94 DAS and DVAS, 107 110

More information

CENG 783. Special topics in. Deep Learning. AlchemyAPI. Week 11. Sinan Kalkan

CENG 783. Special topics in. Deep Learning. AlchemyAPI. Week 11. Sinan Kalkan CENG 783 Special topics in Deep Learning AlchemyAPI Week 11 Sinan Kalkan TRAINING A CNN Fig: http://www.robots.ox.ac.uk/~vgg/practicals/cnn/ Feed-forward pass Note that this is written in terms of the

More information

Generative Adversarial Network: a Brief Introduction. Lili Mou

Generative Adversarial Network: a Brief Introduction. Lili Mou Generative Adversarial Network: a Brief Introduction Lili Mou doublepower.mou@gmail.com Outline Generative adversarial net Conditional generative adversarial net Deep generative image models using Laplacian

More information

HENet: A Highly Efficient Convolutional Neural. Networks Optimized for Accuracy, Speed and Storage

HENet: A Highly Efficient Convolutional Neural. Networks Optimized for Accuracy, Speed and Storage HENet: A Highly Efficient Convolutional Neural Networks Optimized for Accuracy, Speed and Storage Qiuyu Zhu Shanghai University zhuqiuyu@staff.shu.edu.cn Ruixin Zhang Shanghai University chriszhang96@shu.edu.cn

More information

arxiv: v2 [cs.cv] 2 Dec 2017

arxiv: v2 [cs.cv] 2 Dec 2017 Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networks Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jiebo Luo Department of Computer Science, University of Rochester,

More information

RECENT years have witnessed the rapid growth of image. SSDH: Semi-supervised Deep Hashing for Large Scale Image Retrieval

RECENT years have witnessed the rapid growth of image. SSDH: Semi-supervised Deep Hashing for Large Scale Image Retrieval SSDH: Semi-supervised Deep Hashing for Large Scale Image Retrieval Jian Zhang, Yuxin Peng, and Junchao Zhang arxiv:607.08477v [cs.cv] 28 Jul 206 Abstract The hashing methods have been widely used for efficient

More information

arxiv: v1 [cs.cv] 4 Feb 2019

arxiv: v1 [cs.cv] 4 Feb 2019 Realistic Image Generation using Region-phrase Attention arxiv:1902.05395v1 [cs.cv] 4 Feb 2019 Wanming Huang University of Technology, Sydney wanming.huang@student.uts.edu.au Abstract The Generative Adversarial

More information

arxiv: v1 [stat.ml] 19 Aug 2017

arxiv: v1 [stat.ml] 19 Aug 2017 Semi-supervised Conditional GANs Kumar Sricharan 1, Raja Bala 1, Matthew Shreve 1, Hui Ding 1, Kumar Saketh 2, and Jin Sun 1 1 Interactive and Analytics Lab, Palo Alto Research Center, Palo Alto, CA 2

More information

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio Presented

More information

arxiv: v3 [cs.cv] 30 Mar 2018

arxiv: v3 [cs.cv] 30 Mar 2018 Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networks Wei Xiong Wenhan Luo Lin Ma Wei Liu Jiebo Luo Tencent AI Lab University of Rochester {wxiong5,jluo}@cs.rochester.edu

More information

Learning to Match. Jun Xu, Zhengdong Lu, Tianqi Chen, Hang Li

Learning to Match. Jun Xu, Zhengdong Lu, Tianqi Chen, Hang Li Learning to Match Jun Xu, Zhengdong Lu, Tianqi Chen, Hang Li 1. Introduction The main tasks in many applications can be formalized as matching between heterogeneous objects, including search, recommendation,

More information

Introduction to Generative Models (and GANs)

Introduction to Generative Models (and GANs) Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com Nov. 2017 Figures adapted from NIPS 2016 Tutorial Generative Adversarial Networks Generative Models: Learning the Distributions

More information

GENERATIVE ADVERSARIAL NETWORK-BASED VIR-

GENERATIVE ADVERSARIAL NETWORK-BASED VIR- GENERATIVE ADVERSARIAL NETWORK-BASED VIR- TUAL TRY-ON WITH CLOTHING REGION Shizuma Kubo, Yusuke Iwasawa, and Yutaka Matsuo The University of Tokyo Bunkyo-ku, Japan {kubo, iwasawa, matsuo}@weblab.t.u-tokyo.ac.jp

More information

Spatial Localization and Detection. Lecture 8-1

Spatial Localization and Detection. Lecture 8-1 Lecture 8: Spatial Localization and Detection Lecture 8-1 Administrative - Project Proposals were due on Saturday Homework 2 due Friday 2/5 Homework 1 grades out this week Midterm will be in-class on Wednesday

More information

Bilinear Models for Fine-Grained Visual Recognition

Bilinear Models for Fine-Grained Visual Recognition Bilinear Models for Fine-Grained Visual Recognition Subhransu Maji College of Information and Computer Sciences University of Massachusetts, Amherst Fine-grained visual recognition Example: distinguish

More information

Convolutional Neural Networks: Applications and a short timeline. 7th Deep Learning Meetup Kornel Kis Vienna,

Convolutional Neural Networks: Applications and a short timeline. 7th Deep Learning Meetup Kornel Kis Vienna, Convolutional Neural Networks: Applications and a short timeline 7th Deep Learning Meetup Kornel Kis Vienna, 1.12.2016. Introduction Currently a master student Master thesis at BME SmartLab Started deep

More information

ECE 599/692 Deep Learning. Lecture 12 GAN - Introduction

ECE 599/692 Deep Learning. Lecture 12 GAN - Introduction ECE 599/692 Deep Learning Lecture 12 AN - Introduction Hairong Qi, onzalez Family Professor Electrical Engineering and Computer Science University of Tennessee, Knoxville http://www.eecs.utk.edu/faculty/qi

More information

Deep Generative Models Variational Autoencoders

Deep Generative Models Variational Autoencoders Deep Generative Models Variational Autoencoders Sudeshna Sarkar 5 April 2017 Generative Nets Generative models that represent probability distributions over multiple variables in some way. Directed Generative

More information

Inception Network Overview. David White CS793

Inception Network Overview. David White CS793 Inception Network Overview David White CS793 So, Leonardo DiCaprio dreams about dreaming... https://m.media-amazon.com/images/m/mv5bmjaxmzy3njcxnf5bml5banbnxkftztcwnti5otm0mw@@._v1_sy1000_cr0,0,675,1 000_AL_.jpg

More information

DEEP STRUCTURED OUTPUT LEARNING FOR UNCONSTRAINED TEXT RECOGNITION

DEEP STRUCTURED OUTPUT LEARNING FOR UNCONSTRAINED TEXT RECOGNITION DEEP STRUCTURED OUTPUT LEARNING FOR UNCONSTRAINED TEXT RECOGNITION Max Jaderberg, Karen Simonyan, Andrea Vedaldi, Andrew Zisserman Visual Geometry Group, Department Engineering Science, University of Oxford,

More information

From attribute-labels to faces: face generation using a conditional generative adversarial network

From attribute-labels to faces: face generation using a conditional generative adversarial network From attribute-labels to faces: face generation using a conditional generative adversarial network Yaohui Wang 1,2, Antitza Dantcheva 1,2, and Francois Bremond 1,2 1 Inria, Sophia Antipolis, France 2 Université

More information

Generative Adversarial Networks (GANs) Based on slides from Ian Goodfellow s NIPS 2016 tutorial

Generative Adversarial Networks (GANs) Based on slides from Ian Goodfellow s NIPS 2016 tutorial Generative Adversarial Networks (GANs) Based on slides from Ian Goodfellow s NIPS 2016 tutorial Generative Modeling Density estimation Sample generation Training examples Model samples Next Video Frame

More information

Conditional DCGAN For Anime Avatar Generation

Conditional DCGAN For Anime Avatar Generation Conditional DCGAN For Anime Avatar Generation Wang Hang School of Electronic Information and Electrical Engineering Shanghai Jiao Tong University Shanghai 200240, China Email: wang hang@sjtu.edu.cn Abstract

More information