Lip Movement Synthesis from Text
|
|
|
- Charla Caldwell
- 5 years ago
- Views:
Transcription
1 Lip Movement Synthesis from Text 1 1 Department of Computer Science and Engineering Indian Institute of Technology, Kanpur July 20, 2017 (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
2 Outline 1 Objective and Motivation 2 Prerequisite Knowledge Generative Adversarial Networks Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks Generating Videos with Scene Dynamics Generative Adversarial Text to Image Synthesis 3 Approach Video Prepossessing Basic Video Generation Network Basic Video Generation with Text Embedding Network Modified Video Generation with Embedding 4 Dataset Experiments 5 Result Visualization (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
3 Objective and Motivation Lip Reading Figure: Lip Reading Procedure (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur)July 20, / 33
4 Objective and Motivation Lip Writing Figure: Lip Writing Procedure (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
5 Objective and Motivation Lip Writing Figure: Lip Writing Procedure Hallucinating lip movement for new words Feature Vector for Lip Reading Tasks (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
6 Prerequisite Knowledge Generative Adversarial Network An Unsupervised Machine Learning algorithm implemented by two neural networks Generator and Discriminator who compete against each other in a zero-sum game framework min G max V (D, G) = E x p D data (x)[log(d(x)] + E z pz (z)[log(1 D(G(z)))] (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
7 Prerequisite Knowledge Deep Convolution Generative Adversarial Network DCGAN was the first attempt at implementing GAN in a Deep Convolutional framework. (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
8 1 Discriminator Training Get image data from the dataset. Find the Cross Entropy loss from the data through the discriminator with a true label. Generate a sample from the generator. Find the Cross Entropy loss from the generated data through the discriminator with a false label. Backpropogate the loss through the discriminator update the discriminator parameters. 2 Generator Training Find the Cross Entropy loss from the generated data through the discriminator with a true label. Backpropogate the loss in the discriminator and find the loss at the image level representation. Backpropogate the above calculated image level loss through the generator network and update its parameters. (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
9 Prerequisite Knowledge Generating Videos with Scene Dynamics G(z) = m(z) f (z) + (1 m(z)) b(z) (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
10 Prerequisite Knowledge Generative Adversarial Text to Image Synthesis (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
11 Approach Video Prepossessing Figure: Dataset Preprocessing steps (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
12 Approach Basic Video Generation Network The lip movement videos did not have any background to them and the only dynamic aspect to them was the lip movement. We simplified the network by just having the Foreground generation Stream of the VideoGAN framework. The training procedure was the standard GAN training procedure. (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
13 Figure: Basic Video Generation Network Generator and Discriminator (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
14 Approach Basic Video Generation with Text Embedding Network For video generation from text embedding we first set up a model which was just an amalgamation of our basic video generator model and Scott Reed s method of appending the embeddings. The embedding is up sampled to 128 sized vector using a fully-connected layer which is then passed through a LeakyReLU layer. This embedding is then appended to the initial noise vector. The discriminator is also updated from the base model for the new task. At the layer when the spatio-temporal dimension of the discriminator is ,the text embedding is again upsampled to 128 dimensions passed through a LeakyReLU layer and then replicated and appended to the discriminator so as to make the new dimension ( ) 4 4. (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
15 Figure: Basic Video Generation with Text Embedding Generator Discriminator (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
16 Basic Video Generation with Text Embedding Network Training Procedure For the Discriminator 1 From the database get video frames, their corresponding text embeddings and a set of fake database videos having different text embeddings. 2 Calculate the error for the batch in the following way. Get error from database video with the corresponding text embedding with label true. Get error from the generated video and the text embedding with label false. Get error from mismatched data video and text embedding with label false. 3 Use this error to backpropogate it through the discriminator network and update the Discriminator parameters. (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
17 Basic Video Generation with Text Embedding Network Training Procedure For the Generator 1 Use the generated video in discriminator training with the text embedding and find the error with the true label. 2 This error is then backpropogated through the discriminator network to find the error at the video level representation. 3 This video level error used for the generator network. Using this error we backpropogate the error through the Generator network and update its parameters. (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
18 Approach Modified Video Generation with Embedding The results generated from the basic model, though were decipherable as lip-movement, they were blurry. We expanded upon the basic model made some changes in the generator and discriminator models as well as made some changes in the training procedure. (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
19 Modified Video Generation with Embedding Generator Figure: Modified Generator (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
20 Modified Video Generation with Embedding Discriminator Figure: Modified Discriminator (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
21 Modified Video Generation with Embedding Network Changes in Training 1 We sampled from a Spherical Gaussian rather than a Uniform distribution for sampling for the generator. 2 Replace ReLU layers with LeakyReLU in both generator and discriminator. 3 Rather than using two target labels (0,1) for false and true we use soft labels (0-0.3) for false and ( ) for true. This leads to better training of the generator and discriminator. 4 The Discriminator was training and moving towards 0 error soon which was causing the Generator to go haywire during training. To avoid this we added Dropout layers in both generator and discriminator for better training. (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
22 Dataset Experiments Grid Dataset The dataset has 34 users saying sentences in the format <command><color ><preposition ><letter ><digit ><adverb >like place blue at F 9 now. Type Number of Words Words command 4 bin, lay, place, set color 4 blue, green, red, white preposition 4 at, by, in, with letter 25 A-Z excluding W digit adverb 4 again, now, please, soon (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
23 Various Datasets for Generation Sub Sampling Dataset: Took the 75 frames of the video, sub sampled 32 frames from it at regular intervals and used the full text embedding associated with them. Multi Word Dataset: Broke down the 2 second videos into 2 parts of almost equal size according to the frames in which the words are spoken. The 2 videos were sub sampled for 32 frames with their corresponding word embedding. One Word Dataset: Comprised of the frames of people saying a single word which were super sampled from the corpus videos with one word embedding. (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
24 Results Basic Video Generation with Sub Sampling Dataset Figure: Basic Video Generation with Sub Sampling Dataset (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
25 Results Basic Embedding model with Sub Sampling Dataset Figure: Basic Embedding model with Sub Sampling Dataset (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
26 Results Modified Embedding Model with Sub Sampling Dataset Figure: Modified Embedding Model with Sub Sampling Dataset (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
27 Results Modified Embedding Model with Multi Word Dataset Figure: Modified Embedding Model with Multi Word Dataset (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
28 Results Modified Embedding Model with One Word Dataset Figure: Modified Embedding Model with One Word Dataset (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
29 Quantitative Results Structural Similarity Index SSIM is Structural Similarity Index introduced in 2004 Z. Wang et.al It measures the similarity in structure of images. The SSIM index is defined as SSIM(x, y) = (2µ xµ y + c 1 )(2σ xy + c 2 ) (µ 2 x + µ 2 y + c 1 )(σ 2 x + σ 2 y + c 2 ) (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
30 Word SSIM Score Word SSIM Score Word SSIM Score a in set again j seven at k sil b l six bin lay soon blue m sp by n t c nine three d now two e o u eight one v f p white five place with four please x g q y green r z h red zero (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
31 Similar Lip Movement Words Word1 Word2 Real Videos Generated Videos u blue a e b bin blue two blue bin in nine Different Lip Movement Words Word1 Word2 Real Videos Generated Videos four d seven t one e four k set place seven place at five Table: SSIM score between Similar and Different Lip Movement Words (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
32 Qualitative Results Figure: Four Eight M (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
33 Qualitative Results Figure: Five Blue B (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
34 Thank You Any Questions? (1Department of Computer Science Lipand Movement Engineering Synthesis Indianfrom Institute Textof Technology, Kanpur) July 20, / 33
(University Improving of Montreal) Generative Adversarial Networks with Denoising Feature Matching / 17
Improving Generative Adversarial Networks with Denoising Feature Matching David Warde-Farley 1 Yoshua Bengio 1 1 University of Montreal, ICLR,2017 Presenter: Bargav Jayaraman Outline 1 Introduction 2 Background
GENERATIVE ADVERSARIAL NETWORKS (GAN) Presented by Omer Stein and Moran Rubin
GENERATIVE ADVERSARIAL NETWORKS (GAN) Presented by Omer Stein and Moran Rubin GENERATIVE MODEL Given a training dataset, x, try to estimate the distribution, Pdata(x) Explicitly or Implicitly (GAN) Explicitly
Generative Adversarial Text to Image Synthesis
Generative Adversarial Text to Image Synthesis Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee Presented by: Jingyao Zhan Contents Introduction Related Work Method
Generative Adversarial Network
Generative Adversarial Network Many slides from NIPS 2014 Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio Generative adversarial
Generative Modeling with Convolutional Neural Networks. Denis Dus Data Scientist at InData Labs
Generative Modeling with Convolutional Neural Networks Denis Dus Data Scientist at InData Labs What we will discuss 1. 2. 3. 4. Discriminative vs Generative modeling Convolutional Neural Networks How to
Alternatives to Direct Supervision
CreativeAI: Deep Learning for Graphics Alternatives to Direct Supervision Niloy Mitra Iasonas Kokkinos Paul Guerrero Nils Thuerey Tobias Ritschel UCL UCL UCL TUM UCL Timetable Theory and Basics State of
arxiv: v1 [cs.cv] 17 Nov 2016
![arxiv: v1 [cs.cv] 17 Nov 2016](/thumbs/86/93859848.jpg "arxiv: v1 [cs.cv] 17 Nov 2016")
Inverting The Generator Of A Generative Adversarial Network arxiv:1611.05644v1 [cs.cv] 17 Nov 2016 Antonia Creswell BICV Group Bioengineering Imperial College London ac2211@ic.ac.uk Abstract Anil Anthony
Autoencoders. Stephen Scott. Introduction. Basic Idea. Stacked AE. Denoising AE. Sparse AE. Contractive AE. Variational AE GAN.
Stacked Denoising Sparse Variational (Adapted from Paul Quint and Ian Goodfellow) Stacked Denoising Sparse Variational Autoencoding is training a network to replicate its input to its output Applications:
Generative Adversarial Nets. Priyanka Mehta Sudhanshu Srivastava
Generative Adversarial Nets Priyanka Mehta Sudhanshu Srivastava Outline What is a GAN? How does GAN work? Newer Architectures Applications of GAN Future possible applications Generative Adversarial Networks
Image Restoration with Deep Generative Models
Image Restoration with Deep Generative Models Raymond A. Yeh *, Teck-Yian Lim *, Chen Chen, Alexander G. Schwing, Mark Hasegawa-Johnson, Minh N. Do Department of Electrical and Computer Engineering, University
From attribute-labels to faces: face generation using a conditional generative adversarial network
From attribute-labels to faces: face generation using a conditional generative adversarial network Yaohui Wang 1,2, Antitza Dantcheva 1,2, and Francois Bremond 1,2 1 Inria, Sophia Antipolis, France 2 Université
An Empirical Study of Generative Adversarial Networks for Computer Vision Tasks
An Empirical Study of Generative Adversarial Networks for Computer Vision Tasks Report for Undergraduate Project - CS396A Vinayak Tantia (Roll No: 14805) Guide: Prof Gaurav Sharma CSE, IIT Kanpur, India
Unsupervised Learning
Deep Learning for Graphics Unsupervised Learning Niloy Mitra Iasonas Kokkinos Paul Guerrero Vladimir Kim Kostas Rematas Tobias Ritschel UCL UCL/Facebook UCL Adobe Research U Washington UCL Timetable Niloy
arxiv: v2 [cs.cv] 14 May 2018
![arxiv: v2 [cs.cv] 14 May 2018](/thumbs/79/79495201.jpg "arxiv: v2 [cs.cv] 14 May 2018")
ContextVP: Fully Context-Aware Video Prediction Wonmin Byeon 1234, Qin Wang 1, Rupesh Kumar Srivastava 3, and Petros Koumoutsakos 1 arxiv:1710.08518v2 [cs.cv] 14 May 2018 Abstract Video prediction models
arxiv: v1 [cs.cv] 1 Aug 2017
![arxiv: v1 [cs.cv] 1 Aug 2017](/thumbs/87/97260731.jpg "arxiv: v1 [cs.cv] 1 Aug 2017")
Deep Generative Adversarial Neural Networks for Realistic Prostate Lesion MRI Synthesis Andy Kitchen a, Jarrel Seah b a,* Independent Researcher b STAT Innovations Pty. Ltd., PO Box 274, Ashburton VIC
Lab meeting (Paper review session) Stacked Generative Adversarial Networks
Lab meeting (Paper review session) Stacked Generative Adversarial Networks 2017. 02. 01. Saehoon Kim (Ph. D. candidate) Machine Learning Group Papers to be covered Stacked Generative Adversarial Networks
GANs for Exploiting Unlabeled Data. Presented by: Uriya Pesso Nimrod Gilboa Markevich
GANs for Exploiting Unlabeled Data Improved Techniques for Training GANs Learning from Simulated and Unsupervised Images through Adversarial Training Presented by: Uriya Pesso Nimrod Gilboa Markevich [
CS 6501: Deep Learning for Computer Graphics. Training Neural Networks II. Connelly Barnes
CS 6501: Deep Learning for Computer Graphics Training Neural Networks II Connelly Barnes Overview Preprocessing Initialization Vanishing/exploding gradients problem Batch normalization Dropout Additional
19: Inference and learning in Deep Learning
10-708: Probabilistic Graphical Models 10-708, Spring 2017 19: Inference and learning in Deep Learning Lecturer: Zhiting Hu Scribes: Akash Umakantha, Ryan Williamson 1 Classes of Deep Generative Models
CS230: Lecture 4 Attacking Networks with Adversarial Examples - Generative Adversarial Networks
Go to www.menti.com and use the code 91 41 37 CS230: Lecture 4 Attacking Networks with Adversarial Examples - Generative Adversarial Networks Kian Katanforoosh Today s outline I. Attacking NNs with Adversarial
AdaDepth: Unsupervised Content Congruent Adaptation for Depth Estimation
AdaDepth: Unsupervised Content Congruent Adaptation for Depth Estimation Introduction Supplementary material In the supplementary material, we present additional qualitative results of the proposed AdaDepth
arxiv: v1 [cs.cv] 8 Jan 2019
![arxiv: v1 [cs.cv] 8 Jan 2019](/thumbs/94/118236696.jpg "arxiv: v1 [cs.cv] 8 Jan 2019")
GILT: Generating Images from Long Text Ori Bar El, Ori Licht, Netanel Yosephian Tel-Aviv University {oribarel, oril, yosephian}@mail.tau.ac.il arxiv:1901.02404v1 [cs.cv] 8 Jan 2019 Abstract Creating an
Generative Adversarial Network: a Brief Introduction. Lili Mou
Generative Adversarial Network: a Brief Introduction Lili Mou doublepower.mou@gmail.com Outline Generative adversarial net Conditional generative adversarial net Deep generative image models using Laplacian
TGANv2: Efficient Training of Large Models for Video Generation with Multiple Subsampling Layers
TGANv2: Efficient Training of Large Models for Video Generation with Multiple Subsampling Layers Masaki Saito Shunta Saito Preferred Networks, Inc. {msaito, shunta}@preferred.jp arxiv:1811.09245v1 [cs.cv]
Lecture 3 GANs and Their Applications in Image Generation
Lecture 3 GANs and Their Applications in Image Generation Lin ZHANG, PhD School of Software Engineering Tongji University Fall 2017 Outline Introduction Theoretical Part Application Part Existing Implementations
arxiv: v2 [cs.cv] 19 Apr 2019
![arxiv: v2 [cs.cv] 19 Apr 2019](/thumbs/96/129293683.jpg "arxiv: v2 [cs.cv] 19 Apr 2019")
arxiv:1809.04789v2 [cs.cv] 19 Apr 2019 Deep Learning-based Image Super-Resolution Considering Quantitative and Perceptual Quality Jun-Ho Choi, Jun-Hyuk Kim, Manri Cheon, and Jong-Seok Lee School of Integrated
Real-time Gesture Pattern Classification with IMU Data
Real-time Gesture Pattern Classification with IMU Data Alex Fu Stanford University Computer Science Department alexfu@stanford.edu Yangyang Yu Stanford University Electrical Engineering Department yyu10@stanford.edu
SiftingGAN: Generating and Sifting Labeled Samples to Improve the Remote Sensing Image Scene Classification Baseline in vitro
1 SiftingGAN: Generating and Sifting Labeled Samples to Improve the Remote Sensing Image Scene Classification Baseline in vitro Dongao Ma, Ping Tang, and Lijun Zhao arxiv:1809.04985v4 [cs.cv] 30 Nov 2018
Conditional DCGAN For Anime Avatar Generation
Conditional DCGAN For Anime Avatar Generation Wang Hang School of Electronic Information and Electrical Engineering Shanghai Jiao Tong University Shanghai 200240, China Email: wang hang@sjtu.edu.cn Abstract
Lecture 19: Generative Adversarial Networks
Lecture 19: Generative Adversarial Networks Roger Grosse 1 Introduction Generative modeling is a type of machine learning where the aim is to model the distribution that a given set of data (e.g. images,
Inverting The Generator Of A Generative Adversarial Network
1 Inverting The Generator Of A Generative Adversarial Network Antonia Creswell and Anil A Bharath, Imperial College London arxiv:1802.05701v1 [cs.cv] 15 Feb 2018 Abstract Generative adversarial networks
Multi-Modal Generative Adversarial Networks
Multi-Modal Generative Adversarial Networks By MATAN BEN-YOSEF Under the supervision of PROF. DAPHNA WEINSHALL Faculty of Computer Science and Engineering THE HEBREW UNIVERSITY OF JERUSALEM A thesis submitted
S+U Learning through ANs - Pranjit Kalita
S+U Learning through ANs - Pranjit Kalita - (from paper) Learning from Simulated and Unsupervised Images through Adversarial Training - Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda
GAN Frontiers/Related Methods
GAN Frontiers/Related Methods Improving GAN Training Improved Techniques for Training GANs (Salimans, et. al 2016) CSC 2541 (07/10/2016) Robin Swanson (robin@cs.toronto.edu) Training GANs is Difficult
Reconstructing Pore Networks Using Generative Adversarial Networks
Reconstructing Pore Networks Using Generative Adversarial Networks Kelly Guan (kmguan@stanford.edu) I. INTRODUCTION Understanding fluid flow in porous media at the microscale is relevant to many fields,
CS230: Deep Learning Winter Quarter 2018 Stanford University
: Deep Learning Winter Quarter 08 Stanford University Midterm Examination 80 minutes Problem Full Points Your Score Multiple Choice 7 Short Answers 3 Coding 7 4 Backpropagation 5 Universal Approximation
Implicit generative models: dual vs. primal approaches
Implicit generative models: dual vs. primal approaches Ilya Tolstikhin MPI for Intelligent Systems ilya@tue.mpg.de Machine Learning Summer School 2017 Tübingen, Germany Contents 1. Unsupervised generative
arxiv: v1 [cs.ne] 11 Jun 2018
![arxiv: v1 [cs.ne] 11 Jun 2018](/thumbs/80/80526645.jpg "arxiv: v1 [cs.ne] 11 Jun 2018")
Generative Adversarial Network Architectures For Image Synthesis Using Capsule Networks arxiv:1806.03796v1 [cs.ne] 11 Jun 2018 Yash Upadhyay University of Minnesota, Twin Cities Minneapolis, MN, 55414
GANViz: A Visual Analytics Approach to Understand the Adversarial Game. Junpeng Wang, Liang Gou, Hao Yang, and Han-Wei Shen, Member, IEEE
JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 1 GANViz: A Visual Analytics Approach to Understand the Adversarial Game Junpeng Wang, Liang Gou, Hao Yang, and Han-Wei Shen, Member, IEEE
Analysis and Synthesis of Texture
Analysis and Synthesis of Texture CMPE 264: Image Analysis and Computer Vision Spring 02, Hai Tao 31/5/02 Extracting image structure by filter banks Q Represent image textures using the responses of a
Data Set Extension with Generative Adversarial Nets
Department of Artificial Intelligence University of Groningen, The Netherlands Data Set Extension with Generative Adversarial Nets Master s Thesis Luuk Boulogne S2366681 Primary supervisor: Secondary supervisor:
arxiv: v2 [cs.cv] 11 Jul 2018
![arxiv: v2 [cs.cv] 11 Jul 2018](/thumbs/94/119557907.jpg "arxiv: v2 [cs.cv] 11 Jul 2018")
DSSLIC: Deep Semantic Segmentation-based Layered Image Compression Mohammad Akbari, Jie Liang School of Engineering Science, Simon Fraser University, Canada akbari@sfu.ca, jiel@sfu.ca Jingning Han Google
Maximum Differentiation Competition: Direct Comparison of Discriminability Models
Maximum Differentiation Competition: Direct Comparison of Discriminability Models Zhou Wang & Eero P. Simoncelli Howard Hughes Medical Institute, Center for Neural Science, and Courant Institute for Mathematical
Introduction to Generative Adversarial Networks
Introduction to Generative Adversarial Networks Ian Goodfellow, OpenAI Research Scientist NIPS 2016 Workshop on Adversarial Training Barcelona, 2016-12-9 Adversarial Training A phrase whose usage is in
Text to Image Synthesis Using Generative Adversarial Networks
Text to Image Synthesis Using Generative Adversarial Networks arxiv:1805.00676v1 [cs.cv] 2 May 2018 Cristian Bodnar Supervisor: Dr Jon Shapiro University of Manchester School of Computer Science A final
Temporal Coherency based Criteria for Predicting Video Frames using Deep Multi-stage Generative Adversarial Networks
Temporal Coherency based Criteria for Predicting Video Frames using Deep Multi-stage Generative Adversarial Networks Prateep Bhattacharjee 1, Sukhendu Das 2 Visualization and Perception Laboratory Department
arxiv: v1 [cs.cv] 4 Feb 2019
![arxiv: v1 [cs.cv] 4 Feb 2019](/thumbs/88/116995423.jpg "arxiv: v1 [cs.cv] 4 Feb 2019")
Realistic Image Generation using Region-phrase Attention arxiv:1902.05395v1 [cs.cv] 4 Feb 2019 Wanming Huang University of Technology, Sydney wanming.huang@student.uts.edu.au Abstract The Generative Adversarial
arxiv: v4 [cs.lg] 1 May 2018
![arxiv: v4 [cs.lg] 1 May 2018](/thumbs/92/110620456.jpg "arxiv: v4 [cs.lg] 1 May 2018")
Controllable Generative Adversarial Network arxiv:1708.00598v4 [cs.lg] 1 May 2018 Minhyeok Lee School of Electrical Engineering Korea University Seoul, Korea 02841 suam6409@korea.ac.kr Abstract Junhee
Face anti-spoofing using Image Quality Assessment
Face anti-spoofing using Image Quality Assessment Speakers Prisme Polytech Orléans Aladine Chetouani R&D Trusted Services Emna Fourati Outline Face spoofing attacks Image Quality Assessment Proposed method
POINT CLOUD DEEP LEARNING
POINT CLOUD DEEP LEARNING Innfarn Yoo, 3/29/28 / 57 Introduction AGENDA Previous Work Method Result Conclusion 2 / 57 INTRODUCTION 3 / 57 2D OBJECT CLASSIFICATION Deep Learning for 2D Object Classification
StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks
1 StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Senior Member, IEEE, Xiaogang Wang, Member, IEEE, Xiaolei Huang, Member,
Deep Fakes using Generative Adversarial Networks (GAN)
Deep Fakes using Generative Adversarial Networks (GAN) Tianxiang Shen UCSD La Jolla, USA tis038@eng.ucsd.edu Ruixian Liu UCSD La Jolla, USA rul188@eng.ucsd.edu Ju Bai UCSD La Jolla, USA jub010@eng.ucsd.edu
Attribute Augmented Convolutional Neural Network for Face Hallucination
Attribute Augmented Convolutional Neural Network for Face Hallucination Cheng-Han Lee 1 Kaipeng Zhang 1 Hu-Cheng Lee 1 Chia-Wen Cheng 2 Winston Hsu 1 1 National Taiwan University 2 The University of Texas
ECE 599/692 Deep Learning. Lecture 12 GAN - Introduction
ECE 599/692 Deep Learning Lecture 12 AN - Introduction Hairong Qi, onzalez Family Professor Electrical Engineering and Computer Science University of Tennessee, Knoxville http://www.eecs.utk.edu/faculty/qi
SYNTHESIS OF IMAGES BY TWO-STAGE GENERATIVE ADVERSARIAL NETWORKS. Qiang Huang, Philip J.B. Jackson, Mark D. Plumbley, Wenwu Wang
SYNTHESIS OF IMAGES BY TWO-STAGE GENERATIVE ADVERSARIAL NETWORKS Qiang Huang, Philip J.B. Jackson, Mark D. Plumbley, Wenwu Wang Centre for Vision, Speech and Signal Processing University of Surrey, Guildford,
Deep Learning. Deep Learning. Practical Application Automatically Adding Sounds To Silent Movies
http://blog.csdn.net/zouxy09/article/details/8775360 Automatic Colorization of Black and White Images Automatically Adding Sounds To Silent Movies Traditionally this was done by hand with human effort
Visual Recommender System with Adversarial Generator-Encoder Networks
Visual Recommender System with Adversarial Generator-Encoder Networks Bowen Yao Stanford University 450 Serra Mall, Stanford, CA 94305 boweny@stanford.edu Yilin Chen Stanford University 450 Serra Mall
Amortised MAP Inference for Image Super-resolution. Casper Kaae Sønderby, Jose Caballero, Lucas Theis, Wenzhe Shi & Ferenc Huszár ICLR 2017
Amortised MAP Inference for Image Super-resolution Casper Kaae Sønderby, Jose Caballero, Lucas Theis, Wenzhe Shi & Ferenc Huszár ICLR 2017 Super Resolution Inverse problem: Given low resolution representation
arxiv: v1 [cs.cv] 16 Jul 2017
![arxiv: v1 [cs.cv] 16 Jul 2017](/thumbs/85/91311140.jpg "arxiv: v1 [cs.cv] 16 Jul 2017")
enerative adversarial network based on resnet for conditional image restoration Paper: jc*-**-**-****: enerative Adversarial Network based on Resnet for Conditional Image Restoration Meng Wang, Huafeng
Stimulus Synthesis for Efficient Evaluation and Refinement of Perceptual Image Quality Metrics
Presented at: IS&T/SPIE s 16th Annual Symposium on Electronic Imaging San Jose, CA, Jan. 18-22, 2004 Published in: Human Vision and Electronic Imaging IX, Proc. SPIE, vol. 5292. c SPIE Stimulus Synthesis
GAN Related Works. CVPR 2018 & Selective Works in ICML and NIPS. Zhifei Zhang
GAN Related Works CVPR 2018 & Selective Works in ICML and NIPS Zhifei Zhang Generative Adversarial Networks (GANs) 9/12/2018 2 Generative Adversarial Networks (GANs) Feedforward Backpropagation Real? z
A New CGAN Technique for Constrained Topology Design Optimization. Abstract
A New CGAN Technique for Constrained Topology Design Optimization M.-H. Herman Shen 1 and Liang Chen Department of Mechanical and Aerospace Engineering The Ohio State University Abstract This paper presents
Conditional Generative Adversarial Networks for Particle Physics
Conditional Generative Adversarial Networks for Particle Physics Capstone 2016 Charles Guthrie ( cdg356@nyu.edu ) Israel Malkin ( im965@nyu.edu ) Alex Pine ( akp258@nyu.edu ) Advisor: Kyle Cranmer ( kyle.cranmer@nyu.edu
Progress on Generative Adversarial Networks
Progress on Generative Adversarial Networks Wangmeng Zuo Vision Perception and Cognition Centre Harbin Institute of Technology Content Image generation: problem formulation Three issues about GAN Discriminate
Defense Data Generation in Distributed Deep Learning System Se-Yoon Oh / ADD-IDAR
Defense Data Generation in Distributed Deep Learning System Se-Yoon Oh / 2017. 10. 31 syoh@add.re.kr Page 1/36 Overview 1. Introduction 2. Data Generation Synthesis 3. Distributed Deep Learning 4. Conclusions
DOMAIN-ADAPTIVE GENERATIVE ADVERSARIAL NETWORKS FOR SKETCH-TO-PHOTO INVERSION
DOMAIN-ADAPTIVE GENERATIVE ADVERSARIAL NETWORKS FOR SKETCH-TO-PHOTO INVERSION Yen-Cheng Liu 1, Wei-Chen Chiu 2, Sheng-De Wang 1, and Yu-Chiang Frank Wang 1 1 Graduate Institute of Electrical Engineering,
CNN for Low Level Image Processing. Huanjing Yue
CNN for Low Level Image Processing Huanjing Yue 2017.11 1 Deep Learning for Image Restoration General formulation: min Θ L( x, x) s. t. x = F(y; Θ) Loss function Parameters to be learned Key issues The
End-To-End Spam Classification With Neural Networks
End-To-End Spam Classification With Neural Networks Christopher Lennan, Bastian Naber, Jan Reher, Leon Weber 1 Introduction A few years ago, the majority of the internet s network traffic was due to spam
DOMAIN-ADAPTIVE GENERATIVE ADVERSARIAL NETWORKS FOR SKETCH-TO-PHOTO INVERSION
2017 IEEE INTERNATIONAL WORKSHOP ON MACHINE LEARNING FOR SIGNAL PROCESSING, SEPT. 25 28, 2017, TOKYO, JAPAN DOMAIN-ADAPTIVE GENERATIVE ADVERSARIAL NETWORKS FOR SKETCH-TO-PHOTO INVERSION Yen-Cheng Liu 1,
Multi-Input Cardiac Image Super-Resolution using Convolutional Neural Networks
Multi-Input Cardiac Image Super-Resolution using Convolutional Neural Networks Ozan Oktay, Wenjia Bai, Matthew Lee, Ricardo Guerrero, Konstantinos Kamnitsas, Jose Caballero, Antonio de Marvao, Stuart Cook,
Deep Learning for Visual Manipulation and Synthesis
Deep Learning for Visual Manipulation and Synthesis Jun-Yan Zhu 朱俊彦 UC Berkeley 2017/01/11 @ VALSE What is visual manipulation? Image Editing Program input photo User Input result Desired output: stay
Infrared Image Colorization based on a Triplet DCGAN Architecture
Infrared Image Colorization based on a Triplet DCGAN Architecture Patricia L. Suárez plsuarez@espol.edu.ec Angel D. Sappa,2 sappa@ieee.org Boris X. Vintimilla boris.vintimilla@espol.edu.ec Escuela Superior
Learning to generate with adversarial networks
Learning to generate with adversarial networks Gilles Louppe June 27, 2016 Problem statement Assume training samples D = {x x p data, x X } ; We want a generative model p model that can draw new samples
Variational Autoencoders. Sargur N. Srihari
Variational Autoencoders Sargur N. srihari@cedar.buffalo.edu Topics 1. Generative Model 2. Standard Autoencoder 3. Variational autoencoders (VAE) 2 Generative Model A variational autoencoder (VAE) is a
Quantitative Evaluation of Generative Adversarial Networks and Improved Training Techniques
Quantitative Evaluation of Generative Adversarial Networks and Improved Training Techniques by Yadong Li to obtain the degree of Master of Science at the Delft University of Technology, to be defended
Deep Learning Approaches to 3D Shape Completion
Deep Learning Approaches to 3D Shape Completion Prafull Sharma Stanford University prafull7@stanford.edu Jarrod Cingel Stanford University jcingel@stanford.edu Abstract This project explores various methods
Hide-and-Seek: Forcing a network to be Meticulous for Weakly-supervised Object and Action Localization
Hide-and-Seek: Forcing a network to be Meticulous for Weakly-supervised Object and Action Localization Krishna Kumar Singh and Yong Jae Lee University of California, Davis ---- Paper Presentation Yixian
A Survey on Detecting Image Visual Saliency
1/29 A Survey on Detecting Image Visual Saliency Hsin-Ho Yeh Institute of Information Science, Acamedic Sinica, Taiwan {hhyeh}@iis.sinica.edu.tw 2010/12/09 2/29 Outline 1 Conclusions 3/29 What is visual
NVIDIA DLI HANDS-ON TRAINING COURSE CATALOG
NVIDIA DLI HANDS-ON TRAINING COURSE CATALOG Valid Through July 31, 2018 INTRODUCTION The NVIDIA Deep Learning Institute (DLI) trains developers, data scientists, and researchers on how to use artificial
Introduction to GAN. Generative Adversarial Networks. Junheng(Jeff) Hao
Introduction to GAN Generative Adversarial Networks Junheng(Jeff) Hao Adversarial Training is the coolest thing since sliced bread. -- Yann LeCun Roadmap 1. Generative Modeling 2. GAN 101: What is GAN?
arxiv: v1 [eess.sp] 23 Oct 2018
![arxiv: v1 [eess.sp] 23 Oct 2018](/thumbs/94/119614192.jpg "arxiv: v1 [eess.sp] 23 Oct 2018")
Reproducing AmbientGAN: Generative models from lossy measurements arxiv:1810.10108v1 [eess.sp] 23 Oct 2018 Mehdi Ahmadi Polytechnique Montreal mehdi.ahmadi@polymtl.ca Mostafa Abdelnaim University de Montreal
Super-Resolution on Image and Video
Super-Resolution on Image and Video Jason Liu Stanford University liujas00@stanford.edu Max Spero Stanford University maxspero@stanford.edu Allan Raventos Stanford University aravento@stanford.edu Abstract
Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform. Xintao Wang Ke Yu Chao Dong Chen Change Loy
Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform Xintao Wang Ke Yu Chao Dong Chen Change Loy Problem enlarge 4 times Low-resolution image High-resolution image Previous
arxiv: v1 [cs.cv] 7 Mar 2018
![arxiv: v1 [cs.cv] 7 Mar 2018](/thumbs/95/125565711.jpg "arxiv: v1 [cs.cv] 7 Mar 2018")
Accepted as a conference paper at the European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN) 2018 Inferencing Based on Unsupervised Learning of Disentangled
Learning to generate 3D shapes
Learning to generate 3D shapes Subhransu Maji College of Information and Computer Sciences University of Massachusetts, Amherst http://people.cs.umass.edu/smaji August 10, 2018 @ Caltech Creating 3D shapes
Stochastic Simulation with Generative Adversarial Networks
Stochastic Simulation with Generative Adversarial Networks Lukas Mosser, Olivier Dubrule, Martin J. Blunt lukas.mosser15@imperial.ac.uk, o.dubrule@imperial.ac.uk, m.blunt@imperial.ac.uk (Deep) Generative
Handwritten Hindi Numerals Recognition System
CS365 Project Report Handwritten Hindi Numerals Recognition System Submitted by: Akarshan Sarkar Kritika Singh Project Mentor: Prof. Amitabha Mukerjee 1 Abstract In this project, we consider the problem
DEEP LEARNING OF COMPRESSED SENSING OPERATORS WITH STRUCTURAL SIMILARITY (SSIM) LOSS
DEEP LEARNING OF COMPRESSED SENSING OPERATORS WITH STRUCTURAL SIMILARITY (SSIM) LOSS ABSTRACT Compressed sensing (CS) is a signal processing framework for efficiently reconstructing a signal from a small
Shape Context Matching For Efficient OCR
Matching For Efficient OCR May 14, 2012 Matching For Efficient OCR Table of contents 1 Motivation Background 2 What is a? Matching s Simliarity Measure 3 Matching s via Pyramid Matching Matching For Efficient
Single Image Super Resolution of Textures via CNNs. Andrew Palmer
Single Image Super Resolution of Textures via CNNs Andrew Palmer What is Super Resolution (SR)? Simple: Obtain one or more high-resolution images from one or more low-resolution ones Many, many applications
arxiv: v1 [cs.cv] 13 Sep 2018
![arxiv: v1 [cs.cv] 13 Sep 2018](/thumbs/86/94100157.jpg "arxiv: v1 [cs.cv] 13 Sep 2018")
arxiv:1809.04789v1 [cs.cv] 13 Sep 2018 Deep Learning-based Image Super-Resolution Considering Quantitative and Perceptual Quality Jun-Ho Choi, Jun-Hyuk Kim, Manri Cheon, and Jong-Seok Lee School of Integrated
CRF Based Point Cloud Segmentation Jonathan Nation
CRF Based Point Cloud Segmentation Jonathan Nation jsnation@stanford.edu 1. INTRODUCTION The goal of the project is to use the recently proposed fully connected conditional random field (CRF) model to
DCGANs for image super-resolution, denoising and debluring
DCGANs for image super-resolution, denoising and debluring Qiaojing Yan Stanford University Electrical Engineering qiaojing@stanford.edu Wei Wang Stanford University Electrical Engineering wwang23@stanford.edu
Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform Supplementary Material
Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform Supplementary Material Xintao Wang 1 Ke Yu 1 Chao Dong 2 Chen Change Loy 1 1 CUHK - SenseTime Joint Lab, The Chinese
Generating Images with Perceptual Similarity Metrics based on Deep Networks
Generating Images with Perceptual Similarity Metrics based on Deep Networks Alexey Dosovitskiy and Thomas Brox University of Freiburg {dosovits, brox}@cs.uni-freiburg.de Abstract We propose a class of
Controllable Generative Adversarial Network
Controllable Generative Adversarial Network arxiv:1708.00598v2 [cs.lg] 12 Sep 2017 Minhyeok Lee 1 and Junhee Seok 1 1 School of Electrical Engineering, Korea University, 145 Anam-ro, Seongbuk-gu, Seoul,
Introduction to Generative Adversarial Networks
Introduction to Generative Adversarial Networks Luke de Oliveira Vai Technologies Lawrence Berkeley National Laboratory @lukede0 @lukedeo lukedeo@vaitech.io https://ldo.io 1 Outline Why Generative Modeling?
Deep generative models of natural images
Spring 2016 1 Motivation 2 3 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models 4 Outline 1 Motivation 2 3 Variational autoencoders
Human Pose Estimation with Deep Learning. Wei Yang
Human Pose Estimation with Deep Learning Wei Yang Applications Understand Activities Family Robots American Heist (2014) - The Bank Robbery Scene 2 What do we need to know to recognize a crime scene? 3
Introduction to Generative Models (and GANs)
Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com Nov. 2017 Figures adapted from NIPS 2016 Tutorial Generative Adversarial Networks Generative Models: Learning the Distributions