Auto-Encoding Variational Bayes
|
|
- Juliet Holmes
- 6 years ago
- Views:
Transcription
1 Auto-Encoding Variational Bayes Diederik P (Durk) Kingma, Max Welling University of Amsterdam Ph.D. Candidate, advised by Max Durk Kingma D.P. Kingma Max Welling
2 Problem class Directed graphical model: x : observed variable z : latent variables (continuous) θ : model parameters pθ(x,z): joint PDF Factorized, differentiable Hard case: intractable posterior distribution pθ(z x) e.g. neural nets as components We want fast approximate posterior inference per datapoint After inference, learning params is easy D.P. Kingma 2
3 Latent variable generative model latent variable model: learn a mapping from some latent variable z to a complicated distribution on x. p(x) = p(x, z) dz where p(x, z) =p(x z)p(z) p(z) = something simple p(x z) =f(z) Can we learn to decouple the true explanatory factors underlying the data distribution? E.g. separate identity and expression in face images z 2 x 2 f Image from: Ward, A. D., Hamarneh, G.: 3D Surface Parameterization Using Manifold Learning for Medial Shape Representation, Conference on Image Processing, Proc. of SPIE Medical Imaging, 2007 IFT6266: Representation (Deep) Learning Aaron Courville x 3 z 1 x
4 Variational autoencoder (VAE) approach Leverage neural networks to learn a latent variable model. p(x) = p(x, z) dz where p(x, z) =p(x z)p(z) p(z) = something simple p(x z) =f(z) z 2 x 2 z : f x 3 z 1 x 1 f(z) : x : IFT6266: Representation (Deep) Learning Aaron Courville 11 11
5 What VAE can do? x2 z2 f x3 z1 MNIST: z2 Frey Face dataset: Expression z2 x1 Face manifold (b) Learned MNIST manifold z1 ns of learned data manifold for generative models with two-dimensional latent AEVB. Since the prior of the latent space is Gaussian, linearly spaced coorrepresentation (Deep) Learning Courville quareift6266: were transformed through the inverse CDF of the Aaron Gaussian to produce ariables z. For each of these values z, we plotted the corresponding generative (a) Learned Pose Frey Face manifoldz1 Figure 4: Visualisations of learned data manifol 12 space, learned with AEVB. Since the prior12of
6 The inference / learning challenge Where does z come from? The classic directed model dilemma. Computing the posterior p(z x) is intractable. We need it to train the directed model. z 2? x 2 z : f x 3 z 1 x 1 f(z) : x : IFT6266: Representation (Deep) Learning Aaron Courville 13 13
7 Auto-Encoding Variational Bayes Idea: Learn neural net to approximate the posterior qφ(z x) with 'variational parameters' φ one-shot approximate inference akin to the recognition model in Wake-Sleep Construct estimator of the variational lower bound which we can optimize jointly w.r.t. φ jointly with θ -> Stochastic gradient ascent D.P. Kingma 4
8 Variational Lower Bound of the marg. lik. D.P. Kingma 5
9 Monte Carlo estimator of the variational bound Can we differentiate through the sampling process w.r.t. φ? D.P. Kingma
10 Variational Autoencoder (VAE) Where does z come from? The classic DAG problem. The VAE approach: introduce an inference machine q φ (z x) that learns to approximate the posterior p θ (z x). - Define a variational lower bound on the data likelihood: p θ (x) L(θ, φ,x) L(,,x)=E q (z x) [log p (x, z) log q (z x)] = E q (z x) [log p (x z) + log p (z) log q (z x)] = D KL (q (z x) p (z)) + E q (z x) [log p (x z)] What is q φ (z x)? IFT6266: Representation (Deep) Learning Aaron Courville 14 14
11 Variational Autoencoder (VAE) Where does z come from? The classic DAG problem. The VAE approach: introduce an inference machine q φ (z x) that learns to approximate the posterior p θ (z x). - Define a variational lower bound on the data likelihood: p θ (x) L(θ, φ,x) L(,,x)=E q (z x) [log p (x, z) log q (z x)] = E q (z x) [log p (x z) + log p (z) log q (z x)] = D KL (q (z x) p (z)) + E q (z x) [log p (x z)] reconstruction term What is q φ (z x)? IFT6266: Representation (Deep) Learning Aaron Courville 14 14
12 Variational Autoencoder (VAE) Where does z come from? The classic DAG problem. The VAE approach: introduce an inference machine q φ (z x) that learns to approximate the posterior p θ (z x). - Define a variational lower bound on the data likelihood: p θ (x) L(θ, φ,x) L(,,x)=E q (z x) [log p (x, z) log q (z x)] = E q (z x) [log p (x z) + log p (z) log q (z x)] = D KL (q (z x) p (z)) + E q (z x) [log p (x z)] regularization term reconstruction term What is q φ (z x)? IFT6266: Representation (Deep) Learning Aaron Courville 14 14
13 VAE Inference model The VAE approach: introduce an inference model q φ (z x) that learns to approximates the intractable posterior p θ (z x) by optimizing the variational lower bound: L(θ, φ,x)= D KL (q φ (z x) p θ (z)) + E qφ (z x) [log p θ (x z)] We parameterize q φ (z x) with another neural network: q φ (z x) =q(z; g(x, φ)) z : p θ (x z) =p(x; f(z,θ)) z : g(x) : f(z) : x : x : IFT6266: Representation (Deep) Learning Aaron Courville 15 15
14 Key reparameterization trick Construct samples z ~ qφ(z x) in two steps: 1. ε ~ p(ε) (random seed independent of φ) 2. z = g(φ, ε, x) (differentiable perturbation) such that z ~ qφ(z x) (the correct distribution) Examples: if q(z x) ~ N(μ(x), σ(x)^2) ε ~ N(0,I) z = μ(x) + σ(x) * ε (approximate) Inverse CDF Much more possibilities (see paper) D.P. Kingma 7
15 Reparametrization trick Adding a few details + one really important trick Let s consider z to be real and q φ (z x) =N (z; µ z (x), σ z (x)) Parametrize z as z = µ z (x)+σ z (x)ϵ z where ϵ z = N (0, 1) (optional) Parametrize x a x = µ x (z)+σ x (z)ϵ x where ϵ x = N (0, 1) µ z (x) σ z (x) z : { { g(z) : f(z) : x : µ x (z) { σ x (z) { IFT6266: Representation (Deep) Learning Aaron Courville 16 16
16 SGVB estimator Really simple and appropriate for differentiation w.r.t. φ and θ! D.P. Kingma
17 Variational auto-encoder x p injected noise ε p(z) and p(x z) (decoder) z q q(z x) = N(μ,σ) (encoder) x D.P. Kingma 11
18 Why reparametrization helps September 19, / 6
19 Training with backpropagation! Due to a reparametrization trick, we can simultaneously train both the generative model p θ (x z) and the inference model q φ (z x) by optimizing the variational bound using gradient backpropagation. Objective function: L(θ, φ,x)= D KL (q φ (z x) p θ (z)) + E qφ (z x) [log p θ (x z)] Forward propagation z x Backward propagation q φ (z x) p θ (x z) ˆx IFT6266: Representation (Deep) Learning Aaron Courville 17 17
20 Auto-Encoding Variational Bayes Online algorithm repeat Backprop (Torch7 / Theano) e.g. Adagrad until convergence Scales to very large datasets! D.P. Kingma 9
21 Model used in experiments (noisy) negative reconstruction error D.P. Kingma regularization terms 10
22 Special case with Gaussian prior and posterior Suppose p(z) = N (z; 0, I ) Suppose q φ (z x) = N (z; µ φ (x), σφ 2(x)) Variational bound L = ln p θ (x) D KL (q φ (z x) p θ (z x) (1) = IE qφ (z x)[ln p θ (x z)] D KL (q φ (z x) p(z)) (2) Closed-form computation of KL divergence D KL (q(z x) p(z)) = D D 2 ln σ j (x) µ j (x) 2 σ j (x) 2 d=1 Deterministic regularization, stochastic data term September 19, / 6
23 Results: Marginal likelihood lower bound D.P. Kingma 12
24 Results: Marginal log-likelihood Monte Carlo EM does not scale well to large datasets D.P. Kingma 13
25 Robustness to high-dimensional latent space D.P. Kingma 14
26 Effect of KL term: component collapse IFT6266: Representation (Deep) Learning Aaron Courville Figure from Laurent Dinh & Vincent Dumoulin 19 19
27 Component collapse & depth Deep model: some component collapse Deeper model: more component collapse IFT6266: Representation (Deep) Learning Aaron Courville Figures from Laurent Dinh & Vincent Dumoulin 20 20
28 Samples from MNIST (simple ancestral sampling) D.P. Kingma 15
29 2D Latent space: Frey Face z2 D.P. Kingma z1 16
30 2D Latent space: MNIST z2 D.P. Kingma z1 17
31 Labeled Faces in the Wild (random samples from generative model) D.P. Kingma 19
32 Conditional generation using M2, central pixels image September 19, / 6
33 Conditional generation: central pixels image September 19, / 6
34 Semi-supervised Learning with Deep Generative Models Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, Max Welling They study two basic approaches: M1: Standard unsupervised feature learning ( self-taught learning ) - Train features z on unlabeled data, train a classifier to map from z to label y. - Generative model: (recall that x = data, z = latent features) p(z) =N (z 0, I); p (x z) =f(x; z, ), z M2: Generative semi-supervised model. p(y) =Cat(y ); p(z) =N (z 0, I); is the multinomial distribution, the cl p (x y, z) =f(x; y, z, ), labels are treated as latent y x z x IFT6266: Representation (Deep) Learning Aaron Courville 23 23
35 Semi-supervised Learning with Deep Generative Models Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, Max Welling M1+M2: Combination semi-supervised model - Train generative semi-supervised model on unsupervised features z1 on unlabeled data, train a classifier to map from z1 to label z1. ead of the raw data. The result is a deep generativ p (x,y,z 1, z 2 )= p(y)p(z 2 )p (z 1 y, z 2 )p (x z 1 ), y and z above, and both and a y z 2 z 1 x IFT6266: Representation (Deep) Learning Aaron Courville 24 24
36 Semi-supervised Learning with Deep Generative Models Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, Max Welling Appoximate posterior (encoder model) - Following the VAE strategy we parametrize the approximate posterior with a high capacity model, like a MLP or some other deep model (convnet, RNN, etc). M1: q (z x) =N (z µ (x), diag( 2 (x))), M2: q (z y, x) =N (z µ (y, x), diag( 2 (x))); q (y x) =Cat(y (x)), µ (x) - and ( 2 (x) are parameterized by deep MLPs, that can share parameters. M1: z M2: z y x x IFT6266: Representation (Deep) Learning Aaron Courville 25 25
37 Semi-supervised Learning with Deep Generative Models Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, Max Welling M2: The lower bound for the generative semi-supervised model. - Objective with labeled data: log p (x,y) E q (z x,y) [log p (x y, z) + log p (y) + log p(z) log q (z x,y)]= L(x,y), - Objective without labels: posterior inference and the resulting bound for handling data points with an unobserved label X log p (x) E q (y,z x) [log p (x y, z) + log p (y) + log p(z) log q (y, z x)] = X q (y x)( L(x,y)) + H(q (y x)) = U(x). y - Semi-supervised objective: X J = X L(x,y)+ X U(x) (x,y) epl x epu X y z x - actually, for classification, they use J = J + E epl (x,y) [ log q (y x)], IFT6266: Representation (Deep) Learning Aaron Courville 26 26
38 Semi-supervised MNIST classification results Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, Max Welling Combination model M1+M2 shows dramatic improvement: Table 1: Benchmark results of semi-supervised classification on MNIST with few labels. N NN CNN TSVM CAE MTC AtlasRBF M1+TSVM M2 M1+M (± 0.95) (± 0.25) (± 1.71) 3.33 (± 0.14) (± 0.049) 4.94 (± 0.13) 2.59 (± 0.05) (± 0.12) 4.24 (± 0.07) 3.60 (± 0.56) 2.40 (± 0.02) (± 0.04) 3.92 (± 0.63) 2.18 (± 0.04) 4 Experimental Results Full MNIST test error: 0.96% (for comparison, current SOTA: 0.78%). IFT6266: Representation (Deep) Learning Aaron Courville 27 27
39 Conditional generation using M2 September 19, / 6
40 Conditional generation using M2 September 19, /6
Deep Generative Models Variational Autoencoders
Deep Generative Models Variational Autoencoders Sudeshna Sarkar 5 April 2017 Generative Nets Generative models that represent probability distributions over multiple variables in some way. Directed Generative
More informationVariational Autoencoders. Sargur N. Srihari
Variational Autoencoders Sargur N. srihari@cedar.buffalo.edu Topics 1. Generative Model 2. Standard Autoencoder 3. Variational autoencoders (VAE) 2 Generative Model A variational autoencoder (VAE) is a
More informationLecture 21 : A Hybrid: Deep Learning and Graphical Models
10-708: Probabilistic Graphical Models, Spring 2018 Lecture 21 : A Hybrid: Deep Learning and Graphical Models Lecturer: Kayhan Batmanghelich Scribes: Paul Liang, Anirudha Rayasam 1 Introduction and Motivation
More informationDeep generative models of natural images
Spring 2016 1 Motivation 2 3 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models 4 Outline 1 Motivation 2 3 Variational autoencoders
More information19: Inference and learning in Deep Learning
10-708: Probabilistic Graphical Models 10-708, Spring 2017 19: Inference and learning in Deep Learning Lecturer: Zhiting Hu Scribes: Akash Umakantha, Ryan Williamson 1 Classes of Deep Generative Models
More informationGAN Frontiers/Related Methods
GAN Frontiers/Related Methods Improving GAN Training Improved Techniques for Training GANs (Salimans, et. al 2016) CSC 2541 (07/10/2016) Robin Swanson (robin@cs.toronto.edu) Training GANs is Difficult
More informationAlternatives to Direct Supervision
CreativeAI: Deep Learning for Graphics Alternatives to Direct Supervision Niloy Mitra Iasonas Kokkinos Paul Guerrero Nils Thuerey Tobias Ritschel UCL UCL UCL TUM UCL Timetable Theory and Basics State of
More informationarxiv: v2 [cs.lg] 31 Oct 2014
Semi-supervised Learning with Deep Generative Models arxiv:1406.5298v2 [cs.lg] 31 Oct 2014 Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, Max Welling Machine Learning Group, Univ. of Amsterdam,
More informationVariational Autoencoders
red red red red red red red red red red red red red red red red red red red red Tutorial 03/10/2016 Generative modelling Assume that the original dataset is drawn from a distribution P(X ). Attempt to
More informationDEEP LEARNING PART THREE - DEEP GENERATIVE MODELS CS/CNS/EE MACHINE LEARNING & DATA MINING - LECTURE 17
DEEP LEARNING PART THREE - DEEP GENERATIVE MODELS CS/CNS/EE 155 - MACHINE LEARNING & DATA MINING - LECTURE 17 GENERATIVE MODELS DATA 3 DATA 4 example 1 DATA 5 example 2 DATA 6 example 3 DATA 7 number of
More informationAuxiliary Deep Generative Models
Downloaded from orbit.dtu.dk on: Dec 12, 2018 Auxiliary Deep Generative Models Maaløe, Lars; Sønderby, Casper Kaae; Sønderby, Søren Kaae; Winther, Ole Published in: Proceedings of the 33rd International
More informationarxiv: v6 [stat.ml] 15 Jun 2015
VARIATIONAL RECURRENT AUTO-ENCODERS Otto Fabius & Joost R. van Amersfoort Machine Learning Group University of Amsterdam {ottofabius,joost.van.amersfoort}@gmail.com ABSTRACT arxiv:1412.6581v6 [stat.ml]
More informationUnsupervised Learning
Deep Learning for Graphics Unsupervised Learning Niloy Mitra Iasonas Kokkinos Paul Guerrero Vladimir Kim Kostas Rematas Tobias Ritschel UCL UCL/Facebook UCL Adobe Research U Washington UCL Timetable Niloy
More informationImplicit generative models: dual vs. primal approaches
Implicit generative models: dual vs. primal approaches Ilya Tolstikhin MPI for Intelligent Systems ilya@tue.mpg.de Machine Learning Summer School 2017 Tübingen, Germany Contents 1. Unsupervised generative
More informationAdversarially Learned Inference
Institut des algorithmes d apprentissage de Montréal Adversarially Learned Inference Aaron Courville CIFAR Fellow Université de Montréal Joint work with: Vincent Dumoulin, Ishmael Belghazi, Olivier Mastropietro,
More informationGenerative Models in Deep Learning. Sargur N. Srihari
Generative Models in Deep Learning Sargur N. Srihari srihari@cedar.buffalo.edu 1 Topics 1. Need for Probabilities in Machine Learning 2. Representations 1. Generative and Discriminative Models 2. Directed/Undirected
More informationDay 3 Lecture 1. Unsupervised Learning
Day 3 Lecture 1 Unsupervised Learning Semi-supervised and transfer learning Myth: you can t do deep learning unless you have a million labelled examples for your problem. Reality You can learn useful representations
More informationExpectation Maximization. Machine Learning 10701/15781 Carlos Guestrin Carnegie Mellon University
Expectation Maximization Machine Learning 10701/15781 Carlos Guestrin Carnegie Mellon University April 10 th, 2006 1 Announcements Reminder: Project milestone due Wednesday beginning of class 2 Coordinate
More informationSemi-Amortized Variational Autoencoders
Semi-Amortized Variational Autoencoders Yoon Kim Sam Wiseman Andrew Miller David Sontag Alexander Rush Code: https://github.com/harvardnlp/sa-vae Background: Variational Autoencoders (VAE) (Kingma et al.
More informationCambridge Interview Technical Talk
Cambridge Interview Technical Talk February 2, 2010 Table of contents Causal Learning 1 Causal Learning Conclusion 2 3 Motivation Recursive Segmentation Learning Causal Learning Conclusion Causal learning
More informationarxiv: v2 [cs.lg] 17 Dec 2018
Lu Mi 1 * Macheng Shen 2 * Jingzhao Zhang 2 * 1 MIT CSAIL, 2 MIT LIDS {lumi, macshen, jzhzhang}@mit.edu The authors equally contributed to this work. This report was a part of the class project for 6.867
More informationarxiv: v1 [cs.lg] 24 Jan 2019
Jaehoon Cha Kyeong Soo Kim Sanghuyk Lee arxiv:9.879v [cs.lg] Jan 9 Abstract Noting the importance of the latent variables in inference and learning, we propose a novel framework for autoencoders based
More informationGradient of the lower bound
Weakly Supervised with Latent PhD advisor: Dr. Ambedkar Dukkipati Department of Computer Science and Automation gaurav.pandey@csa.iisc.ernet.in Objective Given a training set that comprises image and image-level
More informationClassification of 1D-Signal Types Using Semi-Supervised Deep Learning
UNIVERSITY OF ZAGREB FACULTY OF ELECTRICAL ENGINEERING AND COMPUTING MASTER THESIS No. 1414 Classification of 1D-Signal Types Using Semi-Supervised Deep Learning Tomislav Šebrek Zagreb, June 2017. I
More informationSemantic Segmentation. Zhongang Qi
Semantic Segmentation Zhongang Qi qiz@oregonstate.edu Semantic Segmentation "Two men riding on a bike in front of a building on the road. And there is a car." Idea: recognizing, understanding what's in
More informationarxiv: v1 [stat.ml] 10 Dec 2018
1st Symposium on Advances in Approximate Bayesian Inference, 2018 1 7 Disentangled Dynamic Representations from Unordered Data arxiv:1812.03962v1 [stat.ml] 10 Dec 2018 Leonhard Helminger Abdelaziz Djelouah
More informationEnergy Based Models, Restricted Boltzmann Machines and Deep Networks. Jesse Eickholt
Energy Based Models, Restricted Boltzmann Machines and Deep Networks Jesse Eickholt ???? Who s heard of Energy Based Models (EBMs) Restricted Boltzmann Machines (RBMs) Deep Belief Networks Auto-encoders
More informationAutoencoders. Stephen Scott. Introduction. Basic Idea. Stacked AE. Denoising AE. Sparse AE. Contractive AE. Variational AE GAN.
Stacked Denoising Sparse Variational (Adapted from Paul Quint and Ian Goodfellow) Stacked Denoising Sparse Variational Autoencoding is training a network to replicate its input to its output Applications:
More informationAutoencoder. Representation learning (related to dictionary learning) Both the input and the output are x
Deep Learning 4 Autoencoder, Attention (spatial transformer), Multi-modal learning, Neural Turing Machine, Memory Networks, Generative Adversarial Net Jian Li IIIS, Tsinghua Autoencoder Autoencoder Unsupervised
More informationModel Generalization and the Bias-Variance Trade-Off
Charu C. Aggarwal IBM T J Watson Research Center Yorktown Heights, NY Model Generalization and the Bias-Variance Trade-Off Neural Networks and Deep Learning, Springer, 2018 Chapter 4, Section 4.1-4.2 What
More informationDeep Generative Models and a Probabilistic Programming Library
Deep Generative Models and a Probabilistic Programming Library Discriminative (Deep) Learning Learn a (differentiable) function mapping from input to output x f(x; θ) y Gradient back-propagation Generative
More informationBidirectional GAN. Adversarially Learned Inference (ICLR 2017) Adversarial Feature Learning (ICLR 2017)
Bidirectional GAN Adversarially Learned Inference (ICLR 2017) V. Dumoulin 1, I. Belghazi 1, B. Poole 2, O. Mastropietro 1, A. Lamb 1, M. Arjovsky 3 and A. Courville 1 1 Universite de Montreal & 2 Stanford
More informationNeural Networks for Machine Learning. Lecture 15a From Principal Components Analysis to Autoencoders
Neural Networks for Machine Learning Lecture 15a From Principal Components Analysis to Autoencoders Geoffrey Hinton Nitish Srivastava, Kevin Swersky Tijmen Tieleman Abdel-rahman Mohamed Principal Components
More informationIterative Inference Models
Iterative Inference Models Joseph Marino, Yisong Yue California Institute of Technology {jmarino, yyue}@caltech.edu Stephan Mt Disney Research stephan.mt@disneyresearch.com Abstract Inference models, which
More informationDenoising Adversarial Autoencoders
Denoising Adversarial Autoencoders Antonia Creswell BICV Imperial College London Anil Anthony Bharath BICV Imperial College London Email: ac2211@ic.ac.uk arxiv:1703.01220v4 [cs.cv] 4 Jan 2018 Abstract
More information27: Hybrid Graphical Models and Neural Networks
10-708: Probabilistic Graphical Models 10-708 Spring 2016 27: Hybrid Graphical Models and Neural Networks Lecturer: Matt Gormley Scribes: Jakob Bauer Otilia Stretcu Rohan Varma 1 Motivation We first look
More informationLatent Regression Bayesian Network for Data Representation
2016 23rd International Conference on Pattern Recognition (ICPR) Cancún Center, Cancún, México, December 4-8, 2016 Latent Regression Bayesian Network for Data Representation Siqi Nie Department of Electrical,
More informationGrundlagen der Künstlichen Intelligenz
Grundlagen der Künstlichen Intelligenz Unsupervised learning Daniel Hennes 29.01.2018 (WS 2017/18) University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Supervised learning Regression (linear
More informationNeural Networks and Deep Learning
Neural Networks and Deep Learning Example Learning Problem Example Learning Problem Celebrity Faces in the Wild Machine Learning Pipeline Raw data Feature extract. Feature computation Inference: prediction,
More informationAuto-encoder with Adversarially Regularized Latent Variables
Information Engineering Express International Institute of Applied Informatics 2017, Vol.3, No.3, P.11 20 Auto-encoder with Adversarially Regularized Latent Variables for Semi-Supervised Learning Ryosuke
More informationProbabilistic Programming with Pyro
Probabilistic Programming with Pyro the pyro team Eli Bingham Theo Karaletsos Rohit Singh JP Chen Martin Jankowiak Fritz Obermeyer Neeraj Pradhan Paul Szerlip Noah Goodman Why Pyro? Why probabilistic modeling?
More informationLEARNING TO INFER ABSTRACT 1 INTRODUCTION. Under review as a conference paper at ICLR Anonymous authors Paper under double-blind review
LEARNING TO INFER Anonymous authors Paper under double-blind review ABSTRACT Inference models, which replace an optimization-based inference procedure with a learned model, have been fundamental in advancing
More informationLearning a Representation Map for Robot Navigation using Deep Variational Autoencoder
MSc Computational Science Master Thesis Learning a Representation Map for Robot Navigation using Deep Variational Autoencoder by Kaixin Hu April 2018 Supervisor: Peter O Connor Assessor: dhr. dr. E. (Stratis)
More informationMachine Learning. The Breadth of ML Neural Networks & Deep Learning. Marc Toussaint. Duy Nguyen-Tuong. University of Stuttgart
Machine Learning The Breadth of ML Neural Networks & Deep Learning Marc Toussaint University of Stuttgart Duy Nguyen-Tuong Bosch Center for Artificial Intelligence Summer 2017 Neural Networks Consider
More informationCSC412: Stochastic Variational Inference. David Duvenaud
CSC412: Stochastic Variational Inference David Duvenaud Admin A3 will be released this week and will be shorter Motivation for REINFORCE Class projects Class Project ideas Develop a generative model for
More informationAutoencoding Beyond Pixels Using a Learned Similarity Metric
Autoencoding Beyond Pixels Using a Learned Similarity Metric International Conference on Machine Learning, 2016 Anders Boesen Lindbo Larsen, Hugo Larochelle, Søren Kaae Sønderby, Ole Winther Technical
More informationIMPLICIT AUTOENCODERS
IMPLICIT AUTOENCODERS Anonymous authors Paper under double-blind review ABSTRACT In this paper, we describe the implicit autoencoder (IAE), a generative autoencoder in which both the generative path and
More informationCIS 520, Machine Learning, Fall 2015: Assignment 7 Due: Mon, Nov 16, :59pm, PDF to Canvas [100 points]
CIS 520, Machine Learning, Fall 2015: Assignment 7 Due: Mon, Nov 16, 2015. 11:59pm, PDF to Canvas [100 points] Instructions. Please write up your responses to the following problems clearly and concisely.
More informationWhen Variational Auto-encoders meet Generative Adversarial Networks
When Variational Auto-encoders meet Generative Adversarial Networks Jianbo Chen Billy Fang Cheng Ju 14 December 2016 Abstract Variational auto-encoders are a promising class of generative models. In this
More informationUCLA UCLA Electronic Theses and Dissertations
UCLA UCLA Electronic Theses and Dissertations Title Application of Generative Adversarial Network on Image Style Transformation and Image Processing Permalink https://escholarship.org/uc/item/66w654x7
More informationBilevel Sparse Coding
Adobe Research 345 Park Ave, San Jose, CA Mar 15, 2013 Outline 1 2 The learning model The learning algorithm 3 4 Sparse Modeling Many types of sensory data, e.g., images and audio, are in high-dimensional
More informationGENERATIVE ADVERSARIAL NETWORKS (GAN) Presented by Omer Stein and Moran Rubin
GENERATIVE ADVERSARIAL NETWORKS (GAN) Presented by Omer Stein and Moran Rubin GENERATIVE MODEL Given a training dataset, x, try to estimate the distribution, Pdata(x) Explicitly or Implicitly (GAN) Explicitly
More information(University Improving of Montreal) Generative Adversarial Networks with Denoising Feature Matching / 17
Improving Generative Adversarial Networks with Denoising Feature Matching David Warde-Farley 1 Yoshua Bengio 1 1 University of Montreal, ICLR,2017 Presenter: Bargav Jayaraman Outline 1 Introduction 2 Background
More informationCapsule Networks. Eric Mintun
Capsule Networks Eric Mintun Motivation An improvement* to regular Convolutional Neural Networks. Two goals: Replace max-pooling operation with something more intuitive. Keep more info about an activated
More informationAn Efficient Model Selection for Gaussian Mixture Model in a Bayesian Framework
IEEE SIGNAL PROCESSING LETTERS, VOL. XX, NO. XX, XXX 23 An Efficient Model Selection for Gaussian Mixture Model in a Bayesian Framework Ji Won Yoon arxiv:37.99v [cs.lg] 3 Jul 23 Abstract In order to cluster
More informationA Fast Learning Algorithm for Deep Belief Nets
A Fast Learning Algorithm for Deep Belief Nets Geoffrey E. Hinton, Simon Osindero Department of Computer Science University of Toronto, Toronto, Canada Yee-Whye Teh Department of Computer Science National
More informationCS839: Probabilistic Graphical Models. Lecture 10: Learning with Partially Observed Data. Theo Rekatsinas
CS839: Probabilistic Graphical Models Lecture 10: Learning with Partially Observed Data Theo Rekatsinas 1 Partially Observed GMs Speech recognition 2 Partially Observed GMs Evolution 3 Partially Observed
More informationExtracting and Composing Robust Features with Denoising Autoencoders
Presenter: Alexander Truong March 16, 2017 Extracting and Composing Robust Features with Denoising Autoencoders Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre-Antoine Manzagol 1 Outline Introduction
More informationDeep Boltzmann Machines
Deep Boltzmann Machines Sargur N. Srihari srihari@cedar.buffalo.edu Topics 1. Boltzmann machines 2. Restricted Boltzmann machines 3. Deep Belief Networks 4. Deep Boltzmann machines 5. Boltzmann machines
More informationMachine Learning
Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University February 25, 2015 Today: Graphical models Bayes Nets: Inference Learning EM Readings: Bishop chapter 8 Mitchell
More informationarxiv: v2 [cs.lg] 7 Feb 2019
Implicit Autoencoders Alireza Makhzani Vector Institute for Artificial Intelligence University of Toronto makhzani@vectorinstitute.ai arxiv:1805.09804v2 [cs.lg 7 Feb 2019 Abstract In this paper, we describe
More informationTutorial Deep Learning : Unsupervised Feature Learning
Tutorial Deep Learning : Unsupervised Feature Learning Joana Frontera-Pons 4th September 2017 - Workshop Dictionary Learning on Manifolds OUTLINE Introduction Representation Learning TensorFlow Examples
More informationarxiv: v1 [stat.ml] 3 Apr 2017
Lars Maaløe 1 Marco Fraccaro 1 Ole Winther 1 arxiv:1704.00637v1 stat.ml] 3 Apr 2017 Abstract Deep generative models trained with large amounts of unlabelled data have proven to be powerful within the domain
More informationarxiv: v1 [stat.ml] 11 Feb 2018
Paul K. Rubenstein Bernhard Schölkopf Ilya Tolstikhin arxiv:80.0376v [stat.ml] Feb 08 Abstract We study the role of latent space dimensionality in Wasserstein auto-encoders (WAEs). Through experimentation
More informationSupervised Learning for Image Segmentation
Supervised Learning for Image Segmentation Raphael Meier 06.10.2016 Raphael Meier MIA 2016 06.10.2016 1 / 52 References A. Ng, Machine Learning lecture, Stanford University. A. Criminisi, J. Shotton, E.
More informationJOINT MULTIMODAL LEARNING WITH DEEP GENERA-
JOINT MULTIMODAL LEARNING WITH DEEP GENERA- TIVE MODELS Masahiro Suzuki, Kotaro Nakayama, Yutaka Matsuo The University of Tokyo Bunkyo-ku, Tokyo, Japan {masa,k-nakayama,matsuo}@weblab.t.u-tokyo.ac.jp ABSTRACT
More informationProbabilistic Graphical Models
Overview of Part Two Probabilistic Graphical Models Part Two: Inference and Learning Christopher M. Bishop Exact inference and the junction tree MCMC Variational methods and EM Example General variational
More informationClustering K-means. Machine Learning CSEP546 Carlos Guestrin University of Washington February 18, Carlos Guestrin
Clustering K-means Machine Learning CSEP546 Carlos Guestrin University of Washington February 18, 2014 Carlos Guestrin 2005-2014 1 Clustering images Set of Images [Goldberger et al.] Carlos Guestrin 2005-2014
More informationCOMP 551 Applied Machine Learning Lecture 13: Unsupervised learning
COMP 551 Applied Machine Learning Lecture 13: Unsupervised learning Associate Instructor: Herke van Hoof (herke.vanhoof@mail.mcgill.ca) Slides mostly by: (jpineau@cs.mcgill.ca) Class web page: www.cs.mcgill.ca/~jpineau/comp551
More informationTowards Conceptual Compression
Towards Conceptual Compression Karol Gregor karolg@google.com Frederic Besse fbesse@google.com Danilo Jimenez Rezende danilor@google.com Ivo Danihelka danihelka@google.com Daan Wierstra wierstra@google.com
More informationWhat is machine learning?
Machine learning, pattern recognition and statistical data modelling Lecture 12. The last lecture Coryn Bailer-Jones 1 What is machine learning? Data description and interpretation finding simpler relationship
More informationAkarsh Pokkunuru EECS Department Contractive Auto-Encoders: Explicit Invariance During Feature Extraction
Akarsh Pokkunuru EECS Department 03-16-2017 Contractive Auto-Encoders: Explicit Invariance During Feature Extraction 1 AGENDA Introduction to Auto-encoders Types of Auto-encoders Analysis of different
More informationThe Multi-Entity Variational Autoencoder
The Multi-Entity Variational Autoencoder Charlie Nash 1,2, S. M. Ali Eslami 2, Chris Burgess 2, Irina Higgins 2, Daniel Zoran 2, Theophane Weber 2, Peter Battaglia 2 1 Edinburgh University 2 DeepMind Abstract
More informationLecture 19: Generative Adversarial Networks
Lecture 19: Generative Adversarial Networks Roger Grosse 1 Introduction Generative modeling is a type of machine learning where the aim is to model the distribution that a given set of data (e.g. images,
More informationPerceptual Loss for Convolutional Neural Network Based Optical Flow Estimation. Zong-qing LU, Xiang ZHU and Qing-min LIAO *
2017 2nd International Conference on Software, Multimedia and Communication Engineering (SMCE 2017) ISBN: 978-1-60595-458-5 Perceptual Loss for Convolutional Neural Network Based Optical Flow Estimation
More informationClustering web search results
Clustering K-means Machine Learning CSE546 Emily Fox University of Washington November 4, 2013 1 Clustering images Set of Images [Goldberger et al.] 2 1 Clustering web search results 3 Some Data 4 2 K-means
More informationAn Empirical Evaluation of Deep Architectures on Problems with Many Factors of Variation
An Empirical Evaluation of Deep Architectures on Problems with Many Factors of Variation Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio Université de Montréal 13/06/2007
More informationScore function estimator and variance reduction techniques
and variance reduction techniques Wilker Aziz University of Amsterdam May 24, 2018 Wilker Aziz Discrete variables 1 Outline 1 2 3 Wilker Aziz Discrete variables 1 Variational inference for belief networks
More informationCOMP 551 Applied Machine Learning Lecture 16: Deep Learning
COMP 551 Applied Machine Learning Lecture 16: Deep Learning Instructor: Ryan Lowe (ryan.lowe@cs.mcgill.ca) Slides mostly by: Class web page: www.cs.mcgill.ca/~hvanho2/comp551 Unless otherwise noted, all
More informationAutoencoders, denoising autoencoders, and learning deep networks
4 th CiFAR Summer School on Learning and Vision in Biology and Engineering Toronto, August 5-9 2008 Autoencoders, denoising autoencoders, and learning deep networks Part II joint work with Hugo Larochelle,
More informationarxiv: v1 [cs.cv] 17 Nov 2016
Inverting The Generator Of A Generative Adversarial Network arxiv:1611.05644v1 [cs.cv] 17 Nov 2016 Antonia Creswell BICV Group Bioengineering Imperial College London ac2211@ic.ac.uk Abstract Anil Anthony
More informationGenerative and discriminative classification techniques
Generative and discriminative classification techniques Machine Learning and Category Representation 2014-2015 Jakob Verbeek, November 28, 2014 Course website: http://lear.inrialpes.fr/~verbeek/mlcr.14.15
More informationECG782: Multidimensional Digital Signal Processing
ECG782: Multidimensional Digital Signal Processing Object Recognition http://www.ee.unlv.edu/~b1morris/ecg782/ 2 Outline Knowledge Representation Statistical Pattern Recognition Neural Networks Boosting
More informationGenerative Adversarial Networks (GANs)
Generative Adversarial Networks (GANs) Hossein Azizpour Most of the slides are courtesy of Dr. Ian Goodfellow (Research Scientist at OpenAI) and from his presentation at NIPS 2016 tutorial Note. I am generally
More informationClustering algorithms
Clustering algorithms Machine Learning Hamid Beigy Sharif University of Technology Fall 1393 Hamid Beigy (Sharif University of Technology) Clustering algorithms Fall 1393 1 / 22 Table of contents 1 Supervised
More informationarxiv: v1 [stat.ml] 17 Apr 2017
Multimodal Prediction and Personalization of Photo Edits with Deep Generative Models Ardavan Saeedi CSAIL, MIT Matthew D. Hoffman Adobe Research Stephen J. DiVerdi Adobe Research arxiv:1704.04997v1 [stat.ml]
More informationAuxiliary Variational Information Maximization for Dimensionality Reduction
Auxiliary Variational Information Maximization for Dimensionality Reduction Felix Agakov 1 and David Barber 2 1 University of Edinburgh, 5 Forrest Hill, EH1 2QL Edinburgh, UK felixa@inf.ed.ac.uk, www.anc.ed.ac.uk
More informationMachine Learning
Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University March 4, 2015 Today: Graphical models Bayes Nets: EM Mixture of Gaussian clustering Learning Bayes Net structure
More informationarxiv: v2 [cs.lg] 25 May 2016
Adversarial Autoencoders Alireza Makhzani University of Toronto makhzani@psi.toronto.edu Jonathon Shlens & Navdeep Jaitly Google Brain {shlens,ndjaitly}@google.com arxiv:1511.05644v2 [cs.lg] 25 May 2016
More informationDOMAIN-ADAPTIVE GENERATIVE ADVERSARIAL NETWORKS FOR SKETCH-TO-PHOTO INVERSION
DOMAIN-ADAPTIVE GENERATIVE ADVERSARIAL NETWORKS FOR SKETCH-TO-PHOTO INVERSION Yen-Cheng Liu 1, Wei-Chen Chiu 2, Sheng-De Wang 1, and Yu-Chiang Frank Wang 1 1 Graduate Institute of Electrical Engineering,
More informationGenerative Modeling with Convolutional Neural Networks. Denis Dus Data Scientist at InData Labs
Generative Modeling with Convolutional Neural Networks Denis Dus Data Scientist at InData Labs What we will discuss 1. 2. 3. 4. Discriminative vs Generative modeling Convolutional Neural Networks How to
More informationMax-Margin Deep Generative Models
Max-Margin Deep Generative Models Chongxuan Li, Jun Zhu, Tianlin Shi, Bo Zhang Dept. of Comp. Sci. & Tech., State Key Lab of Intell. Tech. & Sys., TNList Lab, Center for Bio-Inspired Computing Research,
More informationarxiv: v2 [cs.lg] 9 Jun 2017
Shengjia Zhao 1 Jiaming Song 1 Stefano Ermon 1 arxiv:1702.08396v2 [cs.lg] 9 Jun 2017 Abstract Deep neural networks have been shown to be very successful at learning feature hierarchies in supervised learning
More informationTackling Over-pruning in Variational Autoencoders
Serena Yeung 1 Anitha Kannan 2 Yann Dauphin 2 Li Fei-Fei 1 Abstract Variational autoencoders (VAE) are directed generative models that learn factorial latent variables. As noted by Burda et al. (2015),
More informationarxiv: v2 [stat.ml] 21 Oct 2017
Variational Approaches for Auto-Encoding Generative Adversarial Networks arxiv:1706.04987v2 stat.ml] 21 Oct 2017 Mihaela Rosca Balaji Lakshminarayanan David Warde-Farley Shakir Mohamed DeepMind {mihaelacr,balajiln,dwf,shakir}@google.com
More informationTopics in AI (CPSC 532L): Multimodal Learning with Vision, Language and Sound. Lecture 12: Deep Reinforcement Learning
Topics in AI (CPSC 532L): Multimodal Learning with Vision, Language and Sound Lecture 12: Deep Reinforcement Learning Types of Learning Supervised training Learning from the teacher Training data includes
More informationAuxiliary Guided Autoregressive Variational Autoencoders
Auxiliary Guided Autoregressive Variational Autoencoders Thomas Lucas, Jakob Verbeek To cite this version: Thomas Lucas, Jakob Verbeek. Auxiliary Guided Autoregressive Variational Autoencoders. 2017.
More informationModeling and Optimization of Thin-Film Optical Devices using a Variational Autoencoder
Modeling and Optimization of Thin-Film Optical Devices using a Variational Autoencoder Introduction John Roberts and Evan Wang, {johnr3, wangevan}@stanford.edu Optical thin film systems are structures
More informationMultimodal Prediction and Personalization of Photo Edits with Deep Generative Models
Multimodal Prediction and Personalization of Photo Edits with Deep Generative Models Ardavan Saeedi Matthew D. Hoffman Stephen J. DiVerdi CSAIL, MIT Google Brain Adobe Research Asma Ghandeharioun Matthew
More informationStochastic Simulation with Generative Adversarial Networks
Stochastic Simulation with Generative Adversarial Networks Lukas Mosser, Olivier Dubrule, Martin J. Blunt lukas.mosser15@imperial.ac.uk, o.dubrule@imperial.ac.uk, m.blunt@imperial.ac.uk (Deep) Generative
More information