Deep generative models of natural images

Transcription

1 Spring 2016

2 1 Motivation 2 3 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models 4

3 Outline 1 Motivation 2 3 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models 4

4 Generative models Have access to x p data (x) through training set Want to learn a model x p model (x) Want p model to be similar to p data Samples drawn from p model reflect structure of p data Samples from true data distribution have high likelihood under p model

5 Why do generative modeling? Unsupervised representation learning Can transfer learned representation so discriminative tasks, retrieval, clustering, etc. Train network with both discriminative and generative criterion Utilize unlabeled data, regularize Understand data Density estimation Data augmentation...

6 Focus of this talk Generative modeling is a HUGE field...i will focus on (a selected set of) deep directed models of natural images

7 Outline 1 Motivation 2 3 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models 4

8 Directed graphical models z x We assume data is generated by: z p(z) x p(x z) z is latent/hidden x is observed (image) Use θ to denote parameters of the generative model

9 Deep directed graphical models P (x) = h P (x h)p (h) Intractable Can t optimize data likelihood directly

10 Bounding the marginal likelihood Recall Jenson s inequality: When f is concave, f(e[x]) E[f(x)] log p(x) = log z = log z z = z p(x, z) p(x, z) q(z) q(z) p(x, z) q(z) q(z) log = L(x; θ, φ) (by Jensons inequality) q(z) log p(x, z) q(z) log q(z) = E q(z) [log p(x, z)] }{{} + H(q(z)) }{{} Expectation of joint distribution Entropy z

11 Evidence Lower BOund (ELBO) Bound is tight when variational approximation matches true posterior: log p(x) L(x; θ, φ) = log p(x) q(z) log z = q(z) log p(x) z = z q(z) log q(z)p(x) p(x, z) = D KL (q(z; φ) p(z x)) z p(x, z) q(z) q(z) log p(x, z) q(z)

12 Summary Assume existence of q(z; φ) Bound log p(x; θ) with L(x; θ, φ) Bound is tight when: D KL (q(z; φ) p(z x)) = 0 q(z; φ) = p(z x)

13 Learning directed graphical models Maximize bound on likelihood of data: max θ N log p(x i ; θ) i=1 max θ,φ 1,...,φ N N L(x i ; θ, φ i ) i=1 Historically, used different φ i for every data point But we ll move away from this soon.. q(z; φ) typically factorized distribution For more info see Blei et al. (2003)

14 New method of learning: approximate inference model Instead of having different variational parameters for each data point, fit a conditional parametric function The output of this function will be the parameters of the variational distribution q(z x) Instead of q(z) we have q φ (z x) ELBO becomes: L(x; θ, φ) = E qφ (z x)[log p θ (x, z)] }{{} + H(q φ (z x)) }{{} Expectation of joint distribution Entropy

15 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models Outline 1 Motivation 2 3 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models 4

16 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models Variational autoencoder Encoder network maps from image space to latent space Outputs parameters of q φ (z x) Decoder maps from latent space back into image space Outputs parameters of p θ (x z) [Kingma & Welling (2013)]

17 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models Variational autoencoder Rearranging the ELBO: L(x; θ, φ) = = = z z z q φ (z x) log q φ (z x) log z p(x, z) q φ (z x) p(x z)p(z) z q φ (z x) q φ (z x) log p(x z) + q φ (z x) log = E q(z x) log p(x z) E q(z x) log q(z x) p(z) = E q(z x) log p(x z) D KL (q(z x) p(z)) }{{}}{{} Reconstruction term Prior term z p(z) q φ (z x)

18 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models Variational autoencoder Inference network outputs parameters of q φ (z x) Generative network outputs parameters of p θ (x z) Optimize θ and φ jointly by maximizing ELBO: L(x; θ, φ) = E q(z x) log p(x z) D KL (q(z x) p(z)) }{{}}{{} Reconstruction term Prior term

19 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models Stochastic gradient variation bayes (SGVB) estimator Reparameterization trick : re-parameterize z q φ (h z) as z = g φ (x, ɛ) with ɛ p(ɛ) For example, with a Gaussian can write z N (µ, σ 2 ) as z = µ + ɛσ 2 with ɛ N (0, 1) [Kingma & Welling (2013); Rezende et al. (2014)]

20 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models Stochastic gradient variation bayes (SGVB) estimator L(x; θ, φ) = E q(z x) log p(x z) D KL (q(z x) p(z)) }{{}}{{} Reconstruction term Prior term Using reparameterization trick we form Monte Carlo estimate of reconstruction term: E qφ (z x) log p θ (x z) = E p(ɛ) log p θ (x g φ (x, ɛ)) 1 L log p θ (x g φ (x, ɛ)) L i=1 where ɛ p(ɛ) KL divergence term can often be computed analytically (eg. Gaussian)

21 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models VAE learned manifold [Kingma & Welling (2013)]

22 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models VAE samples [Kingma & Welling (2013)]

23 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models VAE tradeoffs Pros: Theoretically pleasing Optimizes bound on likelihood Easy to implement Cons: Samples tend to be blurry Maximum likelihood minimizes D KL(p data p model ) [Theis et al. (2016)]

24 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models Generative adversarial networks Don t focus on optimizing p(x), just learn to sample Two networks pitted against one another: Generative model G captures data distribution Discriminative model D distinguishes between real and fake samples [Goodfellow et al. (2014)]

25 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models Generative adversarial networks D is trained to estimate the probability that a sample came from data distribution rather than G G is trained to maximize the probability of D making a mistake min G max E x p D data(x) log D(x) + E z pnoise(z) log(1 D(G(z)))

26 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models GAN samples

27 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models GAN tradeoffs Pros: Very powerful model High quality samples Cons: Tricky to train (no clear objective to track, instability) Can ignore large parts of image space Because approximately minimizing Jenson-Shannon divergence [Goodfellow et al. (2014); Theis et al. (2016); Huszar (2015)] [Theis et al. (2016)]

28 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models Generative moment matching networks Same idea as GANs, but different optimization method Match moments of data and generative distributions Maximum mean discrepancy Estimator for answering whether two samples come from same distribution Evaluate MMD on generated samples [Li et al. (2015); Dziugaite et al. (2015)]

29 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models Generative moment matching networks L MMD 2 = 1 N = 1 N 2 2 NM N φ(x i) 1 M i=1 N N i=1 i =1 M φ(x j) 2 j=1 φ(x i) φ(x i ) 1 M 2 N j = 1 M φ(x i) φ(x j) i=1 Can make use of kernel trick M j=1 j =1 M φ(x j) φ(x j ) If φ is identity, then matching means Complex φ can match higher order moments

30 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models GMMN samples

31 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models GMMN tradeoffs Pros: Theoretically pleasing Cons: Batch size very important Samples aren t great (get better when combined with autoencoder) [Theis et al. (2016)]

32 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models How to evaluate a generative model? Log likelihood on held out data Makes sense when goal is density estimation Many approaches don t have tractable likelihood or it isn t explicitly represented Have to resort to Parzen window estimates [Breuleux et al. (2009)] Quality of samples But a lookup table of training images will succeed here... Best: evaluate in context of particular application See Theis et al. (2016) for more details

33 Outline 1 Motivation 2 3 Variational autoencoders Generative adversarial networks Generative moment matching networks Evaluating generative models 4

34 DRAW: Deep Recurrent Attentive Writer Basic idea: Iteratively construct image Observe image through sequence of glimpses Recurrent encoder and decoder Optimizes variational bound Attention mechanism determines: Input region observed by encoder Output region modified by decoder [Gregor et al. (2015)]

35 DRAW samples

36 Generating images from captions Language model: bidirectional RNN Image model: conditional DRAW network Image sharpening with Laplacian pyramid adversarial network [Mansimov et al. (2016)]

37 Generating images from captions [Mansimov et al. (2016)]

38 Laplacian pyramid of adversarial networks Difficult to generate large images in one shot Break problem up into sequence of manageable steps Samples drawn in coarse-to-fine fashion Each scale is a convnet trained using GAN framework [Denton et al. (2015)]

39 LAPGAN training procedure I 2 I 2 I 3 I 3 I = I 0 I 1 I 1 z 0 z 1 ~ z 2 I 3 l 2 D 3 Real/ Generated? l 0 G 0 h 1 l 1 D 1 ~ h 1 G 1 D 2 G 2 G 3 h 2 ~ h 2 z 3 Real/ Generated? h 0 ~ h 0 Real/Generated? D 0 Real/Generated?

40 LAPGAN samples

41 LAPGAN samples

42 Deep convolutional generative adversarial networks (DCGAN) Radford et al. (2016) propose several tricks to make GAN training more stable

43 DCGAN vector arithmetic

44 Texture synthesis with spatial LSTMs Two dimensional LSTM Sequentially predict pixels in an image conditioned on previous pixels [Theis & Bethge (2015)]

45 Texture synthesis with spatial LSTMs [Theis & Bethge (2015)]

46 Pixel recurrent neural networks Sequentially predict pixels in an image conditioned on previous pixels Uses spatial LSTM [van den Oord et al. (2016)]

47 Appendix References References I Blei, David M., Ng, Andrew Y., & Jordan, Michael I Latent Dirichlet Allocation. Journal of Machine Learning Research. Breuleux, O., Bengio, Y., & Vincent, P Unlearning for better mixing. Technical report, Universite de Montreal. Denton, Emily, Chintala, Soumith, Szlam, Arthur, & Fergus, Rob Deep generative image models using a laplacian pyramid of adversarial networks. In: NIPS. Dziugaite, Gintare Karolina, Roy, Daniel M., & Ghahramani, Zoubin Training generative neural networks via Maximum Mean Discrepancy optimization. In: UAI.

48 Appendix References References II Goodfellow, Ian J., Pouget-Abadie, Jean, Mirza, Mehdi, Xu, Bing, Warde-Farley, David, Ozair, Sherjil, Courville, Aaron C., & Bengio, Yoshua Generative adversarial networks. In: NIPS. Gregor, Karol, Danihelka, Ivo, Graves, Alex, & Wierstra, Daan DRAW: A Recurrent Neural Network For Image Generation. CoRR, abs/ Huszar, F How (not) to train your generative model: schedule sampling, likelihood, adversary? arxiv preprint arxiv: Kingma, Diederik P, & Welling, Max Auto-encoding variational bayes. arxiv preprint arxiv:

49 Appendix References References III Li, Yujia, Swersky, Kevin, & Zemel, Richard Generative Moment Matching Networks. In: ICML. Mansimov, Elman, Parisotto, Emilio, Ba, Jimmy, & Salakhutdinov, Ruslan Generating Images from Captions with Attention. In: ICLR. Radford, Alec, Metz, Luke, & Chintala, Soumith Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. In: ICLR. Rezende, Danilo Jimenez, Mohamed, Shakir, & Wierstra, Daan Stochastic backpropagation and approximate inference in deep generative models. arxiv preprint arxiv: Theis, Lucas, & Bethge, Matthias Generative Image Modeling Using Spatial LSTMs. In: NIPS.

50 Appendix References References IV Theis, Lucas, van den Oord, Aaron, & Bethge, Matthias A note on the evaluation of generative models. In: ICLR. van den Oord, Aaron, Kalchbrenner, Nal, & Kavukcuoglu, Koray Pixel Recurrent Neural Networks. In: ICML.