GAN and Feature Representation. Hung-yi Lee

Transcription

1 GAN and Feature Representation Hung-yi Lee

2 Outline Generator (Decoder) Discrimi nator + Encoder GAN+Autoencoder x InfoGAN Encoder z Generator Discrimi (Decoder) x nator scalar Discrimi z Generator x scalar Encoder z (Decoder) nator BiGAN z x Decoder Encoder x z x z Discrimi nator scalar

3 GAN + Autoencoder

4 Photo Editing

5 Photo Editing z Generator (Decoder) x We can tune z to edit image x How to modify a specific attribute?

6 GAN+Autoencoder We have a generator (input z, output x) However, given x, how can we find z? Learn an encoder (input x, output z) as close as possible x different structures? Generator Encoder z x (Decoder) init fixed Discriminator Autoencoder

7 as close as possible x Generator Encoder z x (Decoder)

8 Attribute Representation CelebA z male = 1 N 1 x male En x 1 N 2 x male En x Female image x En x + z male = z Gen z male image

9 Find the Attributes z z male z male = 0.76

10 Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman and Alexei A. Efros. "Generative Visual Manipulation on the Natural Image Manifold", ECCV, 2016.

11 Basic Idea space of z

12 Back to z Method 1 z = arg min L G z, x T z Gradient Descent Method 2 z as close as possible x T Difference between G(z) and x T Pixel-wise By another network Method 3 x Generator Encoder z x (Decoder) Using the results from method 2 as the initialization of method 1

13 Back to z - Results

14 Editing Photos z 0 is the code of the input image image Using discriminator to check the image is realistic or not z = arg min z U G z + λ 1 z z 0 2 λ 2 D G z Not too far away from the original image Does it fulfill the constraint of editing?

15 Editing Photos - Results

16 Final System

17 Andrew Brock, Theodore Lim, J.M. Ritchie, Nick Weston, Neural Photo Editing with Introspective Adversarial Networks, arxiv preprint, 2017

18 VAE-GAN Anders Boesen, Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, Ole Winther, Autoencoding beyond pixels using a learned similarity metric, ICML Minimize reconstruction error z close to normal Minimize reconstruction error Cheat discriminator Discriminate real, generated and reconstructed images x Encoder z Generator Discrimi (Decoder) x nator scalar as close as possible VAE Discriminator provides the similarity measure GAN

19 ǁ Algorithm Initialize En, De, Dis In each iteration: Sample M images x 1, x 2,, x M from database Generate M codes zǁ 1, zǁ 2,, zǁ M from encoder zǁ i = En x i Generate M images x 1, x 2,, x M from decoder x i = En z i Sample M codes z 1, z 2,, z M from prior P(z) Another kind of Generate M images x 1, x 2,, x M from decoder discriminator: x i = En z i real gen recon Update En to decrease x i x i, decrease KL(P( zǁ i x i ) P(z)) Discriminator Update De to decrease x i x i, increase Dis x i and Dis x i x Update Dis to increase Dis x i, decrease Dis x i and Dis x i

20 VAE+GAN - Sample blurry sharp sharp

21 VAE+GAN - Reconstruction GAN cannot do reconstruction

22 InfoGAN

23 What is InfoGAN? z = z c Generator (Decoder) x Discrimi nator scalar Auto-encoder Predict the code c that generates x Encoder c Parameter sharing (only the last layer is different)

24 Motivation Regular GAN z c Generator (Decoder) x Encoder c c must have clear influence on x, so the encoder can recover c from x c will be easy to interpret A specific dimension c i cannot cooperate with other feature dimensions to have influence.

25

26

27

28

29 BiGAN Jeff Donahue, Philipp Krähenbühl, Trevor Darrell, Adversarial Feature Learning, ICLR, 2017 Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, Aaron Courville, Adversarially Learned Inference, ICLR, 2017

30 BiGAN code z (from prior distribution) code z from encoder or decoder? Encoder Decoder Discriminator Image x (real) Image x (generated) Image x code z

31 Algorithm Initialize encoder En, decoder De, discriminator Dis In each iteration: Sample M images x 1, x 2,, x M from database Generate M codes zǁ 1, zǁ 2,, zǁ M from encoder zǁ i = En x i Sample M codes z 1, z 2,, z M from prior P(z) Generate M codes x 1, x 2,, x M from decoder x i = De z i Update Dis to increase Dis x i, zǁ i, decrease Dis x i, z i Update En and De to decrease Dis x i, ǁ z i, increase Dis x i, z i

32 code z (from prior distribution) code z from encoder or decoder? Encoder Decoder Discriminator Image x Image x (real) (generated) P x, z Q x, z To make P and Q the same Image x code z Evaluate the difference between P and Q Optimal encoder and decoder: En(x ) = z De(z ) = x De(z ) = x En(x ) = z For all x For all z

33 BiGAN Optimal encoder and decoder: En(x ) = z De(z ) = x De(z ) = x En(x ) = z For all x For all z How about? x zǁ Decoder z Encoder x En, De Encoder x Decoder z En, De

34

35

36 Concluding Remarks GAN+Autoencoder x InfoGAN Encoder z Generator Discrimi (Decoder) x nator scalar Discrimi z Generator x scalar Encoder z (Decoder) nator BiGAN z x Decoder Encoder x z x z Discrimi nator scalar

37 Next Time: Energy-based GAN

38 Original Idea Discriminator Data (target) distribution Generated distribution Discriminator leads the generator D x D x D x D x

39 Original Idea When the data distribution and generated distribution is the same. The output of discriminator will be flat everywhere. However, discriminator is often used in pre-training. It contains useful information. We always use the discriminator obtained in the last iteration as the initialization of the next step.

40 Energy-based Model We want to find an evaluation function F(x) Input: object x (e.g. images), output: scalar (how good x is) Real x has high F(x) F(x) can be a network We can find good x by F(x): Generate x with large F(x) x Evaluation Function scalar How to find F(x)? F(x) real data

41 Energybased GAN We want to find an evaluation function F(x) How to find F(x)? F x F x In the end F x F x

42 Energy-based Model Preview: Framework of structured learning (Energy-based Model) ML Lecture 21: Structured Learning - Introduction ML Lecture 22: Structured Learning - Linear Model ML Lecture 23: Structured Learning - Structured SVM ML Lecture 24: Structured Learning - Sequence Labeling Graphical model & Gibbs sampling _2/Lecture/MRF%20(v2).ecm.mp4/index.html