Generative Adversarial Network

Transcription

1 Generative Adversarial Network Many slides from NIPS 2014 Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio

2 Generative adversarial networks New method of training deep generative models Idea: pit a generator and a discriminator against each other Generator tries to draw samples from P(X) Discriminator tries to tell if sample came from the generator or t he real world Both discriminator and generator are deep networks (differentia ble functions) Can train with backprop: train discriminator for a while, then trai n generator, then discriminator, 2

3 Generative? Data: Discriminative model: p D x; θ D Generative model: p G x; θ G True data distribution: p data (x) Train p G x p data (x) p data (x) 3

4 Generative Adversarial Network Counterfeiters vs Police Game IT S FAKE MONEY! IT S REAL MONEY! 4

5 5

6 Generative Adversarial Network Discriminative Model D Generative Model G Sample x Random noise z p(z) 6

7 Generative Adversarial Network Discriminative Model D: tries to distinguish between samples from real data p(x) and generated ones q(x). 1 0 Try to classify the sample x D(x)=0 when x from Data D(x)=1 when x from G Differentiable function represented by a multilayer perceptron with parameters sample x from data sample x from G Generative Model G Try to generate sample x As similar as the real data random noise z p(z) 7

8 8

9 9

10 10

11 11

12 To learn the G s distribution p g over data x, we define a prior on input noise variables p z (z) Represent a mapping to data space as G(z; θ g ) whre G is a differ entiable function (MLP) A second multilayer perceptron D(x; θ d ) that outputs a single sc alar. D(x) represents the probability that x came from the data rather than p g We train D to maximize the probability of assigning the correct l abel to both training example and samples from G. We simultaneously train G to minimize log 1 D(G z ) D and G play the following 2-player minimax game with value fu nction V(G,D): 12

13 Generative Adversarial Network V D, G = E x~pdata(x) log D x min max V D, G G D + E z~pz(z) log(1 D(G(z))) Discriminative Model D 1 0 Generative Model G random noise z p(z) 13

14 Generative Adversarial Network V D, G = E x~pdata(x) log D x min max V D, G G D + E z~pz(z) log(1 D(G(z))) Discriminative Model D 1 0 Fixed G, maximize V: max D V G D = max D E x~pdata(x) log D x From sample x (i), z (i) max D m i=1 log D x i + E z~pz(z) log(1 D(G(z))) + log 1 D G z i Binary Classification (logistic loss): Sample from data: label=1 Sample from generator: label = 0 Generative Model G Stochastic Gradient random noise z p(z) 14

15 Generative Adversarial Network V D, G = E x~pdata(x) log D x min max V D, G G D + E z~pz(z) log(1 D(G(z))) Discriminative Model D 1 0 Fixed D, minimize V(G): min G V D G = min E z~pz(z) log(1 D(G(z))) Try to make D(G(z)) = 1 Generative Model G Stochastic Gradient random noise z p(z) 15

16 Generative Adversarial Network Update D Update G 16

17 GAN Results Nearest training example Generated Samples 17

18 Slide credit: Sangdoo (pil.snu.ac.kr) Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Network, ICLR Alec Radford, Luke Metz and Soumith Chintala 18

19 DCGAN Deep Convolutional Network + GAN Tricks for stable training Experimental Analysis Discriminative Model D 1 0 Generative Model G random noise z p(z) 19

20 Discriminative Model D DCGAN 1 0 Replace model s network to CNN Example of generator G (same as D) Generative Model G random noise z p(z) z: uniform dist. 20

21 Experiments LSUN bedroom dataset 3 million training examples Epoch #1 Epoch #5 21

22 Experiments Input noise z Interpolation form z i to z j z i z j 22

23 Experiments Which activations(feature map) in CNN has representation of window? At feature activations, assign neuron in the window region is 1, otherwise 0. Logistic regression to find window-representative feature map. Window feature map removal 23

24 Experiments Faces scraped human face image from web 3 million images from 10,000 people. Vector arithmetic 24

25 Experiments 25

26 Slide credit: Sangdoo (pil.snu.ac.kr) Generative Adversarial Text to Image Synthesis, ICML Scott Reed*, Zeynep Akata**, Xinchen Yan*, Lajanugen Logeswaran*, Bernt Schiele**, Honglak Lee* * University of Michigan, Ann Arbor, MI, USA (UMICH.EDU) ** Max Planck Institute for Informatics, Saarbrucken, Germany (MPI- INF.MPG.DE) 26

27 Generative Adversarial Text to Image Synthesis 27

28 Generative model What I cannot create, I do not understand Richard Feynman Generating images Image data (e.g. ImageNet): samples from the true data distribution. Generative model (e.g. deep neural network): outputs images, which means samples from the model. 28

29 Review of GAN Counterfeiters vs Police Game IT S FAKE MONEY! IT S REAL MONEY! 29

30 Review of GAN Discriminator Model D Generator Model G Sample x Random noise z p(z) 30

31 Review of GAN Discriminator Model D 1 0 Try to classify the sample x D(x)=0 when x from Data D(x)=1 when x from G (generator) Differentiable function represented by a multilayer perceptron with parameters sample x from data sample x from G Generator Model G Try to generate sample x As similar as the real data random noise z p(z) 31

32 Generative Adversarial Network V D, G = E x~pdata(x) log D x min max V D, G G D + E z~pz(z) log(1 D(G(z))) Discriminator Model D 1 0 Generator Model G random noise z p(z) 32

33 Review of DCGAN Discriminative Model D 1 0 Replace model s network to CNN Example of generator G (same as D) Generative Model G random noise z p(z) z: uniform dist. 33

34 GAN text to image synthesis ψ t : text embedding function (map to 1024 dim) -> Fully-connected layer -> 128 dim Used pre-trained text encoder (can be done end-to-end manner) z~n 0,1 : 100 dim noise vector Text-Conditional GAN 34

35 Conditional GAN 35 Mirza, Mehdi, and Simon Osindero. "Conditional generative adversarial nets."arxiv preprint arxiv: (2014).

36 Text-conditional GAN (naïve) 128 dim 4*4*16 h = ψ t Real image & matched text Fake image & arbitrary text min G max D E x~p data log D x, h + E z~pz log 1 D G z, h, h 36

37 Matching-aware discriminator 128 dim 4*4*16 h = ψ(t) min G max D Real image & matched text E x~pdata log D x, h + E x~pdata log 1 D x, h +E z~pz log 1 D G z, h, h Real image & mismatched text 37 Fake image & matched text

38 Matching-aware Discriminator h = ψ(t) min G max D Real image & matched text E x~pdata log D x, h + E x~pdata log 1 D x, h +E z~pz log 1 D G z, h, h Real image & mismatched text Fake image & matched text 38

39 Learning with Manifold Interpolation h = ψ(t) Real image & matched text Real image & mis-matched text min G max D E x~pdata log D x, h + E x~pdata log 1 D x, h +E z~pz log 1 D G z, h, h Fake image & matched text Additional term to generator to minimize: E t1,t 2 ~p textdata log 1 D G z, h), h, h = βh β h 2 h 1 = ψ t 1, h 2 = ψ t 2 39

40 Experiments

41 Experiments

42 Style Transfer 128 dim 4*4*16 L style = E t,z~n 0,1 z S G z, h S x z 2 2 Finding an inverse mapping from an image to vector 42

43 Style Transfer L style = E t,z~n 0,1 z S G z, h S x z 2 2 Input image: x Style: z = S(x) Generated image: G(z, h) Image Style vector Text description Style transferred image

44 Sentence Interpolation 44

45 Pixel Level Domain Transfer, ECCV

46 Domain transfer 46

47 Whole architecture Converter: deconvnet Discriminator: Real vs. Fake Discriminator : Associated or not 47

48 Dataset 48

49 Results 49

50 Results varying input conditions 50

51 Results inverse setting 51

52 Image to Image Translation 52 Image-to-Image Translation with Conditional Adversarial Networks

53 Image to Image Translation + L1 loss function Low-freq correctness + PatchGAN High-freq correctness 53

54 Image to Image Translation 54

55 Image to Image Translation 55

56 Plug & Play Generative Networks ArXiv

57 Plug & Play Generative Networks 57

58 Plug & Play Generative Networks 58

59 Noiseless joint PPGN-h Update Rule for a feature vector h Training G Training D 59

60 Noiseless joint PPGN-h Encoder network is pre-trained. G and D are trained with standard GAN learning technique. G is not directly used to generate image, but used as a guiding function combined with DAE. 60

61 PPGN results 61

62 PPGN results 62

63 Learning What and Where to Draw NIPS

64 Motivation Generative adversarial what-where nets (GAWWN) Give a bbox Give part locations Give a part location 64

65 Bounding Box Control Generator input: z, text, bbox location 65

66 Bounding Box Control Discriminator input: real/fake image, text, bbox location 66

67 Bounding Box Control Overall structure 67

68 Keypoint-Conditional Control Generator input: z, text, keypoint location (e.g., head in channel 1, left foot in channel 2, 68

69 Keypoint-Conditional Control Discriminator input: real/fake image, text, keypoint location 69

70 Keypoint-Conditional Control Overall structure 70

71 Keypoint Generation There are too much efforts to enter all keypoints (e.g., 15 parts for a bird). Given a subset of keypoints, let s find the remaining keypoints location. Among many ways, they chose to use GAN. Keypoints: k i = x i, y i, v i, i = 1,, K v i = 1 if visible else 0 k 0,1 K 3 User input: s 0,1 K s i = 1 if given else 0 Given probability: 0.1 is use Generated Keypoints: f: R Z+T+3K R 3K MLP (3-layer fully connected network is used) Kepoints Discriminator: Distinguish (k real, t real ) from synthetic (Dind t say what is used. Maybe MLP.) 71

72 Experiments Bbox control (fix text and z / varying bbox) 72

73 Experiments Keypoint control (fix text / use gt keypoints / varying z) Keypoint control (fix text and z / varying beak and tail keypoint positions and generate other keypoints conditionally) 73

74 Experiments Keypoint control (fix text and z / generate all keypoints conditioned on text) 74

75