Introduction to GAN. Generative Adversarial Networks. Junheng(Jeff) Hao

Transcription

1 Introduction to GAN Generative Adversarial Networks Junheng(Jeff) Hao

2 Adversarial Training is the coolest thing since sliced bread. -- Yann LeCun

3 Roadmap 1. Generative Modeling 2. GAN 101: What is GAN? How does it work? 3. Improvement: From GAN to W-GAN 4. GAN Applications 5. Summary

4 Roadmap 1. Generative Modeling 2. GAN 101: What is GAN? How does it work? 3. Improvement: W-GAN 4. GAN Applications 5. Summary

5 Generative vs Discriminative Discriminative Models Learn P(y x) Directly characterizes the decision boundary between classes only Examples: Logistic Regression, SVM, etc Generative Models Learn P(x,y) Characterize how data is generated (distribution of individual class) Examples: Naive Bayes, HMM, etc. Also includes: Autoencoders (VAEs), GAN.

6 What can generative modeling do? Use high-dimensional, complicated probability distributions Missing data, semi-supervised learning, unsupervised learning Multimodal outputs Realistic generation tasks...

7 Review: Maximum Likelihood ML Method: which is equivalent to: that minimizes Kullback Leibler divergence (KL-divergence).

8 Supplementary Slides Proof: Maximum Likelihood = Minimum KL-Divergence

9 From KL Divergence to JS Divergence KL Divergence JS Divergence Both of them are measures of how one probability distribution diverges from a second.

10 Roadmap 1. Generative Modeling 2. GAN 101: What is GAN? How does it work? 3. Improvement: W-GAN 4. GAN Applications 5. Research Frontiers 6. Summary

11 Story: Counterfeiter and Police Counterfeiter: Try to make fake money Police: Allow legitimate money and catch fake money Question: What is the equilibrium?

12 Adversarial Nets Framework

13 Generator Network Characteristics: Must be differentiable (for training) Trainable for any size of z Normally z has lower dimension than x

14 Minimax Game Resembles of Jensen-Shannon Divergence (See later) Discriminator: Cross-entropy Generator: minimize log-probability of D being correct Question: What is the solution to D(x)?

15 Solution: Minimax Game Assume both densities are non-zero everywhere (keep in mind) Solve for where the functional derivatives are zero:

16 GAN: Change Loss Function Heuristic loss function for generator Discriminator: Cross-entropy Generator: maximize log-probability of discriminator being mistaken Advantage: Generator can still learn when discriminator rejects generator samples Disadvantage: Discuss later

17 GAN: Algorithm

18 Example: 1-D GAN

19 Recap: Understanding basic idea of GAN

20 DCGAN: A better design of GAN

21 DCGAN: Results

22 Tips and Tricks of Training a GAN One-side label smoothing Input normalization, Batchnorm Sample from a gaussian distribution rather than uniform distribution Avoid sparse gradient: ReLU, Max pooling Hybrid models Use labels if they are avaible...

23 Other versions of GANs W-GAN Conditional GAN (CGAN) fgan SeqGAN Energy-based GAN Others: Ensemble of GAN

24 Supplementary Slides Generative Adversarial Models vs Adversarial Samples Generative Adversarial Models A mechanism for training a generative model Adversarial Samples A analysis tool for neural network

25 Roadmap 1. Generative Modeling 2. GAN 101: What is GAN? How does it work? 3. Improvement: W-GAN 4. GAN Applications 5. Summary

26 Maximum Likelihood = Good samples?

27 GAN: Problems Difficulty on training Loss of discriminator and generator can not indicate the performance of generated samples. Low diversity of generated samples and mode collapse.

28 Analysis of Loss Function (1) Previous loss function for generator in MinMax s Game: Then we have Which is the form of Jensen-Shannon Divergence.

29 Analysis of Loss Function (2) Now look at second heuristic version: Generator Loss: We already prove: Finally,

30 Intro: Earth Mover Distance What is the distance from Beijing to Los Angeles? And, what is the distance from China to United States? Reference:

31 Intro: Earth Mover Distance Let s move China to United States! Reference:

32 Wasserstein Distance Wasserstein distance (Earth-Mover Distance) Question: Which distance measurement can provide meaningful gradient?

33 Exercise 1 Given two 1-D uniform distribution AB(P1) and CD(P2) in 2-D space, θ controls the distance of P1 and P2. What is the KL(P1 P2), JS(P1 P2), W(P1 P2)?

34 Problem with {JS, KL} divergence Solution: Note: Only Wasserstein Distance can provide effective gradient! JS and KL divergence cannot reflect the similarity between two probability distributions!

35 From Wasserstein to W-GAN Transform the objective: Note: Lipschitz constant K (range restriction factor) Proof in detail:

36 From Wasserstein to W-GAN Using MLP or CNN for fω with parameter ω inside Generator Loss: Discriminator Loss:

37 WGAN: Algorithm

38 A glimpse of results (WGAN-DCGAN)

39 How to measure results?

40 Effect of W-GAN What does W-CAN change? Define new loss function: Wasserstein Distance and apply to WGAN Remove top sigmoid-function layer: Regression Problem Weight clipping in [-c,c] Choice for gradient descent algorithms: RMSProp not Adam What is the advantage of W-CAN? Novel distance design with valid gradient throught training Tackle the problem of mode collapse Good results without well-designed architecture of generator

41 Roadmap 1. Generative Modeling 2. GAN 101: What is GAN? How does it work? 3. Improvement: W-GAN 4. GAN Applications 5. Summary

42 ICCV 2017 Face Rotation based on GAN Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis

43 CVPR 2017 Super-Resolution Image Generation

44 Results: SRGAN

45 Tomer Weiss: Building Information Design Synthesis(BIDS)

46 Tomer Weiss: BIDS: Let GAN help with your design! From Sketch directly to BIM Basic Input modules: Walls, colors and shapes Small dataset (200 pictures samples) Still need techniques of computer graphics (Filters, PointNet...) Credit to Tomer Weiss. See personal page at

47 Tomer Weiss: BIDS: Demo Ongoing Autodesk project under NDA...

48 Others Text generation Drug discovery and biomarker development Speech synthesis / Music generator Image segmentation Image-text translation (description) Medical image... For a summary of recent CAN Collection, please see this Really Awesome GAN at

49 Roadmap 1. Generative Modeling 2. GAN 101: What is GAN? How does it work? 3. Improvement: W-GAN 4. GAN Applications 5. Summary

50 Summary GANs are generative models that use supervised learning to approximate a cost function. GANs are relatively new and still require more research to reach their potential. Better theoretical understanding and training algorithms are strongly needed. GANs are crucial to many different state-of-art methods of image generation and enable many other applications.

51 Before the end: GUN STOPPING GAN VIOLENCE: GENERATIVE UNADVERSARIAL NETWORKS (GUN) Note: If you are interested, please see for this humor.

52 Q&A

53 Thanks! Contact Me: ScAi Research Lab 3551 Boelter Hall University of California, Los Angeles

54 Supplementary Slides Generative models: Taxonomy Maximum Likelihood Explicit density Implicit density Tractable density Approximate density Markov Chain Nonlinear ICA, etc Variational Autoencoder Boltzmann Machine Generative Stochastic Networks Direct GAN

55 Supplementary Slides Example: 1-D CAN (by Aylien from Youtube)