GANs for Exploiting Unlabeled Data. Presented by: Uriya Pesso Nimrod Gilboa Markevich

Transcription

1 GANs for Exploiting Unlabeled Data Improved Techniques for Training GANs Learning from Simulated and Unsupervised Images through Adversarial Training Presented by: Uriya Pesso Nimrod Gilboa Markevich

2 [ ] you could not see an article in the press [about AI] without the picture being Terminator. It was always Terminator, 100 percent. And you see less of that now, and that s a good thing Yann LeCun, Director, Facebook AI

3 Generative Adversarial Networks is the most interesting idea in the last ten years in machine learning. Yann LeCun, Director, Facebook AI

4 Presentation Overview o Motivation o Intro to GANs o Paper 1: Semi Supervised Learning o Paper 2: Simulated and Unsupervised Learning o Conclusion

5 Motivation Compensating for Missing Data o Labeled data is expensive (time, money, effort) o Unlabeled data is cheaper o Unlabeled data still contains information o How can we utilize unlabeled data? o GANs! We will present two methods from two papers

6 Generative Adversarial Networks o What does it do? Generate synthetic data that is indistinguishable from real data o Uses Super-Resolution Text-to-Speech Art Exploiting unlabeled data (for training other networks)

7 Generative Adversarial Networks o Adversarial Networks Opponent Networks Generator Create realistic samples Discriminator Distinguish generated from real o Compete with each other o The only labels are Real/Fake o Gradient Descent, Train in turns Train D Train G

8 Loss Functions o Cross-entropy loss function o o o o o Dx G z D G L L D G, x p x D D G probability of x being real generated sample from noise z network parameters discriminator loss function Generator loss function data z z L, log D log 1 D G ~ ~noise L G D L

9 Improved Techniques for Training GANs Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen

10 Semi-Supervised Learning - Idea o Objective: We want to train a classifier o We have: labeled data => supervised learning unlabeled data => unsupervised learning o Combining labeled and unlabeled data => semi-supervised

11 Semi-Supervised Learning - Idea o Combine GAN and classifier networks. o Add a label for the synthetic data - K+1 Supervised x Classifier y y 1,.., K Unsupervised z G Generator x* G(z) D Discriminator y y Generated,Real Semi-Supervised z G Generator x or x* G(z) D Classifier y y 1,.., K, K 1

12 Semi-Supervised Loss Function o Cross entropy loss over two distributions L x x log p y x, y~ p (, y) model data L log p y, y K 1 supervised x, y~ p ( x, y) model x data data G log p y K 1 x G x L log 1 p y K 1 log p y K 1 unsupervised x~ p ( x) model x~ model x~ model x supervised unsupervised L L pdata x, y - is the probability distribution of the input data. log pmodel y x - is the model probability distribution to predict label y from K labels given data X.

13 Supervised Loss o Where: p x x Classifier y data - is the probability distribution of the input data. log p y x - is the model probability distribution to predict label y from K labels given data X. model y 1,.., K, K 1 L log p y, y K 1 x x x suprvised, y~ p (, y) model data

14 Supervised Loss x Classifier y data y 1,.., K L x y x log p y x suprvised, ~ p (, y) model o pdata x - is the probability distribution of the input data. o log p y x - is the model probability distribution to predict label y from K labels given data X. model

15 Unsupervised Loss z G Generator x p x D D G data x G(z) D Discriminator y z z L, log D log 1 D G D( x) 1 p ( y K 1 x) model ~ ~noise y Generated,Real L log(1 p ( y K 1 x)) log( p ( y K 1 x)) unsuprvised x~ p ( x) model x~ G( z) model data o p ( y K 1 x) model the probability distribution to predict that the data x is unreal.

16 Punchline Classifier GAN L L L supervised unsupervised L log p y, y K 1 supervised x, y~ p ( x, y) model data data x x G x L log 1 p y K 1 log p y K 1 unsupervised x~ p ( x) model x~ model

17 Semi-Supervised Learning Intuition o How does a classifier benefit from unlabeled data? o From the unlabeled data, the classifier learns to focus on the right features, thus reducing generalization error Number Not Number

18 Results

19 Results - MNIST Generated Real

20 Results - MNIST (*) (*) number of labeled samples per class the rest are unlabeled train size: 60,000 test size: 10,000

21 Results CIFAR10 CIFAR10 Generated

22 Results Imagenet DCGAN (Not Ours )

23 Results Imagenet Our Method

24 Learning from Simulated and Unsupervised Images through Adversarial Training CVPR 2017 Best Paper Award Avish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, Russ Webb Apple Inc

25 Motivation o Labeled data is expensive (time, money, effort) o Simulated data is cheap o Alternatively, we could learn using simulated data Simulator Data Set of Labeled Synthetic Images NN

26 Motivation o Drawback: Simulated data may have artifacts o We want more realistic synthetic data o We have unlabeled data o Let s build a refiner

27 Refiner Network / S+U Learning Simulator Data Set of Labeled Synthetic Images

28 SimGAN Architecture

29 SimGAN Architecture

30 Loss Function Discriminator o Adversarial Loss x p D D R real x L, log D log 1 D R x~ x x~ p x simulated o o o o Dx Rx D R L D, probability of x being real refined simulated image network parameters discriminator loss function o o preal psimulated x x pdf of real images pdf of simulated images

31 Loss Function Refiner o Refiner has two goals Generate realistic samples Adversarial Loss Preserve the label Self-regularization, x~ p x R D R L simulated real reg real log D Rx Rx x reg 1 o o o R x refined simulated image o p x pdf of simulated images D R, network parameters simulated R L refiner loss function o weight

32 Minor Improvements o Are we done? o Almost. There are two additional mechanisms: Local Adversarial Loss Using a History of Refined Images

33 Local Adversarial Loss o Discriminator outputs wxh probability map o The adversarial loss is the sum of the loss over the local patches o Localization restrains artifacts the refined image should look real in every patch

34 Local Adversarial Loss

35 Using a History of Refined Images o Problem Discriminator only focuses on the latest refined images Refiner might reintroduce artifacts that the discriminator has forgotten o Solution Buffer refined images

36 Using a History of Refined Images

37 Using a History of Refined Images

38 Results

39 Stages for SimGAN Performance Evaluation 1) Train SimGAN 2) Generate Synthetic Refined Dataset => Qualitative Results 3) Train NN (Estimator) with the Dataset 4) Test NN (Estimator) => Quantitative Results

40 Results - Performance Evaluation using Gaze Estimatior o Simulated images UnityEyes, 1.2M o Real Labeled Dataset MPIIGaze Dataset, 214K Gaze Estimator

41 Process 1. Train SimGAN UnityEyes MPIIGaze (without labels)

42 Process 2. Trained Refiner Generates DB SimGAN Refiner Data Set of Labeled Synthetic Images Data Set of Labeled Refined Synthetic Images

43 Qualitative Results

44 Process 3. Train Gaze Estimator Gaze Estimator Data Set of Labeled Synthetic Refined Images

45 Process 4. Test Gaze Estimator Gaze Estimator CNN Data Set of Real Images Output

46 Quantitative Results

47 Results Performance Evaluation using Hand Pose Estimatior o NYU hand pose dataset, 73,000 training, 8,000 testing Hand Pose Estimator Depth Image Selected Points Coordinates

48 Qualitative Result

49 Quantitative Results

50 Quantitative Results

51 Conclusion o Generative Adversarial Networks are awesome Generator vs. Discriminator o Unlabeled data can be used for supervised learning Semi-Supervised Learning Classifier combined with Discriminator Train GAN with labeled and unlabeled data Simulated and Unsupervised Learning Train Refiner Generate large synthetic refined dataset