Generative Models II. Phillip Isola, MIT, OpenAI DLSS 7/27/18

Transcription

1 Generative Models II Phillip Isola, MIT, OpenAI DLSS 7/27/18

2 What s a generative model? For this talk: models that output high-dimensional data (Or, anything involving a GAN, VAE, PixelCNN, etc) Useful for lots of problems beyond density estimation and sampling random images!

3 What can you do with generative models? 1. Data prediction 2. Domain mapping 3. Representation learning 4. Model-based intelligence

4 What can you do with generative models? 1. Data prediction 2. Domain mapping 3. Representation learning 4. Model-based intelligence

5 Generative Model x z Gaussian noise Synthesized z N (~0, 1) image

6 Conditional Generative Model x y bird Synthesized image

7 Conditional Generative Model x y A yellow bird on a branch Synthesized image

8 Conditional Generative Model y x Synthesized image

9 Data prediction problems ( structured prediction ) Object labeling Edge Detection [Xie et al. 2015, ] [Long et al. 2015, ] Text-to-photo Future frame prediction this small bird has a pink breast and crown [Reed et al. 2014, ] [Mathieu et al. 2016, ]

10 Challenges in data prediction 1. Output is a high-dimensional, structured objects 2. Uncertainty in the mapping, many plausible outputs

11 Properties of generative models 1. Model high-dimensional, structured objects 2. Model uncertainty; a whole distribution of possible outputs

12 Conditional Generative Models Two ways to do it: 1. Model p(x,y), then do inference to get conditional p(y x) arg max y p(x, y) = arg max y p(y x) 2. Directly model p(y x)

13 Image-to-Image Translation Input x Output y Training data n x y o F, n o n,, o arg min F E x,y[l(f(x), y)] Objective function (loss) Neural Network [Zhang et al., ECCV 2016]

14 Image-to-Image Translation Input x Output y F arg min F E x,y[l(f(x), y)] What should I do How should I do it?

15 Designing loss functions Input Output Ground truth

16

17 Designing loss functions Input Zhang et al Ground truth Color distribution cross-entropy loss with colorfulness enhancing term.

18

19 Designing loss functions Be careful what you wish for!

20 Designing loss functions Image colorization L2 regression Super-resolution [Zhang, Isola, Efros, ECCV 2016] L2 regression [Johnson, Alahi, Li, ECCV 2016]

21 Designing loss functions Image colorization Cross entropy objective, with colorfulness term Super-resolution [Zhang, Isola, Efros, ECCV 2016] Deep feature covariance matching objective [Johnson, Alahi, Li, ECCV 2016]

22 ) Universal loss?

23 ) Generated images Generative Adversarial Network (GANs) Generated vs Real (classifier) Real photos [Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville, Bengio 2014]

24 x G G(x) Generator

25 x G G(x) D real or fake? Generator Discriminator G tries to synthesize fake images that fool D D tries to identify the fakes

26 x G G(x) D fake (0.9) y D real (0.1) arg max D E x,y [ log D(G(x)) + log(1 D(y)) ]

27 x G G(x) D real or fake? G tries to synthesize fake images that fool D: arg min G E x,y [ log D(G(x)) + log(1 D(y)) ]

28 x G G(x) D real or fake? G tries to synthesize fake images that fool the best D: arg min G max D E x,y [ log D(G(x)) + log(1 D(y)) ]

29 x G G(x) Loss Function D G s perspective: D is a loss function. Rather than being hand-designed, it is learned.

30 x G G(x) D real or fake? arg min G max D E x,y [ log D(G(x)) + log(1 D(y)) ]

31 x G G(x) D real! ( Aquarius ) arg min G max D E x,y [ log D(G(x)) + log(1 D(y)) ]

32 x G G(x) D real or fake pair? arg min G max D E x,y [ log D(G(x)) + log(1 D(y)) ]

33 x G G(x) D real or fake pair? arg min G max D E x,y[ log D(x,G(x)) + log(1 D(x, y)) ]

34 x G G(x) D fake pair arg min G max D E x,y[ log D(x,G(x)) + log(1 D(x, y)) ]

35 x G G(x) D real pair arg min G max D E x,y[ log D(x,G(x)) + log(1 D(x, y)) ]

36 x G G(x) D real or fake pair? arg min G max D E x,y[ log D(x,G(x)) + log(1 D(x, y)) ]

37 Conditional GAN Training Details: Loss function

38 Training Details: Loss function Conditional GAN x y G G(x) - Stable training + fast convergence [c.f. Pathak et al. CVPR 2016]

39 BW Color Input Output Input Output Input Output Data from [Russakovsky et al. 2015]

40 #edges2cats [Chris Hesse]

41 Ivy Vitaly

42 Structured Prediction Input x Output ŷ Target y L(ŷ, y) =kŷ yk 2

43 Structured Prediction CRF Each pixel treated as independent Models at pairwise configuration of pixels Y i p(y i x) 1 Z Y i,j p(y i,y j x)

44 Structured Prediction A GAN, with sufficient capacity, samples from the full joint distribution (at equilibrium) Most generative models have this property! Give them sufficient Model joint configuration of all pixels p(y x) capacity and infinite data, and they are the complete solution to prediction problems.

45 Shrinking the capacity: Patch Discriminator D 1/0 N pixels Rather than penalizing if output image looks fake, penalize if each N pixels overlapping patch in output looks fake y [Li & Wand 2016] [Shrivastava et al. 2017] [Isola et al. 2017]

46 Labels Facades Input 1x1 Discriminator Data from [Tylecek, 2013]

47 Labels Facades Input 16x16 Discriminator Data from [Tylecek, 2013]

48 Labels Facades Input 70x70 Discriminator Data from [Tylecek, 2013]

49 Labels Facades Input Full image Discriminator Data from [Tylecek, 2013]

50 Patch Discriminator D 1/0 Rather than penalizing if output image N pixels looks fake, penalize if each overlapping patch in output looks fake N pixels Faster, fewer parameters More supervised observations y Applies to arbitrarily large images

51 Shrinking the capacity: model misspecification [Theis et al. 2016]

52 Mode covering versus mode seeking [Larsen et al. 2016]

53 Modeling multiple possible outputs x G G(x)

54 Modeling multiple possible outputs????? Input Possible outputs

55 Conditional Variational Autoencoders [Sohn, Yan, Lee, NIPS 2015]

56 Example from [Babaeizadeh et al., ICLR 2018] see also [Walker et al., ECCV 2016], [Xue*, Wu*, et al., NIPS 2016]

57 x Observation y Target ŷ Reconstruction z Example from [Babaeizadeh et al., ICLR 2018] see also [Walker et al., ECCV 2016], [Xue*, Wu*, et al., NIPS 2016]

58 x Observation y Target ŷ Reconstruction z Example from [Babaeizadeh et al., ICLR 2018] see also [Walker et al., ECCV 2016], [Xue*, Wu*, et al., NIPS 2016]

59 x Observation y Target ŷ Reconstruction z z learns to encode the missing information necessary to predict y from x, i.e. the direction in which the purple box moves Example from [Babaeizadeh et al., ICLR 2018] see also [Walker et al., ECCV 2016], [Xue*, Wu*, et al., NIPS 2016]

60 x Observation ŷ Prediction z z N (~0, 1) see also [Walker et al., ECCV 2016], [Xue*, Wu*, et al., NIPS 2016]

61 x Observation ŷ Prediction z z N (~0, 1) see also [Walker et al., ECCV 2016], [Xue*, Wu*, et al., NIPS 2016]

62 x Observation ŷ Prediction z z N (~0, 1) see also [Walker et al., ECCV 2016], [Xue*, Wu*, et al., NIPS 2016]

63 x Observation ŷ Prediction z z N (~0, 1) see also [Walker et al., ECCV 2016], [Xue*, Wu*, et al., NIPS 2016]

64 x G y InfoGAN [Chen et al. 2016] BiCycleGAN [Zhu et al., NIPS 2017] z q(z y) Encourages z to relay information about the target.

65 Labels Randomly generated facades [BiCycleGAN, Zhu et al., NIPS 2017]

66 What can you do with generative models? 1. Data prediction 2. Domain mapping 3. Representation learning 4. Model-based intelligence

67 What can you do with generative models? 1. Data prediction 2. Domain mapping 3. Representation learning 4. Model-based intelligence

68 Paired data

69 Paired data Unpaired data Jun-Yan Zhu Taesung Park

70 x G G(x) D real or fake pair? arg min G max D E x,y[ log D(x,G(x)) + log(1 D(x, y)) ]

71 x G G(x) D real or fake pair? arg min G max D E x,y[ log D(x,G(x)) + log(1 D(x, y)) ] No input-output pairs!

72 Gaussian Target distribution z Y

73 Horses Zebras X Y

74 x G G(x) D real or fake? arg min G max D E x,y [ log D(G(x)) + log(1 D(y)) ] Usually loss functions check if output matches a target instance GAN loss checks if output is part of an admissible set

75 x G G(x) D Real!

76 x G G(x) D Real too! Nothing to force output to correspond to input

77 Cycle-Consistent Adversarial Networks [Zhu et al. 2017], [Yi et al. 2017], [Kim et al. 2017]

78 Cycle-Consistent Adversarial Networks

79 Cycle Consistency Loss

80 Cycle Consistency Loss

81

82

83 Failure case

84 Failure case

85 Why does CycleGAN work?

86 Slide credit: Ming-Yu Liu

87 Slide credit: Ming-Yu Liu

88 Simplicity hypothesis [Galanti, Wolf, Benaim, 2018]

89 Cycle Loss upper bounds Conditional Entropy Conditional Entropy High Conditional Entropy Low Conditional Entropy ALICE: Towards Understanding Adversarial Learning for Joint Distribution Matching [Li et al. NIPS 2017]. Also see [Tiao et al. 2018] CycleGAN as Approximate Bayesian Inference

90 Cycle Loss upper bounds Conditional Entropy Conditional Entropy ALICE: Towards Understanding Adversarial Learning for Joint Distribution Matching [Li et al. NIPS 2017]. Also see [Tiao et al. 2018] CycleGAN as Approximate Bayesian Inference

91 CycleGAN at School Course assignment code and handout designed by Prof. Roger Grosse for CSC321 Intro to Neural Networks and Machine Learning at University of Toronto.

92 Can other generative models do this? Unsupervised image-to-image translation networks [Liu, Breuel, Kautz, 2017]

93 Domain Adaptation [Tzeng et al. 2014]

94 Sim2real Simulated data Real data,,? [Richter*, Vineet* et al. 2016] [Krähenbühl et al. 2018]

95 CycleGAN Training data, [Hoffman, Tzeng, Park, Zhu, Isola, Saenko, Darrell, Efros, 2018]

96 CycleGAN Training data, [Hoffman, Tzeng, Park, Zhu, Isola, Saenko, Darrell, Efros, 2018]

97 CycleGAN FCN Training data, [Hoffman, Tzeng, Park, Zhu, Isola, Saenko, Darrell, Efros, 2018]

98 Medical domain adaptation Input MR Generated CT Ground truth CT MRI reconstruction [Quan et al.] arxiv: Cardiac MR images from CT [Chartsias et al. 2017]

99 What can you do with generative models? 1. Data prediction 2. Domain mapping 3. Representation learning 4. Model-based intelligence

100 What can you do with generative models? 1. Data prediction 2. Domain mapping 3. Representation learning 4. Model-based intelligence

101 Representation Learning X Coral Fish Image Compact mental representation

102 Representation Learning compressed image code (vector z) X Image

103 Representation Learning compressed image code (vector z) X ˆX Image Reconstructed image Autoencoder

104 Autoencoder X F ˆX = F(X) Image Reconstructed image

105 X F ˆX = F(X) Image Reconstructed image Fish Coral

106 MNIST manifold learned by a VAE [Kingma & Welling, 2014]

107 Progressive GAN [Karras et al., 2018]

108 Progressive GAN [Karras et al., 2018]

109 Data compression X ˆX Data Data

110 Data prediction X 1 ˆX2 Some data Other data see also Predictive coding, Denoising autoencoders [Vincent et al., 2008]

111 Grayscale image: L channel Semantics? Higherlevel abstraction? Color information: ab channels L ab [Zhang, Isola, Efros, ECCV 2016]

112

113 Instructive failure

114 Instructive failure

115 Deep Net Electrophysiology [Zeiler & Fergus, ECCV 2014] [Zhou et al., ICLR 2015]

116 Stimuli that drive selected neurons (conv5 layer) faces dog faces flowers

117 Representation Learning with Contrastive Predictive Coding [van den Oord, Li, and Vinyals, 2018]

118 Representation Learning with Contrastive Predictive Coding [van den Oord, Li, and Vinyals, 2018] Sound representation Image representation Each color represents a different speaker

119 What can you do with generative models? 1. Data prediction 2. Domain mapping 3. Representation learning 4. Model-based intelligence

120 What can you do with generative models? 1. Data prediction 2. Domain mapping 3. Representation learning 4. Model-based intelligence

121 Model-based intelligence Predict consequences of my actions. Then use that predictive model to plan good actions (or learn good policies). Can be faster, safer, and more abstracted than directly trying actions in real world (or in simulator). Yann LeCun s cake

122 Deep Visual Foresight For Robotic Planning [Finn & Levine 2017] 1. Train model to predict future frame given action 2. Specify target future frame 3. Use prediction model to pick action that maximizes probability of reaching target frame c.f. [Weber et al., 2017], [ ]

123 World Models [Ha & Schmidhuber 2018] 1. Train an RNN to simulate a video game 2. Do policy optimization using the RNN simulate trajectories, rather than the actual game (policy reasons on top of latent space)

124 Incentivizing exploration in reinforcement learning with deep predictive models [Stadie et al., 2015] 1. Train model to predict next state given action and previous state: p(s t+1 s t,a t ) 2. Reward agent for reaching states that surprise it: r t = log p(s t+1 s t,a t ) [Pathak et al., 2017]

125 What can you do with generative models? 1. Data prediction 2. Domain mapping 3. Representation learning 4. Model-based intelligence

126 Thank you!