Generative Networks. James Hays Computer Vision

Transcription

1 Generative Networks James Hays Computer Vision

2 Interesting Illusion: Ames Window

3 Recap Unsupervised Learning Style Transfer A Neural Algorithm of Artistic Style Leon A. Gatys, Alexander S. Ecker, Matthias Bethge. CVPR 2016.

4 Colorful Image Colorization Richard Zhang, Phillip Isola, Alexei (Alyosha) Efros richzhang.github.io/colorization

5 Ansel Adams, Yosemite Valley Bridge

6 Ansel Adams, Yosemite Valley Bridge Our Result

7 Grayscale image: L channel Color information: ab channels L ab

8 Semantics? Higher-level Grayscale image: L channel abstraction? Concatenate (L,ab) L ab Free supervisory signal

9 Inherent Ambiguity Grayscale

10 Inherent Ambiguity Our Output Ground Truth

11 Better Loss Function Colors in ab space (continuous) Regression with L2 loss inadequate Use multinomial classification Class rebalancing to encourage learning of rare colors

12 Better Loss Function Colors in ab space (discrete) Regression with L2 loss inadequate Use multinomial classification Class rebalancing to encourage learning of rare colors

13

14 Better Loss Function Regression with L2 loss inadequate log 10 probability Histogram over ab space Use multinomial classification Class rebalancing to encourage learning of rare colors

15 Better Loss Function Regression with L2 loss inadequate log 10 probability Histogram over ab space Use multinomial classification Class rebalancing to encourage learning of rare colors

16 Classification Parametric L2 Regression Nonparametric Hertzmann et al. In SIGGRAPH, Welsh et al. In TOG, Irony et al. In Eurographics, Liu et al. In TOG, Chia et al. In ACM Gupta et al. In ACM, Hand-engineered Features Deep Networks Deshpande et al. Cheng et al. In ICCV Dahl. Jan Iizuka et al. In SIGGRAPH, Charpiat et al. In ECCV Larsson et al. In ECCV [Concurrent]

17 Network Architecture conv1 conv2 conv3 conv4 conv5 fc6 fc7 lightness 64 ab color L

18 Network Architecture lightness conv1 conv2 conv3 conv4 64 conv5 à trous [1]/dilated [2] conv6 à trous/dilated conv7 conv8 ab color L [1] Chen et al. In arxiv, [2] Yu and Koltun. In ICLR, 2016

19 Ground Input Truth L2 Regression Class w/ Rebalancing

20 Failure Cases

21 Biases

22 Evaluation Visual Quality Representation Learning Quantitative Qualitative Per-pixel accuracy Perceptual realism Semantic interpretability Low-level stimuli Legacy grayscale photos Task generalization ImageNet classification Task & dataset generalization PASCAL classification, detection, segmentation Hidden unit activations

23 Evaluation Visual Quality Representation Learning Quantitative Qualitative Per-pixel accuracy Perceptual realism Semantic interpretability Low-level stimuli Legacy grayscale photos Task generalization ImageNet classification Task & dataset generalization PASCAL classification, detection, segmentation Hidden unit activations

24 Evaluation Visual Quality Representation Learning Quantitative Qualitative Per-pixel accuracy Perceptual realism Semantic interpretability Low-level stimuli Legacy grayscale photos Task generalization ImageNet classification Task & dataset generalization PASCAL classification, detection, segmentation Hidden unit activations

25 Perceptual Realism / Amazon Mechanical Turk Test

26

27 clap if fake clap if fake

28 Fake, 0% fooled

29

30 clap if fake clap if fake

31 Fake, 55% fooled

32

33 clap if fake clap if fake

34 Fake, 58% fooled

35 from Reddit /u/sherysantucci

36 Recolorized by Reddit ColorizeBot

37 Photo taken by Reddit /u/timteroo, Mural from street artist Eduardo Kobra

38 Recolorized by Reddit ColorizeBot

39 Perceptual Realism Test 50% AMT Labeled Real [%] Ground Truth Random Larsson et al Ours (L2) Ours (class) Ours (full) images tested per algorithm %

40 Input Ground Truth Output

41 Does the method work on legacy black and white photos?

42 Thylacine, Dr. David Fleay, extinct in 1936.

43 Thylacine, Dr. David Fleay, extinct in 1936.

44 Amateur Family Photo, 1956.

45 Amateur Family Photo, 1956.

46 Henri Cartier-Bresson, Sunday on the Banks of the River Seine, 1938.

47 Henri Cartier-Bresson, Sunday on the Banks of the River Seine, 1938.

48 Dorothea Lange, Migrant Mother, 1936.

49 Dorothea Lange, Migrant Mother, 1936.

50 Additional Information Demo Reddit ColorizeBot Type colorizebot under any image post Code Website full paper, user examples, visualizations

51 For the full paper, additional examples and our model: richzhang.github.io/colorization

52 Small Sample of other Generative Networks of interest DCGAN. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks Alec Radford, Luke Metz, Soumith Chintala Generative Adversarial Text-to-Image Synthesis [PDF][Supplement][BibTex][Code] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee. ICML Pix2Pix: Image-to-Image Translation with Conditional Adversarial Nets. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros