Visual Material Traits Recognizing Per-Pixel Material Context

Transcription

1 Visual Material Traits Recognizing Per-Pixel Material Context Gabriel Schwartz Ko Nishino 1 / 22

2 2 / 22

3 Materials as Visual Context What tells us this road is unsafe? 3 / 22

4 Material Category Recognition Methods Give single predictions for the entire image Adelson [1] Liu et al. [4] Hu et al. [3] Sharan et al. [6] Images from [2]. 4 / 22

5 Material Category Recognition Methods Give single predictions for the entire image Require object information object mask bounding box Adelson [1] Liu et al. [4] Hu et al. [3] Sharan et al. [6] Images from [2]. 4 / 22

6 Material Category Recognition Methods Give single predictions for the entire image Require object information object mask bounding box Predict categories that are really object properties Adelson [1] Liu et al. [4] Hu et al. [3] Sharan et al. [6] Images from [2]. 4 / 22

7 Material Category Recognition Methods Give single predictions for the entire image Require object information object mask bounding box Predict categories that are really object properties Object information not always available Adelson [1] Liu et al. [4] Hu et al. [3] Sharan et al. [6] Images from [2]. 4 / 22

8 Intra-Class Appearance Variability Images from [7]. 5 / 22

9 Visual Material Traits: Characteristic Material Properties Image from [5]. 6 / 22

10 Visual Material Traits: Characteristic Material Properties Fuzzy Organic Smooth Material traits are locally-recognizable material properties. Image from [5]. 6 / 22

11 Visual Material Trait Appearances What material properties can we see locally? Images from [7]. 7 / 22

12 Visual Material Trait Appearances What material properties can we see locally? Certain properties are easy to describe Images from [7]. 7 / 22

13 Visual Material Trait Appearances What material properties can we see locally? Certain properties are easy to describe Shiny Images from [7]. 7 / 22

14 Visual Material Trait Appearances What material properties can we see locally? Certain properties are easy to describe Shiny Smooth 7 / 22

15 Visual Material Trait Appearances What material properties can we see locally? Certain properties are easy to describe Shiny Smooth Some are more challenging 7 / 22

16 Visual Material Trait Appearances What material properties can we see locally? Certain properties are easy to describe Shiny Smooth Some are more challenging Fuzzy? Soft? 7 / 22

17 Visual Material Trait Appearances What material properties can we see locally? Certain properties are easy to describe Shiny Smooth Some are more challenging Fuzzy? Soft? How do we represent these traits? 7 / 22

18 Learning to Represent Material Traits Learn features that model the appearance of material traits 8 / 22

19 Learning to Represent Material Traits Learn features that model the appearance of material traits Features should be: 8 / 22

20 Learning to Represent Material Traits Learn features that model the appearance of material traits Features should be: Fast to compute 8 / 22

21 Learning to Represent Material Traits Learn features that model the appearance of material traits Features should be: Fast to compute Able to be extracted anywhere 8 / 22

22 Learning to Represent Material Traits Learn features that model the appearance of material traits Features should be: Fast to compute Able to be extracted anywhere Discriminative 8 / 22

23 Learning to Represent Material Traits Learn features that model the appearance of material traits Features should be: Fast to compute Able to be extracted anywhere Discriminative Convolution filters may satisfy all of these properties 8 / 22

24 Learning to Represent Material Traits Learn features that model the appearance of material traits Features should be: Fast to compute Able to be extracted anywhere Discriminative Convolution filters may satisfy all of these properties How do we learn them? 8 / 22

25 Learning Filters for Trait Representation Convolutional Autoencoder (CAE) model for feature learning Find optimal filters (W) s.t. they: min W,W 9 / 22

26 Learning Filters for Trait Representation Convolutional Autoencoder (CAE) model for feature learning Find optimal filters (W) s.t. they: Model trait patches E i = h (W I i + b e ) R i = W E i + b r 1 h (x) = 1 min W,W N 0 1 N I i R i 2 F i=1 9 / 22

27 Learning Filters for Trait Representation Convolutional Autoencoder (CAE) model for feature learning Find optimal filters (W) s.t. they: Model trait patches Form a sparse encoding E i = h (W I i + b e ) R i = W E i + b r 1 h (x) = 1 min W,W N 0 1 N I i R i 2 F +α p 1 N i=1 N 2 E i i=1 F 9 / 22

28 Learning Filters for Trait Representation Convolutional Autoencoder (CAE) model for feature learning Find optimal filters (W) s.t. they: Model trait patches Form a sparse encoding Have constrained magnitude E i = h (W I i + b e ) R i = W E i + b r 1 h (x) = 1 min W,W N 0 1 N I i R i 2 F +α p 1 N i=1 N 2 E i +β i=1 F ( ) W 2 F + W 2 F 9 / 22

29 Learning Filters for Trait Representation Convolutional Autoencoder (CAE) model for feature learning Find optimal filters (W) s.t. they: Model trait patches Form a sparse encoding Have constrained magnitude E i = h (W I i + b e ) R i = W E i + b r 1 h (x) = 1 min W,W N 0 1 N I i R i 2 F +α p 1 N i=1 N 2 E i +β i=1 F ( ) W 2 F + W 2 F 9 / 22

30 Learned Filters Filter Responses Convolution Filters Input Images 10 / 22

31 Learned Filters + Supplemental Nonlinear Features Filter Responses Convolution Filters Describe appearances CAE cannot: Input Images 10 / 22

32 Learned Filters + Supplemental Nonlinear Features Filter Responses Convolution Filters Describe appearances CAE cannot: Input Images Color Histograms 10 / 22

33 Learned Filters + Supplemental Nonlinear Features Filter Responses Convolution Filters Input Images Describe appearances CAE cannot: Color Histograms HOG 10 / 22

34 LBP 10 / 22 Learned Filters + Supplemental Nonlinear Features Filter Responses Convolution Filters Input Images Describe appearances CAE cannot: Color Histograms HOG

35 Material Trait Recognition Process Training data: Flickr Materials Database (FMD) [7] images with trait annotations Learn Filters 11 / 22

36 Material Trait Recognition Process Training data: Flickr Materials Database (FMD) [7] images with trait annotations Learn Filters Select Features Trait CAE Oriented HOG LBP Color Histograms Shiny Fuzzy Transparent (13 Material Traits ) Total Uses / 22

37 Material Trait Recognition Process Training data: Flickr Materials Database (FMD) [7] images with trait annotations Learn Filters Select Features Train Per-Trait Classifiers Trait CAE Oriented HOG LBP Color Histograms Shiny Fuzzy Transparent (13 Material Traits ) Total Uses / 22

38 Material Trait Recognition Process Training data: Flickr Materials Database (FMD) [7] images with trait annotations Learn Filters Select Features Train Per-Trait Classifiers Extract Features Trait CAE Oriented HOG LBP Color Histograms Shiny Fuzzy Transparent (13 Material Traits ) Total Uses / 22

39 Material Trait Recognition Process Training data: Flickr Materials Database (FMD) [7] images with trait annotations Learn Filters Select Features Train Per-Trait Classifiers Extract Features Recognize Traits Trait CAE Oriented HOG LBP Color Histograms Shiny Fuzzy Transparent (13 Material Traits ) Total Uses / 22

40 Material Trait Recognition Accuracy Accuracy Fuzzy 4 Shiny Smooth Soft Striped Metallic Organic Translucent Transparent Rough Liquid Woven Manmade 12 / 22

41 Patch Recognition Results Shiny Fuzzy Metallic Soft Smooth Liquid Rough Woven False Positives True Positives Shiny Fuzzy Metallic Soft Smooth Liquid Rough Woven 13 / 22

42 Per-Pixel Material Trait Maps Image from [5]. 14 / 22

43 Per-Pixel Material Trait Maps Fuzzy Organic Smooth Image from [5]. 14 / 22

44 Per-Pixel Material Trait Maps Image from [7]. 15 / 22

45 Per-Pixel Material Trait Maps Shiny Metallic Smooth Image from [7]. 15 / 22

46 What can we do with these material traits? 16 / 22

47 Material Recognition via Trait Distributions Smooth Rough Organic Soft Foliage Stone Plastic p ( trait i category j ) 17 / 22

48 Material Recognition Accuracy: Flickr Fabric Foliage Glass Leather Metal Paper Plastic Stone Water Wood Fabric Foliage Glass Leather Metal Paper Plastic Stone Water Wood Average: 49.2% [Liu et al.] (w/o obj): 42.6% (w/obj): 57.1% Images from [7]. 18 / 22

49 Material Recognition Accuracy: ImageNet Foliage Glass 0.8 Paper Plastic Stone Water Wood Foliage Glass Paper Plastic Stone Water Average: 60.5% Wood Images from [2]. 19 / 22

50 Segmentation with Material Traits Baseline NCuts Images from [5]. With Traits 20 / 22

51 Summary Material traits: may be recognized locally and accurately 21 / 22

52 Summary Material traits: may be recognized locally and accurately have distributions that encode material categories 21 / 22

53 Summary Material traits: may be recognized locally and accurately have distributions that encode material categories segment images into intuitively separate regions 21 / 22

54 Summary Material traits: may be recognized locally and accurately have distributions that encode material categories segment images into intuitively separate regions Future work: Discover new traits Improve applications 21 / 22

55 References E. H. Adelson. On Seeing Stuff: The Perception of Materials by Humans and Machines. In SPIE, pages 1 12, J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, D. Hu, L. Bo, and X. Ren. Toward Robust Material Recognition for Everyday Objects. In BMVC, pages , C. Liu, L. Sharan, E. H. Adelson, and R. Rosenholtz. Exploring Features in a Bayesian Framework for Material Recognition. In CVPR, pages , D. Martin, C. Fowlkes, D. Tal, and J. Malik. A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics. In ICCV, pages L. Sharan, C. Liu, R. Rosenholtz, and E. H. Adelson. Recognizing Materials Using Perceptually Inspired Features. International Journal of Computer Vision, L. Sharan, R. Rosenholtz, and E. Adelson. Material Perception: What Can You See in a Brief Glance? Journal of Vision, 9(8):784, / 22