Learning the Ecological Statistics of Perceptual Organization

Size: px

Start display at page:

Download "Learning the Ecological Statistics of Perceptual Organization"

Ilene Haynes
5 years ago
Views:

1 Learning the Ecological Statistics of Perceptual Organization Charless Fowlkes work with David Martin, Xiaofeng Ren and Jitendra Malik at University of California at Berkeley 1

2 How do ideas from perceptual organization relate to natural scenes? 2

3 Perceptual Organization Grouping Figure/Ground 3

4 How do ideas from perceptual organization relate to natural scenes? Can we define these cues for real images? Are these cues ecologically valid? [Brunswik & Kamiya 1953] How informative are different cues? What do they have to offer applications of segmentation and recognition? 4

5 Statistics of Natural Scenes P(image features) Edges/Filters/Coding: Ruderman 94/97, Dong/Atick 95, Olshausen/Field 96, Bell/Sejnowski 97, Hateren/Schaaf 98, Buccigrossi/Simoncelli 99, Alvarez/Gousseau/Morel 99, Huang/Mumford 99 P(organization image features) Proximity of Similars: Brunswik/Kamiya 53 Good Continuation: Geisler, et. al. 01, Ren/Malik 02 Similarity, Proximity: Fowlkes/Martin/Malik 01,03 Convexity, Size, Lower-Region: Fowlkes/Martin/Malik 02 5

6 Berkeley Segmentation DataSet [BSDS] D. Martin, C. Fowlkes, D. Tal, J. Malik. "A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics", ICCV,

7 Figure-Ground Labeling - start with 200 segmented images of natural scenes - boundaries labeled by at least 2 different human subjects - subjects agree on 88% of contours labeled 7

8 Overview Grouping Local Grouping Cues Global Integration Figure/Ground Local Figure/Ground Cues Global Integration 8

9 Overview Grouping Local Grouping Cues Global Integration Figure/Ground Local Figure/Ground Cues Global Integration 9

10 Similarity Cues a) distance b) region cues (patch similarity) c) boundary cues (intervening contour) What image measurements allow us to gauge the probability that pixels i and j belong to the same group? 10

11 Learning Pairwise Affinities Sij indicator variable as to whether pixels i and j were marked as belonging to the same group by human subjects. Wij our estimate of the likelihood that pixel i and j belong to the same group conditioned on the image measurements. Use the ground truth given by human segmentations to calibrate cues. Learn statistically optimal cue combination in a supervised learning framework Ecological Statistics: Measure the relative power of different cues for natural scenes C. Fowlkes, D. Martin, J. Malik. "Learning Affinity Functions for Image Segmentation: Combining Patch-based and Gradient-based Approaches", CVPR,

12 Original Image Texture Brightness Color L* a* b* Boundary Processing Region Processing Textons D E Proximity C B A χ 2 χ 2 W ij C A B 12

13 Original Image Boundary Processing D χ 2 E D. Martin, C. Fowlkes, J. Malik. "Learning to Detect Natural Image Boundaries Using Local Brightness, Color and Texture Cues", TPAMI 26 (5) p

14 Non-Boundaries Boundaries I B C T 14

15 Gradient Features 1976 CIE L*a*b* colorspace Brightness Gradient BG(x,y,r,θ) Difference of L* distributions Color Gradient CG(x,y,r,θ) Difference of a*b* distributions Texture Gradient TG(x,y,r,θ) Difference of distributions of V1-like filter responses (x,y) r θ ( gi hi ) χ ( g, h) = 2 i g i + h i 15

16 Two Decades of Local Boundary Detection 16

17 P b Images I Image Canny 2MM Us Human 17

18 How good are humans locally? Off-Boundary On-Boundary Algorithm: r = 9, Humans: r = {5,9,18} Fixation(2s) -> Patch(200ms) -> Mask(1s) 18

19 Man versus Machine: D. Martin, C. Fowlkes, L. Walker, J. Malik. "Local Boundary Detection in Natural Images: Matching Human and Machine Performance", ECVP,

20 Original Image Texture Brightness Color L* a* b* Boundary Processing Region Processing Textons D E Proximity C B A χ 2 χ 2 W ij C A B 20

21 Region Processing Textons C. Fowlkes, D. Martin, J. Malik. "Learning Affinity Functions for Image Segmentation: Combining Patch-based and Gradient-based Approaches", CVPR, 2003 C B χ 2 A C A B 21

22 Patch Features Use histogram based representation of local appearance. Brightness Similarity Color Similarity Texture Similarity (x i,y i ) r Adapt window to local boundary information to avoid straddling contours (x j,y j ) 22

23 Evaluating Affinity Estimates Image Region Cues Contour Cues Estimated Affinity (W) Further Processing Groundtruth Affinity (S) Evaluate Human Segmentations 23

24 Two Evaluation Measures Estimate W ij Groundtruth S ij 1. Precision-Recall of same-segment pairs Precision is P(S ij =1 W ij > t) Recall is P(W ij > t S ij = 1) 2. Mutual Information between W and S p(s,w) log [p(s)p(w) / p(s,w)] 24

25 Individual Features Patches Gradients 25

26 Cue Combination Models Classification Trees Top-down splits to maximize entropy, error bounded Density Estimation Adaptive bins using k-means Logistic Regression, 3 variants Linear and quadratic terms Confidence-rated generalization of AdaBoost (Schapire&Singer) Hierarchical Mixtures of Experts (Jordan&Jacobs) Up to 8 experts, initialized top-down, fit with EM Support Vector Machines (libsvm, Chang&Lin) Gaussian kernel, ν-parameterization Logistic with quadratic terms is sufficient (performs as well as any classifier we tried 26

27 Combining Cues 27

28 Affinity Model vs. Human Segmentation 28

29 Findings Both Edges and Patches provide useful independent information. Texture gradients can be quite powerful Color patches better than gradients Brightness gradients better than patches. Proximity is a result, not a cause of grouping 29

30 Overview Grouping Local Grouping Cues Global Integration Figure/Ground Local Figure/Ground Cues Global Integration 30

Wu/Leahy 1993, Sarkar/Boyer 1996, Shi/Malik 1997,

31 Segmentation from Pairwise Affinities Image Affinities (Wij) Eigenvectors Segmentation Zahn 1971, Urquhart 1982, Scott/Longuet-Higgins 1990, Wu/Leahy 1993, Sarkar/Boyer 1996, Shi/Malik 1997, Felzenszwalb/Huttenlocher 1998, Perona/Freeman 1998, Gdalyahu/Weinshall/Werman 1999, Jermyn/Ishikawa

32 32

33 33

34 Extract Pb Compute Eigenvectors Gradient of eigenvectors 34

35 Evaluating the power of globalization 35

36 Evaluating the power of globalization 36

37 Moving Beyond Pixels Pixels are too fine grained Not scale invariant No explicit control over connectedness Hard to incorporate mid/high-level shape information such as continutiy or familiarity 37

38 Moving Beyond Pixels Superpixels give up little information while reducing complexity Allows for more interesting features Mori, Ren, Efros & Malik CVPR 04 Ren, Fowlkes & Malik CVPR 05? 38

39 Overview Grouping Local Grouping Cues Global Integration Figure/Ground Local Figure/Ground Cues Global Integration 39

40 Local Cues for Figure/Ground Assume we have a perfect segmentation Can we predict which region a contour belongs to based on it s local shape? Size/Surroundedness Convexity Lower Region C. Fowlkes, D. Martin, J. Malik. "On Measuring the Ecological Validity of Local Figure-Ground Cues", ECVP, (September 2003). 40

41 Size and Surroundedness [Rubin 1921] G p F Size(p) = log(area F / Area G ) 41

42 Convexity [Metzger 1953, Kanizsa and Gerbino 1976] Conv G = percentage of straight lines that lie completely within region G G p F Convexity(p) = log(conv F / Conv G ) 42

43 Lower Region [Vecera, Vogel & Woodman 2002] θ p center of mass LowerRegion(p) = θ G 43

44 Figural regions tend to be convex 44

45 Figural regions tend to lie below ground regions 45

46 Size Lower Region Convexity 46

47 Power of cue depends on support of the analysis window. 47

48 Power of cue depends on support of the analysis window. 48

49 Upper Bounding Local Performance Present human subjects with local shapes, seen through an aperture. 49

50 Preliminary Local Human Performance 50

51 Extension to Real Images Build up library of prototypical contour configurations by clustering local shape descriptors Geometric Blur [Berg & Malik 01] Train a classifier which uses similarities to these prototype shapes to predict figure/ground label 51

52 Shapemes Classifier using 64 shapeme features: 61% 52

53 Overview Grouping Local Grouping Cues Global Integration Figure/Ground Local Figure/Ground Cues Global Integration 53

54 Globalization of Figure/Ground Measurements Averaging local shapeme cue over human-marked boundaries: 71% Prior over junction types and label continuity: 79% 54

55 Future Work F/G globalization without using human segmentation Propogate labels on superpixel/cdt graph Role of figure/ground in recognition or segmentation? Show computational benefit of focusing recognition efforts on figural regions first. Simultaneous segmentation and figure/ground assignment on superpixel/cdt graph. 55

56 Future Work Segmentation Segmentation Figure/Ground Recognition Figure/Ground Recognition 56

57 Conclusion Rational design of bottom-up grouping cues. Ecological statistics of grouping and figure/ground cues. Characterization of spectral clustering and other globalization algorithms. 57

58 THE END 58

Part III: Affinity Functions for Image Segmentation

Part III: Affinity Functions for Image Segmentation Charless Fowlkes joint work with David Martin and Jitendra Malik at University of California at Berkeley 1 Q: What measurements should we use for constructing