EECS 442 Computer vision. Object Recognition

Transcription

1 EECS 442 Computer vision Object Recognition Intro Recognition of 3D objects Recognition of object categories: Bag of world models Part based models 3D object categorization

2 Challenges: intraclass variation

3 Usual Challenges: Variability due to: View point Illumination Occlusions

4 Representation Basic properties 2D Bag of Words (BoW) models; 2D Star/ISM models; Multiview models; Learning Generative & Discriminative BoW models Probabilistic Hough voting Generative multiview models Recognition Classification with BoW Detection via Hough voting

5 Representation Basic properties 2D Bag of Words (BoW) models; 2D Star/ISM models; Multiview models; Learning Generative & Discriminative BoW models Probabilistic Hough voting Generative multiview models Recognition Classification with BoW Detection via Hough voting

6 definition of BoW Independent features histogram representation codewords dictionary

7 learning Representation recognition feature detection & representation codewords dictionary image representation category models (and/or) classifiers category decision

8 Learning and Recognition 1. Discriminative method: NN SVM 2.Generative method: graphical models category models (and/or) classifiers

9 Discriminative classifiers category models Model space Class 1 Class N

10 Discriminative models Nearest neighbor Neural networks 10 6 examples Shakhnarovich, Viola, Darrell 2003 Berg, Berg, Malik LeCun, Bottou, Bengio, Haffner 1998 Rowley, Baluja, Kanade 1998 Support Vector Machines Latent SVM Structural SVM Boosting Guyon, Vapnik, Heisele, Serre, Poggio Felzenszwalb 00 Ramanan 03 Viola, Jones 2001, Torralba et al. 2004, Opelt et al. 2006, Courtesy of Vittorio Ferrari Slide credit: Kristen Grauman Slide adapted from Antonio Torralba

11 Support vector machines Find hyperplane that maximizes the margin between the positive and negative examples Support vectors: x i w b 1 x w b Distance between point i and hyperplane: w Margin = 2 / w Solution: w y x i i i i Support vectors Credit slide: S. Lazebnik Margin Classification function (decision boundary): w x b y x x i i i i b

12 Support vector machines Classification w x b y x x i i i i b Test point if x w b 0 class 1 if x w b 0 class 2 C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, 1998

13 Datasets that are linearly separable work out great: Nonlinear SVMs 0 x But what if the dataset is just too hard? 0 x We can map it to a higherdimensional space: x 2 0 x Slide credit: Andrew Moore

14 Nonlinear SVMs General idea: the original input space can always be mapped to some higherdimensional feature space where the training set is separable: Φ: x φ(x) lifting transformation Slide credit: Andrew Moore

15 Overfitting A simple dataset. Two models Linear Nonlinear

16 Overfitting Let s get more data. Simple model has better generalization.

17 Overfitting As complexity increases, the model overfits the data Training loss decreases Real loss increases We need to penalize model complexity = to regularize Loss Model complexity Real loss Training loss

18 Learning and Recognition 1. Discriminative method: NN SVM 2.Generative method: graphical models category models (and/or) classifiers

19 Generative models 1. Naïve Bayes classifier Csurka Bray, Dance & Fan, Hierarchical Bayesian text models (plsa and LDA) Background: Hoffman 2001, Blei, Ng & Jordan, 2004 Object categorization: Sivic et al. 2005, Sudderth et al Natural scene categorization: FeiFei et al. 2005

20 Object categorization: the statistical viewpoint Discriminative methods model posterior Generative methods model likelihood and prior Bayes rule: p( zebra image) p( no zebra image) p( image zebra) p( image no zebra) p( zebra) p( no zebra) posterior ratio likelihood ratio prior ratio

21 Some notations w: a collection of all N codewords in the image w = [w1,w2,,wn] c: category of the image

22 the Naïve Bayes model p(c w) ~ p(c)p(w c) p( c) p( w1,, w c) N Prior prob. of the object classes Image likelihood given the class c w N

23 the Naïve Bayes model p (c w) ~ p(c)p(w c) p( c) p( w1,, w c) Prior prob. of the object classes Image likelihood given the class p( c) N N n 1 p( w n c) Likelihood of n th visual word given the class Assume that each feature (codewords) is conditionally independent given the class p(w, 1,w N c) N i 1 p(w c) i

24 the Naïve Bayes model p (c w) ~ p(c)p(w c) p( c) p( w1,, w c) Prior prob. of the object classes Image likelihood given the class p( c) N N n 1 p( w n c) Likelihood of n th visual word given the class

25 Classification/Recognition c arg max c p( c w) p( c) N n 1 p( w n c) Object class decision p( wi c1 ) How do we learn P(w i c j )? From empirical frequencies of code words in images from a given class p( wi c2)

26 Csurka et al. 2004

27 E = 28% E = 15%

28 Summary: Generative models Naïve Bayes Unigram models in document analysis Assumes conditional independence of words given class Parameter estimation: frequency counting

29 Other generative BoW models Hierarchical Bayesian topic models (e.g. plsa and LDA) Object categorization: Sivic et al. 2005, Sudderth et al Natural scene categorization: FeiFei et al. 2005

30 Generative vs discriminative Discriminative methods Computationally efficient & fast Generative models Convenient for weakly or unsupervised, incremental training Prior information Flexibility in modeling parameters

31 Weakness of BoW the models No rigorous geometric information of the object components It s intuitive to most of us that objects are made of parts no such information Not extensively tested yet for View point invariance Scale invariance Segmentation and localization unclear

32 EECS 442 Computer vision Object Recognition Intro Recognition of 3D objects Recognition of object categories: Bag of world models Part based models 3D object categorization

33 Representation Basic properties 2D Bag of Words (BoW) models; 2D Star/ISM models; Multiview models; Learning Generative & Discriminative BoW models Probabilistic Hough voting Generative multiview models Recognition Classification with BoW Detection via Hough voting

34 Problem with bagofwords All have equal probability for bagofwords methods Location information is important

35 Part Based Representation Object as set of parts Model: Relative locations between parts Appearance of part Figure from [Fischler & Elschlager 73]

36 History of Parts and Structure approaches Fischler & Elschlager 1973 Yuille 91 Brunelli & Poggio 93 Lades, v.d. Malsburg et al. 93 Cootes, Lanitis, Taylor et al. 95 Amit & Geman 95, 99 Perona et al. 95, 96, 98, 00, 03, 04, 05 Ullman et al. 02 Felzenszwalb & Huttenlocher 00, 04 Crandall & Huttenlocher 05, 06 Leibe & Schiele 03, 04 Many papers since 2000

37 Deformations A B C D

38 Presence / Absence of Features occlusion

39 Background clutter

40 Sparse representation Computationally tractable (10 5 pixels parts) But throw away potentially useful image information

41 Discriminative Parts need to be distinctive to separate from other classes

42 Hierarchical representations Pixels Pixel groupings Parts Object Images from [Amit98,Bouchard05]

43 Different connectivity structures Fergus et al. 03 FeiFei et al. 03 Crandall et al. 05 Leibe 05; Felzenszwalb 09 O(N 6 ) O(N 2 ) O(N 3 ) Crandall et al. 05 Felzenszwalb & Huttenlocher 00 O(N 2 ) Csurka 04 Vasconcelos 00 Bouchard & Triggs 05 Carneiro & Lowe 06 from Sparse Flexible Models of Local Features Gustavo Carneiro and David Lowe, ECCV 2006

44 Stochastic Grammar of Images S.C. Zhu et al. and D. Mumford

45 Context and Hierarchy in a Probabilistic Image Model Jin & Geman (2006) e.g. animals, trees, rocks e.g. contours, intermediate objects e.g. linelets, curvelets, T junctions e.g. discontinuities, gradient animal head instantiated by tiger head animal head instantiated by bear head

46 Different connectivity structures Fergus et al. 03 FeiFei et al. 03 Crandall et al. 05 Leibe 05; Felzenszwalb 09 O(N 6 ) O(N 2 ) O(N 3 ) Crandall et al. 05 Felzenszwalb & Huttenlocher 00 O(N 2 ) Csurka 04 Vasconcelos 00 Bouchard & Triggs 05 Carneiro & Lowe 06 from Sparse Flexible Models of Local Features Gustavo Carneiro and David Lowe, ECCV 2006

47 Star models by Latent SVM Felzenszwalb, McAllester, Ramanan, 08 Source code:

48 Star models by Latent SVM Search strategy: Sliding Windows Simple but large computational complexity (x,y, S,, N of classes)

49 Implicit shape models by generalized Hough voting B. Leibe, A. Leonardis, and B. Schiele, Combined Object Categorization and Segmentation with an Implicit Shape Model, ECCV Workshop on Statistical Learning in Computer Vision 2004

50 Basic properties Representation How to represent an object category; which classification scheme? Learning How to learn the classifier, given training data Recognition How the classifier is to be used on novel data

51 Object representation: Constellation of parts w.r.t object centroid B. Leibe, A. Leonardis, and B. Schiele, Combined Object Categorization and Segmentation with an Implicit Shape Model, ECCV Workshop on Statistical Learning in Computer Vision 2004

52 Object representation: How to capture constellation of parts? Using Hough Voting B. Leibe, A. Leonardis, and B. Schiele, Combined Object Categorization and Segmentation with an Implicit Shape Model, ECCV Workshop on Statistical Learning in Computer Vision 2004

53 Basic properties Representation How to represent an object category; which classification scheme? Learning How to learn the classifier, given training data Recognition How the classifier is to be used on novel data

54 Hough Transform Formulation Parts in query image vote for a learnt model Significant aggregations of votes correspond to models Complexity : # parts * # votes Significantly lower than brute force search (e.g., sliding window detectors) Popular for detecting parameterized shapes Hough 59, Duda&Hart 72, Ballard 81, Slide Modified from S. Maji

55 Hough transform P.V.C. Hough, Machine Analysis of Bubble Chamber Pictures, Proc. Int. Conf. High Energy Accelerators and Instrumentation, 1959 Given a set of points, find the curve or line that explains the data points best y x

56 Hough transform P.V.C. Hough, Machine Analysis of Bubble Chamber Pictures, Proc. Int. Conf. High Energy Accelerators and Instrumentation, 1959 Given a set of points, find the curve or line that explains the data points best y (x 1, y 1 ) m y 1 = m x 1 + n x y = m x + n Hough space n

57 Hough transform P.V.C. Hough, Machine Analysis of Bubble Chamber Pictures, Proc. Int. Conf. High Energy Accelerators and Instrumentation, 1959 Issue : parameter space [m,n] is unbounded Use a polar representation for the parameter space y x xcos ysin Hough space

58 Hough transform experiments features votes

59 Hough transform experiments Noisy data features votes IDEA: introduce a grid a count intersection points in each cell Issue: Grid size needs to be adjusted

60 Hough transform experiments features votes Issue: spurious peaks due to uniform noise

61 Hough transform conclusions Good: All points are processed independently, so can cope with occlusion/outliers Some robustness to noise: noise points unlikely to contribute consistently to any single bin Bad: Spurious peaks due to uniform noise Tradeoff noisegrid size (hard to find sweet point)

62 Hough transform experiments Courtesy of TKK Automation Technology Laboratory

63 Credit slide: C. Grauman

64 Generalized Hough transform What if want to detect arbitrary shapes defined by boundary points and a reference point? [Dana H. Ballard, Generalizing the Hough Transform to Detect Arbitrary Shapes, 1980] Credit slide: C. Grauman

65 Generalized Hough transform To detect the model shape in a new image: For each edge point Index into table with its gradient orientation θ Use retrieved r vectors to vote for position of reference point Peak in this Hough space is reference point with most supporting edges Assuming translation is the only transformation here, i.e., orientation and scale are fixed. Credit slide: C. Grauman

66 Example Query Circle model rx ry P1 = 0 R = [rx,ry] = [1,0] C1 = P1 + R P2 = 45 R = [rx,ry] = [.7,.7] C2 = P2 + R Pk = 180 R = [rx,ry] = [1, 0] Ck = Pk + R

67 Implicit shape models Instead of indexing displacements by gradient orientation, index by visual codeword Visual codebook is used to index votes for object position [center] and scale CW rx ry B. Leibe, A. Leonardis, and B. Schiele, Combined Object Categorization and Segmentation with an Implicit Shape Model, ECCV Workshop on Statistical Learning in Computer Vision 2004

68 Implicit shape models Instead of indexing displacements by gradient orientation, index by visual codeword Visual codebook is used to index votes for object position [center] and scale CW rx ry ?? 1 3 B. Leibe, A. Leonardis, and B. Schiele, Combined Object Categorization and Segmentation with an Implicit Shape Model, ECCV Workshop on Statistical Learning in Computer Vision 2004

69 Implicit shape models Instead of indexing displacements by gradient orientation, index by visual codeword Visual codebook is used to index votes for object position [center] and scale CW rx ry N B. Leibe, A. Leonardis, and B. Schiele, Combined Object Categorization and Segmentation with an Implicit Shape Model, ECCV Workshop on Statistical Learning in Computer Vision 2004

70 Implicit shape models: Training 1. Build codebook of patches around extracted interest points using clustering Credit slide: S. Lazebnik

71 Implicit shape models: Training 1. Build codebook of patches around extracted interest points using clustering 2. Map the patch around each interest point to closest codebook entry 3. For each codebook entry, store all positions relative to object center [center is given] and scale [bounding box is given] Credit slide: S. Lazebnik

72 [Leibe, Leonardis, Schiele, SLCV 04; IJCV 08] Implicit Shape Model Recognition Interest Points Matched Codebook Entries Probabilistic Voting y s x 3D Voting Space (continuous) 22Nov11 74

73 [Leibe, Leonardis, Schiele, SLCV 04; IJCV 08] Implicit Shape Model Recognition Interest Points Matched Codebook Entries Probabilistic Voting y s x 3D Voting Space (continuous) Segmentation Backprojected Hypotheses Backprojection of Maxima 76

74 Probabilistic Hough Transform Image Feature Codebook match Object Position f = features l = location of the features. C = codebook entry O = object class x = object center Learnt using a max margin formulation Maji et al, CVPR 2009 Detection Score Position Posterior distribution of the centroid given the Codeword Ci observed at location lj. Codeword Match confidence (or weight) of the codeword Ci.

75 Computer Perceptual Vision and Sensory WS 08/09 Augmented Computing Example: Results on Cows Original image

76 Computer Perceptual Vision and Sensory WS 08/09 Augmented Computing Example: Results on Cows Interest Original points image

77 Computer Perceptual Vision and Sensory WS 08/09 Augmented Computing Example: Results on Cows Matched Interest Original points patches image

78 Computer Perceptual Vision and Sensory WS 08/09 Augmented Computing Example: Results on Cows Prob. Votes

79 Computer Perceptual Vision and Sensory WS 08/09 Augmented Computing Example: Results on Cows 1 st hypothesis

80 Computer Perceptual Vision and Sensory WS 08/09 Augmented Computing Example: Results on Cows 2 nd hypothesis

81 Computer Perceptual Vision and Sensory WS 08/09 Augmented Computing Example: Results on Cows 3 rd hypothesis

82 Computer Perceptual Vision and Sensory WS 08/09 Augmented Computing Example Results: Chairs Dining room chairs Office chairs

83 Computer Perceptual Vision and Sensory WS 08/09 Augmented Computing You Can Try It At Home Linux binaries available Including datasets & several pretrained detectors 88 Source: Bastian Leibe 22Nov11

84 89 Extension: Learning Feature Weights Subhransu Maji and Jitendra Malik, Object Detection Using a MaxMargin Hough Transform, CVPR 2009 Weights can be learned optimally using a maxmargin framework. Important Parts blue (low), dark red (high)

85 Learned Weights (ETHZ shape) Naïve Bayes MaxMargin Influenced by clutter (rare structures) Important Parts blue (low), dark red (high) 95

86 Learned Weights (UIUC cars) Naïve Bayes MaxMargin Important Parts blue (low), dark red (high) 96

87 Detection Results (ETHZ dataset) 1.0 False Positives Per Window 97

88 Conclusions Pros: Works well for many different object categories Both rigid and articulated objects Flexible geometric model Can recombine parts seen on different training examples Learning from relatively few (50100) training examples Optimized for detection, good localization properties Cons: Needs supervised training data Object bounding boxes for detection Segmentations for topdown segmentation No discriminative learning

89 Object detection and 3d shape recovery from a single image M. Sun, B. Xu, G. Bradski, S. Savarese, ECCV 2010 M. Sun, S. Kumar, G. Bradski, S. Savarese, 3DIMPVT 2011 Object categories Arbitrary Texture, Topology Uncontrolled Environment Uncalibrated Camera

90 Depth Encoded Hough Voting s d : Object location x : Part with center location l and scale s

91 Depth Encoded Hough Voting Position Posterior Codeword likelihood Scale related to Depth C = Codebook f = Part appearance(feature), l = 2D location s = Scale of the part, d = Depth information, j j j j j i j j j i j i j j j i ds d l s p d s l C O p f C p d s l O C x p ), ( ),,, ( ) ( ),,,, (, Detection Score

92 Object detection and 3d shape recovery from a single image 3d reconsctruction

93 Object detection and 3d shape recovery from a single image 3d pose estimation

94 Object detection and 3d shape recovery from a single image 3d modeling

95 Object detection and 3d shape recovery from a single image CAD Alignment Texture Completion 105

96

97 Summary Part based models enable more descriptive characterization of objects Semantically useful regions Depth occlusions ISM models Generative Enable segmentation Require large supervision 22Nov11 107