Maximally Stable Extremal Regions and Local Geometry for Visual Correspondences

Transcription

1 Maximally Stable Extremal Regions and Local Geometry for Visual Correspondences Michal Perďoch Supervisor: Jiří Matas Center for Machine Perception, Department of Cb Cybernetics Faculty of Electrical Engineering Czech Technical University in Prague Ph.D. thesis defense 2 nd December 2011

2 The Correspondence Problem Given two images of the same scene or object, find a pair(s) of locations that are projections of the same physical entity 2/23

3 Correspondence Problem Arises in 3D Reconstruction Panorama stitching Image retrieval and also in registration, tracking, optical flow, etc 3/23

4 The Local Features Approach Overview of the approach 1. Detection of distinguished regions 2. Geometric and photometric normalization and description 3. Matching or indexing 4. Geometry verification Detection (MSERs) Description Matching Verification Local geometry + Descriptor = Local feature 4/23

5 The Local Features Approach Overview of the approach 1. Detection of distinguished regions MaximallyStableExtremalRegions (MSERs) 2. Geometric and photometric normalization and description 3. Matching or indexing 4. Geometry verification 5/23

6 Maximally Stable Extremal Regions Baseline method [Matas et al. BMVC 02] Given an ordering of pixel intensities, a union find based algorithm enumerates all thresholdings of an image and tracks 4 connected components Extremal Regions (ER) Maximally Stable Extremal Regions (MSERs) selected among ER as local extrema of a stability function 6/23

7 Maximally Stable Extremal Regions Contribution 1 Proposed new stability functions that reduces MSER s sensitivity to blur Verified on standard datasets baseline * proposed First published in the thesis. 7/23

8 Maximally Stable Extremal Regions Contribution 2 Proposed an implementation that achieves twofold speedup over the baseline method by reducing the memory footprint of the algorithm by reducing the computational costs of the output phase >10Mpixel satellite images First published in the thesis. 8/23

9 The Local Features Approach Overview of the approach 1. Detection of distinguished regions MaximallyStableExtremalRegions (MSERs) 2. Geometric and photometric normalization and description 3. Matching or indexing 4. Geometry verification 9/23

10 The Local Features Approach Overview of the approach 1. Detection of distinguished regions MaximallyStableExtremalRegions (MSERs) Stable Affine Frames 2. Geometric and photometric normalization and description 3. Matching or indexing 4. Geometry verification 10/23

11 MSER + Local Affine Frames Problem 1 Partially occluded MSERs cannot be matched Solution Local Affine Frames (LAFs) [Obdržálek and Matas BMVC 02] One of the constructions concavities of an extremal region MSER detection Concavities on MSERs LAFs from concavities Problem 2 MSER + LAF method the stability of whole region is required Solution lower the sensitivity and get over represented image, or Published in: M. Perďoch, J. Matas, J. and Š. Obdržálek: Stable Affine Frames on Isophotes. In ICCV 07. WoS: 1 citation, Google scholar: 5 citations 11/23

12 Stable Affine Frames Contribution 3 A new concept of Stable Affine Frames that relaxes the necessity of a stable region Frames constructed on all extremal regions and stable ones are selected Detect all ER Concavities on ERs Stable LAFs selection Properties Provides better repeatability and coverage of the image than MSER+LAF More time consuming Published in: M. Perďoch, J. Matas, J. and Š. Obdržálek: Stable Affine Frames on Isophotes. In ICCV 07. WoS: 1 citation, Google scholar: 5 citations 12/23

13 The Local Features Approach Overview of the approach 1. Detection of distinguished regions MaximallyStableExtremalRegions (MSERs) Stable Affine Frames 2. Geometric and photometric normalization and description 3. Matching or indexing 4. Geometry verification 13/23

14 The Local Features Approach Overview of the approach 1. Detection of distinguished regions MaximallyStableExtremalRegions (MSERs) Stable Affine Frames Efficient Local Geometry Representation 2. Geometric and photometric normalization and description 3. Matching or indexing 4. Geometry verification 14/23

15 Efficient Representation of Local Geometry Overview of an Image Retrieval System with Spatial Verification Region detection Bag of words, Video Google [Šivic 03] Local appearance Visual vocabulary Indexed images word1,,word word2,, word 1 word, word 1, word 2,2, wordword, word8,8,... word word... word 88, word word word,word , word ,word ,word d L l geometry Local g millions of images Representation Representation 5 or 6 numbers per feature = 50GB for 5M images!!! 1 number per feature = 10GB for 5M images >> Published in: M.Perďoch, O. Chum, and J. Matas: Efficient Representation of Local Geometry for Large Scale Object Retrieval. CVPR WoS: 12 citations, Google scholar: 34 citations 15/23

16 Efficient Representation of Local Geometry Contribution 4 A method for learning a geometric vocabulary that captures the variety of ellipse shapes in a small set ( ) of ellipse prototypes Local geometry Compressed geometry millions of images Geom. vocabulary x 1,y 1,E x1 x 1,y 1,E 1 x 1 x,y 2,y 1,E 1 1 x,y 2,E 2,y 1,E 2,E x 2 xx,y 3,y 2,E 2,y 3,E 3,y 2,E 5 3,E x 3,y ,E 3,y 3,E xx N...,y N,E N,y N,E x N,y N,E N,y N,E N Representation ~1 number per feature = 10GB for 5M images!!! Training set of ellipses Published in: M.Perďoch, O. Chum, and J. Matas: Efficient Representation of Local Geometry for Large Scale Object Retrieval. CVPR WoS: 12 citations, Google scholar: 34 citations 16/23

17 The Local Features Approach Overview of the approach 1. Detection of distinguished regions MaximallyStableExtremalRegions (MSERs) Stable Affine Frames Efficient Local Geometry Representation 2. Geometric and photometric normalization and description 3. Matching or indexing 4. Geometry verification 17/23

18 The Local Features Approach Overview of the approach 1. Detection of distinguished regions MaximallyStableExtremalRegions (MSERs) Stable Affine Frames Efficient Local Geometry Representation 2. Geometric and photometric normalization and description 3. Matching or indexing Higher orderstructures symmetries 4. Geometry verification 18/23

19 Applications Symmetry Detection Contribution 5 Hypothesize an axis of symmetry from a single LAF correspondence and collect evidence with Hough transform for fast detection of bilateral symmetry y Published in: H. Cornelius, M. Perďoch, J. Matas, and G. Loy: Efficient Symmetry Detection Using Local Affine Frames. SCIA WoS: 4 citations, Google Scholar: 7 citations 19/23

20 The Local Features Approach Overview of the approach 1. Detection of distinguished regions MaximallyStableExtremalRegions (MSERs) Stable Affine Frames Efficient Local Geometry Representation 2. Geometric and photometric normalization and description 3. Matching or indexing Higher orderstructures symmetries 4. Geometry verification 20/23

21 The Local Features Approach Overview of the approach 1. Detection of distinguished regions MaximallyStableExtremalRegions (MSERs) Stable Affine Frames Efficient Local Geometry Representation 2. Geometric and photometric normalization and description 3. Matching or indexing Higher orderstructures symmetries 4. Geometry verification RANSAC 21/23

22 Applications RANSAC Contribution 6 Model of epipolar geometry from two LAF correspondences RANSAC a robust algorithm that estimates model in presence of outliers 7 point to point p correspondences => model of Epipolar pp Geometry (EG) EG from three LAF correspondences works [Chum and Matas DAGM 03] 6 points to estimate EG with fixed focal length [Stewenius et al. CVPR 05] E proportion of inliers m sample size Proportion of inliers - E m 50% 45% 40% 35% 30% 25% 20% 15% Published in: M. Perďoch, J. Matas, and O. Chum: Epipolar Geometry from Two Correspondences. ICPR WoS: 3 citations, Google Scholar: 4 citations 22/23

23 Summary Contributions Improvements to the detection, matching and verification stages of the correspondence problem. Applications The state of the art MSER implementation, widely used State of the art text in the wild detection and recognition based on the fast MS(ER)s implementation (Google Award 2011) Efficient geometry representation allowed a fast image retrieval from 5 million images on a single machine Image retrieval system acquired by Samsung (2011) Stats of M. Perďoch ISI Web of Knowledge (WoS) 37 citations, h index: 3 Google Scholar 163 citations, h index: 5. 23/23

24 Questions Have the algorithms or improvements been incorporated into a bigger software product? If so, how important have they proven to be? How did they influence the overall performance of that product? Have also the results from Chapter 3 been published in some way? 24/23