Part II: Modern Descriptors for High Matching Performance

Transcription

1 CVPR 2017 Tutorials: Local Feature Extraction and Learning for Computer Vision Part II: Modern Descriptors for High Matching Performance Bin Fan National Laboratory of Pattern Recognition Chinese Academy of Sciences

2 Outline Handcrafted Descriptors Non-linear Scale Space Intensity Order Pooling Using Multiple Scales Learned Descriptors Traditional Learning Convolutional Neural Networks (CNN)

3 Handcrafted Descriptors Non-linear Scale Space Intensity Order Pooling Using Multiple Scales Learned Descriptors Traditional Learning Convolutional Neural Networks (CNN)

4 KAZE Features Core idea: Use non-linear scale space to improve localization accuracy and distinctiveness Identical pipeline to SURF: 1. Scale space construction 2. Keypoint detection: based on the determinant of Hessian matrix 3. Dominant orientation estimation: sliding circular segment of 60 degree is used to find the longest x and y derivative vector. 4. Descriptor construction Pablo F. Alcantarilla, Adrien Bartoli, and Andrew J. Davison. KAZE Features. ECCV, 2012.

5 Scale space: Linear VS. Non-linear Gaussian scale space (linear). When scale evolves, image structures become blurry. Non-linear scale space. When scale evolves, image structures with strong edge responses are not affected.

6 Non-linear Scale Space Can be understood as something similar to SIFT s Gaussian scale space discretization Each scale (in pixel unit) is transferred into a time unit for non-linear scale space construction.

7 Non-linear Scale Space Non-linear scale space is constructed by iteratively applying diffusion filtering that depends on local gradient strength. Additive Operator Splitting (AOS)

8 Making the computation fast Additive Operator Splitting (AOS) is computational expensive Using Fast Explicit Diffusion (FED) as an alternative 2 n + n ti+ 1 ti = τ τ max max τ j = 3 2 2j + 1 2cos ( π ) 4n + 2 Pablo F. Alcantarilla, Jesus Nuevo, and Adrien Bartoli. Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces. BMVC, 2013.

9 Intensity Order Pooling Traditional methods rely on geometric locations to aggregate low-level features to construct the descriptor. Geometric locations need an (unreliable)estimated orientation to align. SIFT/SURF: Spatial Pooling based on Geometric locations Bin Fan, Fuchao Wu, and Zhanyi Hu. Rotationally Invariant Descriptors using Intensity Order Pooling. IEEE TPAMI, 2012.

10 Intensity Order Pooling Pooling based on intensity orders is rotationally invariant. Encode some spatial information too. Invariant to monotonic intensity changes Spatial Pooling based on Intensity Orders Bin Fan, Fuchao Wu, and Zhanyi Hu. Rotationally Invariant Descriptors using Intensity Order Pooling. IEEE TPAMI, 2012.

11 Combined with various low-level cues MROGH [Fan et al. CVPR 11] With gradient orientation distribution MRRID [Fan et al. PAMI 12] With center symmetric local binary patterns LIOP [Wang et al. ICCV 11] With LIOP features OIOP [Wang et al. PAMI 16] With OIOP features

12 P 3 X i 4 X i X i 2 X i 1 X i MROGH Rotation Invariant Gradient Dx X = I X I X 1 3 ( i) ( i) ( i ) 2 4 ( i) = ( i ) ( i ) Dy X I X I X Magnitude is linearly assigned to two neighbor orientation bins like SIFT Rotation Invariant CS-LBP Using 8 neighbor points Intensity test among opposite points 16D CS-LBP vector MRRID Bin Fan, Fuchao Wu, and Zhanyi Hu. Rotationally Invariant Descriptors using Intensity Order Pooling. IEEE TPAMI, 2012.

13 Using Multiple Regions D 1 Descriptor Computing D = [ ] D2 D3 D4

14 Local Intensity Order Pattern Y( Xi) = ( I( Xi), I( Xi ), I( Xi ), I( Xi )) = (86,217,102,151) γγ YY XX ii P 3 X i 4 X i 2 X i i 1 X i = (1,4,2,3 relative intensity relationship X ππ 1,2,3,4 1 1,2,4,3 2 1,3,2,4 3 1,3,4,2 4 1,4,2,3 5 1,4,3,2 6 2,1,3,4 7 2,1,4, φφ(ππ)... 4,3,1,2 23 4,3,2, possible ordering LIOP D Desc. Zhenhua Wang, Bin Fan, and Fuchao Wu. Local Intensity Order Pattern for Feature Description. ICCV, 2011.

15 Overall Intensity Order Pattern Problem:LIOP relies on neighbor points intensity order, which is not robust to noise. OIOP solution:according to the overall intensity distribution in a local patch, quantize neighbor points intensities, and use the quantization results to construct the descriptor. Key:How to quantize? Zhenhua Wang, Bin Fan, Gang Wang, and Fuchao Wu. Exploring Local and Overall Ordinal Information for Robust Feature Description. IEEE TPAMI, 2016.

16 Quantization methods Naïve average quantization Quantization distribution of each ordinal bin. It is largely related to the ordinal pooling region. Limited discriminative ability.

17 Quantization methods Learn the quantization percentiles For each ordinal pooling region, learning the best quantization. Enhanced discriminative ability.

18 OIOP: Descriptor Construction P 3 X i 4 X i X i 2 X i 1 X i 1. For each X i in the patch, sampling N neighbor points 2. Each neighbor point s intensity is quantized to M numbers 3. Represent X i by a N M-based number 4. M N possible numbers, one-one correspond to a M N -D vector 5. Intensity order based pooling for these M N -D vector, obtain the OIOP descriptor M = 4,N = 3, suggested by the authors

19 Using Multiple Scales Remember that MROGH/MRRID use multiple scales to compute descriptors separately and concatenate them together. D 1 Descriptor Computing D = [ ] D2 D3 D4

20 Using Multiple Scales DSP-SIFT: pooling SIFT descriptors across scales D 1 Descriptor/SIFT Computing D = [ ] D2 D3 D4 Jingming Dong and Stefano Soatto. Domain-Size Pooling in Local Descriptors: DSP-SIFT. CVPR, 2015.

21 Using Multiple Scales Scale-Less SIFT (SLS): pursuit a subspace representation of SIFT across scales D 1 Descriptor/SIFT Computing D = [ ] Subspace Representation D2 D3 D4 Tal Hassner, Shay Filosof, Viki Mayzels, and Lihi Zelnik-Manor, SIFTing through Scales. IEEE TPAMI, 2017.

22 Handcrafted Descriptors Non-linear Scale Space Intensity Order Pooling Using Multiple Scales Learned Descriptors Traditional Learning Convolutional Neural Networks (CNN)

23 ASR Descriptor Affine Subspace Representation Descriptor Core idea: learn a subspace representation of the patch under different viewpoints. Incorporate robustness to viewpoint changes in descriptor. Zhenhua Wang, Bin Fan, and Fuchao Wu. Affine Subspace Representation for Feature Description. ECCV, 2014.

24 ASR Descriptor Making the method feasible Affine warping is expensive Extremely computational expensive to do for all patches!!! Efficient extraction of the subspace through linear approximation and offline learning

25 ASR Descriptor Feasible Implementation Decompose Patch (cheap) Pre-computed offline

26 Brown s Descriptor Learning Discriminative Descriptor Learning by Optimizing Lowlevel Feature, Spatial Pooling, and Feature Embedding Normalized Patch Learning Smooth Low-level feature extraction Spatial pooling Post process Projection Descriptor M. Brown, G. Hua and S. Winder. Discriminative learning of local image descriptors. IEEE TPAMI, 2011.

27 Pre-defined low level features: gradient-based, filter bank based Pre-defined spatial poolings: SIFT-like, DAISY-like, GLOH-like Optimized combination of low level feature + spatial pooling Objective: maximize the area under ROC Projection: PCA, LDE Low level feature Spatial pooling dim PCA dim 1st Steerable filters bank DAISY-like nd Gradient orientation map DAISY-like PCA is better than LDE for projecting descriptor!

28 VGG s Descriptor Learning Pre-defined units for spatial pooling Each pooling unit contributes to a part of the descriptor Select a small number of pooling units to construct the descriptor Convex Learning objective K. Simonyan, A. Vedaldi, and A. Zisserman. Learning Local Feature Descriptors Using Convex Optimisation. IEEE TPAMI, 2014.

29 MatchNet Simultaneously learn the descriptor and the metric Siamese feature descriptor network Metric network on top Cross-entropy loss, transfer matching problem to classification problem X. Han, T. Leung, Y. Jia, R. Sukthankar, and A.C. Berg. Matchnet: Unifying feature and metric learning for patch-based matching. CVPR, 2015.

30 MatchNet 1 Two steps for usage 1. Using feature network to extract descriptors for all patches 2. Combining descriptor pairs, input to metric network 2 Computational efficient compared to directly input all patch pairs.

31 DeepCompare Core idea: Using CNN + metric network to learn similarity function for a pair of patches 2 - channel Siamese 2 - channel 2-stream S. Zagoruyko and N. Komodakis. Learning to compare image patches via convolutional neural networks. CVPR, 2015.

32 DeepCompare 2 channel(2ch) No direct notion of descriptor: Simply considers the two patches of an input pair as a 2-channel image Computational expensive: Pair-wise operation, can not re-use descriptor of each patch

33 DeepCompare Siamese MatchNet Siamese feature descriptor network Metric network on top

34 DeepCompare 2ch-2stream Consists of two separate streams, central and surround (CS), allowing the network to process at two different resolutions.

35 Global Loss Net Nonmatch Pairs Dist. Distribution Sim. Distribution Core idea: Reduce the proportion of false positive and false negative, i.e., blue shaded area. Objective: 1. Minimize the variance of two distributions. 2. Maximize the margin between the mean values of two distributions. V. Kumar B G, G. Carneiro, and I. Reid. Learning local image descriptors with deep siamese and triplet convolutional networks by minimising global loss functions. CVPR, 2016.

36 Global Loss Net (,, ), 1,, + x x x i= N ( i i Triplet samples: i i i, match pair, nonmatching pair 1Global Loss (L2 distance): δ 2 +, µ + J = δ + δ + λ µ µ max(0, + t) + + d = f( x ) f( x ), d = f( x ) f( x ) i i i i i i are variance and mean of d +, i δ 2, µ x, x + ) x, x ) ( i i are variance and mean of d i 2Triplet Local Loss: J di 2 = max(0,1 ) + d + m i 3Global Loss(Similarity): J x x x x + i i i i = δ+ + δ + λ max(0, µ µ + + m), di = g(, ), di = g(, ) Without metric net : TNet-Tloss(2) < TNet-TGLoss(2 + 1) With metric net:snet-gloss(3) < CS SNet-Gloss(3, 2ch-2stream)

37 DeepDesc Use Euclidean distance,direct substitution of SIFT loss:minimize pairwise hinge loss Edgar Simo-Serra, Eduard Trulls, Luis Ferraz, Iasonas Kokkinos, Pascal Fua, and Francesc Moreno-Noguer. Discriminative learning of deep convolutional feature point descriptors. ICCV, 2015.

38 DeepDesc Network Architecture Subtractive normalization Tanh 7x7 3x 3 Pooling 3x 3 Activation Conv (x64) Normalization 5x5 Pooling Conv (x32) 5x5 Activation L2 pool Normalization 7x7 2 x 2 Pooling 64x64 Activation 7x7 5x5 128-D vector 3x 3 Conv (x128) Only 3 convolutional layers, simple. Use hard negative mining to alleviate the problem of imbalanced positive and negative samples, key to good performance.

39 TFeat Core idea: Smallest negative distance within the triplet should be larger than the positive distance. CNN Structure * min λ ( +, ) Ranking-based: λ µ Ratio-based: * * ( +, ) = max(0, + + ) 2 * + λ ( +, ) = * + + V. Balntas, E. Riba, D. Ponsa and K. Mikolajczyk. Learning local feature descriptors with triplets and shallow convolutional neural networks. BMVC, 2016.

40 L2-Net L2-Net Architecture Fully convolutional architecture Batch normalization after convolution Local response normalization before output Input 32x32 patch, output 128D vector CS L2-Net: two separate L2-Net deal with the central and the whole patch Yurun Tian, Bin Fan, and Fuchao Wu. L2-Net: Deep Learning of Discriminative Patch Descriptor in Euclidean Space. CVPR, 2017.

41 L2-Net Core ideas 1. Progressive sampling: in training stage, a batch contains lots of negatives while only a few positives, in accord with matching scenario. 2. Using the concept of relative minimal distance of matching pairs, which is the characteristics of NN matching. 3. Incorporate the supervision information from intermediate layers into the learning objective, improving generalization. 4. Constrain the compactness of the learned descriptor, reducing overfitting.

42 L2-Net Progressive sampling: each batch, randomly select one patch from matching pairs to form nonmatching pairs, improving the number of non-matching pairs. Advantage: consisting of more negatives compared to pair-wise and triplet samples

43 L2-Net Learning Objectives 1. Relative minimal distance of matching pairs 2. Less correlation among different dimensions 3. Relative maximal similar intermediate feature maps of matching pairs

44 L2-Net Relative minimal distance d d d d i 1p d d d d i 2 p d d d d j1 j2 d d d d p1 p2 ji pi jp pp Euclidean distance Ideally, d ii should be the smallest one in the ith row, and also for the ith column. c s = exp(2 d ) / exp(2 d ) i ii m mi r s = exp(2 d ) / exp(2 d ) i ii m im E 1 1 ( log r log c = i i i i ) 2 s + s

45 L2-Net Less correlation among dims Y = [ b,, b,, b ] T s s s s 1 i q The correlation between the ith and jth dims: Minimize these correlations:

46 L2-Net Relative maximal similarity For intermediate feature maps: s s s Fs 1 2 p = [ f, f,, f ], s = 1, 2 Their similarity is defined by inner product: Ideally, similarity of matching pairs should be the largest, i.e., g ii should be the largest one in the ith row and ith column. r vi = exp( gii ) / exp( gim) m c vi = exp( gii ) / exp( gmi ) m E 1 3 ( log r log c = i i i i ) 2 v + v

47 L2-Net Relative maximal similarity A: Using intermediate feature maps of the first and last conv layers B: Without using intermediate feature maps C: Learned BN parameters D: Using all intermediate feature maps The 3 rd objective should be only used with intermediate feature maps of the first and last conv layers.

48 L2-Net Binary L2-Net output distribution of L2-Net mean values of Binary L2-Net The output of L2-Net approximates a Gaussian distribution, which is very suitable for binary descriptors. By applying sign() to L2-Net, we obtain the Binary L2-Net.

49 Learn to assign orientations Is considering only descriptors wise? Learned Orientations Dominant Gradient Orientations Kwang Moo Yi, Yannick Verdie, Pascal Fua, and Vincent Lepetit. Learning to Assign Orientations to Feature Points. In IEEE CVPR 2016.

50 Learn to assign orientations Learning Framework Feature Desc Descriptor Orientation Orient Estimator Minimize descriptor distance, e.g. SIFT Feature Desc Descriptor

51 Learn to assign orientations Improved results with good orientations Average performance map (higher the better)

52 LIFT Learned Invariant Feature Transform (LIFT) Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. LIFT: Learned invariant feature transform. In ECCV 2016.

53 LIFT Training requires various patches Keypoint VS Non-Keypoint Matching Keypoints Matching Keypoints VS Non-Matching Keypoints

54 LIFT Quadruplet Siamese Network P1, P2: corresponding keypoints. P3: non-corresponding keypoint. P4: non-keypoint.

55 LIFT Loss function detector orientation descriptor detector detector descriptor orientation

56 LIFT Each component is meant for each other SIFT descriptor when used with SIFT keypoints LIFT descriptor when used with SIFT keypoints LIFT descriptor when used with LIFT keypoints 0 Descriptor performance (NN map)

57 Take home messages For patch level datasets (e.g. Brown s dataset), learning based methods generally outperform hand-crafted ones. For image level dataset (e.g. VGG dataset), performance gap between learning based methods and hand-crafted methods is not significant. CNN based methods are dominant in the learning based methods. CNN based methods operating on the Euclidean space is highly required for wider application. Learning for feature detector has been largely ignored, but it is of significant important.

58 Thanks! Coffee Break