Improving Shape retrieval by Spectral Matching and Meta Similarity

Transcription

1 1 / 21 Improving Shape retrieval by Spectral Matching and Meta Similarity Amir Egozi (BGU), Yosi Keller (BIU) and Hugo Guterman (BGU) Department of Electrical and Computer Engineering, Ben-Gurion University of the Negev

2 Talk Outline 2 / 21 Problem Statement Related works The proposed scheme Local shape descriptors Matching algorithm Similarity measure Meta-similarity Experimental results Conclusions

3 Talk Outline 2 / 21 Problem Statement Related works The proposed scheme Local shape descriptors Matching algorithm Similarity measure Meta-similarity Experimental results Conclusions

4 Problem statement 3 / 21 S = {s i } n i=1, and Q = {q i} m i=1, s i, q i R 2, are two shapes Ψ(S, Q) [, 1] quantifies the shape similarity. Invariant to similarity transformation (translation, rotation, and isotropic scaling) Robust to articulation Resilient to boundary noise and non-linear deformation

5 Talk Outline 3 / 21 Problem Statement Related works The proposed scheme Local shape descriptors Matching algorithm Similarity measure Meta-similarity Experimental results Conclusions

6 Related work: 4 / 21 1 Feature extraction Curvature [Fischler and Wolf TPAMI 94] Shape-contexts (SC) [Belongie et al. TPAMI 2] Inner-distance SC (ID-SC) [Ling and Jacobs TPAMI 7] 2 Correspondences Hungarian algorithm [Munkres 1957] Dynamic Programming [Ling and Jacobs TPAMI 7] 3 Agglomerate local similarities into a global similarity measure

7 Talk Outline 4 / 21 Problem Statement Related works The proposed scheme Local shape descriptors Matching algorithm Similarity measure Meta-similarity Experimental results Conclusions

8 Talk Outline 4 / 21 Problem Statement Related works The proposed scheme Local shape descriptors Matching algorithm Similarity measure Meta-similarity Experimental results Conclusions

9 Local Shape Descriptors Shape contexts [Belongie et al. TPAMI 3 ] 5 / 21 For s i S, a 2D histogram of the relative distances and orientations to the other (n 1) points: h i (k) = # { s i = s j (s i s j ) bin(k) }

10 Local Shape Descriptors Inner-distance shape contexts [Ling and Jacobs TPAMI 7] 6 / 21 Properties: Robust to articulation Capturing part structures ID-SC - An extension to shape contexts Euclidean distance is replaced by the inner-distance

11 Local Shape Descriptors 7 / 21 Utilize the ID-SC to obtain a set of candidate assignments E = {(s i, q i )}, s.t. d(φ(s i ), Φ(q i )) < T Retain the k NN For each point s i S.

12 Local Shape Descriptors 7 / 21 Utilize the ID-SC to obtain a set of candidate assignments E = {(s i, q i )}, s.t. d(φ(s i ), Φ(q i )) < T Retain the k NN For each point s i S.

13 Local Shape Descriptors 7 / 21 Utilize the ID-SC to obtain a set of candidate assignments E = {(s i, q i )}, s.t. d(φ(s i ), Φ(q i )) < T Retain the k NN For each point s i S. Example Let S = Q = 1, E = S Q = 1,, for k = 5, E = S k = 5.

14 Talk Outline 7 / 21 Problem Statement Related works The proposed scheme Local shape descriptors Matching algorithm Similarity measure Meta-similarity Experimental results Conclusions

15 Assignment problems Quadratic assignment problem (QAP) Definition A pair-wise affinity function, Ω : E E R +, measures the cost of a pair of individual assignments. Ω(e i, e j ) = exp { D S(s i, s j ) D Q (q i, q j ) 2 σ }, σ > 8 / 21

16 Quadratic Assignment Problem Affinity matrix structure 9 / 21 This affinity measure is: Purely (intrinsic) geometrical measure Invariant to: translation, rotation and reflection. Not invariant to: scaling (uniform and affine)

17 Assignment problems Quadratic assignment problem (QAP) a ij = Ω(e i, e j ) = exp { D S(s i, s j ) D Q (q i, q j ) 2 σ }, σ > Definition The affinity matrix consists of all pairwise affinities, A = (a ij ) R N N. 1 / 21

18 Quadratic Assignment Problem Affinity matrix structure (cont.) 11 / 21 Serialization constraint - keeps only affinities such that: ( ) ( ) s i, s j qi, q j < max

19 Quadratic assignment problem (QAP) 12 / 21 Goal - find M E which maximizes Ω(e i, e j ) = x T Ax, e i M,e j M and obeys the matching constraints. The assignment set M can be represented by a binary vector. X = x = ( 1 1 1) T

20 QAP solutions Spectral matching [Leordeanu and Hebert, ICCV 5] 13 / 21 The optimization problem x = arg max x T Ax s.t. Cx = b, x {, 1} N x This optimization problem is NP-complete! more

21 QAP solutions Spectral matching [Leordeanu and Hebert, ICCV 5] 13 / 21 The optimization problem x = arg max x T Ax s.t. Cx = b, x {, 1} N x This optimization problem is NP-complete! So, relax the binary constraint and the matching constraints [Leordeanu and Hebert, ICCV 5] more ẑ = arg max z T Az s.t. z T z = 1, z R N z Since A is symmetric, the maximum is achieved by the leading eigenvector of A. The matching constraints are enforced at the discretization stage.

22 Talk Outline 13 / 21 Problem Statement Related works The proposed scheme Local shape descriptors Matching algorithm Similarity measure Meta-similarity Experimental results Conclusions

23 Similarity measure - Conclusion 14 / 21 Given the estimated matching vector x {, 1} N, the similarity measure is given by: Properties: Ψ(S, Q) = x T A x Ψ(S, Q) measure the intrinsic geometrical distortion between the two shapes. Invariance to translation, rotation and isotropic scaling (by normalization). Robust to articulation and boundary noise.

24 Talk Outline 14 / 21 Problem Statement Related works The proposed scheme Local shape descriptors Matching algorithm Similarity measure Meta-similarity Experimental results Conclusions

25 Graph based recognition 15 / 21 Basic idea My friend s friend is my friend X

26 Shape Meta-Similarity 16 / 21 Related works: Graph Transduction [Xiang et al. TPAMI 1] x n+1 = Mx n x n+1 (i ) = 1; Contextual dissimilarity measure [Jegou et al. TPAMI 1] Improves bag-of-features based image retrieval

27 Shape Meta-Similarity 17 / 21 Our approach Instead of node similarity use structural similarity Given Ψ ij = Ψ(S i, S j ), for all i and j, Shape Meta-descriptor Λ i R+ N is defined as, Ψ ij Λ i = Sj N Ψ i ij if S j N i otherwise. The Meta-Similarity is Ψ M ij = Λ i Λ j 1

28 Talk Outline 17 / 21 Problem Statement Related works The proposed scheme Local shape descriptors Matching algorithm Similarity measure Meta-similarity Experimental results Conclusions

29 Experimental results MPEG7 Shape-CE-1 18 / 21 MPEG7 Shape-CE-1: 14 images from 7 categories, with 2 images per category.

30 Experimental results 1 MPEG7 Shape-CE-1 Retrival rates - one-to-one similarity Algorithm Performance Visual Parts [Latecki et al. CVPR ] 76.45% Shape Contexts [Belongie et al. TPAMI 2] 76.51% MDS+SC+DP [Ling and Jacobs TPAMI 7] 84.35% Planar Graph cuts [Schmidt et al. CVPR 9] 85% IDSC+DP [Ling and Jacobs TPAMI 7] 85.4% IDSC+DP+EMD-L 1 [Ling and Okada TPAMI 7] 86.56% GM+IDSC 87.47% GM+SC 88.11% Contour Flexibility [Xu et al. TPAMI 9] 89.31% 19 / 21

31 Experimental results 2 MPEG7 Shape-CE-1 2 / 21 Retrival rates - Graph-based similarity Algorithm Performance Graph Transduction [Yang et al. ECCV 8] 91% GM+IDSC+Meta Descriptor 91.46% GM+SC+Meta Descriptor 92.51% Locally Constrained Diffusion [Yang et al. CVPR 9] 93.32%

32 Talk Outline 2 / 21 Problem Statement Related works The proposed scheme Local shape descriptors Matching algorithm Similarity measure Meta-similarity Experimental results Conclusions

33 Shape similarity measure - Conclusions 21 / 21 We presented a new approach for measuring the similarity between shapes. It measure the intrinsic geometrical distortion between the two shapes. Better for articulated objects. We present an efficient meta-descriptor and meta-similarity.

34 Thank you! 22 / 21

35 Related work: Intermediate between local and global 23 / 21 Bending invariant signatures [Elad and Kimmel TPAMI 3] Idea: Embed geodesic distance into Euclidean space via Multi-dimensional scaling (MDS)

36 Related work: Intermediate between local and global 23 / 21 Bending invariant signatures [Elad and Kimmel TPAMI 3] Introduced for 2D manifolds, later to 2D shapes [Ling and Jacobs, TPAMI 7] and [Bronstein et al, IJCV 8]. Back

37 Background: Local Shape Descriptors Inner-distance shape contexts [Ling and Jacobs TPAMI 7] 24 / 21 An extension to shape contexts Euclidean distance is replaced directly with the inner-distance Shape Contexts (SC) Distance π/2 π 3π/2 2π 5

38 Background: Local Shape Descriptors Inner-distance shape contexts [Ling and Jacobs TPAMI 7] 24 / 21 An extension to shape contexts Euclidean distance is replaced directly with the inner-distance Shape Contexts (SC) Distance θ π/2 π 3π/2 2π 5

39 Background: Local Shape Descriptors Inner-distance shape contexts [Ling and Jacobs TPAMI 7] 24 / 21 An extension to shape contexts Euclidean distance is replaced directly with the inner-distance Shape Contexts (SC) Distance θ π/2 π 3π/2 2π 4

40 Background: Local Shape Descriptors Inner-distance shape contexts [Ling and Jacobs TPAMI 7] 24 / 21 An extension to shape contexts Euclidean distance is replaced directly with the inner-distance Shape Contexts (SC) Distance π/2 π 3π/2 2π

41 Background: Local Shape Descriptors Inner-distance shape contexts [Ling and Jacobs TPAMI 7] 24 / 21 An extension to shape contexts Euclidean distance is replaced directly with the inner-distance Shape Contexts (SC) Distance π/2 π 3π/2 2π Inner-distance SC Distance π/2 π 3π/2 2π

42 Background: Local Shape Descriptors Inner-distance shape contexts [Ling and Jacobs TPAMI 7] 24 / 21 An extension to shape contexts Euclidean distance is replaced directly with the inner-distance Shape Contexts (SC) Distance π/2 π 3π/2 2π Inner-distance SC Distance π/2 π 3π/2 2π

43 Background: Local Shape Descriptors Inner-distance shape contexts [Ling and Jacobs TPAMI 7] 24 / 21 An extension to shape contexts Euclidean distance is replaced directly with the inner-distance Shape Contexts (SC) Distance π/2 π 3π/2 2π Inner-distance SC Distance π/2 π 3π/2 2π back

44 QAP solutions Spectral matching [Leordeanu and Hebert, ICCV 5] 25 / 21 The optimization problem: ẑ = arg max z T Az s.t. z T z = 1, z R nm z Since A is symmetric, by Rayleigh-Ritz theorem [Horn and Johnson, 1985]: λ max = max z T Az z T z=1 And, ẑ is the eigenvector corresponds to λ max. By Perron-Frobenius theorem, since A is non-negative (element-wise) ẑ is strictly positive - ẑ i > for all i.

45 QAP solutions Discretization 26 / 21 Finally, extract a binary approximation from the continuous solution by solving: back x = arg min x ẑ T x s.t. Cx = b, x {, 1} N Greedy [Leordeanu and Hebert, ICCV 5] Hungarian [Munkres, 57]. optimal score. Used by [Belongie et al. TPAMI 3 ] for shape matching. Dynamic programming

46 Experimental results Correspondences /

47 Experimental results 2 Articulate dataset 4 images from 8 objects. Each column contains five images from the same object with different degrees of articulation. Top1 Top2 Top3 Top4 Top5 DP+IDSC 4/4 4/4 34/4 35/4 27/4 SM 4/4 4/4 36/4 31/4 26/4 PM 4/4 4/4 35/4 31/4 27/4 SM - Spectral Matching, PM - Probability Marginalization 28 / 21

48 Experimental Results 3 Kimia silhouette Algorithm 1st 2nd 3rd 4th 5th 6th 7th 8th 9th 1th DP+IDSC SC SM MP SM - Spectral Matching, PM - Probability Marginalization 29 / 21