Shape Matching for 3D Retrieval and Recognition. Agenda. 3D collections. Ivan Sipiran and Benjamin Bustos

Transcription

1 Shape Matching for 3D Retrieval and Recognition Ivan Sipiran and Benjamin Bustos PRISMA Research Group Department of Computer Science University of Chile Agenda Introduction Applications Datasets Shape retrieval contests Techniques Future directions Final remarks 3D collections

2 3D devices 3D applications 3D as multimedia The same problems as other media Representation Storage Analysis Processing Content-based matching or?

3 The problem with matching? Non-rigid matching? Partial matching Agenda Introduction Applications Datasets Shape retrieval contests Techniques Future directions Final remarks Applications Craniofacial research Atmosukarto et al., Pattern Recognition, 2010?

4 Applications 3D protein retrieval and classification Paquet and Viktor, Proc. VLDB, 2008 Applications 3D retrieval for museums Goodall et al., Proc. CIVR, 2004 Applications Human ear recognition in 3D Chen and Bhanu, IEEE TPAMI, 2009

5 Applications CAD/CAM You and Tsai, Int. J. Adv. Manuf. Tech., 2010 Applications Archaeology Huang et al., ACM TOG, 2006 Applications 3D video sequences Huang et al., IJCV, 2010

6 Applications 3D face recognition Bronstein et al., IJCV, 2005 Agenda Introduction Applications Datasets Shape retrieval contests Techniques Future directions Final remarks Datasets Princeton Shape Benchmark 1814 models: 907 training, 907 testing Available in:

7 Datasets Purdue Engineering Shape Benchmark Mechanical parts Available in: Datasets TOSCA dataset 80 objects with non-rigid transformations Available in: Agenda Introduction Applications Datasets Shape retrieval contests Techniques Future directions Final remarks

8 Shape Retrieval Contests (SHREC) Competitions started in 2006 To date: 40+ tracks presented Each track has a dataset and evaluation tools Good initiative to evaluate algorithms and make comparisons with the state of the art. SHREC Examples CAD models (2008) Using the ESB benchmark Six participants Available in: SHREC Examples Generic shape retrieval (2009) 720 objects organized in 40 classes 22 algorithms evaluated Available in:

9 SHREC Examples Feature detection and description (2010) Three shapes, 9 transformations in 5 levels of strenght. Goal: measure the repeatability of local features Available in: SHREC Examples Face scans (2011) Training set: 60 models Test set: 650 scans Available in: SHREC Examples Non-rigid retrieval (2011) 600 objects with non-rigid transformations Available in:

10 SHREC Examples Sketch-based 3D models retrieval (2012) 400 3D models 250 hand-drawn sketches Available in: SHREC Examples Large-scale partial shape retrieval (2013) 360 models, 7200 partial queries Available in: SHREC Examples Retrieval of Objects Captured with Low-Cost Depth-Sensing Cameras (2013) 192 models captured with Kinect Available in:

11 Agenda Introduction Applications Datasets Shape retrieval contests Techniques Future directions Final remarks Techniques Generic shape retrieval Shape recognition Non-rigid shape retrieval Shape correspondences Techniques Generic shape retrieval Shape recognition Non-rigid shape retrieval Shape correspondences

12 The global approach Transform a 3D object into a numeric/symbolic representation Compare two objects through their representations. Depth-buffer descriptor Image-based descriptor Pose normalization Depth-buffer construction Fourier transformation Selection of coefficients Depth-buffer descriptor Pose normalization Continuos PCA

13 Depth-buffer descriptor Pose normalization Continuous PCA Depth-buffer descriptor Construction Project the object into the cube's faces Depth-buffer descriptor Fourier transformation

14 Depth-buffer descriptor Selection of coefficients As depth-buffers are real, coefficient posses the symmetry property. Select coefficients whose indices satisfy for some natural number k. PANORAMA descriptor Image-based descriptor Pose normalization (Continuous PCA) Projection Fourier and Wavelet transformations PANORAMA descriptor Projection

15 PANORAMA descriptor Transformations Fourier Haar and Coiflet wavelets Features computed over sub-images of the DWT Global + Local approach Trying to take advantage of the local information in shapes Global + Local approach We need discriminative and robust partitions. Local features-based approach

16 Step 1 : Detection of interest points Harris 3D algorithm Step 1 : Detection of interest points Meshes with bad triangulations Control of resolution to improve triangulations Adaptive clustering of keypoints in Euclidean space Near points: same cluster Far points: different cluster

17 Partitioning and description Extract the patch enclosed by a sphere containing a cluster. We use a kd-tree to efficiently search vertices in the enclosing sphere. An object is represented as Matching Given two objects O and Q, with their representations The distance is i j O Q

18 Matching: Optimization The optimum The distance Matching Solved using binary programming with Linear approach is not geometrically consistent Quadratic approach i i j O Q

19 Quadratic programming Now, we consider the distance between parts ds Quadratic approach solved with Binary Quadratic Programming with Results using SHREC'2009 Generic Shape Retrieval

20 Class-by-class Class-by-class High variability inside classes Difficult for representations

21 Experiments can be tested online in publisher's website Link: Techniques Generic shape retrieval Shape recognition Non-rigid shape retrieval Shape correspondences Spin images Robust local descriptor It is based on how points are distributed on a surface

22 Spin images A local basis is constructed from An oriented point p The normal n The tangent plane P through p and perpendicular to n Spin images Any point q can be represented in this basis The coordinate of q in the spin image is computed from (, ) Spin images Computing positions Accumulation with bilinear weights

23 Spin images Spin images Matching Given two spin images Similarity Techniques Generic shape retrieval Shape recognition Non-rigid shape retrieval Shape correspondences

24 Non-rigid shape retrieval Models with a non-rigid transformation Several approaches Elegant theory: spectral domain Shape Google Represent a 3D model as a quantized vector of spectral descriptors Given a surface S, the heat diffusion process is governed by the heat equation Shape Google The fundamental solution of heat equation is the heat kernel, represented as Donde i and vi are the eigenvalues and eigenvectos of the Laplace-Beltrami operator, respectively.

25 Shape Google A representation for a point can be obtained Using values for t, we can get a descriptor which is called Heat Kernel Signature Shape Google Heat kernel signatures Shape Google Heat kernel signatures are sensitive to scale A scale-invariant variant has been proposed It uses finite differences and Fourier coefficients for removing the scale dependency of HKS

26 Shape Google Compute a descriptor for each vertex in a mesh Given the entire collection of descriptors, perform a k-means clustering to find a dictionary Quantize the descriptors of a shape using the dictionary Shape Google SQFD for Retrieval Unlike bag of features, this approach is local for defining the signatures Signature Quadratic Form Distance Final representation only depends on the object information It is possible to measure the distance between objects with representations of different sizes

27 SQFD for retrieval Object is represented as a set of features Let us suppose the existence of a local partitioning The signature is defined as SQFD for retrieval Given two signatures SQFD is defined as SQFD for retrieval Three approaches for computing the signatures All descriptors of an object Descriptors of keypoints Geodesic clusters Adaptive clustering for computing the local partitioning We use the Heat Kernel Signatures as descriptors

28 SQFD for retrieval All vertices SQFD for retrieval Descriptor of keypoints SQFD for retrieval Geodesic clusters

29 SQFD for retrieval Examples Open problems Query specification Efficiency and large-scale retrieval Object representation Partial matching Automatic 3D object annotation Final remarks Comprehensive literature on 3D retrieval Well-established background for applications Cheap 3D devices will benefit real-world applications Active research community SHREC