Computing and Processing Correspondences with Functional Maps

Transcription

1 Computing and Processing Correspondences with Functional Maps SIGGRAPH 2017 course Maks Ovsjanikov, Etienne Corman, Michael Bronstein, Emanuele Rodolà, Mirela Ben-Chen, Leonidas Guibas, Frederic Chazal, Alex Bronstein

2 General Overview Overall Objective: Create tools for computing and analyzing mappings between geometric objects. 2

3 General Overview Overall Objective: Create tools for computing and analyzing mappings between geometric objects. Rather than comparing pointson objects it is often easier to compare real-valued functionsdefined on them. Such maps can be represented as matrices. 3

4 Course Overview Course Website: or Course Notes: Linked from the website. Or use Attention: (significantly) more material than in the lectures Sample Code: See Sample Code link on the website. New: demo code for basic operations. 4

5 Course Schedule 2:00pm 2:45pm Introduction(Maks) Introduction to the functional maps framework. 2:50pm 3:35pm Computing Functional Maps (Etienne) Optimization methods for functional map estimation. 3:40pm - 4:25pm Functional Vector Fields (Miri) From functional maps to tangent vector fields and back. 4:30pm -5:15pm Maps in Shape Collections (Leo) Networks of maps, shape variability, learning. 5:15pm - Wrapup, Q&A (all) 5

6 What is a Shape? Continuous: a surface embedded in 3D. Discrete: a graph embedded in 3D (triangle mesh). Common assumptions: Connected. Manifold. Without Boundary. 6

7 What is a Shape? Continuous: a surface embedded in 3D. Discrete: a graph embedded in 3D (triangle mesh). 5k 200k triangles 7 Shapes from the FAUST, SCAPE, and TOSCA datasets

8 Overall Goal Given two shapes, find correspondences between them. Finding the best map between a pair of shapes. 8

9 Why Shape Matching? Given a correspondence, we can transfer: texture and parametrization segmentation and labels deformation Sumner et al. 04. Other applications: shape interpolation, reconstruction... 9

10 RigidShape Matching Given a pair of shapes, find the optimal Rigid Alignment between them. The unknowns are the rotation/translation parameters of the source onto the target shape. 10

11 Iterative Closest Point (ICP) Classical approach: iterate between finding correspondences and finding the transformation: M N example in 2D Given a pair of shapes, Mand, iterate: x i M 1. For each find nearestneighbor. 2. Find optimal transformation R,t minimizing: X arg min krx i +t y i k 2 2 R,t i N y i N 11

12 Iterative Closest Point Classical approach: iterate between finding correspondences and finding the transformation: M N Given a pair of shapes, Mand, iterate: x i M 1. For each find nearestneighbor. 2. Find optimal transformation R,t minimizing: X arg min krx i +t y i k 2 2 R,t i N y i N 12

13 Iterative Closest Point Classical approach: iterate between finding correspondences and finding the transformation: M N Given a pair of shapes, Mand, iterate: x i M 1. For each find nearestneighbor. 2. Find optimal transformation R,t minimizing: X arg min krx i +t y i k 2 2 R,t i N y i N 13

14 Iterative Closest Point Classical approach: iterate between finding correspondences and finding the transformation: M N Given a pair of shapes, Mand, iterate: x i M 1. For each find nearestneighbor. 2. Find optimal transformation R,t minimizing: X arg min krx i +t y i k 2 2 R,t i N y i N 14

15 Iterative Closest Point Classical approach: iterate between finding correspondences and finding the transformation: M N Given a pair of shapes, Mand, iterate: x i M 1. For each find nearestneighbor. 2. Find optimal transformation R,t minimizing: X arg min krx i +t y i k 2 2 R,t i N y i N 15

16 Iterative Closest Point Classical approach: iterate between finding correspondences and finding the transformation: M N 1. Finding nearest neighbors: can be done with spacepartitioning data structures (e.g., KD-tree). 2. Finding the optimal transformation R,t minimizing: X arg min krx i +t y i k 2 2 R SO(3),t R 3 i Can be done efficiently via SVD decomposition. Arun et al., Least- Squares Fitting of Two 3-D Point Sets

17 Non-Rigid Shape Matching Unlike rigid matching with rotation/translation, there is no compact representation to optimize for in non-rigid matching. 17

18 Non-Rigid Shape Matching Main Questions: What does it mean for a correspondence to be good? How to compute it efficiently in practice? 18

19 Isometric Shape Matching Deformation Model: Good maps must preserve geodesic distances. N d M (x,y) M d N (T(x),T(y)) Geodesic: length of shortest path lying entirely on the surface. 19

20 Isometric Shape Matching Approach: N d N (T(x),T(y)) d M (x,y) M Find the point mapping by minimizing the distance distortion: X T opt = argmin kd M (x,y) d N (T(x),T(y))k T x,y The unknowns are point correspondences. 20

21 Isometric Shape Matching Approach: N d N (T(x),T(y)) d M (x,y) M Find the point mapping by minimizing the distance distortion: X T opt = argmin kd M (x,y) d N (T(x),T(y))k T x,y Problem: The space of possible solutions is highly non-linear, non-convex.

22 Functional Map Representation We would like to define a representation of shape maps that is more amenable to direct optimization. 1. A compact representation for natural maps. 2. Inherently global and multi-scale. 3. Handles uncertainty and ambiguity gracefully. 4. Allows efficient manipulations (averaging, composition). 5. Leads to simple (linear) optimization problems. 22

23 Functional Approach to Mappings Given two shapes and a pointwisemap T : N M M T N The map T induces a functional correspondence: T F (f)=g, where g = f T Functional maps: a flexible representation of maps between shapes, O., Ben-Chen, Solomon, Butscher, Guibas, SIGGRAPH

24 Functional Approach to Mappings Given two shapes and a pointwisemap T : N M f : M R T F T F (f)=g : N R The map T induces a functional correspondence: T F (f)=g, where g = f T Functional maps: a flexible representation of maps between shapes, O., Ben-Chen, Solomon, Butscher, Guibas, SIGGRAPH

25 Functional Approach to Mappings Given two shapes and a pointwisemap T : N M f : M R T F T F (f)=g : N R The map T induces a functional correspondence: T F (f)=g, where g = f T Functional maps: a flexible representation of maps between shapes, O., Ben-Chen, Solomon, Butscher, Guibas, SIGGRAPH

26 Functional Approach to Mappings Given two shapes and a pointwisemap T : N M f : M R T F T F (f)=g : N R The induced functional correspondence is linear: T F (α 1 f 1 +α 2 f 2 )=α 1 T F (f 1 )+α 2 T F (f 2 ) 26

27 Functional Map Representation Given two shapes and a pointwisemap T : N M f : M R T F T F (f)=g : N R The induced functional correspondence is complete. 27

28 Observation Assume that both: f L 2 (M),g L 2 (N) f : M R T F g : N R M N Express both f and T F (f) in terms of basis functions: f = X i a i φ M i g = T F (f)= X j b j φ N j Since T F is linear, there is a linear transformation from {a i } to {b j }.

29 Functional Map Representation Choice of Basis: Eigenfunctions of the Laplace-Beltrami operator: φ i = λ i φ i (f)= divr(f) Generalization of Fourier bases to surfaces. Ordered by eigenvalues and provide a natural notion of scale. λ 0 =0 λ 1 =2.6 λ 2 =3.4 λ 3 =5.1 λ 4 =7.6 29

30 Functional Map Representation Choice of Basis: Eigenfunctions of the Laplace-Beltrami operator: φ i = λ i φ i Generalization of Fourier bases to surfaces. Ordered by eigenvalues and provide a natural notion of scale. Can be computed efficiently, with a sparse matrix eigensolver. 30

31 Functional Map Representation Since the functional mapping T F is linear: T F (α 1 f 1 +α 2 f 2 )=α 1 T F (f 1 )+α 2 T F (f 2 ) T F = T F can be represented as a matrix C, given a choice of basis for function spaces. Functional maps: a flexible representation of maps between shapes, O., Ben-Chen, Solomon, Butscher, Guibas, SIGGRAPH

32 Functional Map Definition Functional map: matrix C that translates coefficients from Φ M to Φ N. 32

33 Example 1 M T N Given two shapes with n M,n N points and a map: T : N M T : n N n M matrix encoding the map T, one 1 per column with zeros everywhere else. If functions are represented as vectors (in the hat basis), the functional map is given by matrix-vector product: g = T T f C = T T 33

34 Example 2 Given two shapes with n M,n N points and a map: T : N M T : n N n M matrix encoding the map T, one 1 per column with zeros everywhere else. If functions are represented in the reduced basis: Φ M : n M k M Φ N : n N k N k M matrix of the first eigenfunctions of as columns. k N M N matrix of the first eigenfunctions of as columns. The functional map matrix: C = Φ + N TT Φ M + : left pseudo-inverse. C = Φ T NT T Φ M if Φ T NΦ N = Id C = Φ T NA N T T Φ M if Φ T NA N Φ N = Id 34

35 Example Maps in a Reduced Basis Triangle meshes with pre-computed pointwise maps (a) source (b) ground-truth map (c) left-to-right map (d) head-to-tail map Good maps are close to being diagonal 35

36 Reconstructing from LB basis Map reconstruction error using a fixed size matrix. 27.9k vertices reconstruction error Number of basis (eigen)-functions 36

37 Functional Map algebra 1. Map composition becomes matrix multiplication. 2. Map inversion is matrix inversion (in fact, transpose). 3. Algebraic operations on functional maps are possible. E.g. interpolating between two maps with C = αc 1 +(1 α)c 2. 37

38 Shape Matching In practice we do not know C. Given two objects our goal is to find the correspondence.? How can the functional representation help to compute the map in practice? 38

39 Matching via Function Preservation Suppose we don t know C. However, we expect a pair of functions f : M R and g : N R to correspond. Then, C must be s.t. where Ca b f = P i a iφ M i,g= P i b iφ N i. Given enough {a,b} pairs, we can recover C through a linear least squares system. 39

40 Map Constraints Suppose we don t know C. However, we expect a pair of functions f : M R and g : N R to correspond. Then, C must be s.t. Ca b Function preservation constraint is general and includes: Attribute (e.g., color) preservation. Descriptor preservation (e.g. Gauss curvature). Landmark correspondences (e.g. distance to the point). Part correspondences (e.g. indicator function). 40

41 CommutativityConstraints Regularizations: Commutativity with other operators: S M C S N CS M = S N C Note that the energy: kcs M S N Ck 2 F is quadratic in C. 41

42 Regularization Lemma 1: The mapping is isometric, if and only if the functional map matrix commutes with the Laplacian: C M = N C Implies that exact isometries result in diagonal functional maps. Functional maps: a flexible representation of maps between shapes, O., Ben-Chen, Solomon, Butscher, Guibas, SIGGRAPH

43 Regularization Lemma 2: The mapping is locally volume preserving, if and only if the functional map matrix is orthonormal: C T C = Id Map-Based Exploration of Intrinsic Shape Differences and Variability, Rustamov et al., SIGGRAPH

44 Regularization Lemma 3: If the mapping is conformalif and only if: C T 1 C = 2 Map-Based Exploration of Intrinsic Shape Differences and Variability, Rustamov et al., SIGGRAPH

45 Basic Pipeline Given a pair of shapes M,N : 1. Compute the first k (~80-100) eigenfunctions of the Laplace- Beltrami operator. Store them in matrices: 2. Compute descriptor functions (e.g., Wave Kernel Signature) on M,N. Express them in Φ M,Φ N, as columns of : A,B 3. Solve M, N : C diagonal matrices of eigenvalues of LB operator Φ M,Φ N C opt = argminkca Bk 2 +kc M N Ck 2 4. Convert the functional map to a point to point map T. C opt 45

46 Conversion to point-to-point Given a functional map C, we would like to convert to to a point-to-point map. f : M R T F T F (f)=g : N R Option 1: declare T(x) = argmax Problems: high computational complexity, low accuracy. y Φ N Cδ x O(n M n N ) 46

47 Conversion to point-to-point Given a functional map C, we would like to convert to to a point-to-point map. f : M R T F T F (f)=g : N R Option 2: declare T(x) = argmin y kδ y Cδ x k 2 Advantages: computational complexity O(n M logn N ), higher accuracy (e.g., works with the identity map). 47

48 Incorporating Orthonormality In many practical situations we would expect a volumepreserving map, which implies: C T C = Id Option: use post-processing to enforce this constraint. Iterate: 1. Compute the point-to-point map T. 2. Solve for the functional map: arg min C, s.t. C T C=Id x M Exactly the same objective as ICP, but in higher dimension. Can use the same method! X kcδ x δ T(x) k

49 Results A very simple method that puts together many constraints and uses 100 basis functions gives reasonable results: Functional maps: a flexible representation of maps between shapes, O., Ben-Chen, Solomon, Butscher, Guibas, SIGGRAPH

50 Results A very simple method that puts together many constraints and uses 100 basis functions gives reasonable results: radius radius 0.05 Functional maps: a flexible representation of maps between shapes, O., Ben-Chen, Solomon, Butscher, Guibas, SIGGRAPH

51 Segmentation Transfer without P2P To transfer functions we do not need to convert functional to pointwise maps. E.g. we can also transfer segmentations: for each segment, transfer its indicator function, and for each point pick the segment that gave the highest value. 51