Geometric Registration for Deformable Shapes 3.3 Advanced Global Matching

Geometric Registration for Deformable Shapes 3.3 Advanced Global Matching Correlated Correspondences [ASP*04] A Complete Registration System [HAW*08]

In this session Advanced Global Matching Some practical applications of the optimization presented in the last session Correlated Correspondences [ASP*04]: Applies MRF model A Complete Registration System [HAW*08]: Applies Spectral matching to filter correspondences 2

Correlated correspondences Correspondence between data and model meshes Model mesh is a template; i.e. data is a subset of model = Template (Model) Data Result Not a registration method; just computes corresponding points between data/model meshes Non-rigid ICP [Hanhel et al. 2003] (using the outputted correspondences) used to actually generate the registration results seen in the paper 3

Basic approach A joint probability model represents preferred correspondences Define a probability of each correspondence set between data/model meshes Find the correspondence with the highest probability using Loopy Belief Propagation (LBP) [Yedidia et al. 2003] 2 main components (next parts of the tal) Probability model Optimization

Joint Probability Model P({ c}) = 1 Z Compatibility constraints Involves pair of correspondences Represents prior nowledge of which correspondence sets maes sense A. Minimize the amount of deformation induced by the correspondences B. Preserve the geodesic distances in model and data Singleton constraints, l ψ Involves a single correspondence C. Corresponding points have same feature descriptor values l ( c, c l ) ψ ( c ) 5

Compatibility 1: Deformation potential Penalize unnatural deformations Edges lengths should stay the same l ij l ij l ij z l l ij x j z x i Corresponding points in data mesh In model mesh 6

Compatibility 1: Deformation potential Penalize unnatural deformations Edges should twist little as possible di j Is the direction from to in s coord system x i x j d i j d i j, d j i d j i x i z d i j d j i z l x i di j x j d j i Corresponding points in data mesh In model mesh 7

Encoding the preference Zero-mean Gaussian noise model for length and twists Define potential ψ dfor each edge ( z, zl ) in the data mesh ( c, c l ) are correspondence variables indicating what is the corresponding point in the model mesh for z, z respectively l ψ d ( c = i, cl = j) = G( l ij lij ) G( d i j di j ) G( d j i d j i ) Caveat: additional rotation needed to measure twist c = i For each possibility of precompute aligning rotation matrices via rigid ICP on surrounding local patch Expand corresp. variables to be site/rotation pairs

Compatibility 2: Geodesic distance potential Penalize large changes in geodesic distance Geodesically nearby points should stay nearby Enforced for each edge in the data mesh x j z dist Geodesic ( z, z l z l ) p x i dist Geodesic ( xi, x j ) If > 3.5p prob assigned 0 otherwise prob assigned 1 Adjacent points in data mesh Corresponding points in model mesh 9

Compatibility 2: Geodesic distance potential Penalize large changes in geodesic distance Geodesically far points should stay far away Enforced for each pair of points in the data mesh whose geodesic distance is > 5p x j z z l distgeodesic( z, zl ) > 5 p x i dist Geodesic ( xi, x j ) If < 2p prob assigned 0 otherwise prob assigned 1 Adjacent points in data mesh Corresponding points in model mesh 10

Singletons: Local surface signature potential Spin images gives matching score for each individual correspondence Compute spin images & compress using PCA gives surface signature s xi at each point x i Discrepancy between sz (data) and s x (model) i Zero-mean Gaussian noise model sz z x i s x i Compare Spin Images Data mesh Model mesh 11

Model summary Get Pairwise Marov Random Field (MRF) Pointwise potential for each pt in data Pairwise potential for each edge in data Far geodesic potentials for each pair of points > 5p apart Data mesh Model mesh

Quic intro: Marov Random Fields Joint probability function visualized by a graph Prob. = Product of the potentials at all edges ψ ψ l ( c ) ( c, cl ) (ex) Surface signature potential (ex) Deformation, geodesic distance potential 1 P ({ c}) = ψ l ( c, cl ) ψ ( c ) Z, l Observed nodes Hidden nodes 13

Loopy Belief Propagation (LBP) Compute marginal probability for each variable Pic variable value that maximizes the marginal prob. Usual way to compute marginal probabilities (tabulate and sum up) taes exponential time BP is a dynamic programming approach to efficiently compute marginal probabilities Exact for tree MRFs, approximate for general MRFs 14

Loopy Belief Propagation (LBP) Basic idea Marginals at node proportional to product of pointwise potential and incoming messages b ( c c a ) = φ ( c ) l N ( ) x i m l ( c ) c b ma mb mc c c c md c d 15

Loopy Belief Propagation (LBP) Basic idea Compute these messages (at each edge) and we are done m l ( c ) φ ( c ) ψ ( c, c m l l l l all values of cl q N ( l) \ ) q l ( c l ) x j c l l all values of φ ( c ) ψ ( c, c ) c l l l ml l c 16

Loopy Belief Propagation (LBP) Basic idea Compute these messages (at each edge) and we are done m l ( c ) φ ( c ) ψ ( c, c l l l l all values of cl q N ( l) \ ) m q l ( c l ) x j c a ma l c l l all values of φ ( c ) ψ ( c, c ) c l l l l c c b mb l ml mc l c c md l c d 17

Loopy Belief Propagation (LBP) Loops: iterate until messages converge φa ( a ) ψ ab( c Start with initial values (ex: c a, cb ) ) values Apply message update rule until convergence all c a of c a ma b mb a mc a ma c c b mc b mb c Convergence not guaranteed, but wors well in practice c c 19

Results & Applications Efficient, coarse-to-fine implementation Xeon 2.4 GHz CPU, 1.5 mins for arm, 10 mins for puppet Correspondences on human body models Finding articulated parts Interpolation between poses

Next topic: HAW*08 An application to the spectral matching method of last session A good illustration of how a matching method fits into a real registration pipeline A pairwise method Deform the source shape to match the target shape Gray = source Yellow = target

Overview Performs both correspondence and deformation Correspondence Deformation Correspondences based on improving closest points After finding correspondences, deform to move shapes closer together Re-tae correspondences from the deformed position Deform again, and repeat until convergence 22

Overview Performs both correspondence and deformation Correspondence Deformation 5 basic steps 1.Closest points 2.Improve by feature matching 3.Filter by spectral matching 4.Expand sparse set 5.Fine-tune target locations 23

Overview Performs both correspondence and deformation Correspondence Deformation 2 basic steps 1.Fit per-cluster rigid transformation 2.Sparse least-squares solve for deformed positions Occasional step: Increase cluster size 24

Detailed Overview Sampling Whole process wors with reduced sample set Correspondence & Deformation Examine each step in more detail Discussion Discuss pros/cons 25

Sample for robustness & efficiency Coarse to fine approach Use uniform subsampling of the surface and its normals Improve efficiency, can improve robustness to local minima Let s mae it more concrete s i Sample set denoted In correspondence: for each s i, find corresponding target points t i In deformation: given t i, find deformed sample positions that match while preserving local shape detail s i t i 26

Correspondence Step #1 Find closest points For each source sample, find the closest target sample s = sample point on source t = sample point on target Target (yellow) Source (gray) arg min t Tˆ s t 2 Usually pretty bad Closest point correspondences 27

Correspondence Step #2 Improve by feature matching Search target s neighbors to see if there s better feature match, replace target Let f(s) be feature value of s Target (yellow) Source (gray) t arg min f ( s) f ( t ) 2 t N ( t) Iterate until we stop moving If we move too much, discard correspondence Much better, but still outliers Feature-matched correspondences 28

Correspondence Step #3 Filter by spectral matching (First some preprocessing) Construct -nn graph on both src & tgt sample set ( = 15) Length of shortest path on graph gives approx. geodesic distances on src & tgt d s, s ) d ( t, t ) g ( i j g i j Goal is to filter these ---------- and eep a subset which is geodesically consistent Target (yellow) Source (gray) Feature-matched correspondences 29

Correspondence Step #3 Filter by spectral matching Construct affinity matrix M using these shortest path distances Consistency term & matrix c ij d g ( si, s j ) d g ( ti, t j ) min{, }, c = 1 d ( t, t ) d ( s, s ) = ii g i j g i j Target (yellow) Source (gray) Threshold c 0 = 0.7 gives how much error in consistency we are willing to accept Feature-matched correspondences 30

Correspondence Step #3 Filter by spectral matching Apply spectral matching: find eigenvector with largest eigenvalue score for each correspondence Iteratively add corresp. with largest score while consistency with the rest is above c_0 Gives ernel correspondences Target (yellow) Source (gray) Filtered matches usually sparse Filtered correspondences 31

Correspondence Step #4 Expand sparse set Lots of samples have no target position For these, find best target position that respects geodesic distances to ernel set Target (yellow) Source (gray) t = arg min e ( s, t) i K i t N ( t, T) e (,) st = d (, ss) d (, tt ) K g g ( s, t ) K g j 2 Expanded correspondences 32

Correspondence Step #4 Expand sparse set Lots of samples have no target position Compute confidence weight based only how well it respects geodesic distances to ernel set Target (yellow) Source (gray) w i ek( si, ti) = exp( ) 2e e = 1 K ( s, t ) K e ( s, t ) K Red = not consistent --- Blue = very consistent Expanded correspondences 33

Correspondence Step #5 Fine-tuning So far, target points restricted to be points in target samples Not accurate when shapes are close together Relax this restriction and let target points become any point in the original point cloud Replace target sample with a closer neighbor in the original point cloud Target (yellow) Source (gray) Expanded correspondences 34

Deformation Solved by energy minimization (least squares) Last step gave target positions Now find deformed sample positions target positions Two basic criteria: t i that match Match correspondences: should be close to Shape should preserve detail (as-rigid-as-possible) Combine to give energy term: s i t i E = λ E + λ E s i corr corr rigid rigid t i 35

Correspondence matching term Combination of point-to-point (α=0.6) and point-toplane (β=0.4) metrics Weighted by confidence weight w i of the target position E w α β ' 2 ' T 2 corr = i si ti + (( si ti) ni) s S i Point-to-point Point-to-plane 36

Shape preservation term Deformed positions should preserve shape detail Form an extended cluster C ~ for each sample point: the sample itself and its neighbors For each C ~ find the rigid transformation (R,T) from sample positions to their deformed locations E = Rs+ T s ' i i s C i s When solving for, constrain them to move rigidly i according to each cluster that it s associated with 2 37

Clusters for local rigidity Initially each cluster contains a single sample point Every 10 iterations (of correspondence & deformation), combine clusters that have similar rigid transformations (forming larger rigid parts) 38

Advantages of features & clustering Source + Target Without Features Without Clustering With Both 39

Results 40

Results 41

Results Efficient, robust method 42

Conclusion Correlated correspondence Robust method for matching correspondences Measure how much the correspondence maes sense Probability model optimized using LBP Requires a template If model is incomplete, then there is no correct corresponding point to assign

Conclusion Non-rigid registration under isometric deformations Improve closest point correspondences using features and spectral matching Deform shape while preserving local rigidity of clusters Iteratively estimate correspondences and deformation until convergence Robust, efficient method Relies on geodesic distances (problematic when holes are too large) 44