Accurate Reconstruction by Interpolation Leow Wee Kheng Department of Computer Science School of Computing National University of Singapore International Conference on Inverse Problems and Related Topics 14 Aug 2018 Leow Wee Kheng (NUS) Reconstruction by Interpolation 1 / 35
Motivation Motivation Reconstruction of 3D models and images is important and challenging. Craniofacial Surgery: How does his normal skull look like? How to reconstruct the normal skull model? Leow Wee Kheng (NUS) Reconstruction by Interpolation 2 / 35
Motivation Face Recognition: Who is she? How does she look like? How to recover the unoccluded face image? Leow Wee Kheng (NUS) Reconstruction by Interpolation 3 / 35
Motivation General Idea: Find a model that fits the input data as best as possible. A convenient definition of best is least error, e.g., Given input data v(p), for points p in a selected non-defective part S, find a model f(p) that minimizes the sum-sqaured error E: E = p S(f(p) v(p)) 2. (1) Then, the fitted model is the reconstruction result. target p f v skull model surface point reconstructed position input position face image image point reconstructed colour input colour Leow Wee Kheng (NUS) Reconstruction by Interpolation 4 / 35
Motivation Conventional wisdom: Least-error result is the optimal result. Or is it? Least-error result (Eq. 1) is actually an approximation of input data: there are non-zero errors even for reconstruction of non-defective parts. Surface type Error Over-fitting very complex very small very large complex small large. simple large small very simple. very large very small plane very large none. Leow Wee Kheng (NUS) Reconstruction by Interpolation 5 / 35
Motivation For some application problems, fitting to data needs to tbe perfect, if possible. skull reconstruction aircraft body manufacturing Leow Wee Kheng (NUS) Reconstruction by Interpolation 6 / 35
Interpolation Interpolation Approximation Problem Given input data v(p), for points p in a selected non-defective part S, find a model f(p) that minimizes the sum-sqaured error E: E = p S(f(p) v(p)) 2. Interpolation Problem Given input data v(p), for points p in a selected non-defective part S, find a model f(p) such that f(p) = v(p), for all p S. Leow Wee Kheng (NUS) Reconstruction by Interpolation 7 / 35
Interpolation Approximation Interpolation q i q i p i p i Minimizes error to the points. Non-zero error. Easier problem to solve. Passes through the points exactly. Zero error. Harder problem to solve. Leow Wee Kheng (NUS) Reconstruction by Interpolation 8 / 35
Interpolation Example: Consider n+1 points p i = (x i,y i ), i = 0,1,...,n on a curve. Let s fit the points with a polynomial of degree (order) d: y i = a 0 +a 1 x i +a 2 x 2 i + +a d x d i, for i = 0,1,...,n. (2) In matrix form, 1 x 0 x 2 0 x d 0 1 x 1 x 2 1 x d 1....... 1 x n x 2 n x d n a 0 a 1. a d = y 0 y 1. y n (3) If d > n, no unique solution. If d = n, matrix has inverse: interpolating solution. If d < n, matrix has pseudo-inverse: approximating solution. Leow Wee Kheng (NUS) Reconstruction by Interpolation 9 / 35
Interpolation Interpolation vs. Approximation y 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 x 0 0.5 1 1.5 2 2.5 3 3.5 control points (n = 6) degree n interpolation degree n-1 approximation degree n-3 approximation Leow Wee Kheng (NUS) Reconstruction by Interpolation 10 / 35
Surface Interpolation Surface Interpolation Surface interpolation methods: General spline surface Thin plate spline Laplacian deformation Leow Wee Kheng (NUS) Reconstruction by Interpolation 11 / 35
Surface Interpolation Thin Plate Spline Thin Plate Spline Thin plate spline (TPS) warping [Bookstein89] Analogous to bending of a thin metal sheet. Impose smoothness by minimizing bending energy. q i p i TPS maps points p i = [x i1 x id ] on reference surface to desired positions q i = [v i1 v id ] exactly. Leow Wee Kheng (NUS) Reconstruction by Interpolation 12 / 35
Surface Interpolation Thin Plate Spline The function f that minimizes bending energy takes the form [Bookstein89] f( x) = a x+ n w i U( x p i ). (4) i=1 U(r) is increasing function of distance r, e.g., U(r) = r 2 logr. 1st term is affine (linear) transformation. 2nd term is nonlinear warping. Consider j-th dimension of v ij, dropping j for notational convenience. For interpolation, want to find a and w such that, for each p i, v i = f( p i ) = a p n i + w j U( p i p j ). (5) j=1 Leow Wee Kheng (NUS) Reconstruction by Interpolation 13 / 35
Surface Interpolation Thin Plate Spline Eq. 5 can be written in matrix form [ ][ K P w P 0 a ] [ v = 0 ]. (6) For points in general position, P has independent columns. Then, [ ] [ ] 1 [ ] w K P v = a P. (7) 0 0 It is possible to solve for all dimensions at the same time: Pack w j, a j and v j for all dimensions j = 1,...,d together: and apply Eq. 7. w = [w 1 w d ], a = [a 1 a d ], v = [v 1 v d ]. (8) Leow Wee Kheng (NUS) Reconstruction by Interpolation 14 / 35
Surface Interpolation Laplacian Deformation Laplacian Deformation Laplacian Deformation [Sorkine2004, Masuda2006] Preserve shape: local surface curvature and surface normal. d i p i Laplacian deformation maps points p i on reference surface to desired positions d i exactly. Leow Wee Kheng (NUS) Reconstruction by Interpolation 15 / 35
Surface Interpolation Laplacian Deformation Laplacian operator estimates surface curvature and normal at vertex i: L(p i ) = p i 1 N i j N i p j. (9) Shape is preserved by minimizing the difference of Laplacian operators L(p 0 i ) before and L(p i) after shape deformation: L(p i ) L(p 0 i) 2, (10) This difference is organized into a matrix form for all mesh vertices: Ax b 2. (11) The positional constraints are organised into a matrix form Cx = d (12) Then, Laplacian deformation solves the problem: minimize Ax b 2 subject to Cx = d. (13) Leow Wee Kheng (NUS) Reconstruction by Interpolation 16 / 35
Surface Interpolation Laplacian Deformation Applying QR factorization, C can be decomposed as C = [ Q 1 Q 2 ] R = [ I3m 0 0 I 3(n m) ][ I3m 0 ]. (14) Organize A as A = [ A 1 A 2 ], (15) where A 1 is a 3n 3m matrix and A 2 is a 3n 3(n m) matrix. Then, the soluion of x is given by [ ] d x = Q 1 d+q 2 v =. (16) v where v = (A 2 A 2 ) 1 A 2 (b A 1 d) (17) Leow Wee Kheng (NUS) Reconstruction by Interpolation 17 / 35
Surface Interpolation Laplacian Deformation Comparions TPS Laplacian Deformation approach miminize energy preserve shape shape constraints hard to impose easy to impose positional constraints given in v given in d determines K no need to compute d more constraints larger K larger d larger matrix smaller matrix longer run time shorter run time Leow Wee Kheng (NUS) Reconstruction by Interpolation 18 / 35
Skull Reconstruction Skull Reconstruction When positional constraints conflit, interpoating methods produce flipped surfaces. conflicts Laplacian TPS Approximating methods don t adhere to positional constraints strictly; do not produce flipped surfaces but have non-zero errors. Leow Wee Kheng (NUS) Reconstruction by Interpolation 19 / 35
Flip Avoidance Skull Reconstruction Flip Avoidance Consider two points p and q on a mesh surface, whose corresponding target positions are p and q. Correspondence vectors v(p) = (p,p ), v(q) = (q,q ). If v(p) and v(q) don t cross, then no surface flipping. When v(p) and v(q) meet at a point, they form a triangle. Then, v(p) cosθ(p;q)+ v(q) cosθ(q;p) = p q. (18) Leow Wee Kheng (NUS) Reconstruction by Interpolation 20 / 35
Skull Reconstruction Flip Avoidance Let D denote upper bound: v(p) D, p. Then, v(p) and v(q) will not cross if This condition can be simplified as cosθ(p;q)+cosθ(q;p) < p q D. (19) cosθ(p;q) < p q q p, cosθ(q;p) < 2D 2D. (20) The simplest case is p q > 2D. Leow Wee Kheng (NUS) Reconstruction by Interpolation 21 / 35
Skull Reconstruction Flip Avoidance Let C denote the set of corresponding vectors. Simple No-Crossing Condition There is no crossing if, for all pairs (p,p ) and (q,q ) in C, p q > 2D. General No-Crossing Condition There is no crossing if, for each (p,p ) C, cosθ(p;q) < p q 2D, q N(p) = {q p q 2D} and (q,q ) C. Leow Wee Kheng (NUS) Reconstruction by Interpolation 22 / 35
Skull Reconstruction Flip Avoidance Use manually marked landmarks to ensure anatomical correctness. reference model target model Leow Wee Kheng (NUS) Reconstruction by Interpolation 23 / 35
Skull Reconstruction Flip Avoidance FAIS: Flip-Avoiding Interpolating Surface Input: Reference F, target T, manually marked correspondence C. Output: Reconstructed model R. Rigidly register F to T using manually marked correspondence C. Non-rigidly register F to T using C, then set R as registered F. for k from 1 to K do Find C, nearby corresponding points with similar surface normals. Apply simple no-crossing condition to choose a sparse subset C + from C C. Non-rigidly register R to T with C +. Find C, nearest corresponding points within range. Apply general no-crossing condition to choose a dense subset C + from C C. Non-rigidly register R to T with C +. Leow Wee Kheng (NUS) Reconstruction by Interpolation 24 / 35
Skull Reconstruction Test Results Test Results Test on non-defective targets. Laplacian deformation yields more corresponding points than TPS. Laplacian deformation runs faster with more corresponding points. At last step, not enough memory space to run TPS. Leow Wee Kheng (NUS) Reconstruction by Interpolation 25 / 35
Skull Reconstruction Test Results Test on non-defective targets. FAIS has very small error on non-defective parts. FAIS-0 slightly larger error on non-defective parts. Other methods errors are about 10 times larger. Leow Wee Kheng (NUS) Reconstruction by Interpolation 26 / 35
Skull Reconstruction Test Results Test on non-defective targets. FAIS has smallest error on defective parts. FAIS-0 slightly larger error on defective parts. Other methods errors are more 3 times larger. Leow Wee Kheng (NUS) Reconstruction by Interpolation 27 / 35
Skull Reconstruction Test Results Test on synthetic defective targets. Leow Wee Kheng (NUS) Reconstruction by Interpolation 28 / 35
Skull Reconstruction Test Results Comparison of FAIS and ASM results. Target FAIS ASM Leow Wee Kheng (NUS) Reconstruction by Interpolation 29 / 35
Skull Reconstruction Test Results Test on real defective targets. Leow Wee Kheng (NUS) Reconstruction by Interpolation 30 / 35
Skull Reconstruction Test Results Reconstructino is robust to x-ray metal artifacts Leow Wee Kheng (NUS) Reconstruction by Interpolation 31 / 35
Face Image Deocclusion Face Image Deocclusion For face image deocclusion, apply Robust PCA matrix completion. Leow Wee Kheng (NUS) Reconstruction by Interpolation 32 / 35
Summary Summary Conventional wisdom: least-error solution is optimal. For some problems, interpolation produces more accurate results than approximation. Surface interpolation can be tricky: surface flipping and self-intersection. TPS runs slower with more positional constraints. Laplacian deformation runs faster with more positional constraints. Well-designed interpolation algorithm can be fast and still accurate. Leow Wee Kheng (NUS) Reconstruction by Interpolation 33 / 35
Reference Reference 1. F. L. Bookstein, Principal warps: Thin-plate splines and the decomposition of deformations, IEEE Trans. on Pattern Analysis and Machine Intelligence, 11(6):567 585, 1989. 2. G. Wahba, Spline Models for Observational Data, Society for Industrial and Applied Mathematics, Philadelphia, Pennsylvania, 1990. 3. M. J. D. Powell, A thin plate spline method for mapping curves into curves in two dimensions, Proc. Biennial Conf. on Computational Techniques and Applications: CTAC95, 43 57, 1995. 4. O. Sorkine, D. Cohen-Or, Y. Lipman, M. Alexa, C. Rössl, and H. P. Seidel, Laplacian surface editing, In Proc. Eurographics/ACM SIGGRAPH Symp. Geometry Processing, pages 175 184, 2004. 5. H. Masuda, Y. Yoshioka, Y. Furukawa, Interactive mesh deformation using equality-constrained least squares, Computers & Graphics, 30(6):936 946, 2006. Leow Wee Kheng (NUS) Reconstruction by Interpolation 34 / 35
Reference 1. S. Xie, W. K. Leow, H. Lee and T. C. Lim. Flip-Avoiding Interpolating Surface Registration for Skull Reconstruction. International Journal of Medical Robotics and Computer Assisted Surgery, 14(4), Aug 2018. 2. S. Xie, W. K. Leow and T. C. Lim. Laplacian Deformation with Symmetry Constraints for Reconstruction of Defective Skulls. In Proc. Int. Conf. on Computer Analysis of Images and Patterns, Aug 2017. 3. S. Xie and W. K. Leow. Flip-Avoiding Interpolating Surface Registration for Skull Reconstruction. In Proc. Int. Conf. on Pattern Recognition, Dec 2016. 4. W. K. Leow, G. Li, J. Lai, T. Sim, V. Sharma. Hide and Seek: Uncovering Facial Occlusion with Variable-Threshold Robust PCA. In Proc. IEEE Winter Conf. on Applications of Computer Vision, Mar 2016. Leow Wee Kheng (NUS) Reconstruction by Interpolation 35 / 35