Multidimensional scaling

Transcription

1 Multidimensional scaling Lecture 5 Alexander & Michael Bronstein tosca.cs.technion.ac.il/book Numerical geometry of non-rigid shapes Stanford University, Winter 2009

2 Cinderella 2.0 2

3 If it doesn t fit, you must acquit! 3 A glove allegedly dropped by Simpson at the crime scene A pivotal point of the trial: O.J. Simpson is unable to put on the glove

4 Metric model Shape = metric space Similarity = isometry w.r.t. 4 EXTRINSIC GEOMETRY Euclidean metric Invariant to rigid motion Extrinsic geometry is not invariant under inelastic deformations

5 Metric model Shape = metric space Similarity = isometry w.r.t. 5 EXTRINSIC GEOMETRY Euclidean metric Invariant to rigid motion INTRINSIC GEOMETRY Geodesic metric Invariant to inelastic deformation

6 6 Intrinsic vs. extrinsic similarity EXTRINSIC SIMILARITY Part of the same metric space INTRINSIC SIMILARITY Two different metric spaces SOLUTION: Find a representation of and in a common metric space

7 Canonical forms 7 Isometric embedding

8 8 Canonical form distance INTRINSIC SIMILARITY EXTRINSIC SIMILARITY Compute canonical OF CANONICAL forms FORMS = INTRINSIC SIMILARITY

9 Mapmaker s problem 9

10 10 Mapmaker s problem A sphere has non-zero curvature, therefore, it is not isometric to the plane (a consequence of Theoremaegregium) Karl Friedrich Gauss ( )

11 Linial s example 11 A A 1 1 B C D C C 2 2 D D C A A 1 1 B B D 1 1 B Conclusion: generally, isometric embedding does not exist! No contradiction to Nash embedding theorem: Nash guarantees that any Riemannian structure can be realized as a length metric induced by Euclidean metric. We try to realize it using restricted Euclidean metric

12 Minimum distortion embedding 12 Minimum distortion embedding

13 Multidimensional scaling 13 Find an embedding into that distorts the distances the least by solving the optimization problem where are the coordinates of the canonical form The function measuring the distortion of distances is called stress The problem is called multidimensional scaling (MDS)

14 Stress The stress is a function of the canonical form coordinates the distances and 14 L 2 -stress L p -stress L -stress (distortion)

15 Matrix expression of L 2 -stress 15 Some notation: - an matrix of canonical form coordinates (each row corresponds to a point) Shorthand notation for Euclidean distances Write the stress as 1 2

16 Term 1 16

17 17 Term 1 Matrices commute under trace operator where is and matrix with elements

18 18 Term 2 For and zero otherwise

19 Term 2 19 where is and matrix-valued function with elements

20 LS-MDS 20 Minimization of L 2 -stress is called least squares MDS (LS-MDS) variables Non-linear Non-convex (this liable to local convergence) Optimum defined up to Euclidean transformation

21 21 Gradient of L 2 -stress Quadratic in Nonlinear in Constant in Recall exercise in optimization

22 Complexity 22 Stress computation complexity: Stress gradient computation complexity: Steepest descent with line search may be prohibitively expensive (requires multiple evaluations of the stress and the gradient)

23 Observation 23 By Cauchy-Schwarz inequality for all and Equality is achieved for Write it differently:

24 Observation 24 Consider the nonlinear term in the stress

25 25 Majorizing inequality Equality is achieved for [de Leeuw, 1977] We have a quadratic majorizing function to use in iterative majorization algorithm! Jan de Leeuw

26 Iterative majorization 26 Construct a majorizing function satisfying. Majorizing inequality: is convex for all

27 27 Iterative majorization Start with some Find such that Update iterate Until convergence Increment iteration counter Solution

28 28 Iterative majorization The majorizing function is quadratic! Analytic expression for the minimim Analytic expression for pseudoinverse:

29 29 SMACOF algorithm Start with some Update iterate Increment iteration counter Until convergence Solution The algorithm is called SMACOF (Scaling by Minimizing A COnvex Function)

30 SMACOF algorithm vs steepest descent 30 SMACOF is a constant-step gradient descent! Majorization guarantees monotonically decreasing sequence of stress values (untypical behavior for constant-step steepest descent) No guarantee of global convergence Iteration cost:

31 31 MATLAB intermezzo SMACOF algorithm Canonical forms

32 Examples of canonical forms 32 Near-isometric deformations of a shape Canonical forms

33 Examples of canonical forms 33 Visualization of intrinsic similarity

34 Weighted stress 34 where are some fixed weights Matrix expression where is and matrix with elements SMACOF algorithm can be used to minimize weighted stress!

35 Variations on the stress theme 35 Generic form of the stress where is some norm For example, gives the L p -stress The necessary condition for to be the minimizer of is

36 Variations on the stress theme 36 Idea: minimize weighted stress instead of the generic stress and choose the weights such that the two are equivalent If the weights are selected as the minimizers of and will coincide Since is unknown in advance, iterate!

37 37 Iteratively reweighted LS Start with some and Find using as an initialization Update weights Until convergence Increment iteration counter The algorithm is called Iteratively Reweighted Least Squares (IRLS)

38 L -stress 38 Optimization problem can be written equivalently as a constrained problem is called an artificial variable

39 Complexity, bis 39 MDS N=1000 points x5 less points x25 lower complexity MDS N=200 points

40 Multiresolution MDS 40 Bottom-up approach: solve coarse level MDS problem to initialize fine level problem Reduce complexity (less fine-level iterations) Reduce the risk of local convergence (good initialization) Can be performed on multiple resolution levels

41 Two-resolution MDS 41 Grid Data Interpolation operator to transfer solution from coarse level to fine level Can be represented as an sparse matrix Solve fine level problem Interpolate Solve coarse level problem

42 Multiresolution MDS 42 Solve finest level problem Interpolate Solve -st level problem Interpolate Solve coarsest level problem

43 Multiresolution MDS algorithm 43 Hierarchy of grids Hierarchy of data Interpolation operators Start with coarsest-level initialization Solve the l-th level MDS problem For using as initialization Interpolate to next level Solve finest level problem using as initialization

44 Top-down approach 44 Start with a fine-level initialization Decimate fine-level initialization to the coarse grid Typically Solve the coarse-level MDS problem using as initialization Improve the fine-level solution propagating the coarse-level error

45 A problem 45 Fine level Coarse level

46 Correction 46 The fine- and the coarse-level minimizers do not coincide! is called a residual Force consistency of the coarse- and fine-level problems by introducing a correction term into the stress which gives the gradient

47 Correction 47 Fine level Coarse level

48 Correction 48 In order to guarantee consistency, must satisfy at Chain rule which gives the correction term Verify: is the minimizer of coarse-level problem with correction,

49 Modified stress 49 Another problem: the coarse grid problem is unbounded for Can grow arbitrarily Fix the translation ambiguity by adding a quadratic penalty to the stress is called the modified stress [Bronstein et al., 05] The modified coarse grid problem is bounded

50 Two-grid MDS 50 Iteratively improve the fine-level solution using coarse grid residual External iteration is called cycle Correct fine level solution Decimate Interpolate Solve corrected coarse level problem

51 Two-grid MDS 51 Perform a few optimization iterations at fine level between cycles (relaxation) Relax Correct fine level solution Decimate Interpolate Solve corrected coarse level problem

52 52 Two-grid MDS algorithm Start with fine-level initialization Produce an improved fine-level solution by making a few iterations on initialized with Decimate fine-level solution Correction: Solve the coarse-level corrected MDS problem Until convergence using as initialization Transfer residual: Update iterate Solution

53 Multigrid MDS 53 Apply the two-grid algorithm recursively Different cycles possible, e.g. V-cycle Relax Relax Decimate Relax Relax Interpolate correct Decimate Solve coarsest level problem Interpolate correct

54 54 MATLAB intermezzo Multigrid MDS

55 Convergence example 55 Stress Time (sec) Convergence of SMACOF and Multigrid MDS (N=2145)

56 How to accelerate convergence? 56 Let be a sequence of iterates produced by some optimization algorithms converging to Denote by the remained Construct a new sequence converging faster to in the sense

57 Vector extrapolation 57 Construct the new sequence as a transformation of iterates Particular choice: linear combination Question: how to choose the coefficients? Ideally, hence

58 Reduced rank extrapolation (RRE) 58 Problem: depends on unknown Eliminate this dependence by considering the difference which ideally should vanish. Result: solve the constrained linear system of equations in variables Typically hence the system is over-determined and must be solved approximately using least-squares

59 59 Optimization with vector extrapolation Start with some and Generate iterates using optimization on initialized by Extrapolate from Update Until convergence Increment iteration counter

60 60 Safeguarded optimization with vector extrapolation Start with some and Generate iterates using optimization on initialized by Extrapolate from If else Until convergence Increment iteration counter