Computer Vision in a Non-Rigid World

Transcription

1 2 nd Tutorial on Computer Vision in a Non-Rigid World Dr. Lourdes Agapito Prof. Adrien Bartoli Dr. Alessio Del Bue Non-rigid Structure from Motion

2 Non-rigid structure from motion To extract non-rigid 3D models from uncalibrated video sequences

3 Why is this important? Motion capture/animation HCI Laparoscopy Augmented reality (Pilet et al. 2008) Cloth animation

4 Structure from Motion Pipeline Obtain a set of 2D trajectories from an image stream Image tracking/matching problem Form a complete measurement matrix given the 2D coordinates Matrix Completion or NRSfM with missing data If multiple objects, assign each point to each shape Motion Segmentation from 2D trajectories

5 2D point tracking/matching Obtain a set of 2D trajectories from an image stream Image tracking and/or image matching. KLT (Lucas & Kanade, 1981), SIFT (Lowe, 1999) Several algorithms freely available online in C/C++ (OpenCV) and in Matlab (vlfeat with Matlab wrapper)

6 Missing data Form a complete measurement matrix given the 2D coordinates Matrix Completion or NRSfM with missing data Matrix completion problem (Hogben and Wangsness, 2006) Marques and Costeira. Estimating 3D shape from degenerate sequences with missing data. CVIU 2009

7 Motion Segmentation If multiple objects, assign each point to each shape Motion Segmentation from 2D trajectories Extract the multi-body parts by analysing the 2D motion. AKA as subspace assignment problem. Good tutorial on Motion Segmentation From Feature Trajectories, L. appella, 2008.

8 Overview The Factorization Framework 3D deformable models Affine cameras and deformations The metric constraints Computational methods

9 How? Global 3D Reconstruction Aim: To design model-free algorithms which exploit the complete image data of the object shape. u v u v u v 1 P 1 P u v 2 P 2 P M u v u v 3 P 3 P u v u v 4 P 4 P u v u v 5 P 5 P S

10 Factorization Method Image measurements can describe different objects: Rigid Articulated Deformable Under different camera viewing conditions: Orthographic, Weak Perspective, Para-Perspective Full Perspective However, we model shapes with the same framework: W = M S

11 Overview The Factorization Framework 3D deformable models Affine cameras and deformations The metric constraints Computational methods iterative Structure Matrix S

12 Non-rigid models for SfM Two requirements for the models: Models are data-driven They are tuned on the specific subject. They require training. Models are Meshless We cannot assume to have a dense sampling of the given shape in the video sequence.

13 Low rank bases model A 3D shape at a frame f can be expressed as a linear combination of a set of K basis shapes (low-rank bases model statistical prior) R l 1 S 1 l 2 S 2 l 3 S 3 l 4 S 4 Why linear bases? T Inference is computationally stable (PCA like approach). Link to signal processing literature (image denoising, robust factorization). They fit elegantly into the factorization framework.

14 W = M S What is in S? S contains the parameterisation of the 3D points position. Rigid P S 1 P P Non-Rigid P 2P 2P K1 K1 K1 K 2 K 2 K 2 KP KP KP 3K 6 x Px P

15 The Factorization Framework Measurement Matrix Motion Matrix Structure Matrix u v u v u v 1 P 1 P Wu = M S v P 2 P = M 1 M 2 M k P 1P 1P 2 P 2P 2P u v F 1 F u v FP FP K1 K1 K1 K 2 K 2 K 2 KP KP KP 2F x P 2F x 3K 3K x P Bregler, Hertzmann, Biermann (BHB). Recovering non-rigid shapes from image streams. CVPR 2000

16 Overview The Factorization Framework 3D deformable models Affine cameras and deformations Motion Matrix The metric constraints Computational methods iterative M

17 The motion matrix The Motion matrix represents the time-varying components of the deforming object i.e. camera projection matrix and deformation weights M 11 M 12 M1k M = M 1 M 2 M k M 11 = M2k M 21 M 22 M F1 M F2 MFk 2F x 3 2F x 3 2F x 3 M id 2 x 3 The id 2 x 3 block represents the motion component for the frame i and deformation mode d.

18 The motion blocks M id 2 x 3 Each 2 x 3 block contains two components, the camera projection matrix R i and the scalar coefficient weight l id l id R i Each R i project the respective basis shape d weighted by l id into the 2D image plane P 1P 1P u v u = 1 P... v1 P 2 x p l i1 R i 2 x 3k l ik R i K1 K1 K1 K 2 K 2 K 2 KP KP KP 3k x p

19 Affine camera projection matrix 1.Orthographic: R i R i T = 2x2 identity matrix 2 x 3 3 x 2 2.Weak perspective: R i R i T 0 0 a = a 2x2 diagonal matrix 2 x 3 3 x 2 3.Para perspective: R i R i T a c c b = 2 x2 full matrix 2 x 3 3 x 2 Kanatani et al. Uncalibrated factorization using a variable symmetric affine camera. IEICE 2007

20 NRSFM full model l 11 R 1 l 12 R 1 R l 1k P 1P 1P = l 21 R 2 l 22 R 2 R 2 l 2k P 2P 2P l F1 R F l F2 R F l Fk R F K1 K1 K1 K 2 K 2 K 2 KP KP KP F x P 2F x 3K 3K x P

21 Overview The Factorization Framework 3D deformable models Affine cameras and deformations The metric constraints Computational methods Q Q -1

22 Ambiguity in NRSfM Motion Matrix Arbitrary Transformation Structure Matrix W = M Q Q -1 S 3k x 3k 3k x 3k 2F x 3k 3K x P

23 The problem: Computing the transformation matrix Q: The factorization of the matrix W is only true up to a transformation matrix Q. Given a full rank 3K x 3K matrix Q: M new -1 W = M Q Q S S new Forcing the repetitive structure and orthogonality constraints in the motion matrix is not trivial. In the following we assume the simpler orthographic camera case: R i R i T = 2x2 identity matrix 2 x 3 3 x 2

24 1 st Constraint: The metric constraints in NRSfM K orthogonality constraints at a generic frame i : T l i1 l ik R i R i l i1 R i T = l i1 0 1 l ik R i = l ik The motion matrix is coherent to our model if these constraints are valid for every frame in M. 2 nd Constraint: Repetitive block structure of the same 2 x 3 camera matrix R i l i1 R i l i2 R i l ik R i 2 x 3k

25 NRSfM solution The solution to the problem of NRSfM can be summarized by finding a set of linear 3D basis shapes which describe the deformations of the shape: K1 K1 K K 2 K 2 K 2 1P 1P 1P KP KP KP and a motion matrix for each frame with non-linear constraints as: l i1 R i l i2 R i l ik R i

26 Overview The Factorization Framework 3D deformable models Affine cameras and deformations The metric constraints Computational methods

27 Computational methods W = M S We divide the algorithms in two major classes: (Partial) Closed-form methods No general closed-form solution for both 1 st and 2 nd constraints. The single closed-form solution requires prior information. The solution may not enforce exactly 1 st and 2 nd constraints. Based on iterative methods (i.e. non-linear optimization, EM) Sensitive to initialisation Can be trapped into local minima Enforces exactly 1st and 2nd constraints

28 Closed-form methods Solve for the problem globally: 1. First find an affine low-rank decomposition of the two matrix components M and S: W SVD M affine S affine 2. Find a 3K x 3K mixing matrix Q which imposes the metric and repetitive blocks constraints: W M affine Q Q -1 S affine = M S

29 Affine factorization: the rank constraint The measurement matrix W gathers the 2D image coordinates of a set of P points tracked throughout F frames. W is a rank deficient matrix. This property is exploited to truncate W into rank r using SVD. W U D V T u v u v F 1 F u v u v 1 P 1 P FP FP = 2F x P 2F x r r x r r x P

30 Motion & Shape matrices U D V T W = D D 2F x r r x r r x r r x P

31 Motion & Shape Ambiguity Motion Matrix Arbitrary Transformation Structure Matrix W = M affine Q Q -1 S affine 2F x r 1 1 r x P 1

32 New Motion & Shape matrices Measurement Matrix New Motion Matrix New Structure Matrix W = M new S new 2F x P 2F x r r x P

33 Computing the mixing matrix Q Main problem, compute the matrix Q given both constraints in M given by the NRSfM model. The constraints for the ortographic camera matrix are given in the following form: l i1 R i 2 x 3k l ik R i Thus our aim is to compute a full rank matrix Q that gives: Q M affine = l i1 R i l ik R i 2 x 3k 3k x 3k 2 x 3k

34 iao et al. approach In some cases the metric constraints are not sufficient to compute Q or the solution space contains invalid solutions. Frame 1 R 1 R 2 Find K image frames in which Frame 2 the 3D shape is independent R and force the basis constraints. Frame 3 3 Following the condition of independency, the motion matrix has a specific structure l 1 R Frame 4... l Frame F-2 k R 4 R Motion Matrix L = l l 1 1 R... l Frame F-1 R... l Frame F k k R R A Closed-Form Solution to Non-Rigid Shape and Motion Recovery. iao, Chai, Kanade. ECCV 2004.

35 iao et al. approach iao et al. introduce a new set of basis constraints (independent bases). This results in a new set of equations that can be used to solve for the mixing matrix Q. They also prove that such basis constraints form a closed-form solution for the mixing matrix Q. However: Finding a set of independent bases is not uniquely defined and the solution varies dramatically if choosing a different set of bases. A solution can be computed only in the case of no missing data in the measurements matrix W. J. iao, J. Chai and T. Kanade. A Closed-Form Solution to Non-Rigid Shape and Motion Recovery. IJCV 2006

36 Iterative methods - Previous work Non-linear optimization schemes: Torresani et al. CVPR 2001: Alternating least squares. Brand CVPR 2001: Flexible factorization. Brand CVPR 2005: Weak basis constraints. Del Bue et al. IVC 2006: Bundle adjustment Wang et al PRL Rotation constrained powerfactorization Paladini et al. CVPR Motion Manifold projections Del Bue et al (ECCV 2010). Bilinear factorisation with ALM (BALM) Use of priors: Del Bue et al. AMFG 2005 (affine), CVPR 2006 (projective) Bartoli et al. CVPR Coarse to fine shape model. Torresani et al. PAMI Gaussian priors, linear dynamics.

37 Solution with iterative methods min R i S k l i,k Σi, j ij ij 2 R i Σ d l i,d S d 2 We perform nonlinear optimization by minimizing a geometric error cost function S 1 u v u = 1 P... v1 P l i1 R i l ik R i 2 x p 2 x 3k S k 3k x p

38 Bundle Adjustment min R i S k l i,k Σi i, j 2 ij R iji Σ k l i,k S k 2 We perform nonlinear optimization by minimizing a geometric error cost function The Bundle Adjustment method: Minimize the cost function with a Levenberg-Marquadt algorithm Exploit the sparseness of the Jacobian function matrix to decrease computation and memory requirements. Del Bue, Smeraldi, Agapito. Non-rigid SfM using ranklet-based tracking and non-linear optimization. IVC 2007

39 Levenberg-Marquadt minimization Mixture of Gauss-Newton and Gradient descent Behaves like Gauss-Newton when close to the minimum (quadratic region) Gradient descent when the prediction is poor Depends on a parameter λ that controls the mixture of Gauss- Newton and Gradient descent minimise J J T

40 Jacobian matrix sparseness L 11 L 12 L 21 L 22 L 31 L 32 Exploit the sparseness of the Jacobian matrix to decrease computation and memory requirements.

41 BA how to: In NRSfM, Bundle Adjustment is successful only if applied wisely : Requirements: Parameterisation Initialisation Regularisation Proposed Solution: Use of quaternions to represent rotations Initialise each basis one after the other. Smooth the deformable shape variation through the frames (Temporal smoothness)

42 Parameterisation Parameterisation: Use of quaternions to describe rotations. It is extremely important to use one of the minimal representation for rotations. Unit Quaternions (4 parameters) Angle/Axis (3 parameters) Euler angles (3 parameters) The quaternion is a 4-vector with unitary norm i.e. It lies on a hypersphere

43 Parameterisation It is extremely important to use one of the minimal representation for rotations. Unit Quaternions (4 parameters) Angle/Axis (3 parameters) Euler angles (3 parameters) Angle axis is the most compact formulation however having more non-linearities for the rotation formula:

44 Parameterisation It is extremely important to use one of the minimal representation for rotations. Unit Quaternions (4 parameters) Angle/Axis (3 parameters) Euler angles (3 parameters) Euler angles are a composition of 3 rotations each parametrised. They are affected by Gimbal Lock, a set of singularities in the rotation representation.

45 Regularisation Smoothness priors can improve the 3D reconstruction. In general the most used priors are temporal smoothness (physical) and constraints on the deformation weights l id (statistical and physical). Notice that the low-rank bases model is already considered as a statistical prior! min R i S k l i,k Σi, j where ij B i = Σ k R i Σ k l i,k S k l i,k S k λ B i B i+1 Σi 2 + φ R i R i+1 Σi

46 Initialisation Often neglected in the previous literature, the initialisation of the optimisation parameters plays a very important role. Step 1: Find a mean 3D shape from the non-rigid image data in W by considering the shape rigid i.e. K=1 Step 2: Keeping fixed the camera projection matrix R and the first mean basis S 1 estimate the remaining bases shapes l 11 l F1 R 1 R F S 1 l 12 l 1K l FK S 1 S 1

47 Some experiments Front Side Top

48 Previous work Non-linear optimization schemes: Torresani et al. CVPR 2001: Alternating least squares. Brand CVPR 2001: Flexible factorization. Brand CVPR 2005: Weak basis constraints. Del Bue et al. IVC 2007: Bundle adjustment. Wang et al PRL Rotation constrained powerfactorization Paladini et al. CVPR Motion Manifold projections Del Bue et al. ECCV Bilinear factorisation with ALM (BALM) Use of priors: Del Bue et al. AMFG 2005 (affine), CVPR 2006 (affine) Bartoli et al. CVPR Coarse to fine shape model. Torresani et al. PAMI Gaussian priors, linear dynamics.

49 Coarse to fine approach A coarse to fine approach uses a series of nested minimization problems which iteratively adds 3D deformation modes. The idea is that the modes capture decreasingly important details in the deformation. Advantages: - Automatically selects the number of deformation modes (K) - It can handle missing data - It uses priors during the incremental modes estimation: temporal smoothness, proximity measure, and the ordering of deformations modes. Bartoli et al. Coarse-to-Fine Low-Rank Structure-from-Motion. CVPR 2008

50 Modes estimation W 1 i ;j = R i S 1 + t i 1. First estimate the mean shape using Rigid SfM (Tomasi and Kanade, IJCV 1992). This gives the mean shape with respect to the group of Euclidean transformations and perspective projection (see the Deformotion paper [ezzi and Soatto, IJCV 03]) Set k=1 W k + 1 i ;j = W k i ;j + l i ;k + 1 R i S k + 1;j 2. Find the deformation mode S k+1 and configuration weight l k+1 with Linear and Nonlinear Least Squares. The algorithm uses surface shape and temporal smoothness priors. 3. Stop or increase k and loop to step 2 Bartoli et al. Coarse-to-Fine Low-Rank Structure-from-Motion. CVPR 2008

51 Stopping Criterion The reprojection error always decreases with the number of modes The method computes a v-fold cross-validation score This score either stabilizes or increases when superfluous modes are added Bartoli et al. Coarse-to-Fine Low-Rank Structure-from-Motion. CVPR 2008

52 Previous work Non-linear optimization schemes: Torresani et al. CVPR 2001: Alternating least squares. Brand CVPR 2001: Flexible factorization. Brand CVPR 2005: Weak basis constraints. Del Bue and Agapito IJCV 2006: Bundle adjustment, stereo. Wang et al PRL Rotation constrained powerfactorization Paladini et al. CVPR Motion Manifold projections Del Bue et al. ECCV Bilinear factorisation with ALM (BALM) Use of priors: Del Bue et al. AMFG 2005 (affine), CVPR 2006 (affine) Bartoli et al. CVPR Coarse to fine shape model. Torresani et al. PAMI Gaussian priors, linear dynamics.

53 Torresani et al. Approach The algorithm is essentially a reformulation of Expectation- Maximization (EM) algorithm adpted to the NRSfM problem. E-STEP: Estimate the latent variables L i for each frame, i = 1 F L i = l i1 l i2 l ik M-STEP: Estimate the remaining variables (camera motion and bases) Notice that this step sums up to a non-linear cost function because the orthonormality constraints. This is solved with a single Gauss-Newton step. Torresani et al. Non-Rigid Structure-From-Motion: Estimating Shape and Motion with Hierarchical Priors

54 Torresani et al. Priors Two contributions: The standard linear basis model (PCA) is upgraded to a probabilistic PCA (PPCA) model. This results in placing a statistical prior with a Gaussian distribution over the deformation weights l id. L i = l i1 l i2 l ik ~ N (0, I) A linear dynamics model is introduce which enforce temporal consistency on the deformation weights l id. L i = G L i-1 + n i n i ~ N (0, ) Torresani et al. Non-Rigid Structure-From-Motion: Estimating Shape and Motion with Hierarchical Priors

55 Torresani et al. Results

56 And now the second part...