Vision 3D articielle Multiple view geometry

Vision 3D articielle Multiple view geometry Pascal Monasse monasse@imagine.enpc.fr IMAGINE, École des Ponts ParisTech

Contents Multi-view constraints Multi-view calibration Incremental calibration Global calibration Other methods

Reminder: Triangulation Let us write again the binocular formulae: λx = K(RX + T ) λ x = K X Write Y T = ( X T 1 λ λ ) : ( ) KR KT x 03 K 0 3 0 3 x Y = 0 6 (6 equations 5 unknowns+1 epipolar constraint) We can then recover X.

Multi-linear constraints Bilinear constraints: fundamental matrix x T Fx = 0. There are trilinear constraints: x = x T T i i x, which are not combinations of bilinear contraints All constraints involving more than 3 views are combinations of 2- and/or 3-view constraints.

Trilinear constraints ( Write λ i x i = K i Ri ) T i X Write as AY = 0 with Y = ( X 1 λ 1 ) T λ n Look at the rank of A...

Trilinear constraints Assume R 1 = Id and T 1 = 0. We write A of size 3n (n + 4): K 1 0 x 1 0 0 K 2 R 2 K 2 T 2 0 x 2 0 A =.......... K n R n K n T n 0 0 x n Subtracting from 3rd column the rst column multiplied by K 1 1 x 1, rank of A = rank of A with: K 1 0 0 0 0 A K 2 R 2 K 2 T 2 K 2 R 2 K 1 1 = x 1 x 2 0.......... K n R n K n T n K n R n K 1 1 x 1 0 x n Since K 1 is invertible, we have to look at the rank of the lower-right 3(n 1) (n + 1) submatrix.

Trilinear constraints Rank of A=3+rank of B with K 2 T 2 K 2 R 2 K 1 1 x 1 x 2 0 0 0 K 3 T 3 K 3 R 3 K 1 1 x 1 0 x 3 0 0 B =............. K n 1T n 1 K n 1R n 1K 1 1 x 1 0 0 x n 1 0 K n T n K n R n K 1 1 x 1 0 0 0 x n Size of B: 3(n 1) (n + 1).

Trilinear constraints DB has same rank as B since D is full rank 3(n 1): x2 T 0 0 0 0 x3 T 0 0........... 0 0 xn 1 T 0 D = 0 0 0 x T n [x 2 ] 0 0 0 0 [x 3 ] 0 0........... 0 0 [x n 1] 0 0 0 0 [x n ] D has size 4(n 1) 3(n 1) It is easy to check that the kernel of D is {0}.

Trilinear constraints We get: x2 T K 2T 2 x2 T K 2R 2 K 1 1 x 1 x2 T x 2 0 0 x3 T K 3T 3 x3 T K 3R 3 K 1 1 x 1 0 x3 T x 3 0 DB =.......... x T K n nt n x T K n nr n K 1 1 x 1 0 0 x T x n n M 1 M 2 0... 0 0 Since x T x i i > 0, rank of B = (n 1)+ Rank of M (size 3(n 1) 2): [x 2 ] K 2 R 2 K 1 1 x 1 [x 2 ] K 2 T 2 M =.. [x n ] K n R n K 1 1 x 1 [x n ] K n T n We should have: rank of M = 1, so that rank of A = n + 3. Write that 2 2 submatrices of M should have determinant 0

Trilinear constraints Proposition Let M a 3n 2 matrix written in blocks of 3 rows: a 1 b 1 M =.. a n b n Rank of M < 2 i i, j, a i b T j b i a T j = 0.

Trilinear constraints Proposition Let M a 3n 2 matrix written in blocks of 3 rows: a 1 b 1 M =.. a n b n Rank of M < 2 i i, j, a i b T j b i a T j = 0. Proof: The case b = 0 is trivial, assume b 0. We have a = λb, and ai b T b j i a T = λ(b j i b T b j i b T ) = 0. j We have (ai b T b j i a T ) kl = a k j i bl j bk i al = a k b k i i j a l b l. We j j get that all 2 2 submatrices of M have null determinant.

Trilinear constraints Proposition Let M a 3n 2 matrix written in blocks of 3 rows: a 1 b 1 M =.. a n b n Rank of M < 2 i i, j, a i b T j b i a T j = 0. For i = j, ([x i ] K i R i K 1 1 x 1) ([x i ] K i T i ) = 0 amounts to ([x i ] K i T i ) T (K i R i K 1 1 x 1) = xi K i T i K i R i K 1 = 0 or x T i 1 x 1 K T [T i i ] R i K 1 1 x 1 = 0 (epipolar constraint) [x i ] K i R i K 1 1 x 1([x j ] K j T j ) T [x i ] K i T i ([x j ] K j R j K 1 1 x 1) T = 0 ) [x i ] ( k x k 1 T k ij [x j ] = 0 (9 trilinear constraints)

Trilinear constraints A triplet (x 1, x i, x j ) imposes at most 4 independent constraints on T k because of the cross-products. ij Rank of M = 0 (multiple solutions X ) means i, [x i ] K i R i K 1 1 x 1 = [x i ] K i T i = 0 so that K i R i K 1 1 x 1 and K i T i are proportional, hence x 1 = λk 1 R T i T i (epipole in image 1 wrt image i), so that all camera centers are aligned and X is on this line.

Contents Multi-view constraints Multi-view calibration Incremental calibration Global calibration Other methods

Incremental multi-view calibration 1. Compute two-view correspondences 2. Build tracks (multi-view correspondences) 3. Start from initial pair: compute F, deduce R, T and 3D points (known K ) 4. Add image with common points. 5. Estimate pose (R, T ) 6. Add new 3D points 7. Bundle adjustment 8. Go to 4 See open source software Bundler: SfM for Unordered Image Collections (http://www.cs.cornell.edu/~snavely/bundler/)

Incremental multi-view calibration

Bundle adjustment We have the equations x ij = DK ( R j T j ) Xi i: 3D point index j: view index x ij : 2D projection in view j of point X i D: geometric distortion model K : internal parameters of camera Minimize by Levenberg-Marquardt the error E = ij d(x ij, DK ( R j T j ) Xi ) 2

Global calibration Compute E ij, essential matrices between all views i and j Extract R ij and T ij from E ij Rotation alignment: recover {R i }, global rotation with respect to R 0 = Id, such that R i = R ij R j for all i, j [Martinec&Pajdla CVPR 2007]: write R ij as unitary quaternions, nd minimum of q i q ij q j 2 with ( q1 T ) q T T n = n ij But this does not ensure q i = 1, condition for a quaternion to represent a rotation... No exact solution since R ij can have an error. How to close loops? What about outliers among the R ij? Translation alignment: recover {T i }, global translation with respect to T 0 = 0, such that T i = R ij T j + λ ij T ij

Global calibration Compute E ij, essential matrices between all views i and j Extract R ij and T ij from E ij Rotation alignment: recover {R i }, global rotation with respect to R 0 = Id, such that R i = R ij R j for all i, j Translation alignment: recover {T i }, global translation with respect to T 0 = 0, such that T i = R ij T j + λ ij T ij [Moulon&Monasse&Marlet ICCV 2013]: solve the linear programme min γ with λ ij 1, T i R ij T j λ ij T ij γ {T i },{λ ij },γ

Some results Orangerie dataset

Some results Opera Garnier dataset

Contents Multi-view constraints Multi-view calibration Incremental calibration Global calibration Other methods

Visual hull We assume we are able to segment the object of interest in each view From the silhouette, we can restrict the location inside a cone Intersect cones from all views The result is called the visual hull Source: Wikipedia http://en.wikipedia.org/wiki/visual_hull

Epipolar plane imagery A technique for depth estimation from a movie with controlled motion I Assume a uniform motion of camera along the horizontal line I Consider 2D cuts (x,, y, t ) of the volume I Edges move along lines, whose slope is the disparity I Advantage: large baseline between distant time steps (accurate estimation) and small baseline between close times steps (easier tracking) Source: http://www.informatik.uni-konstanz.de/cvia/ research/light-field-analysis/consistent-depth-estimation/

Software Bundler (2008, open source, University of Washington) PhotoScan (2010, commercial, Agisoft) VisualSfM (2011, open source, University of Wahington) OpenMVG (2012, open source, École des Ponts ParisTech) ColMap (2016, open source, University of North Carolina)

Conclusion Multi-view reconstruction is an active and lively eld of research, but less explored than 2-view stereo correspondence Project: openmvg (incremental and global pipelines) Homography Fundamental matrix Essential matrix Resection Triangulation