Rigid Multiview Varieties

Transcription

1 Joe Kileel University of California, Berkeley January 9, 2016 Nonlinear Algebra JMM, Seattle Joe Kileel

2 Preprint arxiv: Michael Joswig Bernd Sturmfels André Wagner

3 Algebraic vision Multiview geometry studies 3D scene reconstruction from images. Foundations in projective geometry. Algebraic vision bridges to algebraic geometry (combinatorial, computational, numerical,...). Oct 8 9, 2015, Berlin May 2 6, 2016, San Jose

4 What is a camera? A camera is a full rank 3 4 real matrix A. Determines a projection P 3 P 2 ; X AX thought of as taking a picture. A choice of point C P 3 (center), plane π P 3 (viewing plane), and coordinates on π gives a camera.

5 Multiview variety Given n cameras A = (A 1,..., A n ) in generic position, their multiview variety V A is the closure of the image of the rational map: P 3 P 2 P 2 P 2 X (A 1 X, A 2 X,..., A n X).

6 Multiview variety Given n cameras A = (A 1,..., A n ) in generic position, their multiview variety V A is the closure of the image of the rational map: P 3 P 2 P 2 P 2 X (A 1 X, A 2 X,..., A n X). Space of n consistent views of one world point.

7 Multiview variety Given n cameras A = (A 1,..., A n ) in generic position, their multiview variety V A is the closure of the image of the rational map: P 3 P 2 P 2 P 2 X (A 1 X, A 2 X,..., A n X). Space of n consistent views of one world point. Irreducible threefold isomorphic to P 3 blown-up at n points.

8 Multiview variety Given n cameras A = (A 1,..., A n ) in generic position, their multiview variety V A is the closure of the image of the rational map: P 3 P 2 P 2 P 2 X (A 1 X, A 2 X,..., A n X). Space of n consistent views of one world point. Irreducible threefold isomorphic to P 3 blown-up at n points. Prime ideal I A R[u i0, u i1, u i2 : i = 1,..., n] is Z n -multihomogeneous.

9 Equations For which u j and u k, does: { A jx = λ ju j A k X = λ k u k have a nonzero solution in X, λ j, λ k? Rewrite as: X [ ] B jk λ j = 0 where B jk Aj u j 0 := A λ k 0 u k k 6 6

10 Equations For which u j and u k, does: { A jx = λ ju j A k X = λ k u k have a nonzero solution in X, λ j, λ k? Rewrite as: X [ ] B jk λ j = 0 where B jk Aj u j 0 := A λ k 0 u k k 6 6 Theorem (Heyden-Aström 1997) For n 4, the ( n 2) bilinear forms det(b jk ) where 1 j < k n cut out V A set-theoretically.

11 Equations For which u j and u k, does: { A jx = λ ju j A k X = λ k u k have a nonzero solution in X, λ j, λ k? Rewrite as: X [ ] B jk λ j = 0 where B jk Aj u j 0 := A λ k 0 u k k 6 6 Theorem (Heyden-Aström 1997) For n 4, the ( n 2) bilinear forms det(b jk ) where 1 j < k n cut out V A set-theoretically. Theorem (Aholt-Sturmfels-Thomas 2013) These ( ( n 2) bilinear forms and n 3) trilinear forms minimally generate IA. Those and ( n 4) quadrilinear forms are a universal Gröbner basis.

12 Rigid multiview variety Given n cameras A = (A 1,..., A n ) in generic position, their rigid multiview variety W A is the closure of the image of the rational map: V (Q) P 3 P 3 ( (P 2 ) n (P 2 ) n (X, Y ) (A1 X,... A n X), (A 1 Y,... A n Y ) ), Q(X, Y ) = (X 0Y 3 Y 0X 3) 2 + (X 1Y 3 Y 1X 3) 2 + (X 2Y 3 Y 2X 3) 2 X 2 3 Y 2 3. X Y u 1 v 1 u 2 v2 f 1 f 2

13 Rigid multiview variety Given n cameras A = (A 1,..., A n ) in generic position, their rigid multiview variety W A is the closure of the image of the rational map: V (Q) P 3 P 3 ( (P 2 ) n (P 2 ) n (X, Y ) (A1 X,... A n X), (A 1 Y,... A n Y ) ), Q(X, Y ) = (X 0Y 3 Y 0X 3) 2 + (X 1Y 3 Y 1X 3) 2 + (X 2Y 3 Y 2X 3) 2 X 2 3 Y 2 3. X Y u 1 v 1 u 2 v2 f 1 f 2 Irreducible 5-fold inside V A V A. Prime ideal J A in R[u i0, u i1, u i2, v i0, v i1, v i2 : i = 1,..., n] is Z 2n -multihomogeneous.

14 Equations Write Q(X, Y ) = T (X, X, Y, Y ), where T (,,, ) is a quadrilinear form. Theorem (Joswig-K.-Sturmfels-Wagner 2015) The octics coming from two pairs of cameras: T ( 5B j 1k 1 i 1 (u), 5B j 1k 1 i 2 (u), 5C j 2k 2 i 3 (v), 5C j 2k 2 i 4 (v) ) cut out W A as a subvariety of V A V A set-theoretically. For this, 16 suffice.

15 Triangulation From two views of one world point X, recover X by intersecting back-projected lines. Works unless X is collinear with centers.

16 Triangulation From two views of one world point X, recover X by intersecting back-projected lines. Works unless X is collinear with centers. For 1 j < k n and 1 i 6, let: [ ] B jk Aj u j 0 (u) = A k 0 u k 6 6

17 Triangulation From two views of one world point X, recover X by intersecting back-projected lines. Works unless X is collinear with centers. For 1 j < k n and 1 i 6, let: [ ] B jk Aj u j 0 (u) = A k 0 u k 6 6 B jk i (u) be the 5 6 matrix that is B jk (u) with its i th row removed

18 Triangulation From two views of one world point X, recover X by intersecting back-projected lines. Works unless X is collinear with centers. For 1 j < k n and 1 i 6, let: [ ] B jk Aj u j 0 (u) = A k 0 u k 6 6 B jk i (u) be the 5 6 matrix that is B jk (u) with its i th row removed 5B jk i (u) be the height 6 column of signed maximal minors of B jk i (u)

19 Triangulation From two views of one world point X, recover X by intersecting back-projected lines. Works unless X is collinear with centers. For 1 j < k n and 1 i 6, let: [ ] B jk Aj u j 0 (u) = A k 0 u k 6 6 B jk i (u) be the 5 6 matrix that is B jk (u) with its i th row removed 5B jk i 5B jk i (u) be the height 6 column of signed maximal minors of B jk i (u) (u) be the height 4 column consisting of the top of 5B jk i (u)

20 Triangulation From two views of one world point X, recover X by intersecting back-projected lines. Works unless X is collinear with centers. For 1 j < k n and 1 i 6, let: [ ] B jk Aj u j 0 (u) = A k 0 u k 6 6 B jk i (u) be the 5 6 matrix that is B jk (u) with its i th row removed 5B jk i 5B jk i C jk (v), C jk i (u) be the height 6 column of signed maximal minors of B jk i (u) (u) be the height 4 column consisting of the top of 5B jk i (u) (v), 5C jk i (v) and 5C jk i (v) be the analogs with v.

21 Equations Write Q(X, Y ) = T (X, X, Y, Y ), where T (,,, ) is a quadrilinear form. Theorem (Joswig-K.-Sturmfels-Wagner 2015) The octics coming from two pairs of cameras: T ( 5B j 1k 1 i 1 (u), 5B j 1k 1 i 2 (u), 5C j 2k 2 i 3 (v), 5C j 2k 2 i 4 (v) ) cut out W A as a subvariety of V A V A set-theoretically. For this, 16 suffice.

22 Equations Write Q(X, Y ) = T (X, X, Y, Y ), where T (,,, ) is a quadrilinear form. Theorem (Joswig-K.-Sturmfels-Wagner 2015) The octics coming from two pairs of cameras: T ( 5B j 1k 1 i 1 (u), 5B j 1k 1 i 2 (u), 5C j 2k 2 i 3 (v), 5C j 2k 2 i 4 (v) ) cut out W A as a subvariety of V A V A set-theoretically. For this, 16 suffice. Sketch. These octics vanish on W A. Conversely:

23 Equations Write Q(X, Y ) = T (X, X, Y, Y ), where T (,,, ) is a quadrilinear form. Theorem (Joswig-K.-Sturmfels-Wagner 2015) The octics coming from two pairs of cameras: T ( 5B j 1k 1 i 1 (u), 5B j 1k 1 i 2 (u), 5C j 2k 2 i 3 (v), 5C j 2k 2 i 4 (v) ) cut out W A as a subvariety of V A V A set-theoretically. For this, 16 suffice. Sketch. These octics vanish on W A. Conversely: For n 3, show one of B 12 1, B 12 2, B 12 1, B 13 2 has rank 5, similarly with C.

24 Equations Write Q(X, Y ) = T (X, X, Y, Y ), where T (,,, ) is a quadrilinear form. Theorem (Joswig-K.-Sturmfels-Wagner 2015) The octics coming from two pairs of cameras: T ( 5B j 1k 1 i 1 (u), 5B j 1k 1 i 2 (u), 5C j 2k 2 i 3 (v), 5C j 2k 2 i 4 (v) ) cut out W A as a subvariety of V A V A set-theoretically. For this, 16 suffice. Sketch. These octics vanish on W A. Conversely: For n 3, show one of B 12 1, B 12 2, B 12 1, B 13 2 has rank 5, similarly with C. For n = 2, need special geometric argument because of world points collinear with centers.

25 Computations Conjecture (Joswig-K.-Sturmfels-Wagner 2015) J A is minimally generated by 4 9 n6 2 3 n n n n2 1 3 n polynomials, coming from two triples of cameras, and their number per symmetry class of degrees is: ( ) : 1 2 ( ) n 2 ( ) : 3 2 ( n n ) 2)( 3 ( ) : 9 (n) 2 2 ( ) : 1 n 2( ) n ( ) : 1 2 ( ) n ( ) : 1 2n ( )( n 1 n ) 3) ( ) : 1 (n ( ) : 3 2n ( n 2 )( 3 n ) 2 3

26 Computations Conjecture (Joswig-K.-Sturmfels-Wagner 2015) J A is minimally generated by 4 9 n6 2 3 n n n n2 1 3 n polynomials, coming from two triples of cameras, and their number per symmetry class of degrees is: ( ) : 1 2 ( ) n 2 ( ) : 3 2 ( n n ) 2)( 3 ( ) : 9 (n) 2 2 ( ) : 1 n 2( ) n ( ) : 1 2 ( ) n ( ) : 1 2n ( )( n 1 n ) 3) ( ) : 1 (n ( ) : 3 2n ( n 2 )( 3 n ) 2 3 Computational proof. Up to n = 5, when there are 4940 minimal generators.

27 Generalizations Images of four coplanar world points. Images of rigid world triangles. Proposed approach to images of unlabeled world points.

28 References [1] C. Aholt, B. Sturmfels and R. Thomas: A Hilbert scheme in computer vision, Canadian Journal of Mathematics 65 (2013) [2] D. Grayson and M. Stillman: Macaulay2, a software system for research in algebraic geometry, available at [3] R. Hartley and A. Zisserman: Multiple View Geometry in Computer Vision, Cambridge University Press, [4] A. Heyden and K. Aström: Algebraic properties of multilinear constraints, Mathematical Methods in the Applied Sciences 20 (1997) [5] M. Joswig, J. Kileel, B. Sturmfels and A. Wagner:, arxiv: [6] B. Li: Images of rational maps of projective spaces, arxiv: [7] E. Miller and B. Sturmfels: Combinatorial Commutative Algebra, Graduate Texts in Mathematics, Springer Verlag, New York, 2004.

29 Thank you!