Projective geometry for 3D Computer Vision

Transcription

1 Subhashis Banerjee Computer Science and Engineering IIT Delhi Dec 16, 2015

2 Overview Pin-hole camera Why projective geometry? Reconstruction

3 Computer vision geometry: main problems Correspondence problem: Match image projections of a 3D configuration. Reconstruction problem: Recover the structure of the 3D configuration from image projections. Re-projection problem: Is a novel view of a 3D configuration consistent with other views? (Novel view generation)

4 An infinitely strange perspective Parallel lines in 3D space converge in images. The line of the horizon is formed by infinitely distant points (vanishing points). Any pair of parallel lines meet at a point on the horizon corresponding to their common direction. All intersections at infinity stay constant as the observer moves.

5 3D reconstruction from pin-hole projections La Flagellazione di Cristo (1460) Galleria Nazionale delle Marche by Piero della Francesca ( ) (Robotics Research Group, Oxford University, 2000)

6 Pin-hole camera The effects can be modelled mathematically using the linear perspective or a pin-hole camera (realized first by Leonardo?) If the world coordinates of a point are (X, Y, Z) and the image coordinates are (x, y), then The model is non-linear. x = fx /Z and y = fy /Z

7 In terms of projective coordinates where, λ x y 1 x y 1 = P 2 and are homogeneous coordinates. f f X Y Z 1 X Y Z 1 P3

8 Euclidean and Affine geometries Given a coordinate system, n-dimensional real affine space is the set of all points parameterized by x = (x 1,..., x n ) t R n. An affine transformation is expressed as x = Ax + b where A is a n n (usually) non-singular matrix and b is a n 1 vector representing a translation. By SVD A = UΣV T = (UV T )(VΣV T ) = R(θ)R( φ)σr(φ) where where [ λ1 0 Σ = 0 λ 2 ]

9 Euclidean and Affine geometries In the special case of when A is a rotation (i.e., AA t = A t A = I, then the transformation is Euclidean. An affine transformation preserves parallelism and ratios of lengths along parallel directions. An Euclidean transformation, in addition to the above, also preserves lengths and angles. Since an affine (or Euclidean) transformation preserves parallelism it cannot be used to describe a pinhole projection.

10 Spherical geometry The space S 2 : S 2 = { x R 3 : x = 1 } lines in S 2 : Viewed as a set in R 3 this is the intersection of S 2 with a plane through the origin. We will call this great circle a line in S 2. Let ξ be a unit vector. Then, l = { x S 2 : ξ t x = 0 } is the line with pole ξ.

11 Spherical geometry Lines in S 2 cannot be parallel. Any two lines intersect at a pair of antipodal points. A point on a line: Two points define a line: Two lines define a point: l x = 0 or l T x = 0 or x T l = 0 l = p q x = l m

12 Projective geometry The projective plane P 2 is the set of all pairs {x, x} of antipodal points in S 2. Two alternative definitions of P 2, equivalent to the preceding one are 1. The set of all lines through the origin in R The set of all equivalence classes of ordered triples (x 1, x 2, x 3 ) of numbers (i.e., vectors in R 3 ) not all zero, where two vectors are equivalent if they are proportional.

13 Projective geometry The space P 2 can be thought of as the infinite plane tangent to the space S 2 and passing through the point (0, 0, 1) t.

14 Projective geometry Let π : S 2 P 2 be the mapping that sends x to {x, x}. The π is a two-to-one map of S 2 onto P 2. A line of P 2 is a set of the form πl, where l is a line of S 2. Clearly, πx lies on πl if and only if ξ t x = 0. Homogeneous coordinates: In general, points of real n-dimensional projective space, P n, are represented by n + 1 component column vectors (x 1,..., x n, x n+1 ) R n+1 such that at least one x i is non-zero and (x 1,..., x n, x n+1 ) and (λx 1,..., λx n, λx n+1 ) represent the same point of P n for all λ 0. (x 1,..., x n, x n+1 ) is the homogeneous representation of a projective point.

15 Canonical injection of R n into P n Affine space R n can be embedded in P n by (x 1,..., x n ) (x 1,..., x n, 1) Affine points can be recovered from projective points with x n+1 0 by (x 1,..., x n ) ( x 1 x n+1,..., x n x n+1, 1) ( x 1 x n+1,..., x n x n+1 ) A projective point with x n+1 = 0 corresponds to a point at infinity. The ray (x 1,..., x n, 0) can be viewed as an additional ideal point as (x 1,..., x n ) recedes to infinity in a certain direction. For example, in P 2, lim (X /T, Y /T, 1) = lim (X, Y, T ) = (X, Y, 0) T 0 T 0

16 Lines in P 2 A line equation in R 2 is a 1 x 1 + a 2 x 2 + a 3 = 0 Substituting by homogeneous coordinates x i = X i /X 3 we get a homogeneous linear equation (a 1, a 2, a 3 ) (X 1, X 2, X 3 ) = 3 a i X i = 0, X P 2 i=1 A line in P 2 is represented by a homogeneous 3-vector (a 1, a 2, a 3 ). A point on a line: a X = 0 or a T X = 0 or X T a = 0 Two points define a line: l = p q Two lines define a point: x = l m

17 The line at infinity The line at infinity (l ): is the line of equation X 3 = 0. Thus, the homogeneous representation of l is (0, 0, 1). The line (u 1, u 2, u 3 ) intersects l at the point ( u 2, u 1, 0). Points on l are directions of affine lines in the embedded affine space (can be extended to higher dimensions).

18 Conics in P 2 A conic in affine space (inhomogeneous coordinates) is ax 2 + bxy + cy 2 + dx + ey + f = 0 Homogenizing this by replacements x = X 1 /X 3 and y = Y 1 /Y 3, we obtain ax bx 1 X 2 + cx dx 1 X 3 + ex 2 X 3 + fx 2 3 = 0 which can be written in matrix notation as X T CX = 0 where C is symmetric and is the homogeneous representation of a conic.

19 Conics in P 2 The line l tangent to a conic C at any point x is given by l = Cx. x t Cx = 0 = (C 1 l) t C((C 1 l) = l t C 1 l = 0 (because C t = C 1 ). This is the equation of the dual conic.

20 Conics in P 2 The degenerate conic of rank 2 is defined by two line l and m as C = lm t + ml t Points on line l satisfy l t x = 0 and are hence on the conic because (x t l)(m t x) + (x t m)(l t x) = 0. (Similarly for m). The dual conic xy t + yx t represents lines passing through x and y.

21 Projective basis Projective basis: A projective basis for P n is any set of n + 2 points no n + 1 of which are linearly dependent. Canonical basis: , 1.,... 0., }{{}}{{}}{{} points at infinity along each axis origin unit point

22 Projective basis Change of basis: Let e 1, e 2,..., e n+1, e n+2 be the standard basis and a 1, a 2,..., a n+1, a n+2 be any other basis. There exists a non-singular transformation [T] (n+1) (n+1) such that: Te i = λ i a i, i = 1, 2...., n + 2 T is unique up to a scale.

23 Homography The invertible transformation T : P n P n is called a projective transformation or collineation or homography or perspectivity and is completely determined by n + 2 point correspondences. Preserves straight lines and cross ratios Given four collinear points A 1, A 2,A 3 and A 4, their cross ratio is defined as A 1 A 3 A 2 A 4 A 1 A 4 A 2 A 3 If A 4 is a point at infinity then the cross ratio is given as A 1 A 3 A 2 A 3 The cross ratio is independent of the choice of the projective coordinate system.

24 Homography

25 Homography

26 Projective mappings of lines If the points x i lie on the line l, we have l T x i = 0. Since, l T H 1 Hx i = 0 the points Hx i all lie on the line H T l. Hence, if points are transformed as x i = Hx i, lines are transformed as l = H T l.

27 Projective mappings of conics Note that a conic is represented (homogeneously) as x T Cx = 0 Under a point transformation x = Hx the conic becomes x T Cx = x T [H 1 ] T CH 1 x = x T H T CH 1 x = 0 This is the quadratic form of x T C x with C = H T CH 1. This gives the transformation rule for a conic.

28 The affine subgroup In an affine space A n an affine transformation defines a correspondence X X given by: X = AX + b where X, X and b are n-vectors, and A is an n n matrix. Clearly this is a subgroup of the projective group. Its projective representation is [ ] C c T = 0 T n t 33 where A = 1 t 33 C and b = 1 t 33 c.

29 Affine transformations preserve the plane/line at infinity [ A b 0 t 1 ] ( ) x 1 x 2 = A x1 x A general projective transformation moves points at infinity to finite points.

30 The Euclidean subgroup The absolute conic: The conic Ω is intersection of the quadric of equation: n+1 xi 2 i=1 = x n+1 = 0 with π In a metric frame π = (0, 0, 0, 1) T, and points on Ω satisfy X1 2 + X X 3 3 } = 0 X 4 For directions on π (with X 4 = 0), the absolute conic Ω can be expressed as (X 1, X 2, X 3 )I(X 1, X 2, X 3 ) T = 0 The absolute conic, Ω, is fixed under a projective transformation H if and only if H is an Euclidean transformation.

31 Affine calibration of a plane

32 Affine calibration of a plane If the imaged line at infinity is l = (l 1, l 2, l 3 ) t, then provided l 3 0 a suitable projective transformation that maps l back to l is H = H A l 1 l 2 l 3

33 Euclidean calibration of a plane How?

34 Pin-hole camera revisited x c y c f = k X c Y c Z c or x c y c f = X c Y c Z c 1 k = f /Z c

35 Intrinsic/internal parameters k u x c = u u 0 k v y c = v 0 v or u v 1 = fk u 0 u 0 0 fk v v x c y c f = K x c y c f

36 External parameters X c = RX w + T or X c Y c Z c 1 [ R = T 0 T 3 1 ] X w Y w Z w 1

37 Calibrated camera u v 1 = K [R T] X w Y w Z w 1 Image to world correspondences of six points in general position gives camera calibration.

38 What does nternal calibration give? An image point x back projects to a ray defined by x and the camera center. Calibration relates the image point to the ray s direction. Suppose points on the ray are written as X = λd in the camera Euclidean frame. Then these points map to the point x = K [I 0] (λd T,?) T Thus, K is the (affine) transformation between x and the ray s direction d = K 1 x measured in the cameras Euclidean frame. The angle between two rays d 1 and d 2 corresponding to image points x 1 and x 2 may be obtained as (by the cosine formula) d T 1 d 2 cos θ = = T d1 d T 1 d2 d 2 x 1 T (K T K 1 )x 1 x1 T (K T K 1 )x 1 x2 T (K T K 1 )x 2

39 Reconstruction Camera recovery Metrology

40 Surfaces of revolution

41 Modeling of structured scenes

42 A walkthrough

43 Multiple views: epipolar geometry epipole: The epipole is the image in one camera of the optical center of the other camera. epipolar plane: is the plane defined by a 3D point and the optical centers. epipolar line: is the line of intersection of the epipolar plane with the image plane.

44 Epipolar constraint Given the two cameras 1 and 2, we have the camera equations: x 1 = P 1 X and x 2 = P 2 X The optical center projects as P i X = 0 Writing P i = [P i P i t i ] where P i is 3 3 non-singular we have that t i is the optical center. [ ] ti [P i P i t i ] = 0 1

45 Epipolar constraint The epipole e 2 in the second image is the projection of the optical center of the first image: [ ] t1 e 2 = P 2 1 The projection of point on infinity along the optical ray < t 1, x 1 > on to the second image is given by: x 2 = P 2 P 1 1 x 1 The epipolar line < e 2, x 2 > is given by the cross product e 2 x 2. If [e 2 ] is the 3 3 antisymmetric matrix representing cross product with e 2, then we have that the epipolar line is given by [e 2 ] P 2 P 1 1 x 1 = Fx 1

46 Epipolar constraint Any point x 2 on this epipolar line satisfies x 2 T Fx 1 = 0 F is called the fundamental matrix. It is of rank 2 and has 7 DOF. Can be computed from 8 (7) point correspondences. Clearly Fe 1 = 0 (degenerate epipolar line) and e 2 T F = 0. The epipoles are obtained as the null spaces of F.

47 Epipolar Geometry: calibrated case Consider the two image scenario where projection matrices are P = K [I 0] P = K [R t] [ ] ( ) P + K 1 0 = 0 T C = 1 [ ] F = P C P P + [ ] = K t K RK 1 = K T [t] RK 1 = K T R[Rt] K 1 = K T [ ] RK T KR T t

48 Epipolar Geometry: calibrated case Where are the epipoles? ( R T ) t e = P = KR T t 1 ( ) e = P 0 = K t 1 Also, the most useful form of F is F = K T [t] RK 1

49 Epipolar Geometry: calibrated case Consider the representation of the fundamental matrix F = K T [t] RK 1 x T F x = 0 x T K T [t] RK 1 x = 0 This can be interpreted as {}}{ x T K T }{{} [t] R K 1 x = 0 Terms denoted are the calibrated image points Central term is the essential matrix

50 Epipolar Geometry: calibrated case Consider camera pairs P = [I 0] and P = [R t] Essential matrix is given by [ ] E = [t] R = R R T t ˆx T E ˆx = 0 As shown earlier, we have E = K T F K

51 Epipolar Geometry: Essential Matrix Essential Matrix properties Has fewer degrees of freedom than the fundamental matrix Defined by rotation and translation, i.e. six degrees of freedom However there s an overall scale ambiguity The essential matrix has 5 degrees of freedom Implies a translation scale ambiguity We can only solve for heading direction and not actual translation

52 Epipolar Geometry: Essential Matrix A 3 3 matrix is an essential matrix if and only if two if its singular values are equal, and the third is zero Essential Matrix properties Rank-2 constraint implies third singular value is zero Canonical form is singular values of (1, 1, 0)

53 Epipolar Geometry: Essential Matrix Proof Consider the decomposition E = [t] R = SR W = W is orthogonal Z is skew-symmetric Z = Skew-symmetric S can always be written as S = kuzu T U is orthogonal Z = diag(1,1,0)w upto scale Implies S = Udiag(1, 1, 0)W U T E = SR = Udiag(1, 1, 0)W U T R Above is a singular value decomposition of E (Q.E.D.)

54 Epipolar Geometry: Essential Matrix Decomposing Essential Matrix SVD of E is U diag(1, 1, 0) V T Ignoring signs, there are two factorisations S = UZU T and R = UW V T S = UZU T and R = UW T V T t is (upto scale) the last column of U Sign of t is ambiguous Results in 4 possible decomposition pairs Need to verify depth positivity of one point to disambiguate Depth positivity will give an unambiguous solution

55 Epipolar geometry: example

56 Epipolar geometry: computation Given corners: Unguided matching: Obtain a small number of seed matches using cross-correlation and pessimistic thresholds.

57 Epipolar geometry: computation Compute epipolar geometry and reject outliers. Guided matching: Search for matches in a band about the epipolar lines.

58 Structure computation: overview

59 Structure computation 1. Compute the fundamental matrix F from x i x i. 2. Decompose F as F = e M. 3. Compute 3D points X i by intersecting back-projected rays using P = [I 0] and P = [ M e ]

60 Projective ambiguity Suppose P and P are two matrices consistent with F, then x = PX and x = P X But, if H is any arbitrary homography of 3-space, then x = (PH 1 )(HX) and x = (P H 1 )(HX ) Thus, we can only recover projective structure modulo an unknown homography. How to find the homography that will upgrade the projective structure to Euclidean?

61 Auto-calibration: Kruppa s equations SVD of the fundamental matrix gives r F = U s } {{ 1 }} 0 0 {{ 0 } A T E V T } {{ } A Setting û = Au and û = A u, we get û T Eû = 0. E is a very special fundamental matrix with the properties 1. The two epipoles are at origin. 2. Corresponding epipolar lines are identical in the two images.

62 Auto-calibration: Kruppa s equations Consider a plane passing through the two camera centers, tangent to the absolute conic. Such a plane will project to a pair of corresponding epipolar lines in the two images, and these two lines will be tangent to the IAC. Let (λ, µ, 0) T be a tangent to the IAC. Since D is the DIAC in the first image, this tangential relationship may be written as (λ, µ, 0)D(λ, µ, 0) T = 0. Similarly (λ, µ, 0)D (λ, µ, 0) T = 0. Enforcing same solutions from the two equations we obtain: d 11 d 11 = d 12 d 12 = d 22 d 22

63 Auto-calibration: Kruppa s equations Writing D = ACA T and D = A CA T, we obtain a T 1 Ca 1 a T 1 Ca 1 = at 1 Ca 2 a T 1 Ca 2 = at 2 Ca 2 a T 2 Ca 2 Define x = (x, y, z) T. (x, 0) is on the absolute conic if and only if x 2 + y 2 + z 2 = 0 or xx T = 0. u = P(x, y, z, 0) T = K(R Rt)(x, y, z, 0) T = KRx. Thus x = R T K 1 u. u is on IAC if xx T = u T K T RR T K 1 u = u T K T K 1 u = 0. u must lie on conic represented by K T K 1. Equivalently, the DIAC KK T = C. K can be compute by Cholesky decomposition.

64 Sparse bundle adjustment Simultaneously refine 3D coordinates describing the scene geometry Parameters of the relative motion. Optical characteristics of the camera. from corresponding image projections of all points. The process minimizes the reprojection error defined by min a j,b i n i=1 j=1 m v ij d(p(a j, b i ), x ij ) 2 (0) where, v ij = 1 if point i is visible in image j. P is the projection matrix, d is the euclidean distance, x ij is projection of a th j point on image b i

65 Sparse bundle adjustment 1. Initially choose two cameras having most inliers using fundamental matrix 2. Obtain structure and motion by minimization using LM. 3. Remove the points and correspondences with large reprojection errors 4. Remove points which violate the cheirality condition 5. Add images that are more consistent with the already recovered cameras. 6. Apply DLT to initialise the new camera internal and external calibration. 7. Go to 2

66 Sparse bundle adjustment

67 Sparse bundle adjustment

68 Build Rome in a day? Sameer Agarwal, Noah Snavely, Ian Simon, Steven M. Seitz and Richard Szeliski, 2009

69 SIFT Features are scale-space maximas in DOGs.

70 SIFT:example

71 SIFT: matching

72 Vocabulary tree

73 Vocabulary tree match graph

74 Divide and conquer: Efficient large scale SFM using graph paritioning We use multiway Normalised Cut (Shi and Malik, 2000): where Ncut(A, B) = cut(a, B) = assoc(a, V ) = cut(a, B) cut(a, B) + assoc(a, V ) assoc(b, V ) u A,v B u A,t V w(u, v) w(u, t)

75 Component-wise reconstruction and registration Incremental bundle adjustment for reconstruction of each component. Estimation of epipolar geometry, rotation and translation of cur edges. Global motion averaging and registration.

76 Segmentation and reconstruction example

77 Reconstruction examples

78 One order of magnitude speed-up Data set Match graph Pairwise Reconstruction Total Time Pairwise Reconstruction Total creation using matching and by us matching by VSFM time vocabulary tree (mins) registration (mins) by VSFM (mins) by VSFM (mins) (mins) (mins) (mins) Rome N/A N/A N/A Hampi St Peter s Basilica Colosseum

79 Dense Reconstruction Initialize voxel space using the sparse reconstruction Voxel labeling: label s indicates voxel on surface front, label a indicates voxel between camera and surface front, label b indicates voxel is behind surface front.

80 Dense Reconstruction Graph-cut formulation using photo consistency measure. α expansion based optimization for multi-label graph-cut.

81 Dense reconstruction: Qutab pillar

82 Dense reconstruction: Hampi pillar

83 Kinect super-resolution

84 Occlusion is a challenge