Descriptive Geometry Meets Computer Vision The Geometry of Two Images (# 82)

Descriptive Geometry Meets Computer Vision The Geometry of Two Images (# 8) Hellmuth Stachel stachel@dmg.tuwien.ac.at http://www.geometrie.tuwien.ac.at/stachel th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil

Table of contents. Remarks on linear images. Geometry of two images. Numerical reconstruction of two images th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil

. Remarks on linear images linear image nonlinear (curved) image th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil

Central projection The central projection (according to A. Dürer) can be generalized by a central axonometry. th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil

Central axonometric principle in space E : U c in the image plane E : U g replacements E PSfrag replacements O E E U U cartesian basis O; E, E, E and points at infinity U, U, U U c E c E c O c E c U c central axonometric reference system O c ; E c, E c, E c ; U c, U c, U c th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil

Definition of linear images There is a unique collinear transformation κ: E E mit O O c, E i E c i, U i U c i, i =,,. Any two-dimensional image of E under a collinear transformation is called linear. = { collinear points have collinear or coincident images cross-ratios of any four collinear points are preserved. th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil

Central projection in coordinates PSfrag replacements Notation: Z... center H... principal point vanishing plane Π v x Π image plane x d... focal length x, x, x... camera frame Z x H c x, x... image coordinate frame x d x th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil

Central projection in coordinates ( x x ) = d ( ) x, or homogeneous x x ξ 0 ξ ξ = 0 0 0 0 d 0 0 0 0 d 0 ξ 0. ξ. Transformation from the camera frame (x, x, x ) into arbitrary world coordinates (x, x, x ) and translation from the particular image frame (x, x ) into arbitrary (x, x ) gives in homogeneous form ξ 0 ξ ξ = 0 0 0 0 0 0 0 0 0 o. R 0 0 0 o }{{ } matrix A 0 0 h d f 0 h 0 d f ξ 0. ξ. th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil

Central projection in coordinates Left hand matrix: (h, h ) are image coordinates of the principal point H, (f, f ) are possible scaling factors, and d is the focal length. These parameters are called the intrinsic calibration parameters. Right hand matrix: R is an orthogonal matrix. The position of the camera frame with respect to the world coordinates defines the extrinsic calibration parameters. Photos with known interior orientation are called calibrated images, others (like central axonometries) are uncalibrated. th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil 8

Unknown interior calibration parameters collinear bundle transformation Z PSfrag replacements Z the bundles Z and Z of the rays of sight are collinear th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil 9

. Geometry of two images Given: Two linear images or two photographs. Wanted: Dimensions of the depicted D-object. Historical Stadtbahn station Karlsplatz in Vienna (Otto Wagner, 89) th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil 0

. Geometry of two images The geometry of two images is a classical subject of Descriptive Geometry. Its results have become standard (Finsterwalder, Kruppa, Krames, Wunderlich, Hohenberg, Tschupik, Brauner, Havlicek, H.S.,... ). Why now? Advantages of digital images: less distorsion, because no paper prints are needed, exact boundary is available, and precise coordinate measurements are possible using standard software. th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil

Computer Vision Why now? The geometry of two images is important for Computer Vision, a topic with the main goal to endow a computer with a sense of vision. Basic problems: Which information can be extracted from digital images? How to preprocess and represent this information? Sensor-guided robots, automatic vehicle control, Big Brother,... th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil

Computer Vision Recent textbooks: Yi Ma, St. Soatto, J. Košecká, S.S. Sastry: An Invitation to -D Vision. Springer-Verlag, New York 00 R. Hartley, A. Zisserman: Multiple View Geometry in Computer Vision. Cambridge University Press 000 Fortunately the authors in the cited book refer to some of these standard results (Krames, Kruppa, Wunderlich) th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil

Geometry of two images (epipolar geometry) Z Z Z Z Z viewing situation π Z Z z l l δ Z π collinear transformations γ γ γ γ γ two images Z l π l π π π Z th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil

Geometry of two images (epipolar geometry) Notations: line z = Z Z... baseline, Z, Z... epipoles (German: Kernpunkte), δ... epipolar plane (it is twice projecting), Z π γ Z Z Z Z l l δ Z γ γ γ γ π z l, l... pair of epipolar lines (German: Kernstrahlen), (, )... corresponding views. Z l π l π π π Z th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil

Epipolar constraint Theorem (synthetic version): For any two linear images of a scene, there is a projectivity between two line pencils Z (δ ) Z (δ ) such that the points, are corresponding they are located on (corresponding =) epipolar lines. Theorem (analytic version): Using homogeneous coordinates for both images, there is a bilinear form β of rank such that two points = x R = (ξ 0 : ξ : ξ ) and = x R = (ξ 0 : ξ : ξ ) are corresponding β(x, x ) = i,j=0 b ij ξ i ξ j = (ξ 0 ξ ξ ) (b ij ) 0 @ ξ 0 ξ ξ A = x T B x = 0. th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil

Epipolar constraint Proof (analytic version): Using homogeneous line coordinates, the projectivity between the line pencils can be expressed as β : (u λ + u λ )R (u λ + u λ )R for all (λ, λ ) R \ {(0, 0)}. x and x are corresponding there is a nontrivial pair (λ, λ ) such that (u λ + u λ ) x = 0 (u λ + u λ ) x = 0. These two linear homogeneous equations in the unknowns (λ, λ ) have a nontrivial solution the determinant vanishes, i.e., β(x, x ) := (u x )(u x ) (u x )(u x ) = i,j=0 b ij ξ i ξ j = 0. There are singular points of this correspondance: Z corresponds to all, and vice versa all points correspond to Z = rk(b ij ) =. th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil

Epipolar constraint in the calibrated case Theorem: In the calibrated case the essential matrix B = (b ij ) is the product of a skew symmetric matrix and an orthogonal one, i.e., B = S R. Z Z π Z z Z Z δ Z π l x x l π Proof: We use both camera frames and the homogeneous coordinates x = Z, x = Z. th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil 8

Epipolar constraint in the calibrated case For transforming the coordinates from the second camera frame into the first one, there is an orthogonal matrix R such that x = z + R x with R T = R and z = (z, z, z ) T = Z Z. The points,, Z, Z are coplanar the triple product of the vectors x, z and x = Z vanishes, i.e., det(x, z, x ) = x (z x ) = 0. Z Z π Z z Z Z δ Z π l x x l π th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil 9

Epipolar constraint in the calibrated case We replace the vector product (z x ) by z (z + R x ) = z R x = S R x mit S = Matrix S is skew symmetric and R is orthogonal. 0 @ 0 z z z 0 z z z 0 A. Hence, the coplanarity of x, x and z is equivalent to 0 = x (z x ) = x T S R }{{} B x, also B = S R. The decomposition of the fundamental matrix B into these two factors defines the relative position of the second camera frame against the first one! th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil 0

Essential matrix Theorem: The essential matrix B has two equal PSfrag replacements singular values σ := σ = σ. z Proof: We have B = S R with orthogonal R. The vector S x = z x is orthogonal zu the orthogonal view x n, where z x = sin ϕ x z = = x n z = σ x n. x ϕ x n z x Π z th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil

Singular value decomposition Theorem: [Singular value decomposition] Any matrix A M(m, n; R) can be decomposed into a product A = U D V T with orthogonal U, V and D = diag(σ,..., σ p ) with D M(m, n; R), σ i 0, and p = min{m, n}. The positive entries in the main diagonal of D are called singular values of A. The singular values of A can be seen as principal distortion factors of the affine transformation represented by A, i.e., the semiaxes of the affine image of the unit sphere. Hence the singular values of an orthogonal projection are (, ). th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil

Singular value decomposition A a x A a 0 a LinAlg U D V T A α(a ) α(a 0 ) α(x) LinAlg α(a ) PSfrag replacements a 0 a a xa th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil α(a 0 ) α(a ) α(a ) α(x) A U D V T A

x A α(a 0 ) α(a ) α(a ) α(x) A U D V T A a 0 a a x A α(a 0 ) α(a ) α(a ) α(x) A A a Singular value decomposition x a 0 a rotation V T Lin LinAlg U D V T A D scaling th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil A α(a ) α(a 0 ) α(a 0 ) α(a α(x) ) α(a ) α(a α(x) ) A U D V T A rotation U a 0 a a xa LinAlg α(a 0 ) α(a ) α(a ) α(x) A U D V T A a 0 a a xa LinAlg α(a 0 ) α(a ) α(a ) α(x) A D scaling

Given: Two either calibrated A or uncalibrated images. a x aa α(a 0 x) α(aa ) α(a ) α(a α(x) ) α(a A ) U D V α(x) T Wanted: viewing situation, i.e., determine the relative position of the two camera frames, and the location of any space point from its images (, ). What means α(a 0 ) reconstruction A D a 0 scaling π a a x A α(a 0 ) α(a ) α(a ) α(x) A D scaling π π π Z π Z Z Z a 0 a Z Z π l α(a l 0 ) π th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil α(a 0 ) α(a ) α(a ) α(x) A U D V T A a 0 a a xa α(a 0 ) α(a ) α(a ) α(x) A U D V T A δ a xa z α(a ) α(a ) α(x) A D scaling

A α(a 0 ) α(a ) α(a ) First fundamental theorem α(x) A U D V T Theorem: A a xa From two uncalibrated images with given projectivity between epipolar lines the depicted object can be reconstructed up to a collinear transformation. α(a 0 ) Sketch of the proof: The two images can be placed in space such that pairs of epipolar lines are intersecting. Then for arbitrary Z, Z on the baseline z = ZZ there is a reconstructed D object. Any other choice of the D viewing situation gives a collinear transform of the D object. a 0 a a x A α(a 0 ) α(a ) α(a ) α(x) scaling A Z π Z Z Z a 0 Z Z π l l π th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil α(a 0 ) α(a ) α(a ) α(x) A U D V T A a 0 a α(a ) α(a ) α(x) A U D V T A δ a a xa z α(a 0 ) α(a ) α(a ) α(x) A D scaling

A α(a 0 ) α(a ) α(a ) Second fundamental theorem α(x) U D V T Theorem (S. Finsterwalder, A 899): a xa From two calibrated images with given projectivity between epipolar lines the depicted object can be reconstructed a 0 up to a similarity. Sketch of the proof: Now in the two bundles of rays the pencils of epipolar planes δ are congruent, and they can be made coincident by a rigid motion. Then relative to the first bundle Z for any Z z there is a reconstructed D object. Any other choice of Z gives a similar D object. A a a x A α(a 0 ) α(a ) α(a ) α(x) A D scaling Z π Z Z Z a 0 Z Z π l l π th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil α(a 0 ) α(a ) α(a ) α(x) A U D V T A a 0 a α(a 0 ) α(a ) α(a ) α(x) A U D V T A δ a a xa z α(a 0 ) α(a ) α(a ) α(x) A D scaling

A D scaling a Determination of epipoles geometric meaning xa Problem of Projectivity: Given: pairs of corresponding points (, ),..., (, ). Wanted: A pair of points (S, S ) (= epipoles) such that there isscaling a projectivity S ([S ],..., [S ]) S ([S ],..., [S ]). S S π th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil U D V T A a 0 a α(a 0 ) α(a ) α(a ) α(x) A D π S S π π

S π π Determination of epipoles geometric meaning Problem of Projectivity: Given: pairs of corresponding points (, ),..., (, ). Wanted: A pair of points (S, S π ) (= epipoles) such that there is a projectivity S S ([S ],..., [S ]) S ([S ],..., [S ]). π th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil S π S S π S S π π

Determination of epipoles analytic solution Theorem: If pairs of corresponding points (, ),..., (, the determination of the epipoles is a cubic problem. ) are given, Proof: pairs of corresponding points give linear homogeneous equations β(x i, x i ) = xt i B x i = 0, i =,...,, for the 9 entries in the ( )-matrix B = (b ij ) called essential matrix. det(b ij ) = 0 gives an additional cubic equation which fixes all b ij up to a common factor. For noisy image points it is recommended to use more than points and methods of least square approximation for obtaining the best fitting matrix B: th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil S S π π S S π π 8

Determination of epipoles analytic solution ) Let A denote the coefficient matrix in the linear system for the entries of B. Then the least square fit for this overdetermined system is an eigenvector for the smallest eigenvalue of the symmetric matrix A T A. ) As an essential matrix needs to have rank, we use the projection into the essential space. This means, the singular value decomposition of B gives a representation B = U diag(σ, σ, σ ) V T with orthogonal U, V and σ σ σ. Then in the uncalibrated case B = U diag(σ, σ, 0) V to the Frobenius norm) and in the calibrated case B = U diag(σ, σ, 0) V T with σ = (σ + σ )/. th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil S S π π S S π π 9 is optimal (with respect

Determination of epipoles analytic solution ) Let A denote the coefficient matrix in the linear system for the entries of B. Then the least square fit for this overdetermined system is an eigenvector for the smallest eigenvalue of the symmetric matrix A T A. ) As an essential matrix needs to have rank, we use the projection into the essential space. This means, the singular value decomposition of B gives a representation B = U diag(σ, σ, σ ) V T with orthogonal U, V and σ σ σ. Then in the uncalibrated case B = U diag(σ, σ, 0) V is optimal (with respect to the Frobenius norm) and in the calibrated case B = U diag(σ, σ, 0) V T with σ = (σ + σ )/. th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil S S π π S S π π 9

π S S π π Step : 8. Numerical reconstruction of two images Specify at least reference points 9 9 8 0 0 π S S π π th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil S S π π 9 8 0 9 0 8 S S π π 0

S S π π Step : Compute the essential matrix S S π π Step : Compute the essential matrix B including the pairs of epipolar lines th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil S S π π S S π π

Step : Factorize B = S.R Theorem: There are exactly two ways of decomposing B = U D V T with D = diag(σ, σ, 0) into a product S R with skew-symmetric S and orthogonal R : Proof: S = ±U R + D U T and R = ±U R T + V T with R + = 0 @ 0 0 0 0 0 0 a) It is sufficient to factorize U D = S R which implies B = S (R V T ), i.e., R = R V T. b) D represents the product of the orthogonal projection into the x x -plane and the scaling with factor σ. The rotation U transforms the x x -plane into the image plane of U D. th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil S S π π A. S S π π

Step : Factorize B = S.R c) Any skew symmetric matrix S represents the product of an orthogonal projection parallel to z, a 90 -rotation about z and a scaling with factor z. d) R + D is skew-symmetric with z = (0, 0, σ). We transform it by U to obtain the required position, i.e., S = ±U (R + D) U T. R + commutes with D, = U D = [ ±U R + D U T ] [±U R T ] +. }{{}}{{} S R e) B represents an orthogonal axonometry; its column vectors are images of an orthonormal frame. We know from Descriptive Geometry that apart from translations there are not more than two different frames with given images. th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil S S π π S S π π

Summary of algorithm ) Specify n > pairs ( i, i ), i =,..., n. ) Set up linear system of equations for the essential matrix B and seek S best fitting matrix (eigenvector of the smallest eigenvalue). ) Compute the closest rank matrix B with two equal singular values. ) Factorize B = S R ; this reveals the relative position of the two camera frames. ) In one of the frames compute the approximate point of intersection between corresponding rays. ) Transform the recovered coordinates into world coordinates. th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil S π π S S π π

Analysis of precision, Remaining problems automated calibration (autofocus and zooming change the focal distance π d), critical configurations. th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil S S π S S π π

S S π π 8 9 9 8 0 original image The solution 0 π π th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil 8 9 9 8 0 S S π π the reconstruction (M : 00) 0 S S π π

S S π π Z front view top view Z 8 8 Position of centers 9 9 9 relative to the depicted object th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil 8 0 0 Z Z S S π π S S π π

Literatur H. Brauner: Lineare Abbildungen aus euklidischen Räumen. Beitr. Algebra Geom., (98). O. Faugeras: Three-Dimensional Computer Vision. A Geometric Viewpoint. π MIT Press, Cambridge, Mass., 90. O. Faugeras, Q.-T. Luong: The Geometry of Multiple Images. MIT Press, Cambridge, Mass., 00. R. Harley, A. Zisserman: Multiple View Geometry in ComputerVision. Cambridge University Press 000. H. Havlicek: On the Matrices of Central Linear Mappings. Math. Bohem., (99). th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil S S π S S π π 8

E. Kruppa: Zur achsonometrischen Methode der darstellenden Geometrie. Sitzungsber., Abt. II, österr. Akad. Wiss., Math.-Naturw. Kl. 9, 8 0 (90). Yi Ma, St. Soatto, J. Košecká, S. Sh. Sastry: An Invitation to -D Vision. Springer-Verlag, New York 00. H. Stachel: Zur Kennzeichnung der Zentralprojektionen nach H. Havlicek. π Sitzungsber., Abt. II, österr. Akad. Wiss., Math.-Naturw. Kl. 0, (99). J. Szabó, H. Stachel, H. Vogel: Ein Satz über die Zentralaxonometrie. Sitzungsber., Abt. II, österr. Akad. Wiss., Math.-Naturw. Kl. 0, (99). J. Tschupik, F. Hohenberg: Die geometrische Grundlagen der Photogrammetrie. In Jordan, Eggert, Kneissl (eds.): Handbuch der Vermessungskunde III a/. 0. Aufl., Metzlersche Verlagsbuchhandlung, Stuttart 9, 9. th International Conference on Geometry and Graphics, August 0, 00, Salvador/Brazil S S π S S π π 9