3D Computer Vision. Class Roadmap. General Points. Picturing the World. Nowadays: Computer Vision. International Research in Vision Started Early!

Transcription

1 3D Computer Vision Adrien Bartoli CNRS LASMEA Clermont-Ferrand, France Søren I. Olsen DIKU Copenhagen, Denmark Lecture 1 Introduction (Chapter 1, Appendices 4 and 5) Class Roadmap (AB, lect. 1-3) Aug. 25: Introduction, modeling of cameras (AB, lect. 4-7) Aug. 27: Numerical optimization ; estimation of two-view relationships (SIO, lect. 8-15) (AB, lect ) Sep. 24: Structure-from-Motion (SIO, lect ) (AB, lect ) Oct. 6: More Structure-from-Motion (AB, lect ) Oct. 8: Vision in deformable environments 2 General Points The world has a 3D structure and is made of objects Human beings know how to describe the world, given millions of pixels (picture elements) available on the retina Each pixel contains information about the incoming light Real world objects do not exist on the retina, but we see them: this is the results of the visual process Picturing the World Prehistoric paintings Lascaux cave, France ~ 14,000 BC The Marriage of the Virgin Raphael Sanzio, 1504 Les Demoiselles d Avignon Pablo Picasso, 1907 Crags and Crevices Jane Frank, International Research in Vision Started Early! Discovery of the pinhole camera Mozi, ~ 400 BC (China) Nowadays: Computer Vision Optical principle of the pinhole camera Aristotle, ~ 330 BC (Greece) First movies Les Frères Lumière, 1895 (France) First camera obscura Alhazen, 1021 (Iraq) Shen Kuo, 1088 (China) X-ray Wilhelm Röntgen, 1895 (Germany) Use as a drawing aid Leonardo da Vinci, 1519 (Italy) First permanent photograph Joseph Niépce, 1826 (France) MRI Raymond Damadian, 1936 (USA) First permanent color photograph James Maxwell, 1861 (UK) 5 6 1

2 What is Vision? Solving the Vision Problem? Real world Images (sets of pixels) Vision is the act of knowing what is where by looking. -Aristotle Vision system Prior knowledge (shape, color, behavior) Visual process (interpretation) Scene description (objects, position, motion) 7 Human Vision is extremely complex Computer Vision does not seek to understand or to reproduce Human Vision, but to make computer algorithms with similar properties Some Computer Vision problems: Object (class) detection or recognition Object tracking In this class, we focus on the problem of perceiving 3D from multiple images 8 The Structure-from-Motion (SfM) Paradigm The Calibrated SfM Paradigm Multiple images of a (rigid) scene Multiple images of a (rigid) scene (Metric) 3D model (Metric) 3D model Internal camera calibration 9 10 The Uncalibrated SfM paradigm Euclidean Structure-from-Motion Multiple images of a rigid scene (Metric) 3D model 1. Pick up one camera (or more ) 2. Take some pictures 3. Compute a Euclidean 3D model of the scene structure and the camera (intrinsic and extrinsic) parameters from those pictures and the following assumptions 1. The scene is rigid 2. Some knowledge on the camera intrinsic parameters (e.g. the focal length)

3 Digitizing the World The Basic Constraint: Frozen Time Measurement (distance, height, speed, ) Editing (insert / remove objects or parts) Rendering (increase the realism of computer graphics models) Retargeting (apply the captured dynamics to other objects) The scene surface The viewing rays for corresponding points intersect in space A camera Two Key Components Feature-Based SfM: Points, Lines, Image matching or registration 3D reconstruction

4 19 20 Mature Parts in Rigid SfM Camera tracking with sparse structure recovery including self-calibration Visual geometry Textbooks: [Hartley and Zisserman 03 ; Faugeras and Luong 01 ; ] Companies: 2d3 (University of Oxford), RealViz (INRIA), evs (University of Verona), SfM Image registration 3D reconstruction Some Open Problems in Rigid SfM Deformable SfM Large scale environments Unstructured image sets Extraction of complex primitives Generative models: lighting, BRDF, optics modeling, super-resolution, Feature descriptors, bridging SfM and recognition

5 The viewing rays for corresponding points do not generally intersect in space Two Strongly Related Main Steps Image matching or registration 4D reconstruction (or just 3D reconstruction) Time SfM Image registration 4D reconstruction An Ill-Posed Problem The images show the same scene under different shapes The scene structure is 4D: it varies in time The registration (image matching) problem is less constrained than in the rigid case The 4D reconstruction problem is ill-posed Which extra constraints / prior knowledge can we use? This will be the topic of the last four lectures! This Week Today: first steps to rigid Structure-from-Motion Modeling a single camera Modeling a pair of cameras Wednesday Least Squares optimization (linear and nonlinear) Application to camera model parameters recovery Robustification: how to deal with erroneous data Resources Includes resources and material (slides) Textbooks Multiple View Geometry in Computer Vision R. Hartley and A. Zisserman Cambridge University Press The Geometry of Multiple Images O. Faugeras and Q.-T. Luong MIT Press M. Pollefeys online tutorial Notation x, α Scalars in italics x, q Vectors in bold fonts P, A Matrices in sans-serif or calligraphic fonts P T, A 1, P Matrix transpose, inverse and pseudo-inverse I Identity matrix 0, 1 Vector of 0 and 1 q Affine coordinates (as opposed to homogeneous coordinates q) Homogeneous equality (equality up to scale) Cross-product of two 3-vectors R r, P r The Euclidean and projective spaces of dimension r vect(h) The row-wise vectorization of matrix H Note: vector are column vectors by default

6 Homogeneous Coordinates F We do not make a difference between physical entities and vectors or matrices, i.e. Q represents a 3D point and its coordinate vector F We use homogeneous coordinates: X Q Y Z, 1 where is equality up to scale F Inhomogeneous or affine coordinates are written Q: Q = X Y or Q Q Z 1 31 Euclidean Transformations Rotation + translation Represented by the rotation matrix R and the translation vector t In affine coordinates: Q 0 = R (3 3) Q + t In homogeneous coordinates: Q 0 R t Q 0 (1 3) 1 (4 4) This is similar in 2D 32 Orthonormal and Rotation Matrices F The orthonormal matrix group O(s) I U is an orthonormal matrix if and only if UU T =I(Note: U T U = UU T =I) I This implies det(u) = ±1 I These matrices form the Lie group O(s) with(s s) thesizeofu F The special orthonormal matrix group SO(s) I Contains the so-called rotation matrices I R is a rotation matrix if and only if RR T =Ianddet(R) =1 I SO(s) O(s) I SO(s) isalsoaliegroup 33 SVD Singular Value Decomposition F Theorem: any matrix A of size (r c) withr c can be decomposed as: A (r c) = U (r c) Σ (c c) V T (c c), where: I U is column orthonormal, i.e. U T U = I but in general UU T 6=I I Σ is diagonal and contains the singular values of A (in descending order) Notes: those are squares of the eigenvalues of A I V O(c) is an orthonormal matrix F The columns of U and V contain the left and right singular vectors of A F SVD is used in Homogeneous Linear Least Squares: min x,kxk 2 =1 kaxk2 Solution: do an SVD of A and take for x the last column of V 34 6