3D Computer Vision Camera Models

Transcription

1 3D Comuter Vision Camera Models Nassir Navab based on a course given at UNC by Marc Pollefeys & the book Multile View Geometry by Hartley & Zisserman July 2, 202 chair for comuter aided medical rocedures deartment of comuter science technische universität münchen

2 Outline of the Comuter Vision Lecture Projective Geometry and ransformations - ransformations with homogeneous coordinates - 2D rojective geometry - 3D rojective geometry - D rojective geometry Estimation - Direct and Iterative Estimation - DL Algorithm - Cost Functions - Robust Estimation (RANSAC) Algorithm Evaluation and Error Analysis - Statistics, Error Proagation Camera Geometry - Camera Models - Camera Calibration Advanced Estimation - Bundle Adjustment - Iterative Closest Point (ICP) Single-View Geometry wo-view Geometry - Eiolar Geometry / Fundamental Matri / Essential Matri - 3D Reconstruction of Cameras and Structure hree-view Geometry - rifocal ensor - Comutation of the rifocal ensor N-View Geometry - N-Linearities and Multile View ensors - N-View Comutational Methods - Factorization - Structure From Motion cam deartment of comuter science technische universität münchen July 2, 202 2

3 Outline Camera Model History of Camera (Camera Obscura) Intensity Images Geometric Parameters of a Finite camera Projective Camera Camera Center, Princial Plane, Ais Plane, Princial Point, Princial Ray Decomosition of Camera Matri Cameras at Infinity Other Camera Models (Pushbroom and Line Cameras) cam deartment of comuter science technische universität münchen July 2, 202 3

4 History :: Camara Obscura htt://en.wikiedia.org/wiki/camera_obscura cam deartment of comuter science technische universität münchen July 2, 202 4

5 History :: Camara Obscura Camera obscura ( Camera obscura) [LL. camera chamber + L. obscurus, obscura, dark.] (Ot.). An aaratus in which the images of eternal objects, formed by a conve lens or a concave mirror, are thrown on a aer or other white surface laced in the focus of the lens or mirror within a darkened chamber, or bo, so that the outlines may be traced. 2. (Photog.) An aaratus in which the image of an eternal object or objects is, by means of lenses, thrown uon a sensitized late or surface laced at the back of an etensible darkened bo or chamber variously modified; - commonly called simly the camera. Websters Dictionary, 93 Camera Obscura, Reinerus Gemma-Frisius, 544, ublished in De Radio Astronomica et Geometrica, 545 from Gernsheim, H., he Origins of Photograhy cam deartment of comuter science technische universität münchen July 2, 202 5

6 History :: Camera Obscura Camera Obscura, Athanasius Kircher, 646 from Gernsheim, H., he Origins of Photograhy Portable 'ent' Camera Obscura, Johannes Keler, 620 from Gernsheim, H., he Origins of Photograhy cam deartment of comuter science technische universität münchen July 2, 202 6

7 History :: Camera Obscura Refle Camera Obscura, Johannes Zahn, 685 ublished in Zahn, J., Oculus Artificialis, ( ) from Gernsheim, H., he Origins of Photograhy cam deartment of comuter science technische universität münchen July 2, 202 7

8 Outline Camera Model History of Camera (Camera Obscura) Intensity Images Geometric Parameters of a Finite camera Projective Camera Camera Center, Princial Plane, Ais Plane, Princial Point, Princial Ray Decomosition of Camera Matri Cameras at Infinity Other Camera Models (Pushbroom and Line Cameras) cam deartment of comuter science technische universität münchen July 2, 202 8

9 Intensity Image hotometric arameters light energy reaching the sensors after being reflected from objects (not discussed in this course) geometric arameters 2D osition in the image on which a 3D oint is rojected (deending on camera geometry, discussed here) cam deartment of comuter science technische universität münchen July 2, 202 9

10 Outline Camera Model History of Camera (Camera Obscura) Intensity Images Geometric Parameters of a Finite camera Projective Camera Camera Center, Princial Plane, Ais Plane, Princial Point, Princial Ray Decomosition of Camera Matri Cameras at Infinity Other Camera Models (Pushbroom and Line Cameras) cam deartment of comuter science technische universität münchen July 2, 202 0

11 Pinhole Camera Model (X, Y,Z) maing between 3D world and 2D image central rojection models are described by matrices with articular roerties (,y) cam deartment of comuter science technische universität münchen July 2, 202

12 cam deartment of comuter science technische universität münchen July 2, Z Y X 0 0 f 0 f Z fy fx z y Z Y X (,y,z) Y, Z,) (X, Homogeneous Coordinates

13 cam deartment of comuter science technische universität münchen July 2, Z Y X f f Z fy fx PX [ ] 0 I P diag(f,f,) Central Projection

14 Princial Point Offset rincial oint (erendicular intersection oint of rincial ais and image lane) ( X, Y,Z) (fx/z +,fy/z + y ) where (,y) are the coordinates of the rincial oint cam deartment of comuter science technische universität münchen July 2, 202 4

15 cam deartment of comuter science technische universität münchen July 2, Z Y X 0 0 f 0 f Z Z fy Z fx y [ ]X 0 K I f f K y is called camera calibration matri Princial Point Offset where

16 Camera Rotation and ranslation inhomogeneous coordinates ~ X cam where R ~ X ~ - C ~ ( X ) ~ X cam X Y Z y z cam cam cam C ~ reresents the oint in world coordinates X ~ R ( X~ -C ~ cam ) reresents the same oint in camera coordinates reresents the coordinates of the camera origin in the world coordinate frame cam deartment of comuter science technische universität münchen July 2, 202 6

17 Camera Rotation and ranslation homogeneous coordinates 4 X, X cam R K[ I 0] X cam X R RC ~ Y R RC ~ Xcam X 0 Z 0 rojection to image lane from camera coordinates [ I 0] cam K X KR[ I C ~ ]X PX rojection to image lane from world coordinates cam deartment of comuter science technische universität münchen July 2, 202 7

18 P Etrinsic and Intrinsic Parameters PX where K KR I f [ ] f C ~ y rojection matri of a general inhole camera with 9 DOF PX intrinsic camera arameters with 3 DOF R, C ~ etrinsic camera arameters with each 3 DOF (camera orientation and osition in world coordinates) cam deartment of comuter science technische universität münchen July 2, 202 8

19 Camera Rotation and ranslation no elicit camera center ~ ~ Xcam RX + t P [ t] K R [ ] K R RC ~ PX where t RC ~ [ from KR I C ~ ]X cam deartment of comuter science technische universität münchen July 2, 202 9

20 cam deartment of comuter science technische universität münchen July 2, y y y y K f f m m K α α CCD Cameras :: Non-Square Piel m y, m number of iels er unit distance 4 DOF [ ] C ~ KR I P 0 DOF

21 Skew Parameter s skew arameter K α s α y y 5 DOF P KR I [ ] C ~ finite rojective camera with DOF cam deartment of comuter science technische universität münchen July 2, 202 2

22 cam deartment of comuter science technische universität münchen July 2, [ ] C KR I RC R s P P y y ~ 0 ~ α α X non-singular Finite Projective Camera :: Summary rojection matri DOF (5+3+3)

23 Finite Projective Camera :: Decomosition of P M P P KR M I [ ] [ C ~ ] KR I C ~ non-singular 33 matri (8 DOF) decomose rojection matri P in K,R,C P [ M ] [ K, R] RQ( M) M 4 4 RQ matri decomosition C ~ cam deartment of comuter science technische universität münchen July 2,

24 Finite Projective Camera :: Summary P M I [ ] [ C ~ ] KR I C ~ where P R 3 4 camera matri P are identical with the set of homogeneous 34 matri for which the left hand 33 submatri is non singular {finite cameras}{p det M 0} {P rank(m)3} If rank(p)3, but rank(m)<3, then camera at infinity if rank(p)<3 the matri maing will be a line or a oint and not a lane (not a 2D image) cam deartment of comuter science technische universität münchen July 2,

25 Outline Camera Model History of Camera (Camera Obscura) Intensity Images Geometric Parameters of a Finite camera Projective Camera Camera Center, Princial Plane, Ais Plane, Princial Point, Princial Ray Decomosition of Camera Matri Cameras at Infinity Other Camera Models (Pushbroom and Line Cameras) cam deartment of comuter science technische universität münchen July 2,

26 Outline Camera Model History of Camera (Camera Obscura) Intensity Images Geometric Parameters of a Finite camera Projective Camera Camera Center, Princial Plane, Ais Plane, Princial Point, Princial Ray Decomosition of Camera Matri Cameras at Infinity Other Camera Models (Pushbroom and Line Cameras) cam deartment of comuter science technische universität münchen July 2,

27 Camera Anatomy camera center column vectors rincial lane ais lane rincial oint rincial ray cam deartment of comuter science technische universität münchen July 2,

28 Camera Center PC 0 P has a D null-sace we will roof that the 4-vector C is the camera center X ( λ) λa + ( λ)c PX λpa + ( λ) PC λpa Finite cameras: Infinite cameras: oints on a line through A and C ~ C M C d C, Md 0 0 since PC 0 all 3D oints on the line are maed on the same 2D image oint, and thus the line is a ray through the camera center cam deartment of comuter science technische universität münchen July 2,

29 Column Vectors i R 3, i,...,4 column vectors are the image oints, which roject the ais directions (X,Y,Z) and the origin eamle for the image of the y-ais (vanishing t) [ ] [ ][ 0 0 0] is the image of the world origin cam deartment of comuter science technische universität münchen July 2,

30 Row Vectors i R 4, i,...,3 reresent geometrically articular world lanes P [ ] row vectors column vectors cam deartment of comuter science technische universität münchen July 2,

31 cam deartment of comuter science technische universität münchen July 2, Z Y X w Row Vectors of the Projection Matri P is defined by the camera center and the line 0 on the image P 2 is defined by the camera center and the line y0 on the image C X Eamle P 2 resectively for P

32 cam deartment of comuter science technische universität münchen July 2, Z Y X y Princial Plane 0 C 3 lane through camera center and arallel to the image lane oints X are imaged on the line at infinity if X is on the rincile lane 0 3 X esecially

33 Princile Point line through camera center and erendicular to rincile lane is the rincile ais intersection of the rincile ais with the image lane is the rincile oint 3 0 P M Pˆ m 3 Pˆ 3 ( 0) 3 32 rincile oint P M Pˆ m 3 normal direction to rincial lane where and P 3 m [ M ] 4 third row of M cam deartment of comuter science technische universität münchen July 2,

34 Princile Ais Vector ambiguity that rincile ais oints towards the front of the camera (ositive direction) m 3 or m 3 P X cam cam [ 0] X cam K I v det ( K) k 3 ( 0,0, ) towards the front of the camera P k cam P cam v k 4 v direction unaffected by scaling [ ] [ M ] P kkr I C ~ since det( R) > 0 4 ( ) 3 v det M m cam deartment of comuter science technische universität münchen July 2,

35 Forward Projection mas a oint in sace on the image lane 3 2 P P PX Points at infinity D ( d,0) [ M ] D Md PD 4 only M effects the rojection of oints at infinity (vanishing oints) cam deartment of comuter science technische universität münchen July 2,

36 Back-rojection to Rays oints on the reconstructed ray PC 0 camera center C + X P P + P ( PP ) (seudo-inverse) + PP I ray is the line formed by those two oints ( λ) P + λc X + D ( ) X µ ( ) ) - M,0 - - M - M 4 M µ + 0 D intersection of the ray with the lane at infinity ( µ - ) cam deartment of comuter science technische universität münchen July 2, C - 4

37 Deth of Points X ( X, Y, Z,) C (C ~,) w P 3 X det M > 0; P m 3 ( X C) m ( ) 3 (PC0) 3 X ~ C ~ (dot roduct) If, then m 3 unit vector ointing in ositive ais direction Suose ( X,Y,Z, ) w(, y,) P. hen deth sign(detm) w ( X;P) 3 m cam deartment of comuter science technische universität münchen July 2,

38 Deth of Points: eamles deth sign(detm) w ( X; P) 3 P ( X,Y,Z, ) w(, y,) m PX P [ I 0] deth ( X ; P) Z P PX R I [ ] C ~ deth(x; P) R ( X ~ 3 C ~ ) cam deartment of comuter science technische universität münchen July 2,

39 Outline Camera Model History of Camera (Camera Obscura) Intensity Images Geometric Parameters of a Finite camera Projective Camera Camera Center, Princial Plane, Ais Plane, Princial Point, Princial Ray Decomosition of Camera Matri Cameras at Infinity Other Camera Models (Pushbroom and Line Cameras) cam deartment of comuter science technische universität münchen July 2,

40 Camera Matri Decomosition Finding the camera center C PC 0 numerically: find right null-sace by SVD of P P 34 U 34 D 44 V 44 P 34 V 44 U 34 D 44 PV 4 0 Algebraically: X det( [ ] 2,3, 4 ) Y det( [ ],3, 4 ) Z det( [,, ]) det( [,, ]) 2 4 where C ( X, Y, Z, ) 2 3 cam deartment of comuter science technische universität münchen July 2,

41 Camera Matri Decomosition Finding the camera center C PC 0 Any lane π going through C will be a linear combination of the three lanes defined by the rows of P. herefore: det ([ π, P ]) 0 π det( P 234 ) π det( P where C ( X, Y, Z, ) 2 34 ) + π det( P 3 24 X det( [ ] 2,3, 4 ) Y det( [ ],3, 4 ) Z det( [,, ]) det( [,, ]) ) π det( P 4 23 ) cam deartment of comuter science technische universität münchen July 2, 202 4

42 Camera Matri Decomosition Finding the camera orientation and internal arameters P [ M ] K[ R ] -MC ~ Decomose RC ~ M KR α s y0 K 0 α y using RQ decomosition ( Q R ) - R - - Q Ambiguity removed by enforcing ositive diagonal entries cam deartment of comuter science technische universität münchen July 2,

43 When is Skew Non-zero? K α s α y y γ arctan(/s) for CCD/CMOS, always s0 Image from image, s 0 ossible (non coinciding rincial ais) resulting camera: HP where H is a 33 homograhy cam deartment of comuter science technische universität münchen July 2,

44 Euclidean vs. Projective Saces general rojective interretation P [ 3 3 homograhy] [ 4 4 homograhy] Meaningful decomosition in K,R,t requires Euclidean image and sace Camera center is still valid in rojective sace Princial lane requires affine image and sace Princial ray requires affine image and Euclidean sace cam deartment of comuter science technische universität münchen July 2,

45 Literature on Camera Models Chater 6 in R. Hartley and A. Zisserman, Multile View Geometry, 2 nd edition, Cambridge University Press, Chater 3 in O. Faugeras, hree-dimensional Comuter Vision, MI Press, 993. Chater 2 in E. rucco and A. Verri, Introductory echniques for 3-D Comuter Vision, Prentice Hall, 998. H. Gernsheim, he Origins of Photograhy, hames and Hudson, 982. A. Shashua. Geometry and Photometry in 3D Visual Recognition, Ph.D. hesis, MI, Nov AIR-40. cam deartment of comuter science technische universität münchen July 2,