ECE Digital Image Processing and Introduction to Computer Vision. Outline

Transcription

1 ECE Digital Image Processing and Introduction to Computer Vision Depart. of ECE, NC State University Instructor: Tianfu (Matt) Wu Spring Recap Outline 2. Modeling Projection and Projection Geometry Three Geometric Problems 1

2 1. Recap, no a computer can see bigger entities From pixels, to neighbors, path, connected components and to regions. 1. Recap, definitions Neighbors of a pixel are defined.r.t. Coordinates and Distance measures 4- / D- / 8-neighbors ( artifacts due to sampling, i.e., digitalized coordinates) Adjacency is defined.r.t. The type of neighbors specified and A set of pixel values specified Ho to generalize this setting? 4- / D- / 8- / m-adjacency Path: a sequence of pixels (in the hole lattice by default) ith successive pixels being adjacent 4- / D- / 8- / m-path Connectivity is defined.r.t. A subset of pixels specified (hich could be the hole lattice) and A path is entirely contained in the subset. Connected components in the subset The subset is a connected set / region if only one connected component exists Image as Graph 2

3 2. Modeling Projection What about 3D relationships? Ho to infer them based on 2D measures? Ho to exploit them in 2D image understanding? Who is taller? Which is closer? Source: D. Forsyth Multivie Geometry A classic book on multi-vie geometry. We only cover some very basic methods here. 3

4 Let s see hat a pinhole camera did in the imaging process. Real camera image is inverted Instead model impossible but more convenient virtual image Source: Simon J.D. Prince Basic principle knon to Mozi ( BCE), Aristotle ( BCE) Draing aid for artists: described by Leonardo da Vinci ( ) Gemma Frisius, 1558 Source: A. Efros 4

5 What is lost? Length Projective Geometry Who is taller? Which is closer? Source: D. Forsyth Length is not preserved A C B Source: D. Forsyth 5

6 What is lost? Length Angles Projective Geometry Parallel? Perpendicular? Source: D. Forsyth Projective Geometry What is preserved? Straight lines are still straight Source: D. Forsyth 6

7 Projection can be tricky Source: Steve Seitz Projection can be tricky Source: Steve Seitz 7

8 Accidental Pinhole Source: A. Torralba and W. Freeman, Accidental Pinhole and Pinspeck Cameras, CVPR 2012 Projective projection: orld coordinates (unit, e.g. m or mm) to image coordinates (pixel) Image coordinate system World coordinate system Assume camera is centered at the origin of the orld coordinate system ith the optical axis exactly aligned ith the!- axis. Ho to compute the transformation? Source: Simon J.D. Prince 8

9 1/19/17 We kne ho to do it. Albrecht Dürer, Mechanical creation of a perspective image, 1525 Let s compute it. f Perspective projective: Compute the relationships beteen ", $ &'( ), *,! hen all are measured in the orld coordinate system. By similar triangles: x= f u y= f v Here, x and y use orld unit (e.g., m or mm) 9

10 Matrix form using homogenous coordinates apple x y <=> x y u 4v 5 <=> u v 7 5 a point in the image plane is a ray in projective space x = f u y = f v Nonlinear f u f 0 0 u 4f v5 = 40 f 05 4v Linear Source: Simon J.D. Prince Matrix multiplication Sensor s density and unit conversion factor the density of the receptors (hich could be different along x and y dims) the scaling from real-orld unit (e.g., mm) to pixel x = y = x f u y f v = x u = y v No, (y, x) presents pixel location x u x 0 0 u 4 y v5 = 4 0 y 05 4v

11 Modeling ske beteen x- and y-axis y = y v x = x u + y = y v v 2 x u + 4 y v x = x u x = y tan = y tan v = v v x 0 u 5 = 4 0 y 05 4v Modeling offset Image coordinate system (pixel) Translation: from the principal point to the left-top corner in the image plane hen the principal point is not the exact center of the image plane. x = x u + y = y v v + y + x x u + v + x x x u 4 y v + y 5 = 4 0 y y 5 4v

12 Intrinsic matrix Image coordinate system World coordinate system Assume camera is centered at the origin of the orld coordinate system ith the optical axis exactly aligned ith the!-axis. Pixel location apple x y 2 4 Intrinsic matrix x x u 0 y y 5 4v World coordinate Intrinsic matrix Image coordinate system World coordinate system 2 4 Assume camera is centered at the origin of the orld coordinate system ith the optical axis exactly aligned ith the!-axis x 1 0 x 1 / y 0 x y y 5 = y y x Denoted by + 2D Translation 2D Shear 2D Scaling 12

13 Extrinsic matrix Image coordinate system World coordinate system Camera is NOT exactly centered at the origin of the orld coordinate system ith the optical axis exactly aligned ith the!-axis. We express the orld points = ), *,!. in the coordinate system of the camera before they are passed through the projection model, using the coordinate transformation: 3D Rotation, denoted by 0 3D Translation, denoted by Τ Extrinsic matrix Image coordinate system World coordinate system Camera is NOT exactly centered at the origin of the orld coordinate system ith the optical axis exactly aligned ith the!-axis. 2 4 u 0 v u u! 11! 12! 13 x 5 = 4! 21! 22! 23 y 5 6v 7 45 =[R T] 6v 7 45! 31! 32! 33 z

14 Full model Image coordinate system World coordinate system Camera is NOT exactly centered at the origin of the orld coordinate system ith the optical axis exactly aligned ith the!-axis u x 4y5 = K 3 3 [R T] 3 4 6v Full model: forard projection Source: Robert Collins, different notations used 14

15 1/19/17 Real Cameras Use lenses, not pinhole Source: Leo Wee Kheng The system of lenses collects light from a larger area and re-focuses it on the image plane. In practice, this leads to a number of deviations from the pinhole model. For example, some parts of the image may be out of focus, hich essentially means that the assumption that a point in the orld maps to a single point in the image is no longer valid. Real Cameras E.g., radial distortion Source: Simon J.D. Prince 15

16 Real Cameras Pinhole model as an approximated model observed data = true data + noise/distortion Image pixel coordinates = pinhole(orld coordinates; +, 0, 1) + noise For simplicity, e model noise ith Gaussian distribution 2345 " 6! 6, +, 0, 1) = 8439(;<'h4>?! 6, +, 0, 1 ; A B C) Three Geometric Problems 1. Learning extrinsic parameters (exterior orientation), 3D rotation matrix 0 D D and 3D translation 1 D F 2. Learning intrinsic parameters (calibration), + D D 3. Inferring 3D points (triangulation / reconstruction) 16

17 Problem 1. Learning extrinsic parameters Given orld points! 6 on a knon object (blue lines), their positions " 6 in the image (circles on image plane), and knon intrinsic parameters +, find the rotation 0 and translation 1 relating the camera and the object. [0 F, 1 F ] [0 B, 1 B ] Maximum log-likelihood estimateion(mle) 0, 1 = arg max O,P Q log 2345 " 6! 6, +, 0, 1) 6 Source: Simon J.D. Prince Problem 2. Learning intrinsic parameters Given orld points! 6 on a knon object (blue lines), their positions " 6 in the image (circles on image plane), find intrinsic parameters +. To that end, e also need to estimate the rotation 0 and translation 1. + F + B [0 F, 1 F ] [0 B, 1 B ] Maximum log-likelihood estimateion(mle) + = arg max [max Q log 2345 " 6! 6, +, 0, 1) ] U O,P 6 Source: Simon J.D. Prince 17

18 Problem 3. Inferring 3D points Given V calibrated cameras in knon positions (i.e., cameras ith knon +, 0, 1) vieing the same 3D point! and knoing the corresponding 2D projections " W, X = 1,, V in the V images, establish the 3D position of the point in the orld. Maximum log-likelihood estimateion(mle)! = arg max [Q log 2345 " W!, + W, 0 W, 1 W )] [ W Source: Simon J.D. Prince Can e anser these questions no? Who is taller? Ho high is the camera? Which is closer? What is the camera rotation? What is the focal length of the camera? Single vie metrology: uncalibrated single vie reconstruction (A. Criminisi, I. Reid, and A. Zisserman, IJCV, 2000). Source: D. Forsyth 18

19 Can e anser these questions no? Who is taller? Ho high is the camera? Which is closer? What is the camera rotation? What is the focal length of the camera? TBD: We ill discuss ho to solve the three problems stated above and single vie metrology in the next lecture. Vanishing points and lines Parallel lines in the orld intersect in the image at a vanishing point Source: D. Forsyth 19