Camera Calibration. COS 429 Princeton University

Transcription

1 Camera Calibration COS 429 Princeton University

2 Point Correspondences What can you figure out from point correspondences? Noah Snavely

3 Point Correspondences X 1 X 4 X 3 X 2 X 5 X 6 X 7 p 1,1 p 1,2 p 1,3 Camera 1 Camera 3 Camera 2 Noah Snavely

4 Camera calibration Suppose we know 3D point positions and correspondences to pixels in every image Can we compute the camera parameters? Noah Snavely

5 Camera triangulation Suppose we know the camera parameters and correspondences between points and pixels in every image? Noah Snavely

6 Camera calibration & triangulation Need to understand how 3D points project into images Noah Snavely

7 Outline Camera model Camera calibration Camera triangulation Structure from motion

8 Outline Camera model Camera calibration Camera triangulation Structure from motion

9 2D Image Basics Origin at center, for now Y axis is up Will often write (u, v) for image coordinates v u

10 3D Geometry Basics 3D points = column vectors Transformations = pre-multiplied matrices

11 3D Geometry Basics Right-handed vs. left-handed coordinate systems x x y z z y

12 Rotation Rotation about the z axis Rotation about x, y axes similar (cyclically permute x, y, z)

13 Arbitrary Rotation Any rotation is a composition of rotations about x, y, and z Composition of transformations = matrix multiplication (watch the order!) Result: orthonormal matrix Each row, column has unit length Dot product of rows or columns = 0 Inverse of matrix = transpose

14 Arbitrary Rotation Rotate around x, y, then z: Don t do this! It s probably buggy! Compute simple matrices and multiply them

15 Scale in x, y, z: Scale

16 Shear Shear parallel to xy plane:

17 Translation Can translation be represented by multiplying by a 33 matrix? No. Proof?

18 Homogeneous Coordinates Add a fourth dimension to each 3D point: To get real (3D) coordinates, divide by w:

19 Translation in Homogeneous Coordinates After divide by w, this is just a translation by (t x, t y, t z )

20 Perspective Projection Map points onto view plane along projectors emanating from center of projection (COP) Center of Projection View Plane Angel Figure 5.9

21 Perspective Projection Compute 2D coordinates from 3D coordinates with similar triangles -z (x,y,z) -z f y (0,0,0) z What are the coordinates of the point resulting from projection of (x,y,z) onto the view plane? View Plane -y

22 Perspective Projection Compute 2D coordinates from 3D coordinates with similar triangles -z (x c,y c,z c ) -z f y (0,0,0) z (x c f/z c, y c f/z c ) View Plane -y

23 Perspective Projection Matrix 4x4 matrix representation: 1 / / s s c c s c c s w f z z f y y z f x x f z w z z y y x x c c c c / ' ' ' ' ' / ' ' / ' ' / ' w z z w y y w x x s s s 1 0 1/ c c c s s s s z y x f w z y x

24 Perspective Projection Properties Points points Lines lines (collinearity preserved) Distances and angles are not preserved Many points along same ray map to same point in image Degenerate cases: Line through focal point projects to a point. Plane through focal point projects to line Plane perpendicular to image plane projects to part of the image. Slide by Kristen Grauman

25 Perspective Effects Far away objects appear smaller Forsyth and Ponce

26 Perspective Effects

27 Perspective Effects Parallel lines in the scene intersect at a point after projection onto image plane Vanishing point Image plane (virtual) Center of projection Scene Slide by Kristen Grauman

28 Perspective Effects Vanishing Points Any set of parallel lines on a plane define a vanishing point The union of all of these vanishing points is the horizon line also called vanishing line Different planes (can) define different vanishing lines v 1 l v 2 Noah Snavely

29 A Camera Model Translate to image center Then unwarp radial distortion Scale to pixel size Perspective projection Camera orientation Camera location 3D point (homogeneous coords) Then perform homogeneous divide, and get (u,v) coords

30 A Camera Model Normalized device coordinates Camera coordinates Image coordinates Pixel coordinates World coordinates

31 A Camera Model Intrinsics Extrinsics

32 Camera parameters Extrinsic: how to map world coordinates to camera coordinates (translation and rotation) Intrinsic: how to map camera coordinates to image coordinates (projection, translation, scale)

33 Camera parameters Extrinsic: how to map world coordinates to camera coordinates (translation and rotation) P c R( P T) w Camera coordinates World coordinates

34 Camera parameters Intrinsic: how to map camera coordinates to image coordinates (projection, translation, scale) X x ( f o s x) x Z Y y ( f o ) y s y Z Coordinates of projected point in image coordinates Coordinates of image point in camera coordinates Coordinates of image center in pixel units Effective size of a pixel (mm)

35 Camera parameters Substituting previous eqns describing intrinsic and extrinsic parameters, can relate pixels coordinates to world points: ) ( ) ( ) ( 3 1 T P R T P R w w x x im f s o x ) ( ) ( ) ( 3 2 T P R T P R w w y y im f s o y R i = Row i of rotation matrix

36 Projection matrix This can be rewritten as a matrix product using homogeneous coordinates: / 0 0 / int y y x x o s f o s f M T R T R T R M r r r r r r r r r ext 1 int w w w ext im im Z Y X w wy wx M M where:

37 Radial Distortion Finally, would like to include transformation to unwarp radial distortion:

38 Radial Distortion Radial distortion cannot be represented by matrix (c u, c v ) is image center, u * img= u img c u, v * img= v img c v, k is first-order radial distortion coefficient

39 A Camera Model Translate to image center Then unwarp radial distortion Scale to pixel size Perspective projection Camera orientation Camera location 3D point (homogeneous coords) Then perform homogeneous divide, and get (u,v) coords

40 Outline Camera model Camera calibration Camera triangulation Structure from motion

41 Camera Calibration Compute intrinsic and extrinsic parameters using observed camera data Main idea Place calibration object with known geometry in the scene Get correspondences Solve for mapping from scene to image: estimate M=M int M ext

42 Camera Calibration Chromaglyphs Courtesy of Bruce Culbertson, HP Labs

43 Camera Calibration Input: 3D 2D correspondences General perspective camera model (no radial distortion, for now) Output: Camera model parameters

44 Direct linear calibration

45 Direct linear calibration Can solve for m ij by linear least squares

46 Advantage: Direct linear calibration Very simple to formulate and solve Disadvantages: Doesn t tell you the camera parameters Doesn t model radial distortion Hard to impose constraints (e.g., known f) Doesn t minimize the right error function Why?

47 Nonlinear Camera Calibration Incorporating additional constraints into camera model No shear, no scale (rigid-body motion) Square pixels Radial distortion etc. These impose nonlinear constraints on camera parameters

48 Application: Sea of Images Aliaga

49 Application: Sea of Images Off-line: Camera calibration Feature correspondence Real-time: Image warping Image composition Large-scale Dense Capture Interactive Walkthroughs Aliaga

50 Sea of Images Capture Hemispherical FOV camera focal point mirror m image plane i p camera Paraboloidal Catadioptric Camera [Nayar97] Aliaga

51 Sea of Images Capture Capture lots of images with hemispherical FOV camera driven on cart Aliaga

52 Sea of Images Calibration Calibrate camera based on positions of fiducials Aliaga

53 Sea of Images Calibration Result is a sea of images with known camera viewpoints spaced a few inches apart Aliaga

54 Sea of Images Correspondence Search for feature correspondences in nearby images A B A B C Captured image viewpoints C Aliaga

55 Sea of Images Rendering Create image from new viewpoint by warping and compositing three nearest images using triangles of feature correspondences A New viewpoint B A B C Captured image viewpoints C Aliaga

56 Sea of Images Rendering Neat data management based on hierarchical encoding to enable interactive walkthrough floor plan New viewpoint Captured image viewpoints Aliaga

57 Bell Labs Museum 900 square ft 9832 images 2.2 inch spacing Princeton Library 120 square ft 1947 images 1.6 inches Personal Office 30 square feet 3475 images 0.7 inches Sea of Images Results Aliaga

58 Sea of Images Results Aliaga

59 Sea of Images Results Aliaga

60 Sea of Images Results Aliaga

61 Outline Camera model Camera calibration Camera triangulation Structure from motion

62 Camera calibration summary Suppose we know 3D point positions and correspondences to pixels in every image We can compute the camera parameters Noah Snavely

63 Camera triangulation Suppose we know the camera parameters and correspondences between points and pixels in every image? Noah Snavely

64 Camera calibration & triangulation Suppose no camera parameters or point positions And have matches between these points and two images Can we compute them both together? Noah Snavely

65 Outline Camera model Camera calibration Camera triangulation Structure from motion

66 Structure from motion Given many images a) figure out where they were all taken from? b) build a 3D model of the scene? Noah Snavely

67 Structure from motion Feature detection Feature description Feature matching Feature correspondence Camera calibration and triangulation Noah Snavely

68 Feature detection Detect features using SIFT [Lowe, IJCV 2004] Noah Snavely

69 Feature detection Detect features using SIFT [Lowe, IJCV 2004] Noah Snavely

70 Feature matching Match features between each pair of images Noah Snavely

71 Feature matching Refine matching using RANSAC to correspondences between each pair Noah Snavely

72 Image connectivity graph (graph layout produced using the Graphviz toolkit: Noah Snavely

73 Structure from motion X 1 X 4 X 5 X 6 X 2 X 3 X 7 minimize g(r, T, X) non-linear least squares p 1,1 p 1,2 p 1,3 Camera 1 R 1,t 1 R 2,t 2 Camera 2 Camera 3 R 3,t 3 Noah Snavely

74 Structure from motion Minimize sum of squared reprojection errors: predicted image location indicator variable: is point i visible in image j? Minimizing this function is called bundle adjustment Optimized using non-linear least squares, e.g. Levenberg-Marquardt observed image location Noah Snavely

75 Problem size What are the variables? How many variables per camera? How many variables per point? Trevi Fountain collection 466 input photos + > 100,000 3D points = very large optimization problem Noah Snavely

76 Structure from motion Noah Snavely

77 Incremental structure from motion Noah Snavely

78 Incremental structure from motion Noah Snavely

79 Incremental structure from motion Noah Snavely

80 Incremental structure from motion Noah Snavely

81 Incremental structure from motion Noah Snavely

82 Structure from Motion Results For photo sets from the Internet, 20% to 75% of the photos were registered Most unregistered photos belonged to different connected components Running time: < 1 hour for 80 photos > 1 week for 2600 photo Noah Snavely

83 Structure from Motion Results Noah Snavely

84 Photo Tourism: Exploring Photo Collections in 3D Noah Snavely Steven M. Seitz University of Washington Richard Szeliski Microsoft Research Noah Snavely

85 15,464 37,383 76,389 Noah Snavely

86 Reproduced with permission of Yahoo! Inc by Yahoo! Inc. YAHOO! and the YAHOO! logo are trademarks of Yahoo! Inc. Noah Snavely

87 Example: Prague Old Town Square Noah Snavely

88 Rendering transitions Noah Snavely

89 Annotations Noah Snavely

90 Annotations Noah Snavely

91 Summary Camera model Camera calibration Camera triangulation Structure from motion Noah Snavely