CV: 3D sensing and calibration

Transcription

1 CV: 3D sensing and calibration Coordinate system changes; perspective transformation; Stereo and structured light MSU CSE 803 1

2 roadmap using multiple cameras using structured light projector 3D transformations general perspective transformation justification of 3x4 camera model MSU CSE 803 2

3 Four Coordinate frames W: world, C,D: cameras, M: object model Need to relate all to each other. MSU CSE 803 3

4 Can we recognize? Is there some object M That can be placed in some location That will create the two images that are observed? Discover/compute what object and what pose MSU CSE 803 4

5 Need to relate frames to compute relate camera to world using rotations and translations project world point into real image using projection scale image point in real image plane to get pixel array coordinates MSU CSE 803 5

6 Stereo configuration 2 corresponding image points enable the intersection of 2 rays in W MSU CSE 803 6

7 Stereo computation MSU CSE 803 7

8 Math for stereo computations need to calibrate both cameras to W so that rays in x,y,z reference same space need to have corresponding points find point of closest approach of the two rays (rays are too far apart à point correspondence error or crude calibration) MSU CSE 803 8

9 Replace camera with projector Can calibrate a projector to W easily. Correspondence now means identifying marks. MSU CSE 803 9

10 Advantages/disadvantages of structured light MSU CSE

11 Grid projected on objects All grid intersects are integral MSU CSE

12 Computing surface normals Surface normals have been computed and then added to the image (augmented reality) MSU CSE

13 Relating coordinate frames need to relate camera frame to world need to rotate, translate, and scale coordinate systems need to project world points to the image plane all the above are modeled using 4x4 matrices and 1x4 points in homogeneous coordinates MSU CSE

14 Translation of 3D point P Point in frame 2 Point in 3D parameters Point in frame 1 MSU CSE

15 Scaling 3D point P MSU CSE

16 Rotation of P about the X-axis MSU CSE

17 Rotate P about the Y-axis MSU CSE

18 Rotate P about the Z-axis Looks same as 2D rotation omitting row, col 4 MSU CSE

19 Arbitrary rotation has orthonormal rows and columns MSU CSE

20 Equivalent rotation form every rotation or combination of rotations is equivalent to ONE rotation of angle theta about a single axis or rotation the axis of rotation and angle of rotation can be determined if three points are known before and after MSU CSE

21 Computing angle-axis form B O A* To rotate A to A*, the axis of rotation is OB that is perpendicular to OA and OA* Compute OB = OA x OA* (unit vectors) A ω Compute ω from OA o OA* (unit vectors) See frame software MSU CSE

22 Example: camera relative to world MSU CSE

23 exercise verify that the 3 x 3 rotation matrix is orthonormal by checking 6 dot products invert the 3 x 3 rotation matrix invert the 4 x 4 matrix verify that the new matrix transforms points correctly from C to W MSU CSE

24 Transformation calculus : notation accounts for transforms Destination frame W Origin frame M W M T Denotes transformation T transforms points from model frame to world frame. (Notation from John Craig, 1986) MSU CSE

25 Apply transformations to points Point in model coordinates W P = W M T M P Point in world coordinates Transformation from model to world coordinates (instance transformation) MSU CSE

26 Matrix algebra enables composition Let M and N be 4 x 4 matrices and let P be a 4 x 1 point M ( N P ) = ( M N ) P we can transform P using N and then transform that by M, or we can multiply matrices M and N and then apply that to point P matrix multiplication is associative (but not commutative) MSU CSE

27 Composing transformations cancel Projection parms. Parameters: rotation and translation A W T = A C T(π) C W T (ω) Two transformations are composed to get one transformation that maps points from the world frame to the frame A MSU CSE

28 Use of transformation calculus Alignment of two congruent triangles (ABC and DEF) via coordinate transformation; Fundamental operation MSU CSE

29 Align corresponding vertices at origin and align first side 0) starting configurations in 3D 1) Translate vertex to origin (do both) 2) Rotate DE to X-axis and AB to X-axis MSU CSE

30 Alignment: last steps 3) Rotate point F about X-axis into XY-plane. Rotate point C in a similar manner. Both triangles should now overlap in 3D MSU CSE

31 Here s what was done Every transformation we applied is invertible. So, we get the inverses to apply to one side. This is W M T MSU CSE

32 Align two congruent triangles in 3D space MSU CSE

33 What use is it? fundamental step in aligning a 3D model to observed 3D points might be lucky to have 3 corners of a rectangular box could have any matching 3D feature points once aligned, other 3D points could be transformed the same way See /user/stockman/software/rst/frames/frame.cpp and look for function align_triangles MSU CSE

34 Deriving form of the camera matrix We have already described what the camera matrix does and what form it has; we now go through the steps to justify it MSU CSE

35 Viewing model points M What s in front of the camera? MSU CSE

36 Math for the steps MSU CSE

37 Perspective transformation: camera origin at the center of projection This transformation uses same units in 3D as in the image plane MSU CSE

38 Perspective projection: camera origin in the real image plane MSU CSE

39 Rigid transformation for change of coordinate frame MSU CSE

40 Relate camera frame to world frame MSU CSE

41 Change scene units to pixels To get into XV or GIMP image coordinates! MSU CSE

42 Final result a 3x4 camera matrix maps 2D image pixels to 3D rays maps 3D rays to 2D image pixels obtain matrix via calibration (easy) obtain matrix via reasoning (hard) do camera calibration exercise MSU CSE