CV: 3D sensing and calibration Coordinate system changes; perspective transformation; Stereo and structured light MSU CSE 803 1
roadmap using multiple cameras using structured light projector 3D transformations general perspective transformation justification of 3x4 camera model MSU CSE 803 2
Four Coordinate frames W: world, C,D: cameras, M: object model Need to relate all to each other. MSU CSE 803 3
Can we recognize? Is there some object M That can be placed in some location That will create the two images that are observed? Discover/compute what object and what pose MSU CSE 803 4
Need to relate frames to compute relate camera to world using rotations and translations project world point into real image using projection scale image point in real image plane to get pixel array coordinates MSU CSE 803 5
Stereo configuration 2 corresponding image points enable the intersection of 2 rays in W MSU CSE 803 6
Stereo computation MSU CSE 803 7
Math for stereo computations need to calibrate both cameras to W so that rays in x,y,z reference same space need to have corresponding points find point of closest approach of the two rays (rays are too far apart à point correspondence error or crude calibration) MSU CSE 803 8
Replace camera with projector Can calibrate a projector to W easily. Correspondence now means identifying marks. MSU CSE 803 9
Advantages/disadvantages of structured light MSU CSE 803 10
Grid projected on objects All grid intersects are integral MSU CSE 803 11
Computing surface normals Surface normals have been computed and then added to the image (augmented reality) MSU CSE 803 12
Relating coordinate frames need to relate camera frame to world need to rotate, translate, and scale coordinate systems need to project world points to the image plane all the above are modeled using 4x4 matrices and 1x4 points in homogeneous coordinates MSU CSE 803 13
Translation of 3D point P Point in frame 2 Point in 3D parameters Point in frame 1 MSU CSE 803 14
Scaling 3D point P MSU CSE 803 15
Rotation of P about the X-axis MSU CSE 803 16
Rotate P about the Y-axis MSU CSE 803 17
Rotate P about the Z-axis Looks same as 2D rotation omitting row, col 4 MSU CSE 803 18
Arbitrary rotation has orthonormal rows and columns MSU CSE 803 19
Equivalent rotation form every rotation or combination of rotations is equivalent to ONE rotation of angle theta about a single axis or rotation the axis of rotation and angle of rotation can be determined if three points are known before and after MSU CSE 803 20
Computing angle-axis form B O A* To rotate A to A*, the axis of rotation is OB that is perpendicular to OA and OA* Compute OB = OA x OA* (unit vectors) A ω Compute ω from OA o OA* (unit vectors) See frame software MSU CSE 803 21
Example: camera relative to world MSU CSE 803 22
exercise verify that the 3 x 3 rotation matrix is orthonormal by checking 6 dot products invert the 3 x 3 rotation matrix invert the 4 x 4 matrix verify that the new matrix transforms points correctly from C to W MSU CSE 803 23
Transformation calculus : notation accounts for transforms Destination frame W Origin frame M W M T Denotes transformation T transforms points from model frame to world frame. (Notation from John Craig, 1986) MSU CSE 803 24
Apply transformations to points Point in model coordinates W P = W M T M P Point in world coordinates Transformation from model to world coordinates (instance transformation) MSU CSE 803 25
Matrix algebra enables composition Let M and N be 4 x 4 matrices and let P be a 4 x 1 point M ( N P ) = ( M N ) P we can transform P using N and then transform that by M, or we can multiply matrices M and N and then apply that to point P matrix multiplication is associative (but not commutative) MSU CSE 803 26
Composing transformations cancel Projection parms. Parameters: rotation and translation A W T = A C T(π) C W T (ω) Two transformations are composed to get one transformation that maps points from the world frame to the frame A MSU CSE 803 27
Use of transformation calculus Alignment of two congruent triangles (ABC and DEF) via coordinate transformation; Fundamental operation MSU CSE 803 28
Align corresponding vertices at origin and align first side 0) starting configurations in 3D 1) Translate vertex to origin (do both) 2) Rotate DE to X-axis and AB to X-axis MSU CSE 803 29
Alignment: last steps 3) Rotate point F about X-axis into XY-plane. Rotate point C in a similar manner. Both triangles should now overlap in 3D MSU CSE 803 30
Here s what was done Every transformation we applied is invertible. So, we get the inverses to apply to one side. This is W M T MSU CSE 803 31
Align two congruent triangles in 3D space MSU CSE 803 32
What use is it? fundamental step in aligning a 3D model to observed 3D points might be lucky to have 3 corners of a rectangular box could have any matching 3D feature points once aligned, other 3D points could be transformed the same way See /user/stockman/software/rst/frames/frame.cpp and look for function align_triangles MSU CSE 803 33
Deriving form of the camera matrix We have already described what the camera matrix does and what form it has; we now go through the steps to justify it MSU CSE 803 34
Viewing model points M What s in front of the camera? MSU CSE 803 35
Math for the steps MSU CSE 803 36
Perspective transformation: camera origin at the center of projection This transformation uses same units in 3D as in the image plane MSU CSE 803 37
Perspective projection: camera origin in the real image plane MSU CSE 803 38
Rigid transformation for change of coordinate frame MSU CSE 803 39
Relate camera frame to world frame MSU CSE 803 40
Change scene units to pixels To get into XV or GIMP image coordinates! MSU CSE 803 41
Final result a 3x4 camera matrix maps 2D image pixels to 3D rays maps 3D rays to 2D image pixels obtain matrix via calibration (easy) obtain matrix via reasoning (hard) do camera calibration exercise MSU CSE 803 42