Perspective Projection Transformation Where does a point of a scene appear in an image?? p p Transformation in 3 steps:. scene coordinates => camera coordinates. projection of camera coordinates into image plane 3. camera coordinates => image coordinates Perspective projection equations are essential for Computer Graphics. For Image Understanding we will need the inverse: What are possible scene coordinates of a point visible in the image? This will follow later. Perspective Projection in Independent Coordinate Sstems It is often useful to describe real-world points, camera geometr and image points in separate coordinate sstems. The formal description of projection involves transformations between these coordinate sstems. camera coordinates image coordinates optical ais scene point v = optical center v image point v p = p p p v p = p p p v p = p p scene (world) coordinates
3D Coordinate Transformation () The new coordinate sstem is specified b a translation and rotation with respect to the old coordinate sstem: v = R (v - v ) R ma be decomposed into 3 rotations about the coordinate aes: R = R R R v is displacement vector R is rotation matri R = Note that these matrices describe coo transforms for positive rotations of the coo sstem. cos α sin α sin α cos α If rotations are performed in the above order: ) γ = rotation angle about -ais ) β = rotation angle about (new) -ais 3) α = rotation angle about (new) -ais ("tilt angle", "pan angle", and "nick angle" for the camera coordinate assignment shown before) R = R = cos β sin β cos γ sin γ sin γ cos γ - sin β cos β 3 3D Coordinate Transformation () B multipling the 3 matrices R, R and R, one gets R = cos β cos γ cos β sin γ - sin β sin α sin β cos γ cos α sin γ sin α sin β sin γ + cos α cos γ sin α cos β cos α sin β cos γ + sin α sin γ cos α sin β sin γ sin α cos γ cos α cos β For formula manipulations, one tries to avoid the trigonometric functions and takes R = r r r 3 r r r 3 r 3 r 3 r 33 Note that the coefficients of R are constrained: A rotation matri is orthonormal: R R T = I (unit matri) 4
Eample for Coordinate Transformation v camera coo sstem: displacement b v rotation b pan angle β = -3 rotation b nick angle α = 45 v = R (v - v ) with R = R R R = R = 3 3 5 Perspective Projection Geometr Projective geometr relates the coordinates of a point in a scene to the coordinates of its projection onto an image plane. Perspective projection is an adequate model for most cameras. optical center focal distance f V p = p p p p p p = f image point scene point v = = optical ais image plane Projection equations: p = f p = f 6 3
Perspective and Orthographic Projection Within the camera coordinate sstem the perspective projection of a scene point onto the image plane is described b f p = f p = p = f (f = focal distance) nonlinear transformation loss of information If all objects are far awa (large ), f/ is approimatel constant => orthographic projection p = s p = s (s = scaling factor) Orthographic projection can be viewed as projection with parallel ras + scaling 7 From Camera Coordinates to Image Coordinates Transform ma be necessar because - optical ais ma not penetrate image plane at origin of desired coordinate sstem - transition to discrete coordinates ma require scaling p = ( p - p ) a p = ( p - p ) b a, b scaling parameters p, p origin of image coordinate sstem Eample: c d c d Image boundaries in camera coordinates: ma = c min = c ma = d min = d Discrete image coordinates: =.. 5 =.. 575 Transformation parameters: p = c p = d a = 5 / (c - c) b = 576 / (d - d) 8 4
Complete Perspective Projection Equation We combine the 3 transformation steps:. scene coordinates => camera coordinates. projection of camera coordinates into image plane 3. camera coordinates => image coordinates p = { f/ [cos β cos γ ( - ) + cos β sin γ ( - ) + sin β ( - )] - p } a p = { f/ [ (- sin α sin β cos γ - cos α sin γ ) ( - ) + ( sin α sin β sin γ + cos α cos γ ) ( - ) + sin α cos β ( - )] - p } b with = (- cos α sin β cos γ + sin α sin γ ) ( - ) + ( - cos α sin β sin γ - sin α cos γ ) ( - ) + cos α cos β ( - ) 9 Homogeneous Coordinates () 4D notation for 3D coordinates which allows to epress nonlinear 3D transformations as linear 4D transformations. Normal: v = R (v - v ) Homogeneous coordinates: v = A v A = R T = r r r 3 r r r 3 r 3 r 3 r 33 - - - (note italics for homogeneous coordinates) Transition to homogeneous coordinates: v T = [ ] => v T = [w w w w] w is arbitrar constant Return to normal coordinates:. Divide components - 3 b 4th component. Omit 4th component 5
Homogeneous Coordinates () Perspective projection in homogeneous coordinates: v p = P v with P = /f and v = w w w w gives v p = w w w w/f Returning to normal coordinates gives v p = f/ f/ f compare with earlier slide Transformation from camera into image coordinates: v p = B v p with B = a - a b - b and v p = w p w p w gives v p = wa( p - ) wb( p - ) w Homogeneous Coordinates (3) Perspective projection can be completel described in terms of a linear transformation in homogeneous coordinates: v p = B P R T v B P R T ma be combined into a single 4 4 matri C : v p = C v In the literature the parameters of these equations ma var because of different choices of coordinate sstems, different order of translation and rotation, different camera models, etc. 6
Inverse Perspective Equations Which points in a scene correspond to a point in the image? p p? Each image point defines a projection ra as the locus of possible scene points (for simplicit in camera coordinates): v λ v p => v λ = λ v v p p origin v = v + R T λ v p 3 equations with the 4 unknowns,,, λ and camera parameters R and v Applications of inverse perspective mapping for e.g. - distance measurements - binocular stereo - camera calibration - motion stereo 3 Binocular Stereo () v optical ais optical ais o u u o b l l l, l b o, o f, f v u, u camera positions (optical center) stereo base camera orientations (unit vectors) focal distances scene point projection ras of scene point (unit vectors) 4 7
Binocular Stereo () Determine distance to v b measuring u and u Formall: α u = b + β u => v = α u + l α and β are overconstrained b the vector equation. In practice, measurements are ineact, no eact solution eists (ras do not intersect). Better approach: Solve for the point of closest approimation of both ras: v = α u + (b + β u ) Solution: α = u T b - (u T u ) (u T b) + l => minimie α u - (b + β u ) - (u T u ) β = (u T u ) ( u T b) - (u T b) - (u T u ) 5 Distance in Digital Images Intuitive concepts of continuous images do not alwas carr over to digital images. Several methods for measuring distance between piels: Eucledian distance D E ((i, j),(h,k)) = (i h) + ( j k) costl computation of square root, can be avoided for distance comparisons Cit-block distance D 4 ((i, j)(h, k)) = i - h + j - k number of horiontal and vertical steps in a rectangular grid Chessboard distance D 8 ((i, j)(h, k)) = ma { i - h, j - k } number of steps in a rectangular grid if diagonal steps are allowed (number of moves of a king on a chessboard) 6 8
Connectivit in Digital Images Connectivit is an important propert of subsets of piels. It is based on adjacenc (or neighbourhood): Piels are 4-neighbours if their distance is D 4 = Piels are 8-neighbours if their distance is D 8 = all 4-neighbours of center piel all 8-neighbours of center piel A path from piel P to piel Q is a sequence of piels beginning at Q and ending at P, where consecutive piels are neighbours. In a set of piels, two piels P and Q are connected, if there is a path between P and Q with piels belonging to the set. A region is a set of piels where each pair of piels is connected. 7 Closed Curve Paradoon solid lines if 8-neighbourhood is used line line a similar paradoon arises if 4-neighbourhoods are used line does not intersect line although it crosses from the outside to the inside 8 9
Geometric Transformations Various applications: change of view point elimination of geometric distortions from image capturing registration of corresponding images artificial distortions, Computer Graphics applications Step : Determine mapping T(, ) from old to new coordinate sstem Step : Compute new coordinates (, ) for (, ) Step 3: Interpolate grevalues at grid positions from grevalues at transformed positions grevalue must be interpolated 9 Polnomial Coordinate Transformations General format of transformation: m m-r = a rk r k r = k= m m-r = b rk r k r = k= Assume polnomial mapping between (, ) and (, ) of degree m Determine corresponding points a) Solve linear equations for a rk, b rk (r, k =... m) b) Minimie mean square error (MSE) for point correspondences Approimation b biquadratic transformation: = a + a + a + a + a + a = b + b + b + b + b + b Approimation b affine transformation: = a + a + a = b + b + b at least 6 corresponding pairs needed at least 3 corresponding pairs needed
Translation, Rotation, Scaling, Skewing Translation b vector t: v = v + t with v = v = t = t t Rotation of image coordinates b angle α: v = R v with R = cos α sin α sin α cos α Scaling b factor a in -direction and factor b in -direction: v = S v with S = a b Skewing b angle β: v = W v with W = tan β Eample of Geometr Correction b Scaling Distortions of electron-tube cameras ma be - % => more than 5 lines for TV images Correction procedure ma be based on - fiducial marks engraved into optical sstem - a test image with regularl spaced marks ideal image actual image Ideal mark positions: mn = a + mb, mn = c + nd Actual mark positions: mn, mn m =... M- n =... N- Determine a, b, c, d such that MSE (mean square error) of deviations is minimied c d b a
Minimiing the MSE Minimie M N E = ( mn mn ) + ( mn mn ) m = M n = N = (a+ mb mn ) + (c+ nd mn ) m = n = From de/da = de/db = de/dc = de/dd = we get: a = MN(M+ ) m 6 b = MN(M ) c = MN(N + ) m 6 d = MN(N ) n m n (M 3m) mn n m (m M + ) mn (N 3n) mn n (n N + ) mn Special case M=N=: a = / ( + ) b = / ( - + - ) c = / ( + ) d = / ( - + - ) 3 Principle of Grevalue Interpolation Grevalue interpolation = computation of unknown grevalues at locations (u v ) from known grevalues at locations ( ) Two was of viewing interpolation in the contet of geometric transformations: A B Grevalues at grid locations ( ) in old image are placed at corresponding locations ( ) in new image: g( ) = g(t( )) => interpolation in new image Grid locations (u v ) in new image are transformed into corresponding locations (u v) in old image: g(u v) = g(t - (u v )) => interpolation in old image We will take view B: Compute grevalues between grid from grevalues at grid locations. 4
Nearest Neighbour Grevalue Interpolation Assign to ( ) grevalue of nearest grid location ( i j ) ( i+ j ) ( i j+ ) ( i+ j+ ) grid locations ( ) location between grid with i i+, j j+ ( i j ) ( i+ j ) ( ) ( i j+ ) ( i+ j+ ) Each grid location represents the grevalues in a rectangle centered around this location: Straight lines or edges ma appear step-like after this transformation: 5 Bilinear Grevalue Interpolation The grevalue at location ( ) between 4 grid points ( i j ) ( i+ j ) ( i j+ ) ( i+ j+ ) is computed b linear interpolation in both directions: g(,)= {( i + )( j+ )g( i j ) + ( i )( j+ )g( i+ j ) + ( i+ i )( j+ i ) ( i+ )( j )g( i j+ ) + ( i )( j )g( i + j+ )} Simple idea behind long formula:. Compute g = linear interpolation of g and g. Compute g 34 = linear interpolation of g 3 and g 4 3. Compute g = linear interpolation of g and g 34 g g g g g 3 g 34 g 4 The step-like boundar effect is reduced. But bilear interpolation ma blur sharp edges. 6 3
Bicubic Interpolation Each grevalue at a grid point is taken to represent the center value of a local bicubic interpolation surface with cross section h 3. - + 3 for < < h 3 = 4-8 + 5-3 for < < otherwise The grevalue at an arbitrar point [u, v] (black dot in figure) can be computed b - 4 horiontal interpolations to obtain grevalues at points [u, j-]... [u, j+] (red dots), followed b - vertical interpolation (between red dots) to obtain grevalue at [u, v]. Note: For an image with constant gevalues g the interpolated grevalues at all points between the grid lines are also g. cross section of interpolation kernel j- j - - j+ j+ i- i i+ i+ 7 4