P Z. parametric surface Q Z. 2nd Image T Z

Size: px

Start display at page:

Download "P Z. parametric surface Q Z. 2nd Image T Z"

Adela Dean
5 years ago
Views:

1 Direct recovery of shae from multile views: a arallax based aroach Rakesh Kumar. Anandan Keith Hanna Abstract Given two arbitrary views of a scene under central rojection, if the motion of oints on a arametric surface is comensated, the residual arallax dislacement eld on the reference image is an eiolar eld. If the surface aligned is a lane, the arallax magnitude at an image oint is directly roortional to the height of the oint from the lane and inversely roortional to its deth from the camera. We exloit the above theorem to infer D height information from oblique aerial D images. We use direct methods to register the aerial images, develo methods to infer height information under the following three conditions: (i) focal length and image center are both known, (ii) only the focal length is known, and (iii) both are unknown. Introduction Traditional methods in motion analysis have exressed the image motion of rigid bodies as a sum of two image dislacement elds: a rotation eld and a eiolar (translation) eld. In this aer, we develo an alternative aroach which is based on decomosing the motion eld into the image motion of a arametric surface and a residual arallax eld. The motion of the surface can be exressed as a arametric motion eld and is estimated using a direct technique []. The direct aroach rovides a more accurate alignment of the surface than using ure \bottom-u" ow-elds. The residual arallax eld is an eiolar eld (see Section ) and is quasi-arametric it is also estimated using a direct method. The arallax magnitudes can be used to infer the D structure of the scene relative to the arametric surface in an object centered reresentation. In section, we develo quasi-arametric direct methods to register aerial images to establish corresondences and infer height information from oblique aerial images under the following three conditions: (i) focal length and image center are known, (ii) focal length is unknown but image center is known and (iii) Readers may contact the authors at David Sarno Research Center, CN00, rinceton, NJ-080, U.S.A.. kumar@sarno.com focal length and image center are both unknown. Finally in section, we resent real data height estimation results, which are an order of magnitude better than the results obtained using the best traditional (ersective rojection) structure for motion algorithms []. ur work is related to the recent work using rojective geometry of [, 6, ] and motion stabilization of []. arametric surfaces In this section, we state the theorem about arallax motion observed after aligning an arbitrary arametric surface or lane between images of two views of a scene. The D oint ~ in the reference camera coordinate system gets maed to the D oint ~ in the insection camera coordinate system by a rigid body transformation: ~ = R( ~ )+ ~ T = R( ~ ; ~ T ) () The maing can be reresented by a rotation (R) followed by a translation ( ~ T )orby a translation ( ~ T ) followed by a rotation (R). With ersective rojection, the image coordinates (x,y) of a rojected oint are given by the vector ~: ~ = x y f = f z () where f is the focal length. Theorem I. Given two views of a scene (ossibly from twodistinct uncalibrated cameras), if the motion corresonding to an arbitrary arametric surface between is comensated (by alying an aroriate D arametric transformation to one of the images and resamling it) then the residual arallax dislacement eld on the reference image lane is an eiolar eld. II (a). Let be a oint not on the surface that is registered, and let be its image in the reference view (see Figure ). Let T denote the baseline vector between the cameras and be the oint where

2 Reference Image q t arametric surface S nd Image Figure : Residual arallax eld after surface alignment is eiolar ' Reference Image Z Z Z T Z q m arametric surface nd Image Figure : Residual arallax magnitude when arametric surface is aligned Reference Image Reference lane q m H T T nd Image Figure : Residual arallax magnitude when lane is aligned the ray connecting to the the second camera center intersects the surface. Then the residual arallax dislacement u at image location can be shown to be u = q ; = T z( z ; z ) z ( z ; T z ) ( ; t ) () where z and z denote the deths of oints and, T z is the z comonent of translation vector T (assumed non-zero), and t denotes the eiole corresonding to T. II (b). If the surface that is aligned is a lane, then the residual arallax dislacement simlies in the case of T z 6=0to: u = HT z( ; t ) T? z ; HT z and in the case of T z =0to: = HT z T? z (q ; t ) () u = ; fh T? z ~ T () where H is the erendicular distance from the oint to the reference lane, T? is the erendicular distance between the second camera center and the reference lane (see Figure ). roof: art I Referring to Figure, let S denote the surface of interest, an environmental oint not on S, and and the two camera centers. The image of on the reference view is. Let the ray intersect the surface S at. The waring rocess would war 0, the image of on the second image to q, the image of on the reference image, since the transformation alied aligns all oints on the surface S. Therefore, the residual arallax vector is q, which is the image of the line. It is immediately obvious from the gure that q lies on the lane, which is the eiolar lane assing through. Since the above argument is true for any oint, the arallax dislacement eld is an eiolar eld. roof: art II For lack ofsacewe do not resent the roof for art II of the theorem. However, the formulae shown in art II can be very easily derived from Figures and using similar triangles and other simle geometric maniulations. Shashua et. al. [] and Sawhney [6] rovide algebraic roofs for the lanar case and arrive at the same formulae for that case. Registration and Interretation To register a lane, we use the hierarchical direct registration technique described in [] with a lanar surface ow eld model. This technique rst constructs a Lalacian yramid from each of the two inut images, and then estimates the motion arameters in a coarse-ne manner. Within each level the Sum of squared dierence (SSD) measure integrated over user selected regions of interest is used as a match measure. This measure is minimized with resect to the quadratic ow eld arameters. The SSD error measure for estimating the ow eld within a region is: E(fug) = X x (I(x t) ; I(x ; u(x) t; )) (6) where x =(x y) denotes the satial image osition of aoint, I the (Lalacian yramid) image intensity and u(x) =(u(x y) v(x y)) denotes the image velocity at that oint, and the sum is comuted over all the oints within the region and fug is used to denote the entire ow eld within that region. The motion eld of a lanar surface can be reresented as: u(x) = x + y x + 8 xy v(x) = x + y xy + 8 y (7)

3 where = 6 N x T x ; N z T z N y T x ; z N x T y + z N y T y ; N z T z f(n z T x + y ) f(n z T y ; x ) ( f y ; N x T z ) (; f x ; N y T z ) 7 (8) In the above equation, (T x T y T z ) denotes the translation vector between the cameras, ( x y z ) denotes the angular-velocity vector, and (N x N y N z ) denotes the normal vector to the lanar surface from the camera center. The reader is referred to [] for further details about this registration technique. The arallax vectors and the direction of translation are simultaneously estimated using the quasiarametric technique described in [].The quasi arametric technique is generally more accurate than using otic ow, but requires an initial estimate for translation. If needed, an initial estimate of the translation direction can be obtained by using the otical ow obtained by using the technique also described in [].. Height Interretation The arallax ow vectors vary directly with height and inversely with deth. To factor out the deth and get the height alone, we useacharacteristic roerty of aerial view images. For aerial view images, the deth of the lane is tyically much greater than the height of objects on the ground. In nadir aerial images a weak ersective rojection aroximation can be used, since the deth of all oints is aroximately the same. Whereas, in an oblique view, there can be considerable deth variation across the image. However, for any single oint in an oblique aerial image, the deth of that image oint is essentially the same as the deth of a ossibly virtual D oint obtained by extending the line of sight ray and intersecting it with the ground lane. Therefore, we can factor out the deth in the arallax equations (,) by estimating the equation of the ground lane (we actually only need to estimate the normal of the lane u to a scale factor). We use the magnitude of the dislacement vectors to infer the magnitude of the height andthe direction of the ow vectors to infer the sign of the height. The magnitude of the dislacement vector = (x w ; x ) +(y w ; y ) for the case where We omit the formulae for the case of Tz =0forlackof sace. T z 6= 0 is given by: = T zh z T? F (9) where F = (x w ; x F ) +(y w ; y F ) is the distance of the oint (x w y w ) from the FE. Noting that N T ~ = f z and using the aerial view roerty equation (9) can be written in the form: I = SHN T ~ (0) where I = F is an image based measurement. S is a roortionality factor which deends solely on the translation vector and its distance to the lane N. The above equation can be rewritten as: H = I (K x + K y + K ) () where K ~ is an unknown vector and its comonents are given by K = SN x, K = SN x and K = fsn z The height H of any image oint can be comuted using equation (). If the focal length and center are both unknown, height of at least three oints are known, these can be used together with equation () to linearly estimate the vector K. ~ We do not need to know the image center, because the unknown oset simly loads the third comonent ofthevector K ~ in Equation. We then use this vector K, ~ again with equation () to determine the height at any other oint, again in a linear fashion. The exerimental results resented in section use this case. If focal length and center are known, we can infer the normal of the lane by using equation (8). This equation relates the quadratic registration arameters to the translation, rotation and normal of the lane, but the translation direction is comuted during the quasi-arametric residual estimation. The translation direction together with equation (8) gives us a linear set of 8 equations for the other 6 unknowns: normal vector N ~ and the rotation vector. ~ Since the translation used in equation (8) is T while the translation we comute from the arallax ow vectors is T, we must invert the quadratic transformation dened by the arameters :: 8 (or directly estimate the inverse quadratic transformation by inter-changing the two images during estimation.) We determine translation only u to a scaling factor and therefore need the height of at least one oint to determine the heights of all other oints in absolute coordinates. Finally, for determining the vector K ~ when focal length is unknownbut imagecenter is knownwe need the height of two oints. This is a combination of the two revious cases. Again for lack of sace this roof is omitted.

4 Results The oblique aerial images were taken in the laboratory by mounting a camera on a triod and simulating an aerial y ath. In the image air (blique forward sequence), (Figures,) the camera is moving forward aroximately 6 inches in the y-z lane of the camera. The camera was tilted at aroximately degrees with resect to gravity and was 69 inches above the ground. The height of the blocks ranged from inch to.9 inches. In Figure (),the numbers written on to of the rectangular at-to blocks are the height of the blocks in inches. Similarly for the triangular blocks, the slant oftheblock (degrees) was written on to of the block. Figure : blique forward aerial view: reference image, Frame. Table : Estimation of heights from oblique aerial views. BL- TRUE AVG. STD. N. CNF. CK HT. HT. DEV. TS THR. in. in. in. * * *CAR The ground lane in the images were registered and the heights comuted using the algorithms described in the revious sections. To comute the height images, we used the height of known oints. In Table, the blocks from which thethreeoints were selected are marked by the symbol \". The height image for the blique forward air can be seen in Figure 6. The bright areas in the images corresond to areas with greater height. There is a a lot of noise around the borders of the images. This corresonds to border areas where there was no information available in one or the other image. These regions are detected by alow condence value obtained from the otic ow estimation algorithm [] and therefore do not ose a roblem. Table lists quantitative results in recovering heights. For each at to in the image, the actual height ofthe block isrovided and the average estimated height and the standard deviation of the heights of the oints on that block are also listed. The number of oints over which the average was comuted and the condence threshold used to select the oints is also listed. For the results listed in the table, we selected all oints belonging to a block by using a condence threshold of 0. In the case of BLCK9 the standard deviation of the obtained heights is comaratively large: 0. for the oblique forward sequence. This large variation is due to regions of uniform intensity in the BLCK9. If we select oints on BLCK9 with condence value greater than 0, (see Table ) that the standard deviation dros considerably. We are able to estimate the height of a large majority of the oints within inches. The best structure from motion algorithms [] would estimate deth of oints tyically to an average accuracy of % or so. which is the same as about inches. The recovery of heights from this data would have about the same accuracy of inches for camera height of69inches. Therefore, our results are an order of magnitude better in accuracy. Figure 7 shows a wire-frame reconstruction of the height ma of the car labeled on the image. The height ma shown in Figure (6) was used to construct the wireframe drawing. The two bums seen on to of the car in the wire-frame drawing are hysically there in the car, they corresond to two semi-transarent ashing signal sirens. References [] Bergen, J. R., Anandan,., Hanna, K.J. and Hingorani, R., \Hiearchical odel-based otion Estimation," roceedings nd Euroean Conference on Comuter Vision-9, Sringer-Verlag, Santa argherita Ligure, Italy, ay 99. [] Hartley, R. and Guta, R., \Comuting matched eiolar rojections," roceedings IEEE Conference The slant of the front window of the car is degrees

HEIGHT A F CAR HEIGHT 0 0 0 60 80 00 Y-AXIS 80 60 0 X-AXIS 0 0 Figure : blique forward aerial view: insection image, Frame Figure 6: blique forward view height ma.

$, \Recovery of egomotion using image stabilization," roceedings IEEE Conference on Comuter Vision and attern Recognition, Seattle, June 99. [] liensis, J. and J. I. Thomas, \Incororating motion error in multi{frame structure from motion," roceedings IEEE Worksho on Visual otion, rinceton, N.$

5 HEIGHT A F CAR HEIGHT Y-AXIS X-AXIS 0 0 Figure : blique forward aerial view: insection image, Frame Figure 6: blique forward view height ma. Figure 7: blique forward aerial view: Wire frame drawing of car. on Comuter Vision and attern Recognition, New York, 99. [] Irani,., Russo, B., and eleg, S., \Recovery of egomotion using image stabilization," roceedings IEEE Conference on Comuter Vision and attern Recognition, Seattle, June 99. [] liensis, J. and J. I. Thomas, \Incororating motion error in multi{frame structure from motion," roceedings IEEE Worksho on Visual otion, rinceton, N.J., ct. 99. [] Shashua, A. and Navab, N., \Relative Ane Structure: Theory and Alication to D reconstruction from ersective views," roceedings IEEE Conference on Comuter Vision and attern Recognition, Seattle, June 99. [6] Sawhney, H.S., \D Geometry from lanar arallax," roceedings IEEE Conference on Comuter Vision and attern Recognition, Seattle, June 99.

Stereo Disparity Estimation in Moment Space

Stereo Disparity Estimation in Moment Space Stereo Disarity Estimation in oment Sace Angeline Pang Faculty of Information Technology, ultimedia University, 63 Cyberjaya, alaysia. angeline.ang@mmu.edu.my R. ukundan Deartment of Comuter Science, University