Joint disparity and motion eld estimation in. stereoscopic image sequences. Ioannis Patras, Nikos Alvertos and Georgios Tziritas y.

Transcription

1 FORTH-ICS / TR-157 December 1995 Joint disparity and motion ed estimation in stereoscopic image sequences Ioannis Patras, Nikos Avertos and Georgios Tziritas y Abstract This work aims at determining four dense eds given two stereoscopic pairs of images in consecutive time instances: two dense veocity eds and two dense disparity eds. The proposed scheme is competed in two stages. During the rst stage, the dense disparity ed of the rst stereoscopic pair is estimated using a mutiscae iterative reaxation agorithm. During the second stage, again using a simiar iterative agorithm, the two dense veocity eds and the disparity ed of the second stereoscopic pair are estimated. The method has been impemented with both synthetic and rea data. y Institute of Computer Science, F.O.R.T.H., and, Computer Science Department, University of Crete, Greece, E-mai: patras@csd.uch.gr, avertos@csd.uch.gr, tziritas@csi.forth.gr

2 1 Introduction There are three genera approaches regarding dynamic stereo vision which are found in the existing bibiography [11]. The rst one consists of initiay soving the stereoscopic probem, which eads to the static determination of objects, foowed by the correspondence of these objects in time. Leung and Huang [5] assume a scene which contains ony one moving object with respect to the stationary receiver (camera), where stereoscopic correspondences are rst determined for a sucient number of points foowed by evauation of their three-dimensiona motion. A simiar approach to the probem is empoyed by Netravai et a. [9] and Mitiche and Bouthemy [8]. Kim and Aggarwa [] use segments of boundaries, instead of points, for which initiay the stereoscopic probem is soved. Motion is determined under the hypothesis that the depicted objects are rigid and, thus, their geometric characteristics remain unchanged in time. Simiar is aso the approach by Zhang and Faugeras [13] which is based on straight ine segments with compete determination and appropriate description in the three-dimensiona space. The second genera approach evauates independenty the two two-dimensiona veocity eds in the sequence of stereoscopic image pairs and then determines and uses the stereoscopic reations which exist between the two veocity eds. Mitiche [7] determines reations among positions and veocities of discrete points which correspond to the stereoscopic pairs. Soution to these equations resuts into the three-dimensiona motion of a set of points which are assumed to beong to the same rigid body. Waxman and Duncan [1] determine reations between the -D veocity eds of the stereoscopic pair and dene the binocuar dierentia motion which is connected to the stereoscopic disparity. Finay, the third approach uses a joint estimation of the two -D veocity eds taking advantage of their stereoscopic reation without seeking the compete 3-D reconstruction of the depicted objects. In this case, where a compete 3-D motion and shape object description is not required, Tamtaoui and Labit [10] introduce stereoscopic reations between the veocity eds of the stereoscopic pair which utiize for the simutaneous evauation of the two veocity eds using a recursive method. This approach seems to produce interesting resuts for appications in 3-D teevision. As it was shown through the previous bibiographic reference, the existing

3 soutions do not utiize the stereoscopic and motion reations simutaneousy; instead, they consider the probem in sequentia stages. It is known, however, that positions at each time instant are connected with dispacements and, furthermore, the reations which connect the rate of change of the stereoscopic disparities with the veocity eds are known. This work aims at an integrated soution to the probem of dynamic stereoscopic vision. A simutaneous estimation of the veocity and disparity eds in a dense structure, that is, for a image points, as opposed to most of the existing methods where ony sparse image descriptions are given, is proposed. At any time instant two motion eds (for the eft and right image sequences) and one disparity ed are computed. The disparity ed of the previous stereoscopic pair is considered as known, that is previousy estimated. In addition, a method which has been shown to be eective in monocuar motion anaysis is used, given that it has been appropriatey adapted to the requirements of the probem. Initiay, a cost function is determined which constraints the dierent eds to be adaptativey smooth. This cost function aso contains known equations regarding veocity and disparity eds in reation to image intensity, as we as reations between veocities and disparities depending on the geometrica mode of the optica system. Minimization of the cost function resuts into estimation of the veocity and disparity eds. This minimization can be achieved using an iterative reaxation agorithm based on the gradient of the cost function. θ z y x B z r yr x r Figure 1: Converging Mode The anaysis is based on a converging (xating) stereoscopic optica system, 3

4 shown in Figure 1, where is the ange between the two optica axes, B is the distance between the two foca points and f is the same foca ength for each camera. For a 3-D point r with eft camera coordinates r = ( ; Y ; Z ) and right camera coordinates r r = ( r ; Y r ; Z r ) the foowing equaity hods true: 6 r Y r Z r = 6 cos 0 sin 0 1 0? sin 0 cos Y Z ?B cos (=) 0 B sin (=) (1) If (x r ; y r ) and (x ; y ) are the perspective projections of r in the right and eft image, respectivey, the foowing reations among the coordinates of the same point r and its projections in the stereoscopic pair are obtained: x r Z r f y r Z r f = x Z f cos + Z sin? B cos = () = y Z f (3) Z r =?x Z f sin + Z cos + B sin = () Examining the ratio y y r it is shown: y y r = Z r Z = x f x r f sin = + cos = sin = + cos = Dening the disparity vector ~ d as ~d = (x r? x ; y r? y ) and assuming that is very sma, then y y r 1; that is, the y-coordinate of d ~ is amost zero. For the remainder of this work it is accepted that a previous assumptions hod true, thus, d ~ is a 1-D vector aong the x-axis. Eqs. ( - ) form the basis for 3-D scene reconstruction foowing the estabishment of image-point correspondence in the stereoscopic pair. For every image-point pair (x r ; y r ); (x ; y ), the soution of those equations provides a depth estimate (Z and Z r ) of the projected 3-D point. Depth can aso be cacuated using an intermediate coordinate system (the cycopean system) which is obtained by rotating < ; Y ; Z > about the y axis with an ange = and then transating B sin (=) aong the x-axis. From the corresponding equations, a cosed form soution for Z (depth) is determined []: Z =?B tan(? ) tan( + ) tan(? ) + tan( + ) (5) where = =?, = arctan(x =f) and = arctan(x r =f).

5 If the ange = 0, we obtain the atera mode. In this case, the equations reating image and 3-D point coordinates are: x r = x? B f Z (6) y r = y (7) where Z = Z r = Z. It is aso noted that the disparity vector ~ d is 1-D aong the x-axis. In Section we present a reguarization method for obtaining a smooth disparity ed from a stereoscopic pair using diusion adaptive functions. In Section 3 the simutaneous estimation of the two motion eds and the current disparity ed from two successive stereoscopic pairs is presented. Section contains experimenta resuts with both synthetic and rea stereoscopic image sequences. Then some concusions are given, as we as directions of future work. Disparity Fied Estimation Soution to the stereoscopic probem consists of determining a dense disparity ed through which every point (x ; y ) in the eft image is matched to a point (x + ; y ) = (x r ; y r ) in the right image. Using the optica-ow preservation principe, it is aso true that I (x ; y ) = I r (x r ; y r ). However, since intensity measurements are not exact and a hypotheses are not absoute, the foowing functiona is minimized [3]: e i = Z Z (I (x; y)? I r (x + ; y)) dxdy (8) In addition, it is required that Z, and thus, is varying smoothy. Consequenty, a smooth-ed functiona is introduced for minimization: e s = Z Z (g( x ) + g( y ))dxdy (9) where x y and g(:) is a function which beongs to DAF (Diusion Adaptive Functions) as described in [6]. g(:) as a member of the DAF famiy, is provided as an aternative to the quadratic reguarizer (:) which has the disadvantage that it imposes the smoothness constraint everywhere and eads to oversmoothing. On the other hand a carefuy chosen g c (:) may reguarize the soution and at the same time preserve the discontinuites. In that framework the interaction function h(:) which is dened such that : g 0 (x) = xh(x) determines the interaction between neighboring pixes. In this work g(:) and h(:) were chosen to be : g(x) = jxj? n (1 + jxj and h(x) = 1 1+ jxj. For the discrete case the tota quantity to be minimized is 5 )

6 given by: i j (I r (i + i;j ; j)? I (i; j)) + i j g( ij? p 0) where N ij = f(i?1; j); (i+1; j); (i; j?1); (i; j+1)g is the -point neighborhood of (i; j). The sum P g( ij? p 0) is an approximation for e s and can be extended to a neighborhood of more than points (e.g., 3 3 where the diagona terms are normaized by p ). is a weight coecient which determines to what degree estimation of the ed is inuenced by the smoothing operator. Minimization of this quantity resuts into the foowing equation: (I r (i + i;j ; j)? I (i; j)) r x(i + i;j ; j) + p 0( ij? ij ) = 0 (10) where p 0 = h( ij? p 0) are the coecients which determine the contribution of each of the neighborhood points in estimating ij, and ij = p 0 p 0 p 0 N ij p 0 Assuming that the magnitude of the ed is reativey sma and image intensity varies smoothy, the foowing reations hod true: I r x(x + ; y) = I r x(x + ; y) I r (x r ; y r ) = I r (x + ; y ) = I r (x + ; y ) + (? )I r x (x + ; y r ) Considering the above, Eq. (10) becomes: (I r + (? )I r x (x + ; y r )) I r x (x + ; y r ) + p 0(? ) = 0 (11) where I r = I r (x + ; y )? I (i; j). In tota, there are as many equations as there are points (i.e., unknowns). The system is formuated into a tri-diagona matrix, thus aowing use of the iterative Gauss-Seide method. The soution at the k th iteration is given by the reation: k i;j = I k?1 i;j? r I r x P (1) p 0 + ( r x) where k?1 is the disparity ed estimated at the (k? 1) iteration. It is particuary important to determine a termination condition for the agorithm, so that convergence is ensured. For this reason, a convergence 6

7 measure is examined periodicay (every ten iterations), which is the average correction in the direction of the sope: ji T = r Ixj r P (i;j) p 0 + (Ix) r The agorithm is terminated when the percentage diminishment of the T amount becomes ess than a threshod. This condition on the diminishment T ratio is proven to be more stabe and reiabe for dierent stereoscopic pairs than a threshod condition on the T amount itsef..1 Mutiscae Agorithm The previousy described agorithm, as a gradient-descent agorithm, can estimate successfuy ony eds of sma disparities. Otherwise, it requires good initia conditions so that it wi not be entrapped and converge to a oca minimum. Thus, it is insucient for rea data where arge disparity vaues are possibe and no prior genera knowedge of the scene depth is avaiabe. 3rd eve nd eve 1st eve (a) (b) Figure : (a) Data pyramid and (b) Fied pyramid Consequenty, a coarse-to-ne mutiscae method in a pyramida form (Fig. ) is impemented, where in each eve () the agorithm is appied to images of submutipe dimensions of the origina ones. Those images are the resut of reduction by ow-pass tering and subsamping. The image dimensions at eve are both reduced by a factor. Convergence of the agorithm at eve impies a dense disparity ed where each of its 1-D vector () corresponds to an initia image region of size. The estimated at eve disparity ed () constitutes the initiaization of the ed (?1) to be estimated at eve? 1. An immediate resut of this reduction is the scae change on the magnitude of the ed to be estimated. Because of the reduction, the 7

8 characteristics (features) to be matched in the two images dier by a smaer absoute vaue (number of pixes). If, for exampe, (x, y ) and (x +, y ) are the coordinates of corresponding points at eve (), then at the higher eve ( + 1) the coordinates wi be ( x, y ) and ( x +, y ), respectivey. This means that at a higher eve the disparity to be estimated has a submutipe magnitude, thus, making the agorithm more ecient. This way, at the higher eves a coarse estimation of the disparity ed is achieved which becomes ner, in exactness and detai, at the ower eves of the pyramid. In the ast eve, the base of the pyramid, the agorithm is appied to the origina images and its convergence provides the na disparity ed. 3 Simutaneous Motion and Disparity Estimation The second stage of this work consists of a simutaneous estimation of the two veocity eds and the disparity ed of the second stereoscopic image pair. 3.1 Optica Fow - Disparity Reations Combining equations 6-7, the foowing reations among the components of the eds to be estimated are derived: and y = y r ) v = v r (13) x r = x + ) u r = u + t+1? t (1) Soving the system of these two equations with respect to t+1 and v r resuts into: v r = v and t+1 = u r? u + t (15) Therefore, to competey determine the requested eds it is sucient to evauate their three components u r, u and v. 3. Proposed Soution To estimate the motion and the second disparity eds, the correspondence between points in the rst stereoscopic image pair is used, as derived in the rst stage by evauating the ed t. In the second stage, the aim is to determine two points in the second pair of images for every pair of points of the rst stereoscopic image pair, so that these four corresponding points are the projections of the same point in space. The rst point pair is the projection at time instant t whie the second is at t + 1. The principe of optica-ow preservation is used; that is, the projections in the four images of 8

9 the same feature of an object in space are of the same ight intensity. Thus, the foowing reations resut: e 1 s = I r t+1? I r t = 0 e s = I t+1? I t = 0 e 3 s = I r t+1? I t+1 = 0 where I and I r t+1 t+1 are the intensities of the eft and right points in the second stereoscopic image pair, whie I t and I r t are the intensities of the corresponding points in the rst stereoscopic image pair. As with the soution to the stereoscopic probem, the minimization of the squared deviation from the optica-ow preservation principe and the minimization of a smoothness measure for the estimated eds are considered. In addition to the eds for direct evauation, a smoothness measure for the quantity (u r?u ) is introduced; that is, the veocity dierence in the two image sequence which in the case of the atera mode is u r? u = f Vz Z. Smoothness of this quantity is achieved with a coecient in reation to the smoothness of eds u r, u and v. The tota quantity to be minimized is: E = (i;j) where Z Z Z Z + ((e 1 s) + (e s) + (e 3 s) )dxdy + Z Z r? u ) + r? u ) ) ) + g(@u ) + ) + ) + g(@v ) + g(@v )! dxdy (16) In the discrete case this quantity becomes: (I ) + (I r ) + (I r ) + (g(u r? ij ur 0) + p g(u? ij u 0) + p g(v? ij v 0) + p g((ur? ij u )? ij (ur? p u 0)) 0 p (i;j) I r = I r t+1? I t+1 I r = I r t+1? I r t I = I t+1? I t Minimizing this ast quantity the foowing equations are obtained: I I x? I r t+1 I x + 1 p 0(u ij? u 0) + p p ((u 0 ij? u r ij)? (u? p ur 0)) = (17) 0 0 p p 0 N ij I r I r r x + I I r + t+1 x p 0(u? ij u 0) + p p ((ur? 0 ij u )? ij (ur? p u 0)) = (18) 0 0 p I I x + I r I r x + (It+1) r (I r x? Ix) + 3 p 0(v ij? v 0) = (19) 0 p 9

10 where the coecients 1 0; p 0; p 3 and p 0 resut from the smoothing of p u ; u r ; v 0 and ( u r? u ), respectivey, and the derivatives of the ight intensities refer to the second stereoscopic image pair (i.e., at time instant t + 1). Let P u p ij = 0 N ij 1 p 0u p 0 + P 1 p 0 and simiary u r ij, whie et v ij be: v ij = P p 0u p 0 P p 0 P 3 p 0v p 0 P 3 p 0 and V ij = 3 p 0(v ij? Assuming that the eds to be estimated are sma in magnitude and that the intensities vary smoothy, the foowing approximations are used: v ij) approximation of the rst derivative of the intensity: I r x I r y r (x t+1 r + u r ; y + v r (x t+1 r + u r ; y + r t+1 (x r + u r ; y + r t+1 (x r + u r ; y + v Simiar approximations hod true for the intensities in the eft image (I x and I y). approximation of the intensities at points (x r + u r ; y + v ) and (x + u ; y +v ) in the eft and right image, respectivey, with rst order Tayor expansion: I r t+1 (x r + u r ; y + v ) I r t+1 (x r + u r ; y + v ) + (u r? u r )I r x + (v r? v r )I r y where Q = I t+1(x + u ; y + v ) I t+1(x + u ; y + v ) + (u? u )I x + (v? v )I y The na form of the equations in the discrete grid are: 6 Q 6 u k? uk?1 u k r? uk?1 r v k? v k?1 (I x) +?I xi r x??i xi r x? = b (0) p 0 I x(i y? I r y) p 0 (I r x) + I r x(i r y? I y) I x(i y? I r y) I r x(i r y? I y) + (I y) + (I r y) + (I r y? I y)

11 and b = 6 I x(i? I r ) + ( I r x(i r + I r ) + ( 0 p ((? 1) 0 p ((? 1) p 0ur p 0 + p 0 u I y (I? I r ) + I r y (I r + I r ) p 0 + p 0 ur p 0 u p 0 p 0 The above system is represented by a positive-denite matrix; thus, its soution is obtained through direct inversion. Experimenta Resuts.1 Stereoscopic Probem Soutions This agorithm was tested with both synthetic and rea data. In the synthetic case, the image of a square is given, with constant disparity = 5 for the square and = 0 for the background. The mean squared error in evauating the ed was , and the mean squared dierence between the right image and its reconstruction using the estimated ed was The argest scae where the ed was estimated was, whie the number of iterations necessary for the agorithm to converge was 150. The vaue of the coecient was 500. A quantitative evauation of the ed is shown in Fig. 3(a), where the corresponding depth map is given (a) (b) Figure 3: Depth map at time (a) t, and (b) t + 1 Rea data consist of a stereoscopic pair shown in Fig.. For = 500, the resuting mean squared error on the intensities was In addition, Fig. 5(a) shows a depth evauation, as it resuts from Eq. (5), of the depicted objects. 11

12 Figure : Stereoscopic image pair at time t. Stereoscopy and Motion The resuts on synthetic data from the second stage of the work are concuded in Tabe 1. Tests on synthetic data were reaized using two categories of images: The second stereoscopic image pair foows from the rst by transating the components U, U r and V = V r which are varied from the eft top to the right bottom point of the image: U from 1.0 to 3.0 U r from 1.0 to.0 V = V r =.0 (constant) The second stereoscopic image pair foows from the rst by transating the depicted square (where = 5 ) as it resuts from the constant eds U = 3:0, U r = 3:0, V = V r = :0. The remaining image is transated by a vector (?1;?1) for both the eft and the right sequence. Category Iterations MSE MSE on Intensities U U r V LL RR RL Tabe 1: Experimenta resuts on synthetic data In Tabe 1 the MSE expresses the mean squared deviation for each of the estimated eds; that is, it resuts from P P (^u ij? U ij ), where ^u is the 1

13 (a) (b) Figure 5: Estimated depth map at time (a) t, and (b) t + 1 estimated ed, whie U is the syntheticay created ed, on which production of the second stereoscopic image pair was based. In the same tabe MSE on intensities are given too, for the eft images (LL), for the right images (RR) and for the stereoscopic pair at t + 1 (RL). A quaitative evauation of the ed t+1 is shown in Fig. 3(b). Impementation of the second stage with rea data consists of two stereoscopic pairs, the rst of which was the one used during the rst stage of this work. The resut of the rst stage (i.e., correspondence between image points in the rst pair) is assumed given and is utiized for the simutaneous estimation of motion and disparity at the next time instant. Quaitativey, the motion of scene objects consists of two components: a camera movement from right to eft, which creates the optica eect of scene objects moving towards the opposite direction movement of the depicted train from right to eft Maximum eve Iterations MSE on Intensities LL RR RL Tabe : Stereoscopy and motion on rea data The estimated motion ed for a scae depth equa to 3 and = 150 is shown in Fig. 6, where the unied motion of the background to the right, 13

14 and the independent motion of the train to the eft, are ceary distinguished. Other numerica resuts are given in Tabe. The estimated motion eds were then segmented into regions using a K-Means type agorithm, in which the number of segments shoud be provided a priori. The distance between a segment s and a point p was dened to be the quantity: k ^Us? U p k, where ^U s is the mean of the veocity vectors at the points that beong to segment s and U p the veocity vector at point p. The resuting segmentation on the eft motion ed is shown in Fig. 6, were the regions that correspond to the train and to the camera's motion are ceary distinguished. The evauated depth map for the time instant t + 1 is shown in Fig. 5(b). (a) (b) Figure 6: (a) Estimated dense veocity ed for the eft camera, and (b) corresponding segmentation 5 Concusions An integrated approach to the probem of dynamic stereoscopic vision was proposed, where veocity and disparity eds in a dense structure are estimated simutaneousy. In both stages of the scheme, where initiay the dense disparity ed of the stereoscopic pair is evauated foowed by estimation of 1

15 the two dense veocity and disparity eds of the second stereoscopic pair, convergence is achieved using a mutiscae iterative reaxation agorithm. Experimenta resuts were presented for both synthetic and rea data. Specicay, the approach was appied to an image sequence of a rea scene, where both the object and the binocuar system are moving. The estimated veocity ed was then segmented using a K-Means type agorithm. Currenty we are working on the reconstruction of the 3-D motion for the dierent segments using robust regression techniques. It woud be aso usefu to incorporate the regression scheme into the segmentation phase, that is to segment the veocity ed into regions that correspond to the same 3-D motion. Since theoretica and experimenta approaches to this probem assume ony a simpe stereoscopic mode (i.e., converging or atera), it woud be usefu to examine other modes such as the axia [1], the teescopic, or a genera one where the two optica systems are reated with non-zero rotations, so that it is possibe to confront the more genera case where the geometry of the stereoscopic mode is varying with time. This optica-system dynamic behavior nds appication in robotics where autonomous mechanisms are desired. References [1] N. Avertos, D. Brazakovic, and R. C. Gonzaez, \Camera geometries for image matching in 3-D machine vision", IEEE Transactions on Pattern Anaysis and Machine Inteigence, Vo. PAMI-11, No. 9, pp , Sept [] E.Grosso and M.Tistarei, \ Active/Dynamic Stereo Vision," IEEE Trans. Patt. Ana. Machine Inte., Vo. 17, No. 9, pp , Sept [3] B. Horn, Robot vision, MIT Press, [] Y. C. Kim and J. K. Aggarwa, \Determining object motion in a sequence of stereo images", IEEE Journa of Robotics and Automation, Vo. 3, No. 6, pp , Dec [5] M. K. Leung and T. S. Huang, \An integrated approach to 3D motion anaysis and object recognition", IEEE Transactions on Pattern Anaysis and Machine Inteigence, Vo. PAMI-13, No. 10, pp , Oct [6] S.Z.Li, \On Dincontinuity-Adaptive Smoothness Priors in Computer Vision", IEEE Trans. Patt. Ana. Machine Inte., Vo. 17, No. 6, pp , June [7] A. Mitiche, \On kineopsis and computation of structure and motion", IEEE Transactions on Pattern Anaysis and Machine Inteigence, Vo. PAMI-8, No. 1, pp , Jan

16 [8] A. Mitiche and P. Bouthemy, \Tracking modeed objects using binocuar images", Computer Vision, Graphics and Image Processing, Vo. 3, pp , [9] A. N. Netravai, T. S. Huang et a, \Agebraic methods in 3D motion estimation from two-view point correspondences", Int. Journa of Imaging Systems and Technoogy, Vo. 1, pp , [10] A. Tamtaoui and C. Labit, \Constrained disparity and motion estimators for 3DTV image sequence coding", Signa Processing: Image Communication, Vo., pp. 5-5, [11] G. Tziritas and C. Labit, Motion anaysis for image sequence coding, Esevier, pp. 366+xxiv, 199. [1] A. M. Waxman and J. H. Duncan, \Binocuar image ows: steps forward stereo-motion fusion", IEEE Transactions on Pattern Anaysis and Machine Inteigence, Vo. PAMI-8, No. 6, pp , Nov [13] Z. Zhang and O. Faugeras, \Estimation of dispacements from two 3-D frames obtained from stereo", IEEE Transactions on Pattern Anaysis and Machine Inteigence, Vo. PAMI-1, No. 1, pp , Dec