Boundary Preserving Joint Estimation of Optical Flow and Disparity in a Sequence of Stereoscopic Images

Transcription

1 Bounday Peeving oint Etimation of Optical Flow and Dipaity in a Sequence of Steeocopic Image H. Weile, A. Mitiche, A. Manoui INRS-EMT Place Bonaventue 8 ue de la Gauchetiee O. Suite 69 Monteal, Quebec H5A 1K6 Abtact The pupoe of thi tudy i to invetigate a new method fo the joint etimation of optical flow and dipaity in a equence of teeocopic image. Fo a equence of two pai of teeocopic image, the poblem involve the etimation of fou field, two field of image motion and two of dipaity. Steeocopy elate the fou field, and thee i no elation between any thee of thee field except though the fouth. We peent a new method whee thee of the field ae etimated independently by aniotopic diffuion to peeve thei boundaie, and the fouth deduced uing the integability containt. Neithe patial no tempoal coepondence i explicitly efeenced. Extention of the cheme ove time equie the etimation of only two field and an application of the integability containt at each time tep. When motion o dipaity in any of the field i of lage extent, the cheme i activated via multieolution. Seveal expeiment confim the advantage of the method both in tem of accuacy and peed of execution. Keywod: Machine viion, motion, dipaity, etimation 1 Intoduction Optical flow and dipaity play a cental ole in eveal image poceing and compute viion poblem uch a coding, image egmentation, motion tacking, and image thee-dimenional intepetation. Thee ae fundamental poblem in ueful application uch a obotic, uveillance, image databae etieval, and video analyi. Optical flow and dipaity have each been ubject to numeou tudie [1][3][2]. Becaue the two field ae elated in teeocopy [4][5], computational benefit may accue fom thei joint etimation [6] [7][8][9][1]. Implementation of joint etimation involve detemining fou field: two motion field and two dipaity field. The teeokinematic containt [4] [5] elate thee of thee field to the fouth. Thee i no elation between any thee of thee field except though the fouth. Theefoe, joint etimation can be done accoding to one of two baic paadigm: (1) The fou field ae etimated imultaneouly uing, fo intance, a vaiational fomulation to minimize an objective function containing tem of confomity to data, egulaization tem of moothne, and a tem to include the teeokinematic containt, (2) thee of the field ae etimated independently, each accoding, fo intance, to a vaiational fomulation a in [1][2], and the fouth field i etimated uing the teeokinematic containt. Etimation along the fit paadigm involve olving a vey lage ytem of equation, nonlinea becaue of the teeokinematic containt tem. Fo intance, thee ae vaiable to etimate with image of ize. Extending the poce ove time equie olving each time uch a lage cale ytem of equation. Etimation along the econd paadigm involve olving independently thee much malle poblem to etimate thee field, followed by an application of the teeokinematic containt to etimate the fouth field. Extending the etimation ove time equie etimating only two field and an application of the teeokinematic containt. Thi, theefoe, affod ignificant computational aving. Peviou tudie have adopted hybid paadigm with no obviou fomal jutification, and have in common that they efeence eithe tempoal o teeocopic coepondence explicitly, equiing the intevention of extenal pocee uch a each o intepolation [6] [7][8][9][1]. In thi tudy, we tate the poblem accoding to the the econd paadigm and in a way that fogoe the need fo explicit efeence to coepondence. Motion and dipaity of mall extent i teated diectly. In the peence of motion o dipaity of lage extent, joint motion-dipaity multieolution etimation i done by a coae-to-fine algoithm we peent in Section 3. The emainde of thi pape i oganized a follow. Section 2 tate the poblem and decibe it eolution. Section 3 explain ou implementation. Section 4 give expeimental eult. Section 5 contain a concluion. 2 Fomulation We define an image equence a a eal function ove a domain, whee i an open ubet of epeenting the patial coodinate domain, i an inteval of time, and! an inteval of R. An image i,

2 e D G +, e p theefoe, a map: "$# %! & (') * +,-') "/. *!+, Fo a fixed value of +213 we obtain a tempoal equence of image and fo two ditinct fixed value + we obtain a teeocopic image equence. Vaiable can be thought of a the paamete of the tajectoy of a equence of image plane in thee plane coodinate domain. Thi definition of an image equence genealize the notion of a teeocopic. image equence. Let *!+,41 5! 5 and *7698. * + 6 : +&, 6 :,. + it coepondant at 6 : +, 6 :,, whee 8 deignate diplacement. The aumption that " doe not change at coeponding point, ". *;6<8. * =+ 6 : +>, 6 :,?=+ 6 : +>, 6 ". * =+>, yield the following motion and dipaity containt: A<"CBED 6 A<"CBHG 6 whee A<" w i the optical velocity vecto, d the dipaity vecto, the " " patial gadient, " F " I of, + and the patial deivative of with epect to and. Becaue 8 8 we alo have the integability containt: (1), (2) Containt (2) i the continuou fom of the teeokinematic containt intoduced in [4][5] fo teeocopic image equence. It paticulaity i that it efeence neithe tempoal no dipaity coepondance explicitly. Conide two conecutive "LKNM F teeocopic image of a "LO&M F teeocopic, equence: "LKNM FQP, the left and ight image " O&M FQP at time, and, the left and ight image at ube-,r D$KNM F &U KQM F DVOM F U O&M F quent time. Let and G F Z X F G FQP, Z X FQP,[R., deignate left and ight optical velocity at time,, and deignate dipaity at and Containt (1) and (2) ugget that the left and ight motion field at time, and the dipaity field at time, can be etimated independently befoe etimating the dipaity field at time,r uing the integability-teeokinematic containt. The left and ight motion field and the dipaity field ae etimated at time, a: D KNM \]!^`_badc gfihkj.ya<" KNM F BED 6 " FKNM F 6ml.!nEAoS`n 62l.!nEA U n D O&M \]!^`_badc gfihkj.ya<" OM F BED 6 " FO&M F 6ml.!nEAoS`n 62l.!nEA U n G \]!^`_badc f hkj.ya<" KNM F BHG 6 " IKQM F 6ml.!nEA<X Z n 6ql.!nHA<X n j whee i a poitive contant to weigh the contibution of the fit tem of the integand (confomity to data) againt the othe two tem (egulaization), and l i a quadatic function a in [1] o a bounday peeving function a in [2]. The Eule-Lagange equation of the fit poblem in (3) ae: j " KNM F.T" KNM F S KQM F 6 " KNM F U KQM F 6 " FKNM F t@vuwnx. l R.nEAoSWKNM F n neaos KQM F n AoS KNM F j " KNM F.T" KNM F S KQM F 6 " KNM F U KQM F 6 " FKNM F t@vuwnx. l R.nEA U KNM F n nea U KQM F n A U KNM F with the bounday condition on : l R.!nHAoS/KNM F n S/KNM F nhays KNM F {z l R.!nEA U KNM F n U KNM F nea U KNM F {z whee n i the unit nomal to. Simila equation can be witten fo the othe two poblem in (3). The Eule-Lagange "LKNM F equation ae dicetized uing " KNM FQP the left image and olved iteatively [1] [2]. A imila pocedue applie fo the etimation " OM F "LO&M offqp the ight motion field uing the ight image and, and fo the etimation " KNM F "LOM F, of the dipaity field at time uing the image and. When the left and ight, motion field and the dipaity,!r field ae etimated at time, the dipaity field at time i deduced uing the integability-teeokinematic containt. Fo motion and dipaity of mall extent, thi done imply by finite diffeence appoximation of the deivative in (2): G D O&M F} D KNM F 6 G F It i impotant to note that extending the etimation ove time equie etimating only two field and an application of the integability containt at each time tep. Thi, theefoe, affod ignificant computational aving. It i alo impotant to note that the vaiable in (4) ae all evaluated at the ame patial coodinate. Thee i explicit efeence to neithe tempoal no dipaity coepondence. Explicit efeence to coepondence would call fo the intevention of complex extenal pocee uch a intepolation to bing evaluation on gid point of the image poitional aay. Howeve, diect iteative eolution of a dicetized veion of the Eule-Lagange equation, and the ue of the integability containt in the fom (4) i valid fo motion and dipaity of mall extent. In the peence of eithe motion o dipaity of lage extent, the etimation of the field in (3), and the application of the integability containt (4) (3) (4)

3 ae pefomed at each level of a multieolution pyamid accoding to a cheme decibed in Section 3. Fo the etimation poblem in (3), we ued the Aubet function, l.t~ b@ L 6 ~ }. With thi function, the egulaization tem of the functional in (3) ha the following effect: In egion whee motion (dipaity) i homogeneou, moothing i done by iotopic diffuion, a with the Hon and Schunk method. At motion (dipaity) boundaie, howeve, and unlike with the Hon and Schunk method, moothing i done by aniotopic diffuion to peeve thee boundaie becaue it i favoed along motion boundaie and inhibited aco [2]. When dicetized, the Eule-Lagange equation coeponding to poblem (3) yield lage cale pae ytem of nonlinea equation that can be olved efficiently via the half-quadatic algoithm [2]. The algoithm can be hown to convege to a unique olution. 3 Multieolution joint etimation All multieolution appoache have in common the ue of pyamid of image. Thee pyamid ae obtained by image filteing and ubampling. We built uch a pyamid uing the aniotopic diffuion Peona-Malik [11] filte to peeve image boundaie, and ubampling by 2. oint etimation i then pefomed on the pyamid uing a coae-tofine tategy. Etimation tat at the coaet level whee motion and dipaity have an extent mall enough to validate a vaiational, gadient-baed independent etimation of thee of the field, and the deduction of the fouth by the integabilty containt. At any ubequent fine level, a coection to the etimate of the peviou level ae calculated uing the integability containt and the ame motion (dipaity) vaiational gadient-baed etimation algoithm. Thi coae-to-fine algoithm i given in Figue 1. 4 Expeimental veification To veify ou method, we an eveal expeiment two of which we decibe hee. The cene in thee expiment conit of a backgound and two object which ae cut out of eal image. Motion and dipaitie ae ynthetic to povide gound tuth fo the pupoe of quantitative evaluation. The equence in the fit expeiment conit of two pai of binocula image, the econg diplayed in Figue 2. The backgound and the object ae given dipaitie in the fit pai of binocula image and ae made to move uch that dipaitie in the econd pai of binocula image ae (-1,) fo the backgound, and (1,1) fo the two object. Thi fit expeiment i to tet the method with motion and dipaitie of mall extent. In the econd expeiment, the equence conit of ten pai of binocula image, the lat one diplayed in Figue 3. We ued dipaitie and motion to poduce dipaitie of lage extent in the lat (tenth) pai of binocula image. Thee ae (-1,) fo the backgound, (- O.Y,?YƒW /* Vaiable and ymbol */ : velocity of point p at time t on level l in the ight image : velocity in the left image : dipaity between both image : bightne in the ight o left image K.T,?YƒW :.T &,?Tƒ{ " O " K.Y &,?TƒW : O : :L: K : coection of the velocity.y,?yƒw N d : coection of the dipaity N ˆ : : lowet and highet level in the pyamid, : ampling ate between two image " R O " R K : image at time t+dt, ceated by moving " R : "/., image with flow field w(t) : image " at time t+dt, ceated by moving image O., S Š with dipaity field d(t) : Aubet/Konpobt/Deiche algoithm [2] /* Etimation on lowet level */ O K :.Y Œ S{Šg.Y"/.Y d &,? "W.T, 6 :,& :.T S{Šg.Y" K.T,E " O.T,& :.T d, 6 O.Y d &, } K.Y d &, 6 :.Y Œ, /* Q d of flow dipaity N ˆ */ fo ( TO ) K O #.Y :.T :.Y,?g " K " O # " R.Y 6, 6 "W.T 6 &, 6.Y 6, " R :.T 6 " O.T 6 &, 6 :.T 6 &, : K : O # :.T 6 S{Šg.Y" R.Y 6, 6 :, "W.T 6, 6 :,& :L:.Y S{Šg.Y" R :.T 6,E " K.Y 6,& /* oint etimation of dipaity and flow */ :L:.Y 6, 6 : O.T 6 &, } : K.T 6, 6 : :.T 6 &, :., 6 :., 6 :, 6 : :., 6 :, /**************************************/ K O #.Y 6, 6 :.T 6 &, :.T :.T 6 &, 6 :L:.Y 6, Table 1. Coae-to-fine joint etimation of optical flow and dipaity 17,-12) fo the object on the left, and (-2,12) fo the othe. Although dipaitie in thee expeiment may not coepond to any kind of eal life cene o camea, the geneated equence do povide adequate tet data, with gound tuth againt which we can contat the etimate of the method. We how the eult of the etimation of dipaitie fo the lat binocula pai of image. Thee eult, hown gaphically in Figue 2 and 3, and quantitatively in Table 2, indicate the efficiency of joint etimation by the method. The top box of Table 2, labelled Actual, how the gound

4 tuth of dipaitie fo the backgound (labelled BG ) and the left and ight object (labelled Obj 1 and Obj 2, fo both expeiment (labelled Expmnt 1 and Expmnt 2 ). epectively). The middle box of the table contain the eult obtained with the joint etimation method, applied incementally via the integability containt. The lowe box (labelled Aubet ) how the eult obtained by diect etimation. Note that fo the econd expeiment, the dipaitie ae jut too lage to be etimated diectly, even with the aitance of multieolution. A can be een fom thee example, the joint etimation method pefom well, and bette than independent, diect etimation, paticulaly in the cae of lage diplacement. Actual BG Obj 1 Obj 2. Expmnt Expmnt Expmnt Expmnt oint etimation BG Obj 1 Obj 2. Expmnt Expmnt Expmnt Expmnt Aubet BG Obj 1 Obj 2. Expmnt Expmnt Expmnt Expmnt Concluion Table 2. Expeimental eult The pupoe of thi tudy wa to invetigate a new method fo the joint etimation of optical flow and dipaity in a equence of teeocopic image. Thi method, which ue the integability containt, and multieolution in the peence of lage diplacement, ha given pomiing eult in a numbe of expeiment. [4] A. Mitiche, On Combining Steeopi and Kineopi fo Space Peception, Fit Intenational Confeence on Atificial Intelligence Application, Denve, CO, 1984, pp [5] A. Mitiche, A Computational Appoach to the Fuion of Steeopi and Kineopi, in W.N. Matin and.k. Aggawal (Ed.) Motion Undetanding: Robot and Human Viion, (Kluwe Academic Publihe, 1988), [6] A. Tamtaoui and C. Labit, Contained dipaity and motion etimation fo 3DTV image equence coding, Signal Poceing: Image Communication, 4, 1991, [7]. Liu and R. Skejanc, Steeo and motion coepondence in a equence of teeo image, Signal Poceing: Image Communication, 5, 1993, [8] Y. Altunbaak, A. M. Tekalp, G. Bozdagi, Simultaneou motion-dipaity etimation and egmentation fom teeo, Fit Intenational Confeence on Image Poceing, Autin, Texa, 1994, pp [9] I. Pata, N. Alveto, and G. Tziita, oint Dipaity and Motion Field Etimation in Steeocopic Image Sequence, Compute Science Dept, Univeity of Geece, FORTH-ICS TR-157, [1] Laganièe, R., Analye tééocinétique d une équence d image: Etimation de champ de mouvement et de dipaité, Ph.D. thei, INRS- Télécommunication, [11] P. Peona and. Malik: Scale-Space and Edge Detection Uing Aniotopic Diffuion, IEEE Tanaction on Patten Analyi and Machine Intelligence, 12(7), 199, Refeence [1] B. K. P Hon and B. G. Schunk, Detemining Optical Flow. Atificial Intelligence, 17, 1981, [2] G. Aubet, R. Deiche, and P. Konpobt, Computing optical flow via vaiational technique. SIAM ounal of Applied Mathematic, 6(1), 199, [3] A. Mitiche and P. Bouthemy, Computation and Analyi of Image Motion: A Synopi of Cuent Poblem and Method, Intenational ounal of Compute Viion 19(1), 1996,

5 .Y.T.>Š.YŠ.Y.T. : Figue 2. Expeimental eult, lage motion: a) left ide image, b) ight ide image, c) eulting dipaity field with oint etimation Figue 1. Expeimental eult, mall motion: a) left ide image, b) ight ide image, c) eulting dipaity field with oint etimation, d) eulting dipaity field with independent etimation