Trinocular Stereo using Shortest Paths and the Ordering Constraint

Transcription

1 Tinocula Steeo using Shotest Paths and the Odeing Constaint Motilal Agawal and Lay S. Davis Deatment of Comute Science, Univesity of Mayland, College Pak, MD 20742, USA Abstact This ae descibes a new algoithm fo disaity estimation using tinocula steeo. The thee cameas ae laced in a ight angled configuation. A gah is then constucted whose nodes eesent the individual ixels and whose edges ae along the eiola lines. Using the well known uniqueness and odeing constaint fo ai by ai matches simultaneously, a ath with the least matching cost is found using dynamic ogamming and the disaity filled along the ath. This ocess is eeated iteatively until the disaity at all the ixels ae filled u. To demonstate the effectiveness of ou aoach, we esent esults fom eal wold images and comae it with the taditional line by line steeo using dynamic ogamming. Keywods: comute vision, tinocula steeo, dynamic ogamming, odeing constaint 1. Intoduction Establishing coesondences is the key oblem in 3D econstuction fom steeo images. The goal of coesondence is to assign matches to each oint in the efeence image. This is done using a measue of similaity based on the intensities at the oints. Steeo coesondence methods may be eithe featue based o ixel based. In featue based coesondence methods, featues such as intensity edges ae extacted and matched fist. The similaity measue used in this case ae coelation windows centeed aound the featues [10]. Subsequently, coesondence fo the emaining oints is detemined by inteolation and knowledge of scene geomety. Pixel based coesondence methods diectly find the coesondence fo each ixel using the ixel intensities. Lately, ixel based coesondence methods have gained oulaity[7, 3, 14]. Pixel based methods have the advantage of giving dense deth mas. When combined with additional constaints and assumtions about the scene, it can yield vey accuate esults. One such widely used and well undestood constaint is the eiola constaint. The eiola constaint aises fom the geomety of the cameas. This constaint esticts the match of a oint in the efeence image to lie along the eiola line in the othe image. It educes the seach sace to a line but the oblem still emains intactable because of the exonential size of the seach sace. This has made it necessay to use additional assumtions about the natue of the scene. Ohta & Kanade and Cox et al. [11, 7] make the uniqueness and odeing assumtion. Uniqueness states that each oint in the efeence image has a unique match, and the odeing constaint equies that the ode of matches is eseved along coesonding eiola lines. Unde these additional assumtions, the coesondence oblem can be solved efficiently using dynamic ogamming fo each eiola line. The odeing constaint is based on the hysical assumtion that scenes ae smooth along eiola lines. Such scenes would not only be smooth along eiola lines, but also acoss eiola lines. Theefoe the matches on adjacent eiola lines ae not indeendent. Ohta and Kanade [11] addess this issue fo the case of featue based coesondence by extacting connected edge segments and efoming an elegant 3D dynamic ogamming. Agawal et al [2] fomulate the oblem as a two stage dynamic ogamming. In the fist stage, dynamic ogamming is used to obtain K best solutions fo each scanline. In the second stage anothe seach is efomed to find solutions fo each ow with maximum smoothness between adjacent eiola lines. Roy & Cox, Ishikawa & Geige [14, 9] have genealized the one dimensional odeing constaint to a two dimensional local cohesivity constaint and fomulated the oblem as finding a minimum cut though a gah. The local cohesivity assumtion means that the disaities tend to be locally simila in all diections and thus acoss eiola lines as well. Howeve this genealization ignoes the stonge eiola constaint. Moe ecently, Zabih et al. [6] have used gah cuts to find local minimum (in a stong sense) of enegy functions that eseves discontinuity and

2 alied it to a ai of steeo images. In this ae, we addess the issue of inteaction between eiola lines though the addition of one moe camea. Using this tinocula camea configuation we fomulate the oblem as a shotest ath oblem and solve it efficiently using dynamic ogamming techniques. One of the key featues of ou fomulation is that it allows us to conside the uniqueness and the odeing constaint in a thee camea setu. Tinocula steeo, and in geneal multiview steeo, has been studied befoe. In [12], the matching is done seaately on the two ais of images and the esults ae meged using elaxation. [15] intoduces additional tinocula constaints and combines the views using a connectionist netwok elaxation algoithm. In [13], featue based matching acoss thee widely seaated views is efomed which simultaneously oututs the tifocal tenso[8] between the views. Point featues such as cones ae detemined and matched acoss the views using a local coelation based similaity measue augmented with a local homogahy estimation. The homogahy ovides the ma between inteest oint neighbohoods fo the coss coelation affinity measue and small baseline steeo can then be subsequently alied. This is then used to estimate the tifocal tenso. Howeve, fo ou algoithm, we assume that the cameas ae aleady calibated and theeby the tifocal tenso and also the camea matices ae known. The oganization of the est of the ae is as follows. Section 2 descibes the geomety of the tinocula steeo and section 3 esents the outline of the algoithm. The fomulation of tinocula steeo as a shotest ath oblem is esented in section 4. In section 5 we esent the esults of ou algoithm on eal images and comae it with a standad widely used steeo algoithm based on dynamic ogamming on single scan lines. 2. Tinocula Steeo Famewok Figue 1 shows the tinocula steeo configuation used. The thee cameas ae laced on the vetices of a ight angle tiangle. This ensues that the eiola lines fo the cente image coesonding to the ight and the to cameas ae mutually eendicula. Each oint in the cente image has disaities and with efeence to the ight and to cameas esectively. Since, in this ideal case the baseline distances of the to and the ight camea elative to the cente camea ae the same, the aangement does not incease the accuacy of the disaity of the oints but hels in educing the eo. Hence it does not matte whethe we efe to disaity o. Fo the emainde of the ae, will denote the disaity with efeence to the ight camea. We assume that the thee cameas ae fully calibated, so that each disaity value of a ixel in the cente image coesonds to a location,, on the line joining the C t to image (It) cente image (Ic) ight image (I) Figue 1. Tinocula Steeo Geomety camea cente to the oint on the image lane. The oint can then be ojected in the ight and to cameas to give the image oints and esectively. In the absence of full calibation, the tifocal tenso [8] can be comuted fom a few known matches in the thee images which can then be used to tansfe oint coesondences between the cente and the ight image to the to image. In eithe case, the fundamental matices fo the image ais can then be easily ecoveed. Unde the assumtion that the image intensity is indeendent of the viewing diection (i.e. the suface is lambetian), the ixel intensities at oints, and in the cente, ight and to images should be identical. Theefoe we define the eo in matching as! "$#!% & (' & )& (' P & (1) This eo function can be used to efom the matching and ideally it will be zeo fo a coect match. Note that since we oject the oint in all the thee views, the cost function takes account of all the thee views simultaneously. This fomulation assumes that the oint is visible in all the thee images. If the oint is occluded in one o moe of the thee views, then the eo will be lage. In that case, the oint will be null matched(occluded) by this cost function. Fomulation of an eo function fo multile views is challenging due to the esence of occlusions[1]. In this ae, we use the simle fomulation fo eo function above and focus on contol issues in tinocula steeo.

3 9 9 = = 3. Algoithm Outline 2. Obtain the matches fo oints on this ath Since the goal of the coesondence seach is to minimize the oveall eo in matching, this citeia can be used to ecove the best ath also. Theefoe the best ath is the ath along which the eo of matching is minimum. In the next section, we illustate how the best ath can be ecoveed using dynamic ogamming in 3D. 4. Shotest Path Fomulation Figue 2. Cental Image - Test set 0 Figue 2 is the cental image fom ou tinocula steeo setu. The eiola lines in the cente image fo the ight and to cameas ae also maked at the image boundaies. Although ou imlementation does not equie the eiola lines to be exactly hoizontal o vetical(which is had to achieve), in the discussions below, to simlify the exosition, we will assume that the eiola lines in the cente image fo the ight and to cameas ae hoizontal and vetical esectively. Suose a oint in the cente image matches a oint in the ight image and let be the coesonding oint in the to image. Let be a oint to the left of on the same ow and ( be to ight of. Also let be a oint above in the same column and be below. Then by the odeing constaint 1. Along the hoizontal eiola line, can have a match only to left of in the ight image and ( can match to a oint to the ight of. 2. Similaly fo the vetical diection, matches fo the oints and must esectively lie above and below in the to image. Matching the hoizontal o the vetical eiola lines indeendently ignoes the odeing constaint in the othe diection. Theefoe the hoizontal and vetical eiola lines should not be matched indeendently. Instead, we oose to match along an inteleaving ath though the image consisting of hoizontal and vetical eiola lines. The hoizontal odeing constaint will be eseved along the hoizontal otions of the ath and the vetical odeing constaint will be eseved along the vetical otions. Thus thee ae two suboblems 1. Identify the ath along which to comute disaities In ode to find the shotest ath, we teat the cente image as an undiected gah. The individual ixels fom the nodes of this gah. The edges of this gah link each node to its fou neighbos. Note that all the thee oints ', and lie on the hoizontal eiola line. Similaly the oints '), and ) lie on the vetical eiola line. We also need to identify the oigin and destination nodes so as to define the endoints of the ath we wish to find. The oint is taken as the oigin and all the oints on the last ow and last column ae the destination nodes. If thee ae disaity levels, each oint in the cente image has ossible disaities. Coesonding to each disaity, we have an eo, which is the cost of assigning disaity to. This eo is comuted using equation 1. In addition may be occluded in one o moe views, we denote this by! and the coesonding cost by "$#%#. Similaly fo oints in the ight and to images, let "$#%#'& and "$#%#'( be the cost of thei being occluded. Let ) ' and '*). Refe to figue 3. We can aive at eithe hoizontally fom ) o vetically fom, which coesonds to a hoizontal and vetical edge esectively. Conside the hoizontal edge fom ),+- to This edge is valid if it eseves the hoizontal odeing constaint in the ight image. i.e. the ojection of the 3D oint ) +- lies to the left of the ojection of the 3D oint coesonding to in the ight image. In figue 3, the edge ).- is invalid. Similaly the vetical edge fom! + is valid if it eseves the vetical odeing constaint in the to image. Each edge has a cost associated with it. Let,/01& ) $+-! be the cost of the hoizontal edge fom ) +- and,/0,(! + the cost of the vetical edge. They ae defined as ) +-! : & ' +- &';<"$#%#%& valid edge?> othewise + & ' A@! : &;B"$#%#'( valid edge othewise

4 Oigin Best Path Coesonding Matched Path q j (q, d q ) d j - d i q i (, d ) x (q, d ) invalid hoizontal edge fom q Cente Image Destination Figue 3. Valid and invalid edges Right Image In othe wods, the cost of an edge which does not eseves the odeing constaint is infinite. Fo a valid edge, the cost is the eo lus a tem which accounts fo the cost of occlusion in the othe image. i.e. if has disaity and ) has disaity +-, then oints between the ojections of and ) +- in the ight image will have no match in the cente image as illustated in figue 3. The otimum cost of taking a hoizontal edge fom ) to is denoted by " & and the otimum cost of taking a vetical edge fom to is denoted by " 8(. Finally, the cost at a node, " is defined as the minimum of the cost of otimum hoizontal o vetical edges. They ae defined ecusively as follows " & " +! " ( " +! " " ) +-,/01& ) +-! "! + :,/0,(! + " " & " ( (6) We scan the image to to bottom, left to ight and at each vetex calculate the costs fo all 3D nodes,. At each node, we also kee tack of the edge which gives us the minimum in equation 6 above and the coesonding hoizontal o vetical node(deending on the diection of the minimum edge). This hels us backtack to the best ath and simultaneously obtain the matches fo oints along that ath. Note that if we estict ou edges to be only vetical o only hoizontal, this would coesond to the single scan line dynamic ogamming solution. At the end of the scanning, we have the cost of all the 3D nodes fo the destination oints. Backtacking fom the destination node with the least cost will thus simultaneously ecove the best ath and disaity of the oints along that ath. This, howeve, gives undue advantage to aths which have fewe numbe of oints. So, we scale the costs of the destination nodes by the actual numbe of oints along that ath and backtack fom the node with the minimum scaled cost Recoveing the dense disaity ma In ode to ecove the disaity at all oints, afte the ath with minimum cost is found, at the next stage we find the next best ath and fill disaities along that ath. Howeve, the matches obtained along the best ath constain matches in othe egions by the hoizontal and vetical odeing constaint. This altes the costs of all othe aths. Hence the costs of all aths need to be ecalculated while taking these constaints into account. In fact, the oiginal second best cost may now even violate the odeing constaint and consequently its cost may now be infinite. In figue 4, oint is matched to in the ight and in the to image. Theefoe matches of all oints, ' must lie to left of in the ight image and matches fo all oints, must lie below in the to image. The oiginal second best ath is. Clealy, violates the odeing constaint at node. Afte the constaints fom the best ath ae taken into account, the new second best ath is and eseves the hoizontal odeing constaint in the ight camea. This ocess is eeated until disaities at all oints ae found. When disaities at all the cuent destination oints ae detemined, the oints in the hoizontal and vetical eiola lines eceding the cuent destination oints become the new destination oints. This is shown by the dotted lines in figue 4.

5 a v v a Cente Image Best ath B Second best ath C New second best ath S Figue 4. Finding next best aths Right Image 4.2. Running Time Fo a image with ixels and disaity levels at each oint, the unning time duing each stage of the algoithm is ). In the wost case, we may have to go though stages of the algoithm. (Note that fo this to haen each new ath that is added will find disaity at one additional new oint.) Theefoe the wost case unning time of the algoithm using thee views is. The dynamic ogamming aoach on seaate eiola lines equies a total unning time of fo a ai of images. Fo eal images of size 320x240, on a entium 400 Mhz machine, the unning time is about one hou. While this is cetainly much slowe than dynamic ogamming on seaate eiola lines, the unning time is comaable to othe algoithms which use maximum flows and gah cuts [5, 4, 14, 9] fo inteactions between eiola lines. 5. Exeiments & Results In this section we comae the esults of steeo obtained by ou method with a standad steeo algoithm. This standad algoithm uses the ight image fo efoming line by line dynamic ogamming as descibed in [7]. But since we have thee images, we include the to image also fo the cost function. Theefoe, the cost function used fo otimization in both the algoithms is the same and they diffe only in the contol stategy of how the disaities ae filled. In ou algoithm, we simultaneously enfoce the odeing constaint along both diections, wheeas the standad steeo enfoces the odeing constaint only along the hoizontal diection. Figues 2, 6(a) & 7(a) show the cente image fo thee image sequences. The eiola lines fom the to and the ight camea have been ovelayed fo efeence and define the egion within which the disaities ae obtained. Figues 5(a), 6(b) & 7(b) ae the disaity mas obtained by standad steeo as discussed above and figues 5(b), 6(c) & 7(c) ae the esults of ou algoithm. Note that the tile like atifacts in the mas ae due to the fact that the eiola lines ae not exactly hoizontal and vetical. This made it necessay to esamle the image in ode to constuct the gah. The ange of disaities in these mas ae fom 10 to 70, and highe intensities coesond to lage disaities. Dak egions in both mas coesond to egions which wee null matched (occlusion). Visual comaison of the the disaity mas show that the standad steeo algoithm tends to oduce a smeaing effect, esecially along edges which ae eendicula to the eiola lines. (In this case vetical edges, since the hoizontal eiola lines have been used.) On the othe hand, ou algoithm is fee of such atifacts. Fom 6(c), it is clea that ou algoithm does not unduly smooth the disaity ma, but eseves sha discontinuities. Also woth noting is the fact that the occluded egions ae coectly identified to lie along the contous of the esons, as exected. Evaluation of steeo algoithms in the absence of gound tuth is a difficult task. Theefoe, we took fou tinocula steeo sets and chose andom oints which wee not occluded(non-null matched) by both methods and veified the matches obtained by hand matching them. Table 1 shows the accuacy atio fo each of the two methods. (A oint was deemed as coectly matched if it was within a distance of two ixels fom manually matched oints) These figues suot ou claim of inceased accuacy ove the standad steeo method.

6 6. Conclusion In this ae, we esented a new algoithm fo tinocula steeo. Using the well known odeing constaint simultaneously in both ais of views, we fomulated the oblem as a shotest ath algoithm. This detemines the ath along which the disaity is filled fist which futhe constains the disaity at the othe locations. The shotest aths ae calculated again and the ocess is caied out eeatedly until disaities at all ositions ae detemined. Exeiments with eal images show imoved accuacy ove single line by line steeo using dynamic ogamming. A majo diection fo futue eseach is to incooate into the algoithm a scheme fo taking into account eos in matching fom the evious stages. Since ou only assumtion is the esevation of the odeing constaint, and at each stage we take the minimum cost ath as the ath along which to detemine disaities, the algoithm cuently has some degee of eo coection built into it. But it does not guaantee that goss eos in the initial stages will not be oagated. Othe aeas of futue eseach include imoving the cost function and also genealizations to moe than thee views. Acknowledgement This eseach has been suoted in at by National Science Foundation gant numbe NSF-EIA and National Institute of Health unde the human bain oject gant numbe NIH The authos would also like to thank the anonymous eviewes fo thei feedback. [6] Y. Boykov, O. Veksle, and R. Zabih. Fast aoximate enegy minimization via gah cuts. In Poc. Intenational Confeence on Comute Vision, ages , [7] I. Cox, S. Hingoani, B. Maggs, and S. Rao. A maximum likelihood steeo algoithm. Comute Vision and Image Undestanding, 63(3): , May [8] R. Hatley and A. Zisseman. Multile View Geomety in Comute Vision. Cambidge Uinvesity Pess, [9] H. Ishikawa and D. Geige. Occlusions, discontinuities, and eiola lines in steeo. In Fifth Euoean Confeence on Comute Vision, LNCS 1406, ages Singe Velag, June [10] T. Kanade and M. Okutomi. A steeo matching algoithm with an adative window: Theoy and exeiment. PAMI, 16(9): , Setembe [11] Y. Ohta and T. Kanade. Steeo by inta and inte-scanline seach using dynamic ogamming. IEEE Tansactions on Patten Analysis and Machine Intelligence, 7(2): , [12] Y. Ohta, M. Watanabe, and K. Ikeda. Imoving deth ma by ight-angled tinocula steeo. In ICPR86, ages , [13] P. Pitchett and A. Zisseman. Wide baseline steeo matching. In Poc. 6th Int l Conf. on Comute Vision, ages , [14] S. Roy and I. Cox. A maximum-flow fomulation of the n- camea steeo coesondence oblem. In Poc. 6th Intl. Confeence on Comute Vision, ages , [15] C. V. Stewat and C. R. Dye. The Tinocula Geneal Suot Algoithm: A Thee-Camea Steeo Algoithm fo Ovecoming Binocula Matching Eos. In Poc 2nd Int. Conf. on Comute Vision, ages , Refeences [1] M. Agawal and L. S. Davis. A obabilistic famewok fo suface econstuction fom multile images. In Poceedings IEEE Confeence on Comute Vision and Patten Recognition, Lihue, Hawaii, Decembe [2] M. Agawal, D. Hawood, R. Duaiswami, L. Davis, and P. Luthe. Thee dimensional ultastuctue fom tansmission electon micoscoe tilt seies. In Poceedings Indian Confeence on Vision Gahics and Image Pocessing, Bangaloe, India, Decembe [3] S. Bichfield and C. Tomasi. Deth discontinuities by ixelto-ixel steeo. In Poceedings Sixth IEEE Intenational Confeence on Comute Vision, ages , Mumbai, India, Januay [4] S. Bichfield and C. Tomasi. Multiway cut fo steeo and motion with slanted sufaces. In Poceedings of the Seventh Intenational Confeence on Comute Vision, ages , Set [5] Y. Boykov, O. Veksle, and R. Zabih. Makov andom fields with efficient aoximations. In Poceedings of the IEEE Confeence on Comute Vision and Patten ecognition(cvpr), ages , 1998.

7 Table 1. Accuacy Comaison of the matches obtained by the two methods Index Total Num. Points Standad Steeo Shotest Path Steeo Set % 95.7% Set % 95.5% Set % 95.9% Set % 93.0% (a) Standad Steeo (b) Shotest Path Steeo Figue 5. Disaity mas fo cental image of test set 0 (Fig 2) (a) Cente Image (b) Standad Steeo (c) Shotest Path Steeo Figue 6. Test set 1 - Disaity mas (a) Cente Image (b) Standad Steeo (c) Shotest Path Steeo Figue 7. Test set 2 - Disaity mas