Dynamic Deph Recoery from Muliple ynchronized Video reams Hai ao, Harpree. awhney, and Rakesh Kumar Deparmen of Compuer Engineering arnoff Corporaion Uniersiy of California a ana Cruz Washingon Road ana Cruz, CA 9564 Princeon, NJ 8543 ao@soe.ucsc.edu {hsawhney, rkumar}@sarnoff.com Absrac his paper addresses he problem of exracing deph informaion of non-rigid dynamic 3D scenes from muliple synchronized ideo sreams. hree main issues are discussed in his conex: (i) emporally consisen deph esimaion, (ii) sharp deph disconinuiy esimaion around objec boundarie and (iii) enforcemen of he global isibiliy consrain. We presen a framework in which he scene is modeled as a collecion of 3D piecewise planar surface paches induced by color based image segmenaion. his represenaion is coninuously esimaed using an incremenal formulaion in which he 3D geomeric, moion, and global isibiliy consrains are enforced oer space and ime. he proposed algorihm opimizes a cos funcion ha incorporaes he spaial color consisency consrain and a smooh scene moion model. Inroducion he problem of recoering deph informaion using images capured simulaneously from muliple iewpoins has been exensiely sudied in he pas. In recen year wih adances in compuing and imaging echnologie capuring muliple synchronized high qualiy ideo sreams has become easier, and he problem of recoering deph maps of dynamic scenes using synchronized capure has receied increasing aenion [Vedula99, Zhang99, Carceroni]. his problem is ermed dynamic deph recoery in his paper. I can be considered as an exension of he radiional sereo compuaion problem where he deph soluion should make images consisen no only across muliple iew bu also across differen ime insans. A sraighforward approach for dynamic deph recoery is o apply a sandard sereo esimaion algorihm a each ime insan. A comprehensie surey on early sereo algorihms can be found in [Dhond89] while a shor lis of newer algorihms consiss of [Belhumeur96, Roy98, Boyko99, Kuulakos99, zeliski99, Lhuillier, Zhang, ao]. he principle underlying hese algorihms is o find a deph soluion ha opimizes an image mach measure across iews. We call his measure spaial mach measure. Howeer, his sraighforward soluion ignores wo new consrains presen in muli-iew image sequences. he firs consrain encodes he geomeric relaionship beween he 3D moion of a scene poin and is D projecions in muliple synchronized images. his relaionship, which is called he scene flow consrain, has been inesigaed in [Vedula99, Zhang99]. By applying his consrain, emporal D image correspondences can be used o infer 3D scene moion and herefore, consrain he deph informaion oer ime. o successfully apply scene flow consrain direcly in deph esimaion, he accuracy of he opical flow is crucial since he effec of unreliable flow a objec boundaries and in unexured regions will propagae ino he final deph map. More reliable resuls may be obained by esimaing parameric moion models in local image regions [Zhang99]. he second consrain arises from he obseraion ha objecs in he scene usually deform or moe smoohly. Applying his consrain helps o obain emporally consisen deph soluions and o eliminae ambiguiies ha usually can no be easily resoled a any single ime insan. Rigid [Zhang9, Adi85, Young9, Hanna93] and non-rigid parameric moion models [Zhang99] hae been employed in preious work. Local parameric moion models can also be esimaed for scenes wih arbirary non-rigid moions [Carceroni]. his paper proposes a dynamic deph recoery mehod in which a scene represenaion ha consiss of piecewise planar surface paches is esimaed wihin an incremenal formulaion. uch a represenaion can be deried based on color segmenaion of inpu images. he proposed formulaion inegraes consrains on geomery, moion, and isibiliy oer boh space and ime. More specifically, he scene surface corresponding o each homogeneous color region in he image is modeled as a 3D plane. he moion of his plane is described using a consan elociy model. he spaial mach measure and he scene flow consrain for his represenaion is inesigaed. Based on he obseraion ha he spaial mach measure depends only on he ou-of-plane moion of each planar surface, a dynamic deph recoery algorihm is deeloped. he proposed mehod enforces he moion model wihou explicily using he scene flow his work was performed while Hai ao was employed by he arnoff Corporaion. his maerial is based upon he work suppored by he Air Force Research Laboraory under Conrac number F36--C-43.
consrain and herefore aoids he propagaion of he errors in opical flow compuaion o deph esimaion. he global maching framework proposed in [ao] is adoped for enforcing he isibiliy consrain and also for iniializing and refining he deph soluion. As a resul, he isibiliy consrain is enforced boh across differen iews and oer differen ime insans. Muli-iew deph recoery of dynamic scenes Figure illusraes he configuraion of a muli-camera ideo capure sysem. Wihou loss of generaliy, we will firs deelop he formulaion for he moion of a single planar srucure. he formulaion can hen be direcly applied o handle a piecewise planar scene descripion. We assume ha boh he inrinsic and exrinsic parameers of he cameras are known. he camera exrinsic parameer namely he posiions and orienaions of he camera are represened wih respec o he camera coordinae sysem of a reference iew. he planar scene surfaces are also represened in his coordinae sysem. he calibraion marices of he M + cameras are K, =,..., M. he roaion and ranslaion (, ) = ( I pair R,) represens he camera pose for he reference camera and he pair R, ) represens he pose ( of he inspecion cameras {,..., M}. For ime insan, only he deph map in he reference iew is esimaed. In our represenaion, each of he homogeneous color segmens in he reference iew corresponds o a planar surface in he world. herefore, he deph map can be deried from he plane parameers ψ = n, d ], s [,..., ], where [, n is he normal ecor of he plane surface in segmen s a ime insan and d, is he disance of ha plane o he reference camera cener. H +, egmen s R +, + + + Reference iew H, H +, + Inspecion iew H, +, Figure. Dynamic deph recoery: synchronized image sequences of a dynamic scene are capured from muliple iewpoins. A any gien ime insan, relaing muliple iews of a scene plane leads o he spaial maching consrain. Mached image poins on he same scene plane across hese iews oer-consrain he plane's geomery. On he oher hand, relaing muliple iews of a scene plane oer ime leads o he planar scene flow consrain. Mached image poins on he same scene plane oer ime oer-consrain he plane's 3D moion and in urn is geomery. he goal of he dynamic deph recoery algorihm is o find a piecewise planar deph map ha opimizes a spaial mach measure and a emporal mach measure subjec o a moion smoohness prior.. paial mach measure Adoping he global maching framework [ao], a mach measure is compued as a funcion of he difference beween he real inspecion iews and he inspecion iews prediced based on he reference image and is deph map. Predicion is done by deph based forward warping in which he global isibiliy is handled using z-buffering. he warping funcion for each segmen is a homography deermined by is plane parameers. More specifically, a ime insan, for he reference iew I,, and he inspecion iew I,, =,..., M, he global mach measure is,, M E ( ψ ) = E( I, I ), () s= = where E I, I ) = g( I, fwarp( I, H )), and, (,,, g () is he sum of squared differences funcion. I should be noiced ha only pixels isible in he inspecion iews are considered. For segmen s, he forward warping funcion fwarp() warps he image of he segmen I, o inspecion iew hrough a homography H, induced by he plane model. his homography can be compued as n H s,, = K R + K, () d R where and are he relaie roaion and ranslaion beween iew and iew. his homography is deermined only by he normal ecor of he plane and he disance of he plane from he camera cener. herefore, i is only affeced by ou-of-plane moions since an in-plane moion leaes he plane normal and he disance unchanged.. cene flow consrain he scene flow consrain relaes he 3D moion of a plane and he resuling D opical flow fields or image correspondences. Le he plane of segmen s undergo a moion described by he roaion/ranslaion pair ( R ). he image projecions of he plane in he +, + reference iew beween imes and + are relaed by he homography: + n H s, +, = K R + + K. (3) d
Also, he homography beween he reference iew a ime and iew a ime + is gien by H H H, (4) +, ν =, ν +, ν, ν where he firs erm is he homography in iew for segmen s beween he wo ime insan which can be compued using (3) in he camera coordinae sysem of iew. he second erm is he spaial homography defined in he preious secion. he emporal mach measure E,s of he segmen s induced by he 3D moion of he plane can now be wrien as E, s ( R M = +, + ) = g( fwarp( I, H +, ), I +,. (5) ) his funcion compues he mach measure beween iew and before and afer he moion. Noe ha he emporal homography consrain for jus one iew is sufficien o sole for he planar moion parameers. herefore he aboe error funcion oer-consrains moion parameers of he plane. I is also clear ha he emporal mach measure is affeced boh by in-plane and ou-of-plane moions..3 Moion smoohness consrain Various moion models such as a consan elociy model or a consan acceleraion model can be employed o consrain he deph esimaion process. We adop a consan roaional/ranslaional elociy model in he local coordinae sysem of each segmen. his coordinae sysem is obained by ranslaing he origin of he reference camera coordinae sysem o he poin on he plane ha corresponds o he cenroid of he image segmen. Using an insananeous moion formulaion, in he Appendix, we show ha his model induces consan elociies in he normal ecor n and he plane disance d also. In oher word consan elociy model of for he moion of he plane resuls in a consan elociy model for is ou-of-plane moion parameers. Deiaions from his model are penalized hrough a moion smoohness cos funcion, which is inegraed in he deph esimaion process. For segmen s, he cos funcion is: E ( ψ o κ, d, + d, ) = +, d n n, + n + n +, =, +, n + ( d +, d, + δ), (6) where κ is he weigh for he second erm, and δ is a small posiie number for aoiding oerflow in diision. he inplane moion smoohness measure E i can also be formulaed. Howeer, since i will no be used in he proposed algorihm, he resul will no be shown here..4 Bach formulaion and incremenal esimaion Wih he aboe geomeric consrains and he moion model defined, esimaing he deph maps from ime o is equialen o finding he plane parameers ha minimize he cos funcion ε = α β = { γeo ( ψ,, ) + λei } s= E = s= ( ψ E, ( R +,, ) + + ) + where he consans α, β, γ, λ [,] are he weighs of each erm. Ideally, all he erms in he aboe funcion should be uilized for finding he opimal deph maps. Howeer, he second erm, i.e., he scene flow consrain, relies on he homographies oer ime which in urn depends on he accuracy of flow or piecewise parameric moion esimaion. I may become unreliable for unexured regions or segmens wih small spaial suppor. herefore, for he algorihm in his paper, we do no use he second erm and rely only on he spaial mach measure and he emporal smoohness of moion o compue he deph maps. he in-plane moion smoohness erm E i is also dropped for he same reason. In his mode, moion esimaes in erms of opical flow are sill uilized bu only for he purpose of esablishing emporal correspondences beween pixels or segmens. his correspondence is subsequenly employed o predic planar parameers using he moion smoohness consrain. uch a simplificaion lowers he requiremen for he accuracy of he opical flow because as long as he corresponding pixels are in he correc segmen, he errors in flow fields will no affec he predicion of he plane parameers. his adanage will be elaboraed furher in ecion 3. he simplified cos funcion (7) is gien by: ε( ψ,, ) = α γ s= = o E ( ψ E ( ψ,,,, ) ) + (7). (8) When he deph informaion before ime is gien or already esimaed, he deph a ime can be compued using an incremenal formulaion. he cos funcion consiss of he spaial mach measure a ime and a moion smoohness measure. More specifically, he cos funcion for he incremenal formulaion is gien by: ε s= = αe ( ψ ) + γ E( ψ, ψ ), (9),,
where ψ,, s is he prediced plane parameers based on he smooh moion model. he funcion E, ψ ) ( ψ, ψ,, represens he difference beween he wo planes s and, which can be compued as he aerage disance ψ beween poins inside a segmen. 3 Algorihm he block diagram of an algorihm based on he incremenal formulaion is shown in Figure. We firs briefly describe he main seps of he algorihm and hen discuss he deails of he algorihm in he following subsecions. he hree seps of he algorihm are: ψ,, ep : Predicing plane parameers s from he preious ime insans. his sep consiss of hree asks: () for each segmen, find he corresponding regions in he preious ime insan () find or esimae plane parameers in hose region and (3) predic he plane parameers a curren ime insan. ep : Iniializaion of he piecewise planar deph represenaion ψ, in he reference iew using he spaial mach measure E only. ep 3: Global deph hypohesis esing. For each segmen ψ,, s, choose eiher s or plane parameers in he iniial deph esimaion as is deph represenaion. Find a locally opimal soluion for he cos funcion in (9) using local search. A greedy hypohesis esing algorihm similar o he one proposed in [ao] is adoped for his purpose. 3. Predicing plane parameers In order o predic he plane parameers of a segmen, is corresponding posiions in he preious images need o be deermined. Based on he deph informaion in hose region he deph represenaion a he curren ime insan is prediced. We propose wo complemenary mehods for finding his emporal image correspondence. he firs mehod is based on emporal color segmenaion and is good for large homogeneous color regions; he second mehod is based on opical flow and works well for exured regions. 3.. egmenaion based mehod For large unexured scene surface he corresponding color segmens a differen ime insans end o hae large oerlapping areas when heir moions are relaiely small. his propery is exploied o rack corresponding segmens oer ime. he algorihm is illusraed in Figure 3a and is described as follows: For any large segmen s a ime insan - Projec he segmen o he nex frame, noed as f A ime insan, find he segmens ha saisfy he following crieria: - 85% of heir areas are coered by f - Hae similar color as he segmen s For hese segmens a ime insan, heir image correspondence a ime insan - is image segmen s. here will be segmens ha hae no correspondence in he preious frame according o he aboe mehod. For hese segmen he opical flow based mehod is applied o find image correspondences. emporal Deph Predicion paial Deph Iniializaion emporal Deph Hypohesis esing Figure. Block diagram of he proposed dynamic deph recoery algorihm. 3.. Opical flow based mehod For any pixel in segmen s a ime insan, is corresponding posiion in he preious frame can be found if he opical flow is aailable. his process can also be used o find is image correspondences in muliple preious frames by concaenaing he flow fields (Figure 3b). ince opical flow is only used o find corresponding regions of a segmen in he preious frames o fi a plane model, as long as he image correspondences are in he righ region, error will no affec he resuling plane parameers. In addiion, errors in opical flow only affec he emporal plane predicion, which is esed amongs a number of differen hypoheses. he erroneous ones will in general no surie. - - egmen s - egmen s f egmen s egmen s Figure 3. (a) Deermine emporal image correspondence based on oerlapping areas of color segmens. egmen s and segmen s a ime hae corresponding segmen s a ime -. (b) Deermine he corresponding pixels in he preious frames using opical flow. (a) (b)
3..3 Predicing plane parameers he image correspondences of a segmen in he preious frames are found eiher by he segmenaion based mehod or by he opical flow based mehod. he associaed plane parameers in he preious frames can hen be obained by using he plane parameers for he racked segmens from he former mehod, or by fiing planes o he deph alues of racked pixels from he laer mehod. Based on he plane parameers in he preious frame he plane parameers a curren ime insan can be prediced using he smooh moion model. For example, o predic plane parameers from wo preious frames and, if we denoe he plane parameers in hose wo frames as [ n,, d, ] and n, d ], he normal ecor a ime is [,, compued as he soluion of (, + n ) / n, + n = n, n () which is n s,, = ( n s,, n s,, I) n. he plane disance parameer is compued as d d d. =,, 3. Iniial deph esimaion he color segmenaion based sereo algorihm [ao] has been implemened o iniialize he deph soluion a he curren ime insan. his algorihm minimizes he spaial mach measure E and considers isibiliy implicily in he global maching crierion. he four seps in his algorihm are () color segmenaion, () correlaion based iniial deph compuaion, (3) plane fiing in each segmen based on he iniial deph, and (4) for each segmen, comparison of he iniial deph represenaion wih deph hypoheses from is neighboring segmen and selecion of he one ha improes E as he soluion. he esimaed iniial deph represenaion for segmen s is denoed as ψ,. (a) (c) (e) (b) (d) (f) 3.3 emporal deph hypohesis esing he hypoheses for each segmen s are he prediced plane parameers ψ s,,, he iniial plane parameers ψ,, and he iniial plane parameers of he neighboring segmens of s. Wih plane parameers of he oher segmens fixed, hese hypoheses are compared using he cos funcion in (9). his process is summarized as follows. A ime insan, for each segmen s in he reference iew Compue he cos funcion (9) for ψ s,,, ψ,, and ψ k,,, k neighboring segmens of s e plane parameer ψ, o be he one wih he lowes cos alue. (g) Figure 4. he dynamic deph recoery algorihm. (a,b) Original ideo frames 67 and 7. (c,d) Color segmenaion in frames 67 and 7. racked segmens are pained using he same color. (e) he magniude of he opical flow compued in frame 7. (f) he prediced deph map in frame 7. (g) he iniial deph map compued using he color segmenaion based sereo algorihm in frame 7. he deph alue of he area below he PingPong paddle is wrong. (h) By combining (f) and (g), he final deph map is compued. he errors in he area below he Ping Pong paddle are correced. (h)
4 Experimenal resuls he proposed mehod has been implemened and esed on synchronized ideo sreams colleced from an eigh-camera ideo capure sysem. All cameras fixae on subjecs abou nine fee away, wih an approximae angular separaion of degrees from each oher. he ideos are capured a 64x48 resoluion in RGB color a 3Hz. he inrinsic parameers and poses of all he cameras are compued using he camera calibraion algorihm of [Zhang]. Figure 4 shows he main seps of he proposed dynamic deph recoery algorihm. Figures 4a and 4b are wo frames capured from he reference iew a differen ime insans. he color segmenaion and he segmen racking resuls are shown in Figures 4c and 4d. Corresponding segmens racked using he mehod described in ecion 3.. are pained wih he same color. For segmens ha are no racked successfully, opical flow based mehod is used o perform pixel leel racking. he opical flow beween he curren frame and he preious frame is compued using a direc mehod [Bergen9]. he magniude of he flow field is shown in Figure 4e. Using he image correspondences compued from he segmenaion based mehod or he opical flow based mehod, he deph represenaion a he curren ime insan is prediced (Figure 4f). his prediced deph map is combined wih he iniial deph map compued using he color segmenaion based sereo algorihm (Figure 4g) o derie he final deph esimaion (Figure 4h). Noe ha he deph error in he highlighed area in he figure below he Ping Pong paddle in he iniial deph map is correced using he prediced deph map. Figure 5 shows a ime sequence of he dynamic deph recoery resuls. he deph maps are emporally consisen and sharp deph boundaries such as he accurae conours around fingers are presered. Compared o he single frame algorihm, improemens in emporal consisency hae been obsered from he resuling deph sequences. Unforunaely, he reducion in emporal scinillaion canno be easily shown in prin. Figure 6 shows he deph esimaion resuls of anoher sequence. he reader is paricularly direced o obsere he sharp delineaion of hin srucure like he finger from he background surface. 5 Discussions Based on a piecewise planar scene represenaion, we sudy he hree consrains applicable o he problem of dynamic deph recoery. he obseraion ha consan elociy moion of a plane causes consan changes in ou-of-plane moion parameers enables a simple algorihm ha enforces moion smoohness across ime wihou using he D opical flow field in he esimaion of 3D homographies and he resuling planar elociies. he opical flow is only used for finding emporal correspondences beween frames for he purpose of predicing plane parameers. ince in-plane moion does no change he prediced plane parameer more errors in he opical flow field can be oleraed. An algorihm based on he global maching framework is proposed. Experimenal resuls show ha emporal consisency in deph esimaion is achieed and sharp deph boundaries are presered. We are furher exploring how he planar consrain on segmens can be combined wih nonparameric dephs and wih oher smoohness consrains on he resuling surface shapes. We plan o ealuae he qualiy of he generaed deph maps by creaing synhesized ideos from noel iewpoins and assessing he spaial and emporal qualiy of he ideos. Figure 5. A deph map sequence esimaed using he proposed dynamic deph recoery algorihm. he lef column shows he firs hree deph maps and he righ column shows laer frames in he sequence. harp deph boundaries such as he finger conours in he firs image are presered een hough he opical flow is blurry. Appendix If a planar scene undergoes moion described by: P = ω P +τ, where ω and τ are he roaion and ranslaion respeciely, hen he ou-of-plane moion of a plane, n P = d, is gien by: n = ω n, and d = n τ. I follows ha for consan elociie ω and τ, he induced elociies for he plane parameers n and d are consan oo.
References [Adi85] G. Adi, Deermining hree-dimensional moion and srucure from opical flow generaed by seeral moing objec IEEE ransacions on Paern Analysis and Machine Inelligence, ol. 7, no. 4, July 985. [Belhumeur96] P. N. Belhumeur, A Bayesian-approach o binocular sereopsi In. Journal of Compuer Vision, ol. 9, no. 3, pp. 37-6, Augus 996. [Bergen9] J. R. Bergen, P. Anandan, K. J. Hanna, and R. Hingorani, Hiearchical model-based moion esimaion, in Proc. European Conference on Compuer Vision (ECCV 9), pp. 37-5, 99. [Boyko99] Y. Boyko, O. Veksler, and R. Zabih, Fas approximae energy minimizaion ia graph cu in Proc. In. Conf. on Compuer Vision (ICCV 99), ep. 999. [Carceroni] R. L. Carceroni and K. N. Kuulako cene capure by surfel sampling: from muli-iew sreams o non-rigid 3D moion, shape and reflecance, in Proc. In. Conf. on Compuer Vision (ICCV ), July. [Dhond89] U. R. Dhond and J. K. Aggarwal, rucure from sereo: a reiew, IEEE ransacions on ysem, Man, and Cyberneic ol. 9, no. 6, pp. 489-5, 989. [Hanna93] K. J. Hanna and Neil E. Okamoo, Combining sereo and moion analysis for direc esimaion of scene srucure, in Proc. In. Conf. on Compuer Vision, pp. 357-65, 993. [Kuulakos99] K. N. Kuulakos and. M. eiz, A heory of shape by space caring, in Proc. In. Conf. on Compuer Vision (ICCV 99), pp. 37-34, 999. [Lhuillier] M. Lhuillier and L. Quan, Edge-consrained join iew riangulaion for image inerpolaion, in Proc. IEEE Conf. on Compuer Vision and Paern Recogniion (CVPR'), June. [Roy98]. Roy and I. J. Cox, "A maximum-flow formulaion of he N-camera sereo correspondence problem", in Proc. In. Conf. on Compuer Vision (ICCV'98), Bombay, India, January 998. [han] Z. Zhang and Y. han, A progressie scheme for sereo maching, Lecure Noes in Compuer cience 8, pringer-verlag, pp. 68-85, March. [zeliski99] R. zeliski, A muli-iew approach o moion and sereo, in Proc. IEEE Conf. on Compuer Vision and Paern Recogniion (CVPR'99), pp. I-57-63, June 999. [ao] H. ao and H.. awhney, Global maching crierion and color segmenaion based ereo, in Proc. Workshop on he Applicaion of Compuer Vision (WACV), pp. 46-53, December. [Vedula99]. Vedula,. Baker, P. Rander, R. Collin and. Kanade, hree-dimensional scene flow, in Proc. In. Conf. on Compuer Vision, pp. II-7 79, ep. 999. [Young9] G. Young and R. Chellappa, 3-D moion esimaion using a sequence of noisy sereo images: model esimaion and uniqueness resul IEEE ransacions on Paern Analysis and Machine Inelligence, ol., pp.735-759, 99. [Zhang9] Z. Zhang and O. Faugera 3D Dynamic cene Analysis: A ereo Based Approach, pringer, Berlin, Heidelberg, 99. [Zhang99] Y. Zhang and C. Kambhameu, Inegraed 3D scene flow and srucure recoery from muliiew image sequence in Proc. IEEE Conf. on Compuer Vision and Paern Recogniion (CVPR ), pp. II-674-68, ouh Carolina, June. [Zhang] Z. Zhang, A flexible new echnique for camera calibraion, IEEE ransacions on Paern Analysis and Machine Inelligence, ol., pp. 33-334,. Figure 6. he deph map sequence esimaed using he proposed dynamic deph recoery algorihm. he lef column shows he original reference images and he righ column shows he esimae deph maps.