An Iterative Scheme for Motion-Based Scene Segmentation

Transcription

1 An Ieraive Scheme for Moion-Based Scene Segmenaion Alexander Bachmann and Hildegard Kuehne Deparmen for Measuremen and Conrol Insiue for Anhropomaics Universiy of Karlsruhe (H), Karlsruhe, Germany Absrac We presen an approach for dense esimaion of moion and deph of a scene conaining a muliple number of differenly moving objecs wih he camera sysem iself being in moion. he esimaes are used o segregae he image sequence ino a number of independenly moving objecs by assigning he objec hypohesis wih maximum a poseriori (MAP) probabiliy o each image poin. Differen o previous approaches in 3-dimensional (3D) scene analysis, we ackle his ask by firs simulaneously esimaing moion and deph for a salien se of feaure poins in a recursive manner. Based on he evolving se of esimaed moion profiles, he scene deph is recovered densely from spaially and emporally separaed views. Given he dense deph map and he se of racked moion esimaes, he likelihood of each image poin o belong o one of he disinc moion profiles can be deermined and dense scene segmenaion can be performed. Wihin our probabilisic model he expecaionmaximizaion (EM) algorihm is used o solve he inheren missing daa problem. A Markov Random Field (MRF) is used o express our expecaions on spaial and emporal coninuiy of objecs. 1. Inroducion his conribuion addresses he deecion of independenly moving objecs in a raffic scene as a sereo camera plaform moves hrough i. Objec deecion is performed based on he relaive moion of exured objecs and he observer. o obain a dense represenaion of he observed scene, objec deecion is formulaed as an image segmenaion ask where each image poin is esed for consisency wih a se of possible hypoheses, each defined by is relaive 3D moion. Alhough a field of acive research for decades, a general and robus soluion o his problem is sill elusive, since he observable 2D image moion is generaed by he combined effecs of camera moion, he moion of independenly moving objecs and scene srucure. As scene srucure basically denoes he deph of a scene poin relaive o he observing camera, in he sequel we use he erm scene deph likewise. Isolaing hese hree facors proves o be a difficul ask as deph disconinuiies and independenly moving objecs boh cause disconinuiies in he image moion field. herefore i is no possible o separae hese facors wihou 3D moion and srucure esimaion. Mehods for finding he moion and srucure of an enire scene can be caegorized ino eiher (i) direc mehods [10], where he unknown moion and srucure parameers are recovered simulaneously from measurable image quaniies a each pixel in he image(s) or (ii) indirec mehods [14], which rely on a sparse se of disinc feaure poins ha are exraced beforehand from he image(s). As for our applicaion, he deecion and segmenaion of moving raffic paricipans, he principle requiremens on our segmenaion scheme are a sufficienly dense represenaion of he scene, i.e. a maximum number of scene poins should be reconsruced, a reliable idenificaion and correcion of erroneous poins, he abiliy o cope wih general objec moion and compuaional efficiency, we pursue he sraegy of (ii) when esimaing he moion componens and hen use he moion esimaes o guide he compuaion of a dense deph and segmenaion map in a direc manner also in regions of he image where here is less informaion. Assuming rigidiy of he individual scene objecs, he 3D moion esimaes can be used as a srong guide o sereo and emporal maching in recovering a dense scene srucure and subsequenly a dense scene segmenaion. Objec moion is esimaed using an EM-based approach ha consiss of a muliple objec racking filer wih probabilisic daa associaion. In he E-sep, he probabiliy of an observaion o belong o each of he objec hypohesis is compued based on he curren sae esimaes. In he M-sep, he sae and error covariance of each objec is robusly updaed in a recursive manner. Based on hese moion esimaes he scene srucure is hen recovered densely from spaially and emporally separaed views. In he segregaion sep, we derive a global cos funcion ha incorporaes he moion and srucure esimaes while considering he fundamenal propery of spaial and emporal label consisency and segregae he image accordingly. c) 2009 IEEE. Personal use of his maerial is permied. Permission from IEEE mus be obained for all oher users, including reprining/ republishing his maerial for adverising or promoional purposes, creaing new collecive works for resale or redisribuion o servers or liss, or reuse of any copyrighed componens of his work in oher works.

2 he remainder of he paper is organized as follows: Secion 2 recapiulaes he moion segmenaion ask and inroduces noaion and consrains used wihin his work. In Secion 3 he segmenaion filer framework is presened. he performance of our approach is illusraed in Secion 4 on real and synheic image daa. 2. Problem Formulaion Following he derivaions of Longue-Higgins and Prazdny [12], for each objec in he scene we consider he equivalen problem of a saionary objec and a moving observer, i.e. we express he enire scene dynamics by a se of rigidly moving objecs, each measured relaive o a camera-fixed coordinae sysem. For each objec, he insananeous rigid body moion of his coordinae sysem is specified by he ranslaional and roaional moion of is origin, = (x, y, z ) and Ω = (ωx, ωy, ωz ), respecively. Formally his can be expressed by he 3D moion field ω () R3, wih he parameerized moion of each poin = (, Y, Z) being ³ (1) ω () =, Y, Z = Ω. he compuaion of he moion field is obviously underdeermined due o he projecion u (x) R2 of ω () ono he image plane. he ask becomes even more ambiious if he observer iself is moving and he scene consiss of a muliple number of differenly moving objecs. Consequenly, he flow field u (x) is generaed by an unknown number of unknown objec moions wih he observer moion superimposing all moion vecors (x saes he collecion of all image poins). A well sudied area in his conex is 2D moion esimaion, which is concerned wih he deerminaion of he flow field u (x). Here, variaional echniques based on he mehod of [8] yield he mos accurae resuls as e.g. presened in [3]. he major limiaion of 2D moion esimaion is he fac ha moion cues presen in he 2D projecion are insufficien o reconsruc he moion presen in he 3D scene. he relaively young area of dense 3D moion esimaion resolves his limiaion by addiionally esimaing he deph of he scene. Here, good resuls have been achieved using he variaional framework. Nex o he compuaionally expensive join esimaion of moion and srucure a he same ime [9], in [15] a mehod is presened ha separaes he problem ino he wo sub-problem scene flow and deph esimaion, resuling in an efficien compuaion of he individual problems. In our approach we inegrae 3D moion and deph esimaion ino one feaure-based approach. Differen o he approaches above, we apply a parameric moion model, expressing he moion field as a collecion of of a rigidly moving objecs wih each objec being specified by is moion parameers. Based on he join esimae of deph and moion on he basis of a sparse se of feaures, we compue a dense deph map of he scene and derive dense scene flow. In he segmenaion sep, each image poin is hen assigned o he objec hypohesis, expressed by a unique se of moion parameers, ha explains he underlying moion bes. Objec sae. In our framework, he 3D moion for each objec eniy is expressed by he coninuous-valued variable θ. We use a facorial represenaion of our sae vecor wih θ = {θ1,..., θj,..., θj }, represening he quaniaive sae informaion for each objec θj = (ωx, ωy, ωz, x, y, z ), independenly. he number of objecs J wihin he curren, discree ime insan is assumed o be fixed. θ is esimaed for each objec j independenly based on a disinc se of feaure poins ha are racked over ime. More deails will be given in Secion 3.2. Vision cues. In our implemenaion we use wo vision cues o obain dense 3D srucure and moion informaion of a scene from muliple images. One is sereo vision, which drasically faciliaes he problem saed above, as he scene is reconsruced from wo views recorded a he same ime, i.e. no consrains (e.g. rigidiy, ec.) need o be imposed. he oher is visual moion, which expresses he displacemen of image poins in emporally separaed views caused by he relaive scene moion in beween. We inegrae sereo vision and visual moion ino one probabilisic framework by inroducing a deph relaed paramerizaion of he spaial and emporal (epipolar) consrains [13]. he approach is based on a deph esimae obained by sereo vision, which grealy simplifies he complexiy of he segmenaion ask. Using he iniial informaion of he sereo reconsrucion, he moion parameers can be obained wihou scalar ambiguiy. Afer obaining he moion parameers, he iniial deph esimaes z can be furher improved by inegraing he moion cue and herefore more epipolar geomery consrains can be used. Our camera seup consiss of a fully calibraed sereo rig wih he world origin a he righ camera (quaniies ha are relaed o he righ camera are indexed wih characer r in he sequel). In normalized image coordinaes x = (x, y, 1), he mapping from an image poin x r in he righ camera o an image poin x in a second camera is x = K RK 1 r x r + K Z. (2) Kr and K sae he camera calibraion marices of he wo cameras respecively. R is he 3x3 roaion marix and he 1x3 ranslaion vecor specifying orienaion and pose of he second camera relaive o he righ camera. I can be seen ha he mapping is divided ino a componen ha depends on he image posiion alone bu no on he deph Z. his erm akes accoun of he camera roaion. he second erm depends on Z bu no on he image posiion and scales wih he amoun of ranslaion beween he cameras, i.e. x

3 moves along he epipolar line as a funcion of he inverse deph, saring from Z = and going in he direcion of he epipole K. Concerning he sereo cue, x represens he corresponding image poin x l in he lef image. Given he inernal and exernal camera parameers, he only unknown in (2) is d = 1/Z and he corresponding poin in he lef image, along he epipolar line, can be parameerized by s (xr, d) = s = xr + dkl. (3) Above, he correspondence search has been drasically simplified by recifying he sereo images such ha he epipolar lines coincide wih he corresponding horizonal scan lines in he warped images, i.e. = (x, 0, 0). For he moion cue, x consiss of he emporally corresponding image poin x +1,r in he righ camera. he epipolar consrain here expresses he geomerical relaionship of poin correspondences beween wo views due o he moion of he sereo camera sysem. he feaure-based esimaion of objec moion θ from poin correspondences is explained in Secion 3.2. An adequae descripion, which formally describes he displacemen of an image poin as a funcion of 3D moion and (inverse) deph, is given by h i 0 x u,i = CΩ Ω + dc, wih C = f 0 f y 2 2 f +x ( xi yi i) (4) f yi f. and CΩ = f 2 +y 2 ( i ) xifyi xi f he corresponding image poin in image G+1 for image poin xr in image G, parameerized by d, hen is m (x,r, d, θ ) = m = x,r + CΩ Ω + dc. (5) he consrains inroduced above are used wihin he moion and srucure esimaion process o efficienly guide he correspondence search and evaluae curren esimaes as presened in he sequel. Observaions. Informaion from he environmen is acquired hrough a sequence of observaions Y0: = (Y0,..., Y ), wih Y = {Y,1,..., Y,i,..., Y,N } being a se of random variables. hroughou he paper, y = {y,1,..., y,i,..., y,n } represens a sample realizaion of Y. N saes he number of image poins. he reconsrucion and segmenaion process are direcly evaluaed on he image gray values g (xi ) = g,i G, assuming observaions o be i.i.d. Gaussians. Concerning he sereo cue, he similariy beween gray value g,i a image posiion x,i in he righ camera ( r is omied in he sequel) and g,si a image posiion si in he lef image, parameerized by z,i, is expressed by à 2! g,i g,si 1 s (6) P (ǫ,i z,i ) = exp. 2σ 2 2πσ he similariy of he moion cue is evaluaed along he emporal epipolar line and is wrien à 2! g,i g,mi 1 m exp P (ǫ,i θ, z,i ) =, (7) 2σ 2 2πσ wih mi saing he corresponding poin in he nex righ image, parameerized by θ. σ saes he error disribuion of moion and sereo cue. Wih his, our observaions consis of he combined error map E, which is composed of he ers ror ǫm and ǫ a each pixel posiion. For opimal moion and deph esimaes, E should reach values close o or equal o zero. Associaion process. he fac ha he scene consiss of a muliple number of differenly moving objecs is considered nex by inroducing a daa associaion process. Obviously, his is a crucial ask for moion esimaing as proposed in his work. Differen o mehods ha are operaing on pixel level o deermine he local moion field, our approach operaes on a global scale in he enire image. A wrong or erroneous assignmen of observaions in he moion esimaion process would lead o heavily corruped reconsrucion and segmenaion resuls. Generally, he differen approaches can be classified as eiher hard or sof daa assignmen mehod. Hard denoes he assignmen of an observaion o one (and only one) hypohesis, whereas sof means he assignmen of an observaion o an objec hypohesis proporional o some weigh. We pose he problem of muliple objec racking and scene segmenaion as incomplee daa problem wih he observaions being he incomplee daa, whereas he objec-associaed observaions sae he complee daa. o capure he unknown relaionship beween an observaion and he objec ha caused i, an associaion process L is inroduced wih is componens being defined n j l,i = 1 if y,i originaed from objec j (8) 0 else. A sample realizaion of he associaion process is defined as binary label field l = (l,1,..., l,n ) wih a label vecor l,i = ej for each observaion i. ej saes a uniy vecor of lengh J. his formalizes our assumpion ha an observaion y,i originaes from exacly one objec j. For noaional convenience we resric our descripion o only one objec insance wihin θ in he sequel and skip he hypohesis index j. If necessary, we will resor o i in he ex. o accoun for observaions wih a low confidence, i.e. all hypoheses seem o be equally unlikely for ha observaion, we furher inroduce an ambiguiy label (AMB) which will be expressed by j = 0. In our esimaion scheme he associaion process is inerpreed as missing daa regarding he observaions. Finally, we define he complee daa hrough ime as E0: = {E0:, L0: }.

4 3. Segmenaion Filer 3.1. E-sep he aim is now o find he opimal moion and deph esimaes and derive from ha a segmenaion ha segregaes he scene ino regions of similar 3D moion. In his conribuion we carry on he work presened in [1], where an ieraive scheme is proposed ha splis he problem saed above ino a se of sub-problems which are hen solved separaely. Concerning he esimaion of θ he EM framework [5] is applied, which consiss of ieraively compuing he expeced complee daa in he E-sep and aferwards esimaing he sae, based on he complee daa, in he M-sep. Differen o he sandard EM algorihm, a penalized maximum likelihood (ML) esimae is obained (see e.g. [6]), leading o a MAP esimae of θ according o he Bayesian recursive updae rule. he presened approach is ime-recursive in ha he moion esimaes from he previous ime insan are used as prior. Based on he moion esimaes, he scene deph z is hen recovered densely from spaially and emporally separaed views. Each EM-loop erminaes wih he segregaion of he image ino a se of disjoin, non-overlapping regions, resuling in a hard scene segmenaion l. hough he approach presened in [1] incorporaes emporal dependencies in he moion esimaes, i ignores hem in he segmenaion and deph esimaion process. herefore, in his conribuion, we exend our exising framework by emporal dependencies of he segmenaion and deph esimaion process, resuling in a spaially and emporally consisen scene segmenaion scheme. Following he noaion of [1], each ieraion of he proposed algorihm consiss of he following seps on he (k + 1)-h ieraion, wih Θ = {θ, z } for noaional convenience. In he E-sep, he condiional expecaions of l are compued based on he acual observaions and sae esimaes, which is equivalen o compuing he probabiliies of an image poin o belong o each of he objec hypoheses By only considering daa from he presen and previous ime sep, he likelihood erm in (9) ges Q(Θ Θ k ) = E[logP (E Θk, E 1 ) E, Θ k ]. (9) A segmenaion n o l k+1 = arg max Q(Θ Θ k ), l (10) is derived from hese condiional expecaions in he image segmenaion sep. he segmenaion is used o derive objec-specific daa, as e.g. he gray value/deph disribuion wihin he objec boundaries. he condiional probabiliies are hen used in he moion updae of he M-sep n o Θ k+1 = arg max Q(Θ Θ k ) + logp (Θ E 1 ), (11) θ,z o weigh he observaions. Scene deph z is recovered densely from spaially and emporally separaed views. k {1,..., K} saes he ieraion index. he wo seps are repeaed unil eiher he parameer esimaes converge or some maximum number of ieraions is reached. logp (E Θ, E 1 ) = logp (E, l Θ, E 1 ) = logp (E l, Θ, E 1 ) + logp (l Θ, E 1 ). (12) Regarding our label prior, spaial dependencies are incorporaed ino he model o accoun for he naural noion ha physical objecs exend in space. he final Q-funcion ges 1 Q(Θ Θ k ) = N E[l,i E, Θ k, E 1 ] i=1 (13) k D(ǫ,i Θ ) V1 (Θ ) E[l,i V2 (Θ )l,n Θ ]. i,n c V2 (Θ ) is a marix of dimension J J wih elemen {j, u} equals o logp (l,i = ej, l,n = eu ) = λ(ej eu ). λ is a regularizaion consan raing he influence of neighbouring sies o he prior erm. his model can be inerpreed as he well known Pos model. A deailed mahemaical derivaion of he single erms can be found in [1]. o capure he srong saisical dependency of he assignmen process in he emporal domain (as image poins wih coheren label value are expeced o follow he underlying, rue objec along a smooh rajecory, parameerized by sae variable θ ) we exend he single elemens of our firs-oder clique V1 ( ) from above in such a way ha V1 (Θ, E 1 ) = [logp (e1 Θ, E 1 ),..., logp (ej Θ, E 1 ),..., logp (ej Θ, E 1 )]. (14) We ake he corresponding fixed label field esimae from he previous ime sep and evaluae he label consisency over ime. herefore, for each image poin a ime, is expeced locaion in he previous image mus be deermined. his is done by deriving he expeced image coordinaes from (5), given sae esimae θ. Wih his simple image warping mehod, he corresponding label value in he previous label field can be deermined. As a measure of label similariy along he expeced objec rajecory we propose o evaluae he similariy of he assumed objec moion in he presen and in he previous image. his is quanified by he ranslaional moion componen θ. he individual elemens of V1 (Θ, E 1 ) hen are ³ ³ logp (ej Θ, E 1 ) = j u Σ 1 j u, (15) wih u being he hypohesis index which has been exraced a he back-projeced posiion in l 1. Equaion (15) quanifies he dissimilariy of he wo sae vecors indexed j and u 1 D(ǫ Θ ),i = [logp (ǫ,i e1, Θ ),..., logp (ǫ,i ej, Θ )] V1 (Θ ) = [logp (e1 Θ ),..., logp (ej Θ )]

5 under he assumpion ha hey are disribued Gaussian. Σ saes a fixed, isoropic covariance marix. Wih his measure we enforce a smooh propagaion of our associaion process over ime. Finally, he poserior probabiliy of he label variable a posiion xi is E[l,i ǫ,i, θ k, z k,i, l 1 ] = π,i, wih he j-h elemen P (ǫ,i l,i = ej, Θ k )P (l,i = ej Θ k ) j π,i = PJ, (16) k k s=1 P (ǫ,i l,i = es, Θ )P (l,i = ej Θ ) expressing he probabiliy ha xi is assigned o objec hypoheses j. Sof daa assignmen. A probabilisic daa associaion measure is obained by applying pseudo-likelihood (PL) approximaion. he PL is evaluaed by resricing he saisical dependencies of he label field in above expression o he local neighborhood Gi of each poin, i.e. k 1 P (l,i Θ ) P (l,i π,u, u Gi, Θ ), wih sep. k 1 π,u being he esimaes from he previous ieraion = arg min l (N i=1 D(ǫ,i Θ k ) + V1 (Θ k, E 1 )+ i,n c l,i V2 (Θ k )l,n Moion esimaion. Differen o direc approaches as i.e. [15, 9], which deermine he moion from he image gray value variaions iself, we use an indirec feaure-based approach in our curren framework. his allows for efficien and robus moion and deph esimaion simulaneously in one filer sep. We show ha, hough we are minimizing a differen error meric when applying feaure-based moion esimaion, also he overall error E converges o low values. For each objec hypohesis independenly, a sae esimae is obained based on a se of M N salien feaure poins,i, i = (1,..., M ) of a rigidly moving objec in 3D space. Following [4], we have applied he idea of he reduced-order observer in order o reduce he dimension of o one sae for each racked poin, encoding is deph ρ,i, i.e.,i = (x,i, ρ,i ). I is assumed ha he corresponding image poins of x,i can be deermined exacly for all scene poins in all views wihin he feaure racking scheme. Deph poins propagae over ime according o ρ,i = (0, 0, 1) [R (Ω ) 1,i + ]. (19) (17) Scene segmenaion. In he segmenaion sep he label ha generaes he highes probabiliy is assigned o each image poin. C quanifies he overall coss of a segmenaion wih each erroneously assigned image poin producing he same coss. Our Bayesian decision rule assigns he hypoheses wih MAP probabiliy o each image poin and herefore minimizes C, i.e. minimizes he number of segmenaion errors. Based on our es saisic π,i, we formulae our segmenaion problem as l k M-sep (18). An opimal labeling is found using a discree energy minimizaion echnique based on he well known graph-cu framework [2]. he opimal labeling l k+1 is hen used wihin a gaing process in he updae sep of our racking filer, resricing he number of valid observaions for a given objec hypohesis o he observaions ha have been assigned o he respecive label. If a rack has no suppor in he curren segregaion sep, i.e. no image poin is assigned o he respecive label, he rack is deleed from he lis of racked objec hypoheses. Wih his, he coordinaes of a scene poin a ime are,i = Π 1 (x,i, ρ,i ), where Π 1 ( ) saes he inverse projecion funcion. Given he se of racked feaure poins, observaions consis of he corresponding image poins in he lef and following righ camera wih image coordinaes x,l,i and x+1,i, i.e. y,i = (x,l,i, x+1,i ). Concerning our observaion model, he image posiion in he following righ image can be prediced from he curren frame using (5), which saes he insananeous velociy field model. Given he curren deph esimae ρ,i, i is also possible o derive he corresponding image coordinae s,i in he curren lef image using (3). Wih his, our combined observaion equaion is defined as h(θ,i, ρ,i ) = (m,i ; s,i ) + r. Observaion noise r is assumed o be a zero-mean, whie Gaussian wih covariance marix R = E[r r ]. he observaion residual is hen v,i R 1,i v,i, wih v,i = ( x+1,i m,i + x,l,i s,i ), expressing he projecion error for each poin x,i ino he following righ and curren lef camera image. We weigh his residual wih he poserior probabiliy π,i of he label variable a he respecive posiion i. By addiionally evaluaing he second erm of (11), we obain a MAP sae esimae considering sae informaion from ime ( 1), i.e. we inegrae he sae evoluion hrough ime ino our esimaion scheme. his is formulaed by he Chapman-Kolmogorov equaion P (θ E 1 ) = R P (θ θ 1 )P (θ 1 E 1 )dθ 1, which expresses he prediced sae disribuion from ime insan ( 1) o based on he a priori sae disribuion P (θ 1 E 1 ) and an appropriae model ha accouns for he sysem dynamics. he model accouns for uncerainies and model errors hrough

6 whie, zero-mean Gaussian process noise q wih error covariance marix Q = E[q q ]. We assume he prior disribuion being embodied in he probabiliy saemen P (θ 1 E 1 ) = N (θ 1, P 1 ), wih θ 1 and P 1 being he mean esimae and covariance of a Gaussian. Given he above equaions, he bes choice for θ ³hen is P (θ E 1 ) = N (θ 1, P 1 ), wih θ 1 = f θ 1 saing he prediced sae based on he previous esimae θ 1. he same holds for he prediced error covariance P 1. In [1], he ieraed exended Kalman filer is presened o find he MAP esimae o his formulaion. Nonlineariies in he sysem model are handled by ieraive relinearizaion of he model equaions wihin he updae sep. Oulier deecion is performed based on a significance es of he error disribuion of v. Besides yielding a robus moion esimae, he oupu of he proposed mehod is also used o iniialize new objec hypoheses. his is done by analyzing he ouliers for paern of similar moion, as disinc moving objecs ha are no conained in he racking process ye, produce coheren groups in he oulier vecor. o idenify hese groups, an ieraive RANSAC approach 2 is chosen. I pariions he ouliers ino poin ses wih he members of each specific se following a moion ha can be approximaed by he same consan moion model over n frames. For any sable oulier poin se, 4 random poins are seleced from he se of ouliers and he 2d homography H is calculaed for all n 1 correspondences. he disance for each puaive correspondences in he oulier se is calculaed using he squared symmeric ransfer error d2ransfer = (xou H 1 x ou )2 + (x ou Hxou )2 as proposed in [7]. o enforce local conneciviy and moion similariy, he disance funcion is also weighed wih he spaial disance and moion vecor disance of ouliers. As he number of maching poins is unknown in his conex, ouliers are assumed o belong o he same group when heir disance d2ransfer o he esimaed homography is below a predefined hreshold. he group ha fulfills d2ransfer < is he new consensus se. For he cos funcion of a consensus se he poins of he consensus se are scored according o heir disance d2ransfer o he model while he ouliers are given a consan weigh. he seps are repeaed unil he number of correspondences is sable and he coss of he consensus are minimal. he resuling consensus se is assumed o be he larges group in he remaining oulier se ha can be approximaed by homographic projecion. Afer soring he group members for laer iniializiaion, hey are removed from he oulier se. he algorihm is repeaed wih he remaining ouliers unil eiher he size of he las found consensus se or he number of remaining feaure poins is below a predefined hresh. he resul is a se of groups which represen he new moion segmens. o avoid an over segmenaion, a maximum 2 Based on he RANSAC oolbox for Malab by M. Zuliani number of groups is defined. he resuling groupings are used o iniialize a new objec hypoheses if he number of spaially clusered image poins exceeds a cerain hreshold. Scene deph esimaion. he final sep wihin one EMcycle is he dense esimaion of he scene deph. For each scene poin, he deph wih MAP probabiliy is deermined z,i,map P (ǫ,i l,i, θ, z,i )P (z θ, E 1 ) (20) Similar o he noaion inroduced above, we formulae his as a discree, combinaorial opimizaion problem wih he likelihood being expressed as daa erm logp (ǫ,i l,i, θ, z,i ) = (ǫ,i l,i, θ ), wih (ǫ,i l,i, θ ) = [logp (ǫ,i z,i = 1, l,i, θ ),... (21)..., logp (ǫ,i z,i =, l,i, θ ))]. Like in he moion esimaion sep, a MAP esimae considering informaion from ime ( 1) is obained by evaluaing he second erm of (11) logp (z,i θ, E 1 ) = V1 (θ, E 1 ), wih V1 (θ, E 1 ) = [logp (z,i = 1 θ, E 1 ),... (22)..., logp (z,i = θ, E 1 )]. emporal coherence is evaluaed by predicing he MAP esimae z 1,i from ( 1) o according o (5) and (19), resuling in he prediced deph esimae z 1,i. Equaion (22) is evaluaed according o he emporal compaibiliy erm V1 (θ, E 1 ) = min( z,i z 1,i, a), assigning low cos o values ha are close o he predicion and high values oherwise. a saes he maximum value of our robus, runcaed linear cos funcion. Spaial smoohness of he reconsrucion resul is guaraneed by modeling z as a MRF wih he same configuraion as he label field. Dispariy esimaion is achieved o pixel accuracy by finding he opimal configuraion of z,i for each image poin, which is equivalen o minimizing he following energy funcional (N (ǫ,i l,i, θ )+ z,map = arg min z V1 (θ, E 1 ) + i=1 i,n c z,i V2 (Θ k )z,n (23) Each image poin can be assigned o one value z,i = z ou of a se of candidae deph values z (1,..., ). Similar o he label erm inroduced above, we enforce spaial and emporal smoohness on he evolving 3D srucure which is also modeled hrough an MRF. A iniializaion, deph map z 1,i 1 and segmenaion l,i consis of he prediced values from K z K 1 and l 1. (23) is compued using a belief propagaion framework similar o he approach of Larsen e al. [11].

7 Figure 1. (a) Righ camera image. (b) Ground ruh dispariy map. (c) Sub-pixel accurae correlaion-based SVS sereo macher (Γ = 0.56, Λ = 0.57) (hp:// (d) Sub-pixel accurae block-based sereo macher (Γ = 0.65, Λ = 0.68). (e) Pixel accurae belief-propagaion framework as presened above (Γ = 1.03, Λ = 0.87). 4. Experimens he performance of our approach has been evaluaed on real and synheic 3 image daa. For each rack, he observaions wihin he M-sep are exraced from he respecive se of racked feaure poins based on a correlaion-based block maching echnique. he filer oupu, represening he ime-smoohed objec parameers of objec j, is hen fed back ino he image segmenaion process. In a subsequen sep, poins ha are no conform wih he labeling are deleed from he lis of racked feaure poins and replaced by new poins, sampled from he labeled region in he image. he sysem sae for each rack is iniialized wih zero velociy and deph ρ1:m0, which is exraced from he iniially esimaed deph map. he number of possible hypoheses J is defined by he momenary number of disinc 6-DoF moion profiles in he scene. Regarding he relaive imporance of daa and smoohness erm he regularizaion facor has been adaped empirically o values beween λ = In Figure 1, he oupu of differen sereo reconsrucion mehods is shown. he esimaion qualiy of he differen approaches has been deermined quaniaively by compuing he average disparpn iy error Γ = 1/N i=1 g es (g:ground ruh, es:esimae). As we are ineresed in a dense scene represenaion, also he oal number of reconsruced scene poins mus be considered when evaluaing he differen approaches. herefore we define Λ = Nes /Ng as a measure of he densiy of our deph map. Our examinaion of he differen sereo maching algorihms showed, ha he block-based approach (Figure 1-(d)) produced slighly beer esimaion resuls (Γ < 0.6, Λ 0.6) compared o he macher based on belief-propagaion (Γ < 0.9, Λ 0.85) (Figure 1-(e)). he crucial benefi of laer approach is he much higher reconsrucion densiy. See capion for deails. Figure 2 depics he mean segmenaion error per pixel, according o (6) and (7), over ime when iniializaion of new objec hypohesis is suppressed (solid blue) and when new objec hypoheses are added o he filer bank for he san3 Downloaded from he.enpeda.. hp:// web sie a Figure 2. Mean segmenaion error per pixel ǫ and mean reconsrucion error per feaure poin v ( Feaure Error (M-Sep) ) for a raffic scene where wo objecs are enering he field of view a frame 30 and 65, respecively. SUPPRESS shows he error propagaion if he iniializaion of objecs wih a disinc moion is suppressed and he ambiguiy label (AMB) is swiched off. AMB>20 shows he error propagaion if AMB is considered in he segmenaion. Image poins wih an error ǫi > 20 are hen assigned o AMB. SD shows he sandard case, where objec hypoheses have been iniialized a frame 32 and 68. dard case (dashed red). he colored, verical bars indicae he ime insan when an objec hypoheses has been added o he filer bank. Figure 3 illusraes he segmenaion pipeline. Afer esimaing scene moion (a) and scene srucure (b), he image is pariioned accordingly (c+d). Figure 4 shows segmenaion resuls of ypical raffic scenarios wih differenly moving objecs. 5. Conclusion In his conribuion we have presened an ieraive procedure for dense esimaion of moion and deph of a scene conaining a muliple number of differenly moving objecs wih he camera sysem iself being in moion. he daa associaion problem has been solved using he EM framework. Wihin he associaion process, which has been implemened as labeling problem, a MRF has been used o ac-

8 Acknowledgemen he auhors graefully acknowledge he conribuion of he German collaboraive research cener SFB/R 28 Cogniive Auomobiles graned by Deusche Forschungsgemeinschaf (DFG). References Figure 3. (a) Esimaed image flow field. (b) Esimaed deph map. (c) Segmenaion resul. Poins ha are coloured red are assigned he ambiguiy label. (d) Original image wih image poins assigned o he objec hypohesis being highlighed in blue. Figure 4. Segmenaion resuls wih deeced objecs being coloured differenly. coun for spaial and emporal relaionships. he EM framework has been adaped for ime-recursive racking of a muliple number of objecs. Based on he moion esimaes, he scene srucure is recovered densely from spaially and emporally separaed views. In he segregaion sep, he image is segregaed ino a se of non-overlapping regions which represen independenly moving objecs. In ongoing work we inegrae relaional classificaion based on Markov logic ino our segmenaion scheme. We assume ha he ineracion of segmenaion/racking wih he resuls from he classificaion sep can be exploied o drive low-level objec deecion schemes ending owards more human-like scene percepion. [1] A. Bachmann. Applying recursive EM o scene segmenaion. In Deusche Arbeisgemeinschaf fuer Musererkennung DAGM e.v., LNCS 5748, pages , Jena, Germany, Sepember Springer-Verlag. [2] Y. Boykov, O. Veksler, and R. Zabih. Fas approximae energy minimizaion via graph cus. IEEE rans. PAMI, 23(11): , Nov [3] A. Bruhn, J. Weicker,. Kohlberger, and C. Schnoerr. Disconinuiy-preserving compuaion of variaional opic flow in real-ime. In Scale-Space, pages , [4] A. Chiuso and S. Soao. Moion and srucure form 2d moion causally inegraed over ime: Anlaysis. In IEEE rans. Roboics and Auomaion, [5] A. Dempser, N. Laird, and D. Rubin. Maximum likelihood from incomplee daa via he EM algorihm. Journal Royal Saisical Sociey, B 39(1):1 38, [6] P. Green. On use of he EM algorihm for penalized likelihood esimaion. Journal Royal Saisical Sociey, 52(3): , [7] R. Harley and A. Zisserman. Muliple View Geomery in Compuer Vision. Cambridge Universiy Press, ISBN: , second ediion, [8] B. Horn and B. Schunck. Deermining opical flow. Arificial Inelligence, 17: , [9] F. Hugue and F. Devernay. A variaional mehod for scene flow esimaion from sereo sequences. In Proc. Inl. Conf. on Compuer Vision, Rio de Janeiro, Brasil, [10] M. Irani and P. Anandan. Abou direc mehods. In ICCV 99: Proceedings of he Inernaional Workshop on Vision Algorihms, pages , London, UK, SpringerVerlag. [11] S. Larsen, M. Philippos, M. Pollefeys, and H. Fuchs. Simplified belief propagaion for muliple view reconsrucion. In 3DPV 06, pages , Washingon, DC, USA, IEEE Compuer Sociey. [12] H. Longue-Higgins and K. Prazdny. he inerpreaion of a moving reinal image. Proceedings of Royal Sociey of London, 208: , [13] C. Srecha and L. Gool. Moion - sereo inegraion for deph esimaion. In In Eur. Conf. on Compuer Vision, pages SpringerVerlag, [14] P. orr and A. Zisserman. Feaure based mehods for srucure and moion esimaion. In Vision Algorihms: heory and Pracice, number 1883 in LNCS, pages Springer-Verlag, [15] A. Wedel, C. Rabe,. Vaudrey,. Brox, U. Franke, and D. Cremers. Efficien dense scene flow from sparse or dense sereo daa. In European Conference on Compuer Vision, pages , Berlin, Heidelberg, Springer-Verlag.