Multi-body Segmentation: Revisiting Motion Consistency Λ

Transcription

1 Mult-body Segmentaton: Revstng Moton Consstency Λ Lh Zelnk-Manor Moshe Machlne Mchal Iran Dept. of Computer Scence and Appled Math The Wezmann Insttute of Scence 76 Rehovot, Israel Abstract Dynamc analyss of vdeo sequences often reles on the segmentaton of the sequence nto regons of consstent motons. Approachng ths problem requres a defnton of whch motons are regarded as consstent. Common approaches to moton segmentaton usually group together ponts or mage regons that have the same moton between successve frames (where the same moton can be 2D, 3D, or non-rgd). In ths paper we defne a new type of moton consstency, whch s based on temporal consstency of behavors across multple frames n the vdeo sequence. Our defnton of consstent temporal behavor s expressed n terms of mult-frame lnear subspace constrants. Ths defnton apples to 2D, 3D, and some non-rgd motons wthout requrng pror model selecton. We further present a mult-frame mult-body segmentaton algorthm whch apples the new moton consstency constrant drectly to mage brghtness measurements, wthout requrng pror correspondence estmaton nor feature trackng. Introducton Common approaches to moton-based segmentaton (e.g., [,, 3, 4, 6, 2, 5, 6, 7]), usually group together ponts whch have the same moton between every par of frames, where the same moton could be ether the same 2D moton, or the same 3D moton, or locally the same mage moton. For example, n [, ] the same moton s modelled by a 2D parametrc transformaton (.e., all ponts grouped together belong to a sngle 2D layer). In other cases (e.g., [3, 4]) the same moton s modelled by a fundamental matrx or a trfocal tensor (.e., all ponts grouped together share the same eppolar geometry and belong to a smple rgd 3D object). The body of work on mult-body factorzaton (e.g., [5, 6, 7, 2]) suggests employng mult-frame lnear subspace constrants to group ponts wth the same 3D rgd moton. In [6, 2], same moton s defned n terms of moton contnuty,.e., neghborng pxels belongng to the same object locally have approxmately the same mage moton. The latter can handle both rgd and non-rgd objects but cannot handle moton dscontnutes (whch occur n the presence of 3D parallax or due to a fragmented object). In all the above mentoned cases, two regons are regarded as movng consstently when they have the same moton between every par of frames. We defne a new type of moton consstency whch s based on temporal constrants: a set of ponts are grouped together as a sngle object when they have a consstent behavor over tme. Our defnton of consstent temporal behavor s expressed n terms of the mult-frame lnear subspace constrants used n [8] for flow estmaton of a sngle object. These subspace constrants allow groupng together ponts movng wth dfferent motons, as long as ther motons change over tme n the same pattern. Our approach provdes a unfed treatment of 2D motons, 3D motons and some non-rgd motons. We further propose an algorthm whch apples these new mult-body subspace constrants drectly to mage brghtness quanttes for segmentng the entre mage (every pxel) nto multple objects. Ths does not requre pror correspondence estmaton or feature trackng (as opposed to [5, 6, 7, 2] whch rely on carefully tracked sparse feature ponts). Ths work was supported by the Israel Scence Foundaton (Grant no. 53/99) and by the European Commsson (IBES Project IST-2-26).

2 . Background and Basc Defntons Let I ;:::;I F denote a sequence of F frames wth N pxels n each frame. Let (u f ;vf ) denote the dsplacement of pxel (x ;y ) n frame I f ( =;:::;N, f =;:::;F). Let and denote two F N matrces constructed from the dsplacements of all the mage ponts across all frames: u u 2 u N v v 2 vn u 2 u 2 2 u 2 N v 2 v 2 2 v 2 N = 6 7 = 6 7 () u F u F 2 u F N v F F N v F 2 vn F F N Each row n these matrces corresponds to a sngle frame, and each column corresponds h to a sngle pont. Stackng the matrces and of Eq. () vertcally results n a 2F N matrx where each column s assocated wth the dsplacements of a sngle pont across all mages/frames. Prevous work on subspace-based mult-body segmentaton/factorzaton (e.g., [5, 6, 7, 2]) decomposed the space spanned by the columns ofh nto lnearly ndependent sub-spaces. Ths was done by permutng and groupng the columns ofh nto sub-matrces of lower ranks, such that the sub-spaces assocated wth h the dfferent sub-matrces are lnearly ndependent. All columns of a sngle such sub-matrx (sub-space) of correspond to all the ponts of a sngle ndependently movng object. Such a decomposton was often obtaned (e.g., [5, 6]) by a factorzaton of the matrxh nto a h product of two matrces: = M 2F (r + +r K )S (r + +r K ) N, where M s the matrx of motons of all objects and S s a block dagonal matrx wth blocks of rank r k. Each block corresponds to the shape of a sngle object and r k s the rank of h that object (.e., the dmensons of ts correspondng sub-space). The permutaton and groupng of columns of to obtan the desred separaton nto ndependently movng objects was obtaned by seekng a block-dagonal structure of S. Whle the mult-body factorzaton approaches assumed rgd motons of objects, Brand [3] and Bregler et al. [4] showed that the moton of some non-rgd objects also resdes n low-dmensonal lnear sub-spaces. In all the above mentoned cases, the sub-space constrants were appled to the matrxh. Boult & Brown [2] and Iran [8] refer to ths matrx as the trajectory matrx because each column contans the dsplacements of a sngle pont across all frames n the sequence,.e., ts trajectory. Iran [8] further nvestgated the meanng of stackng the matrces and horzontally. Ths gves rse to an F 2N matrx [ j ], where each row contans the flow of all ponts between a sngle par of frames (typcally between a reference frame and one of the other frames), and s therefore referred to here as the flow-feld matrx (coned the dsplacement-feld matrx n [8]). Iran showed that sub-space constrants on ths matrx can be used to constran mult-frame correspondence estmaton n a covarance-weghted way [8]. Torresan et al. [5] extended ths approach to non-rgd moton estmaton. Iran and Anandan [9] further showed that the use of [ j ] gves rse to a factorzaton wth drectonal uncertanty. All these assumed that all columns of [ j ] correspond to a sngle (rgd or non-rgd) object/scene. In ths paper we explore the meanng of applyng mult-body factorzaton to the matrx [j ], and show that on one hand ths gves rse to a new nterpretaton of mult-frame moton consstency (Secton 2), and - on the other hand - t supports drect (ntensty based) covarance-weghted mult-body factorzaton wthout a need to resort to pror correspondence estmaton or pont trackng (Secton 3). 2 Revstng Moton Consstency As explaned n Secton. clusterng the columns of the trajectory matrxh (as was done by [5, 6, 7, 2]) captures the dependency between trajectores of dfferent ponts. In ths secton we nvestgate the meanng of clusterng the columns of the flow-feld matrx [ j ], and compare the two approaches. Gven two objects and the matrces ; ; 2 ; 2 of ther mult-pont mult-frame dsplacements, we examne when these objects wll be grouped together as havng consstent motons and when they wll be separated, usng 2

3 (a) (b) (c) (d) Fgure. (a)-(c) The 3 frames of a sequence showng a crcle enlargng. (d) An overlay mage of the three frames and flow vectors for some sample ponts. h subspace constrants on ether or [j ]. Let W, W 2 be ether the trajectory sub-matrces correspondng h h to the two objects fw = and W 2 2 = 2 g or ther flow-feld sub-matrces fw = [ j ] and W 2 = [ 2 j 2 ]g. Let r = rank(w ) and r 2 = rank(w 2 ). In both segmentaton approaches (.e., the exstng subspace- h based segmentaton approaches of or our segmentaton of [j ]) the two objects are grouped together or alternatvely separated nto two objects accordng to the followng (mplct) rank rule : rank([w jw 2 ]) = r + r 2 =) Two separate objects (n all algorthms). max(r ;r 2 ) < rank([w jw 2 ]) <r + r 2 =) Algorthm dependent. rank([w jw 2 ]) = max(r ;r 2 ) =) Sngle object (n all algorthms). (2) We make the followng observatons regardng the rule of Eq. (2): () Although the rank of the flow-feld matrx may be hgher than the rank of the trajectory matrx n some cases and lower n other cases [8], the decson whether two objects wll be grouped together or separated nto two objects depends solely on the relaton between the ranks of the sub-matrces of the sngle objects to the rank of the combned matrx. () It can be shown 2 that f two objects are grouped together n a trajectory based segmentaton ( h ), they wll also be grouped together n a flow-feld based segmentaton [ j ], but not vce versa! Ths has nterestng mplcatons to mult-body segmentaton. () Subspace constrants apply to varous 2D and 3D moton models [8], and some non-rgd motons [3, 4]. The decson whether to group two objects together or separate them s based only on the ranks. Ths does not requre pror selecton of a moton model or a scene geometry. The rank of a matrx s determned by the dmensonalty of the lnear space spanned by ts columns. Ths s also equal to the dmenson of the space spanned by the rows of the matrx (although these spaces are dfferent, ther dmensons are the same). To understand the physcal meanng of each of the segmentaton approaches, we analyze the dependence between columns of the trajectory matrxh, and the dependence between rows of the flow-feld matrx [ j ]. The suggested analyss offers an understandng of the nformaton content of each of these matrces and s ndependent of the specfc segmentaton algorthm used. Fgure explans the notons of lnear dependency of trajectores and lnear dependency of flow-felds of a sngle object. The three frame sequence shows a crcle whch expands from frame to frame (the radus ncreases by a between the frst and the second frames and by another b n the thrd frame). It s clear that the trajectory of the top pont (marked n blue n Fg..d) and the trajectory of the rghtmost pont (marked n red n Fg..d) are lnearly ndependent. The top pont has a trajectory vector [a; b; ; ] T whereas the rghtmost pont has a trajectory vector Ths rank rule bulds upon the observatons prevously made by Boult & Brown [2]. 2 We omt the proof due to lack of space. It wll be added to the fnal (longer) verson of ths paper. 3

4 [; ;a;b] T. The trajectory of any other pont on the crcle can be expressed as a lnear combnaton of these two trajectores. For example take the pont marked n green n Fg..d: ts trajectory vector s: cos(ff)[a; b; ; ] T sn(ff)[; ;a;b] T. Thus we can wrte: 4 5 C 2 2. The trajectory matrx therefore has h 2F N = a b a b rank 2 n ths sequence. We next examne the space spanned by the flow-felds of ths sequence. Observng the flow-felds n the frst par of frames and n the second par of frames, t can be seen that they are lnearly dependent: [u 2 ;:::;u2 N jv2 ;:::;v2 N ]= b a [u ;:::;u N jv ;:::;v N ]. A smlar relaton wll hold for any addtonal par of frames n the expandng crcle (wth dfferent coeffcents dependng on the rate of growth). Therefore, [j ] F 2N = C F [u ;:::;u N jv ;:::;v N ]. Ths mples that the flow-feld matrx [j ] has rank. In other words, for a sngle object thereh exst a set of bass trajectory vectors and a set of bass flow-felds such that the two matrces can be factored nto = B z} z} C and [j ]= z} C z} B. Traj Coeff Flow Coeff Flow bass Traj Bass (columns) In the case of multple objects, as was observed n [2], segmentng accordng to the trajectory matrxh 4 2 h (rows), wll group together all the ponts whose trajectory vectors are spanned by the same bass (.e., B = B 2, or when B and B 2 are lnearly dependent). Thus, for example, two objects movng wth the same 3D moton wll be grouped together as one. nderstandng the meanng of groupng ponts together n the flow-feld matrx s less ntutve: Although the subspace constrants are on the rows of [j ], the groupng of ponts s stll done at the level of columns (every pont has a par of columns n [j ]). When two objects are grouped together ther flow-feld vectors at correspondng frames are concatenated nto longer flow-feld vectors. Two objects wll be grouped together nto a sngle object f they have the same coeffcents n the lnear combnaton of ther ndvdual bass flow-felds (.e., C = C 2, or when C and C 2 are lnearly dependent). Note that the two objects can have completely dfferent bass flow-felds, but they wll be regarded as one as long as the way n whch ther flow-feld change over tme s the same (whch s what the coeffcents capture). Ths s what we refer to n ths paper as consstent temporal behavor. For example, ths allows groupng together ponts wth dfferent motons, as long as the patterns n whch ther motons change over tme s the same. h Examples of Segmentaton s. [ j ] Segmentaton: An llustraton of the dfference between what can be acheved wth a mult-pont trajectory-based segmentaton and a mult-frame flow-feld based segmentaton can be seen n Fgure 2. Fg. 2.a shows a jumpng-jack stck fgure. All the ponts on the left arm move together as a rgd body and have lnearly dependent trajectores (marked n dark-green arrows n Fg. 2.b). The ponts on the rght arm also move rgdly (marked n lght-green arrows n Fg. 2.b), but n a dfferent moton then that of the left arm. Hence, the trajectory matrx of the combned ponts of both arms wll have a hgher rank than any of the trajectory matrces h of each ndvdual arm. Therefore, accordng to the rank rule of Eq. (2), these wll not be grouped together n segmentaton. Smlarly, addng the ponts on the left leg (marked n dark-red arrows n h Fg. 2.b) to the trajectory matrx wll further ncrease the rank, and so wll addng the ponts on the rght leg (marked n lght-red arrows n Fg. 2.b). A segmentaton accordng to the trajectory matrx wll therefore segment the four lmbs nto four separate movng objects. Yet, the flow-feld matrx [ j ], whch contans all the ponts on all the lmbs, wll have the same rank as the flow-feld matrx of any sngle lmb. Even though the flow-felds nduced by the dfferent lmbs are dfferent, they share the same set of coeffcents n the frame-to-frame lnear dependence of flow-felds (.e., C rght arm = C left arm = C rght leg = C left leg ). Therefore, n a segmentaton based on the flow-feld matrx all the lmbs wll be grouped as a sngle object. Fgures 3 and 4 llustrate the dfference between what can be acheved by a trajectory-based segmentaton versus a flow-feld based segmentaton ths tme on real sequences. Each of the tested sequences dsplays a sngle non-rgd object consstng of a group of rgd sub-parts wth dfferent 3D motons. In these examples we tracked 4

5 (a) (b) Fgure 2. (a) Sample frames from a synthetc sequence showng a stck fgure performng jumpng-jacks. (b) Example of the trajectores of sx dfferent ponts on the fgure, one on the left leg, one on the rght leg, two on the left arm and two on the rght arm. ponts on the sub-parts and analyzed the ranks of the trajectory matrces and of the flow-feld matrces both for each separate part as well as for the unon of all parts. Fg. 3.e dsplays the resultng ranks for a vdeo sequence showng a person walkng. Ponts were tracked on three sub-parts: the forearm, the shn (lower-leg) and the head (marked n red n Fgs. 3.a-c). The rank of the trajectory matrx for all the ponts on all parts s hgher than the rank of the trajectory matrx for any of the sngle parts (see Fg. 3.e). On the other hand, the rank of the flow-feld matrx for all the ponts on all parts of ths example equals the rank of the flow-felds matrx for the sngle object wth the hghest rank (see Fg. 3.e). Ths mples that a segmentaton appled to the flow-feld matrx [j ] wll group the forearm, the shn and the head nto a sngle object n ths partcular sequence. The resultng ranks for a sequence showng a hand where ts fve fngers expand and contract smultaneously are dsplayed n Fg. 4. Ponts were tracked on each of the fngertps (marked n red n Fgs. 4.a-b). The rank of the trajectory matrx for all the tracked ponts s hgher than the rank of the trajectory matrx for a sngle fngertp (see Fg. 4.e). Alternatvely, the rank of the flow-feld matrx for all the ponts equals the rank of the flow-feld matrx for a sngle fngertp (see Fg. 4.e). Ths mples that n ths case a segmentaton accordng to the flow-feld matrx wll group all the fngers nto a sngle object. 3 Drect Intensty-Based Mult-Body Segmentaton In ths secton we present an algorthm for mult-body segmentaton of vdeo clps by applyng sub-space constrants of the flow-feld matrx [ j ] drectly to mage brghtness measurements. The segmentaton s embedded n an teratve coarse-to-fne framework and extends the work of Iran [8] nto multple movng objects. Prevous work onh subspace-based segmentaton (e.g., [2, 5, 6, 7]) used low-dmensonalty constrants on the trajectores matrx, and reled on a careful choce of a sparse set of feature ponts whch can be accurately tracked along the mage sequence. These ponts were then segmented nto groups of ponts wth the same rgd 3D moton. The algorthm that s outlned below does not requre pror trackng of ponts nor any pror optcal-flow estmaton, and segments the entre mage (pxel by pxel) nto groups characterzed by consstent behavor over tme. It apples to a wde range of motons, ncludng 2D, 3D and some non rgd motons. We frst algebracally formulate the mult-body constrants on flow-felds, and then show how to translate these constrants from flow-feld to brghtness quanttes. Mult-Body Constrants on the Flow-Feld Matrx: Iran [8, 9] showed that h the flow-feld matrx of a rgd S scene can be factored nto a multplcaton of two matrces: [j ]=[M jm ], where M S ;M contan moton nformaton and S contans shape nformaton. When the scene contans multple movng objects, ther columns are usually not sorted and are mxed n the flow-feld matrx [j ]. Let [ j ~ ~ ] be a matrx obtaned by 5

6 (a) (b) (c) Sngular values of [ ] Sngular values of [ over ] hand leg body all hand leg body all Normalzd sngular value σ k / σ Normalzd sngular value σ k / σ (d) (e) Index of sngular value (k) Index of sngular value (k) Fgure 3. (a)-(c) Sample frames from a sequence showng a person steppng forward. The tracked ponts are marked n red. (d) The trajectores of the tracked ponts dsplayed on a sample mage. (e) Graphs showng the sngular values of the trajectory matrces and of the flow-feld matrces for each object separately and for all ponts together. sortng the columns of [ j ] nto ndependent objects. Then t has the followng form: z } 2 h S 3 S M h z } ~j ~. =[ j j j K j K ]=[M jm j jm K jm K ] ; (3) h 7 5 SK S K where K s the number of objects. Note that the combned shape matrx S of the sorted matrx [ ~ j ~ ] s block dagonal, where each block corresponds to a sngle object. When only the unsorted flow-feld matrx [j ] s avalable, ts columns can be sorted and grouped nto objects (wth temporally consstent behavors) by seekng a block dagonal structure of the total shape matrx S of Eq. (3). Ths can be done usng smlar methods to those used by Gear [6] or Costera-Kanade [5]. Mult-Body Constrants on Brghtness Measurements: Flow estmaton can be hghly naccurate, especally when the scene contans multple movng objects. We next show how the mult-body segmentaton can be appled drectly to mage brghtness measurements gvng rse to a covarance-weghted segmentaton wthout requrng pror flow estmaton. Ths s acheved by frst translatng the flow-based subspace constrants of Eq. (3) nto drect brghtness-based subspace constrants, n a way smlar to that done by Iran [8] for a sngle rgd object and later by Torresan et al. [5] for a sngle non-rgd object. Lucas and Kanade [] estmated the local flow vector [u f vf ] of a pxel at frame f by solvng the equaton: " # [u f vf ] a b ]; (4) b d 6 =[g f hf S

7 h a b where b d =» P P I 2 P x P IxIy IxIy I 2 and [g f y hf ] = [ P I x I t P I y I t ] are measurable mage quanttes, n whch I x and I y are the spatal mage dervatves and I t s the temporal mage dervatve. The summaton s performed over a local wndow around pxel. When all the flow vectors across the entre mage sequence are estmated relatve to a sngle reference mage (whch under Gaussan nose assumptons can be shown to be the posteror nverse-covarance h a b frame then, b d matrx of the flow-vector [u f vf ]) s the same for pxel across all frames (.e., when [uf vf ] s the flow-vector of pxel between the reference-frame and frame f). The collecton of all ndvdual flow equatons (Eq. (4)) of all pxels across all frames can therefore be wrtten n a sngle combned matrx form (whch was referred to n [8] as the Generalzed Lucas Kanade Equaton ): Q z " } # A B [j ] F 2N =[GjH] F 2N (5) B D 2N 2N where A; B and D are dagonal N N matrces constructed from the ndvdual coeffcents a ;b and d of each pxel, and G and H are two F N matrces constructed from the ndvdual g f ;hf. Note that the matrces A; B; D and [GjH] contan only measurable mage quanttes (.e., spatal and temporal dervatves of mages). Iran [8] assumed a smple (rgd) scene. However, when the scene contans multple objects, ther columns are mxed n [GjH]. Let [ Gj ~ H] ~ be a sorted verson of [GjH] nto objects, then: 2 3 h Ak B k B k D k Q [ Gj ~ H]=[G ~ 6 jh j jg K jh K ]=[ j j j K j K ] Q K 7 5 (6) where Q k = are 2N k 2N k matrces and N k s the number of mage ponts belongng to object k. Combnng Eqs. (3) and (6) gves: ~L 2 z h } S 3 ~M z } S Q [ Gj ~ H]= ~ [M jm j jm K jm K ] h. 7 (7) 5 SK Q S K K Eq. (7) mples that the sorted matrx [ Gj ~ H] ~ can be factored nto a product of two matrces [ Gj ~ H]= ~ M ~ L, ~ where ~L s block dagonal. The number of blocks n ths representaton corresponds to the number of movng objects n the scene, and the rank of each block r k characterzes the rank of each object. Brghtness-based segmentaton: Eq. (7) mples that segmentng the entre mage (all pxels) nto ndependent objects can be done by sortng the columns of the brghtness-measurement matrx [GjH]. We obtan such a sortng by fndng the reduced row-echelon form of [GjH] usng a method smlar to that suggested by Gear 3 [6]. However, snce [GjH] s a large F 2N matrx, whch n practce has a much lower rank r, we can frst factor [GjH] nto two rank-r matrces usng SD: [GjH] = M F r L r 2N, and then fnd the reduced row-echelon form of the smaller matrx L. when M s full ranked then the matrx L has the same reduced row-echelon form as [GjH] (see [6] for more detals). Interestngly, for the purpose of segmentaton, accurate knowledge of the rank r and the block ranks r ;:::;r K s not necessary. In fact, the rank of the matrces M and L can be lower than the true rank of [GjH]. Ths s because clusterng (segmentaton) s a compettve process between the dfferent objects whch s often resolved already by a few domnant bass vectors, and may not requre the use of the entre bass. Ths s unque to the 3 The matrx [GjH ] s of sze F 2N where typcally N (the number of pxels) s very large. The segmentaton method proposed by Costera & Kanade [5] s sutable to matrces wth relatvely small dmensons whle the segmentaton method of Gear [6] s more sutable for larger matrces. 7

8 segmentaton task, and s not true for shape and moton recovery tasks, whch requre accurate knowledge of the true ranks r ;:::;r K. The measured mage quanttes a ;b ;d ;g f ;hf of Eq. (4) are obtaned from the lnearzed brghtness constancy equaton. However, ths lnearzaton s a good approxmaton only for small dsplacements (u f ;vf ). To handle larger dsplacements, we apply our segmentaton scheme wthn a mult-frame mult-scale (pyramd) data structure. Large dsplacements at hgh resoluton levels translate to small dsplacements at coarse resoluton levels. Our segmentaton scheme can therefore be appled at coarse resoluton levels, but ths wll only provde coarse segmentaton. To refne t, the process must be propagated to hgher resoluton levels. Ths s acheved by an terate-warp coarse-to-fne framework smlar to the one used n [8] for a sngle rgd scene. Snce warpng requres ntermedate flow estmaton, more accurate knowledge of the ranks r ;:::;r K s requred. These ranks are automatcally detected from the brghtness matrces [G k jh k ] for each of the segmented objects extracted n the prevous teraton. Note that ths does not requre pror model selecton (2D, 3D or non-rgd), as no 3D shape or moton nformaton s recovered (see [8] for more detals). We next summarze the algorthm. Summary of the drect mult-body segmentaton algorthm: () Construct a Gaussan pyramd for all mage frames. (2) For each teraton at each pyramd level (startng at the lowest resoluton level) do: ffl Compute matrces G, H and Q of Eq. (5) from brghtness quanttes. ffl Factorze [GjH] F 2N nto M F r L r 2N thus reducng dmensonalty and nose. ffl Fnd the reduced row echelon form of L. ffl sort the reduced row echelon form of L nto groups of columns whch correspond to the same object. Ths defnes the correct sortng of [GjH]. ffl Project [G k jh k ] (k = ;:::;K) of each object onto a low-dmensonal lnear subspace of dmenson r k to reduce nose and get [ ^G k j ^H k ]. ffl Estmate the dsplacements from the equaton: [ k j k ] Q k =[ ^G k j ^H k ] where k =:::K (see [8]). ffl Warp all frames towards the reference frame accordng to the estmated dsplacements (for all objects). (3) Iterate step 2 several tmes (typcally 4-5) n each resoluton and propagate to the next (hgher) resoluton level n the pyramd. The coarse-to-fne process s often stopped as an ntermedate resoluton level, for computatonal effcency (e.g., at a resoluton n mage wdth and heght,.e., at mage sze). The penalty for that s that the boundares 4 6 of the detected segments (objects) are not recovered accurately. 4 Results We tested our drect mult-body segmentaton algorthm on vdeo sequences wth dfferent types of nduced camera motons (2D planar / 3D), and dfferent types of object motons (rgd / non-rgd). The sequences consst of 4-8 frames each, and wth approxmate mage sze of ο The coarse-to-fne segmentaton process was stopped at a pyramd resoluton correspondng to of the mage sze. 6 The frst sequence conssts of a globe rollng on a track stuated n a 3D scene (Fgs. 5.a-d. See attached sequence globus.mpg). The object s undergong a rotaton and translaton whle the camera s translatng from rght to left, back and forth, nducng 3D parallax effects of the background scene (e.g., note the parallax between the two trpods. They occlude each other n Fg. 5.b and are both vsble n Fg.5.c). Applyng our algorthm to the scene produced a segmentaton nto two separate objects (see Fgs. 5.e-g), one whch corresponds to the globe and the second whch corresponds to the background scene. The background was grouped nto a sngle object even though no explct 3D model was assumed for the background moton. Ths s because although pxels wth dfferent depths have dfferent flows, the pattern of changes over tme n ther flows are the same (.e., they share the same coeffcents over tme n the factorzaton of the [ j ] flow-matrx). 8

9 (a) (b) Sngular values of [ ] Sngular values of [ over ] all fngers sngle fnger all fngers sngle fnger Normalzd sngular value σ k / σ Normalzd sngular value σ k / σ (c) (d) Index of sngular value (k) Index of sngular value (k) Fgure 4. (a)-(b) Sample frames from a sequence of a hand where the fngertps move smultaneously (the hand s squeezng and expandng). The tracked ponts are marked n red. (c) The trajectores of the tracked ponts marked on a sample mage. (d) Graphs showng the sngular values of the trajectory matrces and of the flow-feld matrces for each object separately and for all ponts together. The second example conssts of an outdoor scene n whch a helum balloon drfts up wth changng wnd drectons (Fgs. 6.a-c. See attached sequence baloon.mpg). The camera s agan translatng from left to rght and back causng parallax effects (e.g., the tree trunk s occluded by the woman on the bench n Fg. 6.a and s vsble n Fg.6.b). The result of applyng our algorthm (Fg. 6.d-f) detects the balloon as one object and groups the 3D background scene nto a second object. The thrd sequence shows a hand wth the fngers expandng and contractng makng a squeezng moton, and a plant wth dense leaves n the background (Fgs. 7.a-c. See attached sequence hand.mpg). Each fnger moves n an artculated moton whle the moton of all fngers s non-rgd. In addton, the camera s rotatng around the Z axs. As predcted by the analyss n Secton 2, the flow-feld based segmentaton algorthm grouped together all the fngers nto a sngle object, and all the parts whch undergo pure camera moton (ncludng the arm and the palm of the hand, whch were statonary relatve to the background) nto a second object (Fgs. 7.d-f). The fnal experment was done on a sequence showng a large pece of cloth beng folded and unfolded n a natural wavng moton, whle a branch of leaves s beng waved up and down n front of t (Fgs. 8.a-c. See attached sequence cloth.mpg). The moton of the cloth s non-rgd. The two sdes of the cloth are wavng from outsde towards the center, whle the center top part of the cloth moves vertcally (.e., the top center of the cloth s rased as the cloth s stretched n Fg. 8.a and s lowered as the cloth s folded n Fg. 8.b). However, all the parts of the cloth have consstent temporal behavor. Indeed applyng the segmentaton process of Secton 3 segmented the scene nto two objects, one whch corresponds to the leaves and the second whch corresponds to the cloth (see Fgs. 8.d-f). The textureless background regon (marked n green n Fg. 8.f) was gnored n ths example. Note that the dfferent regons of the cloth were grouped together, eventhough no non-rgd moton model was specfed. In all cases the segmentaton was based on analyss of subspace constrants of [j ] folded nto the mage brghtness measurements. The boundares of the segmentaton results n some of the examples that we have used can be consderably mproved by proceedng to hgher resoluton levels of the pyramd, and by utlzng more 9

10 sophstcated clusterng methods than the nave clusterng approach whch we mplemented. References [] S. Ayer and H. Sawhney. Layered representaton of moton vdeo usng robust maxmum-lkelhood estmaton of mxture models and mdl encodng. In Internatonal Conference on Computer son, pages , 995. [2] T.E. Boult and L.G. Brown. Factorzaton-based segmentaton of motons. In n Proc. of the IEEE Workshop on Moton nderstandng, pages 79 86, 99. [3] M. Brand. Morphable 3d models from vdeo. In IEEE Conference on Computer son and Pattern Recognton, volume II, pages , 2. [4] C. Bregler, A. Hertzmann, and H. Bermann. Recoverng non-rgd 3d shape from mage streams. In IEEE Conference on Computer son and Pattern Recognton, volume II, pages , 2. [5] J. Costera and T. Kanade. A mult-body factorzaton method for moton analyss. In Internatonal Conference on Computer son, pages 7 76, Cambrdge, MA, June 995. [6] C.W. Gear. Multbody groupng from moton mages. Internatonal Journal of Computer son, 2(29):33 5, 998. [7] N. Ichmura. A robust and effcent moton segmentaton based on orthogonal projecton matrx of shape space. In IEEE Conference on Computer son and Pattern Recognton, pages , South-Carolna, June 2. [8] M. Iran. Mult-frame correspondence estmaton usng subspace constrants. Internatonal Journal of Computer son, 22. To appear. A shorter verson appeared n ICC 99. [9] M. Iran and P. Anandan. Factorzaton wth uncertanty. In European Conference on Computer son, pages , Irland, 2. [] M. Iran, B. Rousso, and S. Peleg. Detectng and trackng multple movng objects usng temporal ntegraton. In European Conference on Computer son, pages , Santa Margarta Lgure, May 992. [] B.D. Lucas and T. Kanade. An teratve mage regstraton technque wth an applcaton to stereo vson. In Image nderstandng Workshop, pages 2 3, 98. [2] J. Sh and J. Malk. Moton segmentaton and trackng usng normalzed cuts. In Internatonal Conference on Computer son, Bombay, Inda, January 998. [3] P.H.S. Torr. Geometrc moton segmentaton and model selecton. In J. Lasenby, A. Zsserman, R. Cpolla, and H. Longuet-Hggns, edtors, Phlosophcal Transactons of the Royal Socety A, pages Roy Soc, 998. [4] P.H.S. Torr and A. Zsserman. Concernng bayesan moton segmentaton, model averagng, matchng and the trfocal tensor. In European Conference on Computer son, pages 5 527, 998. [5] L. Torresan, D.B. Yang, E.J. Alexander, and C. Bregler. Trackng and modelng non-rgd objects wth rank constrants. In IEEE Conference on Computer son and Pattern Recognton, volume I, pages 493 5, 2. [6] Y. Wess and E. H. Adelson. Perceptually organzed em: A framework for moton segmentaton that combnes nformaton about form and moton. Techncal Report TR 35, MIT, 994.

11 (a) (b) (c) (d) (e) (f) (g) Fgure 5. (a)-(d) Four sample frames of a sequence showng a globe rollng on a track whle the camera translates (see attached sequence globus.mpg). (e)-(g) Results of the drect mult-body segmentaton algorthm. (e) Shows the frst object, (f) shows the second object and (g) shows an overlay of the segmentaton result on the reference mage. (a) (b) (c) (d) (e) (f) Fgure 6. (a)-(c) Four sample frames of a sequence showng a helum balloon drftng wth the wnd whle the camera translates (see attached sequence baloon.mpg). (d)-(f) Results of the drect mult-body segmentaton algorthm. (d) Shows the frst object, (e) shows the second object and (f) shows an overlay of the segmentaton result on the reference mage.

12 (a) (b) (c) (d) (e) (f) Fgure 7. (a)-(c) Four sample frames of a sequence showng a hand wth the fngers expandng and contractng whle the camera translates and rotates (see attached sequence hand.mpg). (d)-(f) Results of the drect mult-body segmentaton algorthm. (d) Shows the frst object, (e) shows the second object and (f) shows an overlay of the segmentaton result on the reference mage. (a) (b) (c) (d) (e) (f) Fgure 8. (a)-(c) Four sample frames of a sequence showng a cloth movng non-rgdly (folded and stretched) and a branch wavng ndependently, whle the camera remans stll (see attached sequence cloth.mpg). (d)- (f) Results of the drect mult-body segmentaton algorthm. (d) Shows the frst object, (e) shows the second object and (f) shows an overlay of the segmentaton result on the reference mage. 2