Rigid Body Segmentation and Shape Description from Dense Optical Flow Under Weak Perspective

Transcription

1 , VOL. 19, NO. 2, FEBRUARY Rgd Body Segmentaton and Shape Descpton fom Dense Optcal Flow Unde Weak Pespectve Joseph Webe and Jtenda Malk Abstact We pesent an algothm fo dentfyng and tackng ndependently movng gd objects fom optcal flow. Some pevous attempts at segmentaton va optcal flow have focused on fndng dscontnutes n the flow feld. Whle dscontnutes do ndcate a change n scene depth, they do not n geneal sgnal a bounday between two sepaate objects. The poposed method uses the fact that each ndependently movng object has a unque eppola constant assocated wth ts moton. Thus moton dscontnutes based on selfoccluson can be dstngushed fom those due to sepaate objects. The use of eppola geomety allows fo the detemnaton of ndvdual moton paametes fo each object as well as the ecovey of elatve depth fo each pont on the object. The algothm assumes an affne camea whee pespectve effects ae lmted to changes n oveall scale. No camea calbaton paametes ae equed. A Kalman flte based appoach s used fo tackng moton paametes wth tme. Index Tems Optcal flow, eppola constant, fundamental matx, shape fom moton, moton segmentaton, scene pattonng poblem. 1 INTRODUCTION VISUAL moton can povde us wth two vtal peces of nfomaton: the segmentaton of the vsual scene nto dstnct movng objects and shape nfomaton about those objects. We wll examne how the use of eppola geomety unde the assumpton of gdly movng objects can be used to povde both the segmentaton of the vsual scene and the stuctue of the objects wthn t. Eppola geomety tells us that a constant exsts between coespondng ponts fom dffeent vews of a gdly movng object (o camea). Ths eppola constant s unque to each moton. Optcal flow povdes a dense set of coespondences between fames. Theefoe the unque eppola constant can be used to fnd objects undegong sepaate motons gven the optcal flow. Typcally the eppola constant s used fo lage dsplacement motons, but t s equally vald fo optcal flow felds whch we assume epesent small nte-fame dsplacements. An algothm wll be outlned fo segmentng the scene whle smultaneously ecoveng the moton of each object n the scene. Ths algothm makes the assumpton that the scene conssts of connected pecewse-gd objects. The mage then conssts of connected egons, each assocated wth a sngle gd object. Once the moton of gdly movng objects has been detemned, scene stuctue can be obtaned va the same eppola constant. The scene stuctue poblem becomes analogous to steeopss n that object depth s a functon of dstance along the eppola lne. Dense coespondences such as those n optcal flow can lead to ch descptons of the scene geomety. The eppola geomety wll be examned n the context of an af- J. Webe s wth the Engneeng Depatment, The Calfona Insttute of Technology, Pasadena, CA E-mal: webe@vson.caltech.edu. J. Malk s wth Compute Scence Dvson, Unvesty of Calfona at Bekeley, Bekeley, CA E-mal: malk@cs.bekeley.edu. Manuscpt eceved Ma. 9, 1995; evsed Ap. 1, Recommended fo acceptance by A. Sngh. Fo nfomaton on obtanng epnts of ths atcle, please send e-mal to: tanspam@compute.og, and efeence IEEECS Log Numbe P IEEE

2 140 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 2, FEBRUARY 1997 fne camea model whee pespectve effects ae lmted to unfom changes n scale. Unde weak pespectve, the eppola constant equaton becomes lnea n the mage coodnates, thus allowng a least-squaes soluton fo the paametes of the constant. Dffeent egons of the mage epesentng ndependently movng gd objects can then be segmented by the fact that they possess dffeent eppola constants on the moton n the mage plane. Once the paametes of the constant equaton have been ecoveed, they can be used to descbe the thee dmensonal gd moton that each object n the scene has undegone. 2 REVIEW OF PAST WORK Ealy wok on segmentaton va moton looked fo dscontnutes n the dsplacement feld [17], [2] o pecewse affne pattons of the feld [1], [12]. Snce unde geneal pespectve pojectons the moton feld s contnuous as long as the depth of the vewed suface s contnuous, dscontnutes n the flow feld sgnal depth dscontnutes. Unfotunately, the flow feld s dffcult to ecove at dscontnutes. At locatons of depth edges, moton wll ntoduce egons of occluson and dsoccluson whch ae often not explctly modeled n optcal flow outnes. Optcal flow technques based on devatves of the mage functon assume contnuous o affne flow and wll fal at these egons. The fact that eppola geomety mples a lnea constant between the pojected ponts of a gd body as t undegoes an abtay gd tansfomaton has been used fo yeas n photogammety [5] and moe ecently n stuctue fom moton algothms [11], [15], [16]. Moton paametes and shape descptons can also be obtaned fom coespondences between two vews unde weak pespectve pojecton, modulo a elef tansfomaton such as depth scalng [8]. Algothms unde ths model wee mplemented by Shapo et al. [14] and Cenusch-Fas et al. [4]. In Secton 6 we wll see that we can fomulate the segmentaton of the optcal flow feld nto a scene pattonng poblem [10]. The segmentaton poblem s fomulated n tems of a cost functonal whch attempts to balance a numbe of model constants. These constants nclude tems fo fttng a model to the data whle smultaneously mnmzng the numbe of dstnct egons. Thee ae stochastc [6], egon-gowng [7], and contnuaton [10], [3] methods fo fndng solutons to the scene pattonng poblem when t s descbed n tems of a cost functonal. Ou soluton wll use the egon-gowng method descbed n [19] to solve fo the patton. Ths method uses a statstc-based egon gowng algothm whch assumes the soluton s pecewse contnuous n mage coodnates. 3 PROJECTIONS AND RIGID MOTIONS 3.1 The Weak Pespectve Camea The weak pespectve camea pojecton can be wtten as: x = MX + p (1) whee X s the 3D wold coodnate pont and x ts 2D mage pojecton. The 2 3 matx M otates the 3D wold pont nto the camea s efeence fame, scales the axes and pojects onto the mage plane. The vecto p s the mage plane pojecton of the tanslaton algnng the two fames. The smplest fom of the matx M occus when the wold and camea coodnates ae algned and the camea s aspect ato s unty. In ths case M can be wtten f F M = H I K (2) Z ave whee Z ave s the aveage depth of the scene. Ths tansfomaton s a vald appoxmaton to a eal camea only f the vaance of the depth n the vewed scene s small compaed to Z ave. A gd tansfomaton of the wold ponts that takes the pont X to X can be wtten as X = RX + T (3) whee R s a otaton matx wth unt detemnant. Elmnatng the depth component Z between equatons fo x and x we obtan the lnea constant ax + by + cx + dy + e = 0 (4) whee a =- R23, b = R13 c = sr23r11 - sr13r21 d = sr23r12 - sr13r22 e = R23tx - R13ty (5) and the vecto t s sm(t - R(p0) T ). The scale facto s = Z ave /Z ave s the factonal change n aveage depth between fames. Moe detals can be found n [20]. Equaton (4) can be wtten n tems of a specal fom of the Fundamental Matx [5]. (x, y, 1)F(x, y, 1) T = 0 (6) 3.2 Koendenk and van Doon Rotaton Repesentaton A otaton n space can be expessed n a numbe of epesentatons: Eule angles, axs/angle pa, quatenons etc. A patculaly useful epesentaton fo vson was ntoduced by Koendenk and van Doon [8]. In ths epesentaton, the otaton matx s the composton of two specfc otatons: the fst about the vewng decton (cyclootaton) and the second about an axs pependcula to the vewng decton at a gven angle fom the hozontal. Usng ths epesentaton n the fomaton of the Fundamental Matx as n (4) we fnd that bg bg bg bg a = sn cos f, b = sn cos f c =-ssnbg cosbq -fg d = ssn sn q -f bg b g bge x bg y bgj (7) e =- sn t cos f + t sn f Equatons (8) ae dentcal to the ones used n Shapo et al. [14]. We can nvet (8) to fnd the moton paametes s, f, and q gven the elements of the Fundamental Matx (a, b, c, d, e). In the next secton we explan how to estmate these gven the optcal flow. 4 SOLVING FOR THE FUNDAMENTAL MATRIX The eppola constant (4) eques pont coespondences between fames. Equatng pont dsplacements wth optcal flow (u, v), we get (x, y ) = (x, y) + (u, v) and au + bv + c x + d y + e = x n + e = 0 (8) wth x = cuvxy,,, h and n = cabc,,, d h. The eppola constant equaton elements (4) ae elated to the pmed values by c = c + a, d = d + b. The affne eppola constant equaton foces the optcal flow to le on a lne n velocty space. Because of nose, the measued optcal flow may not le on the lne dctated by the eppola constant. We can use weghted least squaes to solve fo the pa-

3 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 2, FEBRUARY ametes cn, eh by mnmzng the weghted dstance n velocty space between the measued optcal flow and the constant lne. The weghtng facto fo each eo tem, w, comes fom the eo covaance of the measued optcal flow, vaev T v j =W v. The followng mnmzaton s smla to the one n Shapo et al. [14]. We defne a Lagange multple, l, on the constant a 2 + b 2 = 1. Ths constant can be wtten as Qn 2 = 1 wth an appopate dagonal matx Q. The functon to be mnmzed s then 2 2 mn n w x n e Qn a, efâ c + h - l e -1j (9) whee the summaton s ove all ponts wth optcal flow measuements. The mnmzaton ove e can be done mmedately by settng e =-x n whee x =Âwx / Âw s the weghted centod of the 4D ponts x. Afte substtutng fo e and dffeentatng we obtan bw - lqgn = 0 (10) T whee the measuement matx W s Âw cx - xh cx - xh. Snce Q has only two nonzeo entes, fndng the value of l whch causes (W - lq) to dop ank nvolves only a quadatc equaton n l. The soluton n s the vecto whch spans the null space of (W - lq). The esultng value of l s equal to the weghted quadatc eo n velocty space. 5 THE CASE OF AFFINE FLOW The soluton fo the Fundamental Matx elements n (11) eques that the matx W - lq have ank thee,.e., the null space has dmenson one. Multple solutons can exst f the optcal flow s affne n mage coodnates. In ths case, a lnea elatonshp exsts between (u, v) and (x, y), and thus W dops ank. The Fundamental Matx cannot be unquely detemned. The nontval causes of affne flow ae ethe coplanaty of the obseved ponts, o f the object moton s a otaton whch contans no otaton n depth (the otaton n the Koendenk and van Doon epesentaton). Snce the optcal flow s coupted by nose, a cteon must be developed fo decdng f a egon contans affne flow. The symmetc matx W - lq should have ank thee and theefoe have thee postve, nonzeo sngula values. A egon s desgnated as contanng affne flow va a ato of sngula values. A theshold on ths ato s used to label egons as contanng affne flow. The magntude of the theshold comes fom the vaance estmate poduced by the optcal flow algothm used. 6 SEGMENTING VIA A REGION-GROWING METHOD We wsh to patton the scene nto dstnct egons, each egon beng labeled by a unque Fundamental Matx. We defne a cost functonal whch balances the cost of labelng each pxel wth a penalty fo havng too many dffeent labelng. We defne as a total cost functonal Ecn; a h = Â Dcnh+ Pcn; ah (11) whee the summaton s ove all pxels n the mage. The vecto n s the estmate of the Fundamental Matx at pxel. In tems of the standad fom of a cost functonal [3], Daf n epesents a goodness of ft tem whch attempts to keep the estmate close to the data, and Pcn; ah s a dscontnuty penalty tem whch tes to lmt the fequency of dscontnutes. The Daf n tem s the weghted sums of squaed dstances n velocty space wth a Lagange multple as defned n the Secton 4. The penalty tem attaches a fxed cost a fo each pxel bodeng a dscontnuty. To solve ths pattonng poblem we wll use the egongowng method descbed n [19]. The algothm begns by fomng small ntal patches of sze 4 4 pxels. Each of these patches then computes ts soluton, n, and eo, D R. Fo a small value of the bounday penalty a, all egons whch can be combned when a statstc, F, s below a fxed confdence level ae meged. Newly fomed egons ae tested fo affne flow solutons. The value of a s nceased allowng fo moe egons to be meged. Ths contnues untl we each the fnal value of a. 7 RECOVERING DEPTH Once we have ecoveed the elements of the Fundamental Matx fo a egon of the mage plane, we can attempt to ecove the depth of each mage pont. Fom Secton 3.1 we fnd that up to an unknown scale facto: 1 Z = bx - ay + dx - cy + Z 2 c a + b 2b g (12) T 2 whee Zc =- dt / es d j wth d = (R 13 R 23 ) T. Z c s a constant fo each object. Theefoe, up to an addtve constant and unknown scale, the depth of each maged pont can be computed gven the elements of the matx F. In the case of affne flow, we know that the object s ethe undegong pue tanslaton o s otatng about an axs paallel to the optcal axs. In ethe case, no depth nfomaton can be obtaned unde othogaphc o weak-pespectve pojecton. Consequently depth ecovey would have to ely on othe cues. 8 OBJECT TRACKING In ode to tack the segmented objects, the algothm takes the pesent segmentaton and foms a pedcton of the segmentaton fo the next flow feld. The segmentaton algothm s un usng ths pedcton mage to fll n the unassgned egons. Ths s epeated fo each new optcal flow feld. The poposed scheme avods havng to un the ente segmentaton algothm fom scatch at each new fame snce t uses the pevous segmentaton as a pedcton. Howeve, ths method eques a coect ntal segmentaton. If two objects ae labeled as a sngle object n the ntal segmentaton they may eman so n subsequent fames. We can use the nfomaton n each new fame to ncease the accuacy of both the shape and moton of each ndependently movng object. We adopt a Kalman flte appoach n whch the moton paametes ae modeled as a slowly vayng pocess. The wok by Soatto et al. [13] addesses the case of estmatng the elements of the Fundamental Matx n a Kalman Flte famewok. Although the wok was fo the full Fundamental Matx, t s easly adapted to the smple affne fom. 9 EXPERIMENTAL RESULTS The algothm was tested on a numbe of synthetc and eal mage sequences. The optcal flow was computed usng the mult-scale dffeental method of Webe and Malk [20]. Flow felds wee about 80% dense wth most estmates mssng fom dscontnuous flow egons. These egons volate the constancy assumpton used by the dffeental method.

4 142 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 2, FEBRUARY Sequence 1 A synthetc sequence was ceated consstng of two textuemapped cubes otatng n space. The magntude of the optcal flow anged fom zeo to about fve pxels/fame. Fo the fst 10 fames of the sequence, the cubes wee otatng about fxed but dffeent otaton axes. Fo the second 10 fames these axes wee swtched. The otaton axes used, as well as a sample mage and optcal flow feld ae shown n Fg. 1. Fg. 3. The ecoveed value of the angle the otaton axs of each cube makes n the mage plane as a functon of fame numbe. Afte 10 fames, the otaton dectons wee swtched. 9.2 Sequence 2 The algothm was un on a eal sequence consstng of a cube placed on a otatng platen. 1 The backgound was statonay. The dsplacements between fames ae vey small n ths sequence, wth the lagest dsplacement on the cube tself beng only 0.5 pxel. The backgound had zeo flow and was labeled as affne. An mage fom the sequence, the computed optcal flow and ecoveed depth map ae shown n Fg. 4. Fg. 1. Two ndependently otatng textue-mapped cubes wee ceated on a Slcon Gaphcs wokstaton. A sngle fame fom the sequence and a sample optcal flow feld s shown on the top ow. No flow estmates wee avalable at the boundaes of the two cubes because such egons volate the constancy assumpton used by the dffeental method. Fo the fst 10 fames, the cubes otated wth otaton axes ndcated n the bottom left fgue. Fo the second 10 fames, the otaton axes wee as ndcated n the bottom ght fgue. The segmentaton algothm found two sepaate movng objects fo each fame. The ntal segmentaton along wth the ntal depth ecoveed fo the smalle cube s shown n Fg. 2. Fg. 4. A sngle fame of a Rubk s Cube on a otatng platen. The optcal flow and ecoveed depth map as seen fom a sde vew ae shown as well. Fg. 2. The bounday between the two ndependently movng objects found by the segmentaton algothm and the pxel depths of the smalle cube. The estmated angle f as a functon of fame numbe fo each cube s shown n Fg. 3. The ognal estmate s good because of the densty of the optcal flow. Subsequent fames do not show much mpovement. The Kalman Flte successvely tacks the change n otaton axs whch occus at fame 10. In ths case, the otaton axs of the cube makes an angle of 90 degees n the mage plane and was ecoveed as such to wthn a few degees. 9.3 Sequence 3 The next mage sequence contans lage plana egons whch poduce egons of affne flow. A fame fom the sequence, an example optcal flow ecoveed and the segmentaton ae shown n Fg. 5. Ths sequence demonstates the algothm s ablty to dentfy egons of affne flow. The boundaes appea egula be- 1. Ths sequence was poduced by Rchad Szelsk at DEC.

5 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 2, FEBRUARY cause no shape pos ae used n the segmentaton algothm. Affne egons wll be labeled as dstnct f the dffeence n affne paametes s lage than the expected vaance n flow due to nose. The theshold used n the segmentaton algothm s bounded by ths nose vaance. Fg. 5. A sngle fame of the moble sequence fom RPI. The backgound conssts of plana tanslatng pattens whle a toy tan taveses the foegound. An example optcal flow ecoveed s also shown. The labeled mage s shown as well. The backgound pats (n gay) wee dentfed as undegong pue tanslatonal moton by the sngula value ato test. The black and whte egons (coespondng to the tan, otatng ball, and tanston egons) wee not labeled as affne. 10 DISCUSSION We have shown that usng just the optcal flow, t s possble to segment an mage nto egons wth a consstent gd moton and detemne the moton paametes fo that gd moton. Futhemoe, the elatve depth of ponts wthn the sepaate egons can be ecoveed fo each pont dsplacement between the mages. The ecovey eques no camea calbaton but does make the assumptons of an affne camea:.e., pespectve effects ae small. The specal fom of eppola geomety fo the case consdeed hee has ts eppoles at nfnty. Pespectve domnant motons can not be ft by the moton paametes. The egon-gowng algothm used fo the smultaneous egon fomaton and moton paamete estmaton was not dependent on ths patcula fom of the geomety. If a ecovey of the full pespectve case was equed, the same algothm could be used. Howeve, the calculaton of the Fundamental Matx fom small dsplacements such as found n optcal flow s not stable [18], [9]. Ths s one of the fundamental lmtatons of usng optcal flow wth an algothm based on the eppola constant. ACKNOWLEDGMENTS Ths eseach was patally suppoted by the PATH poject MOU 83. The authos wsh to thank Paul Debevec fo ceatng the synthetc mage sequence. REFERENCES [1] G. Adv, Detemnng Thee-Dmensonal Moton and Stuctue Fom Optcal Flow Geneated by Seveal Movng Objects, IEEE Tansactons on Patten Analyss and Machne Intellgence, vol. 7, no. 4, pp , [2] M. Black and P. Anandan, Constants fo the Ealy Detecton of Dscontnuty Fom Moton, Poc. Nat l Conf. AI, pp , Boston, [3] A. Blake and A. Zsseman, Vsual Reconstucton. Cambdge, Mass.: MIT Pess, [4] B. Cenusch-Fas, D.B. Coope, Y.P. Hung, and P.N. Belhumeu, Towad a Model-Based Bayesan Theoy fo Estmatng and Recognzng Paametezed 3D Objects Usng Two o Moe Images Taken Fom Dffeent Postons, IEEE Tansactons on Patten Analyss and Machne Intellgence, vol. 11, pp , [5] O. Faugeas, Thee-Dmensonal Compute Vson: A Geometc Vewpont. Cambdge, Mass.: MIT Pess, [6] S. Geman and D. Geman, Stochastc Relaxaton, Gbbs Dstbutons, and the Bayesan Restoaton of Images, IEEE Tansactons on Patten Analyss and Machne Intellgence, vol. 6, pp , Nov [7] S.L. Hoowtz and T. Pavlds, Pctue Segmentaton by a Dected Splt-and-Mege Pocedue, Poc. Second Int l Conf. Patten Recognton, pp , [8] J.J. Koendenk and A.J. van Doon, Affne Stuctue Fom Moton, J. Optcal Soc. Ameca A, vol. 8, no. 2, pp , [9] Q.-T. Luong, R. Deche, O.D. Faugeas, and T. Papadopoulo, On Detemnng the Fundamental Matx: Analyss of Dffeent Methods and Expemental Results, Techncal Repot RR-1894, INRIA, A shote veson appeaed n the Isael Conf. Atfcal Intellgence and Compute Vson. [10] Y.G. Leclec, Constuctng Smple Stable Descptons fo Image Pattonng, Int l J. Compute Vson, vol. 3, pp , [11] H.C. Longuet-Hggns, A Compute Algothm fo Reconstuctng a Scene fom Two Pojectons, Natue, vol. 293, pp , [12] H.-H. Nagel, G. Soche, H. Kollng, and M. Otte, Moton Bounday Detecton n Image Sequences by Local Stochastc Tests, Poc. Thd Euopean Conf. Compute Vson, vol. 2, pp , Stockholm, [13] S. Soatto, R. Fezza, and P. Peona, Recusve Moton Estmaton on the Essental Manfold, Poc. Thd Euopean Conf. Compute Vson, vol. 2, pp , Stockholm, [14] L.S. Shapo, A.P. Zsseman, and M. Bady, Moton Fom Pont Matches Usng Affne Eppola Geomety, Poc. Thd Euopean Conf. Compute Vson, vol. 2, pp , Stockholm, [15] R.Y. Tsa and T.S. Huang, Unqueness and Estmaton of Thee- Dmensonal Moton Paametes of Rgd Objects Wth Cuved Sufaces, IEEE Tansactons on Patten Analyss and Machne Intellgence, vol. 6, pp , [16] C. Tomas and T. Kanade, Shape and Moton Fom Image Steams Unde Othogaphy: A Factozaton Method, Int l J. Compute Vson, vol. 9, no. 2, pp , [17] W. Thompson, K. Mutch, and V. Bezns, Dynamc Occluson Analyss n Optcal Flow Felds, IEEE Tansactons on Patten Analyss and Machne Intellgence, vol. 7, no. 4, pp , [18] J. Weng, N. Ahuja, and T. Huang, Optmal Moton and Stuctue Estmaton, IEEE Tansactons on Patten Analyss and Machne Intellgence, vol. 15, no. 9, pp , [19] J. Webe, Scene Pattonng va Statstc-Based Regon Gowng, IS and T SPIE Symp. Electonc Imagng: Scence and Technology, San Jose, Calf., [20] J. Webe and J. Malk, Rgd Body Segmentaton and Shape Descpton Fom Dense Optcal Flow Unde Weak Pespectve, Poc. Ffth ICCV, pp , Boston, [21] J. Webe and J. Malk, Robust Computaton of Optcal Flow n a Mult-Scale Dffeental Famewok, Int l J. Compute Vson, vol. 14, no. 1, 1995.