Articulated Tree Structure from Motion A Matrix Factorisation Approach

Size: px

Start display at page:

Download "Articulated Tree Structure from Motion A Matrix Factorisation Approach"

Silas Hudson
5 years ago
Views:

1 Artculated Tree Structure from Moton A Matrx Factorsaton Approach arl Scheffler, Konrad H Scheffler, hrstan Omln Department of omputer Scence Unversty of the estern ape 7535 ellvlle, South Afrca cscheffler, kschef, Abstract e present a matrx factorsaton approach for 3D reconstructon of artculated objects from monocular mages Prevous factorsaton methods can solve for multple ndependently movng objects, but fal n the artculated case because of a rank defcency n the nput matrx y extendng the shape and moton matrces used n the case of ndependent objects, we derve a reconstructon algorthm for artculated moton The recovered 3D shape ncludes nformaton about the structure of each artculated segment and the pont of rotaton of each segment relatve to ts parent segment Prelmnary experments on synthetc data ndcate that nose tolerance compares favourably to that reported for the case of ndependently movng objects e also present results quantfyng the effect of nput nose on the reconstructon accuracy 1 Introducton It s possble to use an mage sequence to reconstruct the 3 dmensonal shape and moton of objects n the sequence Ths structurefrommoton (SFM) problem has been solved for the smple case of reconstructng a sngle rgd object [1] and later extended to handle multple ndependent rgd objects [2, 3, 4] and a sngle elastc object [5, 6] e present a factorsaton approach for the case of artculated moton 3D reconstructon has been appled to trackng and buldng models of human faces [5] and buldng a model of a buldng exteror [7] e are developng an artculated reconstructon algorthm as part of a larger system to be used for trackng and reconstructng artculated human moton The system wll be used for vocabulary acquston of South Afrcan Sgn Language Here, we assume that 2D trackng has already taken place, usng a method such as dscussed n [8, 9, 10] The trackng algorthm accepts an mage stream as nput and yelds the 2D coordnate paths of ponts on the mage e address the problem of reconstructng the 3D coordnates of these tracked ponts for the case of artculated moton The matrx factorsaton approach to performng ths reconstructon has the followng general steps: place the observed 2D pont coordnates n an observaton matrx, ; model the underlyng moton as, where s the moton matrx and the shape matrx; fnd ntal values for the shape and moton matrces usng the sngular value decomposton (SVD) and known rank constrants on the matrces these rank constrants depend on the type of moton (eg rgd or nonrgd) beng observed; solve for and usng addtonal constrants on the shape and moton matrces as mposed by the nature of 3D shape and moton The matrx factorsaton approach to solvng the SFM problem has been used n the cases where the observed moton s caused by a sngle rgd object [1], multple ndependently movng rgd objects [2, 3, 4], and a sngle elastc object [5, 6] The ranks of the observaton matrces n these cases are 4, and, respectvely, where s the number of ndependent objects for the second case and the number of bass shapes requred to model the elastc moton of the thrd case e descrbe artculated tree moton and show how the matrx factorsaton approach can be adapted to solve for ths case An artculated tree s a sngle object consstng of a number of segments Each segment s assumed rgd and ts moton dependent on exactly one parent segment See Fgure 1 for an example Left thumb Left upper arm Left lower arm Left hand Left ndex fnger Shoulders Fgure 1: An artculated tree Rght upper arm A chld segment s attached to ts parent at exactly one pont and exhbts free 3D rotaton around that pont If the parent

2 ,,,+,+ 0,,,,,,,+ g g T 2 segment undergoes rotaton or translaton, the pont of rotaton of the chld segment and, hence, the poston and orentaton of the chld segment changes accordngly The chld segment does not exhbt translaton ndependent of ts pont of rotaton An object of ths nature can be descrbed usng a tree structure wth parentchld relatonshps n the tree descrbng segment connectvty Artculated tree moton s a degenerate case of the second case lsted above (rank ) The exstence of degeneraces s mentoned n [4] and dscussed n [3] However, no soluton s presented In the remander of the paper, we descrbe the structure of the shape and moton matrces that allow for artculated reconstructon (Secton 2), derve an algorthm that performs the reconstructon (Secton 3), provde expermental results on the performance of the algorthm (Secton 4), dscuss future work (Secton 5) and conclude wth a dscusson of the results (Secton 6) 2 Matrx Structure In the matrx factorsaton approach, the observaton matrx ( ) s factored nto a moton matrx ( ) and an shape matrx ( ) Here, s the number of observed data frames, the number of observed ponts and the rank of the observaton matrx Each row par n contans the coordnates of a tracked pont on a frame of nput e frst present the exstng models for rgd moton before extendng them to the case of artculated moton 21 Rgd Objects Assumng the weakperspectve camera model, rgd moton of a sngle object can be modeled as wth ; see [1] Each row par n correspondng wth the moton n a sngle frame has the form!# %$'& (1) where ( s the frame number under consderaton, ) s the weak perspectve scalng constant, contans the frst two rows of the rotaton matrx and $*) the translaton of the object The shape matrx has the form where each D:E4: > F / :0;24252<=7 > > > descrbes the 3D coordnates of a sngle pont In the case of multple ndependently movng rgd objects, we have *G wth G the number of objects; see [3, 4] Each H *G matrx s of the form I KJ L/M (3) where each s of the form descrbed n (1) has the block A AA L where each s of the form descrbed n (2) Empty matrx entres are A (2) (4) 22 Trees Artculated tree moton can be modeled wth O H P row par n has the form J ) Q 0 RSR4R Q $U M Here, s the number of segments n the artculated object Each where contans the frst two rows of the rotaton matrx for segment V and $ contans the translaton of the root segment the segment wth no parent The shape matrx has the form XY 0Z'Y [ R4R5R 'Y 0 0\Y [ R4R5R 0\Y T [ R4R5R [5Y T T ] 7=^ ] 74_ ] 74` R4R5R ] 7Sa T Here each b # matrx contans the coordnates of the V th segment n ts local coordnate system The local coordnate system of a segment has orgn at the segment pont of rotaton, e the local moton of a segment can be descrbed as a rotaton only Each X ' Pc Y matrx descrbes the relatonshp between segments V and (see below for detals) The notaton ]Dd for a length e row vector of ones wll be used throughout the rest of the paper Fnally, s the number of ponts n segment V such that f, the total number of ponts n the object If segment V has parent segment, d Y d Y c e 24242S Hh \ c Y c Y ] 75j and d Y k [=l*74j e m n242s2s Voh c Y (7) Here s the column vector descrbng the pont of rotaton of segment V n the local coordnate system of parent segment c Y From ths defnton, notce that the columns of an are all equal The structure and moton of any artculated tree can be descrbed usng shape and moton matrces of the form descrbed here 23 Forests The artculated tree case can be extended to allow artculated forests A forest s smply a collecton of trees, e a number of ndependently movng objects where each object s undergong artculated tree moton The moton matrx s now of the same form as n (3), except that each s an artculated tree moton matrx and G s the number of trees n the forest The shape matrx has the same form as n (4) but wth each an artculated tree shape matrx Snce each has rows where s the number of segments n tree V, the number of rows n (and columns n ) s pf qg Subsequently, the observaton matrx has rank n the nondegenerate case ote that a rgd object s smply an artculated tree wth one segment If each artculated tree n the forest has one segment, the observaton matrx wll have rank GrsG G Ths means that the artculated forest smplfes to the case of G ndependent objects, as dscussed n [3, 4] ecause artculated tree moton has fewer degrees of freedom than ndependent object moton, the observaton matrx A AAAAAA \2 (5) (6)

3 &,,,,,+ $ the former has lower rank X as opposed to G Ths shows that the artculated tree formulaton s a degenerate case of G ndependently movng objects In contrast, the artculated forest formulaton s a generalsaton thereof 24 Lmtatons A lmtaton of ths matrx factorsaton method s that the assgnment of ponts to segments both the number of ponts per segment and the order of the ponts and segments can not be computed automatcally In [4] the assgnment of ponts to ndependent objects was computed usng a symmetrc matrx,, whch contans a nonzero entry at ( V f and only f ponts ( and V belong to the same object y reorderng the rows and columns of such that t has a blockstructure, the ponttoobject assgnment s dscovered Ths method fals n the case of artculated moton The dervaton of the structure of s based on the assumpton that has the blockstructure n (4), whch does not hold for (6) Thus far, no analogy to the matrx has been found for ths case urrently, we crcumvent ths problem by annotatng the frst frame of an mage sequence by hand In practce, ths s feasble snce tracked features need to be grouped n one frame only, wth reconstructon beng performed automatcally from there e are explorng automatc segmentaton solutons n ongong work 3 Algorthm e present here a reconstructon algorthm for the artculated tree case, under the assumpton that observed ponts have been assgned correctly to ther respectve segments ecause of the known ponttosegment assgnment, the observaton matrx can be wrtten as! T & where the ndex of a segment s greater than that of ts parent segment Ths also mples that s the root segment Each of these submatrces now represent the moton of a sngle, rgd segment Reconstructon of the artculated tree takes place segment by segment, from left to rght n the moton (5) and shape (6) matrces e use the method for reconstructon of a sngle, rgd object as orgnally proposed n [1] Frstly, we reconstruct the root segment Ths yelds, $ and n (5) and (6) The translaton $ s reconstructed n the frst step, snce the root segment s the only segment that can undergo free translaton All other segments are attached to a parent segment and can undergo rotaton only ext, we reconstruct each of the remanng segments Despte the fact that each segment can undergo rotaton only, we do fnd that the rgd reconstructon algorthm yelds a translaton component Ths s due to two factors The frst s that the rgd reconstructon algorthm chooses an arbtrary coordnate system for the shape matrx However, for artculated moton, we want the orgn of the coordnate system to be at the pont of rotaton of the shape The rgd reconstructon algorthm yelds and we want Here matrx s a Q ] 75jD2 (9) vector that adjusts the orgn of the shape The second factor s that the pont of rotaton of the segment moves n the global coordnate system As the parent of a segment moves, the pont of rotaton of the chld moves wth t Ths movement s descrbed by (5) and (6) For segment V Y where Y s as defned n (7) and $ ] 74j wth columns equal to the global translaton From (10): h Y (10) s a matrx (11) ext, we perform rgd reconstructon on the left hand sde to get h *2 (12) Here, and are the reconstructed rotaton, shape and translaton matrces for segment V and ] 74j The orgn of s stll ncorrect and we wrte wth ] 74j From the equalty of (11) and (12) and then, from (13) Y ext, we factor $ ] 7 j, and ow we can see that D yeldng the requred rotaton matrx and $! Y 0 R4R4R Y Y# : based on ther defntons Y ] 7 j Y 0\Y Y ] 7 A AAAA (13) (14) (15) (16) (17) (18) (19) whch s an overconstraned system that allows us to solve for Y and Y Y From ths and are constructed th, and, we have all the parameters for segment V To summarse, the reconstructon algorthm conssts of the followng steps Reconstruct, and $ Ths yelds the global translaton For each observaton submatrx V :

4 Subtract the global translaton from Reconstruct and Y and from the overconstraned alculate system (19) alculate ] 75j An algorthm for reconstructon n the artculated forest case wll not be descrbed n detal However, note that the block structure descrbed n Secton 23 (see also (4)) allows for fndng a ponttotree assgnment by usng the matrx, as n [3, 4] Knowng ths assgnment, each tree can be reconstructed ndependently usng the method above 4 Experments For the experments presented here, sets of 3D ponts were generated and projected onto a 2D plane to conform wth the weakperspectve camera projecton The advantages of ths are that no 2D trackng s requred and the ponttosegment assgnment requred by the algorthm s generated together wth the pont data Errors can also be ntroduced n a controlled fashon and the effect on the reconstructon analysed Two data sets were generated The frst approxmates a human arm wth a hand and fngers, second approxmates rgd shoulders wth upper and lower arms (Fgure 2) The frst object conssts of 8 segments and a total of 50 ponts and the second object of 5 segments and 150 ponts, both over 50 frames Unform random nose varyng between 01% and 5% of the mage sze was ntroduced Durng each reconstructon run new random nose was added to the clean dataset, wth 100 runs performed per nose level Fgure 2: The shoulders dataset wth 5 segments, each consstng of 30 ponts The accuracy of the reconstructon was tested by comparng the dstances between the real and reconstructed ponts n each segment Snce the reconstructon algorthm outputs the shape matrx n an arbtrary global coordnate system, the output of the algorthm s adjusted by rotatng and scalng t to match the real (known) shape matrx as closely as possble The mean of the Eucldan dstances between all vertces s then taken as the reconstructon error Ths method for calculatng the reconstructon error s useful when comparng the reconstructon accuracy for dfferent levels of nose n the same nput, but not when comparng accuracy between dfferent objects 41 Results The reconstructon algorthm produced vsually acceptable results for all error rates up to 1% Fgure 3 shows the reconstructed shape of one segment from the second data set for dfferent nose levels e = 0 e = 001 e = 0025 Fgure 3: One reconstructed segment from an nput wth 0%, 1% and 25% nose The frst mage shows how ponts were placed n a spral on a cylnder, wth no nose, there s no reconstructon error The second mage shows the reconstructon wth 1% nose and the cylnder s stll clearly vsble th 25% nose, the thrd reconstructon shows serous deformaton A 1% nose level means that n a q mage, erroneous dsplacements of up to 5 pxels can be handled Ths compares well wth [4] and [3] where Gaussan nose wth a varance of 2 pxels was descrbed as typcal large nose levels on a b mage Fgures 4 and 5 dsplay the accuracy vs error rate of the reconstructons ote that the dataponts approxmate a straght lne on the logarthmc axes Ths mples that the reconstructon error s a polynomal functon of the nput error, snce e d 2 2 In our experments e for the arm data set and e for the shoulders data set The growth of the reconstructon error wth ncreasng nose appears to be approxmately lnear 5 Future work Although ntal tests ndcate that the reconstructon algorthm works well on synthetc data sets, tests on real vdeo data have yet to be performed Ths wll test the robustness of the system and ndcate whether the nose tolerance reported here can handle real data In addton, we plan to address the automatc ponttosegment assgnment problem Ths would allow the reconstructon process to be fully automated Varous moton segmentaton algorthms [11, 12, 13, 14] are beng explored Further mprovements to be explored nclude creatng a realtme varaton on the algorthm presented here, and creatng a system that uses feedback from the reconstructon algorthm to the 2D feature tracker A realtme mplementaton would consst of an approxmaton to the artculated tree usng an ntal batch of nput frames,

5 error nose Fgure 4: The reconstructon error for varous nput error rates on the arm data set error nose Fgure 5: The reconstructon error for varous nput error rates on the shoulders data set followed by refnng hen ponts are lost and acqured by the 2D feature tracker, they are assgned to ther segments automatcally by usng the rank4 constrant on the observaton submatrx of each rgd segment Each vdeo frame after the ntal batch s then used to teratvely update the current 3D reconstructon approxmaton Another possblty for a realtme mplementaton s to explore whether the method used n [15] for rgd reconstructon can be adapted to perform sequental reconstructon on an observaton matrx wth rank greater than 4 Addng feedback from the reconstructon algorthm to the 2D tracker could mprove the accuracy of both the tracker and the reconstructon consderably The trackng of vdeo data s stll a challengng problem the occurrence of depth dscontnutes and undrectonal textures lmt the effectveness of the tracker y usng an estmate of the artculated tree, the postons of the tracked vertces can be adjusted to better satsfy the constrants mposed by artculated tree moton In [5, 6], a method based on rank constrants s used to do just ths for the case of an elastc object 6 onclusons e have proposed a method for performng 3D reconstructon of artculated moton usng a matrx factorsaton approach Matrx factorsaton has been appled to rgd moton and multple ndependent rgd moton n the past; here we successfully handle artculated moton wthn the same framework The formulaton presented here allows for the reconstructon of all artculated segments and, n addton, provdes the ponts of rotaton of all segments Artculated tree moton was dscussed as a specal, degenerate, case of ndependent rgd object moton The artculated tree formulaton was then generalsed to handle artculated forests ndependently movng artculated trees Ths generalsaton was shown to be a generalsaton of ndependent rgd moton also Experments on synthetc data sets ndcate good reconstructon even wth large nput nose (1%) Ths error rate s smlar to that reported n prevous work for the case of ndependent rgd object moton 7 References [1] Tomas,, and Kanade, T, Shape and Moton from Image Streams Under Orthography: A Factorzaton Method, Internatonal Journal of omputer Vson, 9(2): , ovember 1992 [2] Adv, G, Determnng threedmensonal moton and structure from optcal flow generated by several movng objects, IEEE Transactons on Pattern Analyss and Machne Intellgence, 7(4): , July 1985 [3] ostera, J and Kanade, T, A Multbody Factorzaton Method for Independently Movng Objects, arnege Mellon Unversty Techncal Report RITR9730, May 1997 [4] ostera, J and Kanade, T, A Multbody Factorzaton Method for Independently Movng Objects, Internatonal Journal of omputer Vson, 29(3): , September 1998 [5] regler,, Hertzmann, A and ermann, H, Recoverng onrgd 3D Shape from Image Streams, IEEE onference on omputer Vson and Pattern Recognton, 2000 [6] Torresan, L, Yang, D, Alexander, G and regler,, Trackng and Modelng onrgd Objects wth Rank onstrants, IEEE onference on omputer Vson and Pattern Recognton, 2001 [7] Azarbayejan, A J, Galyean, T, Horowtz, and Pentland, A, Recursve Estmaton of AD Model Recovery, Proceedngs of the 2nd ADased Vson orkshop, pp [8] lake, A, Isard, M and Reynard, D, Learnng to Track the Vsual Moton of ontours, Artfcal Intellgence, 78: , 1995 [9] Lucas, D and Kanade, T, An Iteratve Image Regstraton Technque th an Applcaton to Stereo Vson, Proceedngs of the 7th Internatonal Jont onference on Artfcal Intellgence, pp , 1981 [10] Sh, J and Tomas,, Good Features to Track, IEEE omputer Socety onference on omputer Vson and Pattern Recognton, pp , June 1994

6 [11] Iran, M, Rousso, and Peleg, S, omputng Occludng and Transparent Motons, Internatonal Journal of omputer Vson, 12(1):5 16, 1994 [12] Machlne, M, ZelnkManor, L and Iran, M, Multbody Segmentaton: Revstng Moton onsstency, orkshop on Vson and Modellng of Dynamc Scenes, June 2002 [13] Torres, L, Garca, D and Mates, A, A Robust Moton Estmaton and Segmentaton Approach to Represent Movng Images th Layers, Internatonal onference on Acoustcs, Speech, and Sgnal Processng, 4: , 1997 [14] Vasconcelos, and Lppman, A, Emprcal ayesan EMbased Moton Segmentaton, IEEE onference on omputer Vson and Pattern Recognton, pp , 1997 [15] Morta, T, and Kanade, T, A Sequental Factorzaton Method for Recoverng Shape and Moton from Image Streams, IEEE Transactons on Pattern Analyss and Machne Intellgence, 19(8): , August 1997

Structure from Motion

Structure from Motion Structure from Moton Structure from Moton For now, statc scene and movng camera Equvalentl, rgdl movng scene and statc camera Lmtng case of stereo wth man cameras Lmtng case of multvew camera calbraton