3D Reconstruction of a Smooth Articulated Trajectory from a Monocular Image Sequence

Transcription

1 3D Reconstuction of a Smooth Aticulated Tajectoy fom a Monocula Image Sequence Hyun Soo Pak and Yase Sheikh Canegie Mellon Univesity 5 Fobes Ave., Pittsbugh, PA, 523 {hyunsoop,yase}@cs.cmu.edu Abstact An aticulated tajectoy is defined as a tajectoy that emains at a fixed distance with espect to a paent tajectoy. In this pape, we pesent a method to econstuct an aticulated tajectoy in thee dimensions given the two dimensional pojection of the aticulated tajectoy, the 3D paent tajectoy, and the camea pose at each time instant. This is a coe challenge in econstucting the 3D motion of aticulated stuctues such as the human body because endpoints of each limb fom aticulated tajectoies. We simultaneously apply activity-independent spatial and tempoal constaints, in the fom of fixed 3D distance to the paent tajectoy and smooth 3D motion. Thee exist two solutions that satisfy each instantaneous 2D pojection and aticulation constaint (a ay intesects a sphee at up to two locations) and we show that esolving this ambiguity by enfocing smoothness is equivalent to solving a binay quadatic pogamming poblem. A geometic analysis of the econstuction of aticulated tajectoies is also pesented and a measue of the econstuctibility of an aticulated tajectoy is poposed.. Intoduction Reconstucting a moving point in thee dimensions fom a sequence of two dimensional pojections is an ill-posed poblem; any point on the line of pojection connecting the camea s optical cente and an image measuement can be a solution. Yet, humans can effotlessly peceive depth if the 2D points coespond to aticulations of a known skeleton [2]. We study the conjectue that if 3D points move smoothly with a known aticulation stuctue, then it is possible to econstuct thei 3D locations fom thei 2D pojections without any activity-specific pio. The econstuction of an aticulated tajectoy has a fundamental ambigu- Poject page: iccv2/iccv_poject_page.html ity because thee ae two intesecting points that satisfy an aticulation constaint and an image measuement at each time instant [23]: fo a 2D tajectoy of F fames, thee ae 2 F 3D tajectoies that emain at fixed distance to a paent tajectoy. The econstuction of a smooth tajectoy without spatial constaints is also known to be fundamentally ambiguous when the camea tajectoy is smooth [26, 28]. We pesent an algoithm to econstuct a smooth aticulated tajectoy in 3D by simultaneously applying aticulation and smoothness constaints. The algoithm takes as input 2D pojections of the tajectoy, its paent tajectoy in 3D, and the camea pose at each time instant. We pesent a measue of econstuctibility of an aticulated tajectoy which chaacteizes the stability of estimation unde aticulation and smoothness constaints. Each tajectoy is paameteized by coefficients of a tajectoy basis in the spheical coodinate system to enfoce smoothness and aticulation constaints. We show that if a tajectoy is embedded in the tajectoy basis and aticulation constaints ae applied, the econstuction poblem is equivalent to a binay quadatic pogam which is known to be NP-had [5]. A numbe of algoithms exist that poduce an appoximate solution [24, 25, 3] and we use a banchand-bound method to poduce an initialization. We efine the aticulated tajectoies by minimizing epojection eo. The esults ae smooth, length peseved 3D tajectoies. We have applied ou algoithm to ecusively econstuct the 3D motion of a human given the 3D motion of its oot. Two geneal appoaches have been exploed in pio liteatue to econstuct human aticulated body motion. Data-diven appoaches use epositoies of exemplas to ovecome the ambiguity [2,9,34,35,37] and physics-based appoaches use dynamical models of the human body to fit to the image steam [, 36, 38]. Unlike these appoaches, ou appoach econstucts human motion fom puely geometic constaints. Thus, the taget motion is not confined to pe- The paent tajectoy in a skeleton hieachy is the poximal tajectoy to the oot tajectoy and the child tajectoy is the distal tajectoy.

2 2 S 3 R X 2 3 R 3 R O X X 2 O X X 2 X P P 3 P 3 P 3 P P P 2 P P P 2 2 3K P 2 P P P 2 2 3K P K P P S R (a) Fixed length tajectoy (b) Relative tajectoy (c) Tajectoy on a sphee Figue. (a) An aticulated tajectoy is defined as a tajectoy X 2 which peseves distance fom its paent tajectoy X acoss all time instances. (b) The aticulated tajectoy is tansfomed to the elative tajectoy, X 2 X, by collapsing X to the oigin. (c) The aticulated tajectoy lies on a sphee of adius. Thee ae two intesecting points at each time instant between the sphee and the ay connecting the camea s optical cente and an image measuement, which allow 2 F possible 3D tajectoies. defined activities o view points. We defe a detailed discussion of elated wok to the end of the pape and pesent the geomety of the poblem and ou poposed algoithm in the sequel. 2. Geomety of an Aticulated Tajectoy A point tajectoy in 3D without any constaint can be epesented by a seies of points: X = x. x F, y. y F, z. z F, () whee (x i, y i, z i ) is the Catesian coodinate of a point at i th time instant and F is the numbe of fames. If a tajectoy is smooth, it is known that the tajectoy can be expessed by a linea combination of a compact tajectoy basis [2], i.e., X = (Θa x, Θa y, Θa z ) (2) whee Θ is a F K matix composed of a collection of linea tajectoy basis, a is the coefficients o the paametes of a tajectoy, and K is the numbe of basis. If two tajectoies, X and X 2, ae aticulated, the distance between tajectoies emains constant acoss all time instances as shown in Figue (a), i.e., x 2 i + y 2 i + z 2 i = 2, i =,, F, (3) whee X = X 2 X is the elative tajectoy. When a pespective camea captues these two tajectoies, points on the tajectoies at the time instant ae pojected onto the camea plane. The camea epesentation[ in this pape ] is a 3 4 pojection matix, P i = KR i I3 C i whee I3, K, R i, and C i ae a 3 3 identity matix, the uppe tiangula intinsic matix, the camea otation matix, and the camea s optical cente vecto at the i th time instant, espectively. If we tansfom one of the tajectoies, X, to the oigin, O, the othe tajectoy, X 2, maps to the elative tajectoy, X, and a camea, P i, maps to the elative camea pose, P i with espect to X as shown in Figue (b). The tansfomed elative tajectoy lies on a sphee with adius. Thee ae two points intesecting the sphee and the ay connecting the camea s optical cente and an image measuement at each time instant as shown in Figue (c). All intesecting points ae candidate 3D points which the elative tajectoy passes and thus, thee ae 2 F possible elative tajectoies. The epesentation of a elative tajectoy between the aticulated tajectoies fom Equation (2) (Catesian coodinate epesentation) has to meet the additional quadatic equality constaints of Equation (3). Instead of the Catesian coodinate epesentation, we intoduce the spheical coodinate epesentation fo a elative tajectoy to contol the distance between tajectoies, explicitly, i.e., X = (Θa θ, Θa φ, ), (4) whee θ is inclination fom the z axis, φ is azimuth fom the x axis in the xy plane, and is the adius. This epesentation enables us to descibe an aticulated tajectoy pecisely because it satisfies the tempoal constaint and the length constaint simultaneously egadless of paametes by setting the adius constant explicitly. It also enfoces that all imputed points between fames satisfy the aticulation constaint while the Catesian epesentation does not. Fo a topological point of view, the econstuction fom the spheical coodinate system is the mapping of P 2F S 2K R while the econstuction fom the Catesian coodinate system is the mapping of P 2F P 3K as shown in Figue (c).

3 3. Method In this section, we pesent an algoithm fo ecoveing a tajectoy which satisfies spatial and tempoal constaints using the spheical coodinate epesentation of a elative tajectoy pesented in the pevious section. 3.. Objective Function of 3D Reconstuction Fom the spheical coodinate epesentation, we econstuct smooth aticulated tajectoies which minimize the epojection eos: F,P agmin X,, X P d (x ij, ˆx ij ), (5) i,j whee X j is the j th aticulated (o elative) tajectoy paameteized by (Θa θ,j, Θa φ,j, j ), d(, ) is the L 2 distance between two aguments, P is the numbe of aticulated points, and x ij and ˆx ij ae a 2D image measuement and a epojection of the j th point tajectoy at the i th time instant, espectively. If aticulated tajectoies ae sequentially linked, the tajectoies ae X j = f(x R ; X,, X j ), (6) whee f( ) is the fowad kinematic function that takes the oot tajectoy, X R, and all paent elative tajectoies, X,, X j, and outputs the j th tajectoy, X j, in the Catesian coodinate system. The epojection, ˆx ij is ( P i ˆx ij = X j (i) P 3 X i j (i), P2 X ) i j (i) P 3 X, (7) i j (i) whee P l i is the lth ow of the camea pojection matix at the i th time instant and X j (i) is the homogeneous epesentation of the i th point in the j th tajectoy, X j (i) Initialization of Equation (5) The objective function of Equation (5) is highly nonlinea and diect optimization falls into a local minimum. Theefoe, a good initialization of tajectoy paametes is necessay. When the paent joint position and the length between tajectoies ae known, thee ae two intesecting points between a sphee whose oigin is the paent joint position, X p, and a line connecting an image measuement and camea optical cente, C, at each time instant as shown in Figue 2(a). A point lying on the line is C + sv whee s is an unknown scala and v is the diection of the pojection, i.e.,v = R T K [ x T ] T. Then, the intesecting points ae X = C + s v, 2 X = C + s 2 v, (8) whee s,2 = vt C ± (v T C) 2 v 2 ( C 2 2 ) v 2 (9) and C = C X p. Fo each time instant, we have two candidate 3D points though which the econstucted tajectoy must pass. Acoss all time instances, thee ae 2 F possible tajectoies which satisfy the image measuements. Among those tajectoies, we look fo the tajectoy best descibed by the tajectoy basis. Let χ be the elative diection vecto with espect to the paent point as shown in Figue 2(a). Fo each time instant, χ i takes eithe χ i o 2 χ i, i.e., χ i = χ i b i + 2 χ i ( b i ), = ( χ i 2 χ i )b i + 2 χ i, whee b i {, }. () Then, all possible tajectoies can be epesented as: χ χ 2 χ. =... b +. χ F χ 2 F χ F o χ = Eb + F, () whee b is a binay vaiable vecto, χ i and 2 χ i ae two elative diection vectos, and χ i = χ i 2 χ i. Finding the best tajectoy is equivalent to finding the binay vecto, b, which minimizes the following cost, b = agmin ( ΘΘ T I ) (Eb + F) 2, (2) b subject to b {, } F. Note that ΘΘ T I is the pojection opeation onto the null space of the tajectoy basis, Θ. Equation (2) is a quadatic poblem ove binay vaiables. A binay quadatic pogamming poblem is NP-had in geneal. The stuctue of ou poblem does not fall into one of the solvable cases; ou quadatic matix has positive offdiagonal elements [3], is a non-singula matix [3, 4], and cannot be epesented by a ti-/five-diagonal matix [6]. Also, the undelying gaph stuctue is not seies paallel [6]. Thus, in theoy, this is an intactable poblem. Howeve, a numbe of appoaches have been poposed to appoximate a solution of the poblem efficiently using spectal o semidefinite elaxation. A banch-and-bound outine 2 with binay elaxation is one technique fo global optimization. Since ou quadatic matix is positive definite, the objective function behaves convexly in a banched ectangle, which enables us to define a tight lowe bound of the ectangle in polynomial time. 2 hybid/tools/miqp/

4 C x 2 X v 2 χ X P (a) Intesection χ X t =2 t = α Â l ˆB B γ γ C X (b) Relative tajectoy X 2 X Figue 2. (a) Thee ae two solutions, X and 2 X which satisfy the aticulation constaint and an image measuement. (b) Aticulated tajectoy and the camea pose ae tansfomed with espect to the paent tajectoy. Once b is ecoveed, we poject χ = Eb + F onto the tajectoy basis space of the spheical coodinate system to poduce low dimensional paametes, i.e., X = (Θa θ, Θa φ, ). This yields an accuate initialization which can be efined by nonlinea optimization of Equation (5). When the elative tajectoy, X, passes a singula point in the spheical coodinate system in the pocess of pojecting χ onto the spheical coodinate system, a discontinuity of angula tajectoy occus. Fo example, when φ passes fom ɛ > to 2π ɛ, this esults in a discontinuity of the angula tajectoy because φ is defined in the inteval [, 2π). To deal with discontinuous tajectoies, we find the best angula epesentation among all spheical epesentations of χ which peseves local continuity by allowing the domains of θ and φ to be (, ). 4. Reconstuctibility We now exploe the econstuction ambiguity of an aticulated tajectoy and analyze configuations in which the econstuction is accuate. Let X be a known paent tajectoy and X 2 be an aticulated child tajectoy which ae obseved at two time instances as shown in Figue 2(b). The gound tuth elative tajectoy between X and X 2 moves fom A to B. Â and ˆB ae imposto points that satisfy the image measuements as well as the aticulation constaint. In this section, we show that the elationship between the tue tajectoy and the imposto tajectoy inheently detemines the econstuction accuacy. We define a measue of econstuctibility of an aticulated tajectoy, η a, as a citeion to chaacteize econstuction accuacy whee η a = Θ a γ C Θ a γ X, (3) γ X = Θa γx + Θ a γ X, γ C = Θa γc + Θ a γ C, and Θ is the null space of the tajectoy basis. If the econstuctibility of an aticulated tajectoy goes to infinity, thee exists a unique solution and it coesponds to the gound tuth tajectoy. This can be poven by the following. Fo A each time instant, thee ae two intesecting points and an estimation should be one of them: ˆγ = ( b)γ X + bγ C, b = o (4) whee ˆγ is an estimated angle. Fo an estimated angula tajectoy, ˆγ = (I B) γ X + Bγ C, (5) whee B is a diagonal matix whose enties takes eithe o. The best tajectoy epesented by the tajectoy basis minimizes following: agmin Θâ ˆγ 2 (6) â,b = agmin Θâ (I B) γ X Bγ C 2 (7) = â,b agmin Θâ (I B) Θa γx BΘa γc â,b (I B) Θ a γ X BΘ a γ C 2. (8) Reconstuctibility of an aticulated tajectoy goes to infinity when Θ a γ C o Θ a γ X. Fo eithe case, B has to appoach to eliminate the esidual of the null components in Equation (8), which leads to â a γx. Fom the method of Pak et al. [28], if the camea motion is slow o stationay, thee is no way to econstuct an accuate tajectoy using the tajectoy basis because it spans the camea tajectoy well. The econstuctibility of an aticulation states that if the paent tajectoy is independent of the camea tajectoy, the tajectoy econstuction is still possible because mixed motion between the camea and the paent motions induces α motion whee α is the tajectoy of viewing angles fom a camea, α, as shown in Figue 2(b). Even when camea and paent motions ae stationay, the econstuction is possible if γ X Θ because each α is a nonlinea function of γ X, i.e., α = tan (sin γ X /(l + cos γ X )) whee l is the distance between the paent tajectoy and camea tajectoy, and thus α / Θ and γ C / Θ unless l = o l = (i.e., othogaphic pojection) as shown in Figue 2(b). Figue 3(a) shows the distibution of 3D econstuction eo with espect to econstuctibility of an aticulated tajectoy, η a, fom the CMU motion captue data 3. A tajectoy initialized by binay quadatic pogamming is the best fitted tajectoy by the tajectoy basis. When η a is high ( ), 3D econstuction eo of an aticulated tajectoy is low because the gound tuth tajectoy is well descibed by the tajectoy basis and the gound tuth tajectoy and the imposto tajectoy ae well sepaable. In contast, when η a is low ( ), ou solution conveges to the imposto tajectoy because the tajectoy basis spans the imposto tajectoy bette. 3

5 ηa 2 4 (a) Reconstuctibility.2 Eo in an aticulated tajectoy Eo in the paent tajectoy Root tajectoy eo, e/ a (b) Eo in the oot tajectoy.5 Small image measuement noise Lage image measuement noise 3D econstuction eo.8 3D econstuction eo 3D econstuction eo 3D econstuction eo Initial adius scale Small image measuement noise Lage image measuement noise (c) Initialization eo of adius Missing data (%) (d) Effect of missing data Figue 3. (a) The accuacy of the econstuction is high when ηa is geate than whee the tajectoy basis spans the gound tuth tajectoy bette than the imposto tajectoy. (b) Pefomance of ou algoithm against eo in the oot tajectoy, (c) the initialization eo of the adius, (d) amount of missing data ae illustated. 5. Results To validate ou method, we tested it with the HumanEvaII dataset, synthesized tajectoies, and the CMU motion captue data quantitatively and with eal human motion examples taken by video cameas qualitatively. We use the fist K Discete Cosine Tansfom (DCT) basis4 in ode of inceasing fequency and the numbe of basis is chosen manually to span the tajectoy well. 5.. Quantitative Evaluation We compae ou method with the state-of-at human pose estimation [4, 8, 2, 29] using the HumanEva-II dataset5. Subject S2 with camea C is used to econstuct the aticulated tajectoies. Ou method esults in 28.8mm of 3D mean eo with 7.75mm standad deviation. This eo is compaable to the eo of the state-of-at pose estimation algoithms (82mm 2.4mm). It should be noted that while all methods ely on activity specific taining data to econstuct motions, ou appoach uses only activity independent geometic constaints. We geneate synthetic 2D pespective pojections fom synthetic data and the CMU motion captue data and evaluate fo thee aspects: eo in the oot tajectoy, eo in adius of an aticulated tajectoy, and missing data. Fo evaluation of eos in the oot tajectoy and adius, we set the camea stationay and vay eo of the oot tajectoy and adius eo while the oot position is moving. Fo the evaluation of missing data, we atificially emove 2D pojections andomly. We measue 3D econstuction eo of an aticulated tajectoy by vaying the atio between the aveage distance eo of the oot tajectoy, e, and the adius of the aticulated tajectoy, a, as shown in Figue 3(b). The eo in the paent tajectoy is a lowe bound on the econstuction eo 4 Hamidi and Peal [7] have shown that the DCT povides the optimal pefomance to encode the signal unde the fist ode Makov pocesses. Ahkte et al. [2] have empiically justified its optimality on motion captue data. 5 of the aticulated tajectoy. While the vaiance of the distibution fo small oot tajectoy eo (<.2) is low, i.e., the econstuction can be done eliably, the econstuction fom high oot tajectoy eo (>.3) causes high eo in the child tajectoy as well. Fo the evaluation of the eo in adius, we measue 3D econstuction eo fo eoneous adii multiplied by scale6. Figue 3(c) illustates obustness to eoneous initialization. Even though the initial scale is small (i.e., 2 ), the 3D econstuction can be done eliably because befoe solving the binay quadatic pogamming, we adjust the adius of the sphee to intesect with the line of pojection at one point at least. When the initial scale is high (> ), howeve, the econstuction becomes uneliable because the ay intesects with the sphee at all time instances and the optimization falls into a local minimum aound a mis-estimated tajectoy. We also test with the CMU motion captue data fo the evaluation of missing data caused by occlusion o measuement failue. When thee ae missing data, ou spatial and tempoal constaints enable us to impute missing points. Fo this expeiment, we atificially intoduce length eos, image measuement noise, and oot tajectoy eo while the camea is stationay. Ou algoithm poduces an aveage eo7 of 3% fo 5% missing data as shown in Figue 3(d) Qualitative Evaluation We apply ou algoithm to econstuct human body motion in 3D fom 2D pespective pojections. Reconstuction fom a stationay camea and a moving camea ae tested and the statistical anthopometic length atio of the human body is used fo the initialization of length atio with some modifications fo accuate skeleton estimation pupose. The scale of the skeleton is oughly initialized and we manually label image measuements fo aticulated points. Figue 4(a) and Figue 4(b) show the econstuction of the juggling motion and the motion in font of a webcam, 6 Initial adius scale eo means the gound tuth. = X X / X, whee X is the gound tuth tajectoy and X is the estimated tajectoy. 7 eo

6 espectively, in 3D fom a stationay video camea. We poject the 2D oot tajectoy to the unit depth plane and use it as the 3D oot tajectoy because the depth of the oot tajectoy is undedetemined fom a stationay camea. Fo both expeiments, we use the toso as the oot. Fom the oot tajectoy, all aticulated tajectoies ae econstucted ecusively. We also apply ou method to data captued fom a moving camea to ecove the playing cad motion and the yoga motion as shown in Figue 4(c) and Figue 4(d), espectively. Both camea tajectoies ae smooth and well spanned by the tajectoy basis. Fo the econstuction of the oot tajectoy, we choose a elatively igid pat of human body though a sequence and econstuct them using the stuctue fom motion algoithm. Once elative camea poses ae estimated fom the igid pat of the human body, we estimate the similaity tansfom between the elative camea poses and the oiginal camea poses estimated by 3D static stuctue. Head and toso ae used as the oot fo playing cad motion and yoga motion, espectively. 6. Related Wok Points on objects involve spatial and tempoal constaints in geneal. Begle et al. [] intoduced a spatial constaint by assuming that a shape undegoes a defomation which can be well expessed by a linea combination of a compact shape basis. Subsequent wok has discussed ways to select the basis and bilinea optimization schemes that sepaate infomation of camea motion and stuctue [9, 33, 39]. Fo multiple igid bodies, Costeia and Kanade [3] poposed a factoization method using 2D tajectoies in an image steam. Yan and Pollefeys [4] extended this noting that thee is ank loss when two igid bodies ae aticulated. In the tempoal domain, Akhte et al. [2] poposed a linea tajectoy basis using the Discete Cosine Tansfom (DCT) by assuming that an undelying 3D tajectoy moves continuously and smoothly. With a calibated camea, an algebaic solution of tajectoy econstuction in 3D fo specific algebaic cuves (lines and conics) has been poposed by Avidan and Shashua [5] and late it was genealized by Kaminski and Teiche [22]. Pak et al. [28] poposed a linea solution fo the tajectoy econstuction. Pevious methods exploit eithe spatial constaints o tempoal constaints. In ou method, we use the spatial constaint in the fom of aticulation and the tempoal constaint in the fom of smoothness. Fo human motion in paticula, human pose estimation fom a single image by applying a spatial constaint (skeletal stuctue) was poposed by Taylo [32] (paameteization of limb lengths by a scala), by Baon and Kakadiais [7] (joint motion constaint fom the anthopometic statistics), by Paameswaan and Chellappa [27] (camea pose estimation fom head oientation and igidity of toso), and by Agawal and Tiggs [] (silhouette based egession). Human motion estimation fom an image sequence of a monocula camea has been studied as an extension of human pose estimation. Two popula appoaches have been exploed: data diven appoach and physics based appoach. Data diven appoach is to lean low dimensional subspace o latent vaiables that contol undelying human skeletal motion fully using motion captue data o annotated video data. Sidenbladh et al. [8] applied a Bayesian famewok fo 3D human pose tacking using a geneative model of the human body and a pio distibution defined by a tempoal dynamics model. Howe et al. [9] showed Baysian leaning, Choo and Fleet [2] sampled high dimensional taining space fom hybid Monte Calo method, and Utasun et al. [34] used Pinciple Coodinate Analysis (PCA) fo leaning of specific motion (e.g. walking and golfing). Like Taylo s wok [32], Wei and Chai [37] intoduced a geometic solution of motion econstuction using the bone symmetic constaint fom biomechanical data. Valmade and Lucey [35] discussed the validity of Wei and Chai [37] s wok and extended thei algoithm using a stuctue fom motion scheme. Recently, physics based appoaches have eceived attention. Bubake et al. [] have shown econstuction of a bipedal locomotion fom a dynamical model and Vondak et al. [36] have applied multibody dynamics simulation to infe the most plausible human motion in 3D. Wei and Chai [38] have built an inteactive system that integates a dynamical model to captue motion fom a video. Unlike pevious methods, ou appoach elies puely on a geometic intepetation of the aticulation constaint by paameteizing a tajectoy in a way that satisfies both spatial and tempoal constaints simultaneously. 7. Discussion In this pape, we study an aticulated tajectoy which emains at a constant distance with espect to the paent tajectoy. The elative tajectoy is a tajectoy on a sphee and thee ae 2 F tajectoies that meet the spatial constaint and image measuements. Among those tajectoies, we look fo the best tajectoy spanned by the tajectoy basis and we identify that this is equivalent to solving a binay quadatic pogamming poblem. The elative tajectoy obtained by the binay quadatic pogam is paameteized by a compact tajectoy basis in the spheical coodinate system, which satisfies spatial and tempoal constaints, simultaneously. We optimize the tajectoy by minimizing epojection eo. Reconstuction of the aticulated tajectoy is fundamentally limited by the motion induced by the camea and the paent tajectoy and we popose a measue of econstuctibility of an aticulated tajectoy, which chaacteizes the econstuction accuacy. Ou esults show that we ae able to econstuct highly aticulated human motions fom a stationay camea and a moving camea.

7 Acknowledgements The authos wish to thank Tomas Simon and the anonymous eviewes fo thei invaluable comments. This wok was suppoted by NSF gant IIS Refeences [] A. Agawal and B. Tiggs. Recoveing 3D human pose fom monocula images. PAMI, [2] I. Akhte, Y. Sheikh, S. Khan, and T. Kanade. Tajectoy space: A dual epesentation fo nonigid stuctue fom motion. PAMI, 2. 2, 5, 6 [3] K. Allemand, K. Fukuda, T. M. Liebling, and E. Steine. A polynomial case of unconstained zeo-one quadatic optimization. Mathematical Pogamming, 2. 3 [4] M. Andiluka, S. Roth, and B. Schiele. Monocula 3d pose estimation and tacking by detection. In CVPR, 2. 5 [5] S. Avidan and A. Shashua. Tajectoy tiangulation: 3D econstuction of moving points fom a monocula image sequence. PAMI, 2. 6 [6] F. Baahona. A solvable case of quadatic - pogamming. Discete Applied Mathematics, [7] C. Baon and I. A. Kakadiais. Estimating anthopomety and pose fom a single image. In CVPR, 2. 6 [8] M. Begtholdt, J. Kappes, S. Schmidt, and C. Schnö. A study of pats-based object class detection using complete gaphs. IJCV, 2. 5 [9] M. Band. A diect method fo 3D factoization of nonigid motion obseved in 2D. In CVPR, [] C. Begle, A. Hetzmann, and H. Biemann. Recoveing non-igid 3D shape fom image steams. In CVPR, [] M. A. Bubake and D. J. Fleet. The kneed walke fo human pose tacking. In CVPR, 28., 6 [2] K. Choo and D. J. Fleet. People tacking using hybid monte calo filteing. In ICCV, 25., 6 [3] J. P. Costeia and T. Kanade. A multibody factoization method fom independently moving objects. IJCV, [4] J.-A. Feez, K. Fukuda, and T. M. Liebling. Solving the fixed ank convex quadatic maximization in binay vaiables by a paallel zonotope constuction algoithm. Euopean Jounal of Opeations Reseach, [5] M. R. Gaey and D. S. Johnson. Compute and Inteactability: A guide to the theoy of NP-Completeness. Feeman, 979. [6] S. Gu. Polynomial time solvable algoithms to binay quadatic pogamming poblems with q being a ti-diagonal o five-diagonal matix. In WCSP, 2. 3 [7] M. Hamidi and J. Peal. Compaison of the cosine and fouie tansfoms of makov- signal. IEEE Tans. Acoust. Speech, Signal Pocess, [8] M. J. B. Hedvig Sidenbladh and D. J. Fleet. Stochastic tacking of 3D human figues using 2D image motion. In ECCV, 2. 6 [9] N. R. Howe, M. E. Leventon, and W. T. Feeman. Bayesian econstuction of 3D human motion fom single-camea video. In NIPS, 999., 6 [2] Z. L. Husz, A. M. Wallace, and P. R. Geen. Evaluation of a hieachical patitioned paticle filte with action pimitives. In CVPR Wokshop, [2] G. Jansson, S. S. Begstom, and W. Epstein. Peceiving Events and Objects. Lawence Elbaum, 994. [22] J. Y. Kaminski and M. Teiche. A geneal famewok fo tajectoy tiangulation. Jounal of Mathematical Imaging and Vision, [23] H.-J. Lee and Z. Chen. Detemination of 3D human body postues fom a single view. Compute Vision, Gaphics, and Image Pocessing, 985. [24] P. Mez and B. Feisleben. Geedy and local seach heuistics fo unconstained binay quadatic pogamming. Jounal of Heuistics, 22. [25] C. Olsson, A. P. Eiksson, and F. Kahl. Solving lage scale binay quadatic poblems: Spectal methods vs. semidefinite pogamming. In CVPR, 27. [26] K. E. Ozden, K. Conelis, L. V. Eycken, and L. V. Gool. Reconstucting 3D tajectoies of independently moving objects using geneic constaints. CVIU, 24. [27] V. Paameswaan and R. Chellappa. View independent human body pose estimation fom a single pespective image. In CVPR, [28] H. S. Pak, T. Siatoi, I. Matthews, and Y. Sheikh. 3D econstuction of a moving point fom a seies of 2D pojections. In ECCV, 2., 4, 6 [29] P. Peusum, S. Venkatesh, and G. West. A study on smoothing fo paticle-filteed 3D human body tacking. IJCV, 2. 5 [3] J. C. Picad and P. M. Ratliff. Minimal cost cut equivalent netwoks. Management Science, [3] S. Poljak, F. Rendl, and H. Wolkowicz. A ecipe fo semidefinite elaxation fo (,)-quadatic pgamming. Jounal of Global Optimization, 995. [32] C. Taylo. Reconstuction of aticulated objects fom point coespondences in a single uncalibated image. In CVIU, 2. 6 [33] L. Toesani, A. Hetzmann, and C. Begle. Nonigid stuctue-fom-motion: Estimating shape and motion with hieachical pios. PAMI, [34] R. Utasun, D. J. Fleet, and P. Fua. Tempoal motion models fo monocula and multiview 3-D human body tacking. CVIU, 26., 6 [35] J. Valmade and S. Lucey. Deteministic 3D human pose estimation using igid stuctue. In ECCV, 2., 6 [36] M. Vondak, L. Sigal, and O. C. Jenkins. Physic simulation fo pobabilistic motion tacking. In ECCV, 28., 6 [37] X. Wei and J. Chai. Modeling 3D human poses fom uncalibated monocula images. In ICCV, 29., 6 [38] X. Wei and J. Chai. Videomocap: Modeling physically ealistic human motion fom monocula video sequences. SIG- GRAPH, 2., 6 [39] J. Xiao, J. Chai, and T. Kanade. A closed-fom solution to non-igid shape and motion ecovey. IJCV, [4] J. Yan and M. Pollefeys. A factoization-based appoach fo aticulated nonigid shape, motion and kinematic chain ecovey fom video. PAMI, 28. 6

8 (a) Juggling motion fom a stationay camea (b) Motion in font of a webcam fom a stationay camea (c) Playing motion fom a moving camea (d) Yoga motion fom a moving camea Figue 4. (a) Juggling motion, (b) motion in font of the webcam fom a stationay camea, (c) playing cad motion, and (d) yoga motion fom a moving camea. Image measuements ae supeimposed on images in the top ow and 3D econstuction of the motion coesponding to the images ae shown fom diffeent views in the second and the thid ows. The ight-most figues summaize motion by showing whole tajectoies.