Correspondence-free Synchronization and Reconstruction in a Non-rigid Scene

Transcription

1 Correspondence-free Synchronzaton and Reconstructon n a Non-rgd Scene Lor Wolf and Assaf Zomet School of Computer Scence and Engneerng, The Hebrew Unversty, Jerusalem 91904, Israel e-mal: {lwolf,zomet}@cs.huj.ac.l Abstract 3D reconstructon of a dynamc non-rgd scene from features n two cameras usually requres synchronzaton and correspondences between the cameras. These may be hard to acheve due to occlusons, wde base-lne, dfferent zoom scales, etc. In ths work we present an algorthm for reconstructng a dynamc scene from sequences acqured by two uncalbrated non-synchronzed fxed affne cameras. It s assumed that (possbly) dfferent ponts are tracked n the two sequences. The only constrant used to relate the two cameras s that every 3D pont tracked n one sequence can be descrbed as a lnear combnaton of some of the 3D ponts tracked n the other sequence. Such constrant s useful, for example, for artculated objects. We may track some ponts on an arm n the frst sequence, and some other ponts on the same arm n the second sequence. On the other extreme, ths model can be used for generally movng ponts tracked n both sequences wthout knowng the correct permutaton. In between, ths model can cover non-rgd bodes, wth local rgdty constrants. 1

2 We present lnear algorthms for synchronzng the two sequences and reconstructng the 3D ponts tracked n both vews. Outler ponts are automatcally detected and dscarded. The algorthm can handle both 3D objects and planar objects n a unfed framework, therefore avodng numercal problems exstng n other methods. 1 Introducton Many tradtonal algorthms for reconstructng a 3D scene from two or more cameras requre the establshment of correspondences between the mages. Ths becomes challengng n some cases, for example when the cameras have dfferent zoom factors, or large vergence (wde-baselne stereo) [14, 11]. Usng a movng vdeo camera rather than a set of statc cameras helps n overcomng some of the correspondence problems, but may decrease the stablty and accuracy of the reconstructon. Moreover, the reconstructon from a movng camera becomes harder f not mpossble when the scene s not rgd. In ths paper we present an algorthm for reconstructng a non-rgd scene from two fxed vdeo cameras. It s assumed that feature ponts are tracked for each of the cameras, but no correspondences are avalable between the cameras. Moreover, The ponts tracked by the two cameras may be dfferent. Instead we use a weaker assumpton: Every 3D pont tracked n the second sequence can be descrbed as a lnear combnaton of some of the 3D ponts tracked n the frst sequence. The coeffcents of ths lnear combnaton are unknown but fxed throughout the sequence. Snce the cameras vew the same scene, ths assumpton s reasonable. For example, when a pont tracked n the second camera belongs to some rgd part of the scene, t can be expressed as a lnear combnaton of some other ponts on the same part, tracked n the frst camera. Our non-rgdty concept s qute strong. We put no explct lmtaton on the moton of the ponts from one tme to another. The lnear combnaton wth fxed coeffcents assumpton suggests that there exst local rgd structure or that some of the ponts, whch are allowed to move freely, are tracked n both sequences (although we do not know the correspondence between the sequences). Ths paper s dvded to two man sectons. Frst, we show how to synchronze two vdoe 2

3 sequences. 1.1 Prevous work Usng several sequences wthout correspondences between them was done n [4] for algnment n space and tme, and was followed by [18] for synchronzaton and self-calbraton of cameras on a stereo rg. The algorthms suggested compute the moton matrces of each camera ndependently, whch s problematc for dynamc scenes. After computng the moton matrces they combned to express the nter-camera rgdty. Closely related body of work are the class-based approaches [15, 10, 3, 2]. These assume that the ponts n 3D can be expressed as a lnear combnaton of a small morph model bass. The bass can ether be provded as learned pror[10, 15], or computed by the algorthm [3, 2]. Rather than expressng all the ponts as lnear combnatons of some model, n ths work we set some of the ponts as lnear combnaton of other ponts, thus expressng local rgdty of subsets of the features. Stll, the assumptons of the class-based approach and the presented work can be expressed n a smlar way: The matrx contanng the postons of the 3D ponts n dfferent tmes has a low rank. Therefore a sngle-camera factorzaton algorthm desgned for class based settng [3] can be adapted also for reconstructon n our proposed settng. The method proposed n ths paper uses two cameras and can be seen as a compromse between havng the pror 3D model gven to the algorthm [15, 10] and computng the model from the data usng factorzaton [3, 2]. Ths way we get two man advantages over the factorzaton methods. Frst, n factorzaton methods there s an ambguty expressed n an unknown nvertble matrx between the two factors. Second, the proposed method s robust to outlers as t ncludes a way to dscard ourlers by testng each feature separately. Usng factorzaton on data wth outlers degrades the qualty of the results. We elaborate more on ths pont n Secton A soluton for 3D reconstructon of k movng bodes from a sequence of affne vews was presented n [5] and later on n [9] usng factorzaton. Improved solutons were suggested n [17, 16]. These algorthms explot the fact that there are more measurements arsng from each one of the objects than the mnmal number requred to span ths moton (usually the number of bodes s 3

4 small). Ths redundancy s used to dentfy the motons. In ths work the rgdty relatons between ponts s weaker, allowng many dfferent motons of small bodes wth dfferent dmensons. For example, local rgdty constrants of lnes and planes can be exploted to express non-rgd bodes, e.g. trees and clothes. In the experments we compared our method to [5], and showed that even n a k body settng as well t s worthwhle usng a second camera. 2 Formal statement Assume two affne cameras (2 4 matrces) vew dfferent 3D ponts across tme 1 t T. Let P (t) j R 3,j = 1...n, be the 3D ponts tracked by the frst camera. x(t) = y (t) a 1 a 2 a 3 b 1 b 2 b 3 P (t) + Instead of lookng drectly on the pont measurements (x, y) t would be more convenent to look at the flow to some reference frame. (We wll use the frame where t=0 as our reference frame.)the flow s gven as: where dp (t) tme of frame t. u(t) = v (t) = P (t) P (0) a 1 a 2 a 3 b 1 b 2 b 3 ( P (t) P (0) ) = a 4 b 4 a 1 a 2 a 3 b 1 b 2 b 3 dp (t) s the moton of pont from the tme of the reference frame to the We wll use the coordnate system of the frst camera as our world (3D) coordnate system and so our projectons onto the two cameras are gven by: u(t) = dp (t) v (t), û(t) = ˆv (t) â1 â 2 â 3 ˆb 1 ˆb2 ˆb3 (t) dp ˆ Where we denote ponts related to the second camera usng a smlar notaton wth a ˆ hat. The basc constrant we use n ths work s the assumpton that every 3D ponts tracked n the second sequence (t) ˆP, 1 m can be expressed as a lnear combnaton of the ponts tracked 4

5 n the frst sequence: ˆP (t) = ( P (t) 1 P (t) 2... P (t) n ) Q 1. Where Q s the vector specfyng the coeffcents of the lnear combnaton, such that j Q (j) = Note that ths lnear combnaton apples also to the 3D dsplacements Where dz (t) û(t) = ˆv (t) = â1 â 2 â 3 ˆb 1 ˆb2 ˆb3 â1 â 2 â 3 ˆb 1 ˆb2 ˆb3 ( dp (t) 1 dp (t) 2... dp (t) n ) Q = u (t) 1 u (t) 2... u (t) n v (t) 1 v (t) 2... v (t) n dz (t) 1 dz (t) 2... dz (t) n ˆ dp (t) and so: Q (1) are the dsplacements along the Z axs of the ponts tracked n the frst sequence and the last equalty s by substtutng the projecton equaton of the frst camera. Reorganzng the terms we get: û(t) = Q (1)â 1 Q (1)â 2 Q (1)â 3 Q (2)â 1 Q (2)â 2... Q (n)â 3 ˆv (t) Q (1) ˆb 1 Q (1) ˆb 2 Q (1) ˆb 3 Q (2) ˆb 1 Q (2) ˆb 2... Q (n) ˆb 3 u (t) 1 v (t) 1 Z (t) 1. u (t) n v (t) n Z (t) n (2) Let A = (Q (1)â 1, Q (1)â 2, Q (1)â 3, Q (2)â 1, Q (2)â 2,..., Q (n)â 3 ), and B be the smlar expresson nvolvng ˆb 1, ˆb 2, ˆb 3. Let ˆM be the matrx whos rows are A 1,..., A m, B 1,..., B m. and let C be the matrx whos columns are ( u (t) 1, v (t) 1, Z (t) ) 1,..., u (t) n, v n (t), Z n (t) for 1 t T. Note that ˆm s pont-dependent, but does not vary wth tme, whereas the matrx C s tmedependent, but does not depend on ponts n the second sequence. 5

6 Let Û and ˆV be the matrces contanng the flow n the second sequences: Û = Stackng all equatons (2) we get: û (1) 1... û (1) m..... û (T ) 1... û (T ) m, ˆV = ˆv (1) 1... ˆv m (1)..... ˆv (T ) 1... ˆv (T ) m Û ˆV 2m T = ˆM 2m 3n C 3n T (3) A smlar expresson can be easly derved for the frst camera as well: U V 2n T = M 2n 3n C 3n T (4) 3 Synchronzaton and Acton Indexng Gven two sequences of a dynamc scene n the above settng, they are frst synchronzed. Havng synchronzaton, we later descrbe how to compute the structure of the scene across tme. The ablty to synchronze two sequences of the same acton can be used for other dynamc-scene applcatons such as acton ndexng. Two matchng actons should have a mnmal synchronzaton error. Examne the matrx we get by stackng the flow n both sequences sde by sde: E = (U, V, Û, ˆV ). Ths s a matrx of sze T (2n + 2m) and snce we consder long sequences we would expect ts rank to be (2n + 2m). However, ths holds only when there s no connecton between the sequences,.e when they are not synchronzed. Combnng equatons (3) and (4) we get: E = MˆM C 3n T (5) (2m+2m) 3n therefore the rank of E s bounded by 3n. These rank upper bounds are usually not tght, dependng on the rgdty of the scene. For example, when the scene contans k rgd bodes the rank s bounded by 3k. However n ths case 6

7 we expect the rank of the the matrx E to decrease n the synchronzed case at least as much as t drops n the unsynchronzed case. Outlers (ponts whch do not satsfy our assumptons even remotely) pose a dfferent knd of problem for ths rank bound - they mght ncrease the rank of the matrx E. However for outlers we would expect the rank of the matrx E to ncrease both for the synchronzed case and for the unsynchronzed case by the same amount. Therefore n both cases (partally rgd scenes and outlers) we would expect the rank n the synchronzed case to be lower then the rank n the unsynchronzed case). In practce, however, synchronzng by examnng the rank of a matrx s problematc snce due to nose our matrces wll almost always be of full rank. In order to deal wth ths we propose two solutons - the frst one s by examnng the effectve rank of the measurements, and the second s by consderng prncpal angles between the measurements of both sequences. The frst method, nstead of boundng the rank of E by 3n, uses a heurstc to defne ˆn, the effectve rank of (U, V ). We compute the sngular values s 1,..., s h1 of (U, V ), and set N = argmn j { 2j k=1 s k > J} for some threshold J (we used J = 0.99 h 1 k=1 s k). Ths s equvalent to fndng the mnmal rank of a matrx n a gven error bound under Frobenus norm. In order to synchronze we select the synchronzaton for whch the algebrac measure g(e) defned below s mnmzed. The mnmzaton s carred out over all possble synchronzatons where each possble synchronzaton provdes a dfferent matrx E. Let e 1,..., e h2 be all the sngular values of E. Then g s defned by: g(e) = h 2 e k k=3ˆn+1 g measures the amount of energy n the matrx beyond the expected rank bound. It s expected that f the rank bound on E s volated by nose, ths measure would be low, and otherwse, f t s broken by non-synchronzed data, ths measure should be hgh. The second method s based on the dea that for synchronzed sequences the lnear subspaces spanned by the columns of (U, V ) and by the columns of (U, V ) ntersect. ntersecton s 2n. The rank of the A stable way of measurng the amount of ntersecton between two lnear subspaces s by usng the prncple angles [7]. Let U A, U B represent two lnear subspaces of R n. The prncpal angles 7

8 0 θ 1... θ k (π/2) between the two subspaces are unquely defned as: cos(θ k ) = max u U A max v U B u v subject to: u u = v v = 1, u u = 0, v v = 0, = 1,..., k 1 A convenent way to compute the prncpal angles s va the QR factorzaton, as follows. Let A = Q A R A and B = Q B R B where Q s an orthonormal bass of the respectve subspace and R s a upper-dagonal k k matrx wth the Gram-Schmdt coeffcents representng the columns of the orgnal matrx n the new orthonormal bass. The sngular values σ 1,..., σ k of the matrx Q AQ B are the prncpal angles cos(θ ) = σ. As mentoned above the lnear subspaces spanned by the columns of [U, V ] and of [U, V ] ntersect and the rank of the ntersecton s 2n. Hence, the frst 2n prncple angles between these column spaces vansh. In practce we are effected by nose and would expect those frst prncple angles to be close to zero. In order to acheve synchronzaton we consder a functon of the frst few prncple angles (e.g. ther average). Ths functon s computed for every possble dsplacement and the synchronzaton s chosen as to mnmze ths functon. 3.1 Drect synchronzaton usng brghtness measurements The synchronzaton procedures descrbed above are based on havng a set of ponts tracked over tme n each one of the vdeo sequences. The accuracy of our results s therefore effected by the accuracy of the tracker whch we use. Most trackers however have dffcultes mantanng accurate postons for all tracked ponts over tme, especally n dynamc scenes. A method for synchronzng usng only brghtness measurements n the mages s gong to be presented next. Ths method s based on the work of Iran [8] where t was shown that the flow matrces U and V are closely related to matrces computed drectly out of the dervatves of the brghtness values of the mages I(t), 0 t T. These dervatves are just the mage dervatves I x and I y of the reference mage (t=0) and the dervatve over tme I j t = I(j) I(0). The relaton 8

9 - the generalzed Lukas & Kanade constrant - s expressed by the followng equaton: [U, V ] T 2n A, B B, C 2n 2n = [G, H] T 2n Where A,B and C are dagonal n n matrces wth the dagonal elements beng a = k(i x (k)) 2, b = k(i x (k)i y (k)) and c = k(i y (k)) 2 respectvely, where the summaton s at a small wndow around pxel. The (, j)th element of G and H are gven by g j = k(i x (k)i t(k) and h j = k(i y (k)i t(k) where the summaton s over a small neghborhood around pxe j. In practce we compute matrces G and H only for those [ pxels ] for whch there s enough gradent A, B nformaton n ther neghborhood, hence the matrx s nvertble and the column space of B, C U and V s the same as the column space of G and H respectvely. A smlar constrant connects the flow matrces n the second sequence U and V to measurement matrces G and H computed on ths sequence. We can therefore make the observatons below, whch are analogous to the observatons we made earler on the ranks of U, V, U, V. In the synchronzed case: The effectve rank N of the matrx [U, V ] s the same as the effectve rank of the matrx [G, H]. The effectve rank of the matrx [G, H, G, H ] s bounded by 3N. The column spaces of the matrx [G, H] and of the matrx [G.H ] ntersect n a lnear subspace of rank N. These observatons can lead to synchronzaton methods analogous to those presented of the flow matrces. A care should be taken as for the number of frames used. The above generalzed Lukas and Kanade constrant assumes nftsmal moton. Ths assumpton mght hold only a few frames a way from the reference frame. In practce we dvde both sequences to small contnuous fragments (small wndows n tme) of length up to 17 frames. We pck the reference frame as the mddle frame of each fragment, and compute the measurement matrces G and H for each such fragment separately. We then compare each fragment n the frst sequence to each fragment n the second sequence, and compute an algebrac error. 9

10 Let k, k be the number of fragments we extract from each sequence. The comparson of fragments between both sequences results n a matrx of sze k k. A synchronzaton wll appear as the lne n ths matrx whch has the mnmum average value. The offset of the lne determnes the shft of the synchronzaton and the slope of t determnes the rato of frame rates between the sequences. 4 Reconstructon Gven the synchronzaton of the two sequences, the next stage s 3D reconstructon. Let D = null( U ) be a matrx whose columns are all orthogonal to the rows of U 1. The rows of V U V are the rows of C whch correspond to moton along the X and Y axes. Hence the frst two out of every 3 rows n CD vansh (all whch s left s the nformaton regardng the Z coordnates of the dsplacements). By Multplyng both sdes of Eqn. (3) wth D we get: â 3 Q 1 Û ˆb K := D = ˆV ˆMCD 3 Q 1 Z (1) 1... Z (T ) 1 = D â 3 Q m Z (1) n... Z n (T ) ˆb 3 Q m V Observe that each odd row of K equals the next row of K multpled by the rato r = a(3)/b(3). Hence ths rato can be recovered by dvdng the rows. For added robustness we take the medan of all the measurements of r from all ponts. In Secton. 4.1 we elaborate on the robustness of the algorthm. Consder the vector l = ( 1 r ). l s a drecton orthogonal to the projecton of the Z axs to the second mage. Usng ths drecton we elmnate the unknown depths of the ponts. 1 We assume T > 2N, otherwse synchronzaton s not possble. If T < 2n, we take D to be the vector correspondng to the smallest sngular value. 10

11 We multply both sdes of Eqn. 1 wth l and get: ˆ u (t) l ˆ = ( c v (t) 1 c 2 ) u(t) 1 u (t) 2... u (t) n Q v (t) 1 v (t) (6) 2... v n (t) Where c 1 = l (â 1, ˆb 1 ),c 2 = l (â 2, ˆb 2 ), and the thrd row of the projecton matrx just vanshed (l (â 3, ˆb 3 ) = 0). Ths s a blnear system n the unknowns c 1, c 2 and Q. For each we have T equatons. We solve ths system by lnearzaton,.e by defnng a vector of unknowns u convertng the equatons nto a lnear system. = (c 1 Q, c 2 Q ) and From u recoverng Q up to scale s s done by factorng the elements of u as a 2 n rank 1 matrx 2 and the scale s adjusted such that Q = 1. Once the Q are recovered we can recover the ponts n the frst mage correspondng to the ponts tracked n the second mage: (t) ˆP = ( P (t) 1 P (t) 2... P n (t) ) Q = x(t) 1 x (t) 2... x (t) n Q y (t) 1 y (t) 2... y n (t) The problem of reconstructng the ponts tracked n the second mage smple stereo problem n each frame. Reconstructng P (t) from columns are Q, = 1...m. ˆP (t) ˆ P (t) s hence reduced to a s possble by takng the pseudo-nverse of the matrx whose 4.1 Outler Rejecton The bass of our algorthm and many smlar feature based-algorthms s the assumpton that the nput ponts ft some model and were tracked correctly along the sequence. Outler ponts, e.g. ponts whch were not tracked properly, may have n some cases a strong effect on the accuracy of the results. In ths secton we show that outler ponts tracked n the frst camera have neglgble 2 It s better to frst recover c 1, c 2 usng all of u, snce they are not pont-dependent. 11

12 effect on the results, and outler ponts tracked the second camera can be detected automatcally. Hence the algorthm s robust to outler tracks from both sequences. The frst step n the algorthm conssts of computng the matrx D,.e. fndng the subspace orthogonal to the 2D trajectores of all ponts tracked n the frst camera. In case of outler ponts, ths subspace may be reduced, but stll the resultng subspace s orthogonal to all nler trajectores. Therefore the computaton of r = â 3 / ˆb 3 descrbed n the prevous secton s not nfluenced by outlers n the frst sequence. As for the computaton of Q - 3D ponts n the second camera are a combnaton of some of the nlers 3D ponts tracked n the frst camera. The coeffcents of Q correspondng to the outler ponts vansh. Outlers tracks n the frst sequence therefore have lttle effect as long as there are not too many of them. We show next how to elmnate outlers tracks from the second sequence. For each pont n the second mage, several measurements of the rato r can be computed. A smple test, e.g. by the measurng varance of ths value for each pont and across the ponts can be used to reject outlers. In our experments usually 2n > T > 3ˆn,.e. the matrx ˆB had rank T due to nose, but the effectve rank 2N was smaller. For numercal reasons we chose D to be the vector correspondng to the smallest sngular value, and therefore every pont n the second mage had a sngle measurement of the rato r. Outler ponts were chosen by computng the medan of the r values of all ponts, and dscardng ponts wth r value far from the medan. 5 Experments We have tested our algorthm on scenes of movng people. The frst set of experments tested the camera synchronzaton applcaton. Two vdeo sequences were captured by two unsynchronzed cameras vewng the same non-rgd scene. In the frst experment we used tracked ponts, and examned the algebrac error g, as explaned n Secton 3, at dfferent tme shfts, and chose the shft wth the mnmal error. Results are presented n Fg. 1. In the second experment we have used the drect method descrbed above. Snce ths method assumes small mage moton, we appled t to small temporal wndows w(t) (typcally of 15 consecutve frames), and smoothed and decmated the mages. For each such par of wndows w(t 1 ), w(t 2 ) from the two sequences 12

13 resp., we measured the smallest prncple angles between the subspaces defned on these wndows (Lor, the explanaton here depends on how you explan before). A typcal result of ths process s shown n Fg. 2-a. The matrx contans for each par of tmes t 1, t 2 the prncple angles between the lnear spaces computed for w(t 1 ) on the 1st sequence and w(t 2 ) on the 2nd sequence. Snce both t 1, t 2 ncrement by the same shft, the optmal synchronzaton s vsualzed by a 45 o dark lne n the mage. The mean value along such lnes s shown n Fg. 2-b. Correspondng frames of the two synchronzed sequences are shown n Fg. 2-c,d,e,f. We have confrmed these results by manually synchronzng the nput sequences. The second experment tested the reprojecton stage. Usng the proposed algorthm we establshed correspondences between the two cameras, by transferrng tracked ponts from one sequence to the other sequence. The results are presented n Fg. 3. Unfortunately, more than 50% of the tracks were not very good, and our algorthm was not able to handle these tracks as we hoped. So whle n the synthetc experments the algorthm proved to be robust to outlers, n the presented experment we have chosen good tracks n both sequences manually. 5.1 Clusterng In order to show the benefts of usng our algorthm for the settngs of ndependently movng rgd bodes we mplemented the algorthm of Costera and Kanade [5] and appled t to each of the two sequences separately. Based on the results we clustered the ponts to dfferent rgd objects, thus comparng the results on real nosy data. The clusterng method for both algorthms was based on an affnty matrx of the ponts. In our algorthm an affnty matrx can be defned by the normalzed correlaton between the coeffcent vectors Q. In [5] an affnty matrx was defned n a smlar manner. As ponted out n [9], once the affnty matrx s defned, the choce of the clusterng crtera s arbtrary. We have used a software package [1] ncludng several clusterng algorthms, and chose the Ward method whch brought the best results for both algorthms. The results, presented n Fg. 4, show that the use of an addtonal camera n our algorthm mproves the results. 13

14 x a) b) c) d) Fgure 1. Sequences synchronzaton for cameras wth dfferent zoom factors (Fg. 1-a) and for cameras wth large vergence (Fg. 1-b). Correspondng frames from the sequences of the frst graph are shown n c) and d). The frst graph reflects the oscllatons n the nput moton (walkng people). The sequences of the second graph are the same as n Fg. 4 The graphs show the algebrac error g vs. frame offset. 14

15 a) b) c) d) Fgure 2. Sequences synchronzaton for cameras wth large vergence usng drect method. Panel (a) shows the dstance (mnmal prncple angle, see text) between temporal wndows at dfferent temporal shfts n the two sequences. By averagng dagonal drectons n ths matrx, one acheves a score for dfferent synchronzatons, as shown n panel (b). The cyclc nature of the moton can be vewed n the graph. Correspondng frames from the sequences of the frst graph are shown n (c) and (d). 15

16 Fgure 3. Feature reprojecton between two sequences of a non-rgd body. 16

17 (a) (b) (c) (d) (e) (f) Fgure 4. Feature clusterng for k-body scene, a comparson wth the Costera-Kanade algorthm. Fg. 4-(a),(b) show examples of nput mages taken by the two cameras at dfferent tmes. Note the large vergence between the cameras. Fg. 4-(c),(d) show the two clusters found by the method n ths paper. Fg. 4-(e),(f) show the two clusters found by the Costera-Kanade K-body factorzaton algorthm. 17

18 6 Summary We have shown how the correspondence between two vdeo sequences n a non-rgd scene can be solved by usng a smple assumpton relatng the ponts tracked n the two vdeo sequences. The proposed algorthm s smple, lnear and robust to outlers. It combnes the measurement from all frames n a sngle computaton, thus mnmzng the error n all frames smultaneously. In the future we hope to able to apply our methods presented here not only for synchronzaton, but also for acton recognton. Each acton wll be represented by a collecton of small fragments, each one less then half a second long. A dctonary contanng many such fragments and ther assocated actons wll be constructed. In order to recognze an acton we wll compare each such fragment to the fragments n the dctonary, and try to nference the whole acton. References [1] Avalable at kleweg/clusterng/clusterng.html. [2] M. Brand. Morphable 3d models from vdeo. In Computer Vson and Pattern Recognton, Kaua, Hawa, pages II: , [3] C. Bregler, A. Hertzmann, and H. Bermann. Recoverng non-rgd 3d shape from mage streams. In Computer Vson and Pattern Recognton, Hlton Head, SC, pages II: , [4] Y. Casp and M. Iran. Algnment of non-overlappng sequences. In Internatonal Conference on Computer Vson, Vancouver, BC, pages II: 76 83, [5] J. Costera and T. Kanade. A multbody factorzaton method for ndependently movng-objects. Int. J. of Computer Vson, 29(3): , Sept [6] F. Dornaka and R. Chung. Stereo correspondence from moton correspondence. In Computer Vson and Pattern Recognton, pages I:70 75, [7] G. Golub and C.V. Loan, Matrx Computatons, 3rd ed., Johns Hopkns Unversty Press, Baltmore, MD, [8] M. Iran, Mult-Frame Optcal Flow Estmaton Usng Subspace Constrants. In IEEE Internatonal Conference on Computer Vson (ICCV), Corfu, September [9] K. Kanatan. Moton segmentaton by subspace separaton and model selecton. In Internatonal Conference on Computer Vson, Vancouver, BC, pages II: ,

19 [10] M. Leventon and W. Freeman. Bayesan estmaton of 3d human moton. techncal report tr 98-06, mtsubsh electrc research labs, [11] F. Schaffaltzky and A. Zsserman. Vewpont nvarant texture matchng and wde baselne stereo. In Internatonal Conference on Computer Vson, Vancouver, BC, pages II: , [12] P. Sturm and B. Trggs. A factorzaton based algorthm for mult-mage projectve structure and moton. In European Conference on Computer Vson, pages II: , [13] C. Tomas and T. Kanade. Shape and moton from mage streams under orthography: A factorzaton method. Int. J. of Computer Vson, 9(2): , November [14] T. Tuytelaars and L. Van Gool. Wde baselne stereo matchng based on local, affnely nvarant regons. In Brtsh Machne Vson Conference, [15] T. Vetter and T. Poggo. Lnear object classes and mage synthess from a sngle example mage. In IEEE Transactons on Pattern Analyss and Machne Intellgence, 19(7): , July [16] R. Vdal and R. Hartley. Moton Segmentaton wth Mssng Data usng PowerFactorzaton and GPCA. In Computer Vson and Pattern Recognton, [17] L. Zelnk-Manor and M. Iran. Degeneraces, Dependences and ther Implcatons n Mult-body and Mult-Sequence Factorzatons In Computer Vson and Pattern Recognton, June 2003 [18] L. Wolf and A. Zomet. Sequence to sequence self calbraton In European Conference on Computer Vson, Copenhagen, Denmark, pages II: ,