Model reconstruction and pose acquisition using extended Lowe s method

Transcription

1 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 1 Model reconstructon and pose acquston usng extended Lowe s method Mchael Mng-Yuen Chang and Kn-Hong Wong Abstract Fndng the pose and structure of an unknown object from an mage sequence has many applcatons n graphcs, vrtual realty and multmeda processng. In ths paper we address ths problem by usng a two-stage teratve method. Startng from an ntal guess of the structure, the frst stage estmates the pose of the object. The second stage uses the estmated pose nformaton to refne the structure. Ths process s repeated untl the dfference between the observed data and data re-projected from the estmated model s mnmzed. Ths method s a varaton of the classcal bundle adjustment method, but s faster n executon and s smpler to mplement. Synthetc and real data have been tested wth good results. T Index Terms 3D structure acquston, structure from moton, pose estmaton, Lowe s method, bundle adjustment. I. INTRODUCTION hs work nvestgates the problem of fndng both the pose and structure of an unknown object from an mage sequence. Ths lne of research s known as structure from moton (SFM) n the lterature. There are many applcatons related to ths research. For example, n 3D object reconstructon, n creatng a real-lfe scene useful n vrtual realty, and n mxng real scenes wth artfcal objects n augmented realty. Many SFM approaches are based on the method of factorzaton proposed by Tomas and Kanade [37]. The man dea s that the moton of the 2D features depends on the object s structure and the moton parameters (rotaton, translaton) nvolved. Factorzaton provdes a way to recover the structure and moton parameters nvolved n generatng the moton. The method s orgnally desgned for orthographc cameras but later versons are able to handle other projecton models such as weak perspectve, para-perspectve, affne and full perspectve [22][32]. Another common approach s based on eppolar geometry. A par of frames n the mage sequence s used to calculate the fundamental matrx, whch contans nformaton about the camera moton. If the camera ntrnsc parameters of the camera are known, the camera moton (or extrnsc parameters) can be obtaned up to a scale factor. M.Y.Y. Chang s wth the Informaton Engneerng Dept., The Chnese Unversty of Hong Kong, Shatn, Hong Kong. Emal: mchang@e.cuhk.edu.hk K.H. Wong s wth the Computer Scence and Engneerng Dept., The Chnese Unversty of Hong Kong, Shatn, Hong Kong. Emal: khwong@cse.cuhk.edu.hk Ths work was supported by a grant from the Research Grant Councl of Hong Kong Specal Admnstratve Regon. (Project Number. CUHK4389/99E) Frst submtted on 3 Jan

2 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 2 If the parameters are not known, we can stll obtan the camera moton and 3D structure but only up to a partcular projectve camera model. These technques and algorthms are explaned n [40]. If we have a sequence of more than two mages of an object, we can utlze the addtonal nformaton to mprove the result. The Kalman flter based approach s a popular choce [12] for combnng nformaton from an mage sequence. Some researchers work on a class of technques based on space carvng and slhouette ([14], [15]). The concept s to use the nformaton n the nput mages to remove those 3D parts that are nconsstent wth the projectons to obtan the fnal 3D model. However, snce each volume element (VOXEL) s requred to be processed ndependently, t s n general a very slow process and requres a lot of workng memory. Another novel approach s based on the Markov-Chan Monte Carlo (MCMC) method, whch s able to recover the model wth lttle pror knowledge [6]. The drawback s that the computatonal cost s consderable. Models produced by the algorthms descrbed above typcally requre further refnement. Bundle adjustment s the most accurate and common technque n use. It s a global optmzaton technque that ams to reduce the errors between the 2D feature ponts and the predcted feature ponts from the model ([41], [39] and [2]). A system that combnes the use of eppolar geometry, Kalman flterng and bundle adjustment has been reported by Pollefeys [28]. The model obtaned by makng use of eppolar geometry and Kalman flterng s treated as the ntal guess for a bundle adjustment process. The man drawback of bundle adjustment s ts slow speed. Ths problem s partcularly acute f the number of parameters nvolved s large. Ways of mprovng the speed of the process are suggested n [28]. In ths paper, we propose to solve the SFM problem by usng a two-stage bundle adjustment method. It s smlar to the classcal bundle adjustment method, but can run at a faster speed. In our method, each teratve step has two stages. The frst stage uses an approxmate model to estmate the pose of the object. The second stage uses the pose nformaton to refne the model structure. The two stages are executed repeatedly untl the dfference between the observed data and data re-projected from the estmated model s mnmzed. A number of pose estmaton algorthms have been proposed n the lterature [17], [18], [24], [25] and [26]. The pose estmaton algorthm we used s based on the work by Lowe [19]. Lowe s method s a model-based algorthm, whch can estmate the pose of the model provded that the structure of the model s known. Our method can be consdered as an extenson of Lowe s method. In applcatons that requre the trackng of the pose of an unknown object, e.g. the pose of a person s head [16], our algorthm could be used to recover the pose even f a model of the person s head s not avalable. The man contrbuton of ths paper s to propose an effcent and practcal SFM algorthm that can be used by the multmeda communty for model generaton. We also provde explct analytc formulaton n each step of the teratve algorthm. Our system can be used to construct models from mages usng smple web cameras. Both synthetc and real mages have been tested wth good results. In our experment, we constructed a turntable for capturng the rotatonal moton of the objects. Our system was able to reconstruct 3D models of the objects, whch can be vewed at dfferent angles nteractvely by a VRML 3D browser.

3 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 3 Organzaton of the paper s as follows: n secton II, we wll descrbe the theory used n our approach. In III, we wll descrbe the expermental results of ths work. In IV we wll dscuss the results, and V s the concluson. A. Problem settng II. THEORY A camera at the world center O W has a focal length of f. It takes an mage sequence of a model M that has N feature ponts represented by P = { P1, K, P, K, PN}, where P = [ X, Y, Z] s the 3D poston of the th pont. The poston of the model at tme t = 1 serves as the reference poston of the model. The model s moved to new postons by a set of rotaton R t and translaton T t transformatons at tme t = 1, K, Γ. The set of 3D ponts P after these operatons are projected to the mage plane of the camera by the functon g (). A set of correspondng 2D mage ponts q = { q, K, q } at tme t s formed. t 1, t N, t Model M at t=1 P = [ X, Y, Z ] v-axs mage Y-axs Z-axs q = [ u, v ],1,1,1 c (Image center) O w (World center) u-axs f=focal length X-axs Fgure 1 : Perspectve projecton of an object onto an mage Specfcally, the 2D projecton pont qt, can be expressed as q,t = (u,t,v,t ) = g(rp t + T t ), where T Tt = ([ T1, T2, T3] ) t s r11 r12 r13 the translaton vector, R t = r21 r22 r23 s the rotatonal matrx, and r 31 r32 r33 r11x + r12y + r13z + T1 u = f r X + r Y + r Z + T r X + r Y + r Z + T v = f r X + r Y + r Z + T (1) represent the poston of a perspectvely projected pont n the mage plane. The mage formaton process can alternatvely be expressed by q = g( θ, P), where θ represents the pose nformaton ( T, R ).

4 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 4 B. Pose estmaton by Lowe s method The pose estmaton problem can be summarzed as follows. Gven a known model M and a set of correspondng 2D mage ponts q, fnd the pose parameter θ of the model so that q = grp ( + T) for all. Lowe s method [19] provdes a soluton to ths problem. Let θ % be an ntal estmate of the pose and θ be the true pose, so thatθ = % θ + δθ. Expand q = grp ( + T) nto a seres, we have dg( % θ, P ) q = g( θ, P) = g( % θ, P) + δθ + L d % θ (2) Rearrangng the equaton and let e = q g( % θ, P) be the resdual error between the observaton and the predcton, we have, dg( % θ, P ) e = δθ d % θ δθ can now be found because e s measurable and the dervatve of g( % θ, P ) can be calculated. Gven N model ponts, equaton (3) forms a system of lnear equatons. δθ can be found usng least-squares methods. An mproved estmate of the pose s gven by % θ = % + 1 θ + δθ. Ths process of pose estmaton s repeated teratvely untl the resdual error s suffcently small. The followng descrbes an mplementaton of Lowe s method by Trucco and Verr [40]. Let the estmated pose % θ = [ T, T, T, φ, φ, φ ], where φ, φ, φ are the roll, ptch, and yaw (RPY) angles respectvely. For an mage pont q = ( u, v ), the change n ( u, v ) as a result of the change nθ s gven by (3) δ u δ v u u [ ], 3 = Tj + φ j j= 1 Tj φ j v v [ ]. 3 = Tj + φ j j= 1 T j φ j (4) We would lke to fnd Tj and φ j n (4) so as to produce the rght amount of shfts ( δu, δv ) that compensate the error e n (3). Snce e can be measured, f the dervatves n (4) are known, then Tj and j φ can be found. R R R To compute these terms, frst let P = ( X, Y, Z ) be a model pont n the reference poston. Let ( X, Y, Z ) be the poston of the model pont after rotaton, where X = r X + r Y + r Z R R = R = Y r X r Y r Z Z r X r Y r Z And fnally, let ' ' ' (,, ) X Y Z be the model pont after rotaton and translaton, represented by X = X + T ' ' ' R Y = Y + T R R 2 Z = Z + T 3 1

5 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 5 Dfferentate equaton (1) wth respect to the varous terms, we have u f u u X =, = 0, = f T Z T T Z v v f v Y = 0, =, = f T T Z T Z ,, u XY u X X + ZZ u Y φ φ φ R R R R = f, = f, = f 2 '2 1 Z 2 Z 3 Z, v YY + ZZ v X Y v X and = f, = f, = f 2 2 φ Z φ Z φ Z R R R R If the model has N feature ponts, then from (4) we have 2N equatons for 6 unknowns ( T1, T2, T3, φ1, φ2, φ3). A soluton for T and φ can be found usng standard least squares methods [31]. Improved estmates are gven by T% + T andφ% + φ. C. Overvew of the two-stage algorthm for fndng pose and structure from an mage sequence If the structure of a model s known, then one can fnd the pose of the model by Lowe s method. Alternatvely, f the pose of the model s known, then t s possble to recover the model s structure. Based on these deas, the two-stage algorthm proposed here frst uses a rough model to estmate the poses of a sequence of mages by Lowe s method. The nformaton obtaned s then used to refne the model. Ths process s repeated teratvely. D. Steps of our algorthm Steps of our structure from moton algorthm for an mage sequence of N features and Γ frames are descrbed below. 1) Camera calbraton and Feature extracton The camera s calbrated by the tools descrbed n [4]. Then we use the KLT tracker descrbed n [38] to extract the feature set { q, t} from a mage sequence, where = 1, 2,..., N, and t = 1, 2,.., Γ. 2) Model and system ntalzaton The frst stage of our method uses an approxmate model to estmate the pose of an mage sequence. Such a model can readly be obtaned f we could assume the projecton s orthographc. Our algorthm can deal wth perspectve mages, but usng orthographc projecton for the frst guess wll provde an ntal model for us to start the teraton. Based on ths assumpton, the depth of the object should be less (e.g. 1/3 or less) than the dstance between the object and the camera. The unknown model can then be approxmated by a planar object located at a dstance Z from the camera. If the value of Z nt cannot be known exactly, then we can stll recover the model structure, but only up to a scale factor. nt

6 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 6 The orthographc projecton assumpton s used only once at the ntalzaton stage so as to provde the ntal model. Perspectve projecton mappng s used throughout subsequent operatons. In our work, we use the frst mage n the sequence to construct ths ntal planar model. Features n the frst mage are back-projected from the mage plane to an object plane located at a dstance Z nt from the camera. Specfcally, the th mage pont n the frst frame q,1 = [ u,1, v,1] s mapped to a 3D pont [ X, Y, Z ] of the model by Znt Znt X = u, Y v, and Z Znt f = f = (5) The ntal model can also be obtaned n a varety of ways. If the object s geometry s known to be concave or convex, then we can use a concave or convex surface to serve as the ntal guess. We could also replace the planar model by a random model wth a random set of Z nt. If the ntal guess s too poor, then the teratve method wll produce an ncorrect model that has a large resdual error. In ths case, we can use a dfferent random model and repeat the process untl a good model s found. Ths type of RANSAC-lke approach could produce a good result, however, s very tme-consumng. From our experence, we found that a planar surface was good enough for most experments. 3) Frst pass -- Use an estmated model to fnd the pose sequence For t=1 to Γ Use { P1, K, P N } and the correspondng mage features { q1, t, q2, t,.., qn, t} of the t th mage frame to fnd % θ t by Lowe s method End Result: An estmated pose sequence % θ = {% θ1, K, % θ Γ } s found. Descrpton: Based on the planar model obtaned from the ntalzaton, we can estmate the pose % θ t of the t th mage by Lowe s method as descrbed earler. It s mportant to note that the pose for each of the Γ frames can be estmated ndependently. Hence the altogether 6 Γunknown pose parameters can be found n Γ steps. Each step nvolves the estmaton of only 6 unknowns. The same number of unknowns reduces the computatonal cost consderably. 4) Second pass Based on the estmated poses, re- estmate the model from a sequence of mages For =1 to N From the estmated poses {% θ1, K, % θ Γ } and the th mage features n all the mage frames { q,1, q,2,.., q, Γ }, End refne the model pont P by a least-squares method to mnmze resdual mage error. Result: An mproved structure{ P1, K, P N } s found.

7 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 7 Descrpton: Based on the pose sequence {% θ 1, K, % } obtaned from the frst pass, we can now mprove our θ Γ estmaton of the 3D model ponts { P1, K, P N } usng the smlar approach as n equaton (2). Lke step 3, each model pont can be estmated ndependently. The estmaton of the complete model can be dvded nto N ndependent steps. Each step can be computed effcently. As n (2), by keepng % θ constant, and varyng P, we have dg( % θ, P% ) q = g( % θ, P% ) = g( % θ, P% ) + δp% + L dp% t, t t Rearrangng and keepng only the frst order terms, we have dg( θ, P ) dg( θ, P ) dp dg( θ, P ) dp dg( θ, P) dp e = % % δp = % % % δx + % % % δy + % % % % δz, where, t dp% dp% dx dp% dy dp% dz e = q g( % θ, P % ) = [ δu, δv ] (6), t, t t, t, t s the dscrepancy between the poston of the observed mage pont and the predcted value. From (1), dfferentatng u and v wth respect to X, Y, Z, we have δu = a δ X + a δy + a δz, t δv = a δ X + a δy + a δz, t (7) where r11 r31x r12 r32 X r13 r33x a11 = f, 2 a12 = f, 2 a13 = f, 2 Z Z Z Z Z Z r21 ry 31 r22 ry 32 r23 ry 33 a21 = f, a 2 22 = f, a 2 23 = f. 2 Z Z Z Z Z Z For a sequence of Γ mages, we have Γ observatons of the same model pont. Ths gves us 2Γ equatons wth only three unknowns [ δx, δy, δ Z], δu,1 a11 a12 a 13 δ v,1 a21 a22 a23 M M M M δ X δu a a a t, δy δ v = t, a21 a22 a 23 δ Z M M M M δu, Γ a11 a12 a13 δ v, Γ a21 a22 a 23 (8) [ δ X, δy, δ Z ] can be solved from (8) by least-squares methods. An mproved estmate of the th model pont s gven by[ X, Y, Z ] + [ δx, δy, δz ]. If a partcular model pont s not observable n some of the mages, probably due to

8 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 8 occluson, we could dscard the correspondng a mn, parameters n (8). 5) Stoppng rule Error n the model and pose estmaton can be measured drectly by summng up all the re-projecton errors n (6), as error = 1 N N Γ 2 et, (unt pxel) (9) Γ = 1 t= 1 Ths fgure measures the average re-projecton error per pont per frame. The algorthm s termnated f the rate of reducton of the error becomes too small (0.01 pxels). Otherwse the algorthm loops back to step 3. 6) Msmatched Feature flterng step Features are tracked by the KLT tracker. Nevertheless, errors due to measurement or feature msmatches do arse. Errors due to measurement nose could be contaned by usng a standard weghted least squares method to replace the ordnary least squares method used here. The magntude of the re-projecton errors n equaton (6) can be used to form the weghtng matrx n the weghted least squares method. For errors due to feature msmatches, t wll be dffcult to fnd a model pont that can provde a good ft to the observed data. Consequently, some of the re-projecton errors wll be sgnfcantly larger than the rest of the measurement. In ths work, we compute the mean and standard devaton of the magntude of the re-projecton error ( e t, ) n (6). Ponts that le far away from the mean (e.g. fve tmes the standard devaton) are consdered as msmatched features. These ponts are fltered. Step 1 to 6 can be executed many tmes untl no bad feature s found. E. Comparson wth the classcal bundle adjustment method Both classcal bundle adjustment and our two-stage adjustment am at estmatng the pose ( θ ) and model ( P ) N Γ 2 parameters by mnmzng the error = [ qt, g( % θt, P% )]. The man dfference between the two methods s that the = 1 t= 1 classcal method estmates the pose and model smultaneously, whle the two-stage method estmates the pose and model separately. Compare wth classcal bundle adjustment, the man advantages of the two-stage method are speed, smplcty, and self-ntalzaton. From our smulaton results (secton III), we dd not fnd any sgnfcant dfference n accuracy between the two methods. For an mage sequence of N model ponts and Γ vews, let x be the state vector representng the 3N + 6Γunknowns, gxbe ( ) the perspectve transformaton functon that yelds the set of 2D mage ponts, and y be the vector representng the observed ponts. Our goal s to fnd an optmal x so that y = gx ( ). Startng from an ntal value x, and assumng that g s locally lnear, then a frst order approxmaton of g( x ) 0 s gven

9 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 9 by g( x ) = g( x0 ) + J, where J s the Jacoban and s the refnement needed to yeld a better estmate. An mproved estmate can be obtaned by applyng x = x + teratvely. The soluton of + 1 can be found by solvng the equaton y = gx ( 0 ) + J usng least-squares methods. Let e = y g( x0 ), so that e = J, we have Τ Τ J J = J e (10) Ths equaton s known as the normal equaton. The sze of J Τ J,, and e are (3 N + 6 Γ ) (3 N + 6 Γ ), (3N + 6 Γ ), and 2NΓ respectvely. If the sze of T J J s large, then (10) s computatonally expensve to solve. Faugeras et al [7] and Pollefeys [29] have descrbed two ways to reduce the computaton cost. Frst, snce J Τ J s a sparse matrx, solvng (10) usng sparse matrx operatons can reduce the cost consderably. Second, t s possble to break J Τ J nto two smaller matrces, each of sze 3N 3N and 6Γ 6Γ. Ths also helps to reduce the computatonal cost. However, f ether N or Γ s large, the computatonal cost s stll consderable. In comparson, the two-stage approach estmates the pose and model parameters separately. In the pose estmaton stage, the estmaton of the 6Γ unknowns s dvded nto Γ ndependent steps. Each step nvolves solvng a normal equaton lke (10) but wth a Jacoban of sze 6 6only. Lkewse, n the model estmaton stage, the estmaton of the 3N unknowns s dvded nto N ndependent steps. Each step nvolves solvng a normal equaton wth a Jacoban of sze 3 3. The computatonal cost of the two-stage approach grows only lnearly wth N and Γ, whch s comparatvely more effcent than classcal bundle adjustment for large N and Γ. Furthermore, our two-stage method only requres the support of the most rudmentary matrx operatons, and does not rely on any sparse matrx operatons. Our method s therefore smpler to mplement than the classcal bundle method. Self-ntalzaton s another advantage of the two-stage approach. Both of the classcal bundle and two-stage bundle adjustments depend on Newton s method. It s well known that the convergence of Newton s method depends crtcally on the ntal value used. For the two-stage method, ths ntal vector s readly avalable f the ntal model can be approxmated by a planar object. We can use ths ntal model to estmate the pose of the next frame, and then use the result to refne the model structure. Ths procedure s repeated untl all vews are covered, whch completes the ntalzaton process. The same process s then used to refne the model structure and the pose estmaton untl the error s suffcently small. The transton from ntalzaton to refnement s seamless. The mplementaton of the system s relatvely smple. For classcal bundle adjustment, t s dffcult to obtan an ntal vector that carres an approxmate value for both the model and poses nformaton. Typcally the vector can only be found by usng other methods [28]. Ths s why classcal bundle adjustment s usually used only as part of a structure-from-moton system, typcally at the last stage, where an approxmate model s already avalable and the man purpose of bundle adjustment s to provde an accurate fnal structure. III. EXPERIMENTAL RESULTS A. Smulaton results and comparson wth classcal bundle method In ths secton, we compare the accuracy and the convergence rate between the classcal bundle adjustment method

10 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 10 and the two-stage method. Ffty ndependent runs were carred out n our smulaton. In each run, a model of 300 random ponts was generated. All ponts led nsde a cube of 0.13 m 3. The center of the model was 0.33 m away from a camera of focal length 6mm. The model was rotated and translated at a rate of [0.2, -0.2, 0.2] degrees and [0.01, 0.01, 0.01] meters per frame. To smulate a real-lfe scenaro, addtonal perturbaton rangng from -0.5 to 0.5 degrees for rotaton and from 0 to 0.04 meters for translatons were added to each frame. To smulate the measurement errors, 2D Gaussan nose (mean=0, σ = 2 pxels) was added to the mage ponts. Each run used a sequence of 30 mage frames. All the sequences were generated by dfferent models but wth the same pose nformaton. The machne used was a 2.4 GHz PC runnng MATLAB 6.5. To provde a far bass for comparson, the same ntal model and pose nformaton were fed to both bundle adjustment methods. Because the classcal bundle method was ll-suted to provde an ntal model, the two-stage method was used to buld the ntal model and pose nformaton. Fg. 2 and 3 show the trackng result. It can be seen that both algorthms were able to track the pose of the model equally well. Fg. 4 compares the convergent rate of the resdual errors (an average of 50 runs) between the methods. The resdual error s defned n (9). In theory, f the model and pose nformaton can be recovered perfectly, then the resdual error should be equal to the standard devaton of the Gaussan nose added,.e., 2 pxels. The result showed that both algorthms dd manage to converge to ths theoretcal lmt n all 50 runs. In fact, the resdual error was even slghtly less than the theoretcal mnmum, whch we beleve was due to the over-fttng of the model and pose. The result clearly showed that the two-stage method had a sgnfcantly faster rate of convergence.

11 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 11 Fgure 2: The pose trackng result of a test. The upper 3 dagrams show the tracked and real yaw, ptch roll angles n degrees.

12 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 12 Fgure 3: The pose trackng result of a test. The above 3 dagrams show the tracked and real translatons of X, Y, Z n pxels (the pxel wdth s 5.42µm n both mage u, v dmenson, the focal length s 6mm).

13 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 13 Fgure 4. Compare the 2D resdual errors aganst CPU tmes of our method and the classcal bundle adjustment method. The lnes represent the average of 50 runs. B. Experment on the msmatched feature flter In ths experment, we tested the msmatch feature flter descrbed n step 6 of our algorthm. To smulate the msmatch nose, nose equal to 14 pxels was added randomly to 5% of the data n secton III-A. We repeated the smulaton of 50 runs and the average result was plotted n Fg. 5. The result showed that the nose added had an mpact on the resdual error. After applyng the flter, the resdual error converged agan to the expected 2 pxel.

14 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 14 Fgure 5. Effects of the msmatch flter. The lnes represent the average of 50 runs. C. Real scene experments (more results are found at ) A number of real objects were tested. Features n the mage sequences were extracted by the KLT [38] tracker. Statonary mage features obtaned from objects on turntables were classfed as the background or nose and were fltered. The feature sets were passed to our algorthm for model reconstructon. For each model bult, we used the frst mage of the sequence to provde the texture for the VRML fle. The results were vewed by a VRML browser. 1) A flask on a turntable

15 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 15 The frst mage of the nput sequence. The last (10 th ) mage of the sequence Fgure 6: Frst column: frst and last mage of the nput sequence. Second and thrd columns: two reconstructed vews and ther wre-frames of the object. (The ntal model for the algorthm s a plane; the moton has lttle translaton and large rotaton.) 2) A box on a turntable The frst mage of the nput sequence. The last (10 th ) mage of the sequence Fgure 7: Frst column: frst and last mage of the nput sequence. Second and thrd columns: two reconstructed vews and ther wre-frames of the object. (The ntal model for the algorthm s a plane; the moton has lttle translaton and large rotaton.)

16 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 16 3) A laboratory scene captured by a camera movng horzontally wth small amount of rotaton The frst mage of the nput sequence. The last (15 th ) mage of the sequence Fgure 8 : Frst column: frst and last mage of the nput sequence taken by a camera translatng horzontally. Second and thrd columns: two reconstructed vews and ther wre-frames of the object. (The ntal model for the algorthm s a plane; the moton has lttle rotaton and large translaton).

17 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 17 4) Improvement made by applyng the msmatch feature flter (step 6 of our algorthm) Fgure 9 : The frst column: Texture and wre-frame of the result after the msmatched feature flter (step 6 of the algorthm) was appled. The second column: Texture and wre-frame of the result when the flter was not appled. It was found that the msmatched feature flter helped to remove some spkes n the reconstructed model.

18 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 18 IV. DISCUSSION A. Effect of ntal guess A good ntal guess s crtcal to the convergence of any Newton-based methods. Pror knowledge of the structure could provde a better ntal guess for the optmzaton. For the knd of objects that we tested, we found that good results could be obtaned by smply usng a planar object as the ntal model. A reasonable guess of the dstance of the object from the camera was also requred, though t was not a crtcal factor. B. Error ambguty It s known that the translaton and rotaton errors have certan correlaton [35]. For example, translaton error n the horzontal (X-axs) drecton can be mxed up wth rotaton around the Y-axs (ptch), snce both can generate a smlar moton of the features. Future work can concentrate on how to reduce ths type of error. One possble soluton s to make use of some pror knowledge of the moton. For example, n the turntable case, we can assume the moton s manly rotatonal; or n a robot navgaton case, the moton s manly translatonal. C. Occluson and omn-drecton surface model reconstructon Our mplementaton requred all features to appear n all frames; hence t was not able to handle occluded ponts or a very long sequence (200 pctures). At present, the system can only fnd a partal model at a tme. Nevertheless, several partal models can be combned n order to produce an omn-drectonal (360 degrees) vew of the object. For example, a camera can capture a 360 degrees rotaton of an object n a long sequence of 200 frames. The frst 10 frames are used to produce the frst partal model. Frame 6 to 15 can be used to produce the second partal model. Common features n the overlappng frames are responsble to fnd the relatve pose between the two partal models so that they can be combned. Repeat ths process untl all the partal models are merged nto an omn-drecton surface model. D. Samplng rate of the nput pctures Snce the system was based on the KLT feature tracker, t was only able to track features that moved n a relatvely small neghborng regon n successve frames. For a casual user that captures the scene by a stll-pcture dgtal camera wth a small samplng rate, unntentonal pannng caused by the user s hand movement may produce a large translatonal moton between successve pctures. As a result, the tracker may fal to track the features. We propose to solve ths problem by algnng the pctures usng a promnent feature pont wthn the scene n order to reduce the effect of the spurous translaton. V. CONCLUSION In ths paper, we developed a method of recoverng the structure and pose sequence of an object from a sequence of mages. The proposed method s a varaton of the classcal bundle adjustment method. Wthn each teraton, our method estmates the structure and poses separately. Ths separaton helps to reduce the sze of the Jacoban n the computaton consderably, makng the method partcularly effcent for scenes that have larger number of feature ponts and longer lengths. In our experment, we found that both the classcal bundle adjustment and our two-stage

19 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 19 methods converged vrtually to the same mnmum. Ths showed that the two-stage method was as accurate as the classcal bundle adjustment method. The separated estmaton of the structure and pose also helps to smplfy the ntalzaton process. To start the two-stage algorthm, we only need to provde an ntal model. The pose sequence can be estmated by usng the model provded. If the model s located not too close to the camera, then we found that the ntal model can be approxmated by a planar object. Pror knowledge of the object may also help to provde a better ntal model. A smple outler flterng scheme s proposed to reduce trackng errors n features extracton. Fnally the model s translated nto the VRML format for vewng and nteractve manpulaton. We beleve our algorthm s sutable for home users to develop models for Internet applcatons such as puttng 3D objects on web pages and games. To demonstrate the feasblty, we have developed a small turntable system for scannng the 3D objects. The results were found to be satsfactory. In augmented realty applcatons, one can use the method to estmate the pose sequence of a scene, so that synthetc objects can be nserted nto the vdeo stream n a realstc way. Another applcaton s head model generaton and head trackng, whch s an actve research topc n computer vson. We shall focus our future research n a number of drectons. The frst possble drecton s to nvestgate how to mprove the feature trackng module so that correct features can be obtaned more relably. Probablstc or statcally approaches, such as Condensaton (Condtonal Densty Propagaton [11], [42]), may help to mprove trackng performances. The second s to explore dfferent constrants for pose trackng. For example, the trajectores of features of an object on a rotatng turntable are known to be confned to a seres of parallel concentrc crcles. The thrd drecton s to combne local partal models so as to construct a full omn-vew surface model. The forth drecton s to nvestgate how to reduce errors caused by rotaton and translaton ambguty. We expect the hgh effcency of our two-stage method wll provde a frm bass to support the varous mprovement schemes mentoned. VI. ACKNOWLEDGMENT The work descrbed n ths work was supported by a grant from the Research Grant Councl of Hong Kong Specal Admnstratve Regon. (Project Number. CUHK4389/99E) VII. REFERENCES [1] H. Araujo, R. Carceron and C. Brown, A Fully Projectve Formulaton for Lowe s Trackng Algorthm", Computer Vson and Image Understandng, Vol. 70, No. 2, May, pp , [2] P. Beardsley and A. Zsserman and D. Murray, "Sequental updatng of projectve and affne structure from moton", Internatonal Journal of Computer Vson, (23), No.3, Jun-July 1997, P [3] D. Brch, KLT: An Implementaton of the Kanade-Lucas-Tomas Feature Tracker, ( ).

20 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 20 [4] Jean-Yves Bouguet, Camera Calbraton Toolbox for Matlab ( [5] A. Chuso, P. Favaro, J. Haln and S. Soatto, Structure from moton causally ntegrated over tme, Pattern Analyss and Machne Intellgence, IEEE Transactons on, Volume: 24 Issue: 4, Aprl 2002 Page(s): [6] F. Dellaert, S.M. Setz, E.C. Thorpe and S. Thrun. Structure from moton wthout correspondence. In Proc. CVPR, pages , June [7] O. Faugeras, Q.T. Luong and T. Papadopoulo, The Geometry of Multple Images: The Laws that Govern the Formaton of Multple Images of a Scene and some of Ther Applcatons, MIT Press. [8] R.M. Haralck, H. Joo, C. Lee, X. Zhuang, V.G. Vadya, and M.B. Km. Pose estmaton from correspondng pont data. IEEE Transactons on Systems, Man, and Cybernetcs, 6(19): , November/December [9] R. Hartley and A Zsserman, Multple Vew Geometry n Computer Vson, Cambrdge Unversty Press. [10] T.S. Huang and A.N. Netraval, "Moton and Structure from Feature Correspondences: A Revew", Proceedngs of the IEEE, Vol. 82, No.2, Feb [11] M. Isard and A. Blake, CONDENSATION -- condtonal densty propagaton for vsual trackng, Proceedngs of IEEE Internatonal Workshop on Recognton, Analyss and Trackng of Faces and Gestures n Real-Tme Systems, 26-27, September 1999, Corfu, Greece. [12] T. Jebara, A, Azarbayejan and Pentland, A. 3D structure from 2D moton, IEEE Sgnal Processng Magazne, Volume: 16 Issue: 3, May 1999 Page(s): [13] F. Kahl, A. Heyden and L. Long, Mnmal projectve reconstructon ncludng mssng data Pattern Analyss and Machne Intellgence, IEEE Transactons on, Volume: 23 Issue: 4, Aprl 2001 Page(s):

21 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 21 [14] K.N. Kutulakos, A Theory of Shape by Space Carvng, Internatonal Journal of Computer Vson 38(3): ; Jul [15] A. Laurentn, The vsual hull concept for slhouette-based mage understandng Laurentn, A., Pattern Analyss and Machne Intellgence, IEEE Transactons on, Volume: 16 Issue: 2 Feb 1994 Page(s): [16] K.S. Lee, K.H. Wong, S.H. Or and Y.F. Fung, "3D Face Modellng From Perspectve-Vews and Contour-Based Generc-Model", Real-Tme Imagng Journal Aprl, [17] M.L. Lu and K.H. Wong, "A Novel Algorthm for Recoverng the 3D Motons of Multple Movng Rgd Objects",14th Internatonal Conference on Pattern recognton (ICPR'98), Brsbane, Australa, Aug [18] M.L. Lu and K.H. Wong, "Pose Estmaton Usng Four Correspondng Ponts", Pattern Recognton Letters, Volume 20, Number 1 January 1999, pp [19] D.G. Lowe, Fttng Parameterzed Three-Dmensonal Models to Images, IEEE Pattern Analyss and Machne Intellgence, Volume: 13 Issue: 5, May 1991 Page(s): [20] C.P. Lu, G.D. Hager, E. Mjolsness, Fast and globally convergent pose estmaton from vdeo mages Pattern Analyss and Machne Intellgence, IEEE Transactons on, Volume: 22 Issue: 6, June 2000 Page(s): [21] T. Morta and T. Kanade. A sequental factorzaton method for recoverng shape and moton from mage streams. IEEE Transactons on Pattern Analyss and Machne Intellgence, 19(8): , [22] T. Morta, T. Kanade, A sequental factorzaton method for recoverng shape and moton from mage streams Pattern Analyss and Machne Intellgence, IEEE Transactons on, Volume: 19 Issue: 8, Aug Page(s):

22 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 22 [23] J. Olenss, Y. Genc, Fast and accurate algorthms for projectve mult-mage structure from moton, Pattern Analyss and Machne Intellgence, IEEE Transactons on, Volume: 23 Issue: 6, June 2001 Page(s): [24] S.H. Or, K.H.Wong and T.T.Wong. "On Usng Longuet Hggns Equaton n Pose Estmaton Framework by Lowe." Proceedngs of the Internatonal Conference on Imagng Scence, Systems, and Technology, pp , on 6 June 1999, Las Vegas USA. [25] S.H. Or, K.H.Wong, T.K. Lao and T.T.Wong. "An mage based pose approach to pose estmaton", Invted conference paper, 1999 Internatonal Symposum on Sgnal Processng and Intellgent Symposum (ISSPIS'99), Nov , 1999 Guangzhou, Chna.(R). [26] S.H. Or, W.S. Luk, K.H. Wong, I.Kng, "An effcent teratve pose estmaton algorthm", Image and Vson Computng Journal, Volume 16, Issue 5, pp , May [27] C.J. Poelman, Kanade, T, A paraperspectve factorzaton method for shape and moton recovery, n IEEE Trans. Pattern Anal. Machne Intell., vol. PAMI-19, No. 3, pp , March [28] M. Pollefeys, Self-calbraton and metrc 3D reconstructon from uncalbrated mage sequences, PhD. thess, K.U.Leuven, [29] M. Pollefeys, Tutoral on 3D Modelng from Images, June [30] L. Quan and Z. Lan, Lnear N-Pont Camera Pose Determnaton, n IEEE Trans. Pattern Anal. Machne Intell., vol. PAMI-21, No. 8, pp , August [31] G. Strang, Introducton to Lnear Algebra, 2nd ed., Wellesley-Cambrdge Press, [32] P. Sturm B. Trggs, A Factorzaton Based Algorthm for Mult-Image Projectve Structure and Moton, EECV 96. [33] Z. Sun, M. Tekalp, A. Navab, and N. Ramesh, V., Interactve optmzaton of 3D shape and 2D correspondence usng multple geometrc constrants va POCS, Pattern Analyss and Machne Intellgence, IEEE Transactons on,volume: 24 Issue: 4, Aprl 2002 Page(s):

23 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 23 [34] R. Szelsk and S. B. Kang, "Recoverng 3D, shape and moton from mage streams usng non-lnear least squares", Journal of Vsual Communcaton and Image Representaton", vol. 5, No. 1, PP ", [35] R. Szelsk and S. B. Kang, "Shape Ambgutes n Structure from Moton, IEEE Transactons on Pattern Analyss and Machne Intellgence, vol. 19, no.5, May [36] R. Szelsk and SB Kang, Recoverng 3D, shape and moton from mage streams usng non-lnear least squares", Journal of Vsual Communcaton and Image Representaton", vol. 5, No. 1, PP ", [37] C. Tomas and T. Kanade, Shape and Moton from Image Streams under Orthography: A Factorzaton Method, Internatonal Journal of Computer Vson, Vol. 9, No. 2, 1992, pp [38] C. Tomas and T. Kanade. Detecton and Trackng of Pont Features, Carnege Mellon Unversty Techncal Report, CMU-CS , Aprl [39] B. Trggs, P. McLauchlan, R. Hartley and A. Ftzgbbonm, Bundle Adjustment - A Modern Synthess, In Proceedngs of the Internatonal Workshop on Vsual Algorthm: Thoery and Practce, P.P , Corfu, Greece, Sept [40] E. Trucco and A. Verr, Introductory technues for 3-D compouter vson, Prentce Hall, [41] J. Weng, N. Ahuja and T.S. Huang, Optmal moton and structure estmaton Pattern Analyss and machne Intellgence, IEEE Transactons on,volume: 15 Issue: 9, Sept Page(s): [42] K.H. Wong, S.H. Or, and M.M.Y Chang, Pose trackng for vrtual walk-through envronment constructon Conference Proceedng on the Internatonal Conference on Inverse Problems & Numercs, Cty Unversty of Hong Kong, January 9-12, [43] Z. Zhang and G. Xu, Eppolar Geometry n Stereo, Moton and Object Recognton: A Unfed Approach, Academc Publshers, 1996.

24 Model reconstructon and pose acquston usng extended Lowe s method, by M.M.Y. Chang and K.H. Wong 24 [44] Z. Zhang and Y. Shan. Incremental Moton Estmaton through Local Bundle Adjustment. Techncal Report MSR-TR-01-54, Mcrosoft Research, May ( Prof. Mchael Mng-yuen Chang receved the B.Sc. n electrcal engneerng from Imperal College, London Unversty and the PhD degree n electrcal engneerng from Unversty of Cambrdge n He then joned the Department of Informaton Engneerng, The Chnese Unversty of Hong Kong and s now an Assocate Professor. Hs current research nterest s n character recognton, scentfc vsualzaton and ntellgent nstrumental control. Hs contact address s: The Informaton Engneerng Dept., The Chnese Unversty of Hong Kong, Shatn, Hong Kong. Emal: mchang@e.cuhk.edu.hk Prof. Wong Kn-hong receved the B.Sc. n Electroncs and Computer Engneerng from the Unversty of Brmngham n 1982, and a Ph.D. from the Engneerng Dept. of the Unversty of Cambrdge, U.K. n He was a Croucher research fellow at the Unversty of Cambrdge from 1985 to Prof. Wong joned the Computer Scence Dept. of CUHK n 1988 and s now an Assocate Professor. Hs research nterests are 3D computer vson, vrtual realty mage processng, pattern recognton, mcrocomputer applcatons and computer musc. Hs contact address s: The Computer Scence and Engneerng Dept., The Chnese Unversty of Hong Kong, Shatn, Hong Kong. Emal: khwong@cse.cuhk.edu.hk