Introduction to Multiview Rank Conditions and their Applications: A Review.

Transcription

1 Introducton to Multvew Rank Condtons and ther Applcatons: A Revew Jana Košecká Y Ma Department of Computer Scence, George Mason Unversty Electrcal & Computer Engneerng Department, Unversty of Illnos at Urbana-Champagn e-mal:kosecka@csgmuedu, yma@uucedu ABSTRACT Understandng the representatons of 3D scenes as encoded n multple vews taken by a camera from dfferent vantage ponts s central to many tasks n mage and vdeo analyss These tasks range from recoverng the camera moton, 3D structure of the scene and detecton and characterzaton of multple motons n vdeo We wll demonstrate that the natural representatons of a 3D scene n D mages s n terms of the ncdence relatons among dfferent geometrc prmtves, whch can be concsely characterzed by rank condtons of multvew matrces The proposed rank condtons capture all estng ndependent multlnear constrants and enable truly global geometrc analyss of the multple vews comprsed of dfferent geometrc features In addton to the analyss, we present natural factorzaton based lnear algorthms for structure and moton recovery, mage transfer and matchng across multple vews applcable n both calbrated and uncalbrated settng We wll demonstrate the approach epermentally on a problem of multframe structure and moton recovery usng pont and lne features and ther ncdence relatons 1 INTRODUCTION Analyss, algnment and characterzaton of the content of multple mages of a scene captured by a camera from dfferent vantage ponts s central to many tasks n vdeo and mage analyss Most of the past research on vdeo codng, compresson and multmeda applcatons nvolvng vdeo orgnated n mage processng communty and focused predomnantly on D mage processng technques to encode the nformaton n the mage stream On the other hand large amount of work n computer vson communty has been devoted to the problems of recovery of 3D models of the envronments from multple vews The applcatons range from buldng 3D models from photographs (generally referred to as mage renderng technques) wth applcatons to archtectural preservaton, computer graphcs or specal effects n move ndustry, augmented realty systems, human computer nteracton or object leveodelng to retal purposes The work s supported by NSF grant IIS It s nevtable that t s the 3D structure of the envronment whch gves rse to the vdeo and photography content and hence should be eploted n the analyss It s therefore central to understand and study how s the 3D structure encoded n multple vews of the scene and what s the relatonshp between the projectons of the 3D world and camera dsplacements Consderng scenaros where the observed moton of the objects n the scene and/or camera s rgd, the relatonshps are to a large etent characterzed by varous geometrc constrants between observable geometrc prmtves and rgd body moton Characterzaton of the estng geometrc constrants has a long hstory both n computer vson and photogrammetry The basc formulatons of the ntrnsc geometrc constrants governng perspectve projectons of pont features n two vews orgnated n photogrammetry and were later revved the computer vson communty n early eghtes [1] Natural etensons of relatonshps between two vews s to consder multple vews and dfferent feature prmtves In the computer vson lterature, fundamental and structure ndependent relatonshps between mage features and camera dsplacements were characterzed by the so-called multlnear matchng constrants [, 3, 4, 5] These geometrc relatonshps were used etensvely for feature matchng, pont-lne transfer to a new vew and moton and structure recovery from three vews [6, 7] Ths lne of work culmnated recently n publcaton of two monographs on ths topc [8, 9] In ths paper we present new characterzaton of the estng multvew constrants n terms of rank condtons of approprate multple vew matrces ntroduced n [10, 11] We start frst by ntroducng the rank condtons among multple vews of pont and lne features separately We wll demonstrate n an ntutve way that the rank condtons of these multvew matrces captures the relatonshps among all prevously known multlnear constrants and generalzes prevously studed trlnear constrants nvolvng med pont and lne features to a multvew settng As we wll see the lnear formulaton of the problem wll gve rse to natural algorthms for geometrc feature matchng, feature transfer across

2 multple vews and moton and structure recovery In order to demonstrate the wde applcablty of the framework n the lmted space, we wll focus more on the geometrc ntuton behnd the formulaton and algorthmc aspects, whle omttng the detaled proofs of the statements These can be found n [10, 11] In the last secton we present the applcablty of the multple vew matr of med features for a consstent moton and structure recovery whch properly ncorporate all the ncdence condtons n a scene and outlne conceptual algorthms for mage matchng and feature transfer to a novel vew MULTIPLE VIEWS OF POINTS AND LINES Frst we wll ntroduce basc notaton and concepts used through out the paper An mage (t) = [(t),y(t), 1] T R 3 of a pont p E 3, wth coordnates X =[X, Y, Z, 1] T R 4 relatve to a fed world coordnate frame, taken by a movng camera satsfes the followng relatonshp λ(t)(t) =A(t)Pg(t)X (1) where λ(t) R + s the (unknown) depth of the pont p relatve to the camera frame, A(t) SL(3) s the camera calbraton matr (at tme t), P =[I,0] R 3 4 s the constant projecton matr and g(t) SE(3) s the coordnate transformaton from the world frame to the camera frame at tme t In the above equaton, all, X and g are n homogeneous representaton A straght lne L E 3,defnedbyL = {X X = X 0 + αv}, where v =[v 1,v,v 3, 0] T R 4 s a non-zero vector ndcatng the drecton of the lne, and α R An mage l(t) =[a(t),b(t),c(t)] T R 3 of L takenbythemovng camera then satsfes the followng equaton l(t) T (t) =l(t) T A(t)Pg(t)X =0 () for the mage (t) of any pont on the lne L 1 In a practce we usually have avalable mages of (t) or l(t) at some tme nstances: t 1,t,,t m, whch we denote λ = λ(t ), = (t ), l = l(t ), Π = A(t )Pg(t ) Observng set of ponts and/or lnes n multple vews gves rse to the followng system of equatons λ =Π X =[R,T ]X, (3) l T = l T Π X 0 = l T Π v =0 (4) for =1,,mThe suggestve notaton Π =[R,T ] here, does not necessarly correspond to the actual rotaton and translaton R couldbeanarbtrary3 3 matr Only n the case when the camera s perfectly calbrated does R correspond to the actual camera rotaton 1 So defned l s n fact the vector orthogonal to the plane spanned by the mages of ponts on the lne Strctly speakng, l should be called the comage of the lne and T to the translaton Observe that the unknowns, λ, X and v, whch encode the nformaton about locaton of the pont p or the lne L n R 3 are not ntrnscally avalable from the mages By algebracally elmnatng some of the unknowns from the above equatons, the remanng relatonshps would be between, l and Π only, e between the mages and the camera confguraton These relatonshps are referred to as ntrnsc and provde a startng pont for recoverng the remanng nformaton from mages The elmnaton step s rather straghtforward n the two vew case, where the unknown scales can be elmnated by multplyng both sdes of the equaton by the vector T = T λ = R λ T T T R 1 =0 (5) Note that ths yelds an mplct constrant, so-called eppolar constrant, on the camera dsplacement (R,T ) between two vews, whch can be consequently used for dsplacement recovery [1] Geometrc nterpretaton of the two vew constrant s n Fgure 1 1 z e 1 o e 1 z y o (R, T ) Fgure 1: Two projectons 1, R 3 of a 3-D pont p from two vantage ponts The relatve Eucldean transformaton between the two vantage ponts s gven by (R, T ) SE(3) The ntersecton of the lne (o 1,o ) wth each mage plane s the so-called eppole, thatse 1 and e respectvely However n the multvew case, there are many dfferent, but algebracally equvalent, ways of elmnatng the unknowns and hence characterzng these constrants We here present a unfed and concse way of characterzng the estng constrants n terms ncdence relatons among dfferent geometrc prmtves and the rank condtons of ther assocated multvew matrces 3 RANK CONDITIONS AND INCIDENCE RELATIONS Our prevous work [1] has shown that, multple mages of ponts, lnes or planes are unversally governed by certan rank condtons Such condtons not only concsely capture geometrc constrants among multple mages, but also are the key to reconstructon of the camera moton and scene structure Ths lne of work has allowed us to realze a very mportant prncple about multple vew geometry As usual, for a vector u R 3,weuseû to denote the skew symmetrc matr such that ûv = u v for all v R 3 y

3 To a large etent, multple vew geometry s about studyng how can the ncdence relatons (among ponts, lnes, and planes etc) n 3-D space be epressed and eploted computatonally n multple -D mage measurements As we wll demonstrate the rank condtons of the approprate multvew matrces turn out to be the correct tool for ths purpose Snce the ncdence relatonshps are nvarant to change of vewpont or camera calbraton, and they can be effectvely verfed n mages or be gven as modelng condtons n practce, such knowledge can be and should be eploted f a consstent reconstructon s sought In ths secton, we brefly revew a few classc rank condtons n multple vew geometry and ther correspondng geometrc ntuton We wll not provde any proof for these results snce they can be found n our prevous paper [1] Instead, we try to make the results compact here so that the reader can grasp the essence of ths new formulaton n the shortest tme Wthout loss of generalty, we may assume that the frst camera frame s chosen to be the reference frame That gves the projecton matrces Π, =1,,m the general form: Π 1 =[I,0], Π =[R,T ],, Π m =[R m,t m ], where R R 3 3,=,,ms the frst three columns of Π and T R 3, =,,ms the fourth column of Π 31 Multple mages of a pont For the multple mages 1,,, m of a pont p, as shown n Fgure, t s necessary and suffcent for the p The rank value drops to 0 f and only f all the camera centers o 1,o,,o m and the pont p le on the same straght lne (the so-called rectlnear moton) Notce that for M p to be rank-defcent, t s necessary for any par of vectors T, R 1 to be lnearly dependent Ths gves us the well-known blnear eppolar constrants T T R 1 =0between the th and 1 st vew Hence, the constrant rank(m p )=1consstently generalzes the eppolar constrant for vews to arbtrary m vews The constrants among more then two vews come from addtonal lnear dependences between rows of M p In order to characterze them we eplot the followng lnear algebrac fact: Gven non-zero vectors a 1,, a n, b 1,, b n R 3, the followng matr s rank defcent: a 1 b 1 R 3n (9) a n b n f and only f a b T j b a T j =0for all, j =1,,n Applyng the fact drectly to the matr M p we obtan we obtan the well known trlnear constrant (T T 1 RT j R 1 T T j ) j =0 (10) Hence, the rank condton on matr M p captures all trlnear relatonshps between the th, j th and 1 st vews Notce that the multple vew matr M p beng rankdefcent s equvalent to all ts mnorshavng zero determnant Snce the mnors of M p nvolve three mages only, we may safely conclude that there are n fact no addtonal ndependent relatonshps among four vews 3 Multple mages of a lne For multple mages l 1,,, of a lne L, as shown n Fgure 3, t s necessary and suffcent for the m y 1 o 1 y o y o m Fgure : Lnes etended from the multple mages 1,,, m ntersect at one pont p n 3-D L P m P 1 P l 1 o 1 l 1 o lm o m followng so-called multple vew matr M p R 1 T 3 R T 3 M p = R3(m 1) (7) m R m 1 m T m to satsfy the rank condton rank(m p )=1 (8) Fgure 3: Planes etended from the mages l 1,,, ntersect at one lne L n 3-D followng multple vew matr M l l T R l1 l T T l T 3 R 3 l1 l T 3 T 3 M l = R(m 1) 4 (11) l T mr m l1 l T mt m

4 to satsfy the rank condton rank(m l )=1 (1) The rank value drops to 0 f and only f all the planes P 1, P,,P m concde, or equvalently, all the camera centers and the lne are coplanar The rank condton drectly mples all trlnear constrants among m mages of the lne To see ths more eplctly, notce that for rank(m l )=1, t s necessary for any par of row vectors of M l to be lnearly dependent Ths gves us the wellknown trlnear constrants l T j T j l T R l1 l T T l T j R j l1 =0 (13) among the 1 st, th and j th mages Hence the constrant rank(m l ) = 1 s a generalzaton of the trlnear constrant (for 3 vews) to arbtrary m vews Note that when m =3t s equvalent to the trlnear constrant for lnes, ecept for some rare degenerate cases, where for eample l T T =0for some 33 Multple mages of ntersectng lnes For a set of lnes L 1,L,,L m whch ntersect at pont p and ther mages n multple vews, as shown n Fgure 4, t s necessary and suffcent for the followng L 1 L L m p P 1 P P m m l 1 1 o 1 l 1 o o m Fgure 4: Planes etended from the mage lnes l 1,,, ntersect at one pont p n 3-D multple vew matr M l l T R 1 l T T l T 3 M l = R l 3 1 l T 3 T 3 R(m 1) 4 (14) l T m R 1 l T m T m to satsfy the rank condton rank(m l )= (15) It s nterestng to notce that the rank of M l wll drop back to 1 f the famly of lnes L 1,L,,L m happen to be the same lne L n 3-D; and the rank wll further drop to 0 f the famly of planes P 1, P,,P m happen to collapse nto one Hence a change n the rank value ndeed corresponds to a qualtatve change n the types of 3-D ncdence relatons among the nvolved multple mages The nterplay between ponts and lnes also gves rse to some nterestng rank condtons on med versons of the multple vew matr For nstance, suppose that n the reference vew you observe a pont 1 andntheremanng vews the lnes,, ncdent to that pont, such that n each vew l T =0for =,mtheassocated rank condton for ths case can be derved from the rank condton on M l and the followng matr l T R 1 l T T l T 3 R 3 1 l T 3 T 3 M pl = R(m 1) (16) l T m R m 1 l T m T m then satsfes the rank condton rank(m pl )=1 (17) For the rank condton to be satsfed, t s agan necessary as n the pont case, that al mnors of M pl be zero Ths yelds the followng constrants among arbtrary th, j th vew wth the frst vew as a reference (l T R 1 )(l T j T j) (l T j R j 1 )(l T T )=0 R Ths gves the trlnear pont-lne-lne constrant Note that the ncdence relatonshp s relaed n ths case snce the lnes n the subsequent vews do not have to correspond to each other as long as they ntersect at the same pont We wll demonstrate the use of these types of constrants for structure and moton recovery n the followng secton In prncple, one can arbtrarly decde to choose a pont feature or a lne feature n each mage and the resultng multple vew matr always obeys certan rank condton A general law for ths s gven n [1] 34 Multple mages of a planar scene Another case commonly encountered n practcal stuatons s when the set of ponts s restrcted to le on a plane n R 3 The plane can be descrbed as π T X =0for a vector π =[π 1,π ] R 4 wth π 1 R 3,π R Then for multple mages of a pont p or a lne L on the plane, as shown n Fgure 5, t s necessary and suffcent for the followng multple vew matrces M p and M l R 1 T l T R l1 l T T 3R 3 1 3T 3 l T 3 R 3 l1 l T 3 T 3 M p =,M l = mr m 1 mt m l T mr m l1 l T mt m π 1 1 π π 1 l1 π to satsfy the rank condton rank(m p )=1, rank(m l )=1 (18) That s, wth the etra rows [ π 1 1 π ] or [π 1 l ] 1 π appended to regular M p and M l, respectvely, the rank

5 L π p P 1 P P m l 1 m 1 o 1 l 1 o o m Fgure 5: Planes etended from the mage lnes l 1,,, and the plane π ntersect at one lne L; Lnes etended from the mages 1,,, m and the plane π ntersect at one pont p condtons on M p (M l ) reman the same Agan, the rank value drops to 0 f and only f all the planes π, P 1,,P m collapse nto one The rank condtons on the augmented M p and M ply some etra equatons (by consderng mnors of the sub-matr consstng of the th group of three rows of M p or M l and ts last three rows) { ( R 1 π T π 1) 1 = 0, l T ( R 1 π T π 1) (19) l1 = 0, for =,,m These are nothng but the homography between the th and the 1 st mages of the plane π By now, we can summarze a few characterstcs about ths matr rank approach: 1 The multple vew matr rank captures geometrc (and algebrac) constrants among multple mages n a global fashon, and wth t, we no longer need to break an mage set or sequence nto par-wse or trple-wse ones; To demonstrate conceptually how ths works consder a sequence of 8 wdely separated vews of a desk scene asshownnfgure7 For some of the corner ponts p, we can specfy the ncdence relatonshp of certan lne segments For eample the front corner of the bo n the foreground has three lnes ncdent to t and the one on the back bo only two lnes We depct them n the mage n terms of lne segments Suppose that each pont j has k ncdent edge L 1j L kj,forj = 1,,n From m mages of the scene the multple vew matr M j for each pont p has the followng form j R j 1 j T l 1jT R j 1 l 1jT T l kjt R j 1 l kjt T M j = R (m 1)(k+3) j mr m j 1 j mt m l 1jT m R m j 1 l 1jT m T m l kjt m R m j 1 l kjt m T m (0) where j means the mage of the jth corner n the th vew and l kj means the mage of the k th edge assocated to the j th corner n the th vew Note that one can easly verfy that α j =[λ j 1, 1]T R s n the kernel of M j In addton to the multple mages j 1,j m of the j th corner p tself, the etra rows assocated to the lne features l kj, =,m; k = 1,,k also help to determne the depth scale λ j 1 We can already see one advantage of Dfferent rank values of a multple vew matr drectly correspond to qualtatvely dfferent types of 3-D ncdence relatons among multple mages, and hence, a drop of rank value always mples occurrence of degenerate confguraton 3 The unversalty of the rank condtons suggests the possblty of utlzng all ncdence relatons among alages of all features n a unfed fashon for 3-D reconstructon, and as we wll see n the net secton, they drectly mply a lnear factorzaton algorthm for multple vew reconstructon 4 MOTION AND STRUCTURE RECOVERY The unfed formulaton of the rank condton enable us to solve the problem of moton and structure recovery from multple vews usng both pont and lne features Incdence constrants among ponts and lnes can now be eplctly taken nto account when a global estmaton of moton and structure takes place Fgure 6: 1 st and 7 th frame of the test sequence the rank condton: It can smultaneously handle multple ncdence condtons assocated to the same feature; 3 In prncple, usng the coplanar rank condtons (18), one can further take nto account that the some of the vertces and edges on each sde are coplanar Snce such ncdence condtons between ponts and lnes occur frequently n practce, especally for man-made objects such as buldngs and houses, the use of multple vew matr 3 In fact, any algorthm etractng pont feature essentally reles on eplotng local ncdence condton on multple edge features The structure of the M matr smply reveals a smlar fact wthn a larger scale

6 z for med features s gong to mprove the qualty of overall reconstructon by eplctly takng nto account all the geometrc relatonshps among features of varous types and wth multple measurements In order to estmate α j for each pont we need to know the matr M j,ewe need to know the motons (R,T ) for =,,m From the geometrc meanng of α j =[λ j 1, 1]T we can ntalze α j s f we know only the moton (R,T ) between the frst two vews The two vew dsplacement can be estmated usng the standard 8 pont algorthm [1] Knowng α j s note that each row now becomes lnear n R,T For eample for =we have λ j 1 j R j 1 + j T =0 Snce all the equatons M j α j =0, j =1,,,n (1) become lnear n (R,T ), we can select the approprate ones to solve for the motons (agan) Defne the vectors R = [r 11,r 1,r 13,r 1,r,r 3,r 31,r 3,r 33 ] T R 9 and T = T R 3, =,,m It s then equvalent to solve the followng equatons for =,,m: λ T 1 λ 1 1 l11t 1 T 1 l 11T [ ] R λ 1 1 lk1t 1 T 1 l k1t [ ] P = R T =0 R λ n T n(3+k), () 1 n n 1 T n λ n 1 l 1nT n 1 T l 1nT λ n 1 l knt n 1 T l knt where A B s the Kronecker product of A and B In general, f we have rch enough set of features such that the rank of the matr P s at least 11, there s a unque soluton to ( R, T ) Let T R 3 and R R 3 3 be the (unque) soluton of () n matr form Such a soluton can be obtaned numercally as the egenvector of P assocated to the smallest sngular value Let R = U S V T be the SVD of R Then the soluton of () n R 3 SO(3) s gven by: T = sgn(det(u V T )) T 3 R 3, det(s ) (3) R = sgn(det(u V T )) U V T SO(3) (4) We then have the followng lnear algorthm: Algorthm 1 (Multple vew factorzaton) Gven m( 3) mages j 1,, j m of n( 8) ponts pj, j =1,,n (as the corners of a cube), and the mages l kj,k = 1,, 3 of the three edges ntersectng at p j, estmate the motons (R,T ), =,,mas follows: 1 Intalzaton: s =0 (a) Compute (R,T ) usng the 8 pont algorthm for the frst two vews [1] (b) Compute α j s = [λj 1 /λ1 1, 1]T where λ j 1 s the depth of the j th pont relatve to the frst camera frame Compute ( R, T ) as the egenvector assocated to the smallest sngular value of P, =,,m 3 Compute (R,T ) from (3) and (4) for =,,m 4 Compute the new α j s+1 = αj from (1) Normalze so that λ 1 1,s+1 =1 5 If α s α s+1 >ɛ, for a pre-specfed ɛ>0, then s = s +1and goto Else stop The camera moton s then (R,T ),=,,mand the structure of the ponts (wth respect to the frst camera frame) s gven by the converged depth scalar λ j 1,j = 1,,n Fgure 7: Recovered structure of the scene as seen from a novel vewpont Only the features vsble n the frst frame are vsualzed We have a few comments on the proposed algorthm: 1 The reason to set λ 1 1,s+1 =1s to f the unversal scale It s equvalent to puttng the frst pont at a relatve dstance of 1 to the frst camera center Although the algorthm utlzes only one type of ncdence relatonshps between ponts and lnes captured by the multple vew matr, t can be easly generalzed to nclude addtonal ncdence relatonshps or planar restrctons 3 Although the algorthm s gven for the calbrated case, the frst two steps the factorzaton algorthm work also for the uncalbrated case: n the ntalzaton (R,T ) can smply be the canoncal decomposton of the fundamentaatr; the remanng (R,T ) s computed n step wll dffer from the true ones by the same projectve transformaton 01 y

7 41 Correspondence In the followng secton we wll demonstrate how the rank condton can be used for feature matchng We outlne the test for the pont case only, however the same technque can be appled n an analogous way to other multvew matrces Notce that n the pont case M p R 3(m 1) beng rank-defcent s equvalent to the determnant of M T p M p R beng zero det(m T p M p)=0 (5) where Mp T M p s a functon of the projecton matr Π and mages 1,, m If Π s known and we would lke to test f gven m vectors 1,, m R 3 ndeed satsfy all the constrants that m mages of a sngle 3-D pre-mage should, we only need to test f the above determnant s zero A more numercally robust algorthm s outlned below Algorthm (Multple vew matchng test: pont) Suppose the projecton matr Π assocated to m camera frames are gven Then for gven vectors 1,, m R 3, 1 Compute the matr M p R 3(m 1) as n (7); Compute second egenvalue λ of M T p M p ; 3 If λ ɛ for some pre-fed threshold, the m mage vectors match Smlar matchng test can be developed for the lne case In general, n order to test correspondence usng the above algorthm, we need to know the projecton matr Π, whch represents the multvew camera confguraton Fortunately, n order to recover Π, we only need few correspondng ponts and other technques allow us to do so Once the ntaoton estmate s obtaned, the recovered Π can be used to establsh more correspondences Hence the algorthm can be vewed as a naturaultvew etenson of the commonly used RANSAC based matchng algorthm for two vew geometry 4 Image transfer When the two vews are wdely separated t s qute common that certan features become occluded n some vews, whle stll reman vsble n the others Alternatvely we would lke to have some means how to predct the locaton of a feature wthout eplctly computng the scene structure and back-projectng the 3D entty nto an mage Ths task s often referred as mage transfer and can be also naturally accomplshed by eplotng the rank condton Wthout loss of generalty, suppose we know m 1 mages,, m R 3, of a pont n space, and we want to determne ts 1 st mage 1, based on known transformatons R,T, =,,m Observe the structure of the pont multvew matr M p and let us defne the followng matr R T N p R m T = R3(m 1) 4 (6) m R m m T Note that due to the rank defcency of matr M p assocated wth the pont matr we have [ ] N p 1 =0 (7) λ Hence the vector [ T 1,λ] T s n the null space of N p, whch can be robustly computed usng sngular value decomposton The algorthm s summarzed below: Algorthm 3 (Image transfer: pont) Suppose the projecton matr Π assocated to m camera frames are gven Then for gven vectors,, m R 3, 1 Compute the matr N p R 3(m 1) 4 as n (6); Perform SVD such that N p = USV T and denote v 4 to be the 4 th column of V; 3 Let 1 be the vector formed by the last 3 elements of v 4 and then normalze 1 to have that last coordnate 1 In a parallel way we can develop the relatonshp for mage transfer of a lne to a new vew, based on the rank condton of the matr assocated wth the lne Defne matr l T R l T T N l = l T m R m l T 3 R 3 l T 3 T 3 l T m T m R(m 1) 4 (8) It s easy to see that n general we have rank(n l )= snce N l X =0fand only f X span(x o,v) where X o and v were defned n Secton Hence l T 1 ṽ =0 where ( ) s takng the vector formed by frst three elements So we can perform SVD on N l to obtan USV T Snce the last two columns of V, denoted v 3 and v 4, are lnearly ndependent and N l v 3 = N l v 4 =0 Thus l 1 = kṽ ṽ 4 for some k 0 Ths yelds followng practcal lne transfer algorthm Algorthm 4 (Image transfer: lne) Suppose the projecton matr Π assocated to m camera frames are gven Then for gven vectors,, R 3, 1 Compute the matr N l R(m 1) 4 as n (8); Perform SVD such that N l = USV T, such that N l = USV T,whereU R (m 1) (m 1),S R (m 1) 4 and V R 4 4 ;

8 3 Denote v 3 and v 4 to be the 3 rd and 4 th column of V respectvely; set ṽ 3 to be the vector formed by the frst three elements of v 3 and ṽ 4 be the vector formed by the frst 3 elements of v 4 ; 4 Let l 1 =ṽ 3 ṽ 4 and normalze t [] O Faugeras and B Mourran, On the geometry and algebra of the pont and lne correspondences between N mages, n Internatonal Conference on Computer Vson, 1995, pp [3] B Trggs, Matchng constrants and the jont mage, n Internatonal Conference on Computer Vson, June 1995 [4] A Heyden and K Åström, Algebrac propertes of multlnear constrants, Mathematcal Methods n Appled Scences, vo0, no 13, pp , 1997 Fgure 8: In the fgure (8 th frame) on the rght we overlay two vertcal lnes and two ponts whch were transfered from the 4 th frame (left) where they were clearly vsble 5 DISCUSSIONS AND CONCLUSIONS Ths paper revewed a unfed paradgm recently proposed by the authors whch syntheszes results and eperences n the study of multple vews of pont, lnes and planes It s shown that all geometrc relatonshps among multple mages are captured through a sngle rank condton on certan multple vew matr To a large etent, ths matr rank approach smplfes and unfes multple vew geometry In addton, we can now carry out meanngful global geometrc analyss for many mages wthout gong through a parwse, trple-wse, or quadruplewse relatonshps Compared to conventonaultple vew analyss based on trfocal tensors, the multple vew matr based approach clearly separates meanngful geometrc degeneraces from degeneraces whch may be artfcally ntroduced by the use of algebrac equatons descrbng the constrants In partcular, as shown n ths paper, any confguraton whch causes a further drop of rank n the multple vew matr eactly corresponds to certan geometrc degeneracy The proposed approach wll certanly have mpact on both theoretcal analyss and algorthm development The lnear algorthms gven n ths paper and others [10] only show a straght-forward way of usng the rank condton But our smulaton and epermental results have already shown much better performance then the etant projectve factorzaton algorthm [13], and n fact, the performance has n many cases emulated nonlnear algorthms based on bundle adjustment Recent work has also shown that t s possble to generalze ths matr rank approach to study trajectory trangulaton, curve features, and even dynamcal scenes [14] The full potental of ths approach s yet to be nvestgated REFERENCES [1] H C Longuet-Hggns, A computer algorthm for reconstructng a scene from two projectons, Nature, vo93, pp , 1981 [5] M E Spetsaks and Y Allomonos, A mult-frame approach to vsuaoton percepton, Intl Journal of Computer Vson, vol 16, no 3, pp 45 55, 1991 [6] R I Hartley, Lnes and ponts n three vews - a unfed approach, n Image Understandng Workshop, 1994, pp [7] S Avdan and A Shashua, Novel vew synthess n tensor space, n IEEE Conference on Computer Vson and Pattern Recognton, 1997, pp [8] R Harley and A Zsserman, Multple Vew Geometry n Computer Vson, Cambrdge Unversty Press, 000 [9] O Faugeras and Q-T Luong, Geometry of Multple Images, MIT Press, 001 [10] Y Ma, K Huang, R Vdal, J Kosecka, and S Sastry, Rank condtons of the multple vew matr, n CSL Tech Report, Unversty of Illnos Urbana Champan UILU-ENG 01-14, 001 [11] Y Ma, J Kosecka, and K Huang, Rank defcency condton of the multple vew matr for med pont and lne features, n Asan Conference on Computer Vson, 00 [1] Y Ma, R Vdal, K Huang, J Kosecka, and S Sastry, Rank condtons n multvew geometry and ts applcatons, (submtted to) Internatonal Journal of Computer Vson, 00 [13] P Sturm and B Trggs, A factorzaton based algorthm for mult-mage projectve structure and moton, n European Conference on Computer Vson, 1996, pp [14] K Huang, R Fossum, and Y Ma, Generalzed rank condtons n multple vew geometry wth applcatons to dynamcal scenes, n European Conference on Computer Vson, 00, pp 01 15