Structure from Moton
Structure from Moton For now, statc scene and movng camera Equvalentl, rgdl movng scene and statc camera Lmtng case of stereo wth man cameras Lmtng case of multvew camera calbraton wth unknown target Gven n ponts and camera postons, have 2n equatons and 3n+6 unknowns
Approaches Obtanng pont correspondences Optcal flow Stereo methods: correlaton, feature matchng Solvng for ponts and camera moton onlnear mnmzaton (bundle adjustment) Varous appromatons
Orthographc Appromaton Smplest SFM case: camera appromated b orthographc projecton Perspectve Orthographc
Weak Perspectve An orthographc assumpton s sometmes well appromated b a telephoto lens Weak Perspectve
Consequences of Orthographc Projecton Translaton perpendcular to mage plane cannot be recovered Scene can be recovered up to scale (f weak perspectve)
Orthographc Structure from Moton Method due to Tomas & Kanade, 992 Assume n ponts n 3D space p.. p n Observed at ponts n tme at mage coordnates ( j, j ), =.., j=..n Feature trackng, optcal flow, etc. All ponts vsble n all frames
Orthographc Structure from Moton Wrte down matr of data = n n n n D Ponts Frames Frames
Orthographc Structure from Moton Step : fnd translaton Translaton perpendcular to vewng drecton cannot be obtaned Translaton parallel to vewng drecton equals moton of average poston of all ponts
Orthographc Structure from Moton After fndng translaton, subtract t out (.e., subtract average of each row) = n n n n ~ D
Orthographc Structure from Moton Step 2: tr to fnd rotaton Rotaton at each frame defnes local coordnate aes,, and Then î ĵ kˆ ~ ~ = ˆ p, ~ ˆ ~ = j p j j j j
Orthographc Structure from Moton So, can wrte where R s a rotaton matr and S s a shape matr D = RS ~ [ ] n p p S j j R ~ ~ ˆ ˆ ˆ ˆ T T T T = = = n n n n ~ D
Orthographc Structure from Moton Goal s to factor Before we do, observe that rank( ) should be 3 (n deal case wth no nose) Proof: Rank of R s 3 unless no rotaton Rank of S s 3 ff have noncoplanar ponts Product of 2 matrces of rank 3 has rank 3 Wth nose, rank( ) mght be > 3 D ~ D ~ D ~
Goal s to factor Appl SVD: D ~ SVD D ~ ~ D = UWV nto R and S But should have rank 3 all but 3 of the w should be 0 Etract the top 3 w, together wth the correspondng columns of U and V T
Factorng for Orthographc Structure from Moton After etractng columns, U 3 has dmensons 2 3 (just what we wanted for R) W 3 V 3 T has dmensons 3 n (just what we wanted for S) So, let R * =U 3, S * =W 3 V 3 T
Affne Structure from Moton The and j entres of R * are not, n general, unt length and perpendcular We have found moton (and therefore shape) up to an affne transformaton Ths s the best we could do f we ddn t assume orthographc camera
D ~ Ensurng Orthogonalt Snce can be factored as R * S *, t can also be factored as (R * Q)(Q - S * ), for an Q So, search for Q such that R = R * Q has the propertes we want
Ensurng Orthogonalt Want or Let T = QQ T Equatons for elements of T solve b least squares Ambgut add constrants ( ) ( ) ˆ ˆ T * * T = Q Q 0 ˆ ˆ ˆ ˆ ˆ ˆ * T * T * T * T * T * T = = = j QQ j QQ j QQ = = 0 0 ˆ, 0 0 ˆ * T * T j Q Q
Ensurng Orthogonalt Have found T = QQ T Fnd Q b takng square root of T Cholesk decomposton f T s postve defnte General algorthms (e.g. sqrtm n Matlab)
Orthogonal Structure from Moton Let s recap: Wrte down matr of observatons Fnd translaton from avg. poston Subtract translaton Factor matr usng SVD Wrte down equatons for orthogonalzaton Solve usng least squares, square root At end, get matr R = R * Q of camera postons and matr S = Q - S * of 3D ponts
Results Image sequence [Tomas & Kanade]
Results Tracked features [Tomas & Kanade]
Results Reconstructed shape Top vew Front vew [Tomas & Kanade]
Orthographc Perspectve Wth orthographc or weak perspectve can t recover all nformaton Wth full perspectve, can recover more nformaton (translaton along optcal as) Result: can recover geometr and full moton up to global scale factor
Perspectve SFM Methods Bundle adjustment (full nonlnear mnmzaton) Methods based on factorzaton Methods based on fundamental matrces Methods based on vanshng ponts
Moton Feld for Camera Moton Translaton: Moton feld lnes converge (possbl at )
Moton Feld for Camera Moton Rotaton: Moton feld lnes do not converge
Moton Feld for Camera Moton Combned rotaton and translaton: moton feld lnes have component that converges, and component that does not Algorthms can look for vanshng pont, then determne component of moton around ths pont Focus of epanson / contracton Instantaneous eppole
Fndng Instantaneous Eppole Observaton: moton feld due to translaton depends on depth of ponts Moton feld due to rotaton does not Idea: compute dfference between moton of a pont, moton of neghbors Dfferences pont towards nstantaneous eppole
SVD (Agan!) Want to ft drecton to all v (dfferences n optcal flow) wthn some neghborhood PCA on matr of v Equvalentl, take egenvector of A = Σ( v)( v) T correspondng to largest egenvalue Gves drecton of paralla l n that patch, together wth estmate of relablt
SFM Algorthm Compute optcal flow Fnd vanshng pont (least squares soluton) Fnd drecton of translaton from eppole Fnd perpendcular component of moton Fnd veloct, as of rotaton Fnd depths of ponts (up to global scale)