Structure from motion

Transcription

1 Structure from motio Digital Visual Effects Yug-Yu Chuag with slides by Richard Szeliski, Steve Seitz, Zhegyou Zhag ad Marc Pollefyes

2 Outlie Epipolar geometry ad fudametal matrix Structure from motio Factorizatio method Budle adjustmet Applicatios

3 Epipolar geometry & fudametal matrix

4 The epipolar geometry epipolar geometry demo C,C,x,x ad X are coplaar

5 The epipolar geometry What if oly C,C,x are kow?

6 The epipolar geometry All poits o project o l ad l

7 The epipolar geometry Family of plaes ad lies l ad l itersect at e ad e

8 The epipolar geometry epipolar pole = itersectio of baselie with image plae epipolar geometry demo = projectio of projectio ceter i the other image epipolar plae = plae cotaiig baselie epipolar lie = itersectio of epipolar plae with image

9 The fudametal matrix F R p p C C T=C -C Two referece frames are related via the extrisic parameters p Rp' T

10 The fudametal matrix F p Ep' 0 essetial matrix T Rp' p x y x z y z T T T T T T T Multiply both sides by T p T T Rp' T p p T p T T Rp' T p 0 T

11 The fudametal matrix F p Ep' 0 Let M ad M be the itrisic matrices, the p M x p' M' x ' ( M x x) M E( M' EM ' Fx' 0 x x') x' 0 0 fudametal matrix

12 The fudametal matrix F The fudametal matrix is the algebraic represetatio of epipolar geometry The fudametal matrix satisfies the coditio that for ay pair of correspodig poits x x i the two images x T Fx' 0 x T l 0

13 The fudametal matrix F F is the uique 3x3 rak matrix that satisfies x T Fx =0 for all x x. Traspose: if F is fudametal matrix for (x,x ), the F T is fudametal matrix for (x,x). Epipolar lies: l=fx & l =F T x 3. Epipoles: o all epipolar lies, thus e T Fx =0, x e T F=0, similarly Fe =0 4. F has 7 d.o.f., i.e. 3x3-(homogeeous)-(rak) 5. F is a correlatio, projective mappig from a poit x to a lie l=fx (ot a proper correlatio, i.e. ot ivertible)

14 The fudametal matrix F It ca be used for Simplifies matchig Allows to detect wrog matches

15 Estimatio of F 8-poit algorithm The fudametal matrix F is defied by x Fx' 0 for ay pair of matches x ad x i two images. Let x=(u,v,) T ad x =(u,v,) T, each match gives a liear equatio f F f f 3 f f f 3 f f f uu' f uv' f uf3 vu' f vv' f vf3 u' f3 v' f3 f33 0

16 8-poit algorithm f f f f f f f f f v u v v v u v u v u u u v u v v v u v u v u u u v u v v v v u u u v u u I reality, istead of solvig, we seek f to miimize subj.. Fid the vector correspodig to the least sigular value. 0 Af Af f

17 8-poit algorithm To eforce that F is of rak, F is replaced by F that miimizes subject to. F' F 0 ' det F It is achieved by SVD. Let, where, let the is the solutio. V U F Σ Σ Σ' V U F Σ' '

18 8-poit algorithm % Build the costrait matrix A = [x(,:)'.*x(,:)' x(,:)'.*x(,:)' x(,:)'... x(,:)'.*x(,:)' x(,:)'.*x(,:)' x(,:)'... x(,:)' x(,:)' oes(pts,) ]; [U,D,V] = svd(a); % Extract fudametal matrix from the colum of V % correspodig to the smallest sigular value. F = reshape(v(:,9),3,3)'; % Eforce rak costrait [U,D,V] = svd(f); F = U*diag([D(,) D(,) 0])*V';

19 8-poit algorithm Pros: it is liear, easy to implemet ad fast Cos: susceptible to oise

20 Problem with 8-poit algorithm ~0000 ~0000 ~0000 ~0000 ~00 ~00 ~00 ~00! Orders of magitude differece betwee colum of data matrix least-squares yields poor results f f f f f f f f f v u v v v u v u v u u u v u v v v u v u v u u u v u v v v v u u u v u u

21 Normalized 8-poit algorithm. Trasform iput by,. Call 8-poit o to obtai 3. F T' Τ FT ˆ xˆ, xˆ ˆ ' i x i i Tx i ' xˆ i Fˆ T ' x ' i x' Fx 0 x' ˆ T' FT xˆ 0 Fˆ

22 Normalized 8-poit algorithm ormalized least squares yields good results Trasform image to ~[-,]x[-,] (0,500) (700,500) (-,) (0,0) (,) (0,0) (700,0) (-,-) (,-)

23 Normalized 8-poit algorithm [x, T] = ormalisedpts(x); [x, T] = ormalisedpts(x); A = [x(,:).*x(,:)' x(,:)'.*x(,:)' x(,:)'... x(,:)'.*x(,:)' x(,:)'.*x(,:)' x(,:)'... x(,:)' x(,:)' oes(pts,) ]; [U,D,V] = svd(a); F = reshape(v(:,9),3,3)'; [U,D,V] = svd(f); F = U*diag([D(,) D(,) 0])*V'; % Deormalise F = T'*F*T;

24 Normalizatio fuctio [ewpts, T] = ormalisedpts(pts) c = mea(pts(:,:)')'; % Cetroid ewp(,:) = pts(,:)-c(); % Shift origi to cetroid. ewp(,:) = pts(,:)-c(); meadist = mea(sqrt(ewp(,:).^ + ewp(,:).^)); scale = sqrt()/meadist; T = [scale 0 -scale*c() 0 scale -scale*c() 0 0 ]; ewpts = T*pts;

25 RANSAC repeat select miimal sample (8 matches) compute solutio(s) for F determie iliers util (#iliers,#samples)>95% or too may times compute F based o all iliers

26 Results (groud truth)

27 Results (8-poit algorithm)

28 Results (ormalized 8-poit algorithm)

29 Structure from motio

30 Structure from motio Ukow camera viewpoits structure for motio: automatic recovery of camera motio ad scee structure from two or more images. It is a self calibratio techique ad called automatic camera trackig or matchmovig.

31 Applicatios For computer visio, multiple-view shape recostructio, ovel view sythesis ad autoomous vehicle avigatio. For film productio, seamless isertio of CGI ito live-actio backgrouds

32 Matchmove example # example # example #3

33 Structure from motio D feature trackig 3D estimatio optimizatio (budle adjust) geometry fittig SFM pipelie

34 Structure from motio Step : Track Features Detect good features, Shi & Tomasi, SIFT Fid correspodeces betwee frames Lucas & Kaade-style motio estimatio widow-based correlatio SIFT matchig

35 KLT trackig

36 Structure from Motio Step : Estimate Motio ad Structure Simplified projectio model, e.g., [Tomasi 9] or 3 views at a time [Hartley 00]

37 Structure from Motio Step 3: Refie estimates Budle adjustmet i photogrammetry Other iterative methods

38 Structure from Motio Step 4: Recover surfaces (image-based triagulatio, silhouettes, stereo ) Good mesh

39 Factorizatio methods

40 Problem statemet

41 Notatios 3D poits are see i m views q=(u,v,): D image poit p=(x,y,z,): 3D scee poit : projectio matrix : projectio fuctio q ij is the projectio of the i-th poit o image j ij projective depth of q ij q ij ( p j i ) ( x, y, z) ( x / z, y / z) ij z

42 Structure from motio Estimate ad to miimize ( Π,, Π, p,, p ) m log P( ( Π p ); q m ij j i ij j i w ij j pi if pi is visible 0 otherwise w i view Assume isotropic Gaussia oise, it is reduced to j ) m ( Π,, Πm, p,, p) wij ( Π jpi ) qij j i Start from a simpler projectio model

43 Orthographic projectio Special case of perspective projectio Distace from the COP to the PP is ifiite Image World Also called parallel projectio : (x, y, z) (x, y)

44 SFM uder orthographic projectio D image poit Orthographic projectio icorporatig 3D rotatio 3D scee poit image offset q Πp t Trick 3 3 Choose scee origi to be cetroid of 3D poits Choose image origis to be cetroid of D poits Allows us to drop the camera traslatio: q Πp

45 factorizatio (Tomasi & Kaade) 3 3 p p p q q q projectio of features i oe image: 3 3 m m m m m m p p p Π Π Π q q q q q q q q q projectio of features i m images W measuremet M motio S shape Key Observatio: rak(w) <= 3

46 Factorizatio kow W m M m3 S 3 solve for Factorizatio Techique W is at most rak 3 (assumig o oise) We ca use sigular value decompositio to factor W: W m M' S' m3 3 S differs from S by a liear trasformatio A: W M' S' ( MA )( AS) Solve for A by eforcig metric costraits o M

47 Results

48 Extesios to factorizatio methods Projective projectio With missig data Projective projectio with missig data

49 Budle adjustmet

50 Budle adjustmet 3D poits are see i m views x ij is the projectio of the i-th poit o image j a j is the parameters for the j-th camera b i is the parameters for the i-th poit BA attempts to miimize the projectio error Euclidea distace predicted projectio

51 Leveberg-Marquardt method LM ca be thought of as a combiatio of steepest descet ad the Newto method. Whe the curret solutio is far from the correct oe, the algorithm behaves like a steepest descet method: slow, but guarateed to coverge. Whe the curret solutio is close to the correct solutio, it becomes a Newto s method.

52 Budle adjustmet

53 MatchMove

54 Applicatios of matchmove

55 Jurassic park

56 d3 boujou Eemy at the Gate, Double Negative

57 d3 boujou Eemy at the Gate, Double Negative

58 Photo Tourism

59 VideoTrace

60 Video stabilizatio

61 Refereces Richard Hartley, I Defese of the 8-poit Algorithm, ICCV, 995. Carlo Tomasi ad Takeo Kaade, Shape ad Motio from Image Streams: A Factorizatio Method, Proceedigs of Natl. Acad. Sci., 993. Maolis Lourakis ad Atois Argyros, The Desig ad Implemetatio of a Geeric Sparse Budle Adjustmet Software Package Based o the Leveberg-Marquardt Algorithm, FORTH- ICS/TR N. Savely, S. Seitz, R. Szeliski, Photo Tourism: Explorig Photo Collectios i 3D, SIGGRAPH 006. A. Hegel et. al., VideoTrace: Rapid Iteractive Scee Modellig from Video, SIGGRAPH 007.

62 Project #3 MatchMove It is more about usig tools i this project You ca choose either calibratio or structure from motio to achieve the goal Calibratio Voodoo/Icarus Examples from previous classes, #, #