Structure from Motion

Transcription

1 04/4/ Structure from Motion Comuter Vision CS 543 / ECE 549 University of Illinois Derek Hoiem Many slides adated from Lana Lazebnik, Silvio Saverese, Steve Seitz

2 his class: structure from motion Reca of eiolar geometry Deth from two views Projective structure from motion Affine structure from motion

3 Reca: Eioles Point in left image corresonds to eiolar line l in right image Eiolar line asses through the eiole (the intersection of the cameras baseline with the image lane C C

4 Reca: Fundamental Matri Fundamental matri mas from a oint in one image to a line in the other If and corresond to the same 3d oint X:

5 Reca: Automatically Relating Projections Assume we have matched oints with outliers Homograhy (No ranslation) Corresondence Relation ' H ' H. Normalize image coordinates ~ ~ 2. RANSAC with 4 oints 3. De-normalize: 0 H ~ H Fundamental Matri (ranslation) Corresondence Relation F 0. Normalize image coordinates ~ ~ 2. RANSAC with 8 oints 3. Enforce det F by SVD ~ 4. De-normalize: 0 F ~ F

6 Reca We can get rojection matrices P and P u to a rojective ambiguity P I 0 P e F e e F 0 See HZ Code: function P = vgg_p_from_f(f) [U,S,V] = svd(f); e = U(:,3); P = [-vgg_contres(e)*f e];

7 Reca Fundamental matri song

8 riangulation: Linear Solution Generally, rays C and C will not eactly intersect Can solve via SVD, finding a least squares solution to a system of equations X ' 0 X P 0 PX AX 0 v u v u A Further reading: HZ

9 riangulation: Linear Solution Given P, P,,. Precondition oints and rojection matrices 2. Create matri A 3. [U, S, V] = svd(a) 4. X = V(:, end) Pros and Cons Works for any number of corresonding images Not rojectively invariant v u v u 3 2 P v u v u A 3 2 P Code: htt://

10 riangulation: Non-linear Solution Minimize rojected error while satisfying F=0 Solution is a 6-degree olynomial of t, minimizing Further reading: HZ. 38

11 Projective structure from motion Given: m images of n fied 3D oints ij = P i X j, i =,, m, j =,, n Problem: estimate m rojection matrices P i and n 3D oints X j from the mn corresonding oints ij X j j 3j P 2j Slides from Lana Lazebnik P 2 P 3

12 Projective structure from motion Given: m images of n fied 3D oints ij = P i X j, i =,, m, j =,, n Problem: estimate m rojection matrices P i and n 3D oints X j from the mn corresonding oints ij With no calibration info, cameras and oints can only be recovered u to a 44 rojective transformation Q: X QX, P PQ - We can solve for structure and motion when 2mn >= m +3n 5 For two cameras, at least 7 oints are needed

13 cameras Sequential structure from motion Initialize motion from two images using fundamental matri Initialize structure by triangulation oints For each additional view: Determine rojection matri of new camera using all the known 3D oints that are visible in its image calibration

14 cameras Sequential structure from motion Initialize motion from two images using fundamental matri Initialize structure by triangulation oints For each additional view: Determine rojection matri of new camera using all the known 3D oints that are visible in its image calibration Refine and etend structure: comute new 3D oints, re-otimize eisting oints that are also seen by this camera triangulation

15 cameras Sequential structure from motion Initialize motion from two images using fundamental matri Initialize structure by triangulation oints For each additional view: Determine rojection matri of new camera using all the known 3D oints that are visible in its image calibration Refine and etend structure: comute new 3D oints, re-otimize eisting oints that are also seen by this camera triangulation Refine structure and motion: bundle adjustment

16 Bundle adjustment Non-linear method for refining structure and motion Minimizing rerojection error m E( P, X) D n i j 2, P X ij i j X j P X j j 3j P P 2 X j 2j P 3 X j P 2 P 3

17 Auto-calibration Auto-calibration: determining intrinsic camera arameters directly from uncalibrated images For eamle, we can use the constraint that a moving camera has a fied intrinsic matri Comute initial rojective reconstruction and find 3D rojective transformation matri Q such that all camera matrices are in the form P i = K [R i t i ] Can use constraints on the form of the calibration matri, such as zero skew

18 Summary so far From two images, we can: Recover fundamental matri F Recover canonical cameras P and P from F Estimate 3D ositions (if K is known) that corresond to each iel For a moving camera, we can: Initialize by comuting F, P, X for two images Sequentially add new images, comuting new P, refining X, and adding oints Auto-calibrate assuming fied calibration matri to ugrade to similarity transform

19 Photo synth Noah Snavely, Steven M. Seitz, Richard Szeliski, "Photo tourism: Eloring hoto collections in 3D," SIGGRAPH 2006 htt://hotosynth.net/

20 3D from multile images Building Rome in a Day: Agarwal et al. 2009

21 Structure from motion under orthograhic rojection 3D Reconstruction of a Rotating Ping-Pong Ball Reasonable choice when Change in deth of oints in scene is much smaller than distance to camera Cameras do not move towards or away from the scene C. omasi and. Kanade. Shae and motion from image streams under orthograhy: A factorization method. IJCV, 9(2):37-54, November 992.

22 Affine rojection for rotated/translated camera a 2 a X

23 Affine structure from motion Affine rojection is a linear maing + translation in inhomogeneous coordinates y a a 2 a a 2 22 a a 3 23 X t Y t Z y AX t a 2 a X Projection of world origin. We are given corresonding 2D oints () in several frames 2. We want to estimate the 3D oints (X) and the affine arameters of each camera (A)

24 Ste : Simlify by getting rid of t: shift to centroid of oints for each camera n k ik ij ij n t AX j i n k k j i n k i k i i j i n k ik ij n n n A X X X A t A X t A X j i ij X A 2d normalized oint (observed) 3d normalized oint Linear (affine) maing

25 Suose we know 3D oints and affine camera arameters then, we can comute the observed 2d ositions of each oint mn m m n n n m X X X A A A Camera Parameters (2m3) 3D Points (3n) 2D Image Points (2mn)

26 What if we instead observe corresonding 2d image oints? Can we recover the camera arameters and 3d oints? cameras (2 m) oints (n) n m mn m m n n X X X A A A D ? What rank is the matri of 2D oints?

27 Factorizing the measurement matri AX Source: M. Hebert

28 Factorizing the measurement matri Singular value decomosition of D: Source: M. Hebert

29 Factorizing the measurement matri Singular value decomosition of D: Source: M. Hebert

30 Factorizing the measurement matri Obtaining a factorization from SVD: Source: M. Hebert

31 Factorizing the measurement matri Obtaining a factorization from SVD: A ~ X ~ Source: M. Hebert

32 Affine ambiguity A ~ X ~ S he decomosition is not unique. We get the same D by using any 3 3 matri C and alying the transformations A AC, X C - X hat is because we have only an affine transformation and we have not enforced any Euclidean constraints (like forcing the image aes to be erendicular, for eamle) Source: M. Hebert

33 Eliminating the affine ambiguity Orthograhic: image aes are erendicular and of unit length a a 2 = 0 a 2 = a 2 2 = a 2 a X Source: M. Hebert

34 Solve for orthograhic constraints hree equations for each image i ~ a ~ a ~ a ~ a ~ a ~ a icc i i2cc i2 icc i2 0 where ~ A i ~ a ~ a i i2 Solve for L = CC Recover C from L by Cholesky decomosition: L = CC ~ Udate A and X: A = AC, X = C - ~ X

35 Algorithm summary Given: m images and n tracked features ij For each image i, center the feature coordinates Construct a 2m n measurement matri D: Column j contains the rojection of oint j in all views Row i contains one coordinate of the rojections of all the n oints in image i Factorize D: Comute SVD: D = U W V Create U 3 by taking the first 3 columns of U Create V 3 by taking the first 3 columns of V Create W 3 by taking the uer left 3 3 block of W Create the motion (affine) and shae (3D) matrices: A = U 3 W 3 ½ and X = W 3 ½ V 3 Eliminate affine ambiguity Source: M. Hebert

36 Dealing with missing data So far, we have assumed that all oints are visible in all views In reality, the measurement matri tyically looks something like this: cameras oints One solution: solve using a dense submatri of visible oints Iteratively add new cameras

37 A nice short elanation Class notes from Lischinksi and Gruber htt://

38 Reconstruction results (your HW 4) C. omasi and. Kanade. Shae and motion from image streams under orthograhy: A factorization method. IJCV, 9(2):37-54, November 992.

39 HW 4: Problem 2 summary. Detect interest oints (e.g., Harris) 2 I ( D) I I y ( D) ( I, D) g( I ) 2 I I y ( D) I y ( D). Image derivatives I I y det M trace M Square of derivatives 3. Gaussian filter g( I ) I 2 I y 2 I I y g(i 2 ) g(i y2 ) g(i I y ) 4. Cornerness function both eigenvalues are strong 2 har det[ (, )] [trace( (, )) ] g I D ( I ) g( I y ) [ g( I I y)] [ g( I ) g( I y I D )] 2 5. Non-maima suression 43 har

40 HW 4: Problem 2 summary 2. Corresondence via Lucas-Kanade tracking a) Initialize (,y ) = (,y) b) Comute (u,v) by Original (,y) osition I t = I(, y, t+) - I(, y, t) 2 nd moment matri for feature atch in first image dislacement c) Shift window by (u, v): = +u; y =y +v; d) Recalculate I t e) Reeat stes 2-4 until small change Use interolation for subiel values

41 HW 4: Problem 2 summary 3. Get Affine camera matri and 3D oints using omasi-kanade factorization Solve for orthograhic constraints

42 HW 4: Problem 2 summary is Helful matlab functions: inter2, meshgrid, ordfilt2 (for getting local maimum), svd, chol When selecting interest oints, must choose aroriate threshold on Harris criteria or the smaller eigenvalue, or choose to N oints Vectorize to make tracking fast (inter2 will be the bottleneck) If you get stuck on one art, can the included intermediate results Get tracking working on one oint for a few frames before trying to get it working for all oints Etra roblems Either for fun, or if you weren t able to comlete earlier homeworks Affine verification Missing track comletion Otical flow Coarse-to-fine tracking

43 See you net week Object tracking Action recognition