CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.

Transcription

1 CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt. Session 4 Affine Structure from Motion Mani Golparvar-Fard Department of Civil and Environmental Engineering 329D, Newmark Civil Engineering Lab mgolpar@illinois.edu Department of Civil and Environmental Engineering, University of Illinois at Urbana-Champaign

2 Reminders Term Project (P2) is due Apr 4th! Every group should ONLY prepare a 6-slide PPT: Slide - Introduction to the Engineering problem your work is addressing; Slide 2- Review of previous works, or the works you're following for your own implementation; Slide 3- A summary of your technical solution; Slide 4- Presenting eperimental results and discuss validation approach (if you do not have eperimental results by then, that's fine... prepare a detailed plan as to how you will implement and validate your algorithm; and Slide 5- Plan and schedule of activities for the remainder of the project. Assignment A3 will be out net Tuesday 2 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

3 State-of-the-Art 3 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

4 State-of-the-Art speed-of-light-one-trillion-frames-per-second- 9742/ 4 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

5 Review 3D Reconstruction 3D Reconstruction Using Structured Light 3D reconstruction in Government Contet YXq5sXeAIZKSA&inde=&feature=plcp &list=uu8vcnrtf8yxq5sxeaizksa&lf=plcp&playnet=3 Early Structure from Motions Building Rome in a Day CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

6 Outline Multiple view geometry Affine structure from Motion - Affine structure from motion problem - Algebraic methods - Factorization methods Reading: [HZ] Chapters: 6,4,8 [FP] Chapter: 2 Some slides of this lectures are courtesy of prof Savarese, prof. J. Ponce, prof FF Li, prof S. Lazebnik & prof. M. Hebert CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 6

7 Structure from motion problem X j j M m M mj 2j M 2 Given m images of n fied 3D points ij = M i X j, i =,, m, j =,, n CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 7

8 Structure from motion problem X j j M m M mj 2j M 2 From the mn correspondences ij, estimate: m projection matrices M i n 3D points X j motion structure CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 8

9 Affine structure from motion (simpler problem) Image World Image From the mn correspondences ij, estimate: m projection matrices M i (affine cameras) n 3D points X j 9 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

10 Affine Cameras CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

11 Changing Camera Focal Length Increasing the focal length and distance of the camera in a perspective projection results in an approimation of orthographic projection mera_distance_focal_length.gif CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

12 p q r R Q P O X T K R y s K o y o T R K M 3 3 M Canonical perspective projection matri Affine homography (in 3D) Affine Homography (in 2D) Finite cameras CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 2

13 Transformation in 2D Affinities: y H y t A y' ' a -Preserve: - Parallel lines - Ratio of areas - Ratio of lengths on collinear lines - others - How many DOF? CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard,

14 Transformation in 2D Projective: y H y b v t A y' ' p -Preserve: - cross ratio of 4 collinear points - collinearity - and a few others - How many DOF? CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard,

15 Weak perspective projection When the relative scene depth is small compared to its distance from the camera P ~ ' m y' my where m f z ' is the magnification. 5 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

16 Orthographic (affine) projection When the camera is at a (roughly constant) distance from the scene ' y' y Distance from center of projection to image plane is infinite 6 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

17 Weak-Perspective Projection K y 7 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

18 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 Orthographic Projection K Weak-Perspective Projection K y Scaling function of the distance 8

19 X T K R s K y T R K M T R K M y s K o y o Projective case Affine case Parallel projection matri (points at infinity are mapped as points at infinity) Affine cameras CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 9

20 Weak perspective projection Qingming Festival by the Riverside Zhang Zeduan ~9 AD 2 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

21 X T K R s K y T R K M [3 3affine] [4 4 affine] a a a b M a a a b A b X b AX Euc M b b Z Y X a a a a a a y [Homogeneous] [non-homogeneous image coordinates] b A M M Euc ; P M Euc Affine cameras CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 2

22 Affine cameras p P p M = camera matri To recap: from now on we define M as the camera matri for the affine case p u v AP b M P ; M A b 22 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

23 The Affine Structure-from-Motion Problem Given m images of n fied points P j (=X i ) we can write Problem: estimate the m 24 matrices M i and the n positions P j from the mn correspondences p ij. How many equations and how many unknown? 2m n equations in 8m+3n unknowns Two approaches: - Algebraic approach (affine epipolar geometry; estimate F; cameras; points) - Factorization method 23 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

24 Algebraic analysis (2-view case) p P v p u Homogeneous system Full rank matri; dim =? CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

25 Algebraic analysis (2-view case) where The Affine Fundamental Matri! 25 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

26 Affine Epipolar Geometry Note: the epipolar lines are parallel. CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 26

27 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 27

28 From at least 4 correspondences, we obtain a linear system on the unknown alpha, beta, etc Measurements: u, u, v, v v u v u v u v u n n n n f Computed by least square and by enforcing f = SVD Estimating F CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 28

29 Estimating projection matrices from epipolar constraints If M i and P i are solutions, then M i and P i are also solutions, where and Q is an affine transformation. Proof: 29 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

30 Affine ambiguity Affine PX PQ - Q X A A 3 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

31 Estimating projection matrices from epipolar constraints p P p 3 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

32 Estimating projection matrices from epipolar constraints Choose Q such that Canonical affine cameras A ~ ~ b T A ~ ~ b a d b c T Function of the parameters of F CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 32

33 Estimating projection matrices from epipolar constraints Choose Q such that By re-enforcing the epipolar constraint, we can compute a, b, c, d directly from the measurements 33 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

34 Reminder: Epipolar constraint p P v p u Homogeneous system 34 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

35 Estimating projection matrices from epipolar constraints Choose Q such that A b Re-enforce the Epipolar constraint 35 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

36 Estimating projection matrices from epipolar constraints Choose Q such that A b 36 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

37 Estimating projection matrices from epipolar constraints Linear relationship between measurements and unknown Unknown: a, b, c, d Measurements: u, u, v, v From at least 4 correspondences, we can solve this linear system and compute a, b, c, d (via least square) The cameras can be computed How about the structure? CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 37

38 Estimating the structure from epipolar constraints A b Can be solved by least square again 38 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

39 A factorization method Tomasi & Kanade algorithm C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: A factorization method. IJCV, 9(2):37-54, November 992. Centering the data Factorization 4 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

40 Centering: subtract the centroid of the image points j i n k k j i n k i k i i j i n k ik ij ij ˆ n n n ˆ A X X X A b A X b A X CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 4 A factorization method - factorization

41 Centering: subtract the centroid of the image points j i n k k j i n k i k i i j i n k ik ij ij ˆ n n n ˆ A X X X A b A X b A X CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, A factorization method - factorization

42 Centering: subtract the centroid of the image points n k k j i n k i k i i j i n k ik ij ij n n n ˆ X X A b A X b A X j i ij X A ˆ Assume that the origin of the world coordinate system is at the centroid of the 3D points After centering, each normalized point ij is related to the 3D point X i by CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, A factorization method - factorization

43 A factorization method - Centering the data X ˆ A ij i X j 44

44 A factorization method - factorization D Let s create a 2m n data (measurement) matri: ˆ ˆ ˆ 2 m ˆ ˆ ˆ 2 22 m2 points (n ) ˆ ˆ ˆ n 2n mn cameras (2 m ) 45 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

45 Let s create a 2m n data (measurement) matri: n m mn m m n n X X X A A A D ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ cameras (2 m 3) points (3 n ) The measurement matri D = M S must have rank 3 (it s a product of a 2m3 matri and 3n matri) A factorization method - factorization (2 m n) M S CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 46

46 Factorizing the measurement matri 47 Slide courtesy of M. Hebert CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

47 Factorizing the measurement matri Singular value decomposition of D: 48 Slide courtesy of M. Hebert CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

48 Factorizing the measurement matri Singular value decomposition of D: Since rank (D)=3, there are only 3 non-zero singular values 49 Slide courtesy of M. Hebert CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

49 Factorizing the measurement matri Obtaining a factorization from SVD: What is the issue here? Motion (cameras) structure D has rank>3 because of - measurement noise - affine approimation 5 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

50 Factorizing the measurement matri Obtaining a factorization from SVD: structure D D 5 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

51 Affine ambiguity The decomposition is not unique. We get the same D by using any 3 3 matri C and applying the transformations M MC, S C - S We can enforce some Euclidean constraints to resolve the ambiguity (more on net lecture!) 52 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

52 Algorithm summary Given: m images and n features ij For each image i, center the feature coordinates Construct a 2m n measurement matri D: Column j contains the projection of point j in all views Row i contains one coordinate of the projections of all the n points in image i Factorize D: Compute SVD: D = U W V T Create U 3 by taking the first 3 columns of U Create V 3 by taking the first 3 columns of V Create W 3 by taking the upper left 3 3 block of W Create the motion and shape matrices: M = U 3 W 3 ½ and S = W 3 ½ V 3 T (or M = U 3 and S = W 3 V 3T ) Eliminate affine ambiguity 53 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

53 Reconstruction results 54 C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: A factorization method. IJCV, 9(2):37-54, November CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23

54 Net Lecture Multiple view geometry Perspective structure from Motion 55 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23