Structure from Motion. Lecture-15

Transcription

1 Structure from Motion Lecture-15

2 Shape From X Recovery of 3D (shape) from one or two (2D images).

3 Shape From X Stereo Motion Shading Photometric Stereo Texture Contours Silhouettes Defocus

4 Applications Object Recognition Robotics Computer Graphics Image Retrieval Geo-localization Archeology Sports

5 Microsoft Kinect sensor Data Captured using Microsoft Kinect sensor RGB Camera IR Camera 1 IR Camera 2 Approximately 50,000 gesture samples

6 Gesture Lexicons Diving Signals Referee Signals Nurse Gesture Music Notes Gestures from Depth camera Gestures from RGB camera

7 Test case 1: Torso motion adds noise (devel gestures) Instances of Successful Recognition Instances of Failed Recognition

8 Test Case 2: Improvisations (devel 06 9 gestures) Instances of Successful Recognition Instances of Failed Recognition

9 Test Case 3: Subtle differences (devel gestures) Instances of Successful Recognition Instances of Failed Recognition

10 Humans are able to recover 3D from motion Moving Light Display

11 Shape from Motion

12 Problem Given optical flow or point correspondences, compute 3-D motion (translation and rotation) and shape (depth).

13 Structure from Motion S. Ullman Hanson & Riseman Webb & Aggarwal T. Huang Heeger and Jepson Chellappa Faugeras Zisserman Kanade Pentland Van Gool Pollefeys Seitz & Szeliski Shahsua Irani Vidal & Yi Ma Medioni Fleet Tian & Shah -

14 Photosynth

15 Tomasi and Kanade Factorization Orthographic Projection

16 Assumptions The camera model is orthographic. The positions of P points in F frames (F>=3), which are not all coplanar, and have been tracked. The entire sequence has been acquired before starting (batch mode). Camera calibration not needed, if we accept 3D points up to a scale factor.

17 Input Images KLT Tracks

18 Feature Points Image points (This is not optical flow

19 Mean Normalize Feature Points (A)

20 Orthographic Projection

21 Orthographic Projection 3D world point (C) Orthographic projection i, j, k are unit vectors along X, Y, Z

22 If Origin of world is at the centroid of object points, Second term is zero

23

24 (B) 3XP 2FX3 Rank of S is 3, because points in 3D space are not Co-planar

25 Rank Theorem Without noise, the registered measurement matrix is at most of rank three. Because W is a product of two matrices. The maximum rank of S is 3.

26 Linearly Independence A finite subset of n vectors, v 1, v 2,..., v n, from the vector space V, is linearly dependent if and only if there exists a set of n scalars, a 1, a 2,..., a n, not all zero, such that

27 Rank of a Matrix The column rank of a matrix A is the maximum number of linearly independent column vectors of A. The row rank of a matrix A is the maximum number of linearly independent row vectors of A. The column rank of A is the dimension of the column space of A The row rank of A is the dimension of the row space of A.

28 Example (Row Echelon) Rank is 2

29 How to find Translation? From (A) (E) From (B) Comparing above two eqs From (C) (D) is projection of camera translation along x-axis

30 How to find Translation 2FXP 2FX3 3XP 2FX1 1XP From (D)

31 How to find Translation Projected camera translation can be computed: From (D)

32 Noisy Measurements Without noise, the matrix must be at most of rank 3. When noise corrupts the images, however, will not be of rank 3. Rank theorem can be extended to the case of noisy measurements.

33 Singular Valued Decomposition SVD 2FXP 2FXP PXP PXP

34 Singular Value Decomposition (SVD) Theorem: Any m by n matrix A, for which,can be written as is diagonal mxn mxn nxn nxn are orthogonal

35 Approximate Rank 3 P-3 2F 3 P-3 3 P-3 3 P-3 P

36 Approximate Rank The best rank 3 approximation to the ideal registered measurement matrix.

37 Rank Theorem for noisy measurement The best possible shape and rotation estimate is obtained by considering only 3 greatest singular values of together with the corresponding left, right eigenvectors.

38 Approximate Rank Approximate Rotation matrix Approximate Shape matrix This decomposition is not unique Q is any 3X3 invertable matrix

39 Results..\..\CAP6411\Fall2002\tomasiTr92Figures.pdf

40 Hotel Sequence

41 Results (rotations)

42 Selected Features

43 Reconstructed Shape

44 Comparison

45 House Sequence

46 Reconstructed Walls

47 Further Reading C. Tomasi and T. Kanade. Shape and motion from image streams under orthography---a factorization method. International Journal on Computer Vision, 9(2): , November Computer Vision: Algorithms and Applications, Richard Szeliski, Section 7.3