3D Geometric Computer Vision. Martin Jagersand Univ. Alberta Edmonton, Alberta, Canada

Transcription

1 3D Geometric Computer Vision Martin Jagersand Univ. Alberta Edmonton, Alberta, Canada

2 Multi-view geometry Build 3D models from images Carry out manipulation tasks Training for Amazon manipulation challenge And many other usages

3 Multi-view Geometry Relates 3D world points Camera calibr. 2D image points The reconstruction of 3D models of objects from a collection of 2D images

4 Multi-view Geometry Typical processing pipeline (C. Hernandez MVS tutorial)

5 Multi-view Geometry

6 Middlebury multi-view benchmark : : A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms, Seitz, Curless, Diebel, Szeliski CVPR 2006, vol. 1, pages Cited by 1479 Calibration accuracy on these datasets appears to be on the order of a pixel (a pixel spans about 1/4mm on the object).

7 Dorsal and Ventral Pathways Where/What or Action/Perception? Humans don t internalize detailed 3D maps! Use external world as map. (Hayhoe, Pelz, Rensink, Goodale etc)

8 Course content Video processing: Image motion and tracking 2D projective geometry 3D projective geometry Cameras, their math model and calibration Two-camera stereo 3D reconstruction Multi-view geometry and general 3D reconstruct. Light, surface reflectance and math modeling Computer vision systems

9 Challenges: Geometric size ambiguity

10 Challenges Light / photometric ambiguity

11 Challenges Light / photometric ambiguity

12 Fundamental types of video processing Visual motion detecton Relating two adjacent frames: (small differences): Im(x + x, y + y, t + t) = Im(x, y, t) 1/9/

13 Fundamental types of video processing Visual Tracking / Stabilization Globally relating all frames: (large differences):

14 Intensity images / video non-trivial to get information abs( Image 1 - Image 2) =? Constancy: The physical scene is the same Note: Almost all pixels change! 1/9/2018 How do we make use of this? 14

15 MTF Modular Tracking Framework Open source C++ implementation ROS interface Matlab/Pyhton Cross platform

16 Application: Augmenter Reality

17 Beyond 3D Non-rigid and articulated motion Humans ubiquitous in graphics applications A practical, realistic model requires Skeleton Geometry (manually modeled, laser scanned) Physical simulation for clothes, muscle Texture/appearance (from images) Animation (mocap, simulation, artist)

18 Beyond 3D: Non-rigid and articulated motion PhD work of Neil Birkbeck, Best thesis prize winner

19 Uses of partial information from 2D-3D camera geometry Measurements in single images Visual constraints: verify alignments, detect impossible configurations (Escher paintings) Visual servoing Video tracking Rendering many more Why? Compact, accurate Few relative alignments vs. complete global geometry

20 Geometric Cues Vanishing line Vertical vanishing point (at infinity) Vanishing point Vanishing point Single View Metrology

21 Estimating Height The distance t r b r is known Used to estimate the height of the man in the scene Single View Metrology (559 citations) A. Criminisi, I Reid, A Zisserman International Journal of Computer Vision 40 (2), , 2000

22 Geometry for hand-eye coordination Image-Based Visual Servoing (IBVS)

23 Intro to Image-based Visual Servoing IBVS Initial Image User Desired Image

24 Vision-Based Control (Visual Servoing) : Current Image Features : Desired Image Features

25 Vision-Based Control (Visual Servoing) : Current Image Features : Desired Image Features

26 Vision-Based Control (Visual Servoing) : Current Image Features : Desired Image Features

27 Vision-Based Control (Visual Servoing) : Current Image Features : Desired Image Features

28 Vision-Based Control (Visual Servoing) : Current Image Features : Desired Image Features

29 u,v Image-Space Error v u : Current Image Features : Desired Image Features E = [ ] One point E = [y 0 y ] Pixel coord E= y u y v Many points E = y. 1. y 8 y u y v y. 1. y 8 0

30 Problem: Lots of coordinate frames to calibrate Camera Center of projection Different models Robot and scene Base frame End-effector frame Object

31 Visual specifications Point to Point task error : E = [y y 0 ] y 0 y E = y. 1. y 16 Why 16 elements? y. 1. y 16 0 y y 0

32 Visual specifications 2 Point to Line Line: E pl (y, l) = y l l l y r l r l l = [ y 2 y 3 ] l Note: y s in homogeneous coord. y l = [ ] u l v l k [ y 3 ] [ y 2 ] How to design visual specifications in a principled way?

33 The 2D projective plane l X Projective point Homogeneous coordinates X Y s X Y k k s 0 x y = 1 k X Y Inhomogeneous equivalent 2D projective space models perspective imaging Each 3D ray is a point in P 2 : homogeneous coords. Ideal points P 2 is R 2 plus a line at infinity l X = X Y 0

34 Projective Lines l HZ X Y Z = X X l = A B C Projective line ~ a plane through the origin l T X = X T l = AX + BY + CZ = 0 X = X Y 0 l = Ideal line ~ the plane parallel to the image line at infinity Duality: For any 2d projective property, a dual property holds when the role of points and lines are interchanged. l = X 1 X 2 X = l 1 l 2 The line joining two points The point joining two lines

35 Projective transformations Homographies, collineations, projectivities 3x3 nonsingular H maps P 2 to P 2 8 degrees of freedom determined by 4 corresponding points Transforming Lines? subspaces preserved substitution dual transformation x' x' x' h = h h x T l = 0 x T l = 0 x T H T l = 0 l =H T l h h h h h h x = Hx x x x 1 2 3

36 Homographies a generalization of affine and Euclidean transforms Group Transformation Invariants Distortion Projective 8 DOF Affine 6 DOF Metric 4 DOF Euclidean 3 DOF H H H H P A S E A = T v A = T O sr = T O R = O T t v t 1 t 1 t 1 Cross ratio Intersection Tangency Parallelism Relative dist in 1d Line at infinity Relative distances Angles Dual conic Lengths Areas l C * 2 dof l 2 dof C *

37 How to define a visual task?

38 Visually defined alignment: basic point-to-point: e pp (y) = y 2 - y 1 or (homogenous coord) e pp (y) = y 2 x y 1 point-to-line: e pl (y) = y 1. (y 2 x y 3 )

39 Some more visual alignments parallel lines: e par (l) = (l 1 x l 2 ) x (l 3 x l 4 ) line-to-line: e ll (y) = y 1. (y 3 x y 4 ) + y 2. (y 3 x y 4 ) point-to-ellipse: e pe (y) = y T 1 C ellipse y 1

40 Now: Language for visual alignments What else do we need? Need: 1. Some way of entering alignments in images 2. video tracking to perform servoing! Feature trackers Registration trackers: Download:

41 Visual ambiguity Will the scissors cut the paper in the middle?

42 ambiguity Will the scissors cut the paper in the middle? NO!

43 Task Ambiguity Is the probe contacting the wire?

44 Task Ambiguity Is the probe contacting the wire? NO!

45 Solve the cut in the middle task? Compute paper midpoint. How?

46 Solve the cut in the middle task? Compute paper midpoint. (Are we done yet?) x m = (l 1 x l 2 ) l 1 l 2

47 Solve the cut in the middle task? X = (l 3 x l 4 ) Compute vanishing point X, Intersect X w. midpt X m l 4 l m = (X x X m ) l 3 Alternative formulations?

48 What information do we use to move? Mobile robot navigation: From appearance to metric SLAM Recognizable Locations l Topological Maps l Metric Topological Maps l Fully Metric Maps y Courtesy K. Arras 50 km 200 m 2 km 100 km How about other applications? {W} x

49 Beyond projective camera vision and screen GUI: Pointing

50 Questions? More information: Downloadable renderer+models Capturing software + IEEE VR tutorial text Main references for this talk: Jagersand et al Three Tier Model 3DPVT Jagersand Image-based Animation CVPR 1997 More papers:

51 Making Pizza with my robot 3 rd Prize ICRA 15 Video competition Questions? Change a Lightbulb ICRA 97 Video