Robot Sensing & Perception. METR 4202: Advanced Control & Robotics. Date Lecture (W: 11:10-12:40, ) 30-Jul Introduction 6-Aug

Transcription

1 Robot Sensing & Perception Seeing is forgetting the name of what one sees L. Weschler METR 4202: Advanced Control & Robotics Dr Sura Singh -- Lecture # 6 September metr4202@itee.uq.edu.au School of Information Technolog and Electrical Engineering at the Universit of Queensland Schedule Date Lecture W: 11:10-12: Jul Introduction 6-Aug Representing Position & Orientation & State Frames Transformation Matrices & Affine Transformations 13-Aug Robot Kinematics & Ekka Da 20-Aug Robot Dnamics & Control 27-Aug Robot Motion 3-Sep Sensing & Perception 10-Sep Multiple View Geometr Computer Vision 17-Sep Navigation & Localization + Prof. M. Srinivasan 24-Sep Motion Planning + Control 1-Oct Stud break 8-Oct State-Space Modelling 15-Oct Shaping the Dnamic Response 22-Oct Linear Observers & LQR 29-Oct Applications in Industr & Course Review METR 4202: Robotics 3 September

2 Reference Material UQ Librar epdf UQ Librar Hardcop UQ Librar/SpringerLink METR 4202: Robotics 3 September Quick Outline 1. Perception Camera Sensors 1. Image Formation Computational Photograph 2. Calibration 3. Feature etraction 4. Stereopsis and depth 5. Optical flow METR 4202: Robotics 3 September

3 Sensor Information Laser Vision/Cameras GPS METR 4202: Robotics 3 September Mapping: Indoor robots METR 4202: Robotics 3 September

4 Cameras Wikipedia E-30-Cutmodel METR 4202: Robotics 3 September Perception Making Sense from Sensors METR 4202: Robotics 3 September

5 Perception Perception is about understanding the image for informing latter robot / control action METR 4202: Robotics 3 September Perception Perception is about understanding the image for informing latter robot / control action METR 4202: Robotics 3 September

6 Image Formation: Lens Optics Sec. 2.2 from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September Image Formation: Lens Optics Chromatic Aberration & Vignetting Chromatic Aberration: Vignetting: Sec. 2.2 from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

7 Image Formation: Lens Optics Aperture / Depth of Field METR 4202: Robotics 3 September Image Formation Single View Geometr Corke Ch. 11 METR 4202: Robotics 3 September

8 Image Formation Single View Geometr [II] Hartl & Zisserman Ch. 6 METR 4202: Robotics 3 September Image Formation Single View Geometr Camera Projection Matri = Image point X = World point K = Camera Calibration Matri Perspective Camera as: where: P is 3 4 and of rank 3 METR 4202: Robotics 3 September

9 Image Formation Two-View Geometr [Stereopsis] Fundamental Matri METR 4202: Robotics 3 September Planar Projective Transformations Perspective projection of a plane lots of names for this: homograph colineation planar projective map Easil modeled using homogeneous coordinates 000 z sss 1 image plane To appl a homograph H s' * * s' * * s * * compute p = Hp p = p /s normalize b dividing b third component Slide from Szeliski Computer Vision: Algorithms and Applications p H * * * 1 METR 4202: Robotics 3 September p 9

10 Transformations Forward Warp Inverse Warp Sec. 3.6 from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September Transformations : New Image & : Old Image Euclidean: Distances preserved Similarit Scaled Rotation: Angles preserved Fig. 2.4 from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

11 Transformations [2] Affine : lines remain Projective: straight lines preserved H: Homogenous 33 Matri Fig. 2.4 from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September D Transformations = point in the new or 2 nd image = point in the old image Translation = + t Rotation = R + t Similarit = sr + t Affine = A Projective = A here is an inhomogeneous pt 2-vector is a homogeneous point METR 4202: Robotics 3 September

12 2-D Transformations METR 4202: Robotics 3 September D Transformations Slide from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

13 Image Rectification To unwarp rectif an image solve for H given p and p solve equations of the form: sp = Hp linear in unknowns: s and coefficients of H need at least 4 points Slide from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September D Projective Geometr These concepts generalize naturall to 3D Homogeneous coordinates Projective 3D points have four coords: P = XYZW Dualit A plane L is also represented b a 4-vector Points and planes are dual in 3D: L P=0 Projective transformations Represented b 44 matrices T: P = TP L = L T-1 Lines are a special case Slide from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

14 3D 2D Perspective Projection Image Formation Equations X c Y c Z c f u c u Slide from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September D 2D Perspective Projection Matri Projection camera matri: s * X * Y Z * 1 p s * * * * ΠP s * * * * * It s useful to decompose into T R project A Π s 0 t s t f R 0 I T intrinsics projection orientation position Slide from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

15 Calibration matri Is this form of K good enough? non-square piels digital video skew radial distortion From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September Calibration See: Camera Calibration Toolbo for Matlab Intrinsic: Internal Parameters Focal length: The focal length in piels. Principal point: The principal point Skew coefficient: The skew coefficient defining the angle between the and piel aes. Distortions: The image distortion coefficients radial and tangential distortions tpicall two quadratic functions Etrinsics: Where the Camera image plane is placed: Rotations: A set of 33 rotation matrices for each image Translations: A set of 31 translation vectors for each image METR 4202: Robotics 3 September

16 Camera calibration Determine camera parameters from known 3D points or calibration objects internal or intrinsic parameters such as focal length optical center aspect ratio: what kind of camera? eternal or etrinsic pose parameters: where is the camera? How can we do this? From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September Camera calibration approaches Possible approaches: linear regression least squares non-linear optimization vanishing points multiple planar patterns panoramas rotational motion From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

17 17 Measurements on Planes You can not just add a tape measure! Approach: unwarp then measure Slide from Szeliski Computer Vision: Algorithms and Applications 3 September METR 4202: Robotics 33 Projection Models Orthographic Weak Perspective Affine Perspective Projective * * * * * * * * Π z z t j j j t i i i Π t R Π * * * * * * * * * * * * Π z z t j j j t i i i f Π Slide from Szeliski Computer Vision: Algorithms and Applications 3 September METR 4202: Robotics 34

18 Properties of Projection Preserves Lines and conics Incidence Invariants cross-ratio Does not preserve Lengths Angles Parallelism METR 4202: Robotics 3 September The Projective Plane Wh do we need homogeneous coordinates? Represent points at infinit homographies perspective projection multi-view relationships What is the geometric intuition? A point in the image is a ra in projective space sss z image plane Each point on the plane is represented b a ra sss all points on the ra are equivalent: 1 s s s Slide from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

19 Projective Lines What is a line in projective space? A line is a plane of ras through origin all ras z satisfing: a + b + cz = 0 in vector notation : 0 a b c z l T p A line is represented as a homogeneous 3-vector l Slide from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September Point and Line Dualit A line l is a homogeneous 3-vector a ra It is to ever point ra p on the line: lt p=0 l p 1 p 2 l 1 l 2 What is the line l spanned b ras p 1 and p 2? l is to p 1 and p 2 l = p 1 p 2 l is the plane normal What is the intersection of two lines l 1 and l 2? p is to l 1 and l 2 p = l 1 l 2 Points and lines are dual in projective space ever propert of points also applies to lines p METR 4202: Robotics 3 September

20 Ideal points and lines Ideal point point at infinit p 0 parallel to image plane It has infinite image coordinates ss0 ss0 z image plane z image plane Line at infinit l parallel to image plane Contains all ideal points METR 4202: Robotics 3 September Vanishing Points Vanishing point projection of a point at infinit whiteboard image plane capture architecture vanishing point camera center ground plane Slide from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

21 Fun With Vanishing Points Slide from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September Vanishing Points 2D image plane vanishing point camera center line on ground plane METR 4202: Robotics 3 September

22 Vanishing Points Properties An two parallel lines have the same vanishing point The ra from C through v point is parallel to the lines An image ma have more than one vanishing point image plane vanishing point V camera center C line on ground plane line on ground plane METR 4202: Robotics 3 September Vanishing Lines Multiple Vanishing Points An set of parallel lines on the plane define a vanishing point The union of all of these vanishing points is the horizon line v 1 v 2 METR 4202: Robotics 3 September

23 Two-View Geometr: Epipolar Plane Epipole: The point of intersection of the line joining the camera centres the baseline with the image plane. Equivalentl the epipole is the image in one view of the camera centre of the other view. Epipolar plane is a plane containing the baseline. There is a one-parameter famil a pencil of epipolar planes Epipolar line is the intersection of an epipolar plane with the image plane. All epipolar lines intersect at the epipole. An epipolar plane intersects the left and right image planes in epipolar lines and defines the correspondence between the lines. METR 4202: Robotics 3 September Two-frame methods Two main variants: Calibrated: Essential matri E use ra directions i i Uncalibrated: Fundamental matri F [Hartle & Zisserman 2000] From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

24 Fundamental matri Camera calibrations are unknown F = 0 with F = [e] H = K [t] R K-1 Solve for F using least squares SVD re-scale i i so that i 1/2 [Hartle] e epipole is still the least singular vector of F H obtained from the other two s.v.s plane + paralla projective reconstruction use self-calibration to determine K [Pollefes] From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September Essential matri Co-planarit constraint: R + t [t] [t] R [t] [t] R E = 0 with E =[t] R Solve for E using least squares SVD t is the least singular vector of E R obtained from the other two s.v.s From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

25 Stereo: Epipolar geometr Match features along epipolar lines epipolar line epipolar plane viewing ra Slide from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September Stereo: epipolar geometr for two images or images with collinear camera centers can find epipolar lines epipolar lines are the projection of the pencil of planes passing through the centers Rectification: warping the input images perspective transformation so that epipolar lines are horizontal Slide from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

26 Fundamental Matri The fundamental matri is the algebraic representation of epipolar geometr. METR 4202: Robotics 3 September Fundamental Matri Eample Suppose the camera matrices are those of a calibrated stereo rig with the world origin at the first camera Then: Epipoles are at: METR 4202: Robotics 3 September

27 Summar of fundamental matri properties METR 4202: Robotics 3 September Fundamental Matri & Motion Under a pure translational camera motion 3D points appear to slide along parallel rails. The images of these parallel lines intersect in a vanishing point corresponding to the translation direction. The epipole e is the vanishing point. METR 4202: Robotics 3 September

28 Cool Robotics Share D. Wedge The Fundamental Matri Song METR 4202: Robotics 3 September Rectification Project each image onto same plane which is parallel to the epipole Resample lines and shear/stretch to place lines in correspondence and minimize distortion [Zhang and Loop MSR-TR-99-21] Slide from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

29 Rectification BAD! Slide from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September Rectification GOOD! Slide from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

30 Matching criteria Raw piel values correlation Band-pass filtered images [Jones & Malik 92] Corner like features [Zhang ] Edges [man people ] Gradients [Seitz 89; Scharstein 94] Rank statistics [Zabih & Woodfill 94] Slide from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September Finding correspondences Appl feature matching criterion e.g. correlation or Lucas-Kanade at all piels simultaneousl Search onl over epipolar lines man fewer candidate positions Slide from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

31 Image registration revisited How do we determine correspondences? block matching or SSD sum squared differences d is the disparit horizontal motion How big should the neighborhood be? Slide from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September Neighborhood size Smaller neighborhood: more details Larger neighborhood: fewer isolated mistakes w = 3 w = 20 Slide from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

32 Stereo: certaint modeling Compute certaint map from correlations input depth map certaint map METR 4202: Robotics 3 September Plane Sweep Stereo Sweep famil of planes through volume projective re-sampling of XYZ input image composite virtual camera each plane defines an image composite homograph METR 4202: Robotics 3 September

33 Plane sweep stereo Re-order piel / disparit evaluation loops for ever piel for ever disparit compute cost for ever disparit for ever piel compute cost METR 4202: Robotics 3 September Stereo matching framework For ever disparit compute raw matching costs Wh use a robust function? occlusions other outliers Can also use alternative match criteria METR 4202: Robotics 3 September

34 Stereo matching framework Aggregate costs spatiall Here we are using a bo filter efficient moving average implementation Can also use weighted average [non-linear] diffusion METR 4202: Robotics 3 September Stereo matching framework Choose winning disparit at each piel Interpolate to sub-piel accurac Ed d* d METR 4202: Robotics 3 September

35 Traditional Stereo Matching Advantages: gives detailed surface estimates fast algorithms based on moving averages sub-piel disparit estimates and confidence Limitations: narrow baseline nois estimates fails in tetureless areas gets confused near occlusion boundaries METR 4202: Robotics 3 September Stereo with Non-Linear Diffusion Problem with traditional approach: gets confused near discontinuities New approach: use iterative non-linear aggregation to obtain better estimate provabl equivalent to mean-field estimate of Markov Random Field METR 4202: Robotics 3 September

36 How to get Matching Points? Features Colour Corners Edges Lines Statistics on Edges: SIFT SURF ORB In OpenCV: The following detector tpes are supported: "FAST" FastFeatureDetector "STAR" StarFeatureDetector "SIFT" SIFT nonfree module "SURF" SURF nonfree module "ORB" ORB "BRISK" BRISK "MSER" MSER "GFTT" GoodFeaturesToTrackDetector "HARRIS" GoodFeaturesToTrackDetector with Harris detector enabled "Dense" DenseFeatureDetector "SimpleBlob" SimpleBlobDetector METR 4202: Robotics 3 September Feature-based stereo Match corner interest points Interpolate complete solution Slide from Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

37 Features -- Colour Features Fig: Ch. 10 Robotics Vision and Control Baer Patterns RGB is NOT an absolute metric colour space Also! RGB displa or additive colour does not map to CYMK printing or subtractive colour without calibration Y-Cr-Cb or HSV does not solve this either METR 4202: Robotics 3 September How to get the Features? Still MANY Was Cann edge detector: METR 4202: Robotics 3 September

38 Hough Transform Uses a voting mechanism Can be used for other lines and shapes not just straight lines METR 4202: Robotics 3 September Hough Transform: Voting Space Count the number of lines that can go through a point and move it from the - plane to the a-b plane There is onl a one- infinite number a line! of solutions not a two- infinite set a plane METR 4202: Robotics 3 September

39 Hough Transform: Voting Space In practice the polar form is often used This avoids problems with lines that are nearl vertical METR 4202: Robotics 3 September Hough Transform: Algorithm 1. Quantize the parameter space appropriatel. 2. Assume that each cell in the parameter space is an accumulator. Initialize all cells to zero. 3. For each point in the visual & range image space increment b 1 each of the accumulators that satisf the equation. 4. Maima in the accumulator arra correspond to the parameters of model instances. METR 4202: Robotics 3 September

40 Line Detection Hough Lines [1] A line in an image can be epressed as two variables: Cartesian coordinate sstem: mb Polar coordinate sstem: r θ avoids problems with vert. lines =m+b For each point 1 1 we can write: Each pair rθ represents a line that passes through 1 1 See also OpenCV documentation cv::houghlines METR 4202: Robotics 3 September Line Detection Hough Lines [2] Thus a given point gives a sinusoid Repeating for all points on the image See also OpenCV documentation cv::houghlines METR 4202: Robotics 3 September

41 Line Detection Hough Lines [3] Thus a given point gives a sinusoid Repeating for all points on the image NOTE that an intersection of sinusoids represents a point represents a line in which piel points la. Thus a line can be detected b finding the number of Intersections between curves See also OpenCV documentation cv::houghlines METR 4202: Robotics 3 September Cool Robotics Share -- Hough Transform METR 4202: Robotics 3 September

42 Line Etraction and Segmentation Adopted from Williams Fitch and Singh MTRX 4700 METR 4202: Robotics 3 September Line Formula Adopted from Williams Fitch and Singh MTRX 4700 METR 4202: Robotics 3 September

43 Line Estimation Least squares minimization of the line: Line Equation: Error in Fit: Solution: Adopted from Williams Fitch and Singh MTRX 4700 METR 4202: Robotics 3 September Line Splitting / Segmentation What about corners? Split into multiple lines via epectation maimization 1. Epect assume a number of lines N sa 3 2. Find breakpoints b finding nearest neighbours upto a threshold or simpl at random RANSAC 3. How to know N? Also RANSAC Adopted from Williams Fitch and Singh MTRX 4700 METR 4202: Robotics 3 September

44 of a Point from a Line Segment d D Adopted from Williams Fitch and Singh MTRX 4700 METR 4202: Robotics 3 September Edge Detection Cann edge detector: Pepsi Sequence: Image Data: and Szeliski CS223B-L9 See also: Use of Temporal information to aid segmentation: METR 4202: Robotics 3 September

45 Wh etract features? Object detection Robot Navigation Scene Recognition Steps: Etract Features Match Features Adopted drom S. Lazebnik Gang Hua CS 558 METR 4202: Robotics 3 September Wh etract features? [2] Panorama stitching Step 3: Align images Adopted from S. Lazebnik Gang Hua CS 558 METR 4202: Robotics 3 September

46 Characteristics of good features Repeatabilit The same feature can be found in several images despite geometric and photometric transformations Salienc Each feature is distinctive Compactness and efficienc Man fewer features than image piels Localit A feature occupies a relativel small area of the image; robust to clutter and occlusion Adopted from S. Lazebnik Gang Hua CS 558 METR 4202: Robotics 3 September Finding Corners Ke propert: in the region around a corner image gradient has two or more dominant directions Corners are repeatable and distinctive C.Harris and M.Stephens. "A Combined Corner and Edge Detector. Proceedings of the 4th Alve Vision Conference: pages Adopted from S. Lazebnik Gang Hua CS 558 METR 4202: Robotics 3 September

47 Corner Detection: Basic Idea Look through a window Shifting a window in an direction should give a large change in intensit flat region: no change in all directions Source: A. Efros edge : no change along the edge direction corner : significant change in all directions METR 4202: Robotics 3 September Corner Detection: Mathematics Change in appearance of window w for the shift [uv]: 2 E uv w I u v I I Eu v E32 Adopted from S. Lazebnik Gang Hua CS 558 w METR 4202: Robotics 3 September

48 Corner Detection: Mathematics Change in appearance of window w for the shift [uv]: 2 E uv w I u v I I Eu v E00 Adopted from S. Lazebnik Gang Hua CS 558 w METR 4202: Robotics 3 September Corner Detection: Mathematics Change in appearance of window w for the shift [uv]: 2 E uv w I u v I Window function Shifted intensit Intensit Window function w = or Adopted from S. Lazebnik Gang Hua CS in window 0 outside Gaussian METR 4202: Robotics Source: 3 September R. Szeliski

49 Corner Detection: Mathematics Change in appearance of window w for the shift [uv]: 2 E uv w I u v I We want to find out how this function behaves for small shifts Eu v Adopted from S. Lazebnik Gang Hua CS 558 METR 4202: Robotics 3 September Corner Detection: Mathematics Change in appearance of window w for the shift [uv]: Eu 00 1 Euu 00 E u v E00 [ u v] [ u v] Ev00 2 Euv 00 2 E uv w I u v I We want to find out how this function behaves for small shifts Euv00 u E vv00 v Local quadratic approimation of Euv in the neighborhood of 00 is given b the second-order Talor epansion: Adopted from S. Lazebnik Gang Hua CS 558 METR 4202: Robotics 3 September

50 50 Corner Detection: Mathematics v u E E E E v u E E v u E v u E vv uv uv uu v u ] [ ] [ 00 2 E uv w I u v I Second-order Talor epansion of Euv about 00: v u I I v u I w v u I v u I w v u E v u I I v u I w v u I v u I w v u E v u I I v u I w v u E uv uu u Adopted from S. Lazebnik Gang Hua CS September METR 4202: Robotics 99 Corner Detection: Mathematics v u I w I I w I I w I w v u v u E 2 2 ] [ 2 E uv w I u v I Second-order Talor epansion of Euv about 00: I I w E I I w E I I w E E E E uv vv uu v u Adopted from S. Lazebnik Gang Hua CS September METR 4202: Robotics 100

51 Harris detector: Steps Compute Gaussian derivatives at each piel Compute second moment matri M in a Gaussian window around each piel Compute corner response function R Threshold R Find local maima of response function nonmaimum suppression C.Harris and M.Stephens. A Combined Corner and Edge Detector. Proceedings of the 4th Alve Vision Conference: pages Adopted from S. Lazebnik Gang Hua CS 558 METR 4202: Robotics 3 September Harris Detector: Steps Adopted from S. Lazebnik Gang Hua CS 558 METR 4202: Robotics 3 September

52 Harris Detector: Steps Compute corner response R Adopted from S. Lazebnik Gang Hua CS 558 METR 4202: Robotics 3 September Harris Detector: Steps Find points with large corner response: R>threshold Adopted from S. Lazebnik Gang Hua CS 558 METR 4202: Robotics 3 September

53 Harris Detector: Steps Take onl the points of local maima of R Adopted from S. Lazebnik Gang Hua CS 558 METR 4202: Robotics 3 September Harris Detector: Steps Adopted from S. Lazebnik Gang Hua CS 558 METR 4202: Robotics 3 September

54 Invariance and covariance We want corner locations to be invariant to photometric transformations and covariant to geometric transformations Invariance: image is transformed and corner locations do not change Covariance: if we have two transformed versions of the same image features should be detected in corresponding locations Adopted from S. Lazebnik Gang Hua CS 558 METR 4202: Robotics 3 September RANdom SAmple Consensus 1. Repeatedl select a small minimal subset of correspondences 2. Estimate a solution in this case a the line 3. Count the number of inliers e <Θ for LMS estimate med e 4. Pick the best subset of inliers 5. Find a complete least-squares solution Related to least median squares See also: MAPSAC Maimum A Posteriori SAmple Consensus From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

55 Cool Robotics Share Time! D. Wedge The RANSAC Song METR 4202: Robotics 3 September Scale Invariant Feature Transform Basic idea: Take 1616 square window around detected feature Compute edge orientation angle of the gradient - 90 for each piel Throw out weak edges threshold gradient magnitude Create histogram of surviving edge orientations 0 2 angle histogram Adapted from slide b David Lowe METR 4202: Robotics 3 September

56 SIFT descriptor Full version Divide the 1616 window into a 44 grid of cells 22 case shown below Compute an orientation histogram for each cell 16 cells * 8 orientations = 128 dimensional descriptor Adapted from slide b David Lowe METR 4202: Robotics 3 September Properties of SIFT Etraordinaril robust matching technique Can handle changes in viewpoint Up to about 60 degree out of plane rotation Can handle significant changes in illumination Sometimes even da vs. night below Fast and efficient can run in real time Lots of code available From David Lowe and Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

57 Feature matching Given a feature in I 1 how to find the best match in I 2? 1. Define distance function that compares two descriptors 2. Test all the features in I 2 find the one with min distance From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September Feature distance How to define the difference between two features f 1 f 2? Simple approach is SSDf 1 f 2 sum of square differences between entries of the two descriptors can give good scores to ver ambiguous bad matches f 1 f 2 I 1 I 2 From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

58 Feature distance How to define the difference between two features f 1 f 2? Better approach: ratio distance = SSDf 1 f 2 / SSDf 1 f 2 f 2 is best SSD match to f 1 in I 2 f 2 is 2 nd best SSD match to f 1 in I 2 gives small values for ambiguous matches f 1 f f ' 2 2 I 1 I 2 From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September Evaluating the results How can we measure the performance of a feature matcher? feature distance From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

59 True/false positives true match 200 false match feature distance The distance threshold affects performance True positives = # of detected matches that are correct Suppose we want to maimize these how to choose threshold? False positives = # of detected matches that are incorrect Suppose we want to minimize these how to choose threshold? From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September Levenberg-Marquardt Iterative non-linear least squares [Press 92] Linearize measurement equations Substitute into log-likelihood equation: quadratic cost function in Dm From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

60 Levenberg-Marquardt What if it doesn t converge? Multipl diagonal b 1 + l increase l until it does Halve the step size Dm m favorite Use line search Other ideas? Uncertaint analsis: covariance S = A-1 Is maimum likelihood the best idea? How to start in vicinit of global minimum? From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September Camera matri calibration Advantages: ver simple to formulate and solve can recover K [R t] from M using QR decomposition [Golub & VanLoan 96] Disadvantages: doesn't compute internal parameters more unknowns than true degrees of freedom need a separate camera matri for each new view From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

61 Multi-plane calibration Use several images of planar target held at unknown orientations [Zhang 99] Compute plane homographies Solve for K-TK-1 from Hk s 1plane if onl f unknown 2 planes if fucvc unknown 3+ planes for full K Code available from Zhang and OpenCV From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September Rotational motion Use pure rotation large scene to estimate f estimate f from pairwise homographies re-estimate f from 360º gap optimize over all {KRj} parameters [Stein 95; Hartle 97; Shum & Szeliski 00; Kang & Weiss 99] f=510 f=468 Most accurate wa to get f short of surveing distant points From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

62 SFM: Structure from Motion & Cool Robotics Share this week METR 4202: Robotics 3 September Structure [from] Motion Given a set of feature tracks estimate the 3D structure and 3D camera motion. Assumption: orthographic projection Tracks: u fp v fp f: frame p: point Subtract out mean 2D position i f : rotation s p : position From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

63 Structure from motion How man points do we need to match? 2 frames: Rt: 5 dof + 3n point locations 4n point measurements n 5 k frames: 6k n 2kn alwas want to use man more From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September Measurement equations Measurement equations u fp = i ft s p i f : rotation s p : position v fp = j ft s p Stack them up W = R S R = i 1 i F j 1 j F T S = s 1 s P From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

64 Factorization W = R 2F3 S 3P SVD W = U Λ V Λ must be rank 3 W = U Λ 1/2 Λ 1/2 V = U V Make R orthogonal R = QU S = Q -1 V i ft Q T Qi f = 1 From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September Results Look at paper figures From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

65 Bundle Adjustment What makes this non-linear minimization hard? man more parameters: potentiall slow poorer conditioning high correlation potentiall lots of outliers gauge coordinate freedom From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September Lots of parameters: sparsit Onl a few entries in Jacobian are non-zero From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

66 Sparse Cholesk skline First used in finite element analsis Applied to SfM b [Szeliski & Kang 1994] structure motion fill-in From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September Conditioning and gauge freedom Poor conditioning: use 2nd order method use Cholesk decomposition Gauge freedom fi certain parameters orientation or zero out last few rows in Cholesk decomposition From Szeliski Computer Vision: Algorithms and Applications METR 4202: Robotics 3 September

67 More Cool Robotics Share! METR 4202: Robotics 3 September Cool Robotics Share IV Source: Youtube: Wired How the Tesla Model S is Made METR 4202: Robotics 3 September