Multiple View Geometry in Computer Vision

Similar documents
Epipolar Geometry class 11

3D reconstruction class 11

Projective 2D Geometry

3D Photography: Epipolar geometry

Camera models and calibration

Structure from motion

METR Robotics Tutorial 2 Week 2: Homogeneous Coordinates

Epipolar Geometry Prof. D. Stricker. With slides from A. Zisserman, S. Lazebnik, Seitz

Structure from motion

Computer Vision I - Appearance-based Matching and Projective Geometry

Structure from Motion

Robot Vision: Projective Geometry

calibrated coordinates Linear transformation pixel coordinates

Multiple Views Geometry

Computer Vision I - Algorithms and Applications: Multi-View 3D reconstruction

Camera Calibration. Schedule. Jesus J Caban. Note: You have until next Monday to let me know. ! Today:! Camera calibration

Homogeneous Coordinates. Lecture18: Camera Models. Representation of Line and Point in 2D. Cross Product. Overall scaling is NOT important.

Projective geometry for Computer Vision

3D Photography. Marc Pollefeys, Torsten Sattler. Spring 2015

Automatic Image Alignment (feature-based)

Last lecture. Passive Stereo Spacetime Stereo

Agenda. Rotations. Camera models. Camera calibration. Homographies

Agenda. Rotations. Camera calibration. Homography. Ransac

Computer Vision I - Appearance-based Matching and Projective Geometry

Vision par ordinateur

Lecture 9: Epipolar Geometry

Invariance of l and the Conic Dual to Circular Points C

3D Reconstruction from Two Views

Multiple View Geometry in Computer Vision Second Edition

Stereo and Epipolar geometry

Structure from Motion CSC 767

Unit 3 Multiple View Geometry

Index. 3D reconstruction, point algorithm, point algorithm, point algorithm, point algorithm, 253

3D Computer Vision. Structure from Motion. Prof. Didier Stricker

CS 664 Slides #9 Multi-Camera Geometry. Prof. Dan Huttenlocher Fall 2003

Computer Vision Projective Geometry and Calibration. Pinhole cameras

Camera Geometry II. COS 429 Princeton University

55:148 Digital Image Processing Chapter 11 3D Vision, Geometry

Index. 3D reconstruction, point algorithm, point algorithm, point algorithm, point algorithm, 263

Stereo Vision. MAN-522 Computer Vision

BIL Computer Vision Apr 16, 2014

3D Computer Vision II. Reminder Projective Geometry, Transformations

Week 2: Two-View Geometry. Padua Summer 08 Frank Dellaert

Rectification and Distortion Correction

Feature Based Registration - Image Alignment

55:148 Digital Image Processing Chapter 11 3D Vision, Geometry

Multiple View Geometry in Computer Vision

Multiple View Geometry in computer vision

Structure from Motion

A Summary of Projective Geometry

Projective geometry, camera models and calibration

Part 0. The Background: Projective Geometry, Transformations and Estimation

Augmented Reality II - Camera Calibration - Gudrun Klinker May 11, 2004

Computer Vision. 2. Projective Geometry in 3D. Lars Schmidt-Thieme

Computing F class 13. Multiple View Geometry. Comp Marc Pollefeys

CS231A Midterm Review. Friday 5/6/2016

Two-view geometry Computer Vision Spring 2018, Lecture 10

CS231M Mobile Computer Vision Structure from motion

CS-9645 Introduction to Computer Vision Techniques Winter 2019

CSE 252B: Computer Vision II

COMPARATIVE STUDY OF DIFFERENT APPROACHES FOR EFFICIENT RECTIFICATION UNDER GENERAL MOTION

Epipolar Geometry and the Essential Matrix

Camera model and calibration

Structure from Motion. Introduction to Computer Vision CSE 152 Lecture 10

Computer Vision cmput 428/615

Epipolar Geometry and Stereo Vision

Structure from Motion

Structure from motion

Recovering structure from a single view Pinhole perspective projection

C / 35. C18 Computer Vision. David Murray. dwm/courses/4cv.

Epipolar geometry. x x

Visual Recognition: Image Formation

Instance-level recognition I. - Camera geometry and image alignment

More on single-view geometry class 10

Multiple View Geometry in Computer Vision

Projective geometry for 3D Computer Vision

CS201 Computer Vision Camera Geometry

MAPI Computer Vision. Multiple View Geometry

Robust Geometry Estimation from two Images

EXAM SOLUTIONS. Image Processing and Computer Vision Course 2D1421 Monday, 13 th of March 2006,

Part I: Single and Two View Geometry Internal camera parameters

Visual Odometry. Features, Tracking, Essential Matrix, and RANSAC. Stephan Weiss Computer Vision Group NASA-JPL / CalTech

N-Views (1) Homographies and Projection

Multiple View Geometry in Computer Vision

3D Geometry and Camera Calibration

Pin Hole Cameras & Warp Functions

Camera model and multiple view geometry

Auto-calibration. Computer Vision II CSE 252B

Automatic Image Alignment

Automatic Image Alignment

Motion Estimation and Optical Flow Tracking

Robotics - Projective Geometry and Camera model. Matteo Pirotta

EM225 Projective Geometry Part 2

Lecture 10: Multi-view geometry

Multi-view geometry problems

Structure from Motion. Prof. Marco Marcon

Homographies and RANSAC

CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.

Robotics - Single view, Epipolar geometry, Image Features. Simone Ceriani

CS231A Course Notes 4: Stereo Systems and Structure from Motion

Transcription:

http://www.cs.unc.edu/~marc/ Multiple View Geometry in Computer Vision 2011.09.

2

3

4

5

Visual 3D Modeling from Images 6

AutoStitch http://cs.bath.ac.uk/brown/autostitch/autostitch.html 7

Image Composite Editor http://research.microsoft.com/en-us/um/ redmond/groups/ivm/ice/ 8

Deep Zoom & HD View http://research.microsoft.com/enus/um/redmond/groups/ivm/hdview/ 9

Photosynth http://photosynth.net/ 10

PhotoCity http://photocitygame.com/ 11

Projective Transformations

Homogeneous coordinates Homogeneous representation of lines ax by c 0 a,b,c T ka a,b,c T ~ ka,b,c T ( ) x ( kb) y kc 0, k 0 equivalence class of vectors, any vector is representative Set of all equivalence classes in R 3 (0,0,0) T forms P 2 Homogeneous representation of points x xy,,1 on l a,b,c T if and only if ax by c 0 T x,y, 1 a,b,c x,y, 1 l 0 T T x, y,1 ~ kx, y,1, k 0 The point x lies on the line l if and only if x T l=l T x=0 Homogeneous coordinates x, y, z T Inhomogeneous coordinates x, y T but only 2DOF 13

Points from lines and vice-versa Intersections of lines The intersection of two lines l and l' is x l l' Line joining two points The line through two points x and x' is l x x' Example y 1 x 1 14

Ideal points and the line at infinity Intersections of parallel lines l a, b, c T and l' a, b, c' T l l' b, a,0 T Example b, a a,b tangent vector normal direction x 1 x 2 Ideal points x, 2,0 T 1 x 0,0,1 Line at infinity T l P 2 R 2 l Note that in P 2 there is no distinction between ideal points and others 15

A model for the projective plane exactly one line through two points exactly one point at intersection of two lines 16

Duality x l x T l 0 l T x 0 x l l' l x x' Duality principle: To any theorem of 2-dimensional projective geometry there corresponds a dual theorem, which may be derived by interchanging the role of points and lines in the original theorem 17

Projective 2D Geometry 18

Projective transformations Definition: A projectivity is an invertible mapping h from P 2 to itself such that three points x 1,x 2,x 3 lie on the same line if and only if h(x 1 ),h(x 2 ),h(x 3 ) do. Theorem: A mapping h:p 2 P 2 is a projectivity if and only if there exist a non-singular 3x3 matrix H such that for any point in P 2 reprented by a vector x it is true that h(x)=hx Definition: Projective transformation x' x' x' 1 2 3 h h h 11 21 31 h h h 12 22 32 h h h 13 23 33 x x x 1 2 3 or x' H x 8DOF projectivity=collineation=projective transformation=homography 19

Mapping between planes central projection may be expressed by x =Hx (application of theorem) 20

Removing projective distortion select four points in a plane with know coordinates x' 1 h11x h12y h13 x' 2 h21x h22y h x' y' x' h x h y h x' h x h y h x' y' 3 31 32 h31x h32y h33 h11x h12y h13 h31x h32y h33 h21x h22y h23 33 3 31 32 (linear in h ij ) (2 constraints/point, 8DOF 4 points needed) 23 33 21

22

More Examples 23

Transformation for lines For a point transformation x' H x Transformation for lines l' H -T l 24

Isometries x' y' 1 cos sin 0 sin cos 0 tx x t y y 1 1 1 orientation preserving: orientation reversing: 1 1 x' H E x R T 0 t x 1 R T R I 3DOF (1 rotation, 2 translation) special cases: pure rotation, pure translation Invariants: length, angle, area 25

Similarities 26 1 1 0 0 cos sin sin cos 1 ' ' y x t s s t s s y x y x x 0 x x' 1 t T R H s S I R R T also know as equi-form (shape preserving) metric structure = structure up to similarity (in literature) 4DOF (1 scale, 1 rotation, 2 translation) Invariants: ratios of length, angle, ratios of areas, parallel lines

Affine transformations 27 1 1 0 0 1 ' ' 22 21 12 11 y x t a a t a a y x y x x 0 x x' 1 t T A H A non-isotropic scaling! (2DOF: scale ratio and orientation) 6DOF (2 scale, 2 rotation, 2 translation) Invariants: parallel lines, ratios of parallel lengths, ratios of areas DR R R A 2 1 0 0 D

Projective transformations 28 x v x x' v P T t A H Action non-homogeneous over the plane 8DOF (2 scale, 2 rotation, 2 translation, 2 line at infinity) Invariants: cross-ratio of four points on a line (ratio of ratio) T 2 1, v v v v v s P A S T T T T v t v 0 1 0 0 1 0 t A I K R H H H H T RK tv A s K 1 det K upper-triangular, decomposition unique (if chosen s>0)

Overview Transformations Projective 8dof Affine 6dof Similarity 4dof Euclidean 3dof h h h a a 0 sr sr 0 r11 r21 0 11 21 31 11 21 11 21 h h h a a r r 12 22 32 12 22 0 sr sr 12 22 0 12 0 22 h13 h 23 h 33 tx t y 1 tx t y 1 tx t y 1 Concurrency, collinearity, order of contact (intersection, tangency, inflection, etc.), cross ratio Parallellism, ratio of areas, ratio of lengths on parallel lines (e.g midpoints), linear combinations of vectors (centroids). The line at infinity l Ratios of lengths, angles. The circular points I,J lengths, areas. 29

Line at infinity 30 2 2 1 1 2 1 2 1 0 v x v x v x x x x v A A T t 0 0 0 2 1 2 1 x x x x v A A T t Line at infinity becomes finite, allows to observe vanishing points, horizon, Line at infinity stays at infinity

The line at infinity 31 v 1 v 2 l 1 l 2 l 4 l 3 l v 1 v 2 l 2 1 1 l l v 4 3 2 l l v The line at infinity l is a fixed line under a projective transformation H if and only if H is an affinity l 1 0 0 1 t 0 l l A A H T T A

Affine properties from images projection rectification H PA 1 0 l1 0 1 l 2 0 0 H l 3 A T l l l, l 0 l 1 2 3 3 32

Affine Rectification v 1 v 2 l l 1 l 3 l 2 l 4 v v l l1 2 l3 4 v 1 v 2 1 l 2 l 33

Projective 3D geometry Projective 15dof A T v t v Intersection and tangency Affine 12dof A T 0 t 1 Parallellism of planes, Volume ratios, centroids, The plane at infinity π Similarity 7dof s R T 0 t 1 The absolute conic Ω Euclidean 6dof R 0 T t 1 Volume 34

Camera Calibration

Pinhole Camera Model 36 T T Z fy Z fx Z Y X ) /, / ( ),, ( 1 0 1 0 0 1 Z Y X f f Z fy fx Z Y X 1 0 1 0 1 0 1 1 Z Y X f f Z fy fx

Pinhole Camera Model x K I 0X cam K f f px p y 1 Principal point offset Calibration Matrix 37

Internal Camera Parameters 38

Camera rotation and translation X ~ R X~ -C ~ cam RC ~ X R Y X cam 0 1 Z 1 R 0 RC X 1 x K I 0X cam x KR I C ~ X x PX P K R t t RC ~ 39

Camera Parameter Matrix P 40

CCD Cameras 41

Correcting Radial Distortion of Cameras 42

43

Calibration Process 44

x c ( x c )(1 k r k r k r )cos 2 4 6 u x d x 1 d 2 d 3 d ( y c )(1 k r k r k r )sin 2 4 6 d y 1 d 2 d 3 d y c ( x c )(1 k r k r k r )sin 2 4 6 u y d x 1 d 2 d 3 d ( y c )(1 k r k r k r )cos 2 4 6 d y 1 d 2 d 3 d x y p p m x m y m m x 0 u 1 u 2 6 u m7yu 1 m x m y m m x 3 u 4 u 5 6 u m7yu 1 45

OpenCV http://opencv.willowgarage.com/documentation/camera_c alibration_and_3d_reconstruction.html 46

Camera Calibration Toolbox http://www.vision.caltech.edu/bouguetj/calib_doc/ 47

Camera Calibration 48

Calibrating a Stereo System 49

Robust Multi-camera Calibration http://graphics.stanford.edu/~vaibhav/projects/calibcs205/cs205.html 50

Multi-Camera Self Calibration http://cmp.felk.cvut.cz/~esvoboda/ 51

Multi-Camera Self Calibration 52

ARToolKit Camera Calibration 53

Epipolar Geometry and 3D Reconstruction.

Introduction Computer vision is concerned with the theory behind artificial systems that extract information from images. The image data can take many forms, such as video sequences, views from multiple cameras. Computer vision is, in some ways, the inverse of computer graphics. While computer graphics produces image data from 3D models, computer vision often produces 3D models from image data. Today one of the major problems in Computer vision is correspondence search. 55

Epipolar Geometry http://www.ai.sri.com/~luong/research/meta3dviewer/epipolargeo.html C,C,x,x and X are coplanar 56

Epipolar Geometry epipole 57

Epipolar Geometry All points on p project on l and l 58

Epipolar Geometry Family of planes p and lines l and l Intersection in e and e 59

Epipolar Geometry epipoles e,e = intersection of baseline with image plane = projection of projection center in other image = vanishing point of camera motion direction an epipolar plane = plane containing baseline (1-D family) an epipolar line = intersection of epipolar plane with image (always come in corresponding pairs) 60

The Fundamental Matrix F x' H π x l' e' x' e' H x Fx π H : projectivity=collineation=projective transformation=homography 61

Cross Products 62

Epipolar Lines 63

Three Questions (i) Correspondence geometry: Given an image point x in the first view, how does this constrain the position of the corresponding point x in the second image? (ii) Camera geometry (motion): Given a set of corresponding image points {x i x i }, i=1,,n, what are the cameras P and P for the two views? (iii) Scene geometry (structure): Given corresponding image points x i x i and cameras P, P, what is the position of (their pre-image) X in space? 64

Parameter estimation 2D homography Given a set of (x i,x i ), compute H (x i =Hx i ) 3D to 2D camera projection Given a set of (X i,x i ), compute P (x i =PX i ) Fundamental matrix Given a set of (x i,x i ), compute F (x i T Fx i =0) 65

The fundamental matrix F Algebraic Derivation X λ F e' P P' P x λc l' e' x' e' H x Fx l' P'C P'P x π P x P P I X λ (note: doesn t work for C=C F=0) 66

The fundamental matrix F Correspondence Condition The fundamental matrix satisfies the condition that for any pair of corresponding points x x in the two images x' T Fx 0 67

The Fundamental Matrix Song http://danielwedge.com/fmatrix/ 68

The Fundamental Matrix http://www.cs.unc.edu/~blloyd/comp290-089/fmatrix/ 69

Epipolar geometry: basic equation 70 0 Fx x' T separate known from unknown 0 ' ' ' ' ' ' 33 32 31 23 22 21 13 12 11 f yf xf f y yf y xf y f x yf x xf x 0,,,,,,,,,1, ',, ', ' ',, ', ' T 33 32 31 23 22 21 13 12 11 f f f f f f f f f y x y y y x y x y x x x (data) (unknowns) Af 0 0 f 1 ' ' ' ' ' ' 1 ' ' ' ' ' ' 1 1 1 1 1 1 1 1 1 1 1 1 n n n n n n n n n n n n y x y y y x y x y x x x y x y y y x y x y x x x

the NOT normalized 8-point algorithm 71 0 1 1 1 33 32 31 23 22 21 13 12 11 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 f f f f f f f f f y x y y y y x x x y x x y x y y y y x x x y x x y x y y y y x x x y x x n n n n n n n n n n n n ~10000 ~10000 ~10000 ~10000 ~100 ~100 1 ~100 ~100! Orders of magnitude difference Between column of data matrix least-squares yields poor results

the normalized 8-point algorithm Transform image to ~[-1,1]x[-1,1] (0,500) (700,500) 2 700 0 2 500 1 1 1 (-1,1) (0,0) (1,1) (0,0) (700,0) (-1,-1) (1,-1) Least squares yields good results (Hartley, PAMI 97) 72

The fundamental matrix F F is the unique 3x3 rank 2 matrix that satisfies x T Fx=0 for all x x (i) Transpose: if F is fundamental matrix for (P,P ), then F T is fundamental matrix for (P,P) (ii) Epipolar lines: l =Fx & l=f T x (iii) Epipoles: on all epipolar lines, thus e T Fx=0, x e T F=0, similarly Fe=0 (iv) F has 7 d.o.f., i.e. 3x3-1(homogeneous)-1(rank2) (v) F is a correlation, projective mapping from a point x to a line l =Fx (not a proper correlation, i.e. not invertible) 73

Fundamental matrix for pure translation 74

Fundamental matrix for pure translation 75

Fundamental matrix for pure translation e' e' F H 1 H K RK Example: 0 e' 1,0,0 T F 0 0 x' T Fx 0 y y' 0 0 1 0-1 0 x x' PX K[I 0]X -1 P'X K[I t] K x Z ( X,Y,Z ) T -1 K x/z x' x Kt/Z motion starts at x and moves towards e, faster depending on Z pure translation: F only 2 d.o.f., x T [e] x x=0 auto-epipolar 76

General Motion x' x' T T x' e' Hx 0 e' xˆ 0 K'RK -1 x K't/Z 77

Projective transformation and invariance Derivation based purely on projective concepts xˆ x Hx, xˆ' H'x' Fˆ PX H' -1 PH H X Pˆ Xˆ -1 P'H H X Pˆ' Xˆ -T FH -1 F invariant to transformations of projective 3-space x' P'X P, P' F F P,P' unique not unique Canonical form P [I 0] P' [M m] F m M F e' P' P 78

Projective ambiguity of cameras given F 79 previous slide: at least projective ambiguity this slide: not more! Show that if F is same for (P,P ) and ~ (P,P ), ~ there exists a projective transformation H so that P=HP and P =HP P [I 0] P' [A a] F a A ~ a lemma: ~ a A ~ ka A ~ k 1 A av ~ ~ P [I 0] P' [A ~ ~ a] T af a a A 0 ~ af rank 2 T a A ~ a a ka ~ - A 0 ka ~ - A av A ~ 1 H k I 0 1 T k v k 1 P'H [A a] k I 1 k v T 0 k [ k ~ ~ 1 ~ T A -av ka] P' ~ a ka (22-15=7, ok)

Canonical cameras given F F matrix corresponds to P,P iff P T FP is skew-symmetric X T P' T FPX 0, X F matrix, S skew-symmetric matrix P [I 0] [SF e' ] T Possible choice: P [I 0] P' [SF e' ] T T F[I 0] F S F T e' F P' [[e'] F (fund.matrix=f) e' ] T 0 F S 0 0 T F 0 0 Canonical representation: P [I 0] P' [[e'] F e' v T λe'] 80

The Essential matrix ~ Fundamental matrix for calibrated cameras (remove K) t T E R R[R t] xˆ ' T E Exˆ K' T 0 FK xˆ 5 d.o.f. (3 for R; 2 for t up to scale) K -1 x; xˆ' K -1 x' E is essential matrix if and only if two singularvalues are equal (and third=0) T E Udiag(1,1,0)V 81

3D Reconstruction 82

Epipolar Geometry Underlying structure in set of matches for rigid scenes l T 1 l 2 T m F m1 2 0 Fundamental matrix (3x3 rank 2 matrix) C1 m1 l m1 1 e 1 e 2 P m2 l2 m2 C 2 M L2 L1 Canonical representation: P [I 0] P' [[e'] F e' v T λe'] 1. Computable from corresponding points 2. Simplifies matching 3. Allows to detect wrong matches 4. Related to calibration 83

3D reconstruction of cameras and structure Reconstruction Problem: given x i x i, compute P,P and X i xi PX i x i PX i for all i without additional informastion possible up to projective ambiguity 84

Outline of 3D Reconstruction (i) (ii) (iii) Compute F from correspondences Compute camera matrices P,P from F Compute 3D point for each pair of corresponding points computation of F use x i Fx i =0 equations, linear in coeff. F 8 points (linear), 7 points (non-linear), 8+ (least-squares) computation of camera matrices T use P [I 0] P' [[e'] F e' v λe' ] triangulation compute intersection of two backprojected rays 85

Reconstruction ambiguity: similarity x i PX i -1 PH H X S S i PH -1 S K[R t] R' 0 T - R' λ T t' K[RR' T RR' T t' λt] 86

Reconstruction ambiguity: projective x i PX i -1 PH H X P P i 87

Terminology x i x i Original scene X i Projective, affine, similarity reconstruction = reconstruction that is identical to original up to projective, affine, similarity transformation Literature: Metric and Euclidean reconstruction = similarity reconstruction 88

The projective reconstruction theorem If a set of point correspondences in two views determine the fundamental matrix uniquely, then the scene and cameras may be reconstructed from these correspondences alone, and any two such reconstructions from these correspondences are projectively equivalent x i x i P P P -1 2 1H P,P ', P,P ', 1 1 X1i P theorem from last class 2 2 X2i X except :Fx xf 0-1 2 P 1H 2 HX1-1 HX1 i P1 H HX1 i P1 X1 i xi P2 2i 2 X along same ray of P 2, idem for P 2 two possibilities: X 2i =HX 1i, or points along baseline key result : allows reconstruction from pair of uncalibrated images i i 89

90

Stratified reconstruction (i) (ii) (iii) Projective reconstruction Affine reconstruction Metric reconstruction 91

Projective to affine Remember 2-D case 92

Projective to affine P,P', π T H - A, B, C, D T 0,0,0, 1 T X i π I 0 H π 0,0,0,1 T (if D 0) theorem says up to projective transformation, but projective with fixed p is affine transformation can be sufficient depending on application, e.g. mid-point, centroid, parallellism 93

Translational motion points at infinity are fixed for a pure translation reconstruction of x i x i is on p F [e] [e' ] P [I 0] P [I e'] 94

Scene constraints Parallel lines parallel lines intersect at infinity reconstruction of corresponding vanishing point yields point on plane at infinity 3 sets of parallel lines allow to uniquely determine p remark: in presence of noise determining the intersection of parallel lines is a delicate problem remark: obtaining vanishing point in one image can be sufficient 95

Scene constraints 96

Scene constraints Distance ratios on a line known distance ratio along a line allow to determine point at infinity (same as 2D case) 97

The infinity homography P [M m] P' [M' m' ] -1 H M' M unchanged under affine transformations P [M m] A a 0 1 H M' AA -1 M -1 [MA Ma m] X X ~,0 T affine reconstruction P [I 0] P [H e] x MX ~ x' M' X ~ 98

One of the cameras is affine According to the definition, the principal plane of an affine camera is at infinity to obtain affine recontruction, compute H that maps third row of P to (0,0,0,1) T and apply to cameras and reconstruction e.g. if P=[I 0], swap 3 rd and 4 th row, i.e. H 1 0 0 0 0 1 0 0 0 0 0 1 0 0 1 0 99

Affine to metric identify absolute conic transform so that : X Y Z 0, on π then projective transformation relating original and reconstruction is a similarity transformation 2 in practice, find image of W image w back-projects to cone that intersects p in W 2 2 * note that image is independent of particular reconstruction * 100

Affine to metric given P [M m] ω possible transformation from affine to metric is H A 0-1 0 1 AA T T M ωm 1 (cholesky factorisation) proof: P ω M * M -1 ω PH M -1 M M -1 M -T [MA m] T M AA MAA T T M T * I 0 0 0 101

Orthogonality vanishing points corresponding to orthogonal directions T v ωv2 1 0 vanishing line and vanishing point corresponding to plane and normal direction l ωv 102

Correspondence and RANSAC Algorithm.

Correspondence Search 104

Feature Matching To match the points in one image to the points in the other image by exhaustive search(to match one point in one image to all the points in the other image) is a difficult and long process so some constraints are applied. As geometric constraint to minimize the search area for correspondence. The geometric constraints is provided by the epipolar geometry 105

Harris Detector: Mathematics Change of intensity for the shift [x,y]: E( x, y) w( u, v)[ I( x u, y v) I( x, y)] uv, 2 Window function Shifted intensity Intensity Window function w(u,v) = or 1 in window, 0 outside Gaussian 106

E x y I x y I x x y y 2 (, ) [ (, ) (, )] u x,v y w x I( x, y) I( x, y) [ I x( x, y) I y( x, y)] w y x [ I x( x, y) I y( x, y)] w y 2 2 ( I x( x, y)) I x( x, y) I y( x, y) w w x y x 2 I x( x, y) I y( x, y) ( I y( x, y)) y w w x x yc( x, y) y 2 107

Harris Detector: Mathematics For small shifts [u,v] we have a bilinear approximation: u E( u, v) u, v M v where M is a 22 matrix computed from image derivatives: M 2 I x I xi y w( x, y) 2 xy, I xi y I y 108

( max ) -1/2 ( min) -1/2 Harris Detector: Mathematics Intensity change in shifting window: eigenvalue analysis u E( u, v) u, v M v 1, 2 eigenvalues of M If we try every possible orientation n, the max. change in intensity is 2 Ellipse E(u,v) = const 109

Harris Detector: Mathematics Classification of image points using eigenvalues of M: 2 Edge 2 >> 1 Corner 1 and 2 are large, 1 ~ 2 ; E increases in all directions 1 and 2 are small; E is almost constant in all directions Flat region Edge 1 >> 2 1 110

Harris Detector: Mathematics Measure of corner response: trace 2 R det M k M det M trace M 1 2 1 2 (k empirical constant, k = 0.04-0.06) 111

Harris Detector: Mathematics R depends only on eigenvalues of M R is large for a corner R is negative with large magnitude for an edge R is small for a flat region 2 Edge R < 0 Corner R > 0 Flat R small Edge R < 0 1 112

Harris Detector The Algorithm: Find points with large corner response function R Take the points of local maxima of R 113

Harris Detector : Workflow 114

Compute corner response R Harris Detector : Workflow 115

Find points with large corner response: R>threshold Harris Detector : Workflow 116

Take only the points of local maxima of R Harris Detector : Workflow 117

Harris Detector : Workflow 118

Correlation for Correspondence Search i 0,0 j 7,7 Left image: 1. From the left feature point image we select one feature point. 2. Draw the window(n*n) around, with feature point in the center. 3. Calculate the normalized window using the formula s 1 N N i1 j1 w 1 ( i, j)* w 1 ( i, j) N is the size of window w 1( nor) w ( i, j) 1 ( i, j) s 119 1

Correlation Algorithm Select one feature point from the first image. Draw the window across it(7*7). Normalize the window using the given formula. s w ( i, j)* w ( i, j) N is the size of window 1 1 1 i1 j1 w 1( nor) N N ( i, j) w1 ( i, j) s 1 Find the feature in the right image, that are to be considered in the first image,(this should be done by some distance thresholding) After finding the feature point the window of same dimension should be selected in the second image. The normalized correlstion measure should be calculated using the formula N N s w ( i, j)* w ( i, j) 2 1 2 i1 j1 cormat N N i1 j1 s 2 w ( i, j)* w ( i, j) 2 2 120

RANSAC 121

RANSAC Select sample of m points at random 122

RANSAC Select sample of m points at random Calculate model parameters that fit the data in the sample 123

RANSAC Select sample of m points at random Calculate model parameters that fit the data in the sample Calculate error function for each data point 124

RANSAC Select sample of m points at random Calculate model parameters that fit the data in the sample Calculate error function for each data point Select data that support current hypothesis 125

RANSAC Select sample of m points at random Calculate model parameters that fit the data in the sample Calculate error function for each data point Select data that support current hypothesis Repeat sampling 126

RANSAC Select sample of m points at random Calculate model parameters that fit the data in the sample Calculate error function for each data point Select data that support current hypothesis Repeat sampling 127

RANSAC ALL-INLIER SAMPLE RANSAC time complexity k number of samples drawn N number of data points t M time to compute a single model m S average number of models per sample 128

Feature-space outliner rejection Can we now compute H from the blue points? No! Still too many outliers What can we do?

Matching features What do we do about the bad matches?

RAndom SAmple Consensus Select one match, count inliers

RAndom SAmple Consensus Select one match, count inliers

Least squares fit Find average translation vector

RANSAC for estimating homography RANSAC loop: 1. Select four feature pairs (at random) 2. Compute homography H (exact) 3. Compute inliers where SSD(p i, H p i) < ε 4. Keep largest set of inliers 5. Re-compute least-squares H estimate on all of the inliers

RANSAC for Fundamental Matrix Step 1. Extract features Step 2. Compute a set of potential matches Step 3. do Step 3.1 select minimal sample (i.e. 7 matches) Step 3.2 compute solution(s) for F (verify hypothesis) Step 3.3 determine inliers until (#inliers,#samples)<95% Step 4. Compute F based on all inliers Step 5. Look for additional matches Step 6. Refine F based on all correct matches (generate hypothesis)

RANSAC 136

RANSAC 137

Example: Mosaicking with homographies www.cs.cmu.edu/~dellaert/mosaicking 138

Recognizing Panoramas 139

Recognizing Panoramas 140

THANK YOU!! 141

9/28 10/5 10/12 10/19 Liu Chang photosynth create account Take pictures Construct photosynth Markus Photomodeller Select one small object Make 3d model data Yang Yun PhotoCity Select one small object Make 3d model data Wang Ping PhotoCity Presentation file How to install & use Cygwin, bunder, Matlab CC Liu Pengxin Zhu Zi Jian 142 Wang Yuo Remove Perspective distortion Load image Choose feature points SIFT & ASIFT OpenCV CC PhotoTourism ARToolkit Camera Calibration PhotoTourism Matlab CC OpenCV CC Search others photosynth works Select one point in DSU, take pictures Projector Price? Matlab CC Lab Image Data SIFT Fundamental Matrix Remove PT OpenCV CC PT PhotoTourism ARToolkit Camera Calibration Fundamental Matrix M C C led lamp

Topics in Image Processing

144 2014-04-08

145

146

147 2014-04-08

148 2014-04-08

illisis 149

LYYN 150

151 2014-04-08

faceapi http://www.seeingmachines.com/product/faceapi/ 152

Total Immersion - D Fusion Pro http://www.t-immersion.com/products/dfusionsuite/dfusion-pro 153

D Fusion Pro - Markless Tracking 154

155

Monitoring System 156

Scenarios Clear Vision - Denoising Motion Deblurring SuperResolution Panoramic View Background Modeling 157

Around View Monitor 158

Projective Transformations

Projective 2D Geometry 160

Projective transformations Definition: A projectivity is an invertible mapping h from P 2 to itself such that three points x 1,x 2,x 3 lie on the same line if and only if h(x 1 ),h(x 2 ),h(x 3 ) do. Theorem: A mapping h:p 2 P 2 is a projectivity if and only if there exist a non-singular 3x3 matrix H such that for any point in P 2 reprented by a vector x it is true that h(x)=hx Definition: Projective transformation x' x' x' 1 2 3 h h h 11 21 31 h h h 12 22 32 h h h 13 23 33 x x x 1 2 3 or x' H x 8DOF projectivity=collineation=projective transformation=homography 161

Mapping between planes central projection may be expressed by x =Hx (application of theorem) 162

Removing projective distortion select four points in a plane with know coordinates x' 1 h11x h12y h13 x' 2 h21x h22y h x' y' x' h x h y h x' h x h y h x' y' 3 31 32 h31x h32y h33 h11x h12y h13 h31x h32y h33 h21x h22y h23 33 3 31 32 (linear in h ij ) (2 constraints/point, 8DOF 4 points needed) 23 33 163