Image stitching. Digital Visual Effects Yung-Yu Chuang. with slides by Richard Szeliski, Steve Seitz, Matthew Brown and Vaclav Hlavac

Size: px
Start display at page:

Download "Image stitching. Digital Visual Effects Yung-Yu Chuang. with slides by Richard Szeliski, Steve Seitz, Matthew Brown and Vaclav Hlavac"

Transcription

1 Image stitching Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski, Steve Seitz, Matthew Brown and Vaclav Hlavac

2 Image stitching Stitching = alignment + blending geometrical registration photometric registration

3 Applications of image stitching Video stabilization Video summarization i Video compression Video matting Panorama creation

4 Video summarization

5 Video compression

6 Object removal input video

7 Object removal remove foreground

8 Object removal estimate background

9 Object removal background estimation

10 Panorama creation

11 Why panorama? Are you getting the whole picture? Compact Camera FOV = 50 x 35

12 Why panorama? Are you getting the whole picture? Compact Camera FOV = 50 x 35 Human FOV = 200 x 135

13 Why panorama? Are you getting the whole picture? Compact Camera FOV = 50 x 35 Human FOV = 200 x 135 Panoramic Mosaic = 360 x 180

14 Panorama examples Like HDR, it is a topic of computational photography, seeking ways to build a better camera mostly in software. Most consumer cameras have a panorama mode Mars: Earth:

15 What can be globally aligned? In image stitching, we seek for a matrix to globally warp one image into another. Are any two images of the same scene can be aligned this way? Images captured with the same center of projection A planar scene or far-away scene

16 A pencil of rays contains all views real camera synthetic camera Can generate any synthetic camera view g y y as long as it has the same center of projection!

17 Mosaic as an image reprojection mosaic projection plane The images are reprojected onto a common plane The mosaic is formed on this plane Mosaic is a synthetic wide-angle camera

18 Changing camera center Does it still work? synthetic PP PP1 PP2

19 Planar scene (or a faraway one) PP1 PP3 PP2 PP3 is a projection plane of both centers of projection, so we are OK! This is how big aerial photographs are made

20 Motion models Parametric models as the assumptions on the relation between two images.

21 2D Motion models

22 Motion models Translation Affine Perspective 3D rotation 2 unknowns 6 unknowns 8 unknowns 3 unknowns

23 A case study: cylindrical panorama What if you want a 360 field of view? mosaic projection cylinder

24 Cylindrical panoramas Steps Reproject each image onto a cylinder Blend Output the resulting mosaic

25 Cylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical i l coordinate 3. Compute pairwise alignments 4. Fix up the end-to-end alignment 5. Blending 6. Crop the result and import into a viewer It is required to do radial distortion correction for better stitching results!

26 Taking pictures Kaidan panoramic tripod head

27 Translation model Try to align this in PaintShop Pro

28 Where should the synthetic camera be real camera synthetic camera The projection plan of some camera Onto a cylinder

29 Cylindrical projection Adopted from

30 Cylindrical projection

31 Cylindrical projection Adopted from

32 Cylindrical projection unwrapped cylinder y x θ f x

33 Cylindrical projection y x unwrapped cylinder y f θ x z

34 Cylindrical projection y x unwrapped cylinder y f z s=f gives less distortion x

35 Cylindrical reprojection top-down view Focal length the dirty secret Image 384x300 f = 180 (pixels) f = 280 f = 380

36 A simple method for estimating f w p f d Or, you can use other software, such as AutoStich, to help.

37 Input images

38 Cylindrical warping

39 Blending Why blending: parallax, lens distortion, scene motion, exposure difference

40 Blending

41 Blending

42 Blending

43 Assembling the panorama Stitch pairs together, blend, then crop

44 Problem: Drift Error accumulation small errors accumulate over time

45 Problem: Drift (x 1,y 1 ) (x n,y n ) Solution copy of first image add another copy of first image at the end there are a bunch of ways to solve this problem add displacement of (y 1 y n )/(n -1) to each image after the first compute a global warp: y = y + ax run a big optimization problem, incorporating this constraint best solution, but more complicated known as bundle adjustment

46 End-to-end alignment and crop

47 Viewer: panorama example:

48 Viewer: texture mapped model example:

49 Cylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical i l coordinate 3. Compute pairwise alignments 4. Fix up the end-to-end alignment 5. Blending 6. Crop the result and import into a viewer

50 Determine pairwise alignment? Feature-based methods: only use feature points to estimate parameters We will study the Recognising panorama paper published in ICCV 2003 Run SIFT (or other feature algorithms) for each Run SIFT (or other feature algorithms) for each image, find feature matches.

51 Determine pairwise alignment p =Mp, where M is a transformation matrix, p and p are feature matches It is possible to use more complicated models such as affine or perspective For example, assume M is a 2x2 matrix x' m11 m12 x y' m 21 m 22 y Find M with the least square error n Mp p' i1 2

52 Determine pairwise alignment x m m x ' ' x m y m x y m m y ' ' y m y m x Overdetermined system ' 0 0 ' y x m y x y x ' x m m y x ' n n n x m m y x ' 0 0 n n n y y x

53 Normal equation Given an overdetermined system Ax b the normal equation is that which minimizes the sum of the square differences between left and right sides A T Ax A T b Why?

54 Normal equation 2 ) ( b Ax x E ) ( b a a m b x a a : : : : : : : 1 m b x : : : n nm n b a a... 1 nxm nequations m variables nxm, n equations, m variables

55 Normal equation m j j m b x a x a 1 1 j j j j j j b x a b x a : : : i m j j ij i m j j ij b x a b x a 1 1 : b Ax m j m j b 1 1 : : : n m j j nj n j j nj b x a b x a 1 1 n i m j ij b x a E 2 2 ) ( b Ax x i i j j ij b x a E 1 1 ) ( b Ax x

56 Normal equation 0 2 Ax E ( x ) b E x 1 i n m aij x j bi 1 j 1 n m 2 aij x j bi ai1 i1 j1 n m n a a x 2 a i1 ij j i1 i1 j1 i1 2 b i 2 T T 0 2( A Ax A b ) x E AA Ax A b T T

57 Normal equation Ax b 2 T Ax b Ax b T T Ax b Ax b T T T x A b Ax b x x T T A A T T Ax b Ax T Ax x T A T b b T b T T T A b x A b T T x b b E x T 2A Ax 2A T b

58 Determine pairwise alignment p =Mp, where M is a transformation matrix, p and p are feature matches For translation model, it is easier. E n ' 2 ' 2 m1 xi xi m2 yi yi i1 0 E m 1 What if the match is false? Avoid impact of outliers.

59 RANSAC RANSAC = Random Sample Consensus An algorithm for robust fitting i of models in the presence of many data outliers Compare to robust statistics Given N data points x i, assume that mjority of them are generated from a model with parameters, try to recover.

60 RANSAC algorithm Run k times: How many times? How big? Smaller is better (1) draw n samples randomly (2) fit parameters with these n samples (3) for each of other N-n points, calculate its distance to the fitted model, count the number of inlier points, c Output t with the largest c How to define? Depends on the problem.

61 How to determine k p: probability of real inliers P: probability of success after k trials P 1 (1 p ) n k n samples are all inliers a failure k log(1 P ) log(1 p ) failure after k trials n p k for P= n

62 Example: line fitting

63 Example: line fitting n=2

64 Model fitting

65 Measure distances

66 Count inliers c=3

67 Another trial c=3

68 The best model c=15

69 RANSAC for Homography

70 RANSAC for Homography

71 RANSAC for Homography

72 Applications of panorama in VFX Background plates Image-based lighting i

73 Troy (image-based lighting) id=2195&page=4

74 Spiderman 2 (background plate)

75

76 Cylindrical projection Map 3D point (X,Y,Z) onto a unit cylinder Y Z X unit cylinder Convert to cylindrical coordinates Convert to cylindrical image coordinates unwrapped cylinder cylindrical image

77 3D 2D perspective projection (X c,y c c,,z c c) ) f u c u

78 Reference Richard Szeliski, Image Alignment and Stitching, unpublished draft, R. Szeliski and H.-Y. Shum. Creating full view panoramic image mosaics and texture-mapped models, SIGGRAPH 1997, pp M. Brown, D. G. Lowe, Recognising Panoramas, ICCV 2003.

79 Direct vs feature-based Direct methods use all information and can be very accurate, but they depend on the fragile brightness constancy assumption Iterative approaches require initialization Not robust to illumination change and noise images In early days, direct method is better. Feature based methods are now more robust and potentially faster Even better, it can recognize panorama without initialization

80 TODO Bundle adjustment LM method Direct method vs feature-based method Frame-rate rate image alignment for stabilization Rick s CGA 1995 paper? LM method

81 Project #2 Image stitching camera availability Ti Tripod? rses/cse590ss/currentqtr/projects.htm norama.html

82 blending Alpha-blending Photomontage Poisson blending Adelson s pyramid blending Hdr?

83 3D interpretation p t R K u p Y X t r r r t r r r f f x Z Y t r r r t r r r f z y u

84 Cylindrical warping Given focal length f and image center (x c,y c ) (X,Y,Z) Y Y (sin,h,cos) Z X

85 Cylindrical projection Map 3D point (X,Y,Z) onto cylinder Y Z X unit cylinder Convert to cylindrical coordinates Convert to cylindrical image coordinates unwrapped cylinder cylindrical image

86 Cylindrical reprojection How to map from a cylinder to a planar image? Apply camera projection matrix w = image width, h = image height Y Z X side view Convert to image coordinates divide by third coordinate (w) top-down view image coords

87 Cylindrical projection Y Z X unit cylinder

88 Levenberg-Marquardt Method

89 Alignment a rotation of the camera is a translation of the cylinder! cylinder! y x I y x J I I I I 2 ), ( ), ( y y x x y y x y x y x y x x y x I y x J I y x I y x J I v u I I I I I I, 2,, ), ( ), ( ), ( ), ( y x y x y x,,,

90 LucasKanadeStep void LucasKanadeStep(CByteImage& img1, CByteImage& img2, float t[2]) { // Transform the image Translation(img2, img2t, t); // Compute the gradients and summed error by comparing img1 and img2t double A[2][2], b[2]; for (int y = 1; y < height-1; y++) { // ignore borders for (int x = 1; x < width-1; x++) { // If both have full alphas, then compute and accumulate the error double e = img2t.pixel(x, y, k) - img1.pixel (x, y, k); // Accumulate the matrix entries double gx = 0.5*(img2t.Pixel(x+1, ( y, k) -img2t.pixel(x-1, y, k)); double gy = 0.5*(img2t.Pixel(x, y+1, k) - img2t.pixel(x, y-1, k)); A[0][0] ] += gx*gx; g A[0][1] ] += gx*gy; gy; A[1][0] += gx*gy; A[1][1] += gy*gy; } } b[0] += e*gx; b[1] += e*gy;

91 LucasKanadeStep (cont.) // Solve for the update At=b and update the vector double det = 1.0 / (A[0][0]*A[1][1] - A[1][0]*A[1][0]); } t[0] += (A[1][1]*b[0] - A[1][0]*b[1]) * det; t[1] += (A[0][0]*b[1] - A[1][0]*b[0]) * det;

92 PyramidLucasKanade void PyramidalLucasKanade(CByteImage& img1, CByteImage& img2, float t[2], int nlevels, int nlucaskanadesteps) { CBytePyramid p1(img1); // Form the two pyramids CBytePyramid p2(img2); // Process in a coarse-to-fine hierarchy for (int l = nlevels-1; l >= 0; l--) { t[0] /= (1 << l); // scale the t vector t[1] /= (1 << l); CByteImage& i1 = p1[l]; CByteImage& i2 = p2[l]; } } for (int k = 0; k < nlucaskanadesteps; k++) LucasKanadeStep(i1, i2, t); t[0] *= (1 << l); // restore the full scaling t[1] *= (1 << l);

93 Gaussian pyramid

94 2D Motion models translation: x = x + t x = (x,y) rotation: x = R x + t similarity: affine: x = s R x + t x = A x + t perspective: x H x x = (x,y,1) y (x is a homogeneous coordinate) These all form a nested group (closed under These all form a nested group (closed under composition w/ inv.)

95 Video matting alpha matte

96 Recognising Panoramas 1D Rotations () Ordering matching images

97 Recognising Panoramas 1D Rotations () Ordering matching images

98 Recognising Panoramas 1D Rotations () Ordering matching images

99 Recognising Panoramas 1D Rotations () Ordering matching images 2D Rotations (q, f) Ordering matching images

100 Recognising Panoramas 1D Rotations () Ordering matching images 2D Rotations (q, f) Ordering matching images

101 Recognising Panoramas 1D Rotations () Ordering matching images 2D Rotations (q, f) Ordering matching images

102 Probabilistic model for verification Compare probability that this set of RANSAC inliers/outliers was generated by a correct/false image match Choosing values for p 1, p 0 and p min

103 Recognising Panoramas

104 Overview SIFT Feature Matching Image Matching Bundle Adjustment Multi-band Blending

105 Nearest Neighbour Matching Find k-nn for each feature k number of overlapping images (we use k = 4) Use k-d tree k-d tree recursively bi-partitions data at mean in the dimension of maximum variance Approximate nearest neighbours found in O(nlogn)

106 Overview SIFT Feature Matching Image Matching For each image, use RANSAC to select inlier features from 6 images with most feature matches Bundle Adjustment Multi-band Blending

107 Finding the panoramas

108 Finding the panoramas

109 Finding the panoramas

110 Finding the panoramas

111 Overview SIFT Feature Matching Image Matching Bundle Adjustment Multi-band Blending

112 Homography for Rotation Parameterise each camera by rotation and focal length This gives pairwise homographies

113 Error function Sum of squared projection errors n = #images I(i) = set of image matches to image i F(i, j) = set of feature matches between images i,j r ijk = residual of k th feature match between images i,j Robust error function

114 Overview SIFT Feature Matching Image Matching Bundle Adjustment Multi-band Blending

115 Multi-band Blending Burt & Adelson 1983 Blend frequency bands over range

116 2-band Blending Low frequency ( > 2 pixels) High frequency ( < 2 pixels)

117 Linear Blending

118 2-band Blending

119 Results

120 Distortion No distortion Pin cushion Barrel Radial distortion of the image Caused by imperfect lenses Deviations are most noticeable for rays that pass through the edge of the lens

121 Radial correction Correct for bending in wide field of view lenses

Image stitching. Announcements. Outline. Image stitching

Image stitching. Announcements. Outline. Image stitching Announcements Image stitching Project #1 was due yesterday. Project #2 handout will be available on the web later tomorrow. I will set up a webpage for artifact voting soon. Digital Visual Effects, Spring

More information

Image stitching. Digital Visual Effects Yung-Yu Chuang. with slides by Richard Szeliski, Steve Seitz, Matthew Brown and Vaclav Hlavac

Image stitching. Digital Visual Effects Yung-Yu Chuang. with slides by Richard Szeliski, Steve Seitz, Matthew Brown and Vaclav Hlavac Image stitching Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski, Steve Seitz, Matthew Brown and Vaclav Hlavac Image stitching Stitching = alignment + blending geometrical registration

More information

Mosaics. Today s Readings

Mosaics. Today s Readings Mosaics VR Seattle: http://www.vrseattle.com/ Full screen panoramas (cubic): http://www.panoramas.dk/ Mars: http://www.panoramas.dk/fullscreen3/f2_mars97.html Today s Readings Szeliski and Shum paper (sections

More information

Today s lecture. Image Alignment and Stitching. Readings. Motion models

Today s lecture. Image Alignment and Stitching. Readings. Motion models Today s lecture Image Alignment and Stitching Computer Vision CSE576, Spring 2005 Richard Szeliski Image alignment and stitching motion models cylindrical and spherical warping point-based alignment global

More information

Feature Matching and RANSAC

Feature Matching and RANSAC Feature Matching and RANSAC Recognising Panoramas. [M. Brown and D. Lowe,ICCV 2003] [Brown, Szeliski, Winder, CVPR 2005] with a lot of slides stolen from Steve Seitz, Rick Szeliski, A. Efros Introduction

More information

Object Recognition with Invariant Features

Object Recognition with Invariant Features Object Recognition with Invariant Features Definition: Identify objects or scenes and determine their pose and model parameters Applications Industrial automation and inspection Mobile robots, toys, user

More information

Image Stitching. Slides from Rick Szeliski, Steve Seitz, Derek Hoiem, Ira Kemelmacher, Ali Farhadi

Image Stitching. Slides from Rick Szeliski, Steve Seitz, Derek Hoiem, Ira Kemelmacher, Ali Farhadi Image Stitching Slides from Rick Szeliski, Steve Seitz, Derek Hoiem, Ira Kemelmacher, Ali Farhadi Combine two or more overlapping images to make one larger image Add example Slide credit: Vaibhav Vaish

More information

Announcements. Mosaics. How to do it? Image Mosaics

Announcements. Mosaics. How to do it? Image Mosaics Announcements Mosaics Project artifact voting Project 2 out today (help session at end of class) http://www.destination36.com/start.htm http://www.vrseattle.com/html/vrview.php?cat_id=&vrs_id=vrs38 Today

More information

Announcements. Mosaics. Image Mosaics. How to do it? Basic Procedure Take a sequence of images from the same position =

Announcements. Mosaics. Image Mosaics. How to do it? Basic Procedure Take a sequence of images from the same position = Announcements Project 2 out today panorama signup help session at end of class Today mosaic recap blending Mosaics Full screen panoramas (cubic): http://www.panoramas.dk/ Mars: http://www.panoramas.dk/fullscreen3/f2_mars97.html

More information

CS6670: Computer Vision

CS6670: Computer Vision CS6670: Computer Vision Noah Snavely Lecture 7: Image Alignment and Panoramas What s inside your fridge? http://www.cs.washington.edu/education/courses/cse590ss/01wi/ Projection matrix intrinsics projection

More information

More Mosaic Madness. CS194: Image Manipulation & Computational Photography. Steve Seitz and Rick Szeliski. Jeffrey Martin (jeffrey-martin.

More Mosaic Madness. CS194: Image Manipulation & Computational Photography. Steve Seitz and Rick Szeliski. Jeffrey Martin (jeffrey-martin. More Mosaic Madness Jeffrey Martin (jeffrey-martin.com) CS194: Image Manipulation & Computational Photography with a lot of slides stolen from Alexei Efros, UC Berkeley, Fall 2018 Steve Seitz and Rick

More information

6.098 Digital and Computational Photography Advanced Computational Photography. Panoramas. Bill Freeman Frédo Durand MIT - EECS

6.098 Digital and Computational Photography Advanced Computational Photography. Panoramas. Bill Freeman Frédo Durand MIT - EECS 6.098 Digital and Computational Photography 6.882 Advanced Computational Photography Panoramas Bill Freeman Frédo Durand MIT - EECS Lots of slides stolen from Alyosha Efros, who stole them from Steve Seitz

More information

Image-based Modeling and Rendering: 8. Image Transformation and Panorama

Image-based Modeling and Rendering: 8. Image Transformation and Panorama Image-based Modeling and Rendering: 8. Image Transformation and Panorama I-Chen Lin, Assistant Professor Dept. of CS, National Chiao Tung Univ, Taiwan Outline Image transformation How to represent the

More information

Panoramas. Why Mosaic? Why Mosaic? Mosaics: stitching images together. Why Mosaic? Olivier Gondry. Bill Freeman Frédo Durand MIT - EECS

Panoramas. Why Mosaic? Why Mosaic? Mosaics: stitching images together. Why Mosaic? Olivier Gondry. Bill Freeman Frédo Durand MIT - EECS Olivier Gondry 6.098 Digital and Computational Photography 6.882 Advanced Computational Photography Panoramas Director of music video and commercial Special effect specialist (Morphing, rotoscoping) Today

More information

Image Warping and Mosacing

Image Warping and Mosacing Image Warping and Mosacing 15-463: Rendering and Image Processing Alexei Efros with a lot of slides stolen from Steve Seitz and Rick Szeliski Today Mosacs Image Warping Homographies Programming Assignment

More information

Image warping and stitching

Image warping and stitching Image warping and stitching May 4 th, 2017 Yong Jae Lee UC Davis Last time Interactive segmentation Feature-based alignment 2D transformations Affine fit RANSAC 2 Alignment problem In alignment, we will

More information

Image warping and stitching

Image warping and stitching Image warping and stitching May 5 th, 2015 Yong Jae Lee UC Davis PS2 due next Friday Announcements 2 Last time Interactive segmentation Feature-based alignment 2D transformations Affine fit RANSAC 3 Alignment

More information

CSE 527: Introduction to Computer Vision

CSE 527: Introduction to Computer Vision CSE 527: Introduction to Computer Vision Week 5 - Class 1: Matching, Stitching, Registration September 26th, 2017 ??? Recap Today Feature Matching Image Alignment Panoramas HW2! Feature Matches Feature

More information

Homographies and RANSAC

Homographies and RANSAC Homographies and RANSAC Computer vision 6.869 Bill Freeman and Antonio Torralba March 30, 2011 Homographies and RANSAC Homographies RANSAC Building panoramas Phototourism 2 Depth-based ambiguity of position

More information

Chapter 3 Image Registration. Chapter 3 Image Registration

Chapter 3 Image Registration. Chapter 3 Image Registration Chapter 3 Image Registration Distributed Algorithms for Introduction (1) Definition: Image Registration Input: 2 images of the same scene but taken from different perspectives Goal: Identify transformation

More information

EE795: Computer Vision and Intelligent Systems

EE795: Computer Vision and Intelligent Systems EE795: Computer Vision and Intelligent Systems Spring 2012 TTh 17:30-18:45 FDH 204 Lecture 12 130228 http://www.ee.unlv.edu/~b1morris/ecg795/ 2 Outline Review Panoramas, Mosaics, Stitching Two View Geometry

More information

Midterm Examination CS 534: Computational Photography

Midterm Examination CS 534: Computational Photography Midterm Examination CS 534: Computational Photography November 3, 2016 NAME: Problem Score Max Score 1 6 2 8 3 9 4 12 5 4 6 13 7 7 8 6 9 9 10 6 11 14 12 6 Total 100 1 of 8 1. [6] (a) [3] What camera setting(s)

More information

Image warping and stitching

Image warping and stitching Image warping and stitching Thurs Oct 15 Last time Feature-based alignment 2D transformations Affine fit RANSAC 1 Robust feature-based alignment Extract features Compute putative matches Loop: Hypothesize

More information

Automatic Image Alignment

Automatic Image Alignment Automatic Image Alignment Mike Nese with a lot of slides stolen from Steve Seitz and Rick Szeliski 15-463: Computational Photography Alexei Efros, CMU, Fall 2010 Live Homography DEMO Check out panoramio.com

More information

Automatic Image Alignment

Automatic Image Alignment Automatic Image Alignment with a lot of slides stolen from Steve Seitz and Rick Szeliski Mike Nese CS194: Image Manipulation & Computational Photography Alexei Efros, UC Berkeley, Fall 2018 Live Homography

More information

360 Full View Spherical Mosaic

360 Full View Spherical Mosaic 360 Full View Spherical Mosaic Huang Wenfan Huang Yehui Rong Nan U017865B U017844X U018274R Objective Full spherical mosaic 360 x 180. All images are taken with camera mounted on a tripod. Registration

More information

Targil 10 : Why Mosaic? Why is this a challenge? Exposure differences Scene illumination Miss-registration Moving objects

Targil 10 : Why Mosaic? Why is this a challenge? Exposure differences Scene illumination Miss-registration Moving objects Why Mosaic? Are you getting the whole picture? Compact Camera FOV = 5 x 35 Targil : Panoramas - Stitching and Blending Some slides from Alexei Efros 2 Slide from Brown & Lowe Why Mosaic? Are you getting

More information

Agenda. Rotations. Camera models. Camera calibration. Homographies

Agenda. Rotations. Camera models. Camera calibration. Homographies Agenda Rotations Camera models Camera calibration Homographies D Rotations R Y = Z r r r r r r r r r Y Z Think of as change of basis where ri = r(i,:) are orthonormal basis vectors r rotated coordinate

More information

Agenda. Rotations. Camera calibration. Homography. Ransac

Agenda. Rotations. Camera calibration. Homography. Ransac Agenda Rotations Camera calibration Homography Ransac Geometric Transformations y x Transformation Matrix # DoF Preserves Icon translation rigid (Euclidean) similarity affine projective h I t h R t h sr

More information

Index. 3D reconstruction, point algorithm, point algorithm, point algorithm, point algorithm, 253

Index. 3D reconstruction, point algorithm, point algorithm, point algorithm, point algorithm, 253 Index 3D reconstruction, 123 5+1-point algorithm, 274 5-point algorithm, 260 7-point algorithm, 255 8-point algorithm, 253 affine point, 43 affine transformation, 55 affine transformation group, 55 affine

More information

CS4670: Computer Vision

CS4670: Computer Vision CS467: Computer Vision Noah Snavely Lecture 13: Projection, Part 2 Perspective study of a vase by Paolo Uccello Szeliski 2.1.3-2.1.6 Reading Announcements Project 2a due Friday, 8:59pm Project 2b out Friday

More information

EE795: Computer Vision and Intelligent Systems

EE795: Computer Vision and Intelligent Systems EE795: Computer Vision and Intelligent Systems Spring 2012 TTh 17:30-18:45 FDH 204 Lecture 10 130221 http://www.ee.unlv.edu/~b1morris/ecg795/ 2 Outline Review Canny Edge Detector Hough Transform Feature-Based

More information

Panoramic Image Stitching

Panoramic Image Stitching Mcgill University Panoramic Image Stitching by Kai Wang Pengbo Li A report submitted in fulfillment for the COMP 558 Final project in the Faculty of Computer Science April 2013 Mcgill University Abstract

More information

Automatic Panoramic Image Stitching. Dr. Matthew Brown, University of Bath

Automatic Panoramic Image Stitching. Dr. Matthew Brown, University of Bath Automatic Panoramic Image Stitching Dr. Matthew Brown, University of Bath Automatic 2D Stitching The old days: panoramic stitchers were limited to a 1-D sweep 2005: 2-D stitchers use object recognition

More information

Index. 3D reconstruction, point algorithm, point algorithm, point algorithm, point algorithm, 263

Index. 3D reconstruction, point algorithm, point algorithm, point algorithm, point algorithm, 263 Index 3D reconstruction, 125 5+1-point algorithm, 284 5-point algorithm, 270 7-point algorithm, 265 8-point algorithm, 263 affine point, 45 affine transformation, 57 affine transformation group, 57 affine

More information

Recap. DoF Constraint Solver. translation. affine. homography. 3D rotation

Recap. DoF Constraint Solver. translation. affine. homography. 3D rotation Image Blending Recap DoF Constraint Solver translation affine homography 3D rotation Recap DoF Constraint Solver translation 2 affine homography 3D rotation Recap DoF Constraint Solver translation 2 affine

More information

Camera Calibration. COS 429 Princeton University

Camera Calibration. COS 429 Princeton University Camera Calibration COS 429 Princeton University Point Correspondences What can you figure out from point correspondences? Noah Snavely Point Correspondences X 1 X 4 X 3 X 2 X 5 X 6 X 7 p 1,1 p 1,2 p 1,3

More information

Image Warping. Many slides from Alyosha Efros + Steve Seitz. Photo by Sean Carroll

Image Warping. Many slides from Alyosha Efros + Steve Seitz. Photo by Sean Carroll Image Warping Man slides from Alosha Efros + Steve Seitz Photo b Sean Carroll Morphing Blend from one object to other with a series of local transformations Image Transformations image filtering: change

More information

N-Views (1) Homographies and Projection

N-Views (1) Homographies and Projection CS 4495 Computer Vision N-Views (1) Homographies and Projection Aaron Bobick School of Interactive Computing Administrivia PS 2: Get SDD and Normalized Correlation working for a given windows size say

More information

CS664 Lecture #19: Layers, RANSAC, panoramas, epipolar geometry

CS664 Lecture #19: Layers, RANSAC, panoramas, epipolar geometry CS664 Lecture #19: Layers, RANSAC, panoramas, epipolar geometry Some material taken from: David Lowe, UBC Jiri Matas, CMP Prague http://cmp.felk.cvut.cz/~matas/papers/presentations/matas_beyondransac_cvprac05.ppt

More information

Global Flow Estimation. Lecture 9

Global Flow Estimation. Lecture 9 Motion Models Image Transformations to relate two images 3D Rigid motion Perspective & Orthographic Transformation Planar Scene Assumption Transformations Translation Rotation Rigid Affine Homography Pseudo

More information

CSE 252B: Computer Vision II

CSE 252B: Computer Vision II CSE 252B: Computer Vision II Lecturer: Serge Belongie Scribes: Jeremy Pollock and Neil Alldrin LECTURE 14 Robust Feature Matching 14.1. Introduction Last lecture we learned how to find interest points

More information

Image Based Rendering. D.A. Forsyth, with slides from John Hart

Image Based Rendering. D.A. Forsyth, with slides from John Hart Image Based Rendering D.A. Forsyth, with slides from John Hart Topics Mosaics translating cameras reveal extra information, break occlusion Optical flow for very small movements of the camera Explicit

More information

Image Features: Local Descriptors. Sanja Fidler CSC420: Intro to Image Understanding 1/ 58

Image Features: Local Descriptors. Sanja Fidler CSC420: Intro to Image Understanding 1/ 58 Image Features: Local Descriptors Sanja Fidler CSC420: Intro to Image Understanding 1/ 58 [Source: K. Grauman] Sanja Fidler CSC420: Intro to Image Understanding 2/ 58 Local Features Detection: Identify

More information

Feature Based Registration - Image Alignment

Feature Based Registration - Image Alignment Feature Based Registration - Image Alignment Image Registration Image registration is the process of estimating an optimal transformation between two or more images. Many slides from Alexei Efros http://graphics.cs.cmu.edu/courses/15-463/2007_fall/463.html

More information

Stitching and Blending

Stitching and Blending Stitching and Blending Kari Pulli VP Computational Imaging Light First project Build your own (basic) programs panorama HDR (really, exposure fusion) The key components register images so their features

More information

Instance-level recognition

Instance-level recognition Instance-level recognition 1) Local invariant features 2) Matching and recognition with local features 3) Efficient visual search 4) Very large scale indexing Matching of descriptors Matching and 3D reconstruction

More information

Introduction to Image Processing and Computer Vision. -- Panoramas and Blending --

Introduction to Image Processing and Computer Vision. -- Panoramas and Blending -- Introduction to Image Processing and Computer Vision -- Panoramas and Blending -- Winter 2013/14 Ivo Ihrke Panoramas Mosaics and Panoramas - Outline - Perspective Panoramas - Hardware-Based - Software-Based

More information

Video Mosaics for Virtual Environments, R. Szeliski. Review by: Christopher Rasmussen

Video Mosaics for Virtual Environments, R. Szeliski. Review by: Christopher Rasmussen Video Mosaics for Virtual Environments, R. Szeliski Review by: Christopher Rasmussen September 19, 2002 Announcements Homework due by midnight Next homework will be assigned Tuesday, due following Tuesday.

More information

Overview. Augmented reality and applications Marker-based augmented reality. Camera model. Binary markers Textured planar markers

Overview. Augmented reality and applications Marker-based augmented reality. Camera model. Binary markers Textured planar markers Augmented reality Overview Augmented reality and applications Marker-based augmented reality Binary markers Textured planar markers Camera model Homography Direct Linear Transformation What is augmented

More information

Global Flow Estimation. Lecture 9

Global Flow Estimation. Lecture 9 Global Flow Estimation Lecture 9 Global Motion Estimate motion using all pixels in the image. Parametric flow gives an equation, which describes optical flow for each pixel. Affine Projective Global motion

More information

Instance-level recognition

Instance-level recognition Instance-level recognition 1) Local invariant features 2) Matching and recognition with local features 3) Efficient visual search 4) Very large scale indexing Matching of descriptors Matching and 3D reconstruction

More information

Automatic Image Alignment (feature-based)

Automatic Image Alignment (feature-based) Automatic Image Alignment (feature-based) Mike Nese with a lot of slides stolen from Steve Seitz and Rick Szeliski 15-463: Computational Photography Alexei Efros, CMU, Fall 2006 Today s lecture Feature

More information

Scene Modeling for a Single View

Scene Modeling for a Single View Scene Modeling for a Single View René MAGRITTE Portrait d'edward James with a lot of slides stolen from Steve Seitz and David Brogan, 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 Classes

More information

EE795: Computer Vision and Intelligent Systems

EE795: Computer Vision and Intelligent Systems EE795: Computer Vision and Intelligent Systems Spring 2012 TTh 17:30-18:45 FDH 204 Lecture 14 130307 http://www.ee.unlv.edu/~b1morris/ecg795/ 2 Outline Review Stereo Dense Motion Estimation Translational

More information

Feature Detectors and Descriptors: Corners, Lines, etc.

Feature Detectors and Descriptors: Corners, Lines, etc. Feature Detectors and Descriptors: Corners, Lines, etc. Edges vs. Corners Edges = maxima in intensity gradient Edges vs. Corners Corners = lots of variation in direction of gradient in a small neighborhood

More information

Automatic Image Alignment (direct) with a lot of slides stolen from Steve Seitz and Rick Szeliski

Automatic Image Alignment (direct) with a lot of slides stolen from Steve Seitz and Rick Szeliski Automatic Image Alignment (direct) with a lot of slides stolen from Steve Seitz and Rick Szeliski 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 Today Go over Midterm Go over Project #3

More information

CS4670: Computer Vision

CS4670: Computer Vision CS4670: Computer Vision Noah Snavely Lecture 9: Image alignment http://www.wired.com/gadgetlab/2010/07/camera-software-lets-you-see-into-the-past/ Szeliski: Chapter 6.1 Reading All 2D Linear Transformations

More information

Multiview Stereo COSC450. Lecture 8

Multiview Stereo COSC450. Lecture 8 Multiview Stereo COSC450 Lecture 8 Stereo Vision So Far Stereo and epipolar geometry Fundamental matrix captures geometry 8-point algorithm Essential matrix with calibrated cameras 5-point algorithm Intersect

More information

CS 2770: Intro to Computer Vision. Multiple Views. Prof. Adriana Kovashka University of Pittsburgh March 14, 2017

CS 2770: Intro to Computer Vision. Multiple Views. Prof. Adriana Kovashka University of Pittsburgh March 14, 2017 CS 277: Intro to Computer Vision Multiple Views Prof. Adriana Kovashka Universit of Pittsburgh March 4, 27 Plan for toda Affine and projective image transformations Homographies and image mosaics Stereo

More information

CS664 Lecture #16: Image registration, robust statistics, motion

CS664 Lecture #16: Image registration, robust statistics, motion CS664 Lecture #16: Image registration, robust statistics, motion Some material taken from: Alyosha Efros, CMU http://www.cs.cmu.edu/~efros Xenios Papademetris http://noodle.med.yale.edu/~papad/various/papademetris_image_registration.p

More information

Scene Modeling for a Single View

Scene Modeling for a Single View Scene Modeling for a Single View René MAGRITTE Portrait d'edward James CS194: Image Manipulation & Computational Photography with a lot of slides stolen from Alexei Efros, UC Berkeley, Fall 2014 Steve

More information

Photo Tourism: Exploring Photo Collections in 3D

Photo Tourism: Exploring Photo Collections in 3D Photo Tourism: Exploring Photo Collections in 3D SIGGRAPH 2006 Noah Snavely Steven M. Seitz University of Washington Richard Szeliski Microsoft Research 2006 2006 Noah Snavely Noah Snavely Reproduced with

More information

Dense Image-based Motion Estimation Algorithms & Optical Flow

Dense Image-based Motion Estimation Algorithms & Optical Flow Dense mage-based Motion Estimation Algorithms & Optical Flow Video A video is a sequence of frames captured at different times The video data is a function of v time (t) v space (x,y) ntroduction to motion

More information

DD2423 Image Analysis and Computer Vision IMAGE FORMATION. Computational Vision and Active Perception School of Computer Science and Communication

DD2423 Image Analysis and Computer Vision IMAGE FORMATION. Computational Vision and Active Perception School of Computer Science and Communication DD2423 Image Analysis and Computer Vision IMAGE FORMATION Mårten Björkman Computational Vision and Active Perception School of Computer Science and Communication November 8, 2013 1 Image formation Goal:

More information

calibrated coordinates Linear transformation pixel coordinates

calibrated coordinates Linear transformation pixel coordinates 1 calibrated coordinates Linear transformation pixel coordinates 2 Calibration with a rig Uncalibrated epipolar geometry Ambiguities in image formation Stratified reconstruction Autocalibration with partial

More information

Computer Vision. Recap: Smoothing with a Gaussian. Recap: Effect of σ on derivatives. Computer Science Tripos Part II. Dr Christopher Town

Computer Vision. Recap: Smoothing with a Gaussian. Recap: Effect of σ on derivatives. Computer Science Tripos Part II. Dr Christopher Town Recap: Smoothing with a Gaussian Computer Vision Computer Science Tripos Part II Dr Christopher Town Recall: parameter σ is the scale / width / spread of the Gaussian kernel, and controls the amount of

More information

Warping, Morphing and Mosaics

Warping, Morphing and Mosaics Computational Photograph and Video: Warping, Morphing and Mosaics Prof. Marc Pollefes Dr. Gabriel Brostow Toda s schedule Last week s recap Warping Morphing Mosaics Toda s schedule Last week s recap Warping

More information

Invariant Features from Interest Point Groups

Invariant Features from Interest Point Groups Invariant Features from Interest Point Groups Matthew Brown and David Lowe {mbrown lowe}@cs.ubc.ca Department of Computer Science, University of British Columbia, Vancouver, Canada. Abstract This paper

More information

COSC579: Scene Geometry. Jeremy Bolton, PhD Assistant Teaching Professor

COSC579: Scene Geometry. Jeremy Bolton, PhD Assistant Teaching Professor COSC579: Scene Geometry Jeremy Bolton, PhD Assistant Teaching Professor Overview Linear Algebra Review Homogeneous vs non-homogeneous representations Projections and Transformations Scene Geometry The

More information

Final Exam Study Guide

Final Exam Study Guide Final Exam Study Guide Exam Window: 28th April, 12:00am EST to 30th April, 11:59pm EST Description As indicated in class the goal of the exam is to encourage you to review the material from the course.

More information

1-2 Feature-Based Image Mosaicing

1-2 Feature-Based Image Mosaicing MVA'98 IAPR Workshop on Machine Vision Applications, Nov. 17-19, 1998, Makuhari, Chibq Japan 1-2 Feature-Based Image Mosaicing Naoki Chiba, Hiroshi Kano, Minoru Higashihara, Masashi Yasuda, and Masato

More information

Homework #4: Creating Panoramas

Homework #4: Creating Panoramas Homework #4: Creating Panoramas Assigned: March 21 Due: Wednesday, April 6 The goal of this assignment is to write a simple photo panorama creator. You will take four or more photographs and create a panoramic

More information

Image correspondences and structure from motion

Image correspondences and structure from motion Image correspondences and structure from motion http://graphics.cs.cmu.edu/courses/15-463 15-463, 15-663, 15-862 Computational Photography Fall 2017, Lecture 20 Course announcements Homework 5 posted.

More information

Local Feature Detectors

Local Feature Detectors Local Feature Detectors Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr Slides adapted from Cordelia Schmid and David Lowe, CVPR 2003 Tutorial, Matthew Brown,

More information

Introduction to Computer Vision

Introduction to Computer Vision Introduction to Computer Vision Michael J. Black Nov 2009 Perspective projection and affine motion Goals Today Perspective projection 3D motion Wed Projects Friday Regularization and robust statistics

More information

Optical flow and tracking

Optical flow and tracking EECS 442 Computer vision Optical flow and tracking Intro Optical flow and feature tracking Lucas-Kanade algorithm Motion segmentation Segments of this lectures are courtesy of Profs S. Lazebnik S. Seitz,

More information

Image-Based Rendering and Modeling. IBR Approaches for View Synthesis

Image-Based Rendering and Modeling. IBR Approaches for View Synthesis Image-Based Rendering and Modeling l Image-based rendering (IBR): A scene is represented as a collection of images l 3D model-based rendering (MBR): A scene is represented by a 3D model plus texture maps

More information

Feature Tracking and Optical Flow

Feature Tracking and Optical Flow Feature Tracking and Optical Flow Prof. D. Stricker Doz. G. Bleser Many slides adapted from James Hays, Derek Hoeim, Lana Lazebnik, Silvio Saverse, who in turn adapted slides from Steve Seitz, Rick Szeliski,

More information

Lecture 16: Computer Vision

Lecture 16: Computer Vision CS4442/9542b: Artificial Intelligence II Prof. Olga Veksler Lecture 16: Computer Vision Motion Slides are from Steve Seitz (UW), David Jacobs (UMD) Outline Motion Estimation Motion Field Optical Flow Field

More information

Image Stitching using Harris Feature Detection

Image Stitching using Harris Feature Detection Image Stitching using Harris Feature Detection Shreyas Mistry 1, Prof. Arpita Patel 2 1M. Tech. Student, Electronics & Communication, Chandubhai S. Patel Institute of Technology 2Assi. Professor, Electronics

More information

arxiv: v1 [cs.cv] 28 Sep 2018

arxiv: v1 [cs.cv] 28 Sep 2018 Camera Pose Estimation from Sequence of Calibrated Images arxiv:1809.11066v1 [cs.cv] 28 Sep 2018 Jacek Komorowski 1 and Przemyslaw Rokita 2 1 Maria Curie-Sklodowska University, Institute of Computer Science,

More information

E27 Computer Vision - Final Project: Creating Panoramas David Nahmias, Dan Spagnolo, Vincent Stigliani Professor Zucker Due 5/10/13

E27 Computer Vision - Final Project: Creating Panoramas David Nahmias, Dan Spagnolo, Vincent Stigliani Professor Zucker Due 5/10/13 E27 Computer Vision - Final Project: Creating Panoramas David Nahmias, Dan Spagnolo, Vincent Stigliani Professor Zucker Due 5/10/13 Sources Brown, M.; Lowe, D.G., "Recognising panoramas," Computer Vision,

More information

Uncertainties: Representation and Propagation & Line Extraction from Range data

Uncertainties: Representation and Propagation & Line Extraction from Range data 41 Uncertainties: Representation and Propagation & Line Extraction from Range data 42 Uncertainty Representation Section 4.1.3 of the book Sensing in the real world is always uncertain How can uncertainty

More information

Feature-Based Image Mosaicing

Feature-Based Image Mosaicing Systems and Computers in Japan, Vol. 31, No. 7, 2000 Translated from Denshi Joho Tsushin Gakkai Ronbunshi, Vol. J82-D-II, No. 10, October 1999, pp. 1581 1589 Feature-Based Image Mosaicing Naoki Chiba and

More information

Feature Tracking and Optical Flow

Feature Tracking and Optical Flow Feature Tracking and Optical Flow Prof. D. Stricker Doz. G. Bleser Many slides adapted from James Hays, Derek Hoeim, Lana Lazebnik, Silvio Saverse, who 1 in turn adapted slides from Steve Seitz, Rick Szeliski,

More information

Step-by-Step Model Buidling

Step-by-Step Model Buidling Step-by-Step Model Buidling Review Feature selection Feature selection Feature correspondence Camera Calibration Euclidean Reconstruction Landing Augmented Reality Vision Based Control Sparse Structure

More information

Broad field that includes low-level operations as well as complex high-level algorithms

Broad field that includes low-level operations as well as complex high-level algorithms Image processing About Broad field that includes low-level operations as well as complex high-level algorithms Low-level image processing Computer vision Computational photography Several procedures and

More information

Image-Based Modeling and Rendering

Image-Based Modeling and Rendering Image-Based Modeling and Rendering Richard Szeliski Microsoft Research IPAM Graduate Summer School: Computer Vision July 26, 2013 How far have we come? Light Fields / Lumigraph - 1996 Richard Szeliski

More information

Visual Odometry. Features, Tracking, Essential Matrix, and RANSAC. Stephan Weiss Computer Vision Group NASA-JPL / CalTech

Visual Odometry. Features, Tracking, Essential Matrix, and RANSAC. Stephan Weiss Computer Vision Group NASA-JPL / CalTech Visual Odometry Features, Tracking, Essential Matrix, and RANSAC Stephan Weiss Computer Vision Group NASA-JPL / CalTech Stephan.Weiss@ieee.org (c) 2013. Government sponsorship acknowledged. Outline The

More information

Scene Modeling for a Single View

Scene Modeling for a Single View on to 3D Scene Modeling for a Single View We want real 3D scene walk-throughs: rotation translation Can we do it from a single photograph? Reading: A. Criminisi, I. Reid and A. Zisserman, Single View Metrology

More information

Introduction to Computer Vision

Introduction to Computer Vision Introduction to Computer Vision Michael J. Black Oct 2009 Motion estimation Goals Motion estimation Affine flow Optimization Large motions Why affine? Monday dense, smooth motion and regularization. Robust

More information

Lucas-Kanade Image Registration Using Camera Parameters

Lucas-Kanade Image Registration Using Camera Parameters Lucas-Kanade Image Registration Using Camera Parameters Sunghyun Cho a, Hojin Cho a, Yu-Wing Tai b, Young Su Moon c, Junguk Cho c, Shihwa Lee c, and Seungyong Lee a a POSTECH, Pohang, Korea b KAIST, Daejeon,

More information

Single-view 3D Reconstruction

Single-view 3D Reconstruction Single-view 3D Reconstruction 10/12/17 Computational Photography Derek Hoiem, University of Illinois Some slides from Alyosha Efros, Steve Seitz Notes about Project 4 (Image-based Lighting) You can work

More information

Prof. Feng Liu. Spring /26/2017

Prof. Feng Liu. Spring /26/2017 Prof. Feng Liu Spring 2017 http://www.cs.pdx.edu/~fliu/courses/cs510/ 04/26/2017 Last Time Re-lighting HDR 2 Today Panorama Overview Feature detection Mid-term project presentation Not real mid-term 6

More information

FAST-MATCH: FAST AFFINE TEMPLATE MATCHING

FAST-MATCH: FAST AFFINE TEMPLATE MATCHING Seminar on Sublinear Time Algorithms FAST-MATCH: FAST AFFINE TEMPLATE MATCHING KORMAN, S., REICHMAN, D., TSUR, G., & AVIDAN, S., 2013 Given by: Shira Faigenbaum-Golovin Tel-Aviv University 27.12.2015 Problem

More information

Overview. Video. Overview 4/7/2008. Optical flow. Why estimate motion? Motion estimation: Optical flow. Motion Magnification Colorization.

Overview. Video. Overview 4/7/2008. Optical flow. Why estimate motion? Motion estimation: Optical flow. Motion Magnification Colorization. Overview Video Optical flow Motion Magnification Colorization Lecture 9 Optical flow Motion Magnification Colorization Overview Optical flow Combination of slides from Rick Szeliski, Steve Seitz, Alyosha

More information

Lecture 16: Computer Vision

Lecture 16: Computer Vision CS442/542b: Artificial ntelligence Prof. Olga Veksler Lecture 16: Computer Vision Motion Slides are from Steve Seitz (UW), David Jacobs (UMD) Outline Motion Estimation Motion Field Optical Flow Field Methods

More information

Single View Geometry. Camera model & Orientation + Position estimation. What am I?

Single View Geometry. Camera model & Orientation + Position estimation. What am I? Single View Geometry Camera model & Orientation + Position estimation What am I? Vanishing point Mapping from 3D to 2D Point & Line Goal: Point Homogeneous coordinates represent coordinates in 2 dimensions

More information

Multiple Views Geometry

Multiple Views Geometry Multiple Views Geometry Subhashis Banerjee Dept. Computer Science and Engineering IIT Delhi email: suban@cse.iitd.ac.in January 2, 28 Epipolar geometry Fundamental geometric relationship between two perspective

More information